[CentOS] nfsen daemon dies for unknown reason

Hello,

I am using nfsen 1.3.5 (http://nfsen.sourceforge.net/) on CentOS 5.8 x86_64 (running as a VM on a KVM Host).

For those not knowing it, nfsen is using php and perl and it allows graphing and analysis of (Cisco routers) netflow data captured using nfdump.

Often (about once a day) nfsen crashes for unknown reason. For example:

     # /data/nfsen/bin/nfsen status
     NfSen version: 1.3.5
     NfSen status:
     Collector for (pen thi) port 9995 is running [2688].
     nfsen daemon:  pid: [2690] died for unknown reason.

We are running nfsen by including in /etc/rc.local:

     /data/nfsen/bin/nfsen start

Note that nfdump 1.6.4 (used by nfsen) is running flawlessly, never crashing. When restarting:

     # /data/nfsen/bin/nfsen start
     Starting nfcapd:(pen thi)
     Error( Port: 9995 ): a collector with pid[2688] is already running

     Starting nfsend.

nfsen and nfdump were compiled from source.

Note that I haven't had such issues with any other software, except nc (netcat) which would also hang in the same way. After many troubles and unsuccessful efforts to resolve, I was forced to use socat instead; socat has been running flawlessly for months. (This might be a different issue though.)

Could someone please try to identify the source of these problems and suggest a solution?

I've tried to find more details on the problem (see below). I tried to collect some debug info (I'm not an expert) and in the log I see the following about the first failed run (after about twelve hours):

-------------------------------------------------------------------------------------

Apr 20 21:30:29 monserver nfsen[6933]: connection on UNIX socket
Apr 20 21:30:29 monserver nfsen[6933]: comm server started: 18240
Apr 20 21:30:39 monserver nfsen[18240]: Failed: Timeout reading from socket! at /data/nfsen/libexec/Nfcomm.pm line 1131.
Apr 20 21:30:39 monserver nfsen[6933]: comm child[18240] terminated with no exit value

-------------------------------------------------------------------------------------

When running:

(It seems that the multiple nfsend-comm processes remain from previous runs of nfsen; when nfsen dies "for unknown reason", I run nfsen start and nfsen starts but an nfsend-comm from the previous run remains in the process list.)

# ps axjf
  PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
...
     1 30577 30577 30577 ?           -1 Ss     504   0:07 /data/nfsen/bin/nfsend-comm
     1 21259 21259 21259 ?           -1 Ss     504   0:01 /data/nfsen/bin/nfsend-comm
     1 11816 11816 11816 ?           -1 Ss     504   0:00 /data/nfsen/bin/nfsend-comm
     1 19817 19817 19817 ?           -1 Ss     504   0:01 /data/nfsen/bin/nfsend-comm
     1 25822 25822 25822 ?           -1 Ss     504   0:04 /data/nfsen/bin/nfsend-comm
     1  2131  2131  2131 ?           -1 Ss     504   0:03 /data/nfsen/bin/nfsend-comm
...
     1  6932  6932  6932 ?           -1 Rs     504   2:40 /usr/bin/perl -w /data/nfsen/bin/nfsend
  6932  6933  6933  6933 ?           -1 Ss     504   0:00  \_ /data/nfsen/bin/nfsend-comm


# lsof -p 6932
COMMAND  PID  USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
nfsend  6932 nfsen  cwd    DIR              252,0     4096        2 /
nfsend  6932 nfsen  rtd    DIR              252,0     4096        2 /
nfsend  6932 nfsen  txt    REG              252,0    19200    75700 /usr/bin/perl
nfsend  6932 nfsen  mem    REG              252,0   144776  1301763 /lib64/ld-2.5.so
nfsend  6932 nfsen  mem    REG              252,0  1722328  1301781 /lib64/libc-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    23360  1301798 /lib64/libdl-2.5.so
nfsend  6932 nfsen  mem    REG              252,0   615136  1301995 /lib64/libm-2.5.so
nfsend  6932 nfsen  mem    REG              252,0   145872  1301799 /lib64/libpthread-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    85544  1301997 /lib64/libz.so.1.2.3
nfsend  6932 nfsen  mem    REG              252,0    53448  1301800 /lib64/librt-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    51280    73941 /usr/lib64/libdbi.so.0.0.5
nfsend  6932 nfsen  mem    REG              252,0   690576    78964 /usr/lib64/librrd.so.4.1.4
nfsend  6932 nfsen  mem    REG              252,0  1262480   228801 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
nfsend  6932 nfsen  mem    REG              252,0    12040    72516 /usr/lib64/libXau.so.6.0.0
nfsend  6932 nfsen  mem    REG              252,0    22032    72673 /usr/lib64/libXdmcp.so.6.0.0
nfsend  6932 nfsen  mem    REG              252,0  1099816    72985 /usr/lib64/libX11.so.6.2.0
nfsend  6932 nfsen  mem    REG              252,0   262904  1301946 /lib64/libgobject-2.0.so.0.1200.3
nfsend  6932 nfsen  mem    REG              252,0   114352  1302051 /lib64/libnsl-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    13520  1301945 /lib64/libgmodule-2.0.so.0.1200.3
nfsend  6932 nfsen  mem    REG              252,0    92816  1302056 /lib64/libresolv-2.5.so
nfsend  6932 nfsen  mem    REG              252,0   143144  1302011 /lib64/libexpat.so.0.5.0
nfsend  6932 nfsen  mem    REG              252,0   543824    69252 /usr/lib64/libfreetype.so.6.3.10
nfsend  6932 nfsen  mem    REG              252,0    38528    73002 /usr/lib64/libXrender.so.1.3.0
nfsend  6932 nfsen  mem    REG              252,0   212896    69259 /usr/lib64/libfontconfig.so.1.1.0
nfsend  6932 nfsen  mem    REG              252,0   432992    73825 /usr/lib64/libcairo.so.2.9.2
nfsend  6932 nfsen  mem    REG              252,0   149344    70222 /usr/lib64/libpng12.so.0.10.0
nfsend  6932 nfsen  mem    REG              252,0  1297136    76276 /usr/lib64/libxml2.so.2.6.26
nfsend  6932 nfsen  mem    REG              252,0    39304    74098 /usr/lib64/libpangocairo-1.0.so.0.1400.9
nfsend  6932 nfsen  mem    REG              252,0   192152    73833 /usr/lib64/libpangoft2-1.0.so.0.1400.9
nfsend  6932 nfsen  mem    REG              252,0   265536    69216 /usr/lib64/libpango-1.0.so.0.1400.9
nfsend  6932 nfsen  mem    REG              252,0    18152  1302075 /lib64/libutil-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    48600  1302080 /lib64/libcrypt-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    25920  2701809 /var/cache/fontconfig/beeeeb3dfe132a8a0633a017c99ce0c0-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0 56414128    69193 /usr/lib/locale/locale-archive
nfsend  6932 nfsen  mem    REG              252,0   119224   229644 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
nfsend  6932 nfsen  mem    REG              252,0    21424   229774 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
nfsend  6932 nfsen  mem    REG              252,0    14840   229057 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
nfsend  6932 nfsen  mem    REG              252,0    15880   229770 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so
nfsend  6932 nfsen  mem    REG              252,0    31656   229457 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
nfsend  6932 nfsen  mem    REG              252,0    10800   228792 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
nfsend  6932 nfsen  mem    REG              252,0    37400   229771 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
nfsend  6932 nfsen  mem    REG              252,0    97334   619478 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so
nfsend  6932 nfsen  mem    REG              252,0    44016  2701801 /var/cache/fontconfig/e19de935dec46bbf3ed114ee4965548a-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    10136  2701251 /var/cache/fontconfig/e3ead4b767b8819993a6fa3ae306afa9-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0   647608  1301942 /lib64/libglib-2.0.so.0.1200.3
nfsend  6932 nfsen  mem    REG              252,0    13056   228836 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
nfsend  6932 nfsen  mem    REG              252,0    81800   229709 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
nfsend  6932 nfsen  mem    REG              252,0    18080   228851 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
nfsend  6932 nfsen  mem    REG              252,0    28544  2701803 /var/cache/fontconfig/7ddba6133ef499da58de5e8c586d3b75-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    53880  1301825 /lib64/libnss_files-2.5.so
nfsend  6932 nfsen  mem    REG              252,0    66360  2701404 /var/cache/fontconfig/73a61b34dd8ca4d8a159807604ab432f-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    32184  2701806 /var/cache/fontconfig/b79f3aaa7d385a141ab53ec885cc22a8-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0     7184  2701804 /var/cache/fontconfig/87f5e051180a7a75f16eb6fe7dbd3749-x86-64.cache-2
nfsend  6932 nfsen  mem    REG              252,0    11704   358189 /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so
nfsend  6932 nfsen  mem    REG              252,0   237540   618666 /usr/share/fonts/dejavu-lgc/DejaVuLGCSansMono.ttf
nfsend  6932 nfsen    0r   CHR                1,3      0t0     1204 /dev/null
nfsend  6932 nfsen    1w   CHR                1,3      0t0     1204 /dev/null
nfsend  6932 nfsen    2w   CHR                1,3      0t0     1204 /dev/null
nfsend  6932 nfsen    3u  unix 0xffff8100a7cadc40      0t0 10902452 socket

-------------------------------------------------------------------------------------

I also tried to run gdb at the same time:

# gdb /data/nfsen/bin/nfsend 6932
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
"/data/nfsen/bin/nfsend": not in executable format: File format not recognized
Attaching to process 6932
Reading symbols from /usr/bin/perl...(no debugging symbols found)...done.
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so...done.
Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so
Reading symbols from /usr/lib64/librrd.so.4...done.
Loaded symbols for /usr/lib64/librrd.so.4
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libdbi.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libdbi.so.0
Reading symbols from /usr/lib64/libpng12.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libpangocairo-1.0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpangocairo-1.0.so.0
Reading symbols from /usr/lib64/libpango-1.0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpango-1.0.so.0
Reading symbols from /usr/lib64/libcairo.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcairo.so.2
Reading symbols from /lib64/libgobject-2.0.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgobject-2.0.so.0
Reading symbols from /lib64/libgmodule-2.0.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgmodule-2.0.so.0
Reading symbols from /lib64/libglib-2.0.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libglib-2.0.so.0
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libpangoft2-1.0.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpangoft2-1.0.so.0
Reading symbols from /usr/lib64/libfontconfig.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfontconfig.so.1
Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libXrender.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXrender.so.1
Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libexpat.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libexpat.so.0
Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff95ffd000
0x000000372a09a8d0 in __nanosleep_nocancel () from /lib64/libc.so.6
(gdb) continue
Continuing.
Detaching after fork from child process 7558.
Detaching after fork from child process 7559.
...
<Many identical messages: Detaching after fork from child process xxxxx.>
...
Detaching after fork from child process 18133.
Detaching after fork from child process 18135.

Program received signal SIGPIPE, Broken pipe.
0x000000372ac0df15 in send () from /lib64/libpthread.so.0

(gdb) backtrace full
#0  0x000000372ac0df15 in send () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x000000372c0cf063 in Perl_pp_send () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#2  0x000000372c08a39e in Perl_runops_standard () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#3  0x000000372c0380ba in perl_run () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#4  0x00000000004017bc in main ()
No symbol table info available.

-------------------------------------------------------------------------------------

Does that help at all?

If you would like to guide me to further debug, please provide details on what should I do. (e.g. how to install debug symbols for libperl.so and libpthread.so)

Please advise.

Thanks,
Nick