Hello,
I am using nfsen 1.3.5 (http://nfsen.sourceforge.net/) on CentOS 5.8 x86_64 (running as a VM on a KVM Host).
For those not knowing it, nfsen is using php and perl and it allows graphing and analysis of (Cisco routers) netflow data captured using nfdump.
Often (about once a day) nfsen crashes for unknown reason. For example:
# /data/nfsen/bin/nfsen status NfSen version: 1.3.5 NfSen status: Collector for (pen thi) port 9995 is running [2688]. nfsen daemon: pid: [2690] died for unknown reason.
We are running nfsen by including in /etc/rc.local:
/data/nfsen/bin/nfsen start
Note that nfdump 1.6.4 (used by nfsen) is running flawlessly, never crashing. When restarting:
# /data/nfsen/bin/nfsen start Starting nfcapd:(pen thi) Error( Port: 9995 ): a collector with pid[2688] is already running
Starting nfsend.
nfsen and nfdump were compiled from source.
Note that I haven't had such issues with any other software, except nc (netcat) which would also hang in the same way. After many troubles and unsuccessful efforts to resolve, I was forced to use socat instead; socat has been running flawlessly for months. (This might be a different issue though.)
Could someone please try to identify the source of these problems and suggest a solution?
I've tried to find more details on the problem (see below). I tried to collect some debug info (I'm not an expert) and in the log I see the following about the first failed run (after about twelve hours):
-------------------------------------------------------------------------------------
Apr 20 21:30:29 monserver nfsen[6933]: connection on UNIX socket Apr 20 21:30:29 monserver nfsen[6933]: comm server started: 18240 Apr 20 21:30:39 monserver nfsen[18240]: Failed: Timeout reading from socket! at /data/nfsen/libexec/Nfcomm.pm line 1131. Apr 20 21:30:39 monserver nfsen[6933]: comm child[18240] terminated with no exit value
-------------------------------------------------------------------------------------
When running:
(It seems that the multiple nfsend-comm processes remain from previous runs of nfsen; when nfsen dies "for unknown reason", I run nfsen start and nfsen starts but an nfsend-comm from the previous run remains in the process list.)
# ps axjf PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND ... 1 30577 30577 30577 ? -1 Ss 504 0:07 /data/nfsen/bin/nfsend-comm 1 21259 21259 21259 ? -1 Ss 504 0:01 /data/nfsen/bin/nfsend-comm 1 11816 11816 11816 ? -1 Ss 504 0:00 /data/nfsen/bin/nfsend-comm 1 19817 19817 19817 ? -1 Ss 504 0:01 /data/nfsen/bin/nfsend-comm 1 25822 25822 25822 ? -1 Ss 504 0:04 /data/nfsen/bin/nfsend-comm 1 2131 2131 2131 ? -1 Ss 504 0:03 /data/nfsen/bin/nfsend-comm ... 1 6932 6932 6932 ? -1 Rs 504 2:40 /usr/bin/perl -w /data/nfsen/bin/nfsend 6932 6933 6933 6933 ? -1 Ss 504 0:00 _ /data/nfsen/bin/nfsend-comm
# lsof -p 6932 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nfsend 6932 nfsen cwd DIR 252,0 4096 2 / nfsend 6932 nfsen rtd DIR 252,0 4096 2 / nfsend 6932 nfsen txt REG 252,0 19200 75700 /usr/bin/perl nfsend 6932 nfsen mem REG 252,0 144776 1301763 /lib64/ld-2.5.so nfsend 6932 nfsen mem REG 252,0 1722328 1301781 /lib64/libc-2.5.so nfsend 6932 nfsen mem REG 252,0 23360 1301798 /lib64/libdl-2.5.so nfsend 6932 nfsen mem REG 252,0 615136 1301995 /lib64/libm-2.5.so nfsend 6932 nfsen mem REG 252,0 145872 1301799 /lib64/libpthread-2.5.so nfsend 6932 nfsen mem REG 252,0 85544 1301997 /lib64/libz.so.1.2.3 nfsend 6932 nfsen mem REG 252,0 53448 1301800 /lib64/librt-2.5.so nfsend 6932 nfsen mem REG 252,0 51280 73941 /usr/lib64/libdbi.so.0.0.5 nfsend 6932 nfsen mem REG 252,0 690576 78964 /usr/lib64/librrd.so.4.1.4 nfsend 6932 nfsen mem REG 252,0 1262480 228801 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so nfsend 6932 nfsen mem REG 252,0 12040 72516 /usr/lib64/libXau.so.6.0.0 nfsend 6932 nfsen mem REG 252,0 22032 72673 /usr/lib64/libXdmcp.so.6.0.0 nfsend 6932 nfsen mem REG 252,0 1099816 72985 /usr/lib64/libX11.so.6.2.0 nfsend 6932 nfsen mem REG 252,0 262904 1301946 /lib64/libgobject-2.0.so.0.1200.3 nfsend 6932 nfsen mem REG 252,0 114352 1302051 /lib64/libnsl-2.5.so nfsend 6932 nfsen mem REG 252,0 13520 1301945 /lib64/libgmodule-2.0.so.0.1200.3 nfsend 6932 nfsen mem REG 252,0 92816 1302056 /lib64/libresolv-2.5.so nfsend 6932 nfsen mem REG 252,0 143144 1302011 /lib64/libexpat.so.0.5.0 nfsend 6932 nfsen mem REG 252,0 543824 69252 /usr/lib64/libfreetype.so.6.3.10 nfsend 6932 nfsen mem REG 252,0 38528 73002 /usr/lib64/libXrender.so.1.3.0 nfsend 6932 nfsen mem REG 252,0 212896 69259 /usr/lib64/libfontconfig.so.1.1.0 nfsend 6932 nfsen mem REG 252,0 432992 73825 /usr/lib64/libcairo.so.2.9.2 nfsend 6932 nfsen mem REG 252,0 149344 70222 /usr/lib64/libpng12.so.0.10.0 nfsend 6932 nfsen mem REG 252,0 1297136 76276 /usr/lib64/libxml2.so.2.6.26 nfsend 6932 nfsen mem REG 252,0 39304 74098 /usr/lib64/libpangocairo-1.0.so.0.1400.9 nfsend 6932 nfsen mem REG 252,0 192152 73833 /usr/lib64/libpangoft2-1.0.so.0.1400.9 nfsend 6932 nfsen mem REG 252,0 265536 69216 /usr/lib64/libpango-1.0.so.0.1400.9 nfsend 6932 nfsen mem REG 252,0 18152 1302075 /lib64/libutil-2.5.so nfsend 6932 nfsen mem REG 252,0 48600 1302080 /lib64/libcrypt-2.5.so nfsend 6932 nfsen mem REG 252,0 25920 2701809 /var/cache/fontconfig/beeeeb3dfe132a8a0633a017c99ce0c0-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 56414128 69193 /usr/lib/locale/locale-archive nfsend 6932 nfsen mem REG 252,0 119224 229644 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so nfsend 6932 nfsen mem REG 252,0 21424 229774 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so nfsend 6932 nfsen mem REG 252,0 14840 229057 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so nfsend 6932 nfsen mem REG 252,0 15880 229770 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so nfsend 6932 nfsen mem REG 252,0 31656 229457 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so nfsend 6932 nfsen mem REG 252,0 10800 228792 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so nfsend 6932 nfsen mem REG 252,0 37400 229771 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so nfsend 6932 nfsen mem REG 252,0 97334 619478 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so nfsend 6932 nfsen mem REG 252,0 44016 2701801 /var/cache/fontconfig/e19de935dec46bbf3ed114ee4965548a-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 10136 2701251 /var/cache/fontconfig/e3ead4b767b8819993a6fa3ae306afa9-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 647608 1301942 /lib64/libglib-2.0.so.0.1200.3 nfsend 6932 nfsen mem REG 252,0 13056 228836 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so nfsend 6932 nfsen mem REG 252,0 81800 229709 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so nfsend 6932 nfsen mem REG 252,0 18080 228851 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so nfsend 6932 nfsen mem REG 252,0 28544 2701803 /var/cache/fontconfig/7ddba6133ef499da58de5e8c586d3b75-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 53880 1301825 /lib64/libnss_files-2.5.so nfsend 6932 nfsen mem REG 252,0 66360 2701404 /var/cache/fontconfig/73a61b34dd8ca4d8a159807604ab432f-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 32184 2701806 /var/cache/fontconfig/b79f3aaa7d385a141ab53ec885cc22a8-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 7184 2701804 /var/cache/fontconfig/87f5e051180a7a75f16eb6fe7dbd3749-x86-64.cache-2 nfsend 6932 nfsen mem REG 252,0 11704 358189 /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so nfsend 6932 nfsen mem REG 252,0 237540 618666 /usr/share/fonts/dejavu-lgc/DejaVuLGCSansMono.ttf nfsend 6932 nfsen 0r CHR 1,3 0t0 1204 /dev/null nfsend 6932 nfsen 1w CHR 1,3 0t0 1204 /dev/null nfsend 6932 nfsen 2w CHR 1,3 0t0 1204 /dev/null nfsend 6932 nfsen 3u unix 0xffff8100a7cadc40 0t0 10902452 socket
-------------------------------------------------------------------------------------
I also tried to run gdb at the same time:
# gdb /data/nfsen/bin/nfsend 6932 GNU gdb (GDB) CentOS (7.0.1-42.el5.centos) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or laterhttp://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... "/data/nfsen/bin/nfsend": not in executable format: File format not recognized Attaching to process 6932 Reading symbols from /usr/bin/perl...(no debugging symbols found)...done. Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libutil.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so Reading symbols from /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so...done. Loaded symbols for /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so Reading symbols from /usr/lib64/librrd.so.4...done. Loaded symbols for /usr/lib64/librrd.so.4 Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxml2.so.2 Reading symbols from /usr/lib64/libdbi.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libdbi.so.0 Reading symbols from /usr/lib64/libpng12.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libpng12.so.0 Reading symbols from /usr/lib64/libpangocairo-1.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libpangocairo-1.0.so.0 Reading symbols from /usr/lib64/libpango-1.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libpango-1.0.so.0 Reading symbols from /usr/lib64/libcairo.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcairo.so.2 Reading symbols from /lib64/libgobject-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libgobject-2.0.so.0 Reading symbols from /lib64/libgmodule-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libgmodule-2.0.so.0 Reading symbols from /lib64/libglib-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libglib-2.0.so.0 Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libpangoft2-1.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libpangoft2-1.0.so.0 Reading symbols from /usr/lib64/libfontconfig.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libfontconfig.so.1 Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libfreetype.so.6 Reading symbols from /usr/lib64/libXrender.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXrender.so.1 Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libX11.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libexpat.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libexpat.so.0 Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXau.so.6 Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libXdmcp.so.6 Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so Reading symbols from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff95ffd000 0x000000372a09a8d0 in __nanosleep_nocancel () from /lib64/libc.so.6 (gdb) continue Continuing. Detaching after fork from child process 7558. Detaching after fork from child process 7559. ... <Many identical messages: Detaching after fork from child process xxxxx.> ... Detaching after fork from child process 18133. Detaching after fork from child process 18135.
Program received signal SIGPIPE, Broken pipe. 0x000000372ac0df15 in send () from /lib64/libpthread.so.0
(gdb) backtrace full #0 0x000000372ac0df15 in send () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000372c0cf063 in Perl_pp_send () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #2 0x000000372c08a39e in Perl_runops_standard () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #3 0x000000372c0380ba in perl_run () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so No symbol table info available. #4 0x00000000004017bc in main () No symbol table info available.
-------------------------------------------------------------------------------------
Does that help at all?
If you would like to guide me to further debug, please provide details on what should I do. (e.g. how to install debug symbols for libperl.so and libpthread.so)
Please advise.
Thanks, Nick