[CentOS] CentOS 6.6 - net-snmp process - Too many open files

Wed Dec 31 07:10:03 UTC 2014
Laurent Dumont <admin at coldnorthadmin.com>

Hi gents,

I have a CentOS 6.6 install with net-snmp running. I'm using SNMPv3 Auth 
and Encryption with Observium polling devices every five minutes. On one 
of my Centos server, the net-snmp process will die after 24-36 hours of 
running. It seems to be hitting my system maximum file open limit which 
is set to 1024.

> [root at mail net-snmp-5.7]# ulimit -n
> 1024

That number will keep increasing till it trips the 1024 limit.

> [root at mail ~]# lsof -p 26907 | wc -l
> 1071

> [root at mail net-snmp-5.7]# lsof -p 26907
> snmpd   26907 root  166r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  167r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  168r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  169r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  170r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  171r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  172r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  173r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  174r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  175r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  176r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  177r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  178r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp
> snmpd   26907 root  179r   REG      0,3        0 4026532160 
> /proc/26907/net/sctp/snmp

> #tail /var/log/snmpd.log
> cannot open /proc/net/dev ...
> /proc/partitions: Too many open files
> /proc/stat: Too many open files
> /proc/partitions: Too many open files
> /proc/stat: Too many open files
> could not create socket
> /proc/partitions: Too many open files
> /proc/stat: Too many open files
> cannot open /proc/net/dev ...
> /proc/partitions: Too many open files
> /proc/stat: Too many open files
> /proc/partitions: Too many open files
> /proc/stat: Too many open files

It seems that CentOS is frozen to net-snmp 5.5 and I'm don't really want 
to use anything that I can't easily manage with "yum". The weird thing 
is that I have another server with the same config/CentOS 
version/Net-Snmp version and I have no issues with the snmp process on 
that server. I could raise the open file limit but that just seems 
unnecessary since I don't think that snmp is supposed to work that way.

Anything else I could try to troubleshoot the issue?

Thanks!