Hi gents,
I have a CentOS 6.6 install with net-snmp running. I'm using SNMPv3 Auth and Encryption with Observium polling devices every five minutes. On one of my Centos server, the net-snmp process will die after 24-36 hours of running. It seems to be hitting my system maximum file open limit which is set to 1024.
[root@mail net-snmp-5.7]# ulimit -n 1024
That number will keep increasing till it trips the 1024 limit.
[root@mail ~]# lsof -p 26907 | wc -l 1071
[root@mail net-snmp-5.7]# lsof -p 26907 snmpd 26907 root 166r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 167r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 168r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 169r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 170r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 171r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 172r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 173r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 174r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 175r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 176r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 177r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 178r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp snmpd 26907 root 179r REG 0,3 0 4026532160 /proc/26907/net/sctp/snmp
#tail /var/log/snmpd.log cannot open /proc/net/dev ... /proc/partitions: Too many open files /proc/stat: Too many open files /proc/partitions: Too many open files /proc/stat: Too many open files could not create socket /proc/partitions: Too many open files /proc/stat: Too many open files cannot open /proc/net/dev ... /proc/partitions: Too many open files /proc/stat: Too many open files /proc/partitions: Too many open files /proc/stat: Too many open files
It seems that CentOS is frozen to net-snmp 5.5 and I'm don't really want to use anything that I can't easily manage with "yum". The weird thing is that I have another server with the same config/CentOS version/Net-Snmp version and I have no issues with the snmp process on that server. I could raise the open file limit but that just seems unnecessary since I don't think that snmp is supposed to work that way.
Anything else I could try to troubleshoot the issue?
Thanks!