I have four NFS servers running on Dell hardware (PE2900) under CentOS
5.7, x86_64. The number of NFS clients is about 170.
A few days ago, one of the four, with no apparent changes, stopped
responding to NFS requests for two minutes every half an hour (approx).
Let's call this "the hang". It has been doing this for four days now.
There are no log messages of any kind pertaining to this. The other three
servers are fine, although they are less loaded. Between hangs,
performance is excellent. Load is more or less constant, not peaky.
NFS clients do get the usual "not responding, still trying" message during
a hang.
There are no cron or other jobs that launch every half an hour.
All hardware on the affected server seems to be good. Disk volumes being
served are RAID-5 sets with write-back cache enabled (BBU is good). RAID
controller logs are free of errors.
NFS servers used dual bonded gigabit links in balance-alb mode. Turning
off one interface in the bond made no difference.
Relevant /etc/sysctl.conf parameters:
vm.dirty_ratio = 50
vm.dirty_background_ratio = 1
vm.dirty_expire_centisecs = 1000
vm.dirty_writeback_centisecs = 100
vm.min_free_kbytes = 65536
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.core.netdev_max_backlog = 25000
net.ipv4.tcp_reordering = 127
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_no_metrics_save = 1
The {r,w}mem_{max,default} values are twice what they were previously;
changing these had no effect.
The number of dirty pages is nowhere near the dirty_ratio when the hangs
occur; there may be only 50MB of dirty memory.
A local process on the NFS server is reading from disk at around 40-50
MB/sec on average; this continues unaffected during the hang, as do all
other network services on the host (eg an LDAP server). During the hang
the server seems to be quite snappy in all respects apart from NFS. The
network itself is fine as far as I can tell, and all NFS-related processes
on the server are intact.
NFS mounts on clients are made with UDP or TCP with no difference in
results. A client mount cannot be completed ("timed out") and access to an
already NFS mounted volume stalls during the hang (both automounted and
manual mounts).
NFS block size is 32768 r and w; using 16384 makes no difference.
Tcpdump shows no NFS packets exchanged between client and server during a
hang.
I have not rebooted the affected server yet, but I have restarted NFS
with no change.
Help! I cannot figure out what is wrong, and I cannot find anything amiss.
I'm running out of something but I don't know what it is (except perhaps
brains). Hints, please!
Steve