Every so often, one of our servers will go into what I can only describe
as an undefined state: it pings, but there's zero access - you can't ssh
in, and if I go plug a keyboard and monitor into the server itself, you
can see the monitor's live, it's not the "monitor turned off" color, but
there is zero response to the keyboard. The upshot is that I wind up
having to power cycle it.
Well, it just happened again on one of our servers Friday evening, as I
found this morning. Looking at the logs this morning, I see that sar last
shows
10:20:01 PM all 34.38 0.00 8.29 0.00 0.00
57.33
On of my users dropped me an email at 22:45 that it was "off", and the
last things I see in /var/log/messages are one of those annoying
Feb 21 22:26:23 <server> kernel: INFO: task perl:20596 blocked for more
than 120 seconds.
Feb 21 22:26:23 <server> kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
I also see
Feb 21 22:26:23 <server> kernel: perl D ffffffff80158250 0
20596 20557
which, as I just found by googling perl NOTLD, means that this is in a
kernel uninterruptable state
In addition, in the stack trace, some nfs messages
Feb 21 22:26:23 <server> kernel: [<ffffffff886b58d1>]
:nfs:nfs_wait_bit_uninterruptible+0x0/0xd
So, it *appears* to be either an NFS issue, or a NIC issue. The user's
home directory server is CentOS running 6.5, and the server that hung is
5.10. Mount on the formerly hung server, su-d to his account shows merely
nfs, so I'm guessing it's NFS3. Looking at lsmod and /var/log/dmesg, I see
it's running the tg3 NIC driver.
Anyone else seeing this, and if so, any thoughts on the matter? Note that
I've had this on Penguins, which are all Supermicro, and they're using the
igb NIC driver, but the one this past weekend is a Dell, so it's not just
one system.
mark