[CentOS-virt] VMs died due to hanging httpd processes
dennisml at conversis.de
Sun Dec 12 09:40:35 EST 2010
about an hour ago two web-serving VMs died at the same time with the
following error on the console:
INFO: task httpd:4304 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
httpd D 00af1f714d1112e2 0 4304 22471 4305 4303 (NOTLB)
ffff88006574bdc8 0000000000000282 00000000000041f8 ffff88006574bea8
000000000000000a ffff88009747b820 ffffffff804f4b00 00000000001a5eee
Monitoring show that in a timeframe of about 3 minutes the load on the
systems shot up to over 400 before they died. Since MaxClients is set to
512 I suspect that the processes had a mass-lockup with each process
constantly causing a load of 1 (similar to what happens when a process
hangs on an NFS mount point). One of the two VMs acts as a NFS server and
exports directories to the other VM (but doesn't mount any external NFS
What is strange is that both system locked up at the same time since they
are running on two different physical hosts. The hosts run Centos 5.3 while
the VMs run Centos 5.5 as PV Xen guests.
Since the call trace looks identical on both cases I wonder if anyone has
an idea what exactly went wrong here?
More information about the CentOS-virt