[CentOS] ps locking up

Thu Oct 27 12:58:06 UTC 2011
James Shupe <jshupe at osre.org>

I have a client running a CentOS 6.0 machine with cPanel. The machine is
fully updated with both cPanel (RELEASE) and the OS.

At first, I noticed that after cPanel's dcpumon ran (even once),
applications that depend on ps lock up and iowait jumps to around 50%.
Load averages start out around 20 when this happens and slowly crawl up
into the hundreds. Aside from not being able to run commands like ps and
some nrpe scripts, everything still seems to respond fine even with the
insanely high load. We've had it online with customers hitting it with a
load of 400 waiting for a convenient time to reboot, without complaints.

Clarification: If you run ps, it kills your terminal session. dcpumon,
ps, etc, will hang around and you can see them under top (top doesn't
seem to be affected.) If you try to kill any of these, (-9, anything)
they do not respond. They're indefinitely blocked. They begin producing
"processes being blocked for more than 120 seconds errors" in the logs.
The server runs for days between this happening without issues and it
always seems to be related to dcpumon.

I wrote a script that checks to see if dcpumon exists in crontab and
remove it. The script runs every minute. cPanel's automatic updates tend
to put it back every once in a while, and it's possible that updates ran
and that dcpumon ran before my script could remove it. I see that it
removed it last night (it logs removals) but don't know for sure if it
ran. It probably did.

It's currently running 2.6.32-71.29.1.el6.x86_64 and I am considering
trying vanilla kernel build to see if that corrects the issues. The
hardware is HP DL145G3, and we have several other (non-cPanel) servers
that are identical running CentOS 6.0 without issue.

Any ideas?

Thank you,
-- 
James Shupe, OSRE
developer/ engineer
BSD/ Linux support & hosting
jshupe at osre.org | www.osre.org
O 9032530140 | F 9032530150 | M 9035223425
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.centos.org/pipermail/centos/attachments/20111027/dcca41c7/attachment-0004.sig>