I have a client running a CentOS 6.0 machine with cPanel. The machine is fully updated with both cPanel (RELEASE) and the OS.
At first, I noticed that after cPanel's dcpumon ran (even once), applications that depend on ps lock up and iowait jumps to around 50%. Load averages start out around 20 when this happens and slowly crawl up into the hundreds. Aside from not being able to run commands like ps and some nrpe scripts, everything still seems to respond fine even with the insanely high load. We've had it online with customers hitting it with a load of 400 waiting for a convenient time to reboot, without complaints.
Clarification: If you run ps, it kills your terminal session. dcpumon, ps, etc, will hang around and you can see them under top (top doesn't seem to be affected.) If you try to kill any of these, (-9, anything) they do not respond. They're indefinitely blocked. They begin producing "processes being blocked for more than 120 seconds errors" in the logs. The server runs for days between this happening without issues and it always seems to be related to dcpumon.
I wrote a script that checks to see if dcpumon exists in crontab and remove it. The script runs every minute. cPanel's automatic updates tend to put it back every once in a while, and it's possible that updates ran and that dcpumon ran before my script could remove it. I see that it removed it last night (it logs removals) but don't know for sure if it ran. It probably did.
It's currently running 2.6.32-71.29.1.el6.x86_64 and I am considering trying vanilla kernel build to see if that corrects the issues. The hardware is HP DL145G3, and we have several other (non-cPanel) servers that are identical running CentOS 6.0 without issue.
Any ideas?
Thank you,
Vreme: 10/27/2011 02:58 PM, James Shupe piše:
It's currently running 2.6.32-71.29.1.el6.x86_64 and I am considering trying vanilla kernel build to see if that corrects the issues. The hardware is HP DL145G3, and we have several other (non-cPanel) servers that are identical running CentOS 6.0 without issue.
Any ideas?
You should first try kernel from CR repository, 2.6.32-131.12.1.el6.x86_64. just run "yum install centos-release-cr -y" and update kernel (and other packages if you are up to it). You can always use "yum history undo XXX" command to revert to packages without CR repo enabled.
ELRepo (www.elrepo.org) is close to having kernel-ml package for 6.x., but they are not there yet. 2 weeks ago Alan Bartlett posted that he is close to creating source and binary rpms. You can query on ELRepo mailing list.
On 10/27/2011 07:58 AM, James Shupe wrote:
I have a client running a CentOS 6.0 machine with cPanel. The machine is fully updated with both cPanel (RELEASE) and the OS.
At first, I noticed that after cPanel's dcpumon ran (even once), applications that depend on ps lock up and iowait jumps to around 50%. Load averages start out around 20 when this happens and slowly crawl up into the hundreds. Aside from not being able to run commands like ps and some nrpe scripts, everything still seems to respond fine even with the insanely high load. We've had it online with customers hitting it with a load of 400 waiting for a convenient time to reboot, without complaints.
Clarification: If you run ps, it kills your terminal session. dcpumon, ps, etc, will hang around and you can see them under top (top doesn't seem to be affected.) If you try to kill any of these, (-9, anything) they do not respond. They're indefinitely blocked. They begin producing "processes being blocked for more than 120 seconds errors" in the logs. The server runs for days between this happening without issues and it always seems to be related to dcpumon.
I wrote a script that checks to see if dcpumon exists in crontab and remove it. The script runs every minute. cPanel's automatic updates tend to put it back every once in a while, and it's possible that updates ran and that dcpumon ran before my script could remove it. I see that it removed it last night (it logs removals) but don't know for sure if it ran. It probably did.
It's currently running 2.6.32-71.29.1.el6.x86_64 and I am considering trying vanilla kernel build to see if that corrects the issues. The hardware is HP DL145G3, and we have several other (non-cPanel) servers that are identical running CentOS 6.0 without issue.
Any ideas?
Since you are using cPanel, open up a trouble ticket with them and have them take a look at it.
They are usually very responsive to problems like this and may have seen this before.
-
James Shupe -
Johnny Hughes -
Ljubomir Ljubojevic