[CentOS] Find reason for heavy load

Thu Dec 31 12:26:13 UTC 2009
Noob Centos Admin <centos.admin at gmail.com>

Hi,

> Yes, these figures indicate that you are fairly close to being cpu bound.
>
> What kind of filtering are you doing? If you have any connection
> tracking/state related rules set, you will need to be using a fair
> amount of cpu.

Initially, when the load start going up, I had thought the APF
filtering rules were the problem since the Indian fellow is still
hammering away at the server even now. However, I've since taken the
risk of turning off APF and rely on static iptables rules, which adds
up to less than one screenful on SSH.

I also thought it might had to do with exim/spamassassin but making a
few changes to reduce the number of emails that goes to spamd doesn't
seem to be helping much.

In fact as you can see from the stats, load has gone up even further
since. I've been averaging 10+ for the whole working day. At the
moment it's between 6 to 10 when it should be at 0.3 from past months
of logs.

This is despite the fact most of my clients should be out celebrating
New Year's Eve. From weeks of logs, the Indian spammer is also a very
punctual fellow who should have knock off work about 17 minutes ago.
So there shouldn't be any heavy 'known' activities on the server at
this point.

So I'm quite stumped as to what's chewing up the CPU cycles. I am also
starting to worry if the server's been compromised and is now doing
something I don't want it to be.

I'm probably going to shutdown the mail/httpd services after midnight
when the impact is the least and see how the server reacts for a
couple of minutes with everything else cut off.