[CentOS] Find reason for heavy load

Thu Dec 31 15:28:38 UTC 2009
Chan Chung Hang Christopher <christopher.chan at bradbury.edu.hk>

Noob Centos Admin wrote:
> Hi,
> 
>> Yes, these figures indicate that you are fairly close to being cpu bound.
>>
>> What kind of filtering are you doing? If you have any connection
>> tracking/state related rules set, you will need to be using a fair
>> amount of cpu.
> 
> Initially, when the load start going up, I had thought the APF
> filtering rules were the problem since the Indian fellow is still
> hammering away at the server even now. However, I've since taken the
> risk of turning off APF and rely on static iptables rules, which adds
> up to less than one screenful on SSH.

I do not know about now but I had to unload the modules in question. 
Just clearing the rules was not enough to ensure that the netfilter 
connection tracking modules were not using any cpu at all.

> 
> I also thought it might had to do with exim/spamassassin but making a
> few changes to reduce the number of emails that goes to spamd doesn't
> seem to be helping much.
> 
> In fact as you can see from the stats, load has gone up even further
> since. I've been averaging 10+ for the whole working day. At the
> moment it's between 6 to 10 when it should be at 0.3 from past months
> of logs.
> 
> This is despite the fact most of my clients should be out celebrating
> New Year's Eve. From weeks of logs, the Indian spammer is also a very
> punctual fellow who should have knock off work about 17 minutes ago.
> So there shouldn't be any heavy 'known' activities on the server at
> this point.

/me shrugs. When I was the mta admin at Outblaze Ltd. (messaging 
business now owned by IBM and called Lotus Live) spammers always ensured 
I got called. All they do is just press the big red button (aka start 
the script/system) and then go and play while I would have to deal with 
whatever was started. I remember only one occasion when the spams were 
launched but neutralized very soon because they were pushing a website 
and I found a sample real early and so the anti spam system could just 
dump the spams and knock out accounts being used to send the crap.

> 
> So I'm quite stumped as to what's chewing up the CPU cycles. I am also
> starting to worry if the server's been compromised and is now doing
> something I don't want it to be.
> 
> I'm probably going to shutdown the mail/httpd services after midnight
> when the impact is the least and see how the server reacts for a
> couple of minutes with everything else cut off.

First, try rmmod'ing the netfilter modules after you have cleared away 
the state related rules to make sure that you are only using static 
rules in netfilter...unless you have done that already..