[CentOS] Find reason for heavy load

Wed Dec 30 06:24:51 UTC 2009
Ross Walker <rswwalker at gmail.com>

On Dec 30, 2009, at 1:05 AM, Noob Centos Admin  
<centos.admin at gmail.com> wrote:

> Hi,
>
>> Try blocking the IPs on the router and see if that helps.
>
> Unfortunately the server's in a DC so the router is not under our  
> control.

That sucks, oh well.

>> You can also run iostat and look at the disk usage which also
>> generates load.
>
> I did try iostat and its iowait% did coincide with top's report, which
> is basically in the low 1~2%.
>
> However, iostat reports much lower %user and $system compared to top
> running at the same time so I'm not quite sure if I can rely on its
> figures.

Yes, I'm not sure iostat's CPU numbers represent the full CPU  
utilization, or only the CPU utilization for IO.

>> How many cores does your machine have? Load avg is calculated for a
>> single core, so a quad core would reach 100% utilization at a load of
>> 4, but high iowaits can generate an artificially high load avg as  
>> well
>> (and why one sees greater than 100% utilization).
>
> It's a dual core that's why I was getting concerned since loads above
> 2.0 would imply the system's processing capacity was apparently maxed.
> However, load and percentages don't add up.

They never do because of the time scaled averages.

> For example, now I'm seeing
> top - 14:04:30 up 171 days,  7:14,  1 user,  load average: 3.33,  
> 3.97, 3.81
> Tasks: 246 total,   2 running, 236 sleeping,   0 stopped,   8 zombie
> Cpu(s): 13.3%us, 16.0%sy,  0.0%ni, 67.5%id,  3.0%wa,  0.0%hi,   
> 0.2%si,  0.0%st
>
> iostat
> Linux 2.6.18-128.1.16.el5xen     12/30/2009
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           3.28    0.20    1.16    2.38    0.01   92.97
>
>
>> I really wish load would be broken down as CPU/memory/disk instead of
>> the ambiguous load avg, and show network read/write utilization in
>> ifconfig.
>
> Totally agreed. All the load number is doing is telling me something
> is using up resources somewhere but not a single clue otherwise!
> Confusing, frustrating and worrying at the same time :(

Maybe someone could write a command-line utility that outputs the  
system load broken down into CPU/memory/disk/network. Call it  
'sysload' and take the system configuration into account.

Take a look at your iptables setup, make sure the blocked ip rules are  
checked first before any other and drop the packets without any icmp  
(give em a black hole to stare at).

-Ross