[CentOS] Find reason for heavy load

[Noob Centos Admin]

Hi,

> Try blocking the IPs on the router and see if that helps.

Unfortunately the server's in a DC so the router is not under our control.

> You can also run iostat and look at the disk usage which also
> generates load.

I did try iostat and its iowait% did coincide with top's report, which
is basically in the low 1~2%.

However, iostat reports much lower %user and $system compared to top
running at the same time so I'm not quite sure if I can rely on its
figures.

> How many cores does your machine have? Load avg is calculated for a
> single core, so a quad core would reach 100% utilization at a load of
> 4, but high iowaits can generate an artificially high load avg as well
> (and why one sees greater than 100% utilization).

It's a dual core that's why I was getting concerned since loads above
2.0 would imply the system's processing capacity was apparently maxed.
However, load and percentages don't add up.

For example, now I'm seeing
top - 14:04:30 up 171 days,  7:14,  1 user,  load average: 3.33, 3.97, 3.81
Tasks: 246 total,   2 running, 236 sleeping,   0 stopped,   8 zombie
Cpu(s): 13.3%us, 16.0%sy,  0.0%ni, 67.5%id,  3.0%wa,  0.0%hi,  0.2%si,  0.0%st

iostat
Linux 2.6.18-128.1.16.el5xen     12/30/2009
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.28    0.20    1.16    2.38    0.01   92.97


> I really wish load would be broken down as CPU/memory/disk instead of
> the ambiguous load avg, and show network read/write utilization in
> ifconfig.

Totally agreed. All the load number is doing is telling me something
is using up resources somewhere but not a single clue otherwise!
Confusing, frustrating and worrying at the same time :(