[CentOS] High system load but low cpu usage

[Emmanuel Noobadmin]

On 6/9/11, Steven Tardy <sjt5 at its.msstate.edu> wrote:
> top Cpu(s) line is averaged for all cpus/cores. to display individual
> cpus/cores press:
>     1
> you'll likely see one cpu/core being pegged with iowait.
> to identify the offending process within top press:
>     fj<enter>
> to display the P column(last used CPU).
> watch top for a few minutes to see what is using all of the disk io.

Thanks for these tips, it really helped narrow down the issue. Became
quite clear that cpu 0 was taking up most of the user and sys time,
somewhere in the 10x compared to the other 3.

Based on the VM memory usage, I think I know which VM it is but I'm
going to start pinning it to confirm it's the culprit.

> sar output is averaged over the 10 minute interval.
> for smaller sar time slices edit cron file: /etc/cron.d/sysstat
>
> disks are often swamped by "two things happening at once"...
>    backups
>    migrating a VM
>    database upgrades
>    .rrd average updates

Unfortunately, the VMs are public facing and the offending one has got
a relatively popular Wordpress blog as well as relatively high email
traffic. So it's likely the result of those two things happening at
once. I'm increasing the memory allocation on it and hope maybe more
of the Wordpress content gets cached and see if it helps.

The odd thing is I set the VM to 512MB but a max of 1.5G assuming that
KVM will assign the extra memory as needed but it seems to be stuck at
512MB.