I'm trying to figure out what's causing an average system load of 3+ to 5+ on an Intel quad core. The server has with 2 KVM guests (assigned 1 core and 2 cores) that's lightly loaded (0.1~0.4) each. Both guest/host are running 64bit CentOS 5.6
Originally I suspected maybe it's i/o but on checking, there is very little i/o wait % as well. Plenty of free disk space available on all physical drives and memory sufficient for the usage with barely any swap in use.
While performance/responsiveness of the host and guests doesn't seem to be affected, I'm still concerned about this odd load figures. Would appreciate it if anybody can suggest what else I should be looking at?
Output of various commands on the host:
Top === top - 15:15:39 up 5 days, 8:48, 1 user, load average: 4.76, 4.18, 4.43 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie Cpu(s): 4.1%us, 2.5%sy, 0.0%ni, 76.4%id, 17.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3759496k total, 2813076k used, 946420k free, 357300k buffers Swap: 8193016k total, 5648k used, 8187368k free, 736640k cached
free -m ====== total used free shared buffers cached Mem: 3671 2750 921 0 348 719 -/+ buffers/cache: 1682 1989 Swap: 8000 5 7995
vmstat 3 ====== procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 5648 949072 357096 735788 0 0 105 50 17 22 2 1 95 1 0 0 0 5648 949428 357104 735788 0 0 44 2682 14142 14697 6 4 84 6 0 0 0 5648 949304 357104 735788 0 0 1 324 14047 14753 2 1 96 1 0 1 1 5648 946916 357104 735788 0 0 0 215 14410 14496 2 2 90 6 0 0 0 5648 946520 357104 735788 0 0 23 267 13703 14664 2 2 91 6 0
sar === 02:40:01 PM CPU %user %nice %system %iowait %steal %idle 02:50:01 PM all 2.17 0.00 2.18 4.30 0.00 91.35 03:00:01 PM all 2.47 0.00 2.23 3.57 0.00 91.73 03:10:01 PM all 2.29 0.00 2.07 3.77 0.00 91.87 03:20:01 PM all 2.63 0.00 2.07 3.28 0.00 92.03 Average: all 2.39 0.00 1.95 4.77 0.00 90.89