[CentOS] Weird output of system load

Thu Sep 11 15:05:33 UTC 2014
Mingfei Hua <mfhua at aerohive.com>

This issue happened again, who can help on this? system load up to 72, but /proc/stat only show that 2 processes were running and no process blocked. could it be a bug?

/proc/stat
cpu  5584507 61716 7213361 166019596 244884 146923 396136 0 0
cpu0 4186208 9686 2596560 81685397 112653 146699 365382 0 0
cpu1 1398298 52029 4616801 84334199 132230 224 30753 0 0
intr 1121292737 139 8 0 0 0 0 0 0 0 0 0 0 106 0 0 73 0 0 550904491 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4022511 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 1114886293
btime 1409530629
processes 10263085
procs_running 2
procs_blocked 0
softirq 1469123445 0 380686538 182314 632630211 3763285 0 2 56074856 1852476 393933763
/proc/loadavg
72.27 33.47 12.45 3/1906 13601


On Sep 11, 2014, at 2:54 PM, Mingfei Hua <mfhua at aerohive.com<mailto:mfhua at aerohive.com>> wrote:

All,

One server of mine running CentOS 6.3. The load is very high, but few process in running or blocked.
Followed is the output of /proc/stat, /proc/loadavg and top
/proc/stat
cpu  5351723 61716 6974590 161365578 240734 141769 380525 0 0
cpu0 4016881 9686 2510787 79408769 110721 141549 351075 0 0
cpu1 1334842 52029 4463802 81956808 130013 220 29449 0 0
intr 1081538370 139 8 0 0 0 0 0 0 0 0 0 0 106 0 0 73 0 0 531522246 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3929190 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 1074666354
btime 1409530629
processes 9684115
procs_running 1
procs_blocked 0
softirq 1418158114 0 367862840 177073 609942882 3679591 0 2 53962455 1790447 380742824
/proc/loadavg
11.21 22.50 10.74 1/2536 16895
Top –H
top - 05:08:02 up 10 days,  4:50,  1 user,  load average: 13.24, 23.26, 10.86
Tasks: 2535 total,   1 running, 2534 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.1%us,  4.0%sy,  0.0%ni, 92.5%id,  0.1%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   3925040k total,  3780944k used,   144096k free,    31464k buffers
Swap:  2064376k total,     1196k used,  2063180k free,  3201568k cached


Sincerely,
Mingfei Hua