[CentOS] strange sar statistics: total CPU 90% in userspace, but CPU 0 and 1 only 1%

Hi Centos Users

The server (ProLiant DL380 G5, Centos 4.3) was not responsible anymore
(SSH), fans rotated at full power and the console was blank. After
restart it seems to be fine.

Now I am analyzing sar statistics (in system logs nothing special,
unusual). The hardware monitoring with HP tools didn't work.

# sar -P ALL
Linux 2.6.9-34.ELsmp (cent061)         01/15/2008

12:00:01 AM       CPU     %user     %nice   %system   %iowait     %idle
12:10:01 AM       all     90.43      0.00      5.74      3.83      0.00
12:10:01 AM         0      0.91      0.00      0.07      0.04     98.98
12:10:01 AM         1      1.92      0.00      0.11      0.08     97.88
12:20:01 AM       all     73.85      0.00     24.23      1.92      0.00
12:20:01 AM         0      0.15      0.00      0.05      0.00     99.80
12:20:01 AM         1      0.17      0.00      0.06      0.01     99.76
12:30:01 AM       all     75.28      0.00     22.47      2.25      0.00
12:30:01 AM         0      0.17      0.00      0.04      0.00     99.78
12:30:01 AM         1      0.17      0.00      0.05      0.01     99.77
12:40:01 AM       all     72.09      0.00     24.81      3.10      0.00
12:40:01 AM         0      0.16      0.00      0.04      0.00     99.80
12:40:01 AM         1      0.15      0.00      0.06      0.01     99.78
12:50:01 AM       all     73.31      0.00     24.06      2.63      0.00
12:50:01 AM         0      0.14      0.00      0.04      0.01     99.81
12:50:01 AM         1      0.19      0.00      0.06      0.00     99.75
01:00:01 AM       all     74.06      0.00     24.06      1.88      0.00
01:00:01 AM         0      0.13      0.00      0.04      0.00     99.82
01:00:01 AM         1      0.19      0.00      0.06      0.00     99.74
01:10:01 AM       all     68.02      0.00     24.37      7.61      0.00
01:10:01 AM         0      0.23      0.00      0.09      0.01     99.67
01:10:01 AM         1      0.21      0.00      0.07      0.04     99.68
Average:          all     81.85      0.00     14.45      3.70      0.00
Average:            0      0.27      0.00      0.05      0.01     99.67
Average:            1      0.43      0.00      0.07      0.02     99.48

10:17:55 AM       LINUX RESTART

10:20:01 AM       CPU     %user     %nice   %system   %iowait     %idle
10:30:01 AM       all      0.32      0.00      0.07      0.40     99.21
10:30:01 AM         0      0.35      0.00      0.07      0.44     99.14
10:30:01 AM         1      0.28      0.00      0.08      0.36     99.29
10:40:01 AM       all      0.25      0.00      0.07      0.21     99.47
10:40:01 AM         0      0.28      0.00      0.08      0.20     99.45
10:40:01 AM         1      0.22      0.00      0.07      0.21     99.50
[...]

How its possible that the total CPU is 90% busy, but CPU 1 + 2 only
1%? The server have 2 processors.

cheers
Simon

-- 
XMPP: sjolle at swissjabber.org