[CentOS] Slow performance

Mon Apr 30 21:33:22 UTC 2007
Ashley M. Kirchner <ashley at pcraft.com>

Scott Lamb wrote:
> I think your 9% CPU usage is misleading. It probably means you're 
> using one out of eight processors for 72% of your eight minutes. Add 
> in the iowait time (which should be similar to the time the copy took 
> - 11% of your eight minutes) and you're not far off from explaining 
> the whole time.
>
> In top on Linux, you can hit 'I' to make the CPU percentages go up to 
> 800% instead of 100%. Or '1' to show each processor individually.
    Well, that would make sense...

Cpu0  : 20.2%us,  7.1%sy,  0.0%ni, 72.7%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu2  :  0.0%us,  1.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu4  :  0.0%us,  1.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu5  : 52.5%us, 13.9%sy,  0.0%ni, 28.7%id,  5.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  
0.0%st
Mem:   2074932k total,   745108k used,  1329824k free,    26328k buffers
Swap:  2096440k total,        0k used,  2096440k free,   547884k cached

    And iostat -x 5 shows:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.95    0.00    2.90    0.87    0.00   87.28

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz 
avgqu-sz   await  svctm  %util
sda               0.00     4.39  0.00  0.40     0.00    38.32    
96.00     0.00    0.00   0.00   0.00
sdb             214.77     3.39 22.55  1.40  1898.60    44.71    
81.13     0.05    2.03   1.71   4.09
sdc             207.58     6.19 18.96  2.00  1812.38    71.86    
89.90     0.04    1.83   1.47   3.07
sdd             209.78     0.80 21.16  1.00  1847.50    20.76    
84.32     0.05    2.31   1.70   3.77
sde             214.57     2.00 19.36  1.20  1871.46    31.94    
92.58     0.03    1.60   1.52   3.13
md0               0.00     0.00  0.00  2.79     0.00    22.36     
8.00     0.00    0.00   0.00   0.00
md9               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md8               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md7               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md6               0.00     0.00 73.05  4.59  7356.49    36.73    
95.22     0.00    0.00   0.00   0.00
md5               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md4               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md3               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md1               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00 73.05  7.39  7356.49    59.08    
92.19     0.21    2.59   0.99   7.98
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00


    And when the process is done, some 8 and a half minutes later, 
iostat says:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.02    0.00    0.05    0.00    0.00   99.93

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz 
avgqu-sz   await  svctm  %util
sda               0.00     4.60  0.00  0.40     0.00    40.00   
100.00     0.00    4.50   4.50   0.18
sdb               0.00     0.00  0.20  1.00     1.60    14.40    
13.33     0.03   24.33  19.17   2.30
sdc               0.00     1.80  0.00  1.40     0.00    32.00    
22.86     0.03   13.57  16.86   2.36
sdd               0.00     0.00  0.20  1.00     1.60    14.40    
13.33     0.02   24.00  19.00   2.28
sde               0.00     1.80  0.00  1.40     0.00    33.60    
24.00     0.04   17.71  21.00   2.94
md0               0.00     0.00  0.00  2.60     0.00    20.80     
8.00     0.00    0.00   0.00   0.00
md9               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md8               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md7               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md6               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md5               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md4               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md3               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
md1               0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00  0.00  2.60     0.00    20.80     
8.00     0.03    6.23   7.31   1.90
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00


    So, anyone going to make sense out of that please?

-- 
W | It's not a bug - it's an undocumented feature.
  +--------------------------------------------------------------------
  Ashley M. Kirchner <mailto:ashley at pcraft.com>   .   303.442.6410 x130
  IT Director / SysAdmin / Websmith             .     800.441.3873 x130
  Photo Craft Imaging                       .     3550 Arapahoe Ave. #6
  http://www.pcraft.com ..... .  .    .       Boulder, CO 80303, U.S.A.