[CentOS] Software RAID1 Disk I/O

On 1/5/2010 5:44 PM, Matt wrote:
> I just installed CentOS 5.4 64 bit release on a 1.9ghz CPU with 8gB of
> RAM.  It has 2 Western Digital 1.5TB SATA2 drives in RAID1.
>
> [root at server ~]# df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md2              1.4T  1.4G  1.3T   1% /
> /dev/md0               99M   19M   76M  20% /boot
> tmpfs                 4.0G     0  4.0G   0% /dev/shm
> [root at server ~]#
>
> Its barebones right now.  Nothing really running.  I intended to move
> our current email server over to it eventually.  The thing is slow as
> mud due to disk I/O though.  I have no idea whats going on.
>
> Here is a bit of iostat -x output.
>
> [root at server ~]# iostat -x
> Linux 2.6.18-164.9.1.el5 (server.x.us)   01/05/2010
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>             0.13    0.10    0.03   38.23    0.00   61.51
>
> Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda               0.18     0.33  0.04  0.20    27.77     4.22   132.30
>      3.70 15239.91 2560.21  61.91
> sda1              0.00     0.00  0.00  0.00     0.34     0.00   654.72
>      0.01 12358.84 3704.24   0.19
> sda2              0.18     0.00  0.04  0.00    27.30     0.00   742.93
>      0.30 8065.37 1930.35   7.09
> sda3              0.00     0.33  0.01  0.20     0.14     4.22    21.29
>      3.40 16537.41 3008.03  61.52

sda3 is running "hot" with a 62% utilization

> sdb               0.19     0.33  0.06  0.20    28.39     4.22   126.29
>      2.51 9676.49 862.49  22.27
> sdb1              0.00     0.00  0.00  0.00     0.34     0.00   606.29
>      0.00 2202.03 643.13   0.04
> sdb2              0.18     0.00  0.04  0.00    27.30     0.00   724.25
>      0.10 2579.45 745.76   2.81
> sdb3              0.01     0.33  0.02  0.20     0.75     4.22    22.61
>      2.41 10913.16 988.40  21.74

sdb3 is running at 22% utilization

> Does anyone have an idea what I have wrong here?  This is my first
> software RAID install.  Built a number of Centos servers without RAID
> and they have all worked fine.
>

As mentioned by others, atop & lsof are good for figuring out what is 
touching the disk.  Something writing to the disk (wsec/s) and doing it 
fairly evenly while also reading from the 2nd partition.

You could try hdparm (hdparm -tT /dev/sda) and see what speeds you get 
for the individual drives, then compare that to the speed you get off of 
say /dev/md2.  Or if you want a stronger load that lasts for a minute or 
two, try using dd and dumping to /dev/null (dd if=/dev/sda of=/dev/null 
bs=1M count=1000).  The individual drive speeds should match up pretty 
closely with the Software RAID speed.

A pair of 1.5GB 7200RPM SATA drives should be able to handle up to a few 
hundred thousand to a million or so messages per month (Postfix / 
Dovecot).  As long as that's at least a dual-core 1.9GHz CPU.  YMMV of 
course and watching server performance over the long term will be your 
best bet.