I have configured 2x 500G sata HDD as Software RAID1 with three partitions md0,md1 and md2 with md2 as 400+ gigs
Now it is almost 36 hours the status is
cat /proc/mdstat Personalities : [raid1] md0 : active raid1 hdb1[1] hda1[0] 104320 blocks [2/2] [UU] resync=DELAYED
md1 : active raid1 hdb2[1] hda2[0] 4096448 blocks [2/2] [UU] resync=DELAYED
md2 : active raid1 hdb3[1] hda3[0] 484182912 blocks [2/2] [UU] [==========>..........] resync = 51.8% (251168768/484182912) finish=1975. 7min speed=1964K/sec
unused devices: <none>
I have reniced the md2-resync to -10
Q1. Does is take this long? Q2. How to speed it up
Regards and Thanks
Rajagopal
On Thu, May 7, 2009 at 11:17 PM, Rajagopal Swaminathan < raju.rajsand@gmail.com> wrote:
I have configured 2x 500G sata HDD as Software RAID1 with three partitions md0,md1 and md2 with md2 as 400+ gigs
Now it is almost 36 hours the status is
cat /proc/mdstat Personalities : [raid1] md0 : active raid1 hdb1[1] hda1[0] 104320 blocks [2/2] [UU] resync=DELAYED
md1 : active raid1 hdb2[1] hda2[0] 4096448 blocks [2/2] [UU] resync=DELAYED
md2 : active raid1 hdb3[1] hda3[0] 484182912 blocks [2/2] [UU] [==========>..........] resync = 51.8% (251168768/484182912) finish=1975. 7min speed=1964K/sec
unused devices: <none>
I have reniced the md2-resync to -10
Q1. Does is take this long? Q2. How to speed it up
Google: http://www.google.com/search?hl=en&rlz=1B3GGGL_enUS326US326&q=mdadm+...
-- j
Jeremy Rosengren <jeremy.rosengren@...> writes:
Google:
http://www.google.com/search?hl=en&rlz=1B3GGGL_enUS326US326&q=mdadm+... j
/proc/sys/dev/raid/speed_limit_min /proc/sys/dev/raid/speed_limit_max
[root@localhost ~]# cat /proc/sys/dev/raid/speed_limit_min 10000 [root@localhost ~]# cat /proc/sys/dev/raid/speed_limit_max 1000000
Thanks
Rajagopal
Rajagopal Swaminathan wrote:
I have configured 2x 500G sata HDD as Software RAID1 with three partitions md0,md1 and md2 with md2 as 400+ gigs
Now it is almost 36 hours the status is
cat /proc/mdstat Personalities : [raid1] md0 : active raid1 hdb1[1] hda1[0] 104320 blocks [2/2] [UU] resync=DELAYED
md1 : active raid1 hdb2[1] hda2[0] 4096448 blocks [2/2] [UU] resync=DELAYED
md2 : active raid1 hdb3[1] hda3[0] 484182912 blocks [2/2] [UU] [==========>..........] resync = 51.8% (251168768/484182912) finish=1975. 7min speed=1964K/sec
unused devices: <none>
I have reniced the md2-resync to -10
Q1. Does is take this long? Q2. How to speed it up
depends on your disk controllers. whats `iostat -x 5` say the IO on hdb3 and hda3 is doing? (ignore the first sample, its average since boot, wait for the 2nd and later 5 second samples...). iostat is in the sysstat RPM, sadly, not installed by default.
is the md2 filesystem mounted ? it will go faster if there is zero disk activity on the physical drives.
500gb has to be read and written over the SATA channels, which appear to be in IDE mode (hda instead of sda).... figure 60-80MB/sec peak sustainable, lets round that to 50MB/sec, so thats 500000/50 == 10000 seconds if its running at pure wire speed the whole time. if there's -any- disk contention, it will run MUCH slower. if its only running 2MB/sec (as indicated) somethinig is fubar with the disk IO, maybe its in PIO mode, ouch!
This list never ceases to amaze me with very very quick and educating responses
John R Pierce <pierce@...> writes:
Rajagopal Swaminathan wrote:
depends on your disk controllers. whats `iostat -x 5` say the IO on hdb3 and hda3 is doing? (ignore the first sample, its average since
avg-cpu: %user %nice %system %iowait %steal %idle 0.13 0.00 31.98 0.00 0.00 67.89
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util hda 26.40 0.40 4.20 2.20 3660.80 20.80 575.25 2.74 386.97 148.44 95.00 hdb 0.00 24.80 0.00 6.60 0.00 3988.80 604.36 1.43 217.45 143.24 94.54 md2 0.00 0.00 0.00 2.60 0.00 20.80 8.00 0.00 0.00 0.00 0.00 md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 2.60 0.00 20.80 8.00 0.43 163.85 78.08 20.30
I won't claim I understand everyone of the above columns. I am in the process of reading man pages.
is the md2 filesystem mounted ?
Yes. But no users using the system and it is mounted as / (root). So cant unmount it
500gb has to be read and written over the SATA channels, which appear to be in IDE mode (hda instead of sda).... figure 60-80MB/sec peak sustainable, lets round that to 50MB/sec, so thats 500000/50 == 10000 seconds if its running at pure wire speed the whole time. if there's -any- disk contention, it will run MUCH slower. if its only running 2MB/sec (as indicated) somethinig is fubar with the disk IO, maybe its in PIO mode, ouch!
So
Q1. Where should I look to correct its PIO mode hdparm? BIOS? Q2. If I bring down the system, will the array reconstruction start from beginning or from where it left off before reboot?
BTW it is HP ML 110 G5 box with 4 gigs RAM.
Thanks
Rajagopal
Rajagopal Swaminathan wrote:
This list never ceases to amaze me with very very quick and educating responses
John R Pierce <pierce@...> writes:
Rajagopal Swaminathan wrote:
depends on your disk controllers. whats `iostat -x 5` say the IO on hdb3 and hda3 is doing? (ignore the first sample, its average since
avg-cpu: %user %nice %system %iowait %steal %idle 0.13 0.00 31.98 0.00 0.00 67.89
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util hda 26.40 0.40 4.20 2.20 3660.80 20.80 575.25 2.74 386.97 148.44 95.00 hdb 0.00 24.80 0.00 6.60 0.00 3988.80 604.36 1.43 217.45 143.24 94.54 ...
I won't claim I understand everyone of the above columns. I am in the process of reading man pages.
the import wants there are rsec/s and wsec/s which are read/write sectors/sec, and %util which is %utilization. your disks are 100% busy, and one is reading 3600 sector/sec which is 1.8MB/sec and the other is writing that much.
there's no way in h*** a SATA disk should be that slow, my guess is the SATA controller is runnign in IDE PIO mode. you also have 32% CPU utilization, which is real high for just doing this.
Rajagopal Swaminathan wrote:
A critical question repeated as I can't have this going on over the weekend and have to get the system up before next 18 hours
Q2. If I bring down the system, will the array reconstruction start from beginning or from where it left off before reboot?
it resumes from where it left off. it keeps track of the checkpoint locations via the metadata it stores in the last track of the partition (I think thats where it is)
John R Pierce wrote:
Rajagopal Swaminathan wrote:
A critical question repeated as I can't have this going on over the weekend and have to get the system up before next 18 hours
Q2. If I bring down the system, will the array reconstruction start from beginning or from where it left off before reboot?
it resumes from where it left off. it keeps track of the checkpoint locations via the metadata it stores in the last track of the partition (I think thats where it is)
It should only take a couple of hours anyway unless there is a lot of other activity on the partition. Something must be wrong with the controller or drive.
Les Mikesell wrote:
It should only take a couple of hours anyway unless there is a lot of other activity on the partition. Something must be wrong with the controller or drive.
The IOSTAT output earlier showing the drives 100% busy at 1.8MB/sec tells me they are running in ISA PIO mode. there's a issue with the disk controller support and how its configured.
John R Pierce <pierce@...> writes:
Les Mikesell wrote:
It should only take a couple of hours anyway unless there is a lot of other activity on the partition. Something must be wrong with the controller or drive.
The IOSTAT output earlier showing the drives 100% busy at 1.8MB/sec tells me they are running in ISA PIO mode. there's a issue with the disk controller support and how its configured.
[SOLVED]
Thanks all
I powered down the box and changed the SATA mode in BIOS f4rom Auto to Serial ATA. That did the trick
cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU] resync=DELAYED
md1 : active raid1 sdb2[1] sda2[0] 4096448 blocks [2/2] [UU] resync=DELAYED
md2 : active raid1 sdb3[1] sda3[0] 484182912 blocks [2/2] [UU] [=>...................] resync = 5.6% (27346752/484182912) finish=68.6min speed=110970K/sec
unused devices: <none>
======= iostat -x 5 outpur (4th or 5th one)
avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 2.60 0.00 0.00 97.40
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 1752.89 0.00 259.68 0.00 257584.03 0.00 991.93 2.47 9.52 3.85 99.86 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 1752.89 0.00 259.68 0.00 257584.03 0.00 991.93 2.47 9.52 3.85 99.86 sdb 0.00 1752.69 0.00 259.68 0.00 257609.58 992.02 1.51 5.80 3.85 99.86 sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb3 0.00 1752.69 0.00 259.68 0.00 257609.58 992.02 1.51 5.80 3.85 99.86 md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
=========== hdparm -tT output
hdparm -tT /dev/sda
/dev/sda: Timing cached reads: 27444 MB in 1.99 seconds = 13802.85 MB/sec Timing buffered disk reads: 378 MB in 3.01 seconds = 125.67 MB/sec [root@localhost ~]# hdparm -tT /dev/sdb
/dev/sdb: Timing cached reads: 27116 MB in 1.99 seconds = 13636.74 MB/sec Timing buffered disk reads: 398 MB in 3.01 seconds = 132.35 MB/sec [root@localhost ~]#
Thanks again
Regards
Rajagopal
Rajagopal Swaminathan wrote:
I powered down the box and changed the SATA mode in BIOS f4rom Auto to Serial ATA. That did the trick
avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 2.60 0.00 0.00 97.40
and, note its now not only copying the data nearly 100X faster, its 97% idle. I believe your earlier iostat showed like 60% system time or something.
John R Pierce <pierce@...> writes:
Rajagopal Swaminathan wrote:
avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 2.60 0.00 0.00 97.40
and, note its now not only copying the data nearly 100X faster, its 97% idle. I believe your earlier iostat showed like 60% system time or something.
indeed
From earlier post..
avg-cpu: %user %nice %system %iowait %steal %idle 0.13 0.00 31.98 0.00 0.00 67.89
It through with syncing..
Thanks a lot again
Regards
Rajagopal