Is anyone using RAID6 in production? In moving from hardware RAID on my dual 3ware 7500-8 based systems to md, I decided I'd like to go with RAID6 (since md is less tolerant of marginal drives than is 3ware). I did some benchmarking and was getting decent speeds with a 128KiB chunksize.
So the next step was failure testing. First, I fired off memtest.sh as found at http://people.redhat.com/dledford/memtest.html. Then, I did 'mdadm /dev/md0 -f /dev/sdo1', and it started to rebuild as it should. I cranked up /proc/sys/dev/raid/speed_limit_min to 15000 so that it would reconstruct in a decent amount of time (the default of 1000 was leading to a 53 hour estimate for the recovery).
But memtest.sh started kicking out errors (non-matching diffs). And then I got this:
EXT3-fs error (device md0): ext3_journal_start_sb: Detected aborted journal Remounting filesystem read-only attempt to access beyond end of device md0: rw=0, want=28987566088, limit=4595422208 attempt to access beyond end of device md0: rw=0, want=28987566088, limit=4595422208 attempt to access beyond end of device md0: rw=0, want=28987566088, limit=4595422208
Needless to say it's not giving me that warm fuzzy feeling. The one caveat is that not all the members of my array were the same size -- one disk is 180GB while all the rest are 160GB. I'm going to test overnight with identically sized RAID members, but I also wanted to see if anyone else is using RAID6.
Thanks.
Quoting Joshua Baker-LePain jlb17@duke.edu:
Needless to say it's not giving me that warm fuzzy feeling. The one caveat is that not all the members of my array were the same size -- one disk is 180GB while all the rest are 160GB. I'm going to test overnight with identically sized RAID members, but I also wanted to see if anyone else is using RAID6.
I was testing RAID-5 with identical disk drives. Got the same thing. If I attempt to access it while it rebuilds, sooner or later I get errors. If I reboot while it rebuilds, it doesn't start it after the reboot (forgot the actuall error message). I needed to manually kick it again using mdadm to force-start it. If it was root file system, the machine would probably fail to boot completely. You might want to stick to your 3ware for now.
BTW, any particular reason not to use hardware RAID on 3ware?
I was testing software RAID-5 only because I had an old and slow Adaptec 2400 RAID controller. Software RAID-5 (on an older 1.8GHz CPU) across 4 drives connected to it in JBOD config was about 60% faster than hardware RAID-5. But as I said, because of the described problems, I decided to stick with hardware RAID. Adaptec 2400 is a slow card, but it allowed me to use the array while it was rebuilding (and I could safely reboot machine while it was rebuilding too).
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On Thu, 20 Oct 2005 at 9:43am, Aleksandar Milivojevic wrote
Quoting Joshua Baker-LePain jlb17@duke.edu:
Needless to say it's not giving me that warm fuzzy feeling. The one caveat is that not all the members of my array were the same size -- one disk is 180GB while all the rest are 160GB. I'm going to test overnight with identically sized RAID members, but I also wanted to see if anyone else is using RAID6.
I was testing RAID-5 with identical disk drives. Got the same thing. If I attempt to access it while it rebuilds, sooner or later I get errors. If I reboot while it rebuilds, it doesn't start it after the reboot (forgot the actuall error message). I needed to manually kick it again using mdadm to force-start it. If it was root file system, the machine would probably fail to boot completely. You might want to stick to your 3ware for now.
Wonderful. I'm testing RAID5 now, for which I had high hopes.
BTW, any particular reason not to use hardware RAID on 3ware?
That was the initial plan. But I can't get anything even resembling decent speeds out of them. These systems have 2 7500-8 boards in 'em (on separate PCI buses). I had been running RH7.3 (!!) and XFS with the boards in hardware RAID5 and a software RAID0 stripe across those, and would get >100MB/s writes and >300MB/s reads. With centos-4 and ext3, I was getting ~30MB/s writes and ~200MB/s reads. The reads I'd be OK with, but the write speed is absurd. I tweaked all I could think of, but nothing helped much.
Quoting Joshua Baker-LePain jlb17@duke.edu:
Wonderful. I'm testing RAID5 now, for which I had high hopes.
Well, my guess is that there'll be fix for this by the time next kernel version is out. Obviously it is an bug that should be fixed.
BTW, any particular reason not to use hardware RAID on 3ware?
That was the initial plan. But I can't get anything even resembling decent speeds out of them. These systems have 2 7500-8 boards in 'em (on separate PCI buses). I had been running RH7.3 (!!) and XFS with the boards in hardware RAID5 and a software RAID0 stripe across those, and would get >100MB/s writes and >300MB/s reads. With centos-4 and ext3, I was getting ~30MB/s writes and ~200MB/s reads. The reads I'd be OK with, but the write speed is absurd. I tweaked all I could think of, but nothing helped much.
Yeah, mixing RAIDs like that sometimes can get you into trouble.
I guess 3ware was not rebuilding anything while you were testing? Have you tested the speed of ext3 under RH7.3 (to rule out file system differences influencing your particular test)?
Is the speed OK if you use hardware RAID-5 on one card on its own (not mixing it with sofware RAID-0 like you had under RH-7.3)? If it is, try this configuration instead. Create two hardware RAID-5 volumes (one on each card). Then use fdisk to mark them as Linux LVM (8e). Use pvcreate to create LVM physical volumes on them, put them into volume group (for example create a new one) and make a logical volume on them. Example:
# pvcreate /dev/sda1 # pvcreate /dev/sdb1 # vgcreate -s 32m foobar /dev/sda1 /dev/sdb1 # vgdisplay foo # lvcreate -l nnn -n bar foo # mkfs.ext3 /dev/foo/bar # mount -t ext3 /dev/foo/bar /mnt
Replace sda1 and sdb1 with actual partitions on RAID-5 volumes. Replace "nnn" in lvcreate with the number of free physical extents (PE) as shown by vgdisplay command.
You can also test if striping will make any difference. For example, you might also replace the above lvcreate command with:
# lvcreate -l nnn -i 2 -n bar foo /dev/sda1 /dev/sdb1
You can also give it -I (capital i) option to specify the stripe size, if you don't want to use default value. See "man lvcreate" for more info.
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On Thu, 20 Oct 2005 at 10:27am, Aleksandar Milivojevic wrote
Quoting Joshua Baker-LePain jlb17@duke.edu:
Wonderful. I'm testing RAID5 now, for which I had high hopes.
Well, my guess is that there'll be fix for this by the time next kernel version is out. Obviously it is an bug that should be fixed.
There weren't any RAID6 specific bugs I could find in bugzilla. I'll probably file one once I test on a 2nd system and with the most recent kernel (all my testing has been with 2.6.9-11 as our very excellent campus-based distro hasn't pushed 4.2 yet).
BTW, any particular reason not to use hardware RAID on 3ware?
That was the initial plan. But I can't get anything even resembling decent speeds out of them. These systems have 2 7500-8 boards in 'em (on separate PCI buses). I had been running RH7.3 (!!) and XFS with the boards in hardware RAID5 and a software RAID0 stripe across those, and would get
100MB/s writes and >300MB/s reads. With centos-4 and ext3, I was getting
~30MB/s writes and ~200MB/s reads. The reads I'd be OK with, but the write speed is absurd. I tweaked all I could think of, but nothing helped much.
Yeah, mixing RAIDs like that sometimes can get you into trouble.
It's worked well in the past (obviously), and on my dual 9500 based system. It's the same general setup, and gets >150MB/s writes and nearly 400MB/s reads with centos-4 and ext3.
I guess 3ware was not rebuilding anything while you were testing? Have you tested the speed of ext3 under RH7.3 (to rule out file system differences influencing your particular test)?
No, the 3wares weren't rebuilding. I can't seem to find any RH7.3 ext3 benchmarking I did. I *did* try the centosplus kernel and XFS with the mixed hardware/software RAID setup. It got ~100MB/s writes but the reads went down to ~150MB/s. I just don't trust the stack to handle XFS and NFS on a production server.
Is the speed OK if you use hardware RAID-5 on one card on its own (not mixing it with sofware RAID-0 like you had under RH-7.3)? If it is, try this configuration instead. Create two hardware RAID-5 volumes (one on each card). Then use fdisk to mark them as Linux LVM (8e). Use pvcreate to create LVM physical volumes on them, put them into volume group (for example create a new one) and make a logical volume on them. Example:
I haven't tested a single board yet -- it'll be interesting to see what that yields. I hadn't thought of using LVM -- I may give that a shot. Thanks.
-
Aleksandar Milivojevic -
Joshua Baker-LePain