As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular.
I looked up RAID6 and see the addition or a parity bit.
http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6
RAID 10 is also something I looked at. Striped, then Mirrored
So:
8 x 1.5tb = 12tb
RAID 5 = 12tb - 1.5tb for parity data = 10.5tb space available
RAID 10 = 4 x 1.5 = 6tb - 1.5tb for parity data = 4.5tb per stripe then mirror it.
but with RAID 10, data is safe after many types of failures.
On Friday, March 26, 2010 05:52 AM, Slack-Moehrle wrote:
As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular.
I looked up RAID6 and see the addition or a parity bit.
RAID6 allows you to survive failure of any two disks in the array at the cost of two disks of space.
RAID 10 is also something I looked at. Striped, then Mirrored
They recommend that you mirror and then stripe the mirrors. But that is probably old school now with Neil Brown's raid10 personality. Does anyone do nested raid1+0 setups anymore?
but with RAID 10, data is safe after many types of failures.
Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down.
Christopher Chan wrote:
but with RAID 10, data is safe after many types of failures.
Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down.
if you have a 8 drive raid1+0, and a random drive fails, you can survive any other drive failing *except* the mirror of the failed one. so if a second drive fails, there's only a 1 in 7 chance that its the 'fatal' one. on a 4 drive raid10, its a 1 in 3 chance. meanwhile, a raid10 can rebuild from a hotspare in like an hour, if the system isn't busy, and a few hours in the background if its busy and active.
On Friday, March 26, 2010 09:12 AM, John R Pierce wrote:
Christopher Chan wrote:
but with RAID 10, data is safe after many types of failures.
Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down.
if you have a 8 drive raid1+0, and a random drive fails, you can survive any other drive failing *except* the mirror of the failed one. so if a
That's what I said right? I did not say when a mirror is broken...
second drive fails, there's only a 1 in 7 chance that its the 'fatal' one. on a 4 drive raid10, its a 1 in 3 chance. meanwhile, a raid10 can rebuild from a hotspare in like an hour, if the system isn't busy, and a few hours in the background if its busy and active.
yeah, I thought the raid10 module would be able to rebuild automatically from a hotspare and would therefore be better than using nested raid1+0. I better stop the nested raid1+0 thing...does ananconda support the raid10 module during install yet? I mean rather, is the raid10 module included in the installation initrd yet?
On Mar 25, 2010, at 9:36 PM, Christopher Chan <christopher.chan@bradbury.edu.hk
wrote:
On Friday, March 26, 2010 09:12 AM, John R Pierce wrote:
Christopher Chan wrote:
but with RAID 10, data is safe after many types of failures.
Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down.
if you have a 8 drive raid1+0, and a random drive fails, you can survive any other drive failing *except* the mirror of the failed one. so if a
That's what I said right? I did not say when a mirror is broken...
second drive fails, there's only a 1 in 7 chance that its the 'fatal' one. on a 4 drive raid10, its a 1 in 3 chance. meanwhile, a raid10 can rebuild from a hotspare in like an hour, if the system isn't busy, and a few hours in the background if its busy and active.
yeah, I thought the raid10 module would be able to rebuild automatically from a hotspare and would therefore be better than using nested raid1+0. I better stop the nested raid1+0 thing...does ananconda support the raid10 module during install yet? I mean rather, is the raid10 module included in the installation initrd yet?
No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head.
Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette!
-Ross
Ross Walker wrote:
No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head.
Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette!
You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS.
Benjamin Franz wrote:
Ross Walker wrote:
No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head.
Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette!
You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS.
That's some serious redundancy dude!
Good for reads too...
I have one system with an 8-way RAID1 for the OS.
For real or is that a typo? Is that incase you go on holiday for a week and a drive-dies-a-day?
At Fri, 26 Mar 2010 17:16:04 +0000 CentOS mailing list centos@centos.org wrote:
I have one system with an 8-way RAID1 for the OS.
For real or is that a typo? Is that incase you go on holiday for a week and a drive-dies-a-day?
I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the "8-way RAID1 for the OS" probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks.
Robert Heller wrote:
I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the "8-way RAID1 for the OS" probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week. The machine wouldn't finishing booting until I physically removed the bad drive. The RAID1 was fine afterwards, but the RAID6 had to be manually re-assembled (no corruption, it just wouldn't automatically start until I intervened).
Benjamin Franz wrote:
Robert Heller wrote:
I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the "8-way RAID1 for the OS" probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
What are you running? I think there was a patch that evened out the reads across all members as it would at times solely read from one and then another...
Chan Chung Hang Christopher wrote:
Benjamin Franz wrote:
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
What are you running? I think there was a patch that evened out the reads across all members as it would at times solely read from one and then another...
I'm running fully up-to-date CentOS 5.4, kernel 2.6.18-164.15.1. The test was done with a default run of bonnie++. Watching the disk I/O while it ran suggested that it was only using some of the disks at a time to read (but changing which ones every few seconds).
8 x RAID1
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP pbox16.freerun. 16G 586 95 17263 5 13016 1 2263 97 60793 3 644.2 4 Latency 203ms 188s 1521ms 14714us 557ms 1265ms Version 1.96 ------Sequential Create------ --------Random Create-------- pbox16 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 12418 16 +++++ +++ 10999 12 27730 34 +++++ +++ 29916 32 Latency 109us 638us 681us 200us 25us 39us
8 x RAID6 Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP pbox16.freerun. 16G 611 96 90060 15 55711 14 2407 96 200901 27 522.3 15 Latency 13157us 604ms 1727ms 32420us 142ms 73131us Version 1.96 ------Sequential Create------ --------Random Create-------- pbox16 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 10298 16 +++++ +++ 24208 32 30395 45 +++++ +++ 32511 43 Latency 18504us 598us 613us 87us 25us 121us 1.96,1.96,pbox16,1,1269604576,16G,,611,96,90060,15,55711,14,2407,96,200901,27,522.3,15,16,,,,,10298,16,+++++,+++,24208,32,30395,45,+++++,+++,32511,43,13157us,604ms,1727ms,32420us,142ms,73131us,18504us,598us,613us,87us,25us,121us
On Fri, Mar 26, 2010 at 11:14:18AM -0700, Benjamin Franz wrote:
Robert Heller wrote:
I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the "8-way RAID1 for the OS" probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week.
I've also seen CentOS 5.3 (or 5.4, not sure) crash when a single sata hdd failed. The system was running mdadm RAID-1 mirror, so it shouldn't have been fatal event..
There was kernel oops on the console. too bad I didn't have time to capture it then. System was running AHCI SATA on Intel ICH9 controller, with mdadm software raid.
So there's still need for hardware RAID controllers..
-- Pasi
Pasi Kärkkäinen wrote:
On Fri, Mar 26, 2010 at 11:14:18AM -0700, Benjamin Franz wrote:
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week.
I've also seen CentOS 5.3 (or 5.4, not sure) crash when a single sata hdd failed. The system was running mdadm RAID-1 mirror, so it shouldn't have been fatal event..
There was kernel oops on the console. too bad I didn't have time to capture it then. System was running AHCI SATA on Intel ICH9 controller, with mdadm software raid.
So there's still need for hardware RAID controllers..
I'm not sure that is a good conclusion. The controller *is* a (3ware) hardware RAID controller - but the drive failure caused the 3ware driver to crash. That I wasn't using the controller in HW RAID mode may not be a good indicator that all would have been well if I had been.
At Sat, 27 Mar 2010 11:01:08 -0700 CentOS mailing list centos@centos.org wrote:
Pasi Kärkkäinen wrote:
On Fri, Mar 26, 2010 at 11:14:18AM -0700, Benjamin Franz wrote:
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week.
I've also seen CentOS 5.3 (or 5.4, not sure) crash when a single sata hdd failed. The system was running mdadm RAID-1 mirror, so it shouldn't have been fatal event..
There was kernel oops on the console. too bad I didn't have time to capture it then. System was running AHCI SATA on Intel ICH9 controller, with mdadm software raid.
So there's still need for hardware RAID controllers..
I'm not sure that is a good conclusion. The controller *is* a (3ware) hardware RAID controller - but the drive failure caused the 3ware driver to crash. That I wasn't using the controller in HW RAID mode may not be a good indicator that all would have been well if I had been.
There might be drive failure modes that put the drive controller into an 'odd' state. Still should not cause the software driver to crash -- I'd consider that a software bug. I've never heard of a SCSI HBA driver crashing. I might have encountered drive failure modes that would do things like hang the SCSI bus or otherwise confuse the SCSI HBA (eg causing it to fail to see *other* drives/devices). Since there is no SATA 'bus' (SATA drives are connected point-to-point 'star' fashion), a failed drive should not take the controller out, but I guess it depends on the signaling logic and what sort of logic gates are used for 'drive select'. Of course, a controller like the 3ware hardware RAID controller, configured in JOBD mode, probably looks like a SCSI HBA with bunch of disks on a single SCSI bus.
On Sat, Mar 27, 2010 at 11:01:08AM -0700, Benjamin Franz wrote:
Pasi Kärkkäinen wrote:
On Fri, Mar 26, 2010 at 11:14:18AM -0700, Benjamin Franz wrote:
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week.
I've also seen CentOS 5.3 (or 5.4, not sure) crash when a single sata hdd failed. The system was running mdadm RAID-1 mirror, so it shouldn't have been fatal event..
There was kernel oops on the console. too bad I didn't have time to capture it then. System was running AHCI SATA on Intel ICH9 controller, with mdadm software raid.
So there's still need for hardware RAID controllers..
I'm not sure that is a good conclusion. The controller *is* a (3ware) hardware RAID controller - but the drive failure caused the 3ware driver to crash. That I wasn't using the controller in HW RAID mode may not be a good indicator that all would have been well if I had been.
Heh.. well.. it seems we have many bugs.. :)
-- Pasi
Pasi Kärkkäinen wrote:
On Fri, Mar 26, 2010 at 11:14:18AM -0700, Benjamin Franz wrote:
Robert Heller wrote:
I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the "8-way RAID1 for the OS" probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks
Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition).
In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week.
I've also seen CentOS 5.3 (or 5.4, not sure) crash when a single sata hdd failed. The system was running mdadm RAID-1 mirror, so it shouldn't have been fatal event..
There was kernel oops on the console. too bad I didn't have time to capture it then. System was running AHCI SATA on Intel ICH9 controller, with mdadm software raid.
So there's still need for hardware RAID controllers..
A hardware controller can fail in ways that kill the kernel too.
On 3/26/2010 12:45 PM, Robert Heller wrote:
At Fri, 26 Mar 2010 17:16:04 +0000 CentOS mailing listcentos@centos.org wrote:
I have one system with an 8-way RAID1 for the OS.
For real or is that a typo? Is that incase you go on holiday for a week and a drive-dies-a-day?
I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the "8-way RAID1 for the OS" probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks.
If you are really paranoid you can split things up into raid1 mirrors of 2 drives each mounted into logical places. At the expense of having to mange the space in smaller chunks (and losing half to redundancy) you get the ability to control head contention among jobs and to recover data from any single disk after a problem.
I'm not the only one to sit up and take notice here. The last reply came only 2 minutes after the 8-way mention.
Benjamin Franz wrote:
Ross Walker wrote:
No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head.
Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette!
You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS.
That's some serious redundancy dude!
Good for reads too... _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Mar 26, 2010, at 10:25 AM, Benjamin Franz jfranz@freerun.com wrote:
Ross Walker wrote:
No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head.
Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette!
You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS.
Yes, but you can use a lot less drives to get the same IOPS with Neil's module (which technically isn't RAID10 at all but a whole new RAID level).
-Ross
-
Benjamin Franz -
Chan Chung Hang Christopher -
Christopher Chan
-
James Bensley -
John R Pierce -
Les Mikesell -
Pasi Kärkkäinen -
Robert -
Robert Heller -
Ross Walker -
Slack-Moehrle