On Wed, Sep 30, 2009 at 08:52:08PM -0500, Johnny Hughes wrote: > On 09/24/2009 07:35 AM, Rainer Duffner wrote: > > Well, it depends on the disk-size: > > http://www.enterprisestorageforum.com/technology/features/article.php/3839636 > > This info is VERY relevant ... you will almost ALWAYS have a failure on > rebuild with very large RAID 5 arrays. Since that is a fault in a > second drive, that failure will cause the loss of all the data. I would > not recommend RAID 5 right now ... it is not worth the risk. "Almost always" is very dependent on the disks and size of the array. Let's take a 20TiByte array as an example. Now, the "hard error rate" is an expectation. That means that with an error rate of 1E14 then you'd expect to see 1 error for every 1E14 bits read. If we make the simplifying assumption of any read being equally likely to fail then any single bit read has a 1/1E14 chance of being wrong. (see end of email for more thoughts on this). Now to rebuild a 20Tibyte array you would need to read 20Tibytes of data. The chance of this happening without error is: (1-1/1E14)^(8*20*2^40) = 0.172 ie only 17% of rebuilding a 20TiByte array! That's pretty bad. In fact it's downright awful. Do not build 20TiByte arrays with consumer disks! Note that this doesn't care about the size of the disks nor the number of disks; it's purely based on probability of read error. Now an "enterprise" class disk with an error rate of 1E15 looks better: (1-1/1E15)^(8*10*2^40) = 0.838 or 84% chance of successful rebuild. Better. But probably not good enough. How about an Enterprise SAS disk at 1E16 (1-1/1E16)^(8*12.5*2^40) = 0.981 or 98% Not "five nines", but pretty good. Of course you're never going to get 100%. Technology just doesn't work that way. So, if you buy Enterprise SAS disks then you do stand a good chance of rebuilding a 20TiByte Raid 5. A 2% chance of a double-failure. Do you want to risk your company on that? RAID6 makes things better; you need a triple failure to cause data loss. It's possible, but the numbers are a lot lower. Of course the error rate and other disk characteristics are actually WAGs based on some statistical analysis. There's no actual measurements to show this. Real life numbers appear to show that disks far outlive their expected values. Error rates are much lower than manufacturer claims (excluding bad batches and bad manufacturing, of course!) This is just a rough "off my head" analysis. I'm not totally convinced it's correct (my understanding of error rate could be wrong; the assumption of even failure distribution is likely to be wrong because errors on a disk cluster - a sector is bad, a track is bad etc). But the analysis _feels_ right... which means nothing :-) I currently have 5*1Tbyte consumer disks in a RAID5. That, theoretically, gives me a 27% chance of failure during a rebuild. As it happens I've had 2 bad disks, but they went bad a month apart (I think it is a bad batch!). Each time the array has rebuilt without detectable error. Let's not even talk about Petabyte arrays. If you're doing that then you better have multiple redundancy in place, and **** the expense! Google is a great example of this. -- rgds Stephen