[CentOS] 40TB File System Recommendations

Thu Apr 14 14:42:40 UTC 2011
Peter Kjellström <cap at nsc.liu.se>

On Tuesday, April 12, 2011 02:56:54 PM rainer at ultra-secure.de wrote:
...
> > Steve,
> > I'm managing machines with 30TB of storage for more then two years. And
> > with
> > good reporting and reaction we have never had to run fsck.
> 
> That's not the issue.
> The issue is rebuild-time.
> The longer it takes, the more likely is another failure in the array.
> With RAID6, this does not instantly kill your RAID, as with RAID5 - but I
> assume it will further decrease overall-performance and the rebuild-time
> will go up significantly - adding the the risk.

While I do concede the obvious point regarding rebuild time (raid6 takes from 
long to very long to rebuild) I'd like to point out:

 * If you do the math for a 12 drive raid10 vs raid6 then (using actual data 
from ~500 1T drives on HP cciss controllers during two years) raid10 is ~3x 
more likely to cause hard data loss than raid6.

 * mtbf is not everything there's also the thing called unrecoverable read 
errors. If you hit one while rebuilding your raid10 you're toast while in the 
raid6 case you'll use your 2nd parity and continue the rebuild.

/Peter (who runs many 12 drive raid6 systems just fine)

> Thus, it's generally advisable to do just use RAID10 (in this case, a
> thin-striped array of RAID1-arrays).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.centos.org/pipermail/centos/attachments/20110414/115c64e0/attachment-0003.sig>