[CentOS] 12-15 TB RAID storage recommendations

[John R Pierce]

Boris Epstein wrote:
> On Tue, Apr 13, 2010 at 2:30 PM, Joseph L. Casale
> <jcasale at activenetwerx.com> wrote:
>   
>>> I would like to build a 12-15 TB RAID 5 data server to run under
>>> ContOS. Any recommendations as far as hardware, configuration, etc?
>>>       
>> This comes up often:)
>> Things to keep in mind when using large discs if you suffer a failure,
>> while rebuilding in a degraded state it's not impossible or unlikely to
>> drop another disc loosing it all.
>>
>> Rebuild times especially on busy arrays with large discs take lots of
>> time...
>> _______________________________________________
>> CentOS mailing list
>> CentOS at centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>>     
>
> Thanks Joseph!
>
> In your opinion, if we are talking about 15 2-TB disks what does "lots
> of time" translate to?
>   


well, IF your controller totally screams and can rebuild the drives at 
wire speeds with full overlap, you'll be reading 7 * 2TB of data at 
around 100MB/sec average and writing the XOR of that to the 8th drive in 
a 8 spindle raid5 (14tb total).   just reading one drive at wirespeed is 
2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about 
the shortest it possibly could be done.

if you have any access activity during this rebuild, you can figure on 
nearly doubling that before you even get warm.   if the controller can't 
actually move 800MB/sec internally (believe me, that ain't easy), then 
you can double or quadruple it a few more times.   This 5.5 hours could 
easily stretch to a week.


ANY previously unnoticed bad sector on those other 7 drives renders the 
whole pile bad, so doing weekly raid sweeps is probably a good thing, 
except THOSE will take some multiple of 5.5 hours when run in the 
background.