[CentOS] RAID 6 - opinions

[m.roth at 5-cent.us]

John R Pierce wrote:
> On 4/11/2013 8:36 AM, m.roth at 5-cent.us wrote:
>> I'm setting up this huge RAID 6 box. I've always thought of hot spares,
>> but I'm reading things that are comparing RAID 5 with a hot spare to
>> RAID 6, implying that the latter doesn't need one. I*certainly*  have
enough
>> drives to spare in this RAID box: 42 of 'em, so two questions: should I
>> assign one or more hot spares, and, if so, how many?
>
> John's First Rule of Raid.   when a drive fails 2-3 years downstream,
> replacements will be unavailable.  If you had bought cold spares and
> stored them, odds are too high they will be lost when you need them.
>
> John's Second Rule of Raid.  No single raid should be much over 10-12
> disks, or the rebuild times become truly hellacious.
>
> John's Third Rule of Raid.  allow 5-10% hot spares.
>
> so, with 42 disks, 10% would be ~4 spares, which leaves 38.  5% would be
> 2 spares, allowing 40 disks.
<snip>
> I did some testing of very large raids using LSI Logic  9261-8i MegaRAID
> SAS2 cards driving 36 3TB SATA3 disks.  With 3 x 11 disk RAID6 (and 3
> hot spares), a failed disk took about 12 hours to restripe with the
> rebuilding set to medium priority, and the raid essentially idle.
>
> if you're using XFS on this very large file system (which I *would*
> recommend), do be sure to use a LOT of ram, like 48GB...   while regular
> operations might not need it, XFS's fsck process is fairly memory
> intensive on a very large volume with millions of files.

Ok, listening to all of this, I've also been in touch with a tech from the
vendor*, who had a couple of suggestions: first, two RAID sets with two
global hot spares.

I've just spoken with my manager, and we're going with that, then one of
the tech's other suggestions was three volume sets on top of the two RAID
sets, so we'll have what look like three drives/LUNs of about 13+TB each.

All your comments were very appreciated, and gave me a lot more confidence
in this setup. We will be using ext4, btw - I don't get to try out XFS on
this $$$$$$ baby.

         mark

* Unpaid plug: we bought this from AC&NC: their own price was cheaper than
either of the two resellers I spoke to (three quotes required), they seem
pretty hungry (but have been around a while, given the number of old boxes
we have), and they respond *very* quickly to problems for support.