[CentOS] RAID 6 - opinions

[John R Pierce]

On 4/11/2013 8:36 AM, m.roth at 5-cent.us wrote:
> I'm setting up this huge RAID 6 box. I've always thought of hot spares,
> but I'm reading things that are comparing RAID 5 with a hot spare to RAID
> 6, implying that the latter doesn't need one. I*certainly*  have enough
> drives to spare in this RAID box: 42 of 'em, so two questions: should I
> assign one or more hot spares, and, if so, how many?

John's First Rule of Raid.   when a drive fails 2-3 years downstream, 
replacements will be unavailable.  If you had bought cold spares and 
stored them, odds are too high they will be lost when you need them.

John's Second Rule of Raid.  No single raid should be much over 10-12 
disks, or the rebuild times become truly hellacious.

John's Third Rule of Raid.  allow 5-10% hot spares.

so, with 42 disks, 10% would be ~4 spares, which leaves 38.  5% would be 
2 spares, allowing 40 disks.

40 divided by 4 == 10.   You could format that as 10 raid6's, and stripe 
those (aka raid6+0 or raid60), and use 2 hot spares.
Alternately, 3*13 == 39, leaving three hotspares, so 3 stripes of 13 
disks with 3 hot spares is an alternative.

I did some testing of very large raids using LSI Logic  9261-8i MegaRAID 
SAS2 cards driving 36 3TB SATA3 disks.  With 3 x 11 disk RAID6 (and 3 
hot spares), a failed disk took about 12 hours to restripe with the 
rebuilding set to medium priority, and the raid essentially idle.

if you're using XFS on this very large file system (which I *would* 
recommend), do be sure to use a LOT of ram, like 48GB...   while regular 
operations might not need it, XFS's fsck process is fairly memory 
intensive on a very large volume with millions of files.

-- 
john r pierce                                      37N 122W
somewhere on the middle of the left coast