John R Pierce wrote: > On 4/11/2013 8:36 AM, m.roth at 5-cent.us wrote: >> I'm setting up this huge RAID 6 box. I've always thought of hot spares, >> but I'm reading things that are comparing RAID 5 with a hot spare to >> RAID 6, implying that the latter doesn't need one. I*certainly* have enough >> drives to spare in this RAID box: 42 of 'em, so two questions: should I >> assign one or more hot spares, and, if so, how many? > > John's First Rule of Raid. when a drive fails 2-3 years downstream, > replacements will be unavailable. If you had bought cold spares and > stored them, odds are too high they will be lost when you need them. > > John's Second Rule of Raid. No single raid should be much over 10-12 > disks, or the rebuild times become truly hellacious. > > John's Third Rule of Raid. allow 5-10% hot spares. > > so, with 42 disks, 10% would be ~4 spares, which leaves 38. 5% would be > 2 spares, allowing 40 disks. <snip> > I did some testing of very large raids using LSI Logic 9261-8i MegaRAID > SAS2 cards driving 36 3TB SATA3 disks. With 3 x 11 disk RAID6 (and 3 > hot spares), a failed disk took about 12 hours to restripe with the > rebuilding set to medium priority, and the raid essentially idle. > > if you're using XFS on this very large file system (which I *would* > recommend), do be sure to use a LOT of ram, like 48GB... while regular > operations might not need it, XFS's fsck process is fairly memory > intensive on a very large volume with millions of files. Ok, listening to all of this, I've also been in touch with a tech from the vendor*, who had a couple of suggestions: first, two RAID sets with two global hot spares. I've just spoken with my manager, and we're going with that, then one of the tech's other suggestions was three volume sets on top of the two RAID sets, so we'll have what look like three drives/LUNs of about 13+TB each. All your comments were very appreciated, and gave me a lot more confidence in this setup. We will be using ext4, btw - I don't get to try out XFS on this $$$$$$ baby. mark * Unpaid plug: we bought this from AC&NC: their own price was cheaper than either of the two resellers I spoke to (three quotes required), they seem pretty hungry (but have been around a while, given the number of old boxes we have), and they respond *very* quickly to problems for support.