[CentOS] 40TB File System Recommendations

Tue Apr 12 20:54:31 UTC 2011
Lamar Owen <lowen at pari.edu>

On Tuesday, April 12, 2011 02:51:45 PM John R Pierce wrote:
> On 04/12/11 6:02 AM, Marian Marinov wrote:
> >
> > Yes... but with such RAID10 solution you get only half of the disk space... so
> > from 10 2TB drives you get only 10TB instead of 16TB with RAID6.
> 
> those disks are $100 each.   whats your data worth?

Where can I get an enterprise-class 2TB drive for $100?  Commodity SATA isn't enterprise-class.  SAS is; FC is, SCSI is. A 500GB FC drive with EMC firmware new is going to set you back ten times that, at least.  What's youre data worth indeed, putting it on commodity disk.... :-)

> in this case, the OP is talking about a 40TB array, so thats a TWENTY 
> TWO drive raid.  NOONE I know in the storage business will use larger 
> than a 8 or 10 drive raid set.   

EMC allows RAID groups up to 16 drives on Clariion storage.  I've been doing this with EMC stuff for a while, with RAID6 plus a hotspare per DAE; that's a 14 drive RAID group plus the hotspare on one DAE.  Some systems I forgo the dedicated per-DAE hotspare and spread a 16 drive RAID6 group and a 14 drive RAID6 group across two DAE's with hotspares on other DAE's.  Works ok, and I've had double drive soft failures on a single RAID6 group that successfully hotspared (and back).  This is partially due to the custom EMC firmware on the drives, and the interaction with the storage processor.

Rebuild time is several hours, but with more smaller drives it's not too bad.

> If you really need such a massive 
> volume, you stripe several smaller raidsets, so the raid6 version would 
> be 2 x 12 x 2TB or 24 drives for raid6+0 == 40TB.

Or you do metaLUNs, or similar using LVM.