[CentOS] 40TB File System Recommendations

Tue Apr 12 10:35:25 UTC 2011
Steve Brooks <steveb at mcs.st-and.ac.uk>

On Tue, 12 Apr 2011, Marian Marinov wrote:

> On Tuesday 12 April 2011 10:36:54 Alain Péan wrote:
>> Le 12/04/2011 09:23, Matthew Feinberg a écrit :
>>> Hello All
>>> I have a brand spanking new 40TB Hardware Raid6 array to play around
>>> with. I am looking for recommendations for which filesystem to use. I am
>>> trying not to break this up into multiple file systems as we are going
>>> to use it for backups. Other factors is performance and reliability.
>>> CentOS 5.6
>>> array is /dev/sdb
>>> So here is what I have tried so far
>>> reiserfs is limited to 16TB
>>> ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
>>> support creating ext4 (strange)
>>> Anyone work with large filesystems like this that have any
>>> suggestions/recommendations?
>> Hi Matthew,
>> I would go for xfs, which is now supported in CentOS. This is what I use
>> for a 16 TB storage, with CentOS 5.3 (Rocks Cluster), and it woks fine.
>> No problem with lengthy fsck, as with ext3 (which does not support such
>> capacities). I did not try yet ext4...
>> Alain
> I have Raid6 Arrays with 30TB. We have tested XFS and its write performance
> was really dissapointing. So we looked at Ext4. It is really good for our
> workloads, but it lacks the ability to grow over 16TB. So we crated two
> partitions on the raid with ext4.
> The RAID rebuild time is around 2 days, max 3 if the workload is higher. So I
> presume that for 40TB it will be around 4 days.
> Marian

For interest how much *memory* would you need in your raid management node 
to support "fsck" on a 40TB array. I imagine it would be very high.