[CentOS] 32-bit kernel+XFS+16.xTB filesystem = potential disaster (was:Re: ZFS @ centOS)

[Lamar Owen]

On Monday, April 04, 2011 11:09:29 PM Warren Young wrote:
> I did this test with Bonnie++ on a 3ware/LSI 9750-8i controller, with 
> eight WD 3 TB disks attached.  Both tests were done with XFS on CentOS 
> 5.5, 32-bit.  (Yes, 32-bit.  Hard requirement for this application.) 
[snip]
> For the RAID-6 configuration, I used the 3ware card's hardware RAID, 
> creating a single ~16 TB volume, formatted XFS.
[snip] 
> Dropping to 16.37 TB on the RAID configuration by switching 
> to RAID-6 let us put almost the entire array under a single 16 TB XFS 
> filesystem.

You really, really, really don't want to do this.  Not on 32-bit.  When you roll one byte over 16TB you will lose access to your filesystem, silently, and it will not remount on a 32-bit kernel.  XFS works best on a 64-bit kernel for a number of reasons; the one you're likely to hit first is the 16TB hard limit for *occupied* file space; you can mkfs an XFS filesystem on a 17TB or even larger partition or volume, but the moment the occupied data rolls over the 16TB boundary you will be in disaster recovery mode, and a 64-bit kernel will be required for rescue.

The reason I know this?  I had it happen.  On a CentOS 32-bit backup server with a 17TB LVM logical volume on EMC storage.  Worked great, until it rolled 16TB.  Then it quit working.  Altogether.  /var/log/messages told me that the filesystem was too large to be mounted.  Had to re-image the VM as a 64-bit CentOS, and then re-attached the RDM's to the LUNs holding the PV's for the LV, and it mounted instantly, and we kept on trucking.

There's a reason upstream doesn't do XFS on 32-bit.