[CentOS] Setting up large (12.5 TB) filesystem howto?

Fri Aug 28 14:05:49 UTC 2009
Monty Shinn <montys at videopost.com>

Götz Reinicke - IT-Koordinator wrote:
> Hi,
> 
> I'm trying to set up an iscsi 12.5 TB storage for some data backup.
> 
> Doing so, I had some difficulties to find the right tool, maybe it's
> also a question of the system settings...
> 
> The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi
> connection can be establised.
> 
> fdisk and parted fail to create any information on the device or fail
> completely.
> 
> using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly
> create a logical volume:
> 
> lvdisplay /dev/VolGroup02/lvol0
>   --- Logical volume ---
>   LV Name                /dev/VolGroup02/lvol0
>   VG Name                VolGroup02
>   LV UUID                h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x
>   LV Write Access        read/write
>   LV Status              available
>   # open                 0
>   LV Size                12,73 TB
>   Current LE             3337487
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     256
>   Block device           253:4
> 
> 
> But, I can't create a filesystem on it:
> 
> mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0
> /dev/mapper/VolGroup02-lvol0
> 
> 
> mke2fs 1.39 (29-May-2006)
> mkfs.ext3: Filesystem too large.  No more than 2**31-1 blocks
> 	 (8TB using a blocksize of 4k) are currently supported.
> 
> 
> The limits information provided by red hat say, that RH EL 5.1 supports
> 16 TB filesystems:
> 
> http://www.redhat.com/rhel/compare/
> 
> 	-> Maximum filesystem size (Ext3): 16TB in 5.1
> 
> Using a block size of 8192 gives a warning, that this size is to large
> for that system.
> 
> 
> So my question: What is my missunderstanding or what's wrong with my
> system? Where are the real limits? Do I have to switch the OS to 64 Bit?
> 
> Setting up large Filesystems isn't my staff of life  :-)
> 
> 
> Thanks for amy how to or help of any kind - Best regards,
> 
> 	Götz
> 
> 

Gotz,

YMMV, but XFS is what I would use (and do use) as a filesystem in this 
instance.  There are caveats, some of which are listed here:

1)  Server/storage must be on stable power, backed up by a ups.  I have 
never dealt with XFS filesystem corruption, but I have read anecdotal 
horror stories.
2)  You need to run 64 bit, if for no other reason than memory requirements.
3)  You will need to have a partition that can be used as swap space.  I 
have not found any consistent formula, but I am running multiple servers 
using XFS, with 12gig of ram, 9TB filesystem, and 12 gigs was not enough 
memory to run xfs_check, which you should do on occasion.  I wound up 
making a 20gig swap space on a separate partition while I was running 
xfs_check.  That may have been excessively large, but it worked...
4)  XFS is not (or has not been) part of the standard RHEL distro, so 
you lose the 1:1 aspects of running CentOS when you implement XFS.

My experience has been that XFS is quite a bit faster than ext3, 
especially during file manipulation procedures.  Others may not agree. 
It has proven rock-solid for me, both in the linux world and IRIX.

Below is a link discussing the need for 64bit vs. 32bit (a bit old but 
still relevant):

http://oss.sgi.com/archives/xfs/2005-08/msg00391.html

Hope this helps.

Monty