Götz Reinicke - IT-Koordinator wrote:
Hi,
I'm trying to set up an iscsi 12.5 TB storage for some data backup.
Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings...
The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised.
fdisk and parted fail to create any information on the device or fail completely.
using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume:
lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name /dev/VolGroup02/lvol0 VG Name VolGroup02 LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Access read/write LV Status available # open 0 LV Size 12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto
- currently set to 256
Block device 253:4
But, I can't create a filesystem on it:
mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0
mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported.
The limits information provided by red hat say, that RH EL 5.1 supports 16 TB filesystems:
http://www.redhat.com/rhel/compare/
-> Maximum filesystem size (Ext3): 16TB in 5.1
Using a block size of 8192 gives a warning, that this size is to large for that system.
So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit?
Setting up large Filesystems isn't my staff of life :-)
Thanks for amy how to or help of any kind - Best regards,
Götz
Gotz,
YMMV, but XFS is what I would use (and do use) as a filesystem in this instance. There are caveats, some of which are listed here:
1) Server/storage must be on stable power, backed up by a ups. I have never dealt with XFS filesystem corruption, but I have read anecdotal horror stories. 2) You need to run 64 bit, if for no other reason than memory requirements. 3) You will need to have a partition that can be used as swap space. I have not found any consistent formula, but I am running multiple servers using XFS, with 12gig of ram, 9TB filesystem, and 12 gigs was not enough memory to run xfs_check, which you should do on occasion. I wound up making a 20gig swap space on a separate partition while I was running xfs_check. That may have been excessively large, but it worked... 4) XFS is not (or has not been) part of the standard RHEL distro, so you lose the 1:1 aspects of running CentOS when you implement XFS.
My experience has been that XFS is quite a bit faster than ext3, especially during file manipulation procedures. Others may not agree. It has proven rock-solid for me, both in the linux world and IRIX.
Below is a link discussing the need for 64bit vs. 32bit (a bit old but still relevant):
http://oss.sgi.com/archives/xfs/2005-08/msg00391.html
Hope this helps.
Monty