Hi,
I'm trying to set up an iscsi 12.5 TB storage for some data backup.
Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings...
The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised.
fdisk and parted fail to create any information on the device or fail completely.
using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume:
lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name /dev/VolGroup02/lvol0 VG Name VolGroup02 LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Access read/write LV Status available # open 0 LV Size 12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4
But, I can't create a filesystem on it:
mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0
mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported.
The limits information provided by red hat say, that RH EL 5.1 supports 16 TB filesystems:
http://www.redhat.com/rhel/compare/
-> Maximum filesystem size (Ext3): 16TB in 5.1
Using a block size of 8192 gives a warning, that this size is to large for that system.
So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit?
Setting up large Filesystems isn't my staff of life :-)
Thanks for amy how to or help of any kind - Best regards,
Götz
On Fri, Aug 28, 2009 at 1:03 PM, Götz Reinicke - IT-Koordinatorgoetz.reinicke@filmakademie.de wrote:
I'm trying to set up an iscsi 12.5 TB storage for some data backup.
Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings...
The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised.
...snip...
The limits information provided by red hat say, that RH EL 5.1 supports 16 TB filesystems:
http://www.redhat.com/rhel/compare/
-> Maximum filesystem size (Ext3): 16TB in 5.1
Using a block size of 8192 gives a warning, that this size is to large for that system.
So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit?
That was discussed not so long ago. But I can't remember if it was here in the mailing list or on IRC (or somewhere else). But the conclusion was that you are better of using 64 bit and XFS for such large filesystems, and that you need a 64 bit system anyway for ext3/4 to go over 8TB. Well, that is what I remember the conclusion to be :-)
Regards, Tim
On Fri, Aug 28, 2009 at 7:03 AM, Götz Reinicke - IT-Koordinatorgoetz.reinicke@filmakademie.de wrote:
Hi,
I'm trying to set up an iscsi 12.5 TB storage for some data backup.
Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings...
The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised.
fdisk and parted fail to create any information on the device or fail completely.
using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume:
lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name /dev/VolGroup02/lvol0 VG Name VolGroup02 LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Access read/write LV Status available # open 0 LV Size 12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4
But, I can't create a filesystem on it:
mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0
You have to smack mkfs around a bit to get it to work. The incantations are listed here, along with the various limitations. The long and short of it is that you have to use -F to tell mkfs that you're really, REALLY sure. http://www.bofh-hunter.com/2008/02/11/large-filesystem-creation/
On 28/08/2009, at 10:59 PM, Jim Perrin wrote:
On Fri, Aug 28, 2009 at 7:03 AM, Götz Reinicke - IT-Koordinatorgoetz.reinicke@filmakademie.de wrote:
Hi,
I'm trying to set up an iscsi 12.5 TB storage for some data backup.
Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings...
The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised.
fdisk and parted fail to create any information on the device or fail completely.
using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume:
lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name /dev/VolGroup02/lvol0 VG Name VolGroup02 LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Access read/write LV Status available # open 0 LV Size 12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto
- currently set to 256
Block device 253:4
But, I can't create a filesystem on it:
mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0
You have to smack mkfs around a bit to get it to work. The incantations are listed here, along with the various limitations. The long and short of it is that you have to use -F to tell mkfs that you're really, REALLY sure. http://www.bofh-hunter.com/2008/02/11/large-filesystem-creation/
For a 4K block size I think the maximum ext3 filesystem size is 8TB. You may need to use an 8K block size which gives a maximum filesystem size of 16TB.
Neil
Götz Reinicke - IT-Koordinator wrote:
Hi,
I'm trying to set up an iscsi 12.5 TB storage for some data backup.
Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings...
The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised.
fdisk and parted fail to create any information on the device or fail completely.
using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume:
lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name /dev/VolGroup02/lvol0 VG Name VolGroup02 LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Access read/write LV Status available # open 0 LV Size 12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto
- currently set to 256
Block device 253:4
But, I can't create a filesystem on it:
mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0
mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported.
The limits information provided by red hat say, that RH EL 5.1 supports 16 TB filesystems:
http://www.redhat.com/rhel/compare/
-> Maximum filesystem size (Ext3): 16TB in 5.1
Using a block size of 8192 gives a warning, that this size is to large for that system.
So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit?
Setting up large Filesystems isn't my staff of life :-)
Thanks for amy how to or help of any kind - Best regards,
Götz
Gotz,
YMMV, but XFS is what I would use (and do use) as a filesystem in this instance. There are caveats, some of which are listed here:
1) Server/storage must be on stable power, backed up by a ups. I have never dealt with XFS filesystem corruption, but I have read anecdotal horror stories. 2) You need to run 64 bit, if for no other reason than memory requirements. 3) You will need to have a partition that can be used as swap space. I have not found any consistent formula, but I am running multiple servers using XFS, with 12gig of ram, 9TB filesystem, and 12 gigs was not enough memory to run xfs_check, which you should do on occasion. I wound up making a 20gig swap space on a separate partition while I was running xfs_check. That may have been excessively large, but it worked... 4) XFS is not (or has not been) part of the standard RHEL distro, so you lose the 1:1 aspects of running CentOS when you implement XFS.
My experience has been that XFS is quite a bit faster than ext3, especially during file manipulation procedures. Others may not agree. It has proven rock-solid for me, both in the linux world and IRIX.
Below is a link discussing the need for 64bit vs. 32bit (a bit old but still relevant):
http://oss.sgi.com/archives/xfs/2005-08/msg00391.html
Hope this helps.
Monty
On Fri, 28 Aug 2009 at 1:03pm, Götz Reinicke - IT-Koordinator wrote
fdisk and parted fail to create any information on the device or fail completely.
You can't use fdisk on a volume that large. parted should work fine. What was the error you were getting (exactly)? For a volume that large, you must use a GPT disk label, not the default msdos one.
But, I can't create a filesystem on it:
mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0
mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported.
As has been pointed out, you need to use "-F" to force mkfs.ext3 to make a filesystem bigger than 8TB. IMHO, this is misleading. Filesystems up to 16TB are fully supported in centos >5.1, so I don't see why the upstream vendor left the requirement for "-F" in mkfs.ext3.
So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit?
You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option.
On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePainjlb17@duke.edu wrote:
You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option.
Support for xfs has been added to RHEL 5.4 which will be released any day now.
Akemi
And just to add to the discussion...
We use JFS here for large filesystems. :) (We have some 24TB filesystems in place here using JFS, with no problems like XFS has when it gets corrupted).
On Fri, Aug 28, 2009 at 9:53 AM, David Fixdavidf@mrxfx.com wrote:
And just to add to the discussion...
We use JFS here for large filesystems. :) (We have some 24TB filesystems in place here using JFS, with no problems like XFS has when it gets corrupted).
Because the distro kernel does not have support fort JFS, people who are looking into using JFS would need to install either the centosplus kernel or the jfs kernel module provided by ELRepo ( http://elrepo.org/tiki/kmod-jfs ).
Akemi
On Fri, 28 Aug 2009 at 8:58am, Akemi Yagi wrote
On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePainjlb17@duke.edu wrote:
You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option.
Support for xfs has been added to RHEL 5.4 which will be released any day now.
So it has. I recall looking in the beta release notes when they first came out and not seeing it. So either I just plain missed it or it's been added there since then. In any case, that's great news and something that is *long* overdue.
On Fri, Aug 28, 2009 at 9:58 AM, Joshua Baker-LePainjlb17@duke.edu wrote:
On Fri, 28 Aug 2009 at 8:58am, Akemi Yagi wrote
Support for xfs has been added to RHEL 5.4 which will be released any day now.
So it has. I recall looking in the beta release notes when they first came out and not seeing it. So either I just plain missed it or it's been added there since then. In any case, that's great news and something that is *long* overdue.
Indeed. I believe the beta release notes did not mention xfs. But it is here:
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html-single/Re...
and the 5.4beta kernel does/did have xfs.ko.
Akemi
Joshua Baker-LePain wrote:
On Fri, 28 Aug 2009 at 8:58am, Akemi Yagi wrote
On Fri, Aug 28, 2009 at 8:36 AM, Joshua Baker-LePainjlb17@duke.edu wrote:
You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option.
Support for xfs has been added to RHEL 5.4 which will be released any day now.
So it has. I recall looking in the beta release notes when they first came out and not seeing it. So either I just plain missed it or it's been added there since then. In any case, that's great news and something that is *long* overdue.
Please note that it is not unheard of for them to PULL things that they have in the beta when rolling out the release.
In this case, I certainly hope that they do not do this, but it is possible.
Lets hope they do keep XFS in there.
-
Akemi Yagi -
David Fix -
Götz Reinicke - IT-Koordinator -
Jim Perrin -
Johnny Hughes -
Joshua Baker-LePain -
Monty Shinn -
Neil Muller -
Tim Verhoeven