mkfs.ext4 fails to create 38TB file system on CentOS 6.4 64bit with this error:
mkfs.ext4: Size of device /dev/vg02/vtapes too big to be expressed in 32 bits using a blocksize of 4096.
More details follow:
# uname -a Linux tzbackup 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
# fdisk -l /dev/sdc
Disk /dev/sdc: 41996.7 GB, 41996727091200 bytes 255 heads, 63 sectors/track, 5105811 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
# pvdisplay /dev/sdc --- Physical volume --- PV Name /dev/sdc VG Name vg02 PV Size 38.20 TiB / not usable 4.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 10012799 Free PE 0 Allocated PE 10012799 PV UUID qGIgYy-Q8N2-xUjM-58Zb-uEnu-qFNY-NGjkhx
# vgdisplay /dev/vg02 --- Volume group --- VG Name vg02 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 38.20 TiB PE Size 4.00 MiB Total PE 10012799 Alloc PE / Size 10012799 / 38.20 TiB Free PE / Size 0 / 0 VG UUID T1jUYU-wBTG-u0aN-AtYV-iFf8-S6Hw-zMuacu
# lvdisplay /dev/vg02/vtapes --- Logical volume --- LV Path /dev/vg02/vtapes LV Name vtapes VG Name vg02 LV UUID 6cktBz-ayFL-5XVp-kVEp-4giU-A1Kc-Jjf2UV LV Write Access read/write LV Creation host, time tzbackup, 2013-09-20 13:29:30 -0700 LV Status available # open 0 LV Size 38.20 TiB Current LE 10012799 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:8
# mkfs.ext4 -b 4096 /dev/vg02/vtapes mke2fs 1.41.12 (17-May-2010) mkfs.ext4: Size of device /dev/vg02/vtapes too big to be expressed in 32 bits using a blocksize of 4096.
Any idea?
Thanks,
-- Peter
On 20.09.2013 21:58, Peter Wood wrote:
mkfs.ext4 fails to create 38TB file system on CentOS 6.4 64bit with this error:
mkfs.ext4: Size of device /dev/vg02/vtapes too big to be expressed in 32 bits using a blocksize of 4096.
EXT4 filesystems are limited (on RHEL) to 16TB. You should be using XFS for such large volumes.
XFS worked. Thanks a lot.
On Fri, Sep 20, 2013 at 2:05 PM, Nux! nux@li.nux.ro wrote:
On 20.09.2013 21:58, Peter Wood wrote:
mkfs.ext4 fails to create 38TB file system on CentOS 6.4 64bit with this error:
mkfs.ext4: Size of device /dev/vg02/vtapes too big to be expressed in 32 bits using a blocksize of 4096.
EXT4 filesystems are limited (on RHEL) to 16TB. You should be using XFS for such large volumes.
-- Sent from the Delta quadrant using Borg technology!
Nux! www.nux.ro _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 9/20/2013 2:27 PM, Peter Wood wrote:
XFS worked. Thanks a lot.
for large xfs file systems, be sure to mount with option inode64.
there's one caveat on XFS in Linux, when used as a nfs export... if you JUST export your file system root, then you're fine, but if you like to have multiple seperate exports on the same file system, NFS does NOT like the 64 bit inode numbers, so you HAVE to specify manual arbitrary unique export fsid numbers. I usually just use fsid=1,2,3,... for the exports. AFAIK, they only have to be unique within the file system. there's an fsid=uuid option but I ran into multiple problems in my heterogeneous production environment with this, too.
also, natch, you should ONLY use xfs on a 64bit OS. I think its intentionally disabled on 32bit, but if you were to try and force it, you'd get into data corruptions or kernel panics.
and, a large XFS file system requires quite a bit of RAM to do the xfs_check/repair (equiv to fsck)
Very useful information John. Thank you. (Your email slipped through the cracks and I just saw it by accident. I'm happy I didn't loose it.)
On Fri, Sep 20, 2013 at 3:21 PM, John R Pierce pierce@hogranch.com wrote:
On 9/20/2013 2:27 PM, Peter Wood wrote:
XFS worked. Thanks a lot.
for large xfs file systems, be sure to mount with option inode64.
there's one caveat on XFS in Linux, when used as a nfs export... if you JUST export your file system root, then you're fine, but if you like to have multiple seperate exports on the same file system, NFS does NOT like the 64 bit inode numbers, so you HAVE to specify manual arbitrary unique export fsid numbers. I usually just use fsid=1,2,3,... for the exports. AFAIK, they only have to be unique within the file system. there's an fsid=uuid option but I ran into multiple problems in my heterogeneous production environment with this, too.
also, natch, you should ONLY use xfs on a 64bit OS. I think its intentionally disabled on 32bit, but if you were to try and force it, you'd get into data corruptions or kernel panics.
and, a large XFS file system requires quite a bit of RAM to do the xfs_check/repair (equiv to fsck)
-- john r pierce 37N 122W somewhere on the middle of the left coast
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 2013-09-20, John R Pierce pierce@hogranch.com wrote:
for large xfs file systems, be sure to mount with option inode64.
There are a bunch of FAQ entries at the XFS site about this issue. Here's one of them:
http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F
The other issues about using inode64 and NFS are also nearby on the same page.
So following up on that, I have a question: I have a largish XFS filesystem which is currently not mounted inode64. I would like to know how close I am to filling up the inodes in the first 1TB, so that I know how urgent it is for me to migrate to inode64. Does anybody know a way to query this information out of the filesystem?
and, a large XFS file system requires quite a bit of RAM to do the xfs_check/repair (equiv to fsck)
I have read on the mailing lists that the XFS folks recommend using xfs_repair -n over xfs_check. xfs_repair still needs a lot of RAM, but apparently xfs_check needs much more.
--keith
On 9/26/2013 2:27 PM, Keith Keller wrote:
So following up on that, I have a question: I have a largish XFS filesystem which is currently not mounted inode64. I would like to know how close I am to filling up the inodes in the first 1TB, so that I know how urgent it is for me to migrate to inode64. Does anybody know a way to query this information out of the filesystem?
as I understand it, once there's no free blocks in the first TB, there's no way to create another 32bit inode.
On 09/26/2013 04:27 PM, Keith Keller wrote: <>
So following up on that, I have a question: I have a largish XFS filesystem which is currently not mounted inode64. I would like to know how close I am to filling up the inodes in the first 1TB, so that I know how urgent it is for me to migrate to inode64. Does anybody know a way to query this information out of the filesystem?
only way i can think of to get close would be rather lengthy.
as root user;
ls -iR /|less
and watch for max number, or via a php command that will list inodes into increasing numerical order.
if you fear you are getting close to max, zip up a bunch of files that you do not need, ie, emails, text files, photographs, etc. then delete what you zipped to release the inodes.
if you have another partition with a greater number of free inodes, mv directory(s) to it, then link directory(s) back where you want.
hth.
On 9/26/2013 2:27 PM, Keith Keller wrote:
so that I know how urgent it is for me to migrate to inode64.
you realize, 'migrate' is trivial?
mount -o remount,inode64 /dev/whatever
and replace 'defaults' with 'inode64' on the line in /etc/fstab so it does it next time too.
thats it.
On 2013-09-26, John R Pierce pierce@hogranch.com wrote:
On 9/26/2013 2:27 PM, Keith Keller wrote:
so that I know how urgent it is for me to migrate to inode64.
you realize, 'migrate' is trivial?
Yes. I also realize that I have NFS clients and am exporting a subdirectory, so I want to reorganize my share to export the top-level directory instead. Since this involves a disruptive change to my clients I need to schedule such work, ideally at the same time as I need to do other disruptive work.
--keith
On 9/26/2013 6:54 PM, Keith Keller wrote:
Yes. I also realize that I have NFS clients and am exporting a subdirectory, so I want to reorganize my share to export the top-level directory instead. Since this involves a disruptive change to my clients I need to schedule such work, ideally at the same time as I need to do other disruptive work.
or just specify fsid=1, 2, 3 on the various exports.
On 09/26/2013 06:28 PM, John R Pierce wrote:
On 9/26/2013 2:27 PM, Keith Keller wrote:
so that I know how urgent it is for me to migrate to inode64.
you realize, 'migrate' is trivial?
mount -o remount,inode64 /dev/whatever
and replace 'defaults' with 'inode64' on the line in /etc/fstab so it does it next time too.
interesting. and easier than what i suggested.
i was not aware of usage of 'inode64' until reading your post and reading 'man mount'.
i do not find such in 'man fstab'.
would you please explain on using 'inode64' in 'fstab'.
thank you.
On 2013-09-27, g geleem@bellsouth.net wrote:
would you please explain on using 'inode64' in 'fstab'.
It's a mount option, so goes in the fourth column with the other mount options (comma-separated).
--keith
On 09/26/2013 10:44 PM, Keith Keller wrote:
On 2013-09-27, g geleem@bellsouth.net wrote:
would you please explain on using 'inode64' in 'fstab'.
It's a mount option, so goes in the fourth column with the other mount options (comma-separated).
yes, i understand mount options. i do not find it listed in 'man fstab' for the clone that i am currently using.
looks like i will have to wait until i get centos iso downloaded and installed so i can move out of os i am currently using.
thanks for your reply.
On 9/26/2013 9:30 PM, g wrote:
yes, i understand mount options. i do not find it listed in 'man fstab' for the clone that i am currently using.
looks like i will have to wait until i get centos iso downloaded and installed so i can move out of os i am currently using.
its an xfs specific mount option. see the man pages for mount (8), under *Mount options for xfs*...
On 09/26/2013 11:54 PM, John R Pierce wrote:
On 9/26/2013 9:30 PM, g wrote:
yes, i understand mount options. i do not find it listed in
^^^^^^^^^^^^^
'man fstab' for the clone that i am currently using.
^^^^^^^^^
looks like i will have to wait until i get centos iso downloaded and installed so i can move out of os i am currently using.
its an xfs specific mount option. see the man pages for mount (8), under *Mount options for xfs*...
yes, as stated above;
i understand _mount_options_. i do not find it in "man fstab".
On 09/27/2013 12:53 AM, g wrote:
On 09/26/2013 11:54 PM, John R Pierce wrote:
On 9/26/2013 9:30 PM, g wrote:
yes, i understand mount options. i do not find it listed in
^^^^^^^^^^^^^'man fstab' for the clone that i am currently using.
^^^^^^^^^looks like i will have to wait until i get centos iso downloaded and installed so i can move out of os i am currently using.
its an xfs specific mount option. see the man pages for mount (8), under *Mount options for xfs*...
yes, as stated above;
i understand _mount_options_. i do not find it in "man fstab".
in addition, something crossing my mind, this is breaking op's thread, so for now, lets drop this 'ravel'. :=X
after i finish pulling centos iso, and get 6.4 installed, i will post a question. :=)
thank you all.
On 9/26/2013 10:53 PM, g wrote:
its an xfs specific mount option. see the man pages for mount (8),
under*Mount options for xfs*...
yes, as stated above;
i understand_mount_options_. i do not find it in "man fstab".
from man fstab,
The fourth field, (fs_mntops), describes the mount options associated with the filesystem.
It is formatted as a comma separated list of options. It contains at least the type of mount plus any addi- tional options appropriate to the filesystem type. *For documentation on the available options for non-nfs file** ** systems, see mount(8)*. For documentation on all nfs-specific options have a look at nfs(5). Common for all types of file system are the options noauto (do not mount when "mount -a" is given, e.g., at boot time), user (allow a user to mount), and owner (allow device owner to mount), and comment (e.g., for use by fstab-maintaining programs). The owner and comment options are Linux-specific. For more details, see mount(8).
-
g -
John R Pierce -
Keith Keller -
Nux! -
Peter Wood