we recently deployed some large XFS file systems with centos 6.2 used as NFS servers...
I've had some reports of a problem similar to the one reported here... http://www.linuxquestions.org/questions/red-hat-31/xfs-inode64-nfs-export-no...
these reports are somewhat vague (third indirectly reported via internal corporate channels from sites in the far east).
is anyone familiar with this inode64 stuff and NFS in EL6.2 ?
On 03/02/12 10:05 AM, John R Pierce wrote:
is anyone familiar with this inode64 stuff and NFS in EL6.2 ?
still having show stopping issues with XFS, inode64, and NFS. the fsid=uuid option suggested by some googling doesn't seem to work at all for Solaris clients... http://xfs.org/index.php/XFS_FAQ#Q:_Why_doesn.27t_NFS-exporting_subdirectori...
On nfs server...
[root@server data]# mount /dev/sda1 on /data type xfs (rw,inode64)
[root@server data]# blkid /dev/sda1: UUID="d1bff6f7-9887-44bb-be57-d8628831cf40" TYPE="xfs"
[root@server data]# cat /etc/exports /data/oni64 *(rw,async,fsid=d1bff6f7-9887-44bb-be57-d8628831cf40,no_subtree_check)
[root@server data]# uname -a Linux server 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 04:00:16 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
With the option fsid=uuid we could not mount nfs on client.
root@client1 # mount -F nfs server:/data/oni64 /backups nfs mount: tkddo-clf08:/data/oni64: No such file or directory
root@client1 # mount -F nfs -o vers=4 server:/data/oni64 /backups nfs mount: tkddo-clf08:/data/oni64: No such file or directory
root@client1 # mount -F nfs -o vers=3 server:/data/oni64 /backups nfs mount: mount: /backups: Stale NFS file handle
root@client1 # uname -a SunOS client1 5.10 Generic_138889-08 i86pc i386 i86pc
(this last is the same error we got without the fsid=, when using inode64).
It *does* seem to work if the export uses fsid=# where # is an arbitrary integer, unique for each export, but thats something of a messy workaround.
On Mar 6, 2012, at 4:36 PM, John R Pierce pierce@hogranch.com wrote:
On 03/02/12 10:05 AM, John R Pierce wrote:
is anyone familiar with this inode64 stuff and NFS in EL6.2 ?
still having show stopping issues with XFS, inode64, and NFS. the fsid=uuid option suggested by some googling doesn't seem to work at all for Solaris clients... http://xfs.org/index.php/XFS_FAQ#Q:_Why_doesn.27t_NFS-exporting_subdirectori...
On nfs server...
[root@server data]# mount /dev/sda1 on /data type xfs (rw,inode64)
[root@server data]# blkid /dev/sda1: UUID="d1bff6f7-9887-44bb-be57-d8628831cf40" TYPE="xfs"
[root@server data]# cat /etc/exports /data/oni64 *(rw,async,fsid=d1bff6f7-9887-44bb-be57-d8628831cf40,no_subtree_check)
[root@server data]# uname -a Linux server 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 04:00:16 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
With the option fsid=uuid we could not mount nfs on client.
root@client1 # mount -F nfs server:/data/oni64 /backups nfs mount: tkddo-clf08:/data/oni64: No such file or directory
root@client1 # mount -F nfs -o vers=4 server:/data/oni64 /backups nfs mount: tkddo-clf08:/data/oni64: No such file or directory
root@client1 # mount -F nfs -o vers=3 server:/data/oni64 /backups nfs mount: mount: /backups: Stale NFS file handle
root@client1 # uname -a SunOS client1 5.10 Generic_138889-08 i86pc i386 i86pc
(this last is the same error we got without the fsid=, when using inode64).
It *does* seem to work if the export uses fsid=# where # is an arbitrary integer, unique for each export, but thats something of a messy workaround.
From what I read as well NFS and XFS inode64 do not play well at all. NFS still has a lot of 32-bit logic hard coded in it, both in the server and client.
Is there a need for inode64? Is it too late to back-out of it, or have you pickled the file system already.
If so then the only option is to set fsid manually.
-Ross
On 03/06/12 3:51 PM, Ross Walker wrote:
Is there a need for inode64? Is it too late to back-out of it, or have you pickled the file system already.
as i undertsand it, you need inode64 on any XFS file system over 2TiB, and ours is 74TiB.
On Mar 6, 2012, at 6:59 PM, John R Pierce pierce@hogranch.com wrote:
On 03/06/12 3:51 PM, Ross Walker wrote:
Is there a need for inode64? Is it too late to back-out of it, or have you pickled the file system already.
as i undertsand it, you need inode64 on any XFS file system over 2TiB, and ours is 74TiB.
Ah, as I understood it it was only needed if the number of inodes fills up what can be held by a 32-bit counter, the documentation is a little weak here.
-Ross
----- Original Message ----- | On Mar 6, 2012, at 6:59 PM, John R Pierce pierce@hogranch.com | wrote: | | > On 03/06/12 3:51 PM, Ross Walker wrote: | >> Is there a need for inode64? Is it too late to back-out of it, or | >> have you pickled the file system already. | > | > as i undertsand it, you need inode64 on any XFS file system over | > 2TiB, | > and ours is 74TiB. | | Ah, as I understood it it was only needed if the number of inodes | fills up what can be held by a 32-bit counter, the documentation is | a little weak here. | | -Ross
I've had to use inode64 on far smaller file systems (15TB) due to inode counts (many files) and not file system size.
On 03/06/12 10:40 PM, James A. Peltier wrote:
I've had to use inode64 on far smaller file systems (15TB) due to inode counts (many files) and not file system size.
My understanding of it is, the inode number is its physical position on the disk. if the first 1TB (2^31 * 512) of the disk has no free space to create an inode for a new directory, then that inode requires 64bits.
http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F
On Mar 7, 2012, at 2:17 AM, John R Pierce pierce@hogranch.com wrote:
On 03/06/12 10:40 PM, James A. Peltier wrote:
I've had to use inode64 on far smaller file systems (15TB) due to inode counts (many files) and not file system size.
My understanding of it is, the inode number is its physical position on the disk. if the first 1TB (2^31 * 512) of the disk has no free space to create an inode for a new directory, then that inode requires 64bits.
Ah, ok, if the inode's location on disk goes above the 1TB mark then inode64 is needed, but not the data it points to, that can exist anywhere.
Problem is that both inode and data can exist in that 1TB and the data can push the inodes into 64-bit land.
These days XFS should always be inode64 enabled, given the size of disks, and NFS should have been fixed a long, long time ago.
-Ross
On 03/07/12 6:47 AM, Ross Walker wrote:
These days XFS should always be inode64 enabled, given the size of disks, and NFS should have been fixed a long, long time ago.
yes. problem is, we have clients that are running all sorts of OS's, including older versions of Solaris... this is a manufacturing environment, so many of the clients are actually test equipment and machine controllers and such, you can't just slap a new OS on those.
On 3/7/2012 11:16 AM, John R Pierce wrote:
On 03/07/12 6:47 AM, Ross Walker wrote:
These days XFS should always be inode64 enabled, given the size of disks, and NFS should have been fixed a long, long time ago.
yes. problem is, we have clients that are running all sorts of OS's, including older versions of Solaris... this is a manufacturing environment, so many of the clients are actually test equipment and machine controllers and such, you can't just slap a new OS on those.
Why do those embedded systems have to see a whole 74 TB storage midden? Why not slice off a 1 TB or so LVM slice for them?
On 03/07/12 11:27 AM, Warren Young wrote:
Why do those embedded systems have to see a whole 74 TB storage midden? Why not slice off a 1 TB or so LVM slice for them?
LVM management blows chunks. each of these arrays (we have dozens of these 74TiB storage servers) will end up nearly filled up. trying to manage 74 separate 1TiB file systems or even 10 separate 7TiB systems, and figuring out where there's freespace? no thanks. there's 1000s of those clients writing files to this archival storage, and the goal is to fill up the storage.
the operations folks at these production sites were using Solaris w/ ZFS for this in the previous (smaller) incarnation.
one solution proposed by a field site is to use SMB instead of NFS, as it doesn't seem to have these issues. I personally find SMB for (l)Unix to (l)Unix distasteful.
On Wed, Mar 7, 2012 at 1:46 PM, John R Pierce pierce@hogranch.com wrote:
On 03/07/12 11:27 AM, Warren Young wrote:
Why do those embedded systems have to see a whole 74 TB storage midden? Why not slice off a 1 TB or so LVM slice for them?
LVM management blows chunks. each of these arrays (we have dozens of these 74TiB storage servers) will end up nearly filled up. trying to manage 74 separate 1TiB file systems or even 10 separate 7TiB systems, and figuring out where there's freespace? no thanks. there's 1000s of those clients writing files to this archival storage, and the goal is to fill up the storage.
the operations folks at these production sites were using Solaris w/ ZFS for this in the previous (smaller) incarnation.
one solution proposed by a field site is to use SMB instead of NFS, as it doesn't seem to have these issues. I personally find SMB for (l)Unix to (l)Unix distasteful.
I can see where zfs would be good for this... Does this data have some sort of life cycle where you could give the devices a small space to write with the data periodically processed (server-side) into the large archive? Or could they use an http post, database write, or some other application-level protocol instead of seeing the filesystem directly?
On Mar 7, 2012, at 1:16 PM, John R Pierce pierce@hogranch.com wrote:
On 03/07/12 6:47 AM, Ross Walker wrote:
These days XFS should always be inode64 enabled, given the size of disks, and NFS should have been fixed a long, long time ago.
yes. problem is, we have clients that are running all sorts of OS's, including older versions of Solaris... this is a manufacturing environment, so many of the clients are actually test equipment and machine controllers and such, you can't just slap a new OS on those.
I see, then here's hoping there is a server side fix for this somewhere.
Otherwise, I suggest using Solaris/Nexenta and ZFS for this big storage. Not only do you get an almost unbreakable file system, but industry standard NFS server.
That's where I'd go, especially given the heterogenous clients you have to deal with.
-Ross
-
James A. Peltier -
John R Pierce -
Les Mikesell -
Ross Walker -
Warren Young