On Tue, 21 Jan 2014, Keith Keller wrote: > On 2014-01-21, Steve Brooks <steveb at mcs.st-and.ac.uk> wrote: >> >>> mkfs.xfs -d su=512k,sw=14 /dev/sda >> >> where "512k" is the Stripe-unit size of the single logical device built on >> the raid controller. "14" is from the total number of drives minus two >> (raid 6 redundancy). > > The usual advice on the XFS list is to use the defaults where possible. > But you might want to ask there to see if they have any specific advice. Thanks for the reply Keith. Yes I will ask on the list, I did read that when built on mdadm raid devices it is geared up to tune itself but with hardware raid it may take manual tuning. >> I mounted the filesystem with the default options assuming they would be >> sensible but I now believe I should have specified the "inode64" mount >> option to avoid all the inodes will being stuck in the first TB. >> >> The filesystem however is at 87% and does not seem to have had any >> issues/problems. >> >>> df -h | grep raid >> /dev/sda 51T 45T 6.7T 87% /raidstor > Wow, impressive! I know of a much smaller fs which got bit by this > issue. What probably happened is, as a new fs, the entire first 1TB was > able to be reserved for inodes. Yes and the output of "df -i" shows only Filesystem Inodes IUsed IFree IUse% /dev/sda 2187329088 189621 2187139467 1% So few inodes are used because the data is from "hpc" used to run MHD (magneto hydro-dynamics) simulations on the Sun many of the files are snapshots of the simulation at various instances "93G" in size etc. >> Another question is could I now safely remount with the "inode64" option >> or will this cause problems in the future? I read this below in the XFS >> FAQ but wondered if it have been fixed (backported?) into el6.4? > > I have mounted a large XFS fs that previously didn't use inode64 with > it, and it went fine. (I did not attempt to roll back.) You *must* > umount and remount for this option to take effect. I do not know when > the inode64 option made it to CentOS, but it is there now. Ok so I am sort of wondering for this filesystem if it is actually worth it given lack of inodes does not look like it will be an issue. >> I also noted that "xfs_check" ran out of memory and so after some reading >> noted that it is reccommended to use "xfs_repair -n -vv" instead as it >> uses far less memory. One remark is so why is "xfs_check" there at all? > > The XFS team is working on deprecating it. But on a 51TB filesystem > xfs_repair will still use a lot of memory. Using -P can help, but it'll > still use quite a bit (depending on the extent of any damage and how many > inodes, probably a bunch of other factors I don't know). Yes this bothers me a bit, I issued a " xfs_repair -n -vv" and that told me I only needed "6G" I guess with only a few inodes and a clean filesystem it makes sense. I did read a good solution on the XFS mailing list which seems really neat.. "Add an SSD of sufficient size/speed for swap duty to handle xfs_repair requirements for filesystems with arbitrarily high inode counts. Create a 100GB swap partition and leave the remainder unallocated. The unallocated space will automatically be used for GC and wear leveling, increasing the life of all cells in the drive." Steve