We're looking at getting an HBR (that's a technical term, honkin' big RAID). What I'm considering is, rather than chopping it up into 14TB or 16TB filesystems, of using xfs for really big filesystems. The question that's come up is: what's the state of xfs on CentOS6? I've seen a number of older threads seeing problems with it - has that mostly been resolved? How does it work if we have some *huge* files, and lots and lots of smaller files?
mark
On 2014-05-28, m.roth@5-cent.us m.roth@5-cent.us wrote:
We're looking at getting an HBR (that's a technical term, honkin' big RAID). What I'm considering is, rather than chopping it up into 14TB or 16TB filesystems, of using xfs for really big filesystems. The question that's come up is: what's the state of xfs on CentOS6? I've seen a number of older threads seeing problems with it - has that mostly been resolved?
I have some largish (>20TB) xfs filesystems on CentOS 6, and things seem fine. The one issue I had, quite a while ago (and maybe even in CentOS 5?), was with growing the fs, but I grew one on CentOS 6 recently with no problems.
How does it work if we have some *huge* files, and lots and lots of smaller files?
Define "huge". It seems fine for our use with multi-dozen-GB files (possibly getting to >100GB files) and many small files, but our load is generally not that heavy.
--keith
Keith Keller wrote:
On 2014-05-28, m.roth@5-cent.us m.roth@5-cent.us wrote:
We're looking at getting an HBR (that's a technical term, honkin' big RAID). What I'm considering is, rather than chopping it up into 14TB or 16TB filesystems, of using xfs for really big filesystems. The question that's come up is: what's the state of xfs on CentOS6? I've seen a number of older threads seeing problems with it - has that mostly been resolved?
I have some largish (>20TB) xfs filesystems on CentOS 6, and things seem fine. The one issue I had, quite a while ago (and maybe even in CentOS 5?), was with growing the fs, but I grew one on CentOS 6 recently with no problems.
How does it work if we have some *huge* files, and lots and lots of smaller files?
Define "huge". It seems fine for our use with multi-dozen-GB files (possibly getting to >100GB files) and many small files, but our load is generally not that heavy.
At the moment, files that are tens of gigs, but I would not be at *all* surprised to see another decimal point there in the next year or two. HBR - a Jetstor 742 with 42 4TB drives.... I'm assuming they'll want me to do RAID 6 for this, as we've been doing on other RAIDs.
mark
On 2014-05-28, m.roth@5-cent.us m.roth@5-cent.us wrote:
At the moment, files that are tens of gigs, but I would not be at *all* surprised to see another decimal point there in the next year or two. HBR
- a Jetstor 742 with 42 4TB drives.... I'm assuming they'll want me to do
RAID 6 for this, as we've been doing on other RAIDs.
That's significantly bigger than what I have running, but I have heard of people running XFS on larger filesystems than I have.
If you're making one large filesystem out of that, the warning James mentioned is even more important: you will want a boatload of memory to xfs_repair that fs quickly. The rough rule of thumb I've read (which is probably akin to the 2x RAM==swap guideline) is at least 1GB of memory for every 1TB of storage, but the more the better, I'd wager.
--keith
----- Original Message ----- | We're looking at getting an HBR (that's a technical term, honkin' big | RAID). What I'm considering is, rather than chopping it up into 14TB | or | 16TB filesystems, of using xfs for really big filesystems. The | question | that's come up is: what's the state of xfs on CentOS6? I've seen a | number | of older threads seeing problems with it - has that mostly been | resolved? | How does it work if we have some *huge* files, and lots and lots of | smaller files? | | mark |
It works well and will be the default for RHEL 7, so that should say something. However, with respect to file system size. Be prepared to have a lot of memory for a file system that is large and a lot of files should you ever need to do a file system check. ;)
On 5/28/2014 11:13 AM, m.roth@5-cent.us wrote:
We're looking at getting an HBR (that's a technical term, honkin' big RAID). What I'm considering is, rather than chopping it up into 14TB or 16TB filesystems, of using xfs for really big filesystems. The question that's come up is: what's the state of xfs on CentOS6? I've seen a number of older threads seeing problems with it - has that mostly been resolved? How does it work if we have some*huge* files, and lots and lots of smaller files?
I've had good luck with XFS file systems of 80TB or so for nearline archival storage. thats 36 3TB SAS drives organized as 3 x 11 raid6+0 with 3 hotspares.
found two minor(?) gotchas so far with XFS
1) NFS doesn't like 64bit inodes. you can A) only nfs share the root of the giant XFS file system (this *is* the traditional way, but people from a Windows background seem to like to micromanage their shares), or B) use UUID exports (not compatible with all nfs clients in my experience), or C) specify fsid=NNN for a arbitrary unique NNN for each export on a given server. We opted for C.
2) I just discovered the other night that KVM doesn't like booting disk image files stored on xfs on a 4K sector device (in my case, this was an SSD). solution was to specify cache=writeback, which somehow bypasses O_DIRECT. There's probably other fixes, but that works well enough.
also, there was a bad kernel in 6.3 or something, that had a serious bug with XFS. the fix came out 2-3 weeks after 6.3 was released, but I ran into internal operations people who don't update production systems, if you say you tested something on 6.3, then they use 6.3 forever. They pathologically skip my installation step 2, "yum -y update".
----- Original Message ----- | On 5/28/2014 11:13 AM, m.roth@5-cent.us wrote: | > We're looking at getting an HBR (that's a technical term, honkin' | > big | > RAID). What I'm considering is, rather than chopping it up into | > 14TB or | > 16TB filesystems, of using xfs for really big filesystems. The | > question | > that's come up is: what's the state of xfs on CentOS6? I've seen a | > number | > of older threads seeing problems with it - has that mostly been | > resolved? | > How does it work if we have some*huge* files, and lots and lots of | > smaller files? | | | I've had good luck with XFS file systems of 80TB or so for nearline | archival storage. thats 36 3TB SAS drives organized as 3 x 11 | raid6+0 | with 3 hotspares. | | found two minor(?) gotchas so far with XFS | | 1) NFS doesn't like 64bit inodes. you can A) only nfs share the root | of | the giant XFS file system (this *is* the traditional way, but people | from a Windows background seem to like to micromanage their shares), | or | B) use UUID exports (not compatible with all nfs clients in my | experience), or C) specify fsid=NNN for a arbitrary unique NNN for | each | export on a given server. We opted for C. | | 2) I just discovered the other night that KVM doesn't like booting | disk | image files stored on xfs on a 4K sector device (in my case, this was | an | SSD). solution was to specify cache=writeback, which somehow | bypasses | O_DIRECT. There's probably other fixes, but that works well enough. | | also, there was a bad kernel in 6.3 or something, that had a serious | bug | with XFS. the fix came out 2-3 weeks after 6.3 was released, but I | ran into internal operations people who don't update production | systems, | if you say you tested something on 6.3, then they use 6.3 forever. | They pathologically skip my installation step 2, "yum -y update".
Would this be the xfs_asynd chewing up CPU time bug? If that's the one, it was just spinning and didn't actually cause "problems" except that the load went berserk. In either case, there have been bugs with *every file system* I've used with large volumes, but XFS remains my goto file system for all large volumes as it is the best performance/stability match out there.
On 5/28/2014 12:35 PM, James A. Peltier wrote:
Would this be the xfs_asynd chewing up CPU time bug? If that's the one, it was just spinning and didn't actually cause "problems" except that the load went berserk. In either case, there have been bugs with*every file system* I've used with large volumes, but XFS remains my goto file system for all large volumes as it is the best performance/stability match out there.
no, this bug actually caused a big chunk of directories to disappear after a power failure event. I might be wrong about 6.3, it could have been 6.2 or something.
I agree, XFS is my goto linux file system for large volumes. ZFS is my other favorite, but I still don't trust it on Linux.
-
James A. Peltier -
John R Pierce -
Keith Keller -
m.roth＠5-cent.us