On Mon, Jun 2, 2008 at 2:03 PM, Johnny Hughes johnny@centos.org wrote:
I would also not use XFS in production ... but that is just me.
Interesting, I thought that XFS was fairly safe for use. What would you recommend for filesystems in the 50-500 terabyte range?
(And yes, we do actually run a 70 TB at the moment, so I'm not asking just to annoy you; I'm genuinely interested in your opinion as well as those of others, so feel free to chip in)
BR Bent
On Mon, Jun 2, 2008 at 8:18 AM, Bent Terp bent@nagstrup.dk wrote:
Interesting, I thought that XFS was fairly safe for use. What would you recommend for filesystems in the 50-500 terabyte range?
I would recommend you split it in several smaller (2-4TB) filesystems. Most applications would support this, and with some clever tactics you might overcome this in applications that do not support it directly.
Disadvantages of using one huge filesystem (independent of filesystem type): - If it breaks, you lose *ALL* your data. - If you need to check the filesystem (fsck), it will take ages. - You cannot easily scale horizontally by moving some of the data to a second machine. - It's much harder to tell what is causing performance problems.
I never had any filesystem over 5TB in my life (and I've managed more than 100TB at one site), and these days my "sanity" limit is around 1TB per filesystem.
I'm currently using XFS in production (we have some SuSE machines just for the support of it), but I've had so many problems that would have been avoided if we weren't using XFS, that I'm seriously considering migrating them all to ext3. (Bonus points for getting rid of that SuSE trash and replacing them with some shiny CentOS 5 machines.)
HTH, Filipe
Filipe Brandenburger wrote:
On Mon, Jun 2, 2008 at 8:18 AM, Bent Terp bent@nagstrup.dk wrote:
Interesting, I thought that XFS was fairly safe for use. What would you recommend for filesystems in the 50-500 terabyte range?
I would recommend you split it in several smaller (2-4TB) filesystems. Most applications would support this, and with some clever tactics you might overcome this in applications that do not support it directly.
Disadvantages of using one huge filesystem (independent of filesystem type):
- If it breaks, you lose *ALL* your data.
- If you need to check the filesystem (fsck), it will take ages.
- You cannot easily scale horizontally by moving some of the data to a
second machine.
- It's much harder to tell what is causing performance problems.
I never had any filesystem over 5TB in my life (and I've managed more than 100TB at one site), and these days my "sanity" limit is around 1TB per filesystem.
I'm currently using XFS in production (we have some SuSE machines just for the support of it), but I've had so many problems that would have been avoided if we weren't using XFS, that I'm seriously considering migrating them all to ext3. (Bonus points for getting rid of that SuSE trash and replacing them with some shiny CentOS 5 machines.)
Are these 64 bit machines?
Bent Terp wrote:
On Mon, Jun 2, 2008 at 2:03 PM, Johnny Hughes johnny@centos.org wrote:
I would also not use XFS in production ... but that is just me.
Interesting, I thought that XFS was fairly safe for use. What would you recommend for filesystems in the 50-500 terabyte range?
(And yes, we do actually run a 70 TB at the moment, so I'm not asking just to annoy you; I'm genuinely interested in your opinion as well as those of others, so feel free to chip in)
Assuming you still want those all-in-one file systems then you may want to look at JFS as I have heard good things about both it's stability and performance.
Is there anyone running JFS currently that can attest to that?
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
On Mon, Jun 2, 2008 at 6:44 AM, Ross S. W. Walker rwalker@medallion.com wrote:
Assuming you still want those all-in-one file systems then you may want to look at JFS as I have heard good things about both it's stability and performance.
Is there anyone running JFS currently that can attest to that?
I am not using JFS now, but a year ago when I was at Datallegro, we were using JFS because we had severe problems with XFS on both SuSE and CentOS. IIRC, the main problem was that performance was sluggish with XFS over fiber channel, and XFS was not as stable then as (I would hope) it is now, on either system. The main reason they were using SuSE was that Novell came out with version 10.2 with the 2.6.18 kernel about a month or two before CentOS did, and they (DA) already had a SuSE kernel set up with the infiniband drivers they needed for the hardware.
FTR, Datallegro does massively parallel processing database appliances, like Teradata, but cheaper, faster and in direct competition with Teradata, which has the name and the history. A 5TB DB is really small for both of them, but it was spread out over numerous nodes within the appliance, so there was no specific file system with anything that large all in one place.
HTH
mhr
On Mon, Jun 02, 2008 at 10:14:24AM -0700, MHR wrote:
On Mon, Jun 2, 2008 at 6:44 AM, Ross S. W. Walker rwalker@medallion.com wrote:
Assuming you still want those all-in-one file systems then you may want to look at JFS as I have heard good things about both it's stability and performance.
Is there anyone running JFS currently that can attest to that?
To grandparent poster-- we are using JFS here (CentOS4 x86_64 and 10TB volume via LVM). It's been running flawlessly and gets heavy use as an Oracle backup server.
Ray
-
Bent Terp -
Filipe Brandenburger -
Les Mikesell -
MHR -
Ray Van Dolson -
Ross S. W. Walker