Hi!
During the last weeks I experienced some performance problems with a large file-system on XFS basis. Sometimes for instance ls is painfully. Immidiatly afterwards ls on the same directory is immidiate. I used strace on this ls and found that during the first ls the lstat-calls need approx 0.02s each while during the second ls the are two orders of magnitude faster.
Googling around I stumbled upon some messages similar like this
http://www.opensubscriber.com/message/linux-xfs@oss.sgi.com/1355060.html
which have in common a) they're from around 2006 b) they suggest to increase a mount-option ihashsize. This mount option is listed as deprecated in the current kernel-doc
So my question: does anyone have experience with that kind of performance problem? Do you think it is a XFS problem or are there some other tuning parameters in the kernel that could be modified for instance via /proc?
The reason why I'm asking here is that it is a production file-system so I would be very unpopular if I experiment too much (a couple of reboots is OK ;) )
Bernhard
PS: the situation got worse during the last weeks when the file-system increased in size, so the option that some kind of buffer now is too small and I'm experiencing some kind of thrashing seems very likely to me
On Apr 9, 2010, at 9:59 AM, Bernhard Gschaider <bgschaid_lists@ice- sf.at> wrote:
Hi!
During the last weeks I experienced some performance problems with a large file-system on XFS basis. Sometimes for instance ls is painfully. Immidiatly afterwards ls on the same directory is immidiate. I used strace on this ls and found that during the first ls the lstat-calls need approx 0.02s each while during the second ls the are two orders of magnitude faster.
Googling around I stumbled upon some messages similar like this
http://www.opensubscriber.com/message/linux-xfs@oss.sgi.com/1355060.html
which have in common a) they're from around 2006 b) they suggest to increase a mount-option ihashsize. This mount option is listed as deprecated in the current kernel-doc
So my question: does anyone have experience with that kind of performance problem? Do you think it is a XFS problem or are there some other tuning parameters in the kernel that could be modified for instance via /proc?
The reason why I'm asking here is that it is a production file-system so I would be very unpopular if I experiment too much (a couple of reboots is OK ;) )
Bernhard
PS: the situation got worse during the last weeks when the file-system increased in size, so the option that some kind of buffer now is too small and I'm experiencing some kind of thrashing seems very likely to me
Are you defragging the file system regularly?
How much memory do you have in the system and how big is the file system?
What are the XFS parameters for the file system?
What is the storage setup?
Need the info.
-Ross
On Fri, 9 Apr 2010 10:59:02 -0400 "RW" == Ross Walker rswwalker@gmail.com wrote:
RW> On Apr 9, 2010, at 9:59 AM, Bernhard Gschaider RW> <bgschaid_lists@ice- sf.at> wrote:
>> Hi! >> >> During the last weeks I experienced some performance problems >> with a large file-system on XFS basis. Sometimes for instance >> ls is painfully. Immidiatly afterwards ls on the same directory >> is immidiate. I used strace on this ls and found that during >> the first ls the lstat-calls need approx 0.02s each while >> during the second ls the are two orders of magnitude faster. >> >> Googling around I stumbled upon some messages similar like this >> >> http://www.opensubscriber.com/message/linux-xfs@oss.sgi.com/1355060.html >> >> which have in common a) they're from around 2006 b) they >> suggest to increase a mount-option ihashsize. This mount option >> is listed as deprecated in the current kernel-doc >> >> So my question: does anyone have experience with that kind of >> performance problem? Do you think it is a XFS problem or are >> there some other tuning parameters in the kernel that could be >> modified for instance via /proc? >> >> The reason why I'm asking here is that it is a production >> file-system so I would be very unpopular if I experiment too >> much (a couple of reboots is OK ;) ) >> >> Bernhard >> >> PS: the situation got worse during the last weeks when the >> file-system increased in size, so the option that some kind of >> buffer now is too small and I'm experiencing some kind of >> thrashing seems very likely to me
RW> Are you defragging the file system regularly?
Uups. Never occured to me ("Fragmentation is soooo Windoze") Had a look:
xfs_db> frag actual 6349355, ideal 4865683, fragmentation factor 23.37%
This seems significant.
RW> How much memory do you have in the system and how big is the RW> file system?
Memory on the system is 4Gig (2 DualCore Xenons). The filesystem is 3.5 TB of which 740 Gig are used. Which is the maximum amount used during the one year that the filesystem is being used (that is why the high fragmentation amazes me)
RW> What are the XFS parameters for the file system?
Is this sufficent?
% xfs_info /raid meta-data=/dev/VolGroup00/LogVol05 isize=256 agcount=32, agsize=29434880 blks = sectsz=512 attr=0 data = bsize=4096 blocks=941916160, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0
RW> What is the storage setup?
The filesystem is on a LVM-Volume which sits on a RAID 5 (Hardware RAID) drive
RW> Need the info.
So the way to go forward would be using xfs_fsr on that drive. I read some horror stories about lost files, are these to be taken seriously (I mean they were in some Ubuntu forums ;) )
Any other thoughts on parameters?
Thanks for your time
Bernhard
-
Bernhard Gschaider -
Ross Walker