Just to close this thread and remove any doubt that it might have raised about XFS: the problem was a PEBCAB[1] It was pointed out to me on the XFS-list that the device I used for xfs_db was inconsistent with the info from xfs_info (I was blindly copying the device from the output of df) Footnotes: [1] PEBCAB: Problem exists between chair and keyboard >>>>> On Tue, 13 Apr 2010 11:54:53 +0200 >>>>> "BG" == Bernhard Gschaider <bgschaid_lists at ice-sf.at> wrote: BG> Before I'd try to defragment my whole filesystem (see attached BG> mail for whole story) I figured "Let's try it on some file". BG> So I did >> xfs_bmap /raid/Temp/someDiskimage.iso BG> [output shows 101 extents and 1 hole] BG> Then I defragmented the file >> xfs_fsr /raid/Temp/someDiskimage.iso BG> extents before:101 after:3 DONE >> xfs_bmap /raid/Temp/someDiskimage.iso BG> [output shows 3 extents and 1 hole] BG> and now comes the bummer: i wanted to check the fragmentation BG> of the whole filesystem (just for checking): >> xfs_db -r /dev/mapper/VolGroup00-LogVol04 BG> xfs_db: unexpected XFS SB magic number 0x00000000 xfs_db: read BG> failed: Invalid argument xfs_db: data size check failed BG> cache_node_purge: refcount was 1, not zero (node=0x2a25c20) BG> xfs_db: cannot read root inode (22) BG> THAT output was definitly not there when I did this the last BG> time and therefor the new fragmentation does not make me happy BG> either xfs_db> frag BG> actual 0, ideal 0, fragmentation factor 0.00% BG> The file-system is still mounted and working and I don't dare BG> to do anything about it (am in a mild state of panic) because BG> I think it might not come back if I do. BG> Any suggestions most welcome (am googling myself before I do BG> anything about it). BG> I swear to god: I did not do anything else with the BG> xfs_*-commands than the stuff mentioned above BG> Bernhard BG> From: Bernhard Gschaider <bgschaid_lists at ice-sf.at> Subject: BG> Re: [CentOS] Performance problems with XFS on Centos 5.4 To: BG> CentOS mailing list <centos at centos.org> Date: Mon, 12 Apr 2010 BG> 18:22:24 +0200 Organization: ICE Stroemungsforschung Reply-To: BG> CentOS mailing list <centos at centos.org> >>>>> On Fri, 9 Apr 2010 10:59:02 -0400 >>>>> "RW" == Ross Walker <rswwalker at gmail.com> wrote: RW> On Apr 9, 2010, at 9:59 AM, Bernhard Gschaider RW> <bgschaid_lists at ice- sf.at> wrote: >>> Hi! >>> >>> During the last weeks I experienced some performance problems >>> with a large file-system on XFS basis. Sometimes for instance >>> ls is painfully. Immidiatly afterwards ls on the same >>> directory is immidiate. I used strace on this ls and found >>> that during the first ls the lstat-calls need approx 0.02s >>> each while during the second ls the are two orders of >>> magnitude faster. >>> >>> Googling around I stumbled upon some messages similar like >>> this >>> >>> http://www.opensubscriber.com/message/linux-xfs@oss.sgi.com/1355060.html >>> >>> which have in common a) they're from around 2006 b) they >>> suggest to increase a mount-option ihashsize. This mount >>> option is listed as deprecated in the current kernel-doc >>> >>> So my question: does anyone have experience with that kind of >>> performance problem? Do you think it is a XFS problem or are >>> there some other tuning parameters in the kernel that could be >>> modified for instance via /proc? >>> >>> The reason why I'm asking here is that it is a production >>> file-system so I would be very unpopular if I experiment too >>> much (a couple of reboots is OK ;) ) >>> >>> Bernhard >>> >>> PS: the situation got worse during the last weeks when the >>> file-system increased in size, so the option that some kind of >>> buffer now is too small and I'm experiencing some kind of >>> thrashing seems very likely to me RW> Are you defragging the file system regularly? BG> Uups. Never occured to me ("Fragmentation is soooo Windoze") BG> Had a look: xfs_db> frag BG> actual 6349355, ideal 4865683, fragmentation factor 23.37% BG> This seems significant. RW> How much memory do you have in the system and how big is the RW> file system? BG> Memory on the system is 4Gig (2 DualCore Xenons). The BG> filesystem is 3.5 TB of which 740 Gig are used. Which is the BG> maximum amount used during the one year that the filesystem is BG> being used (that is why the high fragmentation amazes me) RW> What are the XFS parameters for the file system? BG> Is this sufficent? BG> % xfs_info /raid meta-data=/dev/VolGroup00/LogVol05 isize=256 BG> agcount=32, agsize=29434880 blks = sectsz=512 attr=0 data = BG> bsize=4096 blocks=941916160, imaxpct=25 = sunit=0 swidth=0 BG> blks, unwritten=1 naming =version 2 bsize=4096 log =internal BG> bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks, BG> lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 RW> What is the storage setup? BG> The filesystem is on a LVM-Volume which sits on a RAID 5 BG> (Hardware RAID) drive RW> Need the info. BG> So the way to go forward would be using xfs_fsr on that BG> drive. I read some horror stories about lost files, are these BG> to be taken seriously (I mean they were in some Ubuntu forums BG> ;) ) BG> Any other thoughts on parameters? BG> Thanks for your time BG> Bernhard _______________________________________________ BG> CentOS mailing list CentOS at centos.org BG> http://lists.centos.org/mailman/listinfo/centos BG> ---------- BG> _______________________________________________ CentOS mailing BG> list CentOS at centos.org BG> http://lists.centos.org/mailman/listinfo/centos -- --------------------------------------------------------------------------- DI Bernhard F.W. Gschaider --------------------------------------------------------------------------- EMail: Bernhard.Gschaider at ice-sf.at WWW : www.ice-sf.at Jabber : bgschaid at jabber.org Tel: +43(3842)98282-42 Fax: +43(3842)98282-02 ---------------------------------------------------------------------------