On Thu, 2014-02-06 at 11:26 -0500, m.roth at 5-cent.us wrote: > Pat Haley wrote: > > > > My system did not recognize the delaylog option, > > but when I mounted with nobarrier,inode64 > > things worked and I was able to write to the > > array! > > > a) please don't top post. > b) a couple of things: first, is your system on a UPS? barrier is > *supposed* to help make transactions atomic, > so that files/dbs are not left in an undefined state in case of > power blip/outage; second, it will *VERY* > much speed up your NFS writes with nobarrier. We finally started > moving people from home directories on > 5.x to 6.x when we found that... with 5.x, uncompressing and > untarring a 25MB file that expanded to about > 105M, on an NFS-mounted drive was about 30 sec, while on 6.x it ran > around 7 MINUTES. Then we found > nobarrier... and it went down to pretty much what it had been on 5.x. > > mark and please DO NOT forget to eliminate all the unwanted text :-) Thanks, > > > Thanks! > > > > Pat > > > >> change mount to nobarrier,inode64,delaylog > >> > >> ----- Original Message ----- > >> | > >> | Hi: > >> | > >> | > On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley <phaley at mit.edu> wrote: > >> | > > >> | >> Hi, > >> | >> > >> | >> I have a server running under CentOS 5.8 and I appear to be in a > >> | >> situation in which the NFS file server is not recognizing the > >> | >> available > >> | >> space on a particular disk (actually a hardware RAID-6 of 13 2Tb > >> | >> disks). > >> | >> If I try to write to the disk I get the following error message > >> | >> > >> | >> [root at nas-0-1 mseas-data-0-1]# touch dum > >> | >> touch: cannot touch `dum': No space left on device > >> | >> > >> | >> However, if I check the available space, there seems to > >> | >> be plenty > >> | >> > >> | >> [root at nas-0-1 mseas-data-0-1]# df -h . > >> | >> Filesystem Size Used Avail Use% Mounted on > >> | >> /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1 > >> | >> > >> | > > >> | > Maybe you're hitting the allocation of reserved blocks for root? > >> | > With your disk usage of 97% I'd think that could be the case. > >> | > > >> | > You didn't say what file system you're using for that 21TB array, > >> | > so we > >> | > (this list) won't be of too much help without knowing that. > >> | > >> | xfs file system. The fstab line for this array is: > >> | > >> | /dev/sdb1 /mseas-data-0-1 xfs defaults > >> | 1 0 > >> | > >> | > > >> | > tune2fs [0] is your friend > >> | > >> | if I read the man pages correctly tune2fs will not work for xfs. > >> | From xfs_info I get the following for mseas-data-0-1 > >> | > >> | [root at nas-0-1 mseas-data-0-1]# xfs_info . > >> | meta-data=/dev/sdb1 isize=256 agcount=32, > >> | agsize=167846667 blks > >> | = sectsz=512 attr=1 > >> | data = bsize=4096 blocks=5371093344, > >> | imaxpct=25 > >> | = sunit=0 swidth=0 blks, > >> | unwritten=1 > >> | naming =version 2 bsize=4096 > >> | log =internal bsize=4096 blocks=32768, version=1 > >> | = sectsz=512 sunit=0 blks, > >> | lazy-count=0 > >> | realtime =none extsz=4096 blocks=0, rtextents=0 > >> | > >> | Unfortunately, I don't know how to interpret this or if > >> | it is giving relevant information to the question at hand> > >> | > >> | > - use it to determine if there are reserved blocks > >> | > - use it to adjust the settings > >> | > > >> | > [0] > >> | > https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks > >> | > > >> | > > >> | >> [root at nas-0-1 mseas-data-0-1]# df -i . > >> | >> Filesystem Inodes IUsed IFree IUse% Mounted on > >> | >> /dev/sdb1 3290047552 4391552 3285656000 1% > >> | >> /mseas-data-0-1 > >> | >> > >> | >> I don't know if the following is relevant but the disk in question > >> | >> is served as one of 3 bricks in a gluster namespace. > >> | >> > >> | >> Based on the test with touch, which is happening directly > >> | >> at the NFS level, this seems to be an NFS rather than gluster > >> | >> issue. I couldn't find any file in /var/log which had a > >> | >> time that corresponded to the failed touch test and I didn't > >> | >> see anything in dmesg. We have tried rebooting this system. > >> | >> What else should we look at and/or try to resolve or debug > >> | >> this issue? > >> | >> > >> | > > >> | > If you have a non-root shell account on that box, can you write to > >> | > that > >> | > array from the NFS host? > >> | > ( Take NFS out of the equation. ) > >> | > >> | Unfortunately we only have a root account on that box. > >> | > >> | > > >> | > > >> | >> Thanks. > >> | >> > >> | >> Pat > >> | >> > >> | >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > >> | >> Pat Haley Email: phaley at mit.edu > >> | >> Center for Ocean Engineering Phone: (617) 253-6824 > >> | >> Dept. of Mechanical Engineering Fax: (617) 253-8125 > >> | >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ > >> | >> 77 Massachusetts Avenue > >> | >> Cambridge, MA 02139-4301 > >> | >> _______________________________________________ > >> | >> CentOS mailing list > >> | >> CentOS at centos.org > >> | >> http://lists.centos.org/mailman/listinfo/centos > >> | >> > >> | > > >> | > > >> | > > >> | > >> | > >> | -- > >> | > >> | -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > >> | Pat Haley Email: phaley at mit.edu > >> | Center for Ocean Engineering Phone: (617) 253-6824 > >> | Dept. of Mechanical Engineering Fax: (617) 253-8125 > >> | MIT, Room 5-213 http://web.mit.edu/phaley/www/ > >> | 77 Massachusetts Avenue > >> | Cambridge, MA 02139-4301 > >> | _______________________________________________ > >> | CentOS mailing list > >> | CentOS at centos.org > >> | http://lists.centos.org/mailman/listinfo/centos > >> | > >> > > > > > > -- > > > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > > Pat Haley Email: phaley at mit.edu > > Center for Ocean Engineering Phone: (617) 253-6824 > > Dept. of Mechanical Engineering Fax: (617) 253-8125 > > MIT, Room 5-213 http://web.mit.edu/phaley/www/ > > 77 Massachusetts Avenue > > Cambridge, MA 02139-4301 > > _______________________________________________ > > CentOS mailing list > > CentOS at centos.org > > http://lists.centos.org/mailman/listinfo/centos > > > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos CUT OUT TEXT NOT RELEVANT. Many thanks. -- Paul. England, EU. Our systems are exclusively Linux. No Micro$oft Windoze here.