[CentOS] NFS not recognizing available file space

Thu Feb 6 16:26:49 UTC 2014
m.roth at 5-cent.us <m.roth at 5-cent.us>

Pat Haley wrote:
>
> My system did not recognize the delaylog option,
> but when I mounted with nobarrier,inode64
> things worked and I was able to write to the
> array!
>
a) please don't top post.
b) a couple of things: first, is your system on a UPS? barrier is
*supposed* to help make transactions atomic,
       so that files/dbs are not left in an undefined state in case of
power blip/outage; second, it will *VERY*
       much speed up your NFS writes with nobarrier. We finally started
moving people from home directories on
       5.x to 6.x when we found that... with 5.x, uncompressing and
untarring a 25MB file that expanded to about
       105M, on an NFS-mounted drive was about 30 sec, while on 6.x it ran
around 7 MINUTES. Then we found
       nobarrier... and it went down to pretty much what it had been on 5.x.

         mark

> Thanks!
>
> Pat
>
>> change mount to nobarrier,inode64,delaylog
>>
>> ----- Original Message -----
>> |
>> | Hi:
>> |
>> | > On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley <phaley at mit.edu> wrote:
>> | >
>> | >> Hi,
>> | >>
>> | >> I have a server running under CentOS 5.8 and I appear to be in a
>> | >> situation in which the NFS file server is not recognizing the
>> | >> available
>> | >> space on a particular disk (actually a hardware RAID-6 of 13 2Tb
>> | >> disks).
>> | >> If I try to write to the disk I get the following error message
>> | >>
>> | >> [root at nas-0-1 mseas-data-0-1]# touch dum
>> | >> touch: cannot touch `dum': No space left on device
>> | >>
>> | >> However, if I check the available space, there seems to
>> | >> be plenty
>> | >>
>> | >> [root at nas-0-1 mseas-data-0-1]# df -h .
>> | >> Filesystem            Size  Used Avail Use% Mounted on
>> | >> /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
>> | >>
>> | >
>> | > Maybe you're hitting the allocation of reserved blocks for root?
>> | > With your disk usage of 97% I'd think that could be the case.
>> | >
>> | > You didn't say what file system you're using for that 21TB array,
>> | > so we
>> | > (this list) won't be of too much help without knowing that.
>> |
>> | xfs file system.  The fstab line for this array is:
>> |
>> | /dev/sdb1               /mseas-data-0-1         xfs     defaults
>> |        1 0
>> |
>> | >
>> | > tune2fs [0] is your friend
>> |
>> | if I read the man pages correctly tune2fs will not work for xfs.
>> |  From xfs_info I get the following for mseas-data-0-1
>> |
>> | [root at nas-0-1 mseas-data-0-1]# xfs_info .
>> | meta-data=/dev/sdb1              isize=256    agcount=32,
>> | agsize=167846667 blks
>> |           =                       sectsz=512   attr=1
>> | data     =                       bsize=4096   blocks=5371093344,
>> | imaxpct=25
>> |           =                       sunit=0      swidth=0 blks,
>> |           unwritten=1
>> | naming   =version 2              bsize=4096
>> | log      =internal               bsize=4096   blocks=32768, version=1
>> |           =                       sectsz=512   sunit=0 blks,
>> |           lazy-count=0
>> | realtime =none                   extsz=4096   blocks=0, rtextents=0
>> |
>> | Unfortunately, I don't know how to interpret this or if
>> | it is giving relevant information to the question at hand>
>> |
>> | > - use it to determine if there are reserved blocks
>> | > - use it to adjust the settings
>> | >
>> | > [0]
>> | > https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks
>> | >
>> | >
>> | >> [root at nas-0-1 mseas-data-0-1]# df -i .
>> | >> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
>> | >> /dev/sdb1            3290047552 4391552 3285656000    1%
>> | >> /mseas-data-0-1
>> | >>
>> | >> I don't know if the following is relevant but the disk in question
>> | >> is served as one of 3 bricks in a gluster namespace.
>> | >>
>> | >> Based on the test with touch, which is happening directly
>> | >> at the NFS level, this seems to be an NFS rather than gluster
>> | >> issue. I couldn't find any file in /var/log which had a
>> | >> time that corresponded to the failed touch test and I didn't
>> | >> see anything in dmesg.  We have tried rebooting this system.
>> | >> What else should we look at and/or try to resolve or debug
>> | >> this issue?
>> | >>
>> | >
>> | > If you have a non-root shell account on that box, can you write to
>> | > that
>> | > array from the NFS host?
>> | > ( Take NFS out of the equation. )
>> |
>> | Unfortunately we only have a root account on that box.
>> |
>> | >
>> | >
>> | >> Thanks.
>> | >>
>> | >> Pat
>> | >>
>> | >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> | >> Pat Haley                          Email:  phaley at mit.edu
>> | >> Center for Ocean Engineering       Phone:  (617) 253-6824
>> | >> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>> | >> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>> | >> 77 Massachusetts Avenue
>> | >> Cambridge, MA  02139-4301
>> | >> _______________________________________________
>> | >> CentOS mailing list
>> | >> CentOS at centos.org
>> | >> http://lists.centos.org/mailman/listinfo/centos
>> | >>
>> | >
>> | >
>> | >
>> |
>> |
>> | --
>> |
>> | -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> | Pat Haley                          Email:  phaley at mit.edu
>> | Center for Ocean Engineering       Phone:  (617) 253-6824
>> | Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>> | MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>> | 77 Massachusetts Avenue
>> | Cambridge, MA  02139-4301
>> | _______________________________________________
>> | CentOS mailing list
>> | CentOS at centos.org
>> | http://lists.centos.org/mailman/listinfo/centos
>> |
>>
>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:  phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>