[CentOS] NFS not recognizing available file space

Tue Feb 4 20:58:26 UTC 2014
James A. Peltier <jpeltier at sfu.ca>

change mount to nobarrier,inode64,delaylog

----- Original Message -----
| 
| Hi:
| 
| > On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley <phaley at mit.edu> wrote:
| > 
| >> Hi,
| >>
| >> I have a server running under CentOS 5.8 and I appear to be in a
| >> situation in which the NFS file server is not recognizing the
| >> available
| >> space on a particular disk (actually a hardware RAID-6 of 13 2Tb
| >> disks).
| >> If I try to write to the disk I get the following error message
| >>
| >> [root at nas-0-1 mseas-data-0-1]# touch dum
| >> touch: cannot touch `dum': No space left on device
| >>
| >> However, if I check the available space, there seems to
| >> be plenty
| >>
| >> [root at nas-0-1 mseas-data-0-1]# df -h .
| >> Filesystem            Size  Used Avail Use% Mounted on
| >> /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
| >>
| > 
| > Maybe you're hitting the allocation of reserved blocks for root?
| > With your disk usage of 97% I'd think that could be the case.
| > 
| > You didn't say what file system you're using for that 21TB array,
| > so we
| > (this list) won't be of too much help without knowing that.
| 
| xfs file system.  The fstab line for this array is:
| 
| /dev/sdb1               /mseas-data-0-1         xfs     defaults
|        1 0
| 
| > 
| > tune2fs [0] is your friend
| 
| if I read the man pages correctly tune2fs will not work for xfs.
|  From xfs_info I get the following for mseas-data-0-1
| 
| [root at nas-0-1 mseas-data-0-1]# xfs_info .
| meta-data=/dev/sdb1              isize=256    agcount=32,
| agsize=167846667 blks
|           =                       sectsz=512   attr=1
| data     =                       bsize=4096   blocks=5371093344,
| imaxpct=25
|           =                       sunit=0      swidth=0 blks,
|           unwritten=1
| naming   =version 2              bsize=4096
| log      =internal               bsize=4096   blocks=32768, version=1
|           =                       sectsz=512   sunit=0 blks,
|           lazy-count=0
| realtime =none                   extsz=4096   blocks=0, rtextents=0
| 
| Unfortunately, I don't know how to interpret this or if
| it is giving relevant information to the question at hand>
| 
| > - use it to determine if there are reserved blocks
| > - use it to adjust the settings
| > 
| > [0]
| > https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks
| > 
| > 
| >> [root at nas-0-1 mseas-data-0-1]# df -i .
| >> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
| >> /dev/sdb1            3290047552 4391552 3285656000    1%
| >> /mseas-data-0-1
| >>
| >> I don't know if the following is relevant but the disk in question
| >> is served as one of 3 bricks in a gluster namespace.
| >>
| >> Based on the test with touch, which is happening directly
| >> at the NFS level, this seems to be an NFS rather than gluster
| >> issue. I couldn't find any file in /var/log which had a
| >> time that corresponded to the failed touch test and I didn't
| >> see anything in dmesg.  We have tried rebooting this system.
| >> What else should we look at and/or try to resolve or debug
| >> this issue?
| >>
| > 
| > If you have a non-root shell account on that box, can you write to
| > that
| > array from the NFS host?
| > ( Take NFS out of the equation. )
| 
| Unfortunately we only have a root account on that box.
| 
| > 
| > 
| >> Thanks.
| >>
| >> Pat
| >>
| >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
| >> Pat Haley                          Email:  phaley at mit.edu
| >> Center for Ocean Engineering       Phone:  (617) 253-6824
| >> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
| >> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
| >> 77 Massachusetts Avenue
| >> Cambridge, MA  02139-4301
| >> _______________________________________________
| >> CentOS mailing list
| >> CentOS at centos.org
| >> http://lists.centos.org/mailman/listinfo/centos
| >>
| > 
| > 
| > 
| 
| 
| --
| 
| -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
| Pat Haley                          Email:  phaley at mit.edu
| Center for Ocean Engineering       Phone:  (617) 253-6824
| Dept. of Mechanical Engineering    Fax:    (617) 253-8125
| MIT, Room 5-213                    http://web.mit.edu/phaley/www/
| 77 Massachusetts Avenue
| Cambridge, MA  02139-4301
| _______________________________________________
| CentOS mailing list
| CentOS at centos.org
| http://lists.centos.org/mailman/listinfo/centos
| 

-- 
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier at sfu.ca
Website : http://www.sfu.ca/itservices

"I want to inspire people.  I want someone to say "because of you I didn't give up". - Chanda Kaushik