Hi:
On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley phaley@mit.edu wrote:
Hi,
I have a server running under CentOS 5.8 and I appear to be in a situation in which the NFS file server is not recognizing the available space on a particular disk (actually a hardware RAID-6 of 13 2Tb disks). If I try to write to the disk I get the following error message
[root@nas-0-1 mseas-data-0-1]# touch dum touch: cannot touch `dum': No space left on device
However, if I check the available space, there seems to be plenty
[root@nas-0-1 mseas-data-0-1]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1
Maybe you're hitting the allocation of reserved blocks for root? With your disk usage of 97% I'd think that could be the case.
You didn't say what file system you're using for that 21TB array, so we (this list) won't be of too much help without knowing that.
xfs file system. The fstab line for this array is:
/dev/sdb1 /mseas-data-0-1 xfs defaults 1 0
tune2fs [0] is your friend
if I read the man pages correctly tune2fs will not work for xfs. From xfs_info I get the following for mseas-data-0-1
[root@nas-0-1 mseas-data-0-1]# xfs_info . meta-data=/dev/sdb1 isize=256 agcount=32, agsize=167846667 blks = sectsz=512 attr=1 data = bsize=4096 blocks=5371093344, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=1 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0
Unfortunately, I don't know how to interpret this or if it is giving relevant information to the question at hand>
- use it to determine if there are reserved blocks
- use it to adjust the settings
[0] https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks
[root@nas-0-1 mseas-data-0-1]# df -i . Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 3290047552 4391552 3285656000 1% /mseas-data-0-1
I don't know if the following is relevant but the disk in question is served as one of 3 bricks in a gluster namespace.
Based on the test with touch, which is happening directly at the NFS level, this seems to be an NFS rather than gluster issue. I couldn't find any file in /var/log which had a time that corresponded to the failed touch test and I didn't see anything in dmesg. We have tried rebooting this system. What else should we look at and/or try to resolve or debug this issue?
If you have a non-root shell account on that box, can you write to that array from the NFS host? ( Take NFS out of the equation. )
Unfortunately we only have a root account on that box.
Thanks.
Pat
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos