On Tue, Feb 4, 2014 at 1:03 PM, Pat Haley phaley@mit.edu wrote:
Hi,
I have a server running under CentOS 5.8 and I appear to be in a situation in which the NFS file server is not recognizing the available space on a particular disk (actually a hardware RAID-6 of 13 2Tb disks). If I try to write to the disk I get the following error message
[root@nas-0-1 mseas-data-0-1]# touch dum touch: cannot touch `dum': No space left on device
However, if I check the available space, there seems to be plenty
[root@nas-0-1 mseas-data-0-1]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1
Maybe you're hitting the allocation of reserved blocks for root? With your disk usage of 97% I'd think that could be the case.
You didn't say what file system you're using for that 21TB array, so we (this list) won't be of too much help without knowing that.
tune2fs [0] is your friend - use it to determine if there are reserved blocks - use it to adjust the settings
[0] https://wiki.archlinux.org/index.php/ext4#Remove_reserved_blocks
[root@nas-0-1 mseas-data-0-1]# df -i . Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 3290047552 4391552 3285656000 1% /mseas-data-0-1
I don't know if the following is relevant but the disk in question is served as one of 3 bricks in a gluster namespace.
Based on the test with touch, which is happening directly at the NFS level, this seems to be an NFS rather than gluster issue. I couldn't find any file in /var/log which had a time that corresponded to the failed touch test and I didn't see anything in dmesg. We have tried rebooting this system. What else should we look at and/or try to resolve or debug this issue?
If you have a non-root shell account on that box, can you write to that array from the NFS host? ( Take NFS out of the equation. )
Thanks.
Pat
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos