[CentOS] Quota (and disk usage) is incorrectly reported on nfs client mounting XFS filesystem

[Julie Ashworth]

hi all,
I've done more investigation into this problem. I think I 
uncovered a bug, but I want a sanity check... (and please excuse 
me for top posting - I wanted to include the original message
for easier reference).

As a reminder, I have a XFS filesystem exported via nfs4. The
nfs4 clients (all 13) report the quota incorrectly for a user
with >6TB data.
On the NFS server, the 'quota' and 'xfs_quota' commands report 
disk usage correctly.

I modified the this user's quota on the server, and then 
report the output of the 'quota' command below. I start 
with a quota equal to the disk usage (6635021816), and then
decrease it, to the point that the client reports the same
quota value as the server.

Note: the lines separate instances where I adjusted the quota, 
where the top line is 'quota' output on the nfs server, the 
second line is from the nfs client. (The asterisk just 
indicates that the user exceeded the quota block limit).

-----------------------------------------
     Filesystem  blocks      quota   
/dev/mapper/fs   6635021816  6635021816       

     Filesystem  blocks      quota  
     server:/fs  2340054520  2340054520       
-----------------------------------------
     Filesystem  blocks      quota  
/dev/mapper/fs   6635021816* 4294990000 

     Filesystem  blocks      quota 
     server:/fs 2340054520*  22704 
-----------------------------------------
     Filesystem  blocks      quota
/dev/mapper/fs   6635021816* 4294970000

     Filesystem  blocks      quota
     server:/fs 2340054520*  2704
-----------------------------------------
     Filesystem  blocks      quota
/dev/mapper/fs   6635021816* 4294970000

     Filesystem  blocks      quota
     server:/fs  2340054520* 1704
-----------------------------------------
     Filesystem  blocks      quota
/dev/mapper/fs   6635021816* 4294968000       

     Filesystem  blocks      quota  
     server:/fs  2340054520* 704  
-----------------------------------------
     Filesystem  blocks      quota   
/dev/mapper/fs   6635021816* 4294967900       

     Filesystem  blocks      quota   
     server:/fs  2340054520* 604  
-----------------------------------------
     Filesystem  blocks      quota   
/dev/mapper/fs   6635021816* 4294967500       

     Filesystem  blocks      quota   
     server:/fs  2340054520* 204  
-----------------------------------------
     Filesystem  blocks      quota   
/dev/mapper/fs   6635021816* 4294967300       

     Filesystem  blocks      quota   
     server:/fs  2340054520* 4      
-----------------------------------------
     Filesystem  blocks      quota   
/dev/mapper/fs   6635021816* 4294967296 

     <NONE>
-----------------------------------------
     Filesystem  blocks      quota   
/dev/mapper/fs   6635021816* 4294967292 

     Filesystem  blocks      quota   
     server:/fs  2340054520  4294967292
-----------------------------------------

I think it's interesting that the second to last instance
didn't report any quota for this user on the client. Of
course, increasing or decreasing the quota produced output. 

Any insight?

Thanks!
Julie




On 24-06-2011 17.17 -0700, Julie Ashworth wrote:
> hi all,
> I hope you can help me with a strange quota/nfs/XFS behavior...
>  
> I'm using Centos 5.6 on both a nfs server and client. Both are 
> 64-bit, and using a recent kernel:
> 
> NFS server:
> Linux fs2.priv 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> 
> NFS client:
> Linux nx8.priv 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> 
> The NFS server is exporting a XFS filesystem:
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/vgXX-lvXX  24T   16T  8.2T  66% /export
> 
> 
> 
> User foo (for anonymity) added ~3TB of data in the last 2-3 days. 
> On the NFS server, her quota is reported as ~5.8 TB:
> 
> Disk quotas for user foo (uid 1314): 
>      Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
> /dev/mapper/vgXXX-lvXXX
>                 6129681348  7490000000       0         6991914       0       0    
> 
> 
> On a NFS client, her quota is reported as ~1.8TB:
> 
> Disk quotas for user foo (uid 1314): 
>      Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
>     server:/XXXXX   1831504512  3195032704       0         6988049       0       0       
> 
> 
> When I modify her quota on the NFS sever, it's not reflected 
> on the NFS client.
> 
> This problem is specific user foo. There are about 20 other 
> users with quotas on this NFS server, and if I change one of 
> their quotas, the quota/usage is correctly (and instantly) 
> reflected on the NFS client.
> 
> rpc.quotad is configured to use port 875, and I can see this
> is working with strace. And tcpdump shows an established 
> connection:
> 16:50:04.480757 IP nx8.priv.pssc > fs2.priv.875: UDP, length 116
> 16:50:04.481015 IP fs2.priv.875 > nx8.priv.pssc: UDP, length 68
> 
> Any ideas?
> 
> Thanks!
> Best,
> Julie
> 
> 
> -- 
> Julie Ashworth <julie.ashworth at berkeley.edu>
> http://www.neuro.berkeley.edu
> PGP Key ID: 0x17F013D2
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
---end quoted text---

-- 
Julie Ashworth <julie.ashworth at berkeley.edu>
http://www.neuro.berkeley.edu
PGP Key ID: 0x17F013D2