Hello everyone, We have a CentOS 6.3 NFSv4 server and client, and we've run into a situation where the client is unable to list "private" (chmod 700-ed) directories, even if the current user owns the directory in question. A bit more background: we're also using Samba 3.5+Winbind to provide authentication and UID/GID mapping against a Windows 2008 R2 domain controller. I've been asked to get Kerberized NFSv4 working, but this problem occurs on mounts both with and without Kerberos, so I won't torture anyone with the Kerberos part. :-) ID mapping works: [joeuser at nfsclient ~]$ id joeuser uid=56055(joeuser) gid=6502(domain users) groups=6502(domain users),1000001(BUILTIN\users) Mounting directories -o sec=none works: [joeuser at nfsclient ~]$ sudo mount -t nfs4 -o proto=tcp,port=2049 -o sec=none nfsserver.example.com:/testdir /mnt [joeuser at nfsclient ~]$ ls -l /mnt drwx------ 2 user1 domain users 4096 Aug 3 11:43 user1 drwx------ 2 adbinder domain users 4096 Aug 17 15:20 adbinder drwx------ 2 joeuser domain users 4096 Aug 3 11:43 joeuser ... however, we can't seem to track down why this happens: [joeuser at nfsclient ~]$ cd /mnt/joeuser bash: cd: joeuser: Permission denied We have the same issue when mounting -o sec=krb5p, which yields slightly more interesting debug output[2]. Both hosts are running CentOS 6.3: $ uname -a ; cat /etc/redhat-release ; rpm -qa | egrep "(samba|winbind|nfs)" Linux nfsserver 2.6.32-279.2.1.el6.x86_64 #1 SMP Fri Jul 20 01:55:29 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux CentOS release 6.3 (Final) nfs-utils-1.2.3-26.el6.x86_64 nfs-utils-lib-1.1.5-4.el6.x86_64 samba-client-3.5.10-125.el6.x86_64 samba-common-3.5.10-125.el6.x86_64 samba-winbind-3.5.10-125.el6.x86_64 samba-winbind-clients-3.5.10-125.el6.x86_64 Out of curiosity, I tried the same AD/Winbind/NFSv4 setup on Fedora 17 and had the same results when mounting either -o sec=none or -o sec=krb5/i/p: $ uname -a ; cat /etc/redhat-release ; rpm -qa | egrep "(samba|winbind|nfs)" Linux nfsserver-fedora 3.5.2-1.fc17.x86_64 #1 SMP Wed Aug 15 16:09:27 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Fedora release 17 (Beefy Miracle) libnfsidmap-0.25-3.fc17.x86_64 libnfsidmap-debuginfo-0.25-3.fc17.x86_64 nfs-utils-1.2.6-3.fc17.x86_64 nfs-utils-debuginfo-1.2.6-3.fc17.x86_64 samba-3.6.6-92.fc17.1.x86_64 samba-client-3.6.6-92.fc17.1.x86_64 samba-common-3.6.6-92.fc17.1.x86_64 samba-winbind-3.6.6-92.fc17.1.x86_64 samba-winbind-clients-3.6.6-92.fc17.1.x86_64 So... * Is this an ID mapping problem of some kind? Many of the mailing list posts and bug reports that I've read pertaining to this problem usually stem from ID mapping failure, but it seems to be working in our case[4][5]. * Is it possible that this is an AD/Kerberos problem even though we're mounting -o sec=none and Kerberos is supposedly not involved? A recent linux-nfs mailing list post[0] described a similar problem when mounting, but the fix had no effect for us. (Once again, I tried mounting -o sec=none and -o sec=krb5/i/p). * Since we're relying in W2K8R2 for ID mapping, is there something extra we need to do to our least-privilege bind account[3]? We would greatly appreciate any advice that you NFSv4 experts out there could offer. I would be happy to pass along more details/logs/debug/puzzle pieces if needed. Many thanks, -- Derek Warren, IT Services, Research Computing Group, Simon Fraser University [0] http://www.spinics.net/lists/linux-nfs/msg27440.html [1] Preamble to mounting. Again, the same problem happens with or without Kerberos. # net ads join createupn="nfs/nfsserver.example.com at AD.EXAMPLE.COM" createcomputer="OU" -U $BINDUSER%$BINDPASSWD # kinit -k NFSSERVER$ (and NFSCLIENT$, but we're going to mount -o sec=none as mentioned above) # net ads keytab add nfs # klist -ke Keytab name: WRFILE:/etc/krb5.keytab KVNO Principal ---- -------------------------------------------------------------------------- 3 host/nfsserver.example.com at AD.EXAMPLE.COM (des-cbc-crc) 3 host/nfsserver.example.com at AD.EXAMPLE.COM (des-cbc-md5) 3 host/nfsserver.example.com at AD.EXAMPLE.COM (arcfour-hmac) 3 host/NFSSERVER at AD.EXAMPLE.COM (des-cbc-crc) 3 host/NFSSERVER at AD.EXAMPLE.COM (des-cbc-md5) 3 host/NFSSERVER at AD.EXAMPLE.COM (arcfour-hmac) 3 NFSSERVER$@AD.EXAMPLE.COM (des-cbc-crc) 3 NFSSERVER$@AD.EXAMPLE.COM (des-cbc-md5) 3 NFSSERVER$@AD.EXAMPLE.COM (arcfour-hmac) 3 nfs/nfsserver.example.com at AD.EXAMPLE.COM (des-cbc-crc) 3 nfs/nfsserver.example.com at AD.EXAMPLE.COM (des-cbc-md5) 3 nfs/nfsserver.example.com at AD.EXAMPLE.COM (arcfour-hmac) 3 nfs/NFSSERVER at AD.EXAMPLE.COM (des-cbc-crc) 3 nfs/NFSSERVER at AD.EXAMPLE.COM (des-cbc-md5) 3 nfs/NFSSERVER at AD.EXAMPLE.COM (arcfour-hmac) (same `klist -ke` output on the client, too) [2] The syslog output from the same problem, but when mounted -o sec=krb5/i/p: nfsclient rpc.gssd[10256]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt1b) nfsclient rpc.gssd[10256]: handle_gssd_upcall: 'mech=krb5 uid=56055 enctypes=18,17,16,23,3,1,2 ' nfsclient rpc.gssd[10256]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt1b) nfsclient rpc.gssd[10256]: process_krb5_upcall: service is '<null>' nfsclient rpc.gssd[10256]: getting credentials for client with uid 56055 for server nfsserver.example.com nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_56055_od8D5s' being considered, with preferred realm 'AD.EXAMPLE.COM' nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_56055_od8D5s'(joeuser at AD.EXAMPLE.COM) passed all checks and has mtime of 1344544593 nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_56055_DUEN2I' being considered, with preferred realm 'AD.EXAMPLE.COM' nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_56055_DUEN2I'(joeuser at AD.EXAMPLE.COM) passed all checks and has mtime of 1344543679 nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_56055_od8D5s' is our current best match with mtime of 1344544593 nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_0' being considered, with preferred realm 'AD.EXAMPLE.COM' nfsclient rpc.gssd[10256]: CC file '/tmp/krb5cc_0' owned by 0, not 56055 nfsclient rpc.gssd[10256]: using FILE:/tmp/krb5cc_56055_od8D5s as credentials cache for client with uid 56055 for server nfsserver.example.com nfsclient rpc.gssd[10256]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_56055_od8D5s nfsclient rpc.gssd[10256]: creating context using fsuid 56055 (save_uid 0) nfsclient rpc.gssd[10256]: creating tcp client for server nfsserver.example.com nfsclient rpc.gssd[10256]: DEBUG: port already set to 2049 nfsclient rpc.gssd[10256]: creating context with server nfs at nfsserver.example.com nfsclient rpc.gssd[10256]: WARNING: Failed to create krb5 context for user with uid 56055 for server nfsserver.example.com nfsclient rpc.gssd[10256]: WARNING: Failed to create krb5 context for user with uid 56055 for server nfsserver.example.com nfsclient rpc.gssd[10256]: doing error downcall [3] Our minimal-privilege Active Directory binding account for Winbind was created using the following steps: 1. Run ADU&C. View menu=>Advanced Features 2. Create a binding account (e.g., 'megabind') 3. Right-click on the FAS container=>Properties=>Security tab 4. [Advanced]=>[Add...]=>megabind=>[OK] 5. [megabind: This object and all descendant objects...] => Write All Properties: allow => Create Computer objects: allow 6. [OK]/[OK]/[OK] [4] rpc.idmapd debug output when listing /mnt--same results when mounted -o sec=none/-o sec=krb5/i/p: Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: Server : (user) id "56055" -> name "joeuser at ad.example.com" Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfsdcb: authbuf=199.60.0.0/255.255.240.0 authtype=group Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfs4_gid_to_name: calling nsswitch->gid_to_name Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfs4_gid_to_name: nsswitch->gid_to_name returned 0 Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfs4_gid_to_name: final return value is 0 Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: Server : (group) id "6502" -> name "domain users at ad.example.com" Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfsdcb: authbuf=199.60.0.0/255.255.240.0 authtype=user Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfs4_uid_to_name: calling nsswitch->uid_to_name Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfs4_uid_to_name: nsswitch->uid_to_name returned 0 Aug 17 11:28:30 nfsserver rpc.idmapd[1484]: nfs4_uid_to_name: final return value is 0 [5] nfsd debug output: [joeuser at nfsclient /mnt] cd /mnt/joeuser Aug 17 11:28:30 nfsclient kernel: encode_compound: tag= Aug 17 11:28:30 nfsclient kernel: decode_attr_type: type=040000 Aug 17 11:28:30 nfsclient kernel: decode_attr_change: change attribute=17878195753850634499 Aug 17 11:28:30 nfsclient kernel: decode_attr_size: file size=4096 Aug 17 11:28:30 nfsclient kernel: decode_attr_fsid: fsid=(0x64e7963b2d1f48aa/0x98fd26d8b8883c44) Aug 17 11:28:30 nfsclient kernel: decode_attr_fileid: fileid=2117406 Aug 17 11:28:30 nfsclient kernel: decode_attr_fs_locations: fs_locations done, error = 0 Aug 17 11:28:30 nfsclient kernel: decode_attr_mode: file mode=0755 Aug 17 11:28:30 nfsclient kernel: decode_attr_nlink: nlink=6 Aug 17 11:28:30 nfsclient kernel: decode_attr_owner: uid=0 Aug 17 11:28:30 nfsclient kernel: decode_attr_group: gid=0 Aug 17 11:28:30 nfsclient kernel: decode_attr_rdev: rdev=(0x0:0x0) Aug 17 11:28:30 nfsclient kernel: decode_attr_space_used: space used=4096 Aug 17 11:28:30 nfsclient kernel: decode_attr_time_access: atime=1345215537 Aug 17 11:28:30 nfsclient kernel: decode_attr_time_metadata: ctime=1344019704 Aug 17 11:28:30 nfsclient kernel: decode_attr_time_modify: mtime=1344019352 Aug 17 11:28:30 nfsclient kernel: decode_attr_mounted_on_fileid: fileid=0 Aug 17 11:28:30 nfsclient kernel: decode_getfattr_attrs: xdr returned 0 Aug 17 11:28:30 nfsclient kernel: decode_getfattr_generic: xdr returned 0 Aug 17 11:28:30 nfsclient kernel: NFS: nfs_update_inode(0:17/2117406 ct=2 info=0x27e7f) Aug 17 11:28:30 nfsclient kernel: NFS: permission(0:17/2117406), mask=0x1, res=0 Aug 17 11:28:30 nfsclient kernel: NFS: nfs_lookup_revalidate(/joeuser) is valid Aug 17 11:28:30 nfsclient kernel: encode_compound: tag= joeuser: Permission denied. Aug 17 11:28:30 nfsclient kernel: decode_attr_type: type=040000 Aug 17 11:28:30 nfsclient kernel: decode_attr_change: change attribute=8802615669730494470 Aug 17 11:28:30 nfsclient kernel: decode_attr_size: file size=4096 Aug 17 11:28:30 nfsclient kernel: decode_attr_fsid: fsid=(0x64e7963b2d1f48aa/0x98fd26d8b8883c44) Aug 17 11:28:30 nfsclient kernel: decode_attr_fileid: fileid=2121737 Aug 17 11:28:30 nfsclient kernel: decode_attr_fs_locations: fs_locations done, error = 0 Aug 17 11:28:30 nfsclient kernel: decode_attr_mode: file mode=0700 Aug 17 11:28:30 nfsclient kernel: decode_attr_nlink: nlink=2 Aug 17 11:28:30 nfsclient kernel: decode_attr_owner: uid=56055 Aug 17 11:28:30 nfsclient kernel: decode_attr_group: gid=6502 Aug 17 11:28:30 nfsclient kernel: decode_attr_rdev: rdev=(0x0:0x0) Aug 17 11:28:30 nfsclient kernel: decode_attr_space_used: space used=4096 Aug 17 11:28:30 nfsclient kernel: decode_attr_time_access: atime=1345071487 Aug 17 11:28:30 nfsclient kernel: decode_attr_time_metadata: ctime=1345071482 Aug 17 11:28:30 nfsclient kernel: decode_attr_time_modify: mtime=1344019451 Aug 17 11:28:30 nfsclient kernel: decode_attr_mounted_on_fileid: fileid=0 Aug 17 11:28:30 nfsclient kernel: decode_getfattr_attrs: xdr returned 0 Aug 17 11:28:30 nfsclient kernel: decode_getfattr_generic: xdr returned 0 Aug 17 11:28:30 nfsclient kernel: NFS: nfs_update_inode(0:17/2121737 ct=1 info=0x27e7f) Aug 17 11:28:30 nfsclient kernel: NFS: permission(0:17/2121737), mask=0x11, res=-13 Aug 17 11:28:30 nfsclient kernel: NFS: dentry_delete(/joeuser, 8) Aug 17 11:28:30 nfsclient kernel: NFS: permission(0:17/2117406), mask=0x1, res=0 Aug 17 11:28:30 nfsclient kernel: NFS: nfs_lookup_revalidate(/joeuser) is valid Aug 17 11:28:30 nfsclient kernel: NFS: permission(0:17/2121737), mask=0x11, res=-13 Aug 17 11:28:30 nfsclient kernel: NFS: dentry_delete(/joeuser, 8)