Short story: cannot get Kerberized NFSv4 to work. I've googled a great deal and cannot find where I have goofed (and there sure is a lot of misleading and just plain incorrect information out there), so would appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as does ID mapping. We're using NFSv4 in production with sec=sys, but I'm not happy with that. My kerberized NFSv4 attempts are on a separate test cluster.
Longer story (sorry for the length):
All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64, clients are both x86_64 and i686. Two DC's are both i686, running Samba 4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc all good). Samba is at 3.6.9 on all systems except for the DC's.
Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys.
I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with:
# net ads join -U Administrator ... # net ads testjoin
and created the nfs service principals (on the client and NFSv4 server) with:
# net ads keytab add nfs -U Administrator
This all works. I can see that the nfs service principals have been added; on the client abbott.test.cornell.edu, for example:
# net ads keytab list | grep -i nfs 2 DES cbc mode with CRC-32 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU 2 DES cbc mode with RSA-MD5 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU 2 ArcFour with HMAC/md5 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU 2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU 2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU 2 DES cbc mode with CRC-32 nfs/abbott@TITAN.TEST.CORNELL.EDU 2 DES cbc mode with RSA-MD5 nfs/abbott@TITAN.TEST.CORNELL.EDU 2 ArcFour with HMAC/md5 nfs/abbott@TITAN.TEST.CORNELL.EDU 2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott@TITAN.TEST.CORNELL.EDU 2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott@TITAN.TEST.CORNELL.EDU
and on one of the DC's:
# ldbsearch cn=abbott | grep -i nfs servicePrincipalName: NFS/abbott servicePrincipalName: NFS/abbott.test.cornell.edu
and on the client "net ads search '(sAMAccountName=abbott$)'" also works, as does klist:
# klist -ke | grep -i nfs 2 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU (des-cbc-crc) 2 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU (des-cbc-md5) 2 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU (arcfour-hmac) 2 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96) 2 nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96) 2 nfs/abbott@TITAN.TEST.CORNELL.EDU (des-cbc-crc) 2 nfs/abbott@TITAN.TEST.CORNELL.EDU (des-cbc-md5) 2 nfs/abbott@TITAN.TEST.CORNELL.EDU (arcfour-hmac) 2 nfs/abbott@TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96) 2 nfs/abbott@TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and rpc.gssd and rpc.svcgssd are running (although no need for the latter on the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said, NFSv4 works fine without krb5, so I believe the exports file to be correct).
But when I try to mount, I get the catch-all error:
# mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp mount.nfs4: access denied by server while mounting costello.test.cornell.edu:/data
and in /var/log/messages on the same client:
Jun 5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for 'nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU' Jun 5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client 'nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU' not found in Kerberos database while getting initial ticket for principal 'nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU' using keytab 'FILE:/etc/krb5.keytab' Jun 5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for connection to server costello.test.cornell.edu
With tcpdump I can see that the DC is contacted during the mount, but the NFSv4 server is not. The log files on the NFSv4 server are silent.
I have tried (everything was restarted between tests); "no change" means that it still does not work and gives the same exact errors:
- verified that /etc/idmapd.conf on all systems has the same domains and realms. This works anyway with sec=sys.
- reduced the keytab to the DES enctypes for nfs/... on all systems; no change.
- used "allow_weak_crypto=true" in /etc/krb5.conf; no change.
- set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5 des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change.
- tried adding the service principals on the DC with "samba-tool spn add" instead of "net ads keytab add" on the client, and then exporting the keytab to the client; no change.
- add the SPN "nfs/abbott.test.cornell.edu@TITAN.TEST.CORNELL.EDU" and "nfs/abbott@TITAN.TEST.CORNELL.EDU" in case the missing realm was a factor; no change.
- started rpc.gssd on the clients with the -l switch; no change.
- started rpc.svcgssd with the -n switch so that nfs/ service principals are in theory not required; no change.
I do not understand why rpc.gssd canot find the service principal, when it does exist in the database. I've probably missed out some other things that I have tried to no effect. Help!
Steve