Short story: cannot get Kerberized NFSv4 to work. I've googled a great
deal and cannot find where I have goofed (and there sure is a lot of
misleading and just plain incorrect information out there), so would
appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as
does ID mapping. We're using NFSv4 in production with sec=sys, but I'm not
happy with that. My kerberized NFSv4 attempts are on a separate test
cluster.
Longer story (sorry for the length):
All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and
nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64,
clients are both x86_64 and i686. Two DC's are both i686, running Samba
4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems
use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc
all good). Samba is at 3.6.9 on all systems except for the DC's.
Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys.
I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with:
# net ads join -U Administrator ...
# net ads testjoin
and created the nfs service principals (on the client and NFSv4 server)
with:
# net ads keytab add nfs -U Administrator
This all works. I can see that the nfs service principals have been added;
on the client abbott.test.cornell.edu, for example:
# net ads keytab list | grep -i nfs
2 DES cbc mode with CRC-32 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
2 DES cbc mode with RSA-MD5 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
2 ArcFour with HMAC/md5 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
2 DES cbc mode with CRC-32 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 DES cbc mode with RSA-MD5 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 ArcFour with HMAC/md5 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott at TITAN.TEST.CORNELL.EDU
2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott at TITAN.TEST.CORNELL.EDU
and on one of the DC's:
# ldbsearch cn=abbott | grep -i nfs
servicePrincipalName: NFS/abbott
servicePrincipalName: NFS/abbott.test.cornell.edu
and on the client "net ads search '(sAMAccountName=abbott$)'" also works,
as does klist:
# klist -ke | grep -i nfs
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and
rpc.gssd and rpc.svcgssd are running (although no need for the latter on
the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said,
NFSv4 works fine without krb5, so I believe the exports file to be
correct).
But when I try to mount, I get the catch-all error:
# mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp
mount.nfs4: access denied by server while mounting costello.test.cornell.edu:/data
and in /var/log/messages on the same client:
Jun 5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU'
Jun 5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in Kerberos
database while getting initial ticket for principal
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab
'FILE:/etc/krb5.keytab'
Jun 5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for connection to server costello.test.cornell.edu
With tcpdump I can see that the DC is contacted during the mount, but the
NFSv4 server is not. The log files on the NFSv4 server are silent.
I have tried (everything was restarted between tests); "no change" means
that it still does not work and gives the same exact errors:
- verified that /etc/idmapd.conf on all systems has the same domains and
realms. This works anyway with sec=sys.
- reduced the keytab to the DES enctypes for nfs/... on all systems; no
change.
- used "allow_weak_crypto=true" in /etc/krb5.conf; no change.
- set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5
des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change.
- tried adding the service principals on the DC with "samba-tool spn add"
instead of "net ads keytab add" on the client, and then exporting the
keytab to the client; no change.
- add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU" and
"nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm was a
factor; no change.
- started rpc.gssd on the clients with the -l switch; no change.
- started rpc.svcgssd with the -n switch so that nfs/ service principals
are in theory not required; no change.
I do not understand why rpc.gssd canot find the service principal,
when it does exist in the database. I've probably missed out some
other things that I have tried to no effect. Help!
Steve
--
----------------------------------------------------------------------------
Steve Thompson E-mail: smt AT vgersoft DOT com
Voyager Software LLC Web: http://www DOT vgersoft DOT com
39 Smugglers Path VSW Support: support AT vgersoft DOT com
Ithaca, NY 14850
"186,282 miles per second: it's not just a good idea, it's the law"
----------------------------------------------------------------------------