[CentOS] Samba4 and NVSv4

Wed Jun 5 21:58:01 UTC 2013
Steve Thompson <smt at vgersoft.com>

Short story: cannot get Kerberized NFSv4 to work. I've googled a great 
deal and cannot find where I have goofed (and there sure is a lot of 
misleading and just plain incorrect information out there), so would 
appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as 
does ID mapping. We're using NFSv4 in production with sec=sys, but I'm not 
happy with that. My kerberized NFSv4 attempts are on a separate test 
cluster.

Longer story (sorry for the length):

All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and 
nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64, 
clients are both x86_64 and i686. Two DC's are both i686, running Samba 
4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems 
use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc 
all good). Samba is at 3.6.9 on all systems except for the DC's.

Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys.

I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with:

   # net ads join -U Administrator ...
   # net ads testjoin

and created the nfs service principals (on the client and NFSv4 server) 
with:

   # net ads keytab add nfs -U Administrator

This all works. I can see that the nfs service principals have been added; 
on the client abbott.test.cornell.edu, for example:

   # net ads keytab list | grep -i nfs
   2  DES cbc mode with CRC-32	nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
   2  DES cbc mode with RSA-MD5  nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
   2  ArcFour with HMAC/md5	nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
   2  AES-128 CTS mode with 96-bit SHA-1 HMAC	nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
   2  AES-256 CTS mode with 96-bit SHA-1 HMAC	nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
   2  DES cbc mode with CRC-32	nfs/abbott at TITAN.TEST.CORNELL.EDU
   2  DES cbc mode with RSA-MD5	nfs/abbott at TITAN.TEST.CORNELL.EDU
   2  ArcFour with HMAC/md5	nfs/abbott at TITAN.TEST.CORNELL.EDU
   2  AES-128 CTS mode with 96-bit SHA-1 HMAC	nfs/abbott at TITAN.TEST.CORNELL.EDU
   2  AES-256 CTS mode with 96-bit SHA-1 HMAC	nfs/abbott at TITAN.TEST.CORNELL.EDU

and on one of the DC's:

   # ldbsearch cn=abbott | grep -i nfs
   servicePrincipalName: NFS/abbott
   servicePrincipalName: NFS/abbott.test.cornell.edu

and on the client "net ads search '(sAMAccountName=abbott$)'" also works, 
as does klist:

   # klist -ke | grep -i nfs
    2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
    2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
    2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
    2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
    2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
    2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
    2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
    2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
    2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
    2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)

In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and 
rpc.gssd and rpc.svcgssd are running (although no need for the latter on 
the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said, 
NFSv4 works fine without krb5, so I believe the exports file to be 
correct).

But when I try to mount, I get the catch-all error:

   # mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp
   mount.nfs4: access denied by server while mounting costello.test.cornell.edu:/data

and in /var/log/messages on the same client:

Jun  5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for
 	'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU'
Jun  5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client
 	'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in Kerberos
 	database while getting initial ticket for principal
 	'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab
 	'FILE:/etc/krb5.keytab'
Jun  5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for connection to server costello.test.cornell.edu

With tcpdump I can see that the DC is contacted during the mount, but the 
NFSv4 server is not. The log files on the NFSv4 server are silent.

I have tried (everything was restarted between tests); "no change" means 
that it still does not work and gives the same exact errors:

- verified that /etc/idmapd.conf on all systems has the same domains and
   realms. This works anyway with sec=sys.

- reduced the keytab to the DES enctypes for nfs/... on all systems; no
   change.

- used "allow_weak_crypto=true" in /etc/krb5.conf; no change.

- set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5
   des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change.

- tried adding the service principals on the DC with "samba-tool spn add"
   instead of "net ads keytab add" on the client, and then exporting the
   keytab to the client; no change.

- add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU" and
   "nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm was a
   factor; no change.

- started rpc.gssd on the clients with the -l switch; no change.

- started rpc.svcgssd with the -n switch so that nfs/ service principals
   are in theory not required; no change.

I do not understand why rpc.gssd canot find the service principal, 
when it does exist in the database. I've probably missed out some
other things that I have tried to no effect. Help!

Steve
-- 
----------------------------------------------------------------------------
Steve Thompson                 E-mail:      smt AT vgersoft DOT com
Voyager Software LLC           Web:         http://www DOT vgersoft DOT com
39 Smugglers Path              VSW Support: support AT vgersoft DOT com
Ithaca, NY 14850
   "186,282 miles per second: it's not just a good idea, it's the law"
----------------------------------------------------------------------------