[CentOS] Samba4 and NVSv4

Wed Jun 5 22:51:23 UTC 2013

On Wed, Jun 5, 2013 at 5:58 PM, Steve Thompson <smt at vgersoft.com> wrote:

> Short story: cannot get Kerberized NFSv4 to work. I've googled a great
> deal and cannot find where I have goofed (and there sure is a lot of
> misleading and just plain incorrect information out there), so would
> appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as
> does ID mapping. We're using NFSv4 in production with sec=sys, but I'm not
> happy with that. My kerberized NFSv4 attempts are on a separate test
> cluster.
>
> Longer story (sorry for the length):
>
> All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and
> nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64,
> clients are both x86_64 and i686. Two DC's are both i686, running Samba
> 4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems
> use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc
> all good). Samba is at 3.6.9 on all systems except for the DC's.
>
> Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys.
>
> I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with:
>
>    # net ads join -U Administrator ...
>    # net ads testjoin
>
> and created the nfs service principals (on the client and NFSv4 server)
> with:
>
>    # net ads keytab add nfs -U Administrator
>
> This all works. I can see that the nfs service principals have been added;
> on the client abbott.test.cornell.edu, for example:
>
>    # net ads keytab list | grep -i nfs
>    2  DES cbc mode with CRC-32  nfs/
> abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
>    2  DES cbc mode with RSA-MD5  nfs/
> abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
>    2  ArcFour with HMAC/md5     nfs/
> abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
>    2  AES-128 CTS mode with 96-bit SHA-1 HMAC   nfs/
> abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
>    2  AES-256 CTS mode with 96-bit SHA-1 HMAC   nfs/
> abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
>    2  DES cbc mode with CRC-32  nfs/abbott at TITAN.TEST.CORNELL.EDU
>    2  DES cbc mode with RSA-MD5 nfs/abbott at TITAN.TEST.CORNELL.EDU
>    2  ArcFour with HMAC/md5     nfs/abbott at TITAN.TEST.CORNELL.EDU
>    2  AES-128 CTS mode with 96-bit SHA-1 HMAC   nfs/
> abbott at TITAN.TEST.CORNELL.EDU
>    2  AES-256 CTS mode with 96-bit SHA-1 HMAC   nfs/
> abbott at TITAN.TEST.CORNELL.EDU
>
> and on one of the DC's:
>
>    # ldbsearch cn=abbott | grep -i nfs
>    servicePrincipalName: NFS/abbott
>    servicePrincipalName: NFS/abbott.test.cornell.edu
>
> and on the client "net ads search '(sAMAccountName=abbott$)'" also works,
> as does klist:
>
>    # klist -ke | grep -i nfs
>     2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
>     2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
>     2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
>     2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU(aes128-cts-hmac-sha1-96)
>     2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU(aes256-cts-hmac-sha1-96)
>     2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
>     2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
>     2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
>     2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
>     2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
>
> In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and
> rpc.gssd and rpc.svcgssd are running (although no need for the latter on
> the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said,
> NFSv4 works fine without krb5, so I believe the exports file to be
> correct).
>
> But when I try to mount, I get the catch-all error:
>
>    # mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp
>    mount.nfs4: access denied by server while mounting
> costello.test.cornell.edu:/data
>
> and in /var/log/messages on the same client:
>
> Jun  5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for
>         'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU'
> Jun  5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client
>         'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in
> Kerberos
>         database while getting initial ticket for principal
>         'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab
>         'FILE:/etc/krb5.keytab'
> Jun  5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for
> connection to server costello.test.cornell.edu
>
> With tcpdump I can see that the DC is contacted during the mount, but the
> NFSv4 server is not. The log files on the NFSv4 server are silent.
>
> I have tried (everything was restarted between tests); "no change" means
> that it still does not work and gives the same exact errors:
>
> - verified that /etc/idmapd.conf on all systems has the same domains and
>    realms. This works anyway with sec=sys.
>
> - reduced the keytab to the DES enctypes for nfs/... on all systems; no
>    change.
>
> - used "allow_weak_crypto=true" in /etc/krb5.conf; no change.
>
> - set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5
>    des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change.
>
> - tried adding the service principals on the DC with "samba-tool spn add"
>    instead of "net ads keytab add" on the client, and then exporting the
>    keytab to the client; no change.
>
> - add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU" and
>    "nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm was a
>    factor; no change.
>
> - started rpc.gssd on the clients with the -l switch; no change.
>
> - started rpc.svcgssd with the -n switch so that nfs/ service principals
>    are in theory not required; no change.
>
> I do not understand why rpc.gssd canot find the service principal,
> when it does exist in the database. I've probably missed out some
> other things that I have tried to no effect. Help!
>
> Steve
> --
>
> ----------------------------------------------------------------------------
> Steve Thompson                 E-mail:      smt AT vgersoft DOT com
>

What is in your nsswitch.conf? I only ask because you mention it never goes
to the nfs server on the same client.
Can you confirm that the rpc.svcgssd and  rpc.gssd are not crashing after
starting?