On Wed, Jun 5, 2013 at 5:58 PM, Steve Thompson <smt at vgersoft.com> wrote: > Short story: cannot get Kerberized NFSv4 to work. I've googled a great > deal and cannot find where I have goofed (and there sure is a lot of > misleading and just plain incorrect information out there), so would > appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as > does ID mapping. We're using NFSv4 in production with sec=sys, but I'm not > happy with that. My kerberized NFSv4 attempts are on a separate test > cluster. > > Longer story (sorry for the length): > > All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and > nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64, > clients are both x86_64 and i686. Two DC's are both i686, running Samba > 4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems > use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc > all good). Samba is at 3.6.9 on all systems except for the DC's. > > Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys. > > I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with: > > # net ads join -U Administrator ... > # net ads testjoin > > and created the nfs service principals (on the client and NFSv4 server) > with: > > # net ads keytab add nfs -U Administrator > > This all works. I can see that the nfs service principals have been added; > on the client abbott.test.cornell.edu, for example: > > # net ads keytab list | grep -i nfs > 2 DES cbc mode with CRC-32 nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 DES cbc mode with RSA-MD5 nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 ArcFour with HMAC/md5 nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 DES cbc mode with CRC-32 nfs/abbott at TITAN.TEST.CORNELL.EDU > 2 DES cbc mode with RSA-MD5 nfs/abbott at TITAN.TEST.CORNELL.EDU > 2 ArcFour with HMAC/md5 nfs/abbott at TITAN.TEST.CORNELL.EDU > 2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott at TITAN.TEST.CORNELL.EDU > 2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott at TITAN.TEST.CORNELL.EDU > > and on one of the DC's: > > # ldbsearch cn=abbott | grep -i nfs > servicePrincipalName: NFS/abbott > servicePrincipalName: NFS/abbott.test.cornell.edu > > and on the client "net ads search '(sAMAccountName=abbott$)'" also works, > as does klist: > > # klist -ke | grep -i nfs > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU(aes128-cts-hmac-sha1-96) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU(aes256-cts-hmac-sha1-96) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96) > > In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and > rpc.gssd and rpc.svcgssd are running (although no need for the latter on > the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said, > NFSv4 works fine without krb5, so I believe the exports file to be > correct). > > But when I try to mount, I get the catch-all error: > > # mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp > mount.nfs4: access denied by server while mounting > costello.test.cornell.edu:/data > > and in /var/log/messages on the same client: > > Jun 5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for > 'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' > Jun 5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client > 'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in > Kerberos > database while getting initial ticket for principal > 'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab > 'FILE:/etc/krb5.keytab' > Jun 5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for > connection to server costello.test.cornell.edu > > With tcpdump I can see that the DC is contacted during the mount, but the > NFSv4 server is not. The log files on the NFSv4 server are silent. > > I have tried (everything was restarted between tests); "no change" means > that it still does not work and gives the same exact errors: > > - verified that /etc/idmapd.conf on all systems has the same domains and > realms. This works anyway with sec=sys. > > - reduced the keytab to the DES enctypes for nfs/... on all systems; no > change. > > - used "allow_weak_crypto=true" in /etc/krb5.conf; no change. > > - set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5 > des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change. > > - tried adding the service principals on the DC with "samba-tool spn add" > instead of "net ads keytab add" on the client, and then exporting the > keytab to the client; no change. > > - add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU" and > "nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm was a > factor; no change. > > - started rpc.gssd on the clients with the -l switch; no change. > > - started rpc.svcgssd with the -n switch so that nfs/ service principals > are in theory not required; no change. > > I do not understand why rpc.gssd canot find the service principal, > when it does exist in the database. I've probably missed out some > other things that I have tried to no effect. Help! > > Steve > -- > > ---------------------------------------------------------------------------- > Steve Thompson E-mail: smt AT vgersoft DOT com > What is in your nsswitch.conf? I only ask because you mention it never goes to the nfs server on the same client. Can you confirm that the rpc.svcgssd and rpc.gssd are not crashing after starting?