[CentOS] Windows 2008R2 AD, kerberos, NFSv4

Mon Apr 23 21:41:44 UTC 2012
janice.psyop <janice.psyop at gmail.com>

Hi,

I'm trying to set up NFSv4 on two boxes (centos 5.5)  and have it
authenticate against our Windows 2008R2 AD server acting as the KDC.
 (samba/winbind is running ok with "idmap config MYCOMPANY: backend = rid"
so we have identical ids across the servers.)

I can mount my test directory fine via NFSv4 *without* the sec=krb5 option.
 However, once I put the sec=krb5 option in, then I get a mount error:
 "mount.nfs4: Permission denied" and rpc.gssd reports: "Failed to obtain
machine credentials for connection to server"

The computers have an AD computer account and for the service-principal, I
created an AD user account "nfsHostname" and mapped the UPN e.g. NFS/
hostname.mycompany.tv at MYCOMPANY.TV to it using ktpass.

This is the closest post similar to my issue I could find:
http://lists.centos.org/pipermail/centos/2010-July/096378.html    However,
I'm trying not to run the createupn command via smbutils.
Side note:
Eventually we will also be using a HDS nas which doesn't provide us with
samba net utils (e.g. net ads join createupn) only their proprietary
webadmin/cli.  When that nas joined our AD domain, it created a computer
account with SPNs of HOST/HOSTNAME, HOST/hostname.MYCOMPANY.TV and a UPN of
HOST/hostname.mycompany.tv at MYCOMPANY.TV  And the HDS nas only wants
encryption type:  des-cbc-crc:normal.  This is why on my test nfs server
(nas002), I'm trying to use the same limited commands as I would if I were
using the HDS nas.

Any suggestions where to look next or get more verbose info from
kerberos/KDC or the nfs server?  (nothing shows up in either syslog --
plus, I'm not all that familiar with kerberos.)

thanks in advance!
JA.



info:
10.100.1.11  KDC server (Windows 2008 R2, AD)
10.100.1.35  bk001  (nfsv4 client, kernel 2.6.18-194.32.1.el5)
10.100.1.82  nas002 (nfsv4 server, kernel 2.6.18-194.32.1.el5)
10.100.1.99  monitoring server

intsalled on both nfsv4 client and server:
nfs-utils.x86_64 1.0.9-60.el5
nfs-utils-lib.x86_64 1.0.8-7.9.el5
nfs4-acl-tools.x86_64 0.3.3-3.el5
krb5-workstation.x86_64 1.6.1-70.el5
samba (nas002)  3.3.8-0.52.el5_5.2
samba (bk001)   3.5.10-0.107.el5



[root at bk001 ~]# net ads testjoin
Join is OK

[root at bk001 ~]# kinit administrator at MYCOMPANY.TV
Password for administrator at MYCOMPANY.TV:

[root at bk001 ~]# kinit nfs/nas002.mycompany.tv at MYCOMPANY.TV
Password for nfs/nas002.mycompany.tv at MYCOMPANY.TV:

[root at bk001 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: nfs/nas002.mycompany.tv at MYCOMPANY.TV

Valid starting     Expires            Service principal
04/13/12 16:08:51  04/14/12 02:08:51  krbtgt/MYCOMPANY.TV at MYCOMPANY.TV
        renew until 04/16/12 16:08:51


Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached


[root at bk001 ~]# showmount -e nas002.mycompany.tv
Export list for nas002.mycompany.tv:
/array gss/krb5,*


[root at bk001 ~]# mount -v -t nfs4 -o proto=tcp,sec=krb5 nas002.mycompany.tv:/
/mnt/nfs4test
Warning: rpc.idmapd appears not to be running.
         All uids will be mapped to the nobody uid.
Warning: rpc.gssd appears not to be running.
mount: pinging: prog 100003 vers 4 prot tcp port 2049
mount.nfs4: Permission denied

[root at bk001 ~]# ps -elf | egrep 'gss|idmap'
1 S root      2498     1  0  75   0 -  8016 -      Apr12 ?        00:00:00
rpc.gssd -rrrvvvv
1 S root      4575     1  0  76   0 - 14833 -      Apr12 ?        00:00:00
rpc.idmapd -vvv


[root at bk001 ~]# tail /var/log/messages
Apr 13 16:09:09 bk001 rpc.idmapd[4575]: New client: 16
Apr 13 16:09:09 bk001 rpc.gssd[2498]: handling krb5 upcall
Apr 13 16:09:09 bk001 rpc.idmapd[4575]: New client: 17
Apr 13 16:09:09 bk001 rpc.idmapd[4575]: Opened
/var/lib/nfs/rpc_pipefs/nfs/clnt16/idmap
Apr 13 16:09:09 bk001 rpc.gssd[2498]: Using keytab file '/etc/krb5.keytab'
Apr 13 16:09:09 bk001 rpc.gssd[2498]: WARNING: Failed to obtain machine
credentials for connection to server nas002.mycompany.tv
Apr 13 16:09:09 bk001 rpc.gssd[2498]: doing error downcall
Apr 13 16:09:09 bk001 rpc.idmapd[4575]: Stale client: 16
Apr 13 16:09:09 bk001 rpc.idmapd[4575]:  -> closed
/var/lib/nfs/rpc_pipefs/nfs/clnt16/idmap
Apr 13 16:09:09 bk001 rpc.idmapd[4575]: Stale client: 17
Apr 13 16:09:09 bk001 rpc.idmapd[4575]:  -> closed
/var/lib/nfs/rpc_pipefs/nfs/clnt17/idmap
Apr 13 16:09:09 bk001 rpc.gssd[2498]: destroying client clnt17
Apr 13 16:09:09 bk001 rpc.gssd[2498]: destroying client clnt16



tshark capture of commands I performed (above):
[root at bk001 ~]# cat /var/tmp/tshark_041312-1608.out
366   9.948504  10.100.1.35 -> 10.100.1.11  TCP 42564 > kerberos [SYN]
Seq=0 Win=5840 Len=0 MSS=1460 TSV=86719599 TSER=0 WS=7
367   9.948813  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42564 [SYN, ACK]
Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 TSV=396813568 TSER=86719599
368   9.948824  10.100.1.35 -> 10.100.1.11  TCP 42564 > kerberos [ACK]
Seq=1 Ack=1 Win=5840 Len=0 TSV=86719599 TSER=396813568
369   9.948849  10.100.1.35 -> 10.100.1.11  KRB5 AS-REQ
370   9.949976  10.100.1.11 -> 10.100.1.35  KRB5 KRB Error:
KRB5KDC_ERR_PREAUTH_REQUIRED
371   9.949982  10.100.1.35 -> 10.100.1.11  TCP 42564 > kerberos [ACK]
Seq=181 Ack=154 Win=6432 Len=0 TSV=86719600 TSER=396813568
372   9.950031  10.100.1.35 -> 10.100.1.11  TCP 42564 > kerberos [FIN, ACK]
Seq=181 Ack=154 Win=6432 Len=0 TSV=86719600 TSER=396813568
373   9.950288  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42564 [ACK]
Seq=154 Ack=182 Win=65160 Len=0 TSV=396813568 TSER=86719600
374   9.950297  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42564 [RST, ACK]
Seq=154 Ack=182 Win=0 Len=0
444  11.840921  10.100.1.35 -> 10.100.1.11  TCP 42565 > kerberos [SYN]
Seq=0 Win=5840 Len=0 MSS=1460 TSV=86721491 TSER=0 WS=7
446  11.841178  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42565 [SYN, ACK]
Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 TSV=396813757 TSER=86721491
447  11.841185  10.100.1.35 -> 10.100.1.11  TCP 42565 > kerberos [ACK]
Seq=1 Ack=1 Win=5840 Len=0 TSV=86721491 TSER=396813757
448  11.841206  10.100.1.35 -> 10.100.1.11  KRB5 AS-REQ
449  11.842812  10.100.1.11 -> 10.100.1.35  TCP [TCP segment of a
reassembled PDU]
450  11.842817  10.100.1.35 -> 10.100.1.11  TCP 42565 > kerberos [ACK]
Seq=259 Ack=1449 Win=8688 Len=0 TSV=86721493 TSER=396813757
451  11.842819  10.100.1.11 -> 10.100.1.35  KRB5 AS-REP
452  11.842822  10.100.1.35 -> 10.100.1.11  TCP 42565 > kerberos [ACK]
Seq=259 Ack=1518 Win=8688 Len=0 TSV=86721493 TSER=396813757
453  11.842852  10.100.1.35 -> 10.100.1.11  TCP 42565 > kerberos [FIN, ACK]
Seq=259 Ack=1518 Win=8688 Len=0 TSV=86721493 TSER=396813757
454  11.843043  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42565 [ACK]
Seq=1518 Ack=260 Win=65160 Len=0 TSV=396813758 TSER=86721493
455  11.843050  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42565 [RST, ACK]
Seq=1518 Ack=260 Win=0 Len=0
827  21.821693  10.100.1.35 -> 10.100.1.11  TCP 42566 > kerberos [SYN]
Seq=0 Win=5840 Len=0 MSS=1460 TSV=86731472 TSER=0 WS=7
828  21.821920  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42566 [SYN, ACK]
Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 TSV=396814755 TSER=86731472
829  21.821930  10.100.1.35 -> 10.100.1.11  TCP 42566 > kerberos [ACK]
Seq=1 Ack=1 Win=5840 Len=0 TSV=86731472 TSER=396814755
830  21.821958  10.100.1.35 -> 10.100.1.11  KRB5 AS-REQ
831  21.822968  10.100.1.11 -> 10.100.1.35  KRB5 AS-REP
832  21.822974  10.100.1.35 -> 10.100.1.11  TCP 42566 > kerberos [ACK]
Seq=191 Ack=618 Win=6787 Len=0 TSV=86731473 TSER=396814756
833  21.823003  10.100.1.35 -> 10.100.1.11  TCP 42566 > kerberos [FIN, ACK]
Seq=191 Ack=618 Win=6787 Len=0 TSV=86731473 TSER=396814756
835  21.823278  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42566 [ACK]
Seq=618 Ack=192 Win=65160 Len=0 TSV=396814756 TSER=86731473
836  21.823287  10.100.1.11 -> 10.100.1.35  TCP kerberos > 42566 [RST, ACK]
Seq=618 Ack=192 Win=0 Len=0
1472  39.980317  10.100.1.35 -> 10.100.1.82  TCP 40520 > nfs [SYN] Seq=0
Win=5840 Len=0 MSS=1460 TSV=86749629 TSER=0 WS=7
1473  39.980491  10.100.1.82 -> 10.100.1.35  TCP nfs > 40520 [SYN, ACK]
Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=3789493491 TSER=86749629 WS=7
1474  39.980498  10.100.1.35 -> 10.100.1.82  TCP 40520 > nfs [ACK] Seq=1
Ack=1 Win=5888 Len=0 TSV=86749631 TSER=3789493491
1475  39.980533  10.100.1.35 -> 10.100.1.82  NFS V4 NULL Call
1476  39.980701  10.100.1.82 -> 10.100.1.35  TCP nfs > 40520 [ACK] Seq=1
Ack=45 Win=5888 Len=0 TSV=3789493492 TSER=86749631
1477  39.980705  10.100.1.82 -> 10.100.1.35  NFS V4 NULL Reply (Call In
1475)
1478  39.980707  10.100.1.35 -> 10.100.1.82  TCP 40520 > nfs [ACK] Seq=45
Ack=29 Win=5888 Len=0 TSV=86749631 TSER=3789493492
1479  39.980733  10.100.1.35 -> 10.100.1.82  TCP 40520 > nfs [FIN, ACK]
Seq=45 Ack=29 Win=5888 Len=0 TSV=86749631 TSER=3789493492
1480  39.980896  10.100.1.82 -> 10.100.1.35  TCP nfs > 40520 [FIN, ACK]
Seq=29 Ack=46 Win=5888 Len=0 TSV=3789493492 TSER=86749631
1481  39.980901  10.100.1.35 -> 10.100.1.82  TCP 40520 > nfs [ACK] Seq=46
Ack=30 Win=5888 Len=0 TSV=86749631 TSER=3789493492
1482  40.001039  10.100.1.35 -> 10.100.1.82  TCP connendp > nfs [SYN] Seq=0
Win=5840 Len=0 MSS=1460 TSV=86749651 TSER=0 WS=7
1483  40.001210  10.100.1.82 -> 10.100.1.35  TCP nfs > connendp [SYN, ACK]
Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=3789493512 TSER=86749651 WS=7
1484  40.001221  10.100.1.35 -> 10.100.1.82  TCP connendp > nfs [ACK] Seq=1
Ack=1 Win=5888 Len=0 TSV=86749651 TSER=3789493512
1485  40.001244  10.100.1.35 -> 10.100.1.82  NFS V4 NULL Call
1486  40.001409  10.100.1.82 -> 10.100.1.35  TCP nfs > connendp [ACK] Seq=1
Ack=45 Win=5888 Len=0 TSV=3789493512 TSER=86749651
1487  40.001414  10.100.1.82 -> 10.100.1.35  NFS V4 NULL Reply (Call In
1485)
1488  40.001418  10.100.1.35 -> 10.100.1.82  TCP connendp > nfs [ACK]
Seq=45 Ack=29 Win=5888 Len=0 TSV=86749652 TSER=3789493512
1489  40.002363  10.100.1.35 -> 10.100.1.82  TCP connendp > nfs [FIN, ACK]
Seq=45 Ack=29 Win=5888 Len=0 TSV=86749653 TSER=3789493512
1490  40.002526  10.100.1.82 -> 10.100.1.35  TCP nfs > connendp [FIN, ACK]
Seq=29 Ack=46 Win=5888 Len=0 TSV=3789493513 TSER=86749653
1491  40.002532  10.100.1.35 -> 10.100.1.82  TCP connendp > nfs [ACK]
Seq=46 Ack=30 Win=5888 Len=0 TSV=86749653 TSER=3789493513
1493  40.002880  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: New client: 16\n
1497  40.003611  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.ERR:
rpc.gssd[2498]: handling krb5 upcall \n
1498  40.004069  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: New client: 17\n
1499  40.004489  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: Opened /var/lib/nfs/rpc_pipefs/nfs/clnt16/idmap\n
1500  40.004949  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.ERR:
rpc.gssd[2498]: Using keytab file '/etc/krb5.keytab' \n
1501  40.005369  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.ERR:
rpc.gssd[2498]: WARNING: Failed to obtain machine credentials for
connection to server nas002.mycompany.tv \n
1502  40.005829  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.ERR:
rpc.gssd[2498]: doing error downcall \n
1503  40.012862  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: Stale client: 16\n
1504  40.013326  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: \t-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt16/idmap\n
1505  40.013740  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: Stale client: 17\n
1506  40.014157  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.WARNING:
rpc.idmapd[4575]: \t-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt17/idmap\n
1507  40.014621  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.ERR:
rpc.gssd[2498]: destroying client clnt17 \n
1508  40.015082  10.100.1.35 -> 10.100.1.99  Syslog DAEMON.ERR:
rpc.gssd[2498]: destroying client clnt16 \n
[root at bk001 ~]#