[CentOS] ssh failed only with nfs home directory

Tue Jul 14 03:14:46 UTC 2015
Tim Dunphy <bluethundr at gmail.com>

Hey all,

 Having a weird ssh issue I'd like some opinions on.

 If I have my home directory mounted on the NFS server itself, I get
permission denied when I try to ssh into it. The correct permissions and
ownership are on the home directory, ssh directory and the authorized_users
file.

Here's what a verbose ssh session looks like:

#ssh -v bluethundr at nfs1.example.com
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to nfs1.example.com [162.243.109.94] port 22.
debug1: Connection established.
debug1: identity file /Users/TimothyDunphy/.ssh/id_rsa type 1
debug1: identity file /Users/TimothyDunphy/.ssh/id_rsa-cert type -1
debug1: identity file /Users/TimothyDunphy/.ssh/id_dsa type -1
debug1: identity file /Users/TimothyDunphy/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm at openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm at openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA f7:06:1a:56:2f:0e:1b:bd:7b:e6:de:8c:9a:88:ea:09
debug1: Host 'nfs1.example.com' is known and matches the RSA host key.
debug1: Found key in /Users/TimothyDunphy/.ssh/known_hosts:19
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/TimothyDunphy/.ssh/id_rsa
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /Users/TimothyDunphy/.ssh/id_dsa
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

And I see this message in the secure log:

Jul 13 23:09:28 nfsdb1 sshd[15305]: Connection closed by xx.xx.xx.xx
 [preauth]

The IP that I xxx'd out  is my client IP

Here's the permissions and ownership on the directories and files:

#ls -ld /home/bluethundr/ /home/bluethundr/.ssh
/home/bluethundr/.ssh/authorized_keys

drwxr-x---. 37 bluethundr bluethundr 4096 Jul 13 20:57 /home/bluethundr/

drw-------.  3 bluethundr bluethundr 4096 Jun 15 17:22 /home/bluethundr/.ssh

-rw-------.  1 bluethundr bluethundr 2614 Jun 15 17:22
/home/bluethundr/.ssh/authorized_keys

SELinux is set to permissve:

#getenforce
Permissive

If I unmount the nfs home directory I am able to log in:

[root at nfs1:~] #umount -l /home
[root at nfs1:~] #

#ssh bluethundr at nfs1.example.com
Last login: Mon Jul 13 23:08:35 2015 from ool-2f126f64.dyn.optonline.net
-bash-4.2$

The permissions on the non-nfs home directory are the same as the NFS
mounted home directory:

#ls -ld /home/bluethundr/ /home/bluethundr/.ssh
/home/bluethundr/.ssh/authorized_keys
drwxr-x---. 37 bluethundr bluethundr 4096 Jul 13 20:57 /home/bluethundr/
drw-------.  3 bluethundr bluethundr 4096 Jun 15 17:22 /home/bluethundr/.ssh
-rw-------.  1 bluethundr bluethundr 2614 Jun 15 17:22
/home/bluethundr/.ssh/authorized_keys

As soon as I mount it back, the issue returns and I am unable to ssh in:

#ssh bluethundr at nfs1.example.com
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).


I'd really appreciate any ideas you guys may have as to why this is
happening!!

Thanks,
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B