All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from ::ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed
Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
John Lagrue wrote:
All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from ::ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
is selinux enabled?
from a client, try ssh -v 192.168.0.1 -p 2222 and see what kind of debugging errors show up on the client side.
Cameron Showalter wrote:
John Lagrue wrote:
All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from ::ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
is selinux enabled?
from a client, try ssh -v 192.168.0.1 -p 2222 and see what kind of debugging errors show up on the client side.
No obvious errors that I can see :( --------------------------------------------------- [john@lt-131-jdl-f7 ~]$ ssh -v -p 2222 192.168.0.1 OpenSSH_4.5p1, OpenSSL 0.9.8b 04 May 2006 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to 192.168.0.1 [192.168.0.1] port 2222. debug1: Connection established. debug1: identity file /home/john/.ssh/identity type -1 debug1: identity file /home/john/.ssh/id_rsa type -1 debug1: identity file /home/john/.ssh/id_dsa type -1 debug1: Remote protocol version 1.99, remote software version OpenSSH_3.9p1 debug1: match: OpenSSH_3.9p1 pat OpenSSH_3.* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.5 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-cbc hmac-md5 none debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host '[192.168.0.1]:2222' is known and matches the RSA host key. debug1: Found key in /home/john/.ssh/known_hosts:2 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Next authentication method: publickey debug1: Trying private key: /home/john/.ssh/identity debug1: Trying private key: /home/john/.ssh/id_rsa debug1: Trying private key: /home/john/.ssh/id_dsa debug1: Next authentication method: keyboard-interactive debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Next authentication method: password john@192.168.0.1's password: debug1: Authentication succeeded (password). debug1: channel 0: new [client-session] debug1: Entering interactive session. debug1: Sending environment. debug1: Sending env LANG = en_GB.UTF-8 debug1: channel 0: free: client-session, nchannels 1 Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed. debug1: Transferred: stdin 0, stdout 0, stderr 85 bytes in 0.0 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 4007.2 debug1: Exit status -1 -------------------------------------------------------
On Thu, 7 Jun 2007, John Lagrue wrote:
Cameron Showalter wrote:
John Lagrue wrote:
All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from : : ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
This last entry smells to me like it might be a disk-full error. Even if df reports lots of space free, it might be worth using lsof to see whether there are file descriptors open to large (but seemingly deleted) files.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Paul Heinlein Sent: Thursday, June 07, 2007 2:36 PM To: CentOS mailing list Subject: Re: [CentOS] SSH suddenly started failing :(
On Thu, 7 Jun 2007, John Lagrue wrote:
Cameron Showalter wrote:
John Lagrue wrote:
All of a sudden I can no longer ssh into my server
running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]:
authentication failure;
logname= uid=0 euid=0 tty=NODEVssh ruser=
rhost=192.168.0.209 user=john
Jun 7 18:51:46 moray1 sshd[11348]: Accepted password
for john from
: : ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session
opened for user
john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the
same thing every
time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the
specified session
This last entry smells to me like it might be a disk-full error. Even if df reports lots of space free, it might be worth using lsof to see whether there are file descriptors open to large (but seemingly deleted) files.
Or if it is out of inodes on /var but plenty of blocks left...
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
Ross S. W. Walker wrote:
specified session
This last entry smells to me like it might be a disk-full error. Even if df reports lots of space free, it might be worth using lsof to see whether there are file descriptors open to large (but seemingly deleted) files.
Or if it is out of inodes on /var but plenty of blocks left...
Neither of those, as far as I can see. df -i shows I'm only using a few percent of my inodes, and lsof doesn't show anything obvious either :(
JDL
I had a few questions about your failed ssh logins. It appeared to me from the client log that you successfully authenticated but then the session immediately closed.
1. Can you login locally? (Maybe you can only login remotely?) 2. Can others ssh login normally? 3. Can you login as root or as someone else and then do "su -" to yourself?
Maybe your login shell is broken/missing/not-permitted? Maybe your homedir is missing (I don't think that should prevent login, though.)
Dan
Dan Halbert wrote:
I had a few questions about your failed ssh logins. It appeared to me from the client log that you successfully authenticated but then the session immediately closed.
- Can you login locally? (Maybe you can only login remotely?)
- Can others ssh login normally?
- Can you login as root or as someone else and then do "su -" to
yourself?
Maybe your login shell is broken/missing/not-permitted? Maybe your homedir is missing (I don't think that should prevent login, though.)
Dan
I am running a VNC onto the server, and using a window in there can switch users quite happily. But nobody can ssh into the server.
My only thought is that a few days ago I changed the hostname using system-network-config but I am pretty certain that I could still ssh in after doing that. It just stopped working this morning.
I've even tried rebooting in case there were odd processes hanging about, but that's made no difference. At the moment I'm totally stuck!
JDL
On Thu, 7 Jun 2007, John Lagrue wrote:
I am running a VNC onto the server, and using a window in there can switch users quite happily. But nobody can ssh into the server.
My only thought is that a few days ago I changed the hostname using system-network-config but I am pretty certain that I could still ssh in after doing that. It just stopped working this morning.
I've even tried rebooting in case there were odd processes hanging about, but that's made no difference. At the moment I'm totally stuck!
Try a different tack. Get a root shell on the machine in question and fire up sshd in no-fork mode under strace:
strace -o /tmp/ssh.trace sshd -D -p 2222
Then try to ssh into that machine on port 2222 (or whatever you choose). Assuming it fails, close down the temporary sshd and point $PAGER at /tmp/ssh.trace. The failure point will likely be about 90% of the way through the file. (The end of the file will be related to closing down sshd).
Paul Heinlein wrote:
On Thu, 7 Jun 2007, John Lagrue wrote:
I am running a VNC onto the server, and using a window in there can switch users quite happily. But nobody can ssh into the server.
My only thought is that a few days ago I changed the hostname using system-network-config but I am pretty certain that I could still ssh in after doing that. It just stopped working this morning.
I've even tried rebooting in case there were odd processes hanging about, but that's made no difference. At the moment I'm totally stuck!
Try a different tack. Get a root shell on the machine in question and fire up sshd in no-fork mode under strace:
strace -o /tmp/ssh.trace sshd -D -p 2222
Then try to ssh into that machine on port 2222 (or whatever you choose). Assuming it fails, close down the temporary sshd and point $PAGER at /tmp/ssh.trace. The failure point will likely be about 90% of the way through the file. (The end of the file will be related to closing down sshd).
I tried that, and monitored the ssh.trace file with tail-f in another window. There were no obvious signs of error, only what looks like a child starting and then exiting. Mind you, it's been a fair few years since I last looked at a trace file!
On Thu, 7 Jun 2007, John Lagrue wrote:
Paul Heinlein wrote:
Try a different tack. Get a root shell on the machine in question and fire up sshd in no-fork mode under strace:
strace -o /tmp/ssh.trace sshd -D -p 2222
Then try to ssh into that machine on port 2222 (or whatever you choose). Assuming it fails, close down the temporary sshd and point $PAGER at /tmp/ssh.trace. The failure point will likely be about 90% of the way through the file. (The end of the file will be related to closing down sshd).
I tried that, and monitored the ssh.trace file with tail-f in another window. There were no obvious signs of error, only what looks like a child starting and then exiting. Mind you, it's been a fair few years since I last looked at a trace file!
Ugh. Sorry. Try -d instead of -D, so sshd won't fork... You can use up to three -d switches to increase verbosity.
John Lagrue wrote:
I tried that, and monitored the ssh.trace file with tail-f in another window. There were no obvious signs of error, only what looks like a child starting and then exiting. Mind you, it's been a fair few years since I last looked at a trace file!
lace your /etc/profile and ~/.bash_profile files with some `echo "I'm here..."` kind of statements... see if its bombing somewhere
You could try 'nmap localhost' on that machine to ensure that ssh is running on that port.
Could it also be a protocol version problem? Like, the client is using protocol 1, but the sshd is only allowing protocol 2 connections?
JC
On 6/8/07, John R Pierce pierce@hogranch.com wrote:
John Lagrue wrote:
I tried that, and monitored the ssh.trace file with tail-f in another window. There were no obvious signs of error, only what looks like a child starting and then exiting. Mind you, it's been a fair few years since I last looked at a trace file!
lace your /etc/profile and ~/.bash_profile files with some `echo "I'm here..."` kind of statements... see if its bombing somewhere
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Justin Cataldo spake the following on 6/7/2007 4:25 PM:
You could try 'nmap localhost' on that machine to ensure that ssh is running on that port.
Could it also be a protocol version problem? Like, the client is using protocol 1, but the sshd is only allowing protocol 2 connections?
You could also try an rpm -qa --last and see if there might have been an update near the time that it stopped working.
Dan Halbert wrote:
I had a few questions about your failed ssh logins. It appeared to me from the client log that you successfully authenticated but then the session immediately closed.
- Can you login locally? (Maybe you can only login remotely?)
- Can others ssh login normally?
- Can you login as root or as someone else and then do "su -" to
yourself?
Maybe your login shell is broken/missing/not-permitted? Maybe your homedir is missing (I don't think that should prevent login, though.)
Interestingly, I get the the same error even in the VNC window if I type "ssh -p 2222 localhost". It starts and immediately fails!
On 6/7/07, John Lagrue admin@moraystudio.com wrote:
Interestingly, I get the the same error even in the VNC window if I type "ssh -p 2222 localhost". It starts and immediately fails!
Improper permissions on /tmp or /var/tmp maybe?
Jim Perrin wrote:
On 6/7/07, John Lagrue admin@moraystudio.com wrote:
Interestingly, I get the the same error even in the VNC window if I type "ssh -p 2222 localhost". It starts and immediately fails!
Improper permissions on /tmp or /var/tmp maybe?
Nope. Permissions are drwxrwxrwxt on both directories.
John Lagrue wrote:
All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from ::ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
any updates lately? I'm wondering if they may be some conflicts with pam. try disabling pam in /etc/ssh/sshd_config, and restart sshd.
Cameron Showalter wrote:
John Lagrue wrote:
All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from ::ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
any updates lately? I'm wondering if they may be some conflicts with pam. try disabling pam in /etc/ssh/sshd_config, and restart sshd.
I can't recall any updates.
Pam isn't mentioned in /etc/ssh/sshd_config so I have no idea how to disable it :(
John Lagrue wrote:
All of a sudden I can no longer ssh into my server running CentOS 4.5
This is what happens:
[john@lt-131-jdl-f7 ~]$ ssh -Y -p 2222 192.168.0.1 john@192.168.0.1's password: Connection to 192.168.0.1 closed by remote host. Connection to 192.168.0.1 closed.
And yes, the account does exist and the password is correct!
Looking at the logs, I see this:
Jun 7 18:51:37 moray1 sshd(pam_unix)[11348]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=192.168.0.209 user=john Jun 7 18:51:46 moray1 sshd[11348]: Accepted password for john from ::ffff:192.168.0.209 port 57755 ssh2 Jun 7 18:51:46 moray1 sshd(pam_unix)[11352]: session opened for user john by (uid=500) Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed opening loginuid Jun 7 18:51:46 moray1 pam_loginuid[11352]: set_loginuid failed Does any of this make sense?
I've tried it on three different clients and get the same thing every time.
JDL
Jun 7 18:51:46 moray1 sshd[11352]: fatal: PAM session setup failed[14]: Cannot make/remove an entry for the specified session
what does /var/log/secure say?
None of the suggestions made any difference I'm afraid. So in the end I removed openssh_server and reinstalled it, and the problem's gone away.
Thanks to all who tried to help :)
JDL
-
Cameron Showalter -
Dan Halbert -
Jim Perrin -
John Lagrue -
John R Pierce -
Justin Cataldo -
Kay Diederichs -
Paul Heinlein -
Ross S. W. Walker -
Scott Silva