Hi List,
We have a CentOS VPS running a web site in a DC far away. The chap that dev's this site told me he couldn't SFTP in yesterday, his password was being rejected (I went to his desk to confirm and saw it was telling him the password was incorrect but neither him nor me had changed it and we are the only two with access to this VPS). So I logged in as root and reset his password, be he still couldn't log in (same problem, claiming the password was wrong).
[root@server ~]# passwd webdevuser Changing password for user webdevuser. New UNIX password: Retype new UNIX password: passwd: all authentication tokens updates successfully.
I tried to SSH in as the web dev user and it wouldn't let me in. Returning back to my root console window;
[root@server ~]# su - webdevuser [webdevuser@server ~]# passwd Changing password for user webdevuser. Changing password for webdevuser. (current) UNIX password: passwd: Authentication token manipulation error
Firstly; I am stracthing my head as to why his password was no longer working in the first place?
Secondly; Why I can't reset it?
Googling around many people suggest there is a discrepancy between the /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using pwconv to recreate shadow and the same for /etc/groups, deleting gshadow recreating it with grpconv will solve the problem but I still can't login as the web dev user.
Any ideas anyone?
On Wed, Feb 16, 2011 at 7:28 AM, James Bensley jwbensley@gmail.com wrote:
Hi List,
We have a CentOS VPS running a web site in a DC far away. The chap that dev's this site told me he couldn't SFTP in yesterday, his password was being rejected (I went to his desk to confirm and saw it was telling him the password was incorrect but neither him nor me had changed it and we are the only two with access to this VPS). So I logged in as root and reset his password, be he still couldn't log in (same problem, claiming the password was wrong).
[root@server ~]# passwd webdevuser Changing password for user webdevuser. New UNIX password: Retype new UNIX password: passwd: all authentication tokens updates successfully.
I tried to SSH in as the web dev user and it wouldn't let me in. Returning back to my root console window;
[root@server ~]# su - webdevuser [webdevuser@server ~]# passwd Changing password for user webdevuser. Changing password for webdevuser. (current) UNIX password: passwd: Authentication token manipulation error
Firstly; I am stracthing my head as to why his password was no longer working in the first place?
Secondly; Why I can't reset it?
Googling around many people suggest there is a discrepancy between the /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using pwconv to recreate shadow and the same for /etc/groups, deleting gshadow recreating it with grpconv will solve the problem but I still can't login as the web dev user.
Any ideas anyone?
Uh-oh. Has your developer, or you, been editing the /etc/passwd, /etc/shadow, /etc/group, or /etc/gshadow files manually? And do you use NIS or LDAP for authentication? And this is a publicly exposed webserver, right? How fast can you rebuild it if it's been rootkitted?
Check the /etc/shadow and /etc/group for consistent numbers of entries, and /etc/group and /etc/gshadow. Do you have other users who can still log in or not?
On 16/02/11 13:28, James Bensley wrote:
Hi List,
We have a CentOS VPS running a web site in a DC far away. The chap that dev's this site told me he couldn't SFTP in yesterday, his password was being rejected (I went to his desk to confirm and saw it was telling him the password was incorrect but neither him nor me had changed it and we are the only two with access to this VPS). So I logged in as root and reset his password, be he still couldn't log in (same problem, claiming the password was wrong).
[root@server ~]# passwd webdevuser Changing password for user webdevuser. New UNIX password: Retype new UNIX password: passwd: all authentication tokens updates successfully.
I tried to SSH in as the web dev user and it wouldn't let me in. Returning back to my root console window;
[root@server ~]# su - webdevuser [webdevuser@server ~]# passwd Changing password for user webdevuser. Changing password for webdevuser. (current) UNIX password: passwd: Authentication token manipulation error
Firstly; I am stracthing my head as to why his password was no longer working in the first place?
Secondly; Why I can't reset it?
Googling around many people suggest there is a discrepancy between the /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using pwconv to recreate shadow and the same for /etc/groups, deleting gshadow recreating it with grpconv will solve the problem but I still can't login as the web dev user.
Any ideas anyone?
- Could the account have become locked somehow? (passwd -u $user) Or could the account have become expired?
- Are the permissions strict on the users ~/.ssh? (0700 on the directory, and 0600 on any files inside that directory - like authorized_keys ...)
- Is SELinux in Enforced mode and are the SELinux file context correct on /home? (restorecon -rv /home)
Also double check /var/log/messages, /var/log/secure and /var/log/audit/audit.log carefully when trying to log in as that user.
kind regards,
David Sommerseth
On Wed, Feb 16, 2011 at 7:28 AM, James Bensley jwbensley@gmail.com wrote:
[root@server ~]# su - webdevuser [webdevuser@server ~]# passwd Changing password for user webdevuser. Changing password for webdevuser. (current) UNIX password: passwd: Authentication token manipulation error
A lot of things can cause this, including a full /var filesystem :/
Thanks to all for your various replies....
On 16 February 2011 12:50, Nico Kadel-Garcia nkadel@gmail.com wrote:
Check the /etc/shadow and /etc/group for consistent numbers of entries, and /etc/group and /etc/gshadow.
Do you mean duplicate entries? If so there are none of those.
No, I mean the sam enumber of entries.
wc /etc/shadow /etc/passwd
cut -f1 -d: /etc/shasow /etc/passwd | sort | uniq -c
This came back 2 for each user, so no differences.
And actually go line by line down these files, checking for matching usernames, correct layout of ':' separated entries, correct numbers of entries, and blank lines. I've seen serous problems where one or ther other of these files were corrupted by something, especially badly written installer scripts that only edited /etc/passwd directly and ignored /etc/shadow, or which mishandled "$" entries in newly created encrypted passwords.
I'm now going through this although its all looking intact.
Do you have other users who can still log in or not?
There is only the root and web dev user on this box.
Are you *sure*? Can you back this thing up for review and rebuilding? It might be safest to image it for analysis and simply rebuild it.
Yes, but I like to fix things. If I can't fix this I will restore the box but for now I'm going to continue troubleshooting. The root user and web dev user are the only two that have hash value in the passwd file so I would expect this to mean they are the only two users than can actually log in?
On 16 February 2011 12:59, David Sommerseth dazo@users.sourceforge.net wrote:>
- Could the account have become locked somehow? (passwd -u $user) Or
could the account have become expired?
[root@server ~]# passwd -u futuread. Unlocking password for user futuread. paswd: Success.
But I still get access denied.
- Are the permissions strict on the users ~/.ssh? (0700 on the directory,
and 0600 on any files inside that directory - like authorized_keys ...)
If I remove execute permissions form the web dev home folder a website will stop working, its within that users home folder. I.e. virtual site1 is inside the home folder of user 'virtual1' and virtual site2 is within the home folder of the user 'virtual2'. The web dev chap logins in as say virtual1 and edits all sites with that account. There is no .ssh subfolder in the home folder? Could this be the problem? If he saw it in there and deleted it perhaps (although I imagine it would just be recreated if needed?).
- Is SELinux in Enforced mode and are the SELinux file context correct on
/home? (restorecon -rv /home)
[root@server ~]# getenforce Disabled
Also double check /var/log/messages, /var/log/secure and /var/log/audit/audit.log carefully when trying to log in as that user.
/var/log/audit is empty. Is this normal, this VPS comes initially configured from the provider? /var/log/messages and /var/log/secure both just show a generic invalid login attemp:
/var/log/messages: Feb 16 13:53:58 server1882 sshd(pam_unix)[16225]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4 user=webdevuser
/var/log/secure: Feb 16 13:53:50 server1882 sshd[16225]: Failed password for futuread from ::ffff:1.2.3.4 port 1536 ssh2
On 16 February 2011 13:08, Kwan Lowe kwan.lowe@gmail.com wrote:
A lot of things can cause this, including a full /var filesystem :/
Nope, only %75 full (60GB filesystem), there's some room left in her yet ;)
Thanks everyone for your help so far its really appreciated.
On 16 February 2011 13:00, ... wrote:
you realize that there are no passwords in /etc/passwd, so if you delete /etc/shadow and rebuild it using pwconv there will be no passwords in the new /etc/shadow... depending on the exact state, you either won't be able to log in, or the machine will be totally open.
Yes sorry I meant that the other way round :)
i'd suggest looking at the log files (/var/log/secure and .../messages), for indications of why you're having trouble logging in as the other user. you can also, in a terminal window from a "mere mortal" (not root) login, try:
su - <user>
as that may give you some feedback. something like having an invalid shell will cause what you're seeing.
As root, if I 'su - webdevuser' it doesn't prompt me for a password and drops me in as the user, presumably what is intended?
On 16/02/11 15:16, James Bensley wrote:
i'd suggest looking at the log files (/var/log/secure and
.../messages), for indications of why you're having trouble logging in as the other user. you can also, in a terminal window from a "mere mortal" (not root) login, try:
su - <user>
as that may give you some feedback. something like having an invalid shell will cause what you're seeing.
As root, if I 'su - webdevuser' it doesn't prompt me for a password and drops me in as the user, presumably what is intended?
This is normal behaviour. root can su to which ever user without being asked for any password by default.
kind regards,
David Sommerseth
On Wed, Feb 16, 2011 at 6:28 AM, James Bensley jwbensley@gmail.com wrote:
Hi List,
We have a CentOS VPS running a web site in a DC far away. The chap that dev's this site told me he couldn't SFTP in yesterday, his password was being rejected (I went to his desk to confirm and saw it was telling him the password was incorrect but neither him nor me had changed it and we are the only two with access to this VPS). So I logged in as root and reset his password, be he still couldn't log in (same problem, claiming the password was wrong).
[root@server ~]# passwd webdevuser Changing password for user webdevuser. New UNIX password: Retype new UNIX password: passwd: all authentication tokens updates successfully.
I tried to SSH in as the web dev user and it wouldn't let me in. Returning back to my root console window;
[root@server ~]# su - webdevuser [webdevuser@server ~]# passwd Changing password for user webdevuser. Changing password for webdevuser. (current) UNIX password: passwd: Authentication token manipulation error
Firstly; I am stracthing my head as to why his password was no longer working in the first place?
Secondly; Why I can't reset it?
Googling around many people suggest there is a discrepancy between the /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using pwconv to recreate shadow and the same for /etc/groups, deleting gshadow recreating it with grpconv will solve the problem but I still can't login as the web dev user.
Any ideas anyone?
What does /etc/nsswitch.conf look like? Anything other than "files" for passwd, shadow and group? If that's OK, I would start comparing files in /etc/pam.d to a known-good system.
-- Jeff
On 16 February 2011 14:34, ... wrote:
yes, that is what doing an "su - <user>" as *root* will do, which doesn't tell you much. instead of doing this from a root login, do it from a regular account (you don't routinely log in as root i hope
- actually it sounds like you do).
if this works, then the issue isn't with the password or shell.
No other user is allowed to execute /bin/su :)
(This is something the VPS providers have put in place, apart from root, all other users for each virtual site have their shell set to /usr/local/cpanel/bin/jailshell or /usr/local/cpanel/bin/noshell)
by the way, it doesn't sound like the accounts on this machine are set up very well. you should *never* log in as root (that capability should be disabled actually). rather you should log in to a regular, unprivileged, account and su (or sudo) to root only when you need to do something privileged and only for that moment. your developer's access sounds rather odd too, with the seeming lack of separation between the login and the site content.
Its not my server so those aren't my decisions to make. I don't normally allow root ssh, I would have probably installed fail2ban, set up SELinux blah blah blah and many other things but this isn't my VPS, I've just been tasked with it so this is the way it is! :s
On 16 February 2011 14:50, Jeff jlar310@gmail.com wrote:
What does /etc/nsswitch.conf look like? Anything other than "files" for passwd, shadow and group? If that's OK, I would start comparing files in /etc/pam.d to a known-good system.
Thanks for the info, I will look into the pam.d stuff :)
cat /etc/nsswitch.conf ....SNIP..... passwd: files shadow: files group: files
Ok, everything is fixed now. I spoke with the VPS providers;
The jailed shell was removed from the webdev user (and the webmaster user?) and they reset the password. I logged into ssh as the webdev user to change the password and they told me off for trying and said I must do it through WHM/cPanel. I suspect there is some crazy arrangement here I don't know about and there is some link between those two accounts. When I tried (apparently wrongly) to change the webdev users password I still got "passwd: Authentication token manipulation error" but they said to leave it alone?!
I'm just glad its over, thanks everyone for your support :D