Here is the applicable article:
http://www.linux.com/feature/125548
There are links in the above article that explain tests for the system and what is currently known about the rootkit.
Apparently initial access is NOT via any vulnerability but just guessed root passwords.
There are currently 2 methods to see if you are infected:
1. In some cases, the root kit causes you to not be able to create directories starting with a number ... so as root do:
mkdir 1
If it gives you an error similar to this, you are probably infected:
mkdir: cannot create directory `1': No such file or directory
2. Run this command for several minutes while you have windows users connecting to your web server:
tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]{5}.js'"
If you get output from this script, you may be infected.
======================================================== More info:
http://www.cpanel.net/security/notes/random_js_toolkit.html
http://www.finjan.com/Pressrelease.aspx?id=1820&PressLan=1819&lan=3
http://www.pcworld.com/article/id,141358-c,techindustrytrends/article.html
http://www.webhostingtalk.com/showthread.php?t=651748
==========================================================
This does not seem to be caused by a specific vulnerability that CentOS or RHEL or cPanel has, but rather it seems to be caused by compromised root passwords.
There are several recommendations in the above links to prevent becoming infected as well as what to do if you are infected.
While there does not seem to be anything that the CentOS Development Team can "FIX" in relation to this issue ... I thought I would put the information out so that people can test their machines and take action as necessary.
Thanks, Johnny Hughes
Johnny Hughes wrote:
Here is the applicable article:
http://www.linux.com/feature/125548
There are links in the above article that explain tests for the system and what is currently known about the rootkit.
Apparently initial access is NOT via any vulnerability but just guessed root passwords.
There are currently 2 methods to see if you are infected:
- In some cases, the root kit causes you to not be able to create
directories starting with a number ... so as root do:
mkdir 1
If it gives you an error similar to this, you are probably infected:
mkdir: cannot create directory `1': No such file or directory
- Run this command for several minutes while you have windows users
connecting to your web server:
tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]{5}.js'"
If you get output from this script, you may be infected.
======================================================== More info:
http://www.cpanel.net/security/notes/random_js_toolkit.html
http://www.finjan.com/Pressrelease.aspx?id=1820&PressLan=1819&lan=3
http://www.pcworld.com/article/id,141358-c,techindustrytrends/article.html
http://www.webhostingtalk.com/showthread.php?t=651748
==========================================================
This does not seem to be caused by a specific vulnerability that CentOS or RHEL or cPanel has, but rather it seems to be caused by compromised root passwords.
There are several recommendations in the above links to prevent becoming infected as well as what to do if you are infected.
While there does not seem to be anything that the CentOS Development Team can "FIX" in relation to this issue ... I thought I would put the information out so that people can test their machines and take action as necessary.
As a secondary note, the CentOS Security Team (and also the upstream security team) would like to have access to an infected machine for analysis, if anyone is infected and if they can spare the machine for several days for us to analyze (you should change your root passwd and take apache off line ... meaning you would need to stand up another web server to replace this one).
So, if you have a machine for the cause that was infected in the wild that you can spare, you can contact the CentOS Security team at:
security_AT_centos.org
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
Thanks, Johnny Hughes
On Mon, 2008-01-28 at 19:55 -0600, Johnny Hughes wrote:
Johnny Hughes wrote:
Here is the applicable article:
http://www.linux.com/feature/125548
There are links in the above article that explain tests for the system and what is currently known about the rootkit.
Apparently initial access is NOT via any vulnerability but just guessed root passwords.
There are currently 2 methods to see if you are infected:
- In some cases, the root kit causes you to not be able to create
directories starting with a number ... so as root do:
mkdir 1
If it gives you an error similar to this, you are probably infected:
mkdir: cannot create directory `1': No such file or directory
- Run this command for several minutes while you have windows users
connecting to your web server:
tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]{5}.js'"
If you get output from this script, you may be infected.
======================================================== More info:
http://www.cpanel.net/security/notes/random_js_toolkit.html
http://www.finjan.com/Pressrelease.aspx?id=1820&PressLan=1819&lan=3
http://www.pcworld.com/article/id,141358-c,techindustrytrends/article.html
http://www.webhostingtalk.com/showthread.php?t=651748
==========================================================
This does not seem to be caused by a specific vulnerability that CentOS or RHEL or cPanel has, but rather it seems to be caused by compromised root passwords.
There are several recommendations in the above links to prevent becoming infected as well as what to do if you are infected.
While there does not seem to be anything that the CentOS Development Team can "FIX" in relation to this issue ... I thought I would put the information out so that people can test their machines and take action as necessary.
As a secondary note, the CentOS Security Team (and also the upstream security team) would like to have access to an infected machine for analysis, if anyone is infected and if they can spare the machine for several days for us to analyze (you should change your root passwd and take apache off line ... meaning you would need to stand up another web server to replace this one).
So, if you have a machine for the cause that was infected in the wild that you can spare, you can contact the CentOS Security team at:
security_AT_centos.org
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
---- doesn't this almost beg for upstream to make denyhosts a base install and automatically on, just as sshd is automatically on?
Craig
Craig White wrote:
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
doesn't this almost beg for upstream to make denyhosts a base install and automatically on, just as sshd is automatically on?
I've always wondered why a program like sshd didn't rate-limit connection attempts from day one. It's not exactly a new concept, especially for a security-oriented program.
On Jan 28, 2008 10:14 PM, Les Mikesell lesmikesell@gmail.com wrote:
Craig White wrote:
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
doesn't this almost beg for upstream to make denyhosts a base install and automatically on, just as sshd is automatically on?
I've always wondered why a program like sshd didn't rate-limit connection attempts from day one. It's not exactly a new concept, especially for a security-oriented program.
It's a question of scale. For some systems, 30 people logging in is too many. For others, it's 3000. There is no 'right' default value. It should be (and is) left up to the admin and iptables.
On Mon, 2008-01-28 at 22:19 -0500, Jim Perrin wrote:
On Jan 28, 2008 10:14 PM, Les Mikesell lesmikesell@gmail.com wrote:
Craig White wrote:
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
doesn't this almost beg for upstream to make denyhosts a base install and automatically on, just as sshd is automatically on?
I've always wondered why a program like sshd didn't rate-limit connection attempts from day one. It's not exactly a new concept, especially for a security-oriented program.
It's a question of scale. For some systems, 30 people logging in is too many. For others, it's 3000. There is no 'right' default value. It should be (and is) left up to the admin and iptables.
---- conjecture aside, that doesn't stop us from submitting an RFE upstream.
Craig
Jim Perrin wrote:
On Jan 28, 2008 10:14 PM, Les Mikesell lesmikesell@gmail.com wrote:
Craig White wrote:
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
doesn't this almost beg for upstream to make denyhosts a base install and automatically on, just as sshd is automatically on?
I've always wondered why a program like sshd didn't rate-limit connection attempts from day one. It's not exactly a new concept, especially for a security-oriented program.
It's a question of scale. For some systems, 30 people logging in is too many. For others, it's 3000. There is no 'right' default value. It should be (and is) left up to the admin and iptables.
You have to have some default and it might as well be on the secure side since as you suggest you won't be right for everyone. But, you don't have to rate-limit connections in general, you just need a delay after a failed attempt before permitting another attempt from the same place. I thought getty/login always had such a delay to discourage password guessing so it seemed odd for ssh to ignore this risk.
Les Mikesell wrote:
Craig White wrote:
We will work also with the Red Hat Security team and see if we can isolate any issues that might be FIXABLE.
doesn't this almost beg for upstream to make denyhosts a base install and automatically on, just as sshd is automatically on?
I've always wondered why a program like sshd didn't rate-limit connection attempts from day one. It's not exactly a new concept, especially for a security-oriented program.
I actually think RedHat has moved backwards in this area. I'm seeing dictionary attacks on ssh, vsftp, dovecot, samba, virtually every service which might be available out to the web. Gaining access in any of these areas is the first step to a compromised system.
ssh and vsftp seem to be the most often attacked... I have had ssh set to deny all and allow only known IP addresses of known users who need the service... still not perfect by any means, as somewhere along the line someone is going to need access while their connection is dynamic... just hadn't hit that one yet.
I have to wonder about vsftp... Yes it's fast, but I wonder if some of this speed comes from not doing checks that really need to be done, like keeping up with multiple failed logins. Seems like wu and pro both had settings for this within their config files? But, even if we take the UNIX ideal for doing things, the modular approach... I am very surprised that RHEL doesn't appear to have any system within the provided packages which can be set to deal with the various servers in some straight forward manner. Yes, there are programs out there. I'm running one of them. But why are we left with this one shortcoming by upstream?
Sorry, this just seems to be really odd to me. Dealing with each external system, is dealing with yet one more system to follow. Each time, there may be a new issue introduced with regards to a conflict on a server... the whole reason for following upstream as much as possible. Each one also introduces the need to follow another mailing list. It's just not very efficient nor as safe, when compared to yum or up2date updates.
As for changes to passwords. Sure, changing the root password is a great idea. But then, what about all the users? It's absurd to consider making all the users on a hosting server change their passwords once a month, once a year or even once every ten years. They can barely keep up with the one they have and many don't. Most don't know how to configure their email client. Entry into a system from any service opens up a lot of potentials. I really don't get why there is not a system in place to deal with this just as we have selinux, suexec, etc.
John Hinton
Along the lines of staying safe, now is probably a good time to check your password policies.
1. Don't allow root access to ssh. (modify /etc/ssh/sshd_config) 2. restrict root logins to only the local machine. (modify /etc/securetty) 3. Limit users with access to 'su' to the wheel group (use visudo and also modify /etc/pam.d/su) 4. Make sure root is the only one with a uid of 0. ( awk -F: '($3 == "0") {print}' /etc/passwd ) 5. Use pam to require strong passwords. (install/use pam_passwdqc which is part of the base distro, modify /etc/pam.d/system-auth ) 6. Use denyhosts or pam.tally2 to restrict login attempts. 7. use ssh keys.
And above all, because I know many admins slack on this, and I'm guilty of it as well if it's not forced... ROTATE your passwords periodically
The recommended password requirements for root: at least 10 characters with a mix of upper/lower case, special characters, and numbers.
Discussion, and alternate suggestions welcome.
On Mon, 28 Jan 2008 22:36:03 -0500 Jim Perrin jperrin@gmail.com wrote:
And above all, because I know many admins slack on this, and I'm guilty of it as well if it's not forced... ROTATE your passwords periodically
I have never understood this. If I have a good, strong password that nobody knows, how is changing it to another one an improvement over what I already have?
I have never understood this.
Exactly why you should be more proactive, if it matters in your environment.
If I have a good, strong password that nobody knows
Do you know this?
CMIIW, But an example from the windows world would be *if* someone sniffed the hash of an admin login, then took it home and ran it against the rainbow tables, your hosed. That's one weak example of why it's good practice, extrapolate this a million ways for a million angles on this scenario. It's easier to be proactive against what you don't know then reactive against what you just found out:)
Some environments don't need high security, some do and some depend so severely on it that they don't have any access to a public network in any form.
jlc
Frank Cox wrote:
On Mon, 28 Jan 2008 22:36:03 -0500 Jim Perrin jperrin@gmail.com wrote:
And above all, because I know many admins slack on this, and I'm guilty of it as well if it's not forced... ROTATE your passwords periodically
I have never understood this. If I have a good, strong password that nobody knows, how is changing it to another one an improvement over what I already have?
I agree with you.
A company I worked for required rotation of passwords and strong passwords. We fired one of the sysadmins because he had a problem coming in to work late.
Take a wild guess at what we found taped to the bottom of his keyboard. Requiring password rotation increases the occurrences of that issue.
Rotating passwords IMHO should only be done when their is a possibility that the shadow file has been compromised or an employee with root access is dismissed on bad terms.
A better thing to do is disable remote root login, be extremely careful with sudo (it should not be allowed to spawn a shell for any user), and log to a log server rather than local filesystem.
On Jan 28, 2008 9:19 PM, Michael A. Peters mpeters@mac.com wrote:
Frank Cox wrote:
On Mon, 28 Jan 2008 22:36:03 -0500 Jim Perrin jperrin@gmail.com wrote:
And above all, because I know many admins slack on this, and I'm guilty of it as well if it's not forced... ROTATE your passwords periodically
I have never understood this. If I have a good, strong password that nobody knows, how is changing it to another one an improvement over what I already have?
I agree with you.
A company I worked for required rotation of passwords and strong passwords. We fired one of the sysadmins because he had a problem coming in to work late.
Take a wild guess at what we found taped to the bottom of his keyboard. Requiring password rotation increases the occurrences of that issue.
I am sorry but that is a logical fallacy if I have ever seen. I have seen lots of people who come in on time and stay late who have passwords taped to the bottom of their keyboards... and they never had to change their passwords. And I know lots of people who do not do this who have to change their passwords every 90 days.
Rotating passwords comes from the following theories:
1) As in cryptography, you must assume that the attacker knows everything you know and probably something more. 2) You do not know where the attacker is.
Thus for a networked system or a system with multiple users, you must assume that within a certain amount of time, your hashes have been seen. Then you multiply it by the amount of time it would take to 'crack' that hash with precomputed hash tables and/or multi-system hacks. With the price of a cluster of 10,000 botted computers being pretty low.. you can assume that multi-system hacks are possible. Then you look at the value of your data. From that you can come up with how long before your password needs to be rotated.
Using 2-3 factor authentication lowers this risk, and using 1 time passwords also does. However the cost of doing so in training, materials, etc may be more than what you wish to look for.
Rotating passwords IMHO should only be done when their is a possibility that the shadow file has been compromised or an employee with root access is dismissed on bad terms.
A better thing to do is disable remote root login, be extremely careful with sudo (it should not be allowed to spawn a shell for any user), and log to a log server rather than local filesystem.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
General note:
"loose" - pronounced "luce" (like Henry Luce, publisher of Time magazine), rhymes with goose - means not tight, or insecure or, as a verb, to let go of.
"lose" - pronounced "looz," rhymes with booze (I know you'll understand that one! :-) - means to fail to hold on to or keep track of, misplace, etc.
I'm sure it was just a typo, but I see it enough that I just had to say something. Every once in a while it just bugs me - YMMV.
:-)
mhr
Jim Perrin wrote:
Along the lines of staying safe, now is probably a good time to check your password policies.
- Don't allow root access to ssh. (modify /etc/ssh/sshd_config)
why isn't this the default?
- restrict root logins to only the local machine. (modify /etc/securetty)
- Limit users with access to 'su' to the wheel group (use visudo and
also modify /etc/pam.d/su)
same question here.
- Make sure root is the only one with a uid of 0. ( awk -F: '($3 ==
"0") {print}' /etc/passwd ) 5. Use pam to require strong passwords. (install/use pam_passwdqc which is part of the base distro, modify /etc/pam.d/system-auth ) 6. Use denyhosts or pam.tally2 to restrict login attempts. 7. use ssh keys.
[snip]
On Jan 29, 2008 5:52 AM, mouss mouss@netoyen.net wrote:
Jim Perrin wrote:
Along the lines of staying safe, now is probably a good time to check your password policies.
- Don't allow root access to ssh. (modify /etc/ssh/sshd_config)
why isn't this the default?
Taking an educated guess on this one, I'd say to allow configuration after a remote install.
- restrict root logins to only the local machine. (modify /etc/securetty)
- Limit users with access to 'su' to the wheel group (use visudo and
also modify /etc/pam.d/su)
same question here.
For this one I'd guess that it's because by default folks don't get added to wheel. So if an admin forgets to add his own user account, he can no longer gain root with 'su'. He has to walk his happy ass to the console to log in. Everything about the *nix culture points to not walking anywhere except possibly to a pub :-P
Jim Perrin wrote:
On Jan 29, 2008 5:52 AM, mouss mouss@netoyen.net wrote:
Jim Perrin wrote:
Along the lines of staying safe, now is probably a good time to check your password policies.
- Don't allow root access to ssh. (modify /etc/ssh/sshd_config)
why isn't this the default?
Taking an educated guess on this one, I'd say to allow configuration after a remote install.
- restrict root logins to only the local machine. (modify /etc/securetty)
- Limit users with access to 'su' to the wheel group (use visudo and
also modify /etc/pam.d/su)
same question here.
For this one I'd guess that it's because by default folks don't get added to wheel. So if an admin forgets to add his own user account, he can no longer gain root with 'su'. He has to walk his happy ass to the console to log in. Everything about the *nix culture points to not walking anywhere except possibly to a pub :-P
Well ... not to say anything bad about beer, BUT
The real reason is that RHEL does not ship that way, so CentOS does not either.
The bottom line for this and all other questions like it is this:
We clone the configuration of the upstream system on purpose so that CentOS performs as much as possible like the upstream product ... if/when they change the defaults, so will we.
Thanks, Johnny Hughes
Johnny Hughes wrote:
Jim Perrin wrote:
The real reason is that RHEL does not ship that way, so CentOS does not either.
The bottom line for this and all other questions like it is this:
We clone the configuration of the upstream system on purpose so that CentOS performs as much as possible like the upstream product ... if/when they change the defaults, so will we.
Thanks, Johnny Hughes
If you don't like the defaults, get anaconda to change them for you. Or write a script that you run shortly after install to make the changes for you.
Milton Calnek wrote:
If you don't like the defaults, get anaconda to change them for you. Or write a script that you run shortly after install to make the changes for you.
That would be pretty amazing if at the end (or at the beginning) of the install there was some checkbox that said something to the effect of:
"Would you like to maintain compatibility with upstream security defaults or would you like to follow our more sensible recommendations instead?"
And if the user chooses the latter, a much more secure default configuration could be applied. That might go a long way towards helping non-wizard folks to enjoy some measure of additional protection by default. Just a thought.
Best,
Chris
Chris Mauritz wrote:
Milton Calnek wrote:
If you don't like the defaults, get anaconda to change them for you. Or write a script that you run shortly after install to make the changes for you.
That would be pretty amazing if at the end (or at the beginning) of the install there was some checkbox that said something to the effect of:
"Would you like to maintain compatibility with upstream security defaults or would you like to follow our more sensible recommendations instead?"
And if the user chooses the latter, a much more secure default configuration could be applied. That might go a long way towards helping non-wizard folks to enjoy some measure of additional protection by default. Just a thought.
Or, package the more sensible configuration (according to your expert judgement...) in centosplus for easy addition later.
on 1/29/2008 8:00 AM Chris Mauritz spake the following:
Milton Calnek wrote:
If you don't like the defaults, get anaconda to change them for you. Or write a script that you run shortly after install to make the changes for you.
That would be pretty amazing if at the end (or at the beginning) of the install there was some checkbox that said something to the effect of:
"Would you like to maintain compatibility with upstream security defaults or would you like to follow our more sensible recommendations instead?"
And if the user chooses the latter, a much more secure default configuration could be applied. That might go a long way towards helping non-wizard folks to enjoy some measure of additional protection by default. Just a thought.
But again, that breaks upstream compatibility. Besides, all of you know that there are people that click "yes" on every dialog box without reading them. I swear that if you added a dialog box that stated their firstborn would be sacrificed to the IT gods, and recorded the answers, you would get a large percentage of "yes" clicks. And most of those would be unintentional.
on 1/29/2008 3:50 AM Jim Perrin spake the following:
On Jan 29, 2008 5:52 AM, mouss mouss-EcCAZ+sBjEfR7s880joybQ@public.gmane.org wrote:
Jim Perrin wrote:
Along the lines of staying safe, now is probably a good time to check your password policies.
- Don't allow root access to ssh. (modify /etc/ssh/sshd_config)
why isn't this the default?
Taking an educated guess on this one, I'd say to allow configuration after a remote install.
- restrict root logins to only the local machine. (modify /etc/securetty)
- Limit users with access to 'su' to the wheel group (use visudo and
also modify /etc/pam.d/su)
same question here.
For this one I'd guess that it's because by default folks don't get added to wheel. So if an admin forgets to add his own user account, he can no longer gain root with 'su'. He has to walk his happy ass to the console to log in. Everything about the *nix culture points to not walking anywhere except possibly to a pub :-P
You mean I have to walk to the pub, too? ;-D
On Tue, 29 Jan 2008 11:39:28 -0500 Chris Mauritz chrism@imntv.com wrote:
I'm sure somebody somewhere has written a 1 line perl script (and printed it on a T-shirt) that can magically make beer appear in your hands upon execution.
perl -e 'print "Martha!\n"'
?
on 1/29/2008 8:39 AM Chris Mauritz spake the following:
Scott Silva wrote:
You mean I have to walk to the pub, too? ;-D
I'm sure somebody somewhere has written a 1 line perl script (and printed it on a T-shirt) that can magically make beer appear in your hands upon execution.
:)
I tried grep beer and the system went off looking for some. I had to send a break before it would quit looking! I guess I taught it right! ;-P
On Mon, Jan 28, 2008 at 10:36:03PM -0500, Jim Perrin wrote:
Along the lines of staying safe, now is probably a good time to check your password policies.
- Don't allow root access to ssh. (modify /etc/ssh/sshd_config)
- restrict root logins to only the local machine. (modify /etc/securetty)
- Limit users with access to 'su' to the wheel group (use visudo and
also modify /etc/pam.d/su) 4. Make sure root is the only one with a uid of 0. ( awk -F: '($3 == "0") {print}' /etc/passwd ) 5. Use pam to require strong passwords. (install/use pam_passwdqc which is part of the base distro, modify /etc/pam.d/system-auth ) 6. Use denyhosts or pam.tally2 to restrict login attempts. 7. use ssh keys.
And above all, because I know many admins slack on this, and I'm guilty of it as well if it's not forced... ROTATE your passwords periodically
The recommended password requirements for root: at least 10 characters with a mix of upper/lower case, special characters, and numbers.
Discussion, and alternate suggestions welcome.
I will add to that list, change ssh port 22 to somthing else
Regards
Alfredo The Sauce