On 1/2/2012 7:29 AM, Johnny Hughes wrote: > On 01/02/2012 02:04 AM, Craig White wrote: >> On Sun, 2012-01-01 at 14:23 -0800, Bennett Haselton wrote: >>> (Sorry, third time -- last one, promise, just giving it a subject line!) >>> >>> OK, a second machine hosted at the same hosting company has also apparently >>> been hacked. Since 2 of out of 3 machines hosted at that company have now >>> been hacked, but this hasn't happened to any of the other 37 dedicated >>> servers that I've got hosted at other hosting companies (also CentOS, same >>> version or almost), this makes me wonder if there's a security breach at >>> this company, like if they store customers' passwords in a place that's >>> been hacked. (Of course it could also be that whatever attacker found an >>> exploit, was just scanning that company's address space for hackable >>> machines, and didn't happen to scan the address space of the other hosting >>> companies.) >>> >>> So, following people's suggestions, the machine is disconnected and hooked >>> up to a KVM so I can still examine the files. I've found this file: >>> -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl >>> which appears to be a copy of this exploit script: >>> http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html >>> Note the last-mod date of October 21. >>> >>> No other files on the system were last modified on October 21st. However >>> there was a security advisory dated October 20th which affected httpd: >>> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update >>> https://rhn.redhat.com/errata/RHSA-2011-1392.html >>> >>> and a large number of files on the machine, including lots of files in */ >>> usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* , >>> have a last-mod date of October 20th. So I assume that these are files >>> which were updated automatically by yum as a result of the patch that goes >>> with this advisory -- does that sound right? >>> >>> So a couple of questions that I could use some help with: >>> >>> 1) The last patch affecting httpd was released on October 20th, and the >>> earliest evidence I can find of the machine being hacked is a file dated >>> October 21st. This could be just a coincidence, but could it also suggest >>> that the patch on October 20th introduced a new exploit, which the attacker >>> then used to get in on October 21st? >>> (Another possibility: I think that when yum installs updates, it >>> doesn't actually restart httpd. So maybe even after the patch was >>> installed, my old httpd instance kept running and was still vulnerable? As >>> for why it got hacked the very next day, maybe the attacker looked at the >>> newly released patch and reverse-engineered it to figure out where the >>> vulnerabilities were, that the patch fixed?) >>> >>> 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5 >>> weeks by default, it looks like any log entries related to how the attacker >>> would have gotten in on or before October 21st, are gone. (The secure* >>> logs do show multiple successful logins as "root" within the last 4 weeks, >>> mostly from IP addresses in Asia, but that's to be expected once the >>> machine was compromised -- it doesn't help track down how they originally >>> got in.) Anywhere else that the logs would contain useful data? >> ---- > At this point, I would like to point out that if the machine initially > had ssh access only from subnets where you would have logged in from > then the Asian subnets would have been excluded. (Assuming that the > original attack came from there) > > You can also set up openvpn on the server and control ports like ssh to > only be open to you if you are using an openvpn client to connect to the > machine. True but I travel a lot and sometimes need to connect to the machines from subnets that I don't know about in advance. If I used openvpn to connect, and then connected via ssh over openvpn, this seems like essentially security through obscurity :) by just replacing the public ssh daemon with a different public daemon (with a different connection protocol) which an attacker could try to brute-force the same way they could try to brute-force sshd. However it still seems that this would only matter if the attacker got in by brute-forcing the login. If they obtained the ability to run privileged commands any other way, then (1) they could continue to run privileged commands that way anyway, or (2) as their first action they could just remove all the IP address restrictions on ssh connections at which point they could connect normally via ssh from anywhere. So if this only matters when the attacker is trying to brute-force the login, and I still think that a 12-character random password is un-bruteforceable which makes the IP restrictions moot. If I'm wrong, then why? What do you think -- if my password is already a 12-character random string, do think it adds additional security to restrict ssh logins to only subnets that I'm logging in from? If so, then what's a specific scenario where the attacker would be able to break in (or would have a larger chance of breaking in) if I'm not restricting ssh logins by IP, but would not be able to break in if I were restricting ssh logins? >> the particular issue which was patched by this httpd (apache) update was >> to fix a problem with reverse proxy so the first question is did this >> server actually have a reverse proxy configured? >> >> My next thought is that since this particular hacker managed to get >> access to more than one of your machines, is it possible that there is a >> mechanism (ie a pre-shared public key) that would allow them access to >> the second server from the first server they managed to crack? The point >> being that this computer may not have been the one that they originally >> cracked and there may not be evidence of cracking on this computer. >> >> The script you identified would seem to be a script for attacking other >> systems and by the time it landed on your system, it was already broken >> into. > I agree with Craig, the fact that the script is owned by root.root > indicates that it was likely not put there by the apache daemon itself > (unless you have the User and Group variable for your apache set to root > and root). > > This would tend to support Craig's assertion that the script was placed > on the system after the fact and was not the source of the attack or > placed there through an apache flaw. Yes I agreed with Craig too :) Maybe my first post was not clear but I always assumed that they broke in first and put that file there afterwards, not that the file somehow enabled the break-in. > You can look in your apache config file locations to see if /home/ is > listed as a "DocumentRoot" location or if there is a alias that contains > /home/. > > You are also going to need to be able to execute CGI scripts ... so > there is probably a "Options +ExecCGI" or "ScriptAlias" somewhere in a > config file (either in /etc/httpd/conf/http.conf or a *.conf file in > /etc/httpd/conf.d/ ... assuming they did not totally replace apache and > setup a different location to read in config files). OK I checked. No "/home/" in httpd.conf or in any of the .conf files under conf.d. > This does not help you with the breakin ... it does help you find if any > of your other machines are being used for the same thing. > >> There are some tools to identify a hackers access though they are often >> obscured by the hacker... >> >> last # reads /var/log/wtmp and provides a list of users, login date/time >> login duration, etc. Read the man page for last to get other options on >> its usage including the '-f' option to read older wtmp log files if >> needed. >> >> lastb # reads /var/log/btmp much as above but list 'failed' logins >> though this requires pro-active configuration and if you didn't do that, >> you probably will do that going forward. >> >> looking at /etc/passwd to see what users are on your system and then >> search their $HOME directories carefully for any evidence that their >> account was the first one compromised. Very often, a single user with a >> weak password has his account cracked and then a hacker can get a copy >> of /etc/shadow and brute force the root password. >> >> Consider that this type of activity is often done with 'hidden' files& >> directories. This hacker was apparently brazen enough to operate openly >> in /home so it's likely that he wasn't very concerned about his cracking >> being discovered. > Right, but you can find other directories in the apache config files if > those are found. > >> The most important thing to do at this point is to figure out HOW they >> got into your systems in the first place and discussions of SELinux and >> yum updates are not useful to that end. Yes, you should always update >> and always run SELinux but not useful in determining what actually >> happened. >> >> Make a list of all open ports on this system, check the directories, >> files, data from all daemons/applications that were exposed (Apache? >> PHP?, MySQL?, etc.) and be especially vigilant to any directories where >> user apache had write access. >> >> Again though, I am concerned that your first action on your first >> discovered hacked server was to wipe it out and of a notion that it's >> entirely possible that the actual cracking occurred on that system and >> this (and perhaps other servers) are simply free gifts to the hacker >> because they had pre-shared keys or the same root password. > I would also look at that 3rd server very closely since they probably > also had access there if they knew you owned it. Well with all of the break-ins happening at one company, and other problems with them, I've been thinking that I might as well cancel all the servers hosted there anyway. Since I don't have any data stored on them and I can re-create the systems from scratch, might as well re-create them from scratch somewhere else. There's always the possibility that the hosting company stored all of the root passwords somewhere where an attacker found them. Bennett