On Mon, Jan 2, 2012 at 12:04 AM, Craig White <craigwhite at azapple.com> wrote: > On Sun, 2012-01-01 at 14:23 -0800, Bennett Haselton wrote: > > (Sorry, third time -- last one, promise, just giving it a subject line!) > > > > OK, a second machine hosted at the same hosting company has also > apparently > > been hacked. Since 2 of out of 3 machines hosted at that company have > now > > been hacked, but this hasn't happened to any of the other 37 dedicated > > servers that I've got hosted at other hosting companies (also CentOS, > same > > version or almost), this makes me wonder if there's a security breach at > > this company, like if they store customers' passwords in a place that's > > been hacked. (Of course it could also be that whatever attacker found an > > exploit, was just scanning that company's address space for hackable > > machines, and didn't happen to scan the address space of the other > hosting > > companies.) > > > > So, following people's suggestions, the machine is disconnected and > hooked > > up to a KVM so I can still examine the files. I've found this file: > > -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl > > which appears to be a copy of this exploit script: > > http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html > > Note the last-mod date of October 21. > > > > No other files on the system were last modified on October 21st. However > > there was a security advisory dated October 20th which affected httpd: > > > http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update > > https://rhn.redhat.com/errata/RHSA-2011-1392.html > > > > and a large number of files on the machine, including lots of files in */ > > usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* , > > have a last-mod date of October 20th. So I assume that these are files > > which were updated automatically by yum as a result of the patch that > goes > > with this advisory -- does that sound right? > > > > So a couple of questions that I could use some help with: > > > > 1) The last patch affecting httpd was released on October 20th, and the > > earliest evidence I can find of the machine being hacked is a file dated > > October 21st. This could be just a coincidence, but could it also > suggest > > that the patch on October 20th introduced a new exploit, which the > attacker > > then used to get in on October 21st? > > (Another possibility: I think that when yum installs updates, it > > doesn't actually restart httpd. So maybe even after the patch was > > installed, my old httpd instance kept running and was still vulnerable? > As > > for why it got hacked the very next day, maybe the attacker looked at the > > newly released patch and reverse-engineered it to figure out where the > > vulnerabilities were, that the patch fixed?) > > > > 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5 > > weeks by default, it looks like any log entries related to how the > attacker > > would have gotten in on or before October 21st, are gone. (The secure* > > logs do show multiple successful logins as "root" within the last 4 > weeks, > > mostly from IP addresses in Asia, but that's to be expected once the > > machine was compromised -- it doesn't help track down how they originally > > got in.) Anywhere else that the logs would contain useful data? > ---- > the particular issue which was patched by this httpd (apache) update was > to fix a problem with reverse proxy so the first question is did this > server actually have a reverse proxy configured? > I'm not exactly sure how to tell but httpd.conf contains the lines LoadModule proxy_module modules/mod_proxy.so LoadModule cache_module modules/mod_cache.so so I think that means the answer is yes. > My next thought is that since this particular hacker managed to get > access to more than one of your machines, is it possible that there is a > mechanism (ie a pre-shared public key) that would allow them access to > the second server from the first server they managed to crack? The point > being that this computer may not have been the one that they originally > cracked and there may not be evidence of cracking on this computer. > OK, no I don't have pre-shared keys or any other link between them. > The script you identified would seem to be a script for attacking other > systems and by the time it landed on your system, it was already broken > into. Yes. That's what I've been saying to some apparently confused people who thought that the script was used to break into our server :) > There are some tools to identify a hackers access though they are often > obscured by the hacker... > > last # reads /var/log/wtmp and provides a list of users, login date/time > login duration, etc. Read the man page for last to get other options on > its usage including the '-f' option to read older wtmp log files if > needed. > Tried that -- looks like this has been "obscured by the hacker" as you said, since the output says "wtmp begins Sun Jan 1 03:03:28 2012". > lastb # reads /var/log/btmp much as above but list 'failed' logins > though this requires pro-active configuration and if you didn't do that, > you probably will do that going forward. > > looking at /etc/passwd to see what users are on your system and then > search their $HOME directories carefully for any evidence that their > account was the first one compromised. Very often, a single user with a > weak password has his account cracked and then a hacker can get a copy > of /etc/shadow and brute force the root password. > In the secure* logs going back the last four weeks I've seen "successful" logins for users called "ssh" and "bash". As far as I can tell those are not actually standard users on the system, so I assume the attacker created them to give themselves a backdoor in case I changed the root password, so I deleted those users. But when the server was first set up, the only user was root. > Consider that this type of activity is often done with 'hidden' files & > directories. This hacker was apparently brazen enough to operate openly > in /home so it's likely that he wasn't very concerned about his cracking > being discovered. > > The most important thing to do at this point is to figure out HOW they > got into your systems in the first place and discussions of SELinux and > yum updates are not useful to that end. Yes, you should always update > and always run SELinux but not useful in determining what actually > happened. > > Make a list of all open ports on this system, check the directories, > files, data from all daemons/applications that were exposed (Apache? > PHP?, MySQL?, etc.) and be especially vigilant to any directories where > user apache had write access. > portmap and rpc.statd are listening, and I can turn those off. (That's an easy change to make since it reduces complexity and hence the number of things that can go wrong, rather than increasing it.) Although I don't know how much safer that makes me -- I don't know of any data showing how frequently exploits are discovered (whether published or not) in those services, compared to others like httpd that cannot be turned off. (If there are more exploits discovered in httpd than there are in portmap and rpc.statd combined, then presumably turning off portmap and rpc.statd only reduces chance of a breakin by less than 50% -- not trivial, but not reassuring, either.) > > Again though, I am concerned that your first action on your first > discovered hacked server was to wipe it out and of a notion that it's > entirely possible that the actual cracking occurred on that system and > this (and perhaps other servers) are simply free gifts to the hacker > because they had pre-shared keys or the same root password. > > OK that's something to keep in mind as a reason never to use pre-shared keys or a common root password :) but I didn't do that in this case so the two servers should have been as independent of each other as any other two servers on the Internet.