On 01/02/2012 12:27 AM, Bennett Haselton wrote: > On Sun, Jan 1, 2012 at 2:55 PM, Eero Volotinen<eero.volotinen at iki.fi>wrote: > >> 2012/1/2 Bennett Haselton<bennett at peacefire.org>: >>> (Sorry, third time -- last one, promise, just giving it a subject line!) >>> >>> OK, a second machine hosted at the same hosting company has also >> apparently >>> been hacked. Since 2 of out of 3 machines hosted at that company have >> now >>> been hacked, but this hasn't happened to any of the other 37 dedicated >>> servers that I've got hosted at other hosting companies (also CentOS, >> same >>> version or almost), this makes me wonder if there's a security breach at >>> this company, like if they store customers' passwords in a place that's >>> been hacked. (Of course it could also be that whatever attacker found an >>> exploit, was just scanning that company's address space for hackable >>> machines, and didn't happen to scan the address space of the other >> hosting >>> companies.) >>> >>> So, following people's suggestions, the machine is disconnected and >> hooked >>> up to a KVM so I can still examine the files. I've found this file: >>> -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl >>> which appears to be a copy of this exploit script: >>> http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html >>> Note the last-mod date of October 21. >>> >>> No other files on the system were last modified on October 21st. However >>> there was a security advisory dated October 20th which affected httpd: >>> >> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update >>> https://rhn.redhat.com/errata/RHSA-2011-1392.html >>> >>> and a large number of files on the machine, including lots of files in */ >>> usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* , >>> have a last-mod date of October 20th. So I assume that these are files >>> which were updated automatically by yum as a result of the patch that >> goes >>> with this advisory -- does that sound right? >>> >>> So a couple of questions that I could use some help with: >>> >>> 1) The last patch affecting httpd was released on October 20th, and the >>> earliest evidence I can find of the machine being hacked is a file dated >>> October 21st. This could be just a coincidence, but could it also >> suggest >>> that the patch on October 20th introduced a new exploit, which the >> attacker >>> then used to get in on October 21st? >>> (Another possibility: I think that when yum installs updates, it >>> doesn't actually restart httpd. So maybe even after the patch was >>> installed, my old httpd instance kept running and was still vulnerable? >> As >>> for why it got hacked the very next day, maybe the attacker looked at the >>> newly released patch and reverse-engineered it to figure out where the >>> vulnerabilities were, that the patch fixed?) >>> >>> 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5 >>> weeks by default, it looks like any log entries related to how the >> attacker >>> would have gotten in on or before October 21st, are gone. (The secure* >>> logs do show multiple successful logins as "root" within the last 4 >> weeks, >>> mostly from IP addresses in Asia, but that's to be expected once the >>> machine was compromised -- it doesn't help track down how they originally >>> got in.) Anywhere else that the logs would contain useful data? >> >> sshd with root login enabled with very bad password? >> >> > Forgot to mention: the root password was: 1WyJstJZnQ!j (I have since > changed it). > > (I have already practically worn out my keyboard explaining the math behind > why I think a 12-character alphanumeric password is secure enough :) ) > The Errata you posted does not explain (to me at least) how they would get in. They would still need to brut-force the password. So if it was not a password, they used something else. And that VBullliten exploit looks like they used it to attack other sites with old VBullitens, from your server. Doesn't your Hosting company keep connection tracking from their rooters for 6 months, in case "police" requests them? You could track what happened from those logs. Also, have you used Logwatch to send daily log summaries to your mail? They could have had interesting data. Good practice is to setup remote logging by sending logs to server with rsyslog daemon with IP/host separation. It is done in real time, so attacker can only stop the log daemon on broken system and reconfigure them. One of the reasons I use Virtualmin package is because all servers (httpd, postfix, ....) are run under the user that owns that domain. Even if you have only one domain, all services are will lowered privileges, so the damage is lesser then when they are run from root account. -- Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe Google is the Mother, Google is the Father, and traceroute is your trusty Spiderman... StarOS, Mikrotik and CentOS/RHEL/Linux consultant