[CentOS] an actual hacked machine, in a preserved state

Sun Jan 1 22:55:14 UTC 2012
Eero Volotinen <eero.volotinen at iki.fi>

2012/1/2 Bennett Haselton <bennett at peacefire.org>:
> (Sorry, third time -- last one, promise, just giving it a subject line!)
>
> OK, a second machine hosted at the same hosting company has also apparently
> been hacked.  Since 2 of out of 3 machines hosted at that company have now
> been hacked, but this hasn't happened to any of the other 37 dedicated
> servers that I've got hosted at other hosting companies (also CentOS, same
> version or almost), this makes me wonder if there's a security breach at
> this company, like if they store customers' passwords in a place that's
> been hacked.  (Of course it could also be that whatever attacker found an
> exploit, was just scanning that company's address space for hackable
> machines, and didn't happen to scan the address space of the other hosting
> companies.)
>
> So, following people's suggestions, the machine is disconnected and hooked
> up to a KVM so I can still examine the files.  I've found this file:
> -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl
> which appears to be a copy of this exploit script:
> http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html
> Note the last-mod date of October 21.
>
> No other files on the system were last modified on October 21st.  However
> there was a security advisory dated October 20th which affected httpd:
> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update
> https://rhn.redhat.com/errata/RHSA-2011-1392.html
>
> and a large number of files on the machine, including lots of files in */
> usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* ,
> have a last-mod date of October 20th.  So I assume that these are files
> which were updated automatically by yum as a result of the patch that goes
> with this advisory -- does that sound right?
>
> So a couple of questions that I could use some help with:
>
> 1) The last patch affecting httpd was released on October 20th, and the
> earliest evidence I can find of the machine being hacked is a file dated
> October 21st.  This could be just a coincidence, but could it also suggest
> that the patch on October 20th introduced a new exploit, which the attacker
> then used to get in on October 21st?
>    (Another possibility: I think that when yum installs updates, it
> doesn't actually restart httpd.  So maybe even after the patch was
> installed, my old httpd instance kept running and was still vulnerable? As
> for why it got hacked the very next day, maybe the attacker looked at the
> newly released patch and reverse-engineered it to figure out where the
> vulnerabilities were, that the patch fixed?)
>
> 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5
> weeks by default, it looks like any log entries related to how the attacker
> would have gotten in on or before October 21st, are gone.  (The secure*
> logs do show multiple successful logins as "root" within the last 4 weeks,
> mostly from IP addresses in Asia, but that's to be expected once the
> machine was compromised -- it doesn't help track down how they originally
> got in.)  Anywhere else that the logs would contain useful data?

sshd with root login enabled with very bad password?

--
Eero