[CentOS] an actual hacked machine, in a preserved state

Sun Jan 1 23:27:32 UTC 2012
Bennett Haselton <bennett at peacefire.org>

On Sun, Jan 1, 2012 at 2:55 PM, Eero Volotinen <eero.volotinen at iki.fi>wrote:

> 2012/1/2 Bennett Haselton <bennett at peacefire.org>:
> > (Sorry, third time -- last one, promise, just giving it a subject line!)
> >
> > OK, a second machine hosted at the same hosting company has also
> apparently
> > been hacked.  Since 2 of out of 3 machines hosted at that company have
> now
> > been hacked, but this hasn't happened to any of the other 37 dedicated
> > servers that I've got hosted at other hosting companies (also CentOS,
> same
> > version or almost), this makes me wonder if there's a security breach at
> > this company, like if they store customers' passwords in a place that's
> > been hacked.  (Of course it could also be that whatever attacker found an
> > exploit, was just scanning that company's address space for hackable
> > machines, and didn't happen to scan the address space of the other
> hosting
> > companies.)
> >
> > So, following people's suggestions, the machine is disconnected and
> hooked
> > up to a KVM so I can still examine the files.  I've found this file:
> > -rw-r--r-- 1 root root 1358 Oct 21 17:40 /home/file.pl
> > which appears to be a copy of this exploit script:
> > http://archive.cert.uni-stuttgart.de/bugtraq/2006/11/msg00302.html
> > Note the last-mod date of October 21.
> >
> > No other files on the system were last modified on October 21st.  However
> > there was a security advisory dated October 20th which affected httpd:
> >
> http://mailinglist-archive.com/centos-announce/2011-10/00035-CentOSannounce+CESA20111392+Moderate+CentOS+5+i386+httpd+Update
> > https://rhn.redhat.com/errata/RHSA-2011-1392.html
> >
> > and a large number of files on the machine, including lots of files in */
> > usr/lib64/httpd/modules/* and */lib/modules/2.6.18-274.7.1.el5/kernel/* ,
> > have a last-mod date of October 20th.  So I assume that these are files
> > which were updated automatically by yum as a result of the patch that
> goes
> > with this advisory -- does that sound right?
> >
> > So a couple of questions that I could use some help with:
> >
> > 1) The last patch affecting httpd was released on October 20th, and the
> > earliest evidence I can find of the machine being hacked is a file dated
> > October 21st.  This could be just a coincidence, but could it also
> suggest
> > that the patch on October 20th introduced a new exploit, which the
> attacker
> > then used to get in on October 21st?
> >    (Another possibility: I think that when yum installs updates, it
> > doesn't actually restart httpd.  So maybe even after the patch was
> > installed, my old httpd instance kept running and was still vulnerable?
> As
> > for why it got hacked the very next day, maybe the attacker looked at the
> > newly released patch and reverse-engineered it to figure out where the
> > vulnerabilities were, that the patch fixed?)
> >
> > 2) Since the */var/log/httpd/* and /var/log/secure* logs only go back 4-5
> > weeks by default, it looks like any log entries related to how the
> attacker
> > would have gotten in on or before October 21st, are gone.  (The secure*
> > logs do show multiple successful logins as "root" within the last 4
> weeks,
> > mostly from IP addresses in Asia, but that's to be expected once the
> > machine was compromised -- it doesn't help track down how they originally
> > got in.)  Anywhere else that the logs would contain useful data?
>
> sshd with root login enabled with very bad password?
>
>
 Forgot to mention: the root password was: 1WyJstJZnQ!j   (I have since
changed it).

(I have already practically worn out my keyboard explaining the math behind
why I think a 12-character alphanumeric password is secure enough :) )

Bennett