[CentOS] an actual hacked machine, in a preserved state

Mon Jan 2 12:03:05 UTC 2012
Bennett Haselton <bennett at peacefire.org>

On Sun, Jan 1, 2012 at 6:04 PM, Ljubomir Ljubojevic <office at plnet.rs> wrote:

> On 01/02/2012 02:50 AM, Bennett Haselton wrote:
> > I'm not sure what you mean by "an exploit from a web board which is
> > apparently designed to pull outside traffic".  Like Ljubomir said, it
> looks
> > like a script that is used from machine X to DOS attack machine Y, if
> > machine Y has the VBulletin bulletin board software installed on it.  Is
> > that what you meant?:)
> >
> > But anyway, since the file was located at /home/file.pl (and since
> attacker
> > had console access), presumably it wasn't being invoked by the web
> server,
> > only from the command line.  So how would it have made any difference if
> > httpd was running in its own context, if that script was not being
> invoked
> > by httpd?
>
> Nobody of us really knows how they got in. All you will get from this
> mailing list will be speculations, apart from useful instructions how to
> gather as much info as possible.


Speculation is fine, I'm interested in any method that the attacker *could*
have used to get in to the server if it's *logically possible* that the
attack could have had a non-trivial chance of working.  (If they could have
gotten in by method A, and they could have gotten in by method B, then
regardless of which they DID use, I should still try to fix both A and B.)

But that still excludes things like "they got in because you used a
password instead of an ssh key", as I have said many times why a
12-character random alphanumeric password (with 10^24 possible values) is
secure enough even with the most pessimistic assumptions about botnets and
GPUs.



> So there are many possible ways they
> got in including brute force.


Knowing that it was a 12-character random alphanumeric password, do you
still think it could have been done by brute force?


>

As I understood you, you do not use
> neither fail2ban, denyhosts or/and logwatch, and you haven't checked
> those two servers very much in recent months.
>
> What Rilindo is saying is that SELinux might detect exploits while their
> trying to break processes from their routine (allowed by SELinux), and
> all of this (if it happened via exploits) might have been prevented by
> SELinux. You really do have lot of gaps in your security.


A couple of people have pointed out "gaps" in what I was doing for
*detection and analysis* after the fact (don't wipe a machine after it's
hacked, have logs going back further than 4 weeks, have the logs backed up
off-site, etc.), and that's useful.  But as far as I can tell nobody has
really pointed out any "gap" in what I'm doing on the *prevention* side.

To focus on one exact question: What is a *preventive* measure that you
think I am not doing, that you think would reduce the chance of a break-in,
in some specific scenario?

Take fail2ban and denyhosts, which block connections from IPs that make too
many invalid ssh logins.  In the scenario where the attacker is trying to
brute-force the login, then if your ssh login password is already
un-bruteforceable, then blocking connections from IPs that are making too
many invalid attempts, will not reduce the chance of an attacker
brute-force (which is already essentially zero).

Do you think I'm incorrect that fail2ban and/or denyhosts (and switching
from passwords to ssh keys) do not reduce the chance of a breakin, if your
ssh password is a truly random mixed-case-alphanumeric 12-character
string?  If you think this is incorrect, why?




> If I were you,
> I would use all advice's given to you here and secure the rest of your
> servers (SELinux, fail2ban/denyhosts, logwatch, rsyslog, etc..)
>

I tried SELinux but it broke so much needed functionality on the server
that it was not an option.  There are many, many issues where all the forum
discussions about how to solve the problem, just end with "Just turn off
selinux", and nobody apparently has any other idea how to solve it.  But in
any case, what exactly does it do for httpd security, that isn't already
accomplished by having httpd run as the unprivileged "apache" user?  You
said SELinux could prevent an exploit from "breaking a process from its
routine".  But even without SELinux, an attacker who found an exploit that
could take control of httpd and make it try any action he wanted, still
wouldn't be able to actually do anything while running as "apache", would
they?