[CentOS] How to tell if I've been hacked?

Sat Aug 22 16:49:17 UTC 2009
Bill Campbell <centos at celestial.com>

On Fri, Aug 21, 2009, Dave wrote:
>On Tue, Aug 18, 2009 at 3:53 PM, Scott Ehrlich<srehrlich at gmail.com> wrote:
... stuff deleted

>On Tue, Aug 18, 2009 at 6:57 PM, Bill Campbell<centos at celestial.com> wrote:
>> To really know whether a system has been hacked, it's necessary
>> to use something like Tripwire or Aide,
>
>And very carefully. Only that won't help you with memory-only attacks,
>or bios stuff, etc. These tools concentrate on verifying that your
>disk files have not been altered. I don't think they would help with
>an attack that uses free space (guessing here). Also, they are a pain,
>unless your system stays absolutely static, which in effect means, if
>you never use it. Have them ignore your data space, and the hacker can
>exploit that. And even then, linux is constantly updating various
>files in the background, and of course you need to update software to
>keep up with the security patches. You need to track every change of
>every file. I doubt many people have the patience.

One of the problems I've found with tripwire in particular and aide to a
lesser extent is that they (a) tend to be very verbose even when nothing
has changed, and (b) updating their database is fairly complex.  I have
developed a system that we use here and at our client sites that uses the
tripwire formatted configuration files, but maintains its own database, and
produces minimal reports of changes (none of nothing has changed).
Updating its database after changes have been checked and verified is a
simple file ``mv'' command.

I review daily reports from over 50 systems every morning, checking changes
found, usually taking no more than 10 minutes a day.  The key is to keep
the reports simple, and to make updating easy (and to have procedures that
monitor systems to be sure they's still alive and reporting in).

We also remove prelink from our kickstart installs on CentOS systems
because I think that the benefits of prelinking are marginal compared with
the problems it creates tracking system changes.  The changes of prelink
makes on a system can be removed by turning it off then the appropriate
/etc/sysconfig file and waiting a day for the daily maintenance to restore
things to their original condition.

>[snip]
>
>> It's also a good idea to check for executables in places they
>> normally shouldn't be, /tmp, /dev/shm on SuSE systems, /var/tmp,
>> and similar directories where crackers like to hide their work.
>> Often these executes will be in directories with names like ``.. ''
>> (note the trailing space) that look legitimate.
>
>I like this, because it might actually be automated. Of course, you're
>trusting stat or whatever.

Actually I'm trusing the python os.path.walk and ``file'' command
to check for executables.

>[snip]
>
>> You cannot trust tools like ``ps'', ``find'', ``netstat'', and
>> ``lsof'' as these are frequently replaced by ones that are
>> modified to hide the cracker's work.
>
>Naturally we are running aide and tripwire from a CD or other
>read-only medium, why not toss in a copy of these tools as well? Of
>course, if the kernel has been hacked, even that won't save us, but we
>have to take what we can get.

We create a file system initially, the same size as ``/'', and make a copy
of ``/'' in it identical except for the /etc/fstab entry.  This is not
mounted in normal operations, but the system can be booted from it to get
to a clean system.  Of course this must be updated using rsync after
significant changes in the root file system.

The key to all of this is to plan for security and intrusion detection at
the outset.

...
Bill
-- 
INTERNET:   bill at celestial.com  Bill Campbell; Celestial Software LLC
URL: http://www.celestial.com/  PO Box 820; 6641 E. Mercer Way
Voice:          (206) 236-1676  Mercer Island, WA 98040-0820
Fax:            (206) 232-9186  Skype: jwccsllc (206) 855-5792

"I do not feel obliged to believe that the same God who has endowed us
with sense, reason, and intellect has intended us to forego their use."
    -- Galileo Galilei