[CentOS] How to tell if I've been hacked?

Sat Aug 22 23:07:02 UTC 2009
Bill Campbell <centos at celestial.com>

On Sat, Aug 22, 2009, Dave wrote:
>On Sat, Aug 22, 2009 at 6:49 AM, Bill Campbell<centos at celestial.com> wrote:
>> I review daily reports from over 50 systems every morning, checking changes
>> found, usually taking no more than 10 minutes a day.  The key is to keep
>> the reports simple, and to make updating easy (and to have procedures that
>> monitor systems to be sure they's still alive and reporting in).
>
>So how do you track the inevitable changes? Not saying you can't, just
>curious. For me, when I look at a batch of changes, some of them are
>obviously stuff I've done, other stuff not so obvious. I also filter
>reports through a script that sort of does a diff and makes an attempt
>to limit the boilerplate. Sometimes it is a bit too terse.

First off, we don't allow automatic updates on most systems, much
preferring to do them manually making it pretty easy to refresh
the comparison database immediately after the update is complete.
The odds that a cracker will get in and do their dirty deeds
while this are going on are pretty low, and can probably be
ignored.

We handle pretty much all server stuff under the OpenPKG portable
package management system so things like spamassassin, amavisd,
clamav, and postfix are not the distribution versions, but those
from OpenPKG (which are generally updated more quickly then the
distribution's).  A typical occurrence will be that we get an
e-mail saying that clamav is out of date from the nightly
freshclam update, I will pick up the new sources, update the
OpenPKG SRPM for it, and deploy it 40 or so systems running it,
and expect to see a corresponding set of notices the next morning
that files under clamav have changed.

The clusterssh program makes this sort of thing much more efficient
as one can execute shell commands on multiple systems simultaneously.

>> We create a file system initially, the same size as ``/'', and make a copy
>> of ``/'' in it identical except for the /etc/fstab entry.  This is not
>> mounted in normal operations, but the system can be booted from it to get
>> to a clean system.
>
>Wow, elaborate. How do you protect this file system from intruders?
>Exterrnal and powerred off?

That's one way to do it.  We also run a fair number of Linux
servers under VMware so periodic snapshots and backups simplify
the task.

I have not seen many successful cracks of Linux boxes that we
have configured from scratch.  Some basic things can be done to
minimize the chances of cracks.

   + Create the baseline for intrusion detection tools before putting the
     syste on line, and monitor it daily.

   + Configure openssh to refuse password authentication requiring
     authorized_keys access.

   + Configure openssh with tcp_wrappers support, restricting access by IP
     address and/or domain names.  I consider this absolutely mandatory if
     one needs to all username and password authentication.

   + Use fail2ban or similar techniques to quickly block IP addresses that
     are found probing the system (don't forget to look at POP and IMAP
     logs for failed login attempts).

   + Use /bin/false as the standard shell for accounts that don't have good
     reason for shell access.  This does not affect e-mail or most services
     that a typical ISP customer needs.

   + Use OpenVPN for access.  This works well even when in hotels with NAT
     firewalls, and is not easily hacked anonymously.

   + Restrict access of webmin and usermin to local networks so they are
     not vulnerable to outside attack.  These services are available to
     people outside connecting with OpenVPN.

   + Restrict webmail, pop, and imap access to secure connections using
     https, tls, ssl.  We have never been able to get the average ISP
     customer to use good passwords, but every little bit helps.

Bill
-- 
INTERNET:   bill at celestial.com  Bill Campbell; Celestial Software LLC
URL: http://www.celestial.com/  PO Box 820; 6641 E. Mercer Way
Voice:          (206) 236-1676  Mercer Island, WA 98040-0820
Fax:            (206) 232-9186  Skype: jwccsllc (206) 855-5792

bad economics will sink any economy no matter how much they believe this
time things are different. They aren't. -- Arthur Laffer