[CentOS] How to tell if I've been hacked?

Rob Townley rob.townley at gmail.com
Sun Aug 23 17:58:39 UTC 2009

On Sat, Aug 22, 2009 at 6:07 PM, Bill Campbell<centos at celestial.com> wrote:
> On Sat, Aug 22, 2009, Dave wrote:
>>On Sat, Aug 22, 2009 at 6:49 AM, Bill Campbell<centos at celestial.com> wrote:
>>> I review daily reports from over 50 systems every morning, checking changes
>>> found, usually taking no more than 10 minutes a day.  The key is to keep
>>> the reports simple, and to make updating easy (and to have procedures that
>>> monitor systems to be sure they's still alive and reporting in).
>>So how do you track the inevitable changes? Not saying you can't, just
>>curious. For me, when I look at a batch of changes, some of them are
>>obviously stuff I've done, other stuff not so obvious. I also filter
>>reports through a script that sort of does a diff and makes an attempt
>>to limit the boilerplate. Sometimes it is a bit too terse.
> First off, we don't allow automatic updates on most systems, much
> preferring to do them manually making it pretty easy to refresh
> the comparison database immediately after the update is complete.
> The odds that a cracker will get in and do their dirty deeds
> while this are going on are pretty low, and can probably be
> ignored.
> We handle pretty much all server stuff under the OpenPKG portable
> package management system so things like spamassassin, amavisd,
> clamav, and postfix are not the distribution versions, but those
> from OpenPKG (which are generally updated more quickly then the
> distribution's).  A typical occurrence will be that we get an
> e-mail saying that clamav is out of date from the nightly
> freshclam update, I will pick up the new sources, update the
> OpenPKG SRPM for it, and deploy it 40 or so systems running it,
> and expect to see a corresponding set of notices the next morning
> that files under clamav have changed.
> The clusterssh program makes this sort of thing much more efficient
> as one can execute shell commands on multiple systems simultaneously.
>>> We create a file system initially, the same size as ``/'', and make a copy
>>> of ``/'' in it identical except for the /etc/fstab entry.  This is not
>>> mounted in normal operations, but the system can be booted from it to get
>>> to a clean system.
>>Wow, elaborate. How do you protect this file system from intruders?
>>Exterrnal and powerred off?
> That's one way to do it.  We also run a fair number of Linux
> servers under VMware so periodic snapshots and backups simplify
> the task.
> I have not seen many successful cracks of Linux boxes that we
> have configured from scratch.  Some basic things can be done to
> minimize the chances of cracks.
>   + Create the baseline for intrusion detection tools before putting the
>     syste on line, and monitor it daily.
>   + Configure openssh to refuse password authentication requiring
>     authorized_keys access.
>   + Configure openssh with tcp_wrappers support, restricting access by IP
>     address and/or domain names.  I consider this absolutely mandatory if
>     one needs to all username and password authentication.
>   + Use fail2ban or similar techniques to quickly block IP addresses that
>     are found probing the system (don't forget to look at POP and IMAP
>     logs for failed login attempts).
>   + Use /bin/false as the standard shell for accounts that don't have good
>     reason for shell access.  This does not affect e-mail or most services
>     that a typical ISP customer needs.
>   + Use OpenVPN for access.  This works well even when in hotels with NAT
>     firewalls, and is not easily hacked anonymously.
>   + Restrict access of webmin and usermin to local networks so they are
>     not vulnerable to outside attack.  These services are available to
>     people outside connecting with OpenVPN.

Cross Site Attacks (CSRF, XSS) make webmin very vulnerable in this
scenario.  It is a bad idea to use a single browser.  If in Firefox,
you already logged in to webmin and browse to a malicious site (many
reputable sites unknowingly have malicious javascript -- see
HoneyNet), the malicious site could do nasty things via webmin or any
other internal webserver.  Yes, NoScript may help, but NoScript has to
be updated daily and Firefox restarted.

The best practice is to Install three separate browser application
such as Epiphany or Dillo  and only use this for internal websites.
Use Firefox for email.  Use Chrome for everything else.  The idea is
to have completely separate processes using completely separate memory
and harddrive locations.

I don't think there are many malicious variants of InvisibleThings's
BluePill or BlueChicken, but if a malicious variant can elevate itself
to become the Hypervisor, then all of your virtual machines could be
monitored by a HyperKit -- rootkit in the hypervisor.  Again, i don't
know if there are many malicious in-the-wild versions of bluepill, but
if just one malicious vmware image is uploaded to the Amazon EC2, then
every other VM on that same hardware at Amazon can be controlled by a
hyperkit.  InvisibleThings are professional security researchers in
Poland, so the original bluepill isn't malicious.   BlueChicken is to
evade detection by moving bluepill back and forth between the
hypervisor and a vm at will.  Others are going to know more about what
actually is in-the-wild versus in the lab.

>   + Restrict webmail, pop, and imap access to secure connections using
>     https, tls, ssl.  We have never been able to get the average ISP
>     customer to use good passwords, but every little bit helps.
> Bill
> --
> INTERNET:   bill at celestial.com  Bill Campbell; Celestial Software LLC
> URL: http://www.celestial.com/  PO Box 820; 6641 E. Mercer Way
> Voice:          (206) 236-1676  Mercer Island, WA 98040-0820
> Fax:            (206) 232-9186  Skype: jwccsllc (206) 855-5792
> bad economics will sink any economy no matter how much they believe this
> time things are different. They aren't. -- Arthur Laffer
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

More information about the CentOS mailing list