[CentOS] Configuration Compliance auditing for many CentOS 5.x boxes

Thu Feb 2 00:04:14 UTC 2012
Kwan Lowe <kwan.lowe at gmail.com>

On Wed, Feb 1, 2012 at 2:54 PM, Tom H <tom at limepepper.co.uk> wrote:

> Hi CentOS experts,*
> Short Version*
> I would like to produce a weekly report in HTML for each CentOS 5.x
> server we have indicating configuration compliance with some industry
> benchmark. I am looking for a tool or tools to implement this, I am
> happy to use 3rd party proprietary stuff if necessary.

> [snip]

I'm in a similar situation. We have a growing infrastructure of over 300
instances of RHEL4/5/6.  Though not specifically CentOS the tools are the
same. My focus has been on PCI compliance. As of yet we don't have any SOX
systems on Linux, but I expect that will change in the near future.

For PCI compliance there are a few things that we do. The first thing was
to get a handle on the buildout process which we did via kickstart. This
ensured consistency in the builds which previously was done by different
engineers/operators with different skill levels. We validated the standard
image and then used Satellite/Spacewalk to keep track of the versions.

The next step was the daily bit rot and the damage from the application
folks whose sole experience was on desktop or laptop systems (i.e., they
never had to comply with any industry standards). We started by separating
OS from application. This meant not only separate volume groups and mount
points for application files, but also things like ensuring that apps did
not run as root (you'd be amazed how many developers insist that builds
must occur as root).  In just about every case where we allowed application
developers to have root access we ended up with systems that were wildly
out of compliance. In one case a developer installed an entire desktop
suite, including MP3 player and video editing tools, in order to satisfy a
dependency on a single widget library. We don't do that any more. :/

Next was auditing, which I think may apply to your question.

For the basic package setup, Spacewalk or Satellite can track the versions
and allow you to lock the package set. There are also existing scripts that
wrap variations of an 'rpm -qVa' and send the reports back. Tools such as
tripwire are also useful for this. If you have deployed SELinux, you can
effectively even lock the root user from installing or modifying system

For the configurations, we are experimenting with cfengine and puppet. They
allow you to track configuration changes, reset changes, etc..  I've also
used CVS to track configuration files directly.  I.e., checkin the changes
onto a logged administration server then have the production servers
checkout the changes on an on-demand or scheduled basis. This minimizes
on-the-fly configurations that accumulate and take the server out of
compliance.  There are tools to generate reports from cfengine/puppet that
show which configurations have changed, etc..

We are also using the perl test harness to run validations. It's pretty
coding intensive so you'd possibly need a Perl developer initially to
create and to maintain the scripts. The idea is to create the test scripts
in lock step with changes to the kickstart. The harness generates a PASS or
FAIL response depending on the Perl test. For example, for PCI compliance
we have a standard login banner. The test does an MD5 sum against the
target machine's /etc/issue.net and checks it against the stored hash. If
the hashes correspond it passes the test (barring hash collisions of course
:D ).

We are still looking at other methods.