On Feb 1, 2012, at 2:54 PM, Tom H tom@limepepper.co.uk wrote:

Hi CentOS experts,*

Short Version*

I would like to produce a weekly report in HTML for each CentOS 5.x server we have indicating configuration compliance with some industry benchmark. I am looking for a tool or tools to implement this, I am happy to use 3rd party proprietary stuff if necessary.

You could have a weekly cron job on all boxes that does a rpm for all package config files, diff against a "snapshot" copy contained under /var somewhere, email those diffs to a change management system,then save the current files in the snapshot directory.

First run will send the complete configs, all subsequent runs will send the diffs.

Of course you need a change management system that will hold an inventory of systems, those systems' hardware/software inventories and configurations, and track those changes with alerts and reports and such.

I don't know of a good system for doing all that unfortunately, but if you do find one let me know.

-Ross

On Wed, Feb 1, 2012 at 2:54 PM, Tom H tom@limepepper.co.uk wrote:

Hi CentOS experts,*

Short Version*

I would like to produce a weekly report in HTML for each CentOS 5.x server we have indicating configuration compliance with some industry benchmark. I am looking for a tool or tools to implement this, I am happy to use 3rd party proprietary stuff if necessary.

[snip]

I'm in a similar situation. We have a growing infrastructure of over 300 instances of RHEL4/5/6. Though not specifically CentOS the tools are the same. My focus has been on PCI compliance. As of yet we don't have any SOX systems on Linux, but I expect that will change in the near future.

For PCI compliance there are a few things that we do. The first thing was to get a handle on the buildout process which we did via kickstart. This ensured consistency in the builds which previously was done by different engineers/operators with different skill levels. We validated the standard image and then used Satellite/Spacewalk to keep track of the versions.

The next step was the daily bit rot and the damage from the application folks whose sole experience was on desktop or laptop systems (i.e., they never had to comply with any industry standards). We started by separating OS from application. This meant not only separate volume groups and mount points for application files, but also things like ensuring that apps did not run as root (you'd be amazed how many developers insist that builds must occur as root). In just about every case where we allowed application developers to have root access we ended up with systems that were wildly out of compliance. In one case a developer installed an entire desktop suite, including MP3 player and video editing tools, in order to satisfy a dependency on a single widget library. We don't do that any more. :/

Next was auditing, which I think may apply to your question.

For the basic package setup, Spacewalk or Satellite can track the versions and allow you to lock the package set. There are also existing scripts that wrap variations of an 'rpm -qVa' and send the reports back. Tools such as tripwire are also useful for this. If you have deployed SELinux, you can effectively even lock the root user from installing or modifying system packages.

For the configurations, we are experimenting with cfengine and puppet. They allow you to track configuration changes, reset changes, etc.. I've also used CVS to track configuration files directly. I.e., checkin the changes onto a logged administration server then have the production servers checkout the changes on an on-demand or scheduled basis. This minimizes on-the-fly configurations that accumulate and take the server out of compliance. There are tools to generate reports from cfengine/puppet that show which configurations have changed, etc..

We are also using the perl test harness to run validations. It's pretty coding intensive so you'd possibly need a Perl developer initially to create and to maintain the scripts. The idea is to create the test scripts in lock step with changes to the kickstart. The harness generates a PASS or FAIL response depending on the Perl test. For example, for PCI compliance we have a standard login banner. The test does an MD5 sum against the target machine's /etc/issue.net and checks it against the stored hash. If the hashes correspond it passes the test (barring hash collisions of course :D ).

We are still looking at other methods.