On Fri, Dec 7, 2012 at 11:09 AM, Gelen James hahaha_30k@yahoo.com wrote:

Hi all,

Has any one used any log archiving tools? Please suggest one that's your favorite.

Presently I have a project to archive raw log files on hundreds of Centos boxes to a central server and stored there for two years. The basic requirements of the project are:

1, The log files will be sent from hundreds of Centos clients with a client side agent or similar. 2, better there is a client side configuration file which specify log files to archive 3, a server side configuration file specified where to logs from host ( or host groups), and the directory structure to store log files 4, better there is a way to remove duplication of log files, i.e. md5 checksum, time stamp, etc.

Open source or commercial are both fine. And there is no need to do the indexing or other further processing, just raw files is fine.

I've done this for specific applications with some simple shell scripts using rsync over ssh, but it has to mesh with the way the log files roll over and are renamed. I don't think there is a generic tool. If you just want archived daily snapshot copies, you could use backuppc to back up /var/log on the targets. It will compress files and de-dup exact file content matches with hard links to a pooled instance even where the file has been renamed (like logrotate does). That approach doesn't get you a handy way to analyze anything over a long time span or look at more than one file as a stream - but you could have it working in a few hours and it will be very efficient with disk usage.

On Sat, Dec 8, 2012 at 6:37 AM, Leon Fauster leonfauster@googlemail.comwrote:

Am 07.12.2012 um 18:09 schrieb Gelen James:

...

Has any one used any log archiving tools? Please suggest one that's

your favorite.

...

Presently I have a project to archive raw log files on hundreds of

Centos boxes to a central server and stored there for two years. The basic requirements of the project are:

...

1, The log files will be sent from hundreds of Centos clients with a

client side agent or similar.

...

2, better there is a client side configuration file which specify log

files to archive

...

3, a server side configuration file specified where to logs from host (

or host groups), and the directory structure to store log files

...

4, better there is a way to remove duplication of log files, i.e. md5

checksum, time stamp, etc.

...

Open source or commercial are both fine. And there is no need to do the

indexing or other further processing, just raw files is fine.

what about syslog's capability to send all logs to a central syslog server (check /etc/sysconfig/syslog for the server part and /etc/syslog.conf for the client part)?

+1 for a centralized syslog server ... it's the ideal tool for the job

rsyslog is part of the CentOS Base for EL6 syslog-ng is packaged in EPEL (if it's preferred over sysklogd or rsyslog)

applications/scripts can log also via "logger" - like echo test | /usr/bin/logger -i -t "appname" -p user.notice

the archive part can be done on the syslog server centrally (e.g. use rsnapshot to provide daily, weekly, monthly cascades of /var/log)

-- LF

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos