On Fri, Dec 7, 2012 at 11:09 AM, Gelen James <hahaha_30k at yahoo.com> wrote: > Hi all, > > Has any one used any log archiving tools? Please suggest one that's your favorite. > > Presently I have a project to archive raw log files on hundreds of Centos boxes to a central server and stored there for two years. The basic requirements of the project are: > > 1, The log files will be sent from hundreds of Centos clients with a client side agent or similar. > 2, better there is a client side configuration file which specify log files to archive > 3, a server side configuration file specified where to logs from host ( or host groups), and the directory structure to store log files > 4, better there is a way to remove duplication of log files, i.e. md5 checksum, time stamp, etc. > > Open source or commercial are both fine. And there is no need to do the indexing or other further processing, just raw files is fine. I've done this for specific applications with some simple shell scripts using rsync over ssh, but it has to mesh with the way the log files roll over and are renamed. I don't think there is a generic tool. If you just want archived daily snapshot copies, you could use backuppc to back up /var/log on the targets. It will compress files and de-dup exact file content matches with hard links to a pooled instance even where the file has been renamed (like logrotate does). That approach doesn't get you a handy way to analyze anything over a long time span or look at more than one file as a stream - but you could have it working in a few hours and it will be very efficient with disk usage. -- Les Mikesell lesmikesell at gmail.com