Hi,
I want to be certain that my apache and varnish logfiles are in strict date order when rotated. I'd like to run a sort command against them before they're compressed.
I've had a look at the logrotate man page, and it looks like I can use a postrotate/endscript to do this. However, I can see any reference in the documentation for how to operate on the file. All the examples seem to take the form of restarting something, or running some other standalone script. What I want to do is run a sort command against the newly rotated file - but how do I know what it is called?
Any ideas?
S.
On Monday 09 November 2009 10:00:32 am Stephen Nelson-Smith wrote:
I want to be certain that my apache and varnish logfiles are in strict date order when rotated. I'd like to run a sort command against them before they're compressed.
I use the dateext option in my logrotate configuration file so that rotated files have the date appended to the filename. I also compress them so they end up like:
whatever-site-access.log.20090930.gz whatever-site-access.log.20091031.gz whatever-site-error.log.20090930.gz whatever-site-error.log.20091031.gz
Well...these are sorted (within the same type of file: access... error). If you want them strictly sorted by date you'll need to investigate. I'm not sure if logrotate provides any facility in order to manipulate the "current" file being rotated so maybe you'll have to do this via a shells cript & cron (after logs are rotated).
HTH, Jorge
2009/11/9 Jorge Fábregas jorge.fabregas@gmail.com:
On Monday 09 November 2009 10:00:32 am Stephen Nelson-Smith wrote:
I want to be certain that my apache and varnish logfiles are in strict date order when rotated. I'd like to run a sort command against them before they're compressed.
I use the dateext option in my logrotate configuration file so that rotated files have the date appended to the filename. I also compress them so they end up like:
whatever-site-access.log.20090930.gz
...
Yes, I do that too.
If you want them strictly sorted by date you'll need to investigate. I'm not sure if logrotate provides any facility in order to manipulate the "current" file being rotated so maybe you'll have to do this via a shells cript & cron (after logs are rotated).
That would be a shame... it would work, but it would suck.
S.
Stephen Nelson-Smith wrote on Mon, 9 Nov 2009 14:37:18 +0000:
Yes, I do that too.
Then, how are these not in date order? It might help you'd explain the purpose.
Kai
2009/11/9 Kai Schaetzl maillists@conactive.com:
Stephen Nelson-Smith wrote on Mon, 9 Nov 2009 14:37:18 +0000:
Yes, I do that too.
Then, how are these not in date order? It might help you'd explain the purpose.
Because within apache, you can't necessarily guarantees strict chronological order of log entries. If, for example, php takes a couple of seconds to run, the request could come a couple of seconds before the log entry is written
S.
Stephen Nelson-Smith wrote on Mon, 9 Nov 2009 16:45:35 +0000:
Because within apache, you can't necessarily guarantees strict chronological order of log entries. If, for example, php takes a couple of seconds to run, the request could come a couple of seconds before the log entry is written
Is it possible that everyone misunderstood your question? You want to sort the *content* of the log files, so the content is in chronological order? I do not see much value in this. Commonly used log reporting tools are able to cope with limited "displacements" in time. I don't know anybody who cares about that. But you may try mergelog for this purpose. http://mergelog.sourceforge.net/
Kai
2009/11/9 Kai Schaetzl maillists@conactive.com:
Stephen Nelson-Smith wrote on Mon, 9 Nov 2009 16:45:35 +0000:
Because within apache, you can't necessarily guarantees strict chronological order of log entries. If, for example, php takes a couple of seconds to run, the request could come a couple of seconds before the log entry is written
Is it possible that everyone misunderstood your question? You want to sort the *content* of the log files, so the content is in chronological order?
Precisely so.
I do not see much value in this.
My client requires it, so there is value for him :)
But you may try mergelog for this purpose. http://mergelog.sourceforge.net/
I'll check it out.
S.
Stephen Nelson-Smith wrote:
2009/11/9 Kai Schaetzl maillists@conactive.com:
Stephen Nelson-Smith wrote on Mon, 9 Nov 2009 16:45:35 +0000:
Because within apache, you can't necessarily guarantees strict chronological order of log entries. If, for example, php takes a couple of seconds to run, the request could come a couple of seconds before the log entry is written
Is it possible that everyone misunderstood your question? You want to sort the *content* of the log files, so the content is in chronological order?
Precisely so.
I do not see much value in this.
My client requires it, so there is value for him :)
But you may try mergelog for this purpose. http://mergelog.sourceforge.net/
I'll check it out.
There is also a perl script called logresolvemerge.pl that is part of the awstats program that may work - although I'm not sure if either of these will adjust out-of-order source files. They are intended to merge already sorted files into one for analyzers that aren't bright enough to do it themselves. Analog complains about possible duplication but doesn't seem to choke if you just give it a big list of files.
From: Stephen Nelson-Smith stephen@atalanta-systems.com
I want to be certain that my apache and varnish logfiles are in strict date order when rotated. I'd like to run a sort command against them before they're compressed.
I've had a look at the logrotate man page, and it looks like I can use a postrotate/endscript to do this. However, I can see any reference in the documentation for how to operate on the file. All the examples seem to take the form of restarting something, or running some other standalone script. What I want to do is run a sort command against the newly rotated file - but how do I know what it is called?
I am not sure you can retrieve and cycle through the filenames that were being rotated... I would create sections for each logfile to rotate (so you know the filename)... which might be tedious if you have many log files. A script to auto-generate the logrotate conf file in /etc/logrotate.d/ will help...
JD
Stephen Nelson-Smith wrote:
I've had a look at the logrotate man page, and it looks like I can use a postrotate/endscript to do this. However, I can see any reference in the documentation for how to operate on the file. All the examples seem to take the form of restarting something, or running some other standalone script. What I want to do is run a sort command against the newly rotated file - but how do I know what it is called?
How about this as an example -
"/home/user/var/log/httpd/*www*log" { daily rotate 1 nocompress notifempty copytruncate missingok sharedscripts
olddir /usr/local/log/httpd/archives
postrotate DATE=`date --date=Yesterday +%y%m%d` cd /usr/local/log/httpd/archives for FOO in `ls *.1` do mv $FOO `echo $FOO | cut -f1 -d.`.$DATE.log done gzip -9 *.$DATE.log sleep 60 sync logger "[LOGROTATE] Rotated these logs: `echo *.${DATE}.log.gz`" endscript
}
nate
-
John Doe -
Jorge Fábregas -
Kai Schaetzl -
Les Mikesell -
nate -
Stephen Nelson-Smith