[CentOS] Remote-logging nginx? (or other non-syslog-enabled stuff)

Thu Mar 24 22:52:24 UTC 2011
Dr. Ed Morbius <dredmorbius at gmail.com>

on 17:50 Thu 24 Mar, Lamar Owen (lowen at pari.edu) wrote:
> On Thursday, March 24, 2011 05:37:41 pm Dr. Ed Morbius wrote:
> > on 17:14 Thu 24 Mar, Lamar Owen (lowen at pari.edu) wrote:
> > > Prior to PostgreSQL supporting syslog I used [logger] to
> > > pipe PostgreSQL output to syslog.  Worked fine.
> 
> > I haven't, looking at it.
> 
> It is one option that is definitely in vanilla CentOS.

Quite.
 
> > OK.  Any pointers on configuration are greatly appreciated.  Docs, etc.
> 
> Whew.  Large scale remote syslog operation is a large subject; I've
> never had anything large-enough scale to need more than logwatch or
> site-grown scripts to do processing.  The biggest thing to do is set
> up NTP and have three reference time sources (three so that if one is
> wrong you know which one).  Otherwise, log correlation is impossible.

It is.  There've been a few advances in sysadmin practice since the
Nemeth books were first produced, and while there are some titles
dealing with portions of this, codifying practices in docs would be a
wonderful thing.  I've considered (and been approached regarding)
tackling at least parts of this myself.

Useful logging is definitely part of this.
  
> > Yeah, we're aware of that (I mentioned this in my first post to the
> > thread).
> 
> Yep, that you did.
> 
> > We've got a locally-compiled version of nginx, so patching isn't out of
> > the question.  Just looking at all our options.
> 
> While CentOS doesn't provide nginx itself, it does provide tools for
> dealing with logs; I saw several things doing a 'yum list | grep log'
> (I know there's easier ways of doing that; that's just the way I
> prefer to go about it).  Also try grepping a yum list for 'watch' as I
> remember some logwatching stuff.....

Right, and the general solution also generalizes to other tools.
Postgresql (which we aren't using currently) also has its own log
handler (a small frustration of mine with the database).


And I turned up the rsyslogd feature:

    http://www.rsyslog.com/doc/imfile.html                                          
    Text File Input Module

    Module Name:    imfile

    Author: Rainer Gerhards <rgerhards at adiscon.com>

    Description:

    Provides the ability to convert any standard text file into a syslog
    message. A standard text file is a file consisting of printable
    characters with lines being delimited by LF.

    The file is read line-by-line and any line read is passed to
    rsyslog's rule engine. The rule engine applies filter conditons and
    selects which actions needs to be carried out.

    As new lines are written they are taken from the file and processed.
    Please note that this happens based on a polling interval and not
    immediately. The file monitor support file rotation. To fully work,
    rsyslogd must run while the file is rotated. Then, any remaining
    lines from the old file are read and processed and when done with
    that, the new file is being processed from the beginning. If
    rsyslogd is stopped during rotation, the new file is read, but any
    not-yet-reported lines from the previous file can no longer be
    obtained.

    When rsyslogd is stopped while monitoring a text file, it records
    the last processed location and continues to work from there upon
    restart. So no data is lost during a restart (except, as noted
    above, if the file is rotated just in this very moment).

    Currently, the file must have a fixed name and location (directory).
    It is planned to add support for dynamically generating file names
    in the future.

    Multiple files may be monitored by specifying $InputRunFileMonitor
    multiple times. 

-- 
Dr. Ed Morbius, Chief Scientist /            |
  Robot Wrangler / Staff Psychologist        | When you seek unlimited power
Krell Power Systems Unlimited                |                  Go to Krell!