[CentOS] Remote-logging nginx? (or other non-syslog-enabled stuff)

Fri Mar 25 07:00:31 UTC 2011
Dr. Ed Morbius <dredmorbius at gmail.com>

First:  thanks very much for spelling this out, Ilyas.  This was along
the lines of what I'd been considering.  You addressed a number of
concerns I had (e.g.: non-blocking output) which is really helpful.

on 08:39 Fri 25 Mar, Ilyas -- (umask00 at gmail.com) wrote:
> Hi!
> 
> I'm using follow method for remote logging and catch logs from many servers.
> Nginx writes logs into fifo, which created via nginx init script:
> 
> cat /etc/sysconfig/nginx
> ...
> # syslog-ng support for nginx
> if [ ! -p /var/log/nginx/access.log ]; then
>         /bin/rm -f /var/log/nginx/access.log
>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
> fi
> if [ ! -p /var/log/nginx/error.log ] ; then
>         /bin/rm -f /var/log/nginx/error.log
>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
> fi
> /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log
> 
> Nginx just writes to fifo as to file. Nginx has nonblocking output to
> logs and if nobody read fifo nginx dont stop on logs write.

Bingo.
 
> From other side pipe reads syslog-ng.
> cat /etc/syslog-ng/syslog-ng.conf
> ...
> source s_nginx_20 {
>         fifo ("/var/log/nginx/access.log" log_prefix("nginx-access-log: "));
> };
> 
> source s_nginx_21 {
>         fifo ("/var/log/nginx/error.log" log_prefix("nginx-error-log: "));
> };
> ...
> destination d_remote { tcp("remote.example.com", port(514)); };
> ...
> # nginx
> filter f_nginx_20 { match("nginx-access-log: "); };
> filter f_nginx_21 { match("nginx-error-log: "); };
> ...
> # nginx
> log { source(s_nginx_20); filter(f_nginx_20); destination(d_remote); };
> log { source(s_nginx_21); filter(f_nginx_21); destination(d_remote); };
> 

Nice.
 
> To avoid syslog-ng problems on startup (ex. if  fifo does not exists)
> used follow solution:
> cat /etc/sysconfig/syslog-ng
> ...
> # syslog-ng support for nginx
> if [ ! -p /var/log/nginx/access.log ]; then
>         /bin/rm -f /var/log/nginx/access.log
>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
> fi
> if [ ! -p /var/log/nginx/error.log ] ; then
>         /bin/rm -f /var/log/nginx/error.log
>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
> fi
> /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log
> 
> 
> 
> On remote side (remote.example.com):
> cat /etc/syslog-ng/syslog-ng.conf
> ...
> source s_net {
>         udp(ip(0.0.0.0) port(514));
>         tcp(ip(0.0.0.0) port(514) keep-alive(yes) max-connections(128));
> };
> ...
> filter f_nginx_20 { match("nginx-access-log: "); };
> filter f_nginx_21 { match("nginx-error-log: "); };
> ...
> destination d_nginx_20 { file("/var/log/nginx/access.log"); };
> destination d_nginx_21 { file("/var/log/nginx/error.log"); };
> ...
> log { source(s_sys); filter(f_nginx_20); destination(d_nginx_20); };
> log { source(s_sys); filter(f_nginx_21); destination(d_nginx_21); };
> 
> 
> 
> In the same way I catch logs from 20-30 servers to 1 server, approx.
> 300GB gzipped logs per day.

Great.  That also answers the scaling question.  We're comfortably under
that scale for now.

Very, very helpful post, thanks again.
 
> On Thu, Mar 24, 2011 at 11:23 PM, Dr. Ed Morbius <dredmorbius at gmail.com> wrote:
> > I'm looking for suggestions as to a good general method of
> > remote-logging services such as nginx or anything else which doesn't
> > support syslog natively.
> >
> > I'm aware that there's an nginx patch, and we're evaluating this.  It
> > may be the way we fly.
> >
> > However there are other tools which may not have a patch for which
> > remote logging would be useful.  If there's a general soution (something
> > as naive as tailing local logs and firing these off on a regular basis).
> >
> > I've heard rumors of a Perl script used for apache logs.
> >
> > Also that rsyslog supports logging from local files to a remote syslog
> > server, possibly.  I'm RTFMing on that.
> >
> > Thanks in advance.
> >
> > --
> > Dr. Ed Morbius, Chief Scientist /            |
> >  Robot Wrangler / Staff Psychologist        | When you seek unlimited power
> > Krell Power Systems Unlimited                |                  Go to Krell!
> > _______________________________________________
> > CentOS mailing list
> > CentOS at centos.org
> > http://lists.centos.org/mailman/listinfo/centos
> >
> 
> 
> 
> -- 
> Ilyas R. Khasyanov
> Unix/Linux System Administrator
> GPG Key ID: 6EC5EB27 (Changed since 2009-05-12)

-- 
Dr. Ed Morbius, Chief Scientist /            |
  Robot Wrangler / Staff Psychologist        | When you seek unlimited power
Krell Power Systems Unlimited                |                  Go to Krell!