First: thanks very much for spelling this out, Ilyas. This was along the lines of what I'd been considering. You addressed a number of concerns I had (e.g.: non-blocking output) which is really helpful. on 08:39 Fri 25 Mar, Ilyas -- (umask00 at gmail.com) wrote: > Hi! > > I'm using follow method for remote logging and catch logs from many servers. > Nginx writes logs into fifo, which created via nginx init script: > > cat /etc/sysconfig/nginx > ... > # syslog-ng support for nginx > if [ ! -p /var/log/nginx/access.log ]; then > /bin/rm -f /var/log/nginx/access.log > /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log > fi > if [ ! -p /var/log/nginx/error.log ] ; then > /bin/rm -f /var/log/nginx/error.log > /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log > fi > /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log > > Nginx just writes to fifo as to file. Nginx has nonblocking output to > logs and if nobody read fifo nginx dont stop on logs write. Bingo. > From other side pipe reads syslog-ng. > cat /etc/syslog-ng/syslog-ng.conf > ... > source s_nginx_20 { > fifo ("/var/log/nginx/access.log" log_prefix("nginx-access-log: ")); > }; > > source s_nginx_21 { > fifo ("/var/log/nginx/error.log" log_prefix("nginx-error-log: ")); > }; > ... > destination d_remote { tcp("remote.example.com", port(514)); }; > ... > # nginx > filter f_nginx_20 { match("nginx-access-log: "); }; > filter f_nginx_21 { match("nginx-error-log: "); }; > ... > # nginx > log { source(s_nginx_20); filter(f_nginx_20); destination(d_remote); }; > log { source(s_nginx_21); filter(f_nginx_21); destination(d_remote); }; > Nice. > To avoid syslog-ng problems on startup (ex. if fifo does not exists) > used follow solution: > cat /etc/sysconfig/syslog-ng > ... > # syslog-ng support for nginx > if [ ! -p /var/log/nginx/access.log ]; then > /bin/rm -f /var/log/nginx/access.log > /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log > fi > if [ ! -p /var/log/nginx/error.log ] ; then > /bin/rm -f /var/log/nginx/error.log > /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log > fi > /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log > > > > On remote side (remote.example.com): > cat /etc/syslog-ng/syslog-ng.conf > ... > source s_net { > udp(ip(0.0.0.0) port(514)); > tcp(ip(0.0.0.0) port(514) keep-alive(yes) max-connections(128)); > }; > ... > filter f_nginx_20 { match("nginx-access-log: "); }; > filter f_nginx_21 { match("nginx-error-log: "); }; > ... > destination d_nginx_20 { file("/var/log/nginx/access.log"); }; > destination d_nginx_21 { file("/var/log/nginx/error.log"); }; > ... > log { source(s_sys); filter(f_nginx_20); destination(d_nginx_20); }; > log { source(s_sys); filter(f_nginx_21); destination(d_nginx_21); }; > > > > In the same way I catch logs from 20-30 servers to 1 server, approx. > 300GB gzipped logs per day. Great. That also answers the scaling question. We're comfortably under that scale for now. Very, very helpful post, thanks again. > On Thu, Mar 24, 2011 at 11:23 PM, Dr. Ed Morbius <dredmorbius at gmail.com> wrote: > > I'm looking for suggestions as to a good general method of > > remote-logging services such as nginx or anything else which doesn't > > support syslog natively. > > > > I'm aware that there's an nginx patch, and we're evaluating this. It > > may be the way we fly. > > > > However there are other tools which may not have a patch for which > > remote logging would be useful. If there's a general soution (something > > as naive as tailing local logs and firing these off on a regular basis). > > > > I've heard rumors of a Perl script used for apache logs. > > > > Also that rsyslog supports logging from local files to a remote syslog > > server, possibly. I'm RTFMing on that. > > > > Thanks in advance. > > > > -- > > Dr. Ed Morbius, Chief Scientist / | > > Robot Wrangler / Staff Psychologist | When you seek unlimited power > > Krell Power Systems Unlimited | Go to Krell! > > _______________________________________________ > > CentOS mailing list > > CentOS at centos.org > > http://lists.centos.org/mailman/listinfo/centos > > > > > > -- > Ilyas R. Khasyanov > Unix/Linux System Administrator > GPG Key ID: 6EC5EB27 (Changed since 2009-05-12) -- Dr. Ed Morbius, Chief Scientist / | Robot Wrangler / Staff Psychologist | When you seek unlimited power Krell Power Systems Unlimited | Go to Krell!