[CentOS] Remote-logging nginx? (or other non-syslog-enabled stuff)

Fri Mar 25 17:19:08 UTC 2011
Ilyas -- <umask00 at gmail.com>

Hi!

Also note that:
1. logrotate wouldn't rotate fifo/pipes if options `notifempty'
enabled in logrotate profiles.
2. enable buffering in syslog-ng.conf (next - whole list of options in
my config):
options {
        sync (128);
        time_reopen (10);
        log_fifo_size (16384);
        chain_hostnames (yes);
        use_dns (no);
        use_fqdn (yes);
        create_dirs (yes);
        keep_hostname (yes);
        dir_perm(0755);
        perm(0644);
        dir_owner(root);
        dir_group(root);
        owner(root);
        group(root);
        log_msg_size(16384);
};
3. Don't worry about blocking output in some services. If syslog-ng
listen fifo locally (in the same server/vps where working daemon which
logs we want serve) any output will be buffered (with few limits of
free version of syslog-ng) in syslog-ng. Main idea here is that other
side which listen fifo - locally runned syslog-ng.
4. I used and using opensource version of syslog-ng and no have
problems with load. Syslog-ng is very perfect tool for loads.

On Fri, Mar 25, 2011 at 10:00 AM, Dr. Ed Morbius <dredmorbius at gmail.com> wrote:
> First:  thanks very much for spelling this out, Ilyas.  This was along
> the lines of what I'd been considering.  You addressed a number of
> concerns I had (e.g.: non-blocking output) which is really helpful.
>
> on 08:39 Fri 25 Mar, Ilyas -- (umask00 at gmail.com) wrote:
>> Hi!
>>
>> I'm using follow method for remote logging and catch logs from many servers.
>> Nginx writes logs into fifo, which created via nginx init script:
>>
>> cat /etc/sysconfig/nginx
>> ...
>> # syslog-ng support for nginx
>> if [ ! -p /var/log/nginx/access.log ]; then
>>         /bin/rm -f /var/log/nginx/access.log
>>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
>> fi
>> if [ ! -p /var/log/nginx/error.log ] ; then
>>         /bin/rm -f /var/log/nginx/error.log
>>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
>> fi
>> /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log
>>
>> Nginx just writes to fifo as to file. Nginx has nonblocking output to
>> logs and if nobody read fifo nginx dont stop on logs write.
>
> Bingo.
>
>> From other side pipe reads syslog-ng.
>> cat /etc/syslog-ng/syslog-ng.conf
>> ...
>> source s_nginx_20 {
>>         fifo ("/var/log/nginx/access.log" log_prefix("nginx-access-log: "));
>> };
>>
>> source s_nginx_21 {
>>         fifo ("/var/log/nginx/error.log" log_prefix("nginx-error-log: "));
>> };
>> ...
>> destination d_remote { tcp("remote.example.com", port(514)); };
>> ...
>> # nginx
>> filter f_nginx_20 { match("nginx-access-log: "); };
>> filter f_nginx_21 { match("nginx-error-log: "); };
>> ...
>> # nginx
>> log { source(s_nginx_20); filter(f_nginx_20); destination(d_remote); };
>> log { source(s_nginx_21); filter(f_nginx_21); destination(d_remote); };
>>
>
> Nice.
>
>> To avoid syslog-ng problems on startup (ex. if  fifo does not exists)
>> used follow solution:
>> cat /etc/sysconfig/syslog-ng
>> ...
>> # syslog-ng support for nginx
>> if [ ! -p /var/log/nginx/access.log ]; then
>>         /bin/rm -f /var/log/nginx/access.log
>>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/access.log
>> fi
>> if [ ! -p /var/log/nginx/error.log ] ; then
>>         /bin/rm -f /var/log/nginx/error.log
>>         /usr/bin/mkfifo --mode=0640 /var/log/nginx/error.log
>> fi
>> /bin/chown nginx:root /var/log/nginx/access.log /var/log/nginx/error.log
>>
>>
>>
>> On remote side (remote.example.com):
>> cat /etc/syslog-ng/syslog-ng.conf
>> ...
>> source s_net {
>>         udp(ip(0.0.0.0) port(514));
>>         tcp(ip(0.0.0.0) port(514) keep-alive(yes) max-connections(128));
>> };
>> ...
>> filter f_nginx_20 { match("nginx-access-log: "); };
>> filter f_nginx_21 { match("nginx-error-log: "); };
>> ...
>> destination d_nginx_20 { file("/var/log/nginx/access.log"); };
>> destination d_nginx_21 { file("/var/log/nginx/error.log"); };
>> ...
>> log { source(s_sys); filter(f_nginx_20); destination(d_nginx_20); };
>> log { source(s_sys); filter(f_nginx_21); destination(d_nginx_21); };
>>
>>
>>
>> In the same way I catch logs from 20-30 servers to 1 server, approx.
>> 300GB gzipped logs per day.
>
> Great.  That also answers the scaling question.  We're comfortably under
> that scale for now.
>
> Very, very helpful post, thanks again.
>
>> On Thu, Mar 24, 2011 at 11:23 PM, Dr. Ed Morbius <dredmorbius at gmail.com> wrote:
>> > I'm looking for suggestions as to a good general method of
>> > remote-logging services such as nginx or anything else which doesn't
>> > support syslog natively.
>> >
>> > I'm aware that there's an nginx patch, and we're evaluating this.  It
>> > may be the way we fly.
>> >
>> > However there are other tools which may not have a patch for which
>> > remote logging would be useful.  If there's a general soution (something
>> > as naive as tailing local logs and firing these off on a regular basis).
>> >
>> > I've heard rumors of a Perl script used for apache logs.
>> >
>> > Also that rsyslog supports logging from local files to a remote syslog
>> > server, possibly.  I'm RTFMing on that.
>> >
>> > Thanks in advance.
>> >
>> > --
>> > Dr. Ed Morbius, Chief Scientist /            |
>> >  Robot Wrangler / Staff Psychologist        | When you seek unlimited power
>> > Krell Power Systems Unlimited                |                  Go to Krell!
>> > _______________________________________________
>> > CentOS mailing list
>> > CentOS at centos.org
>> > http://lists.centos.org/mailman/listinfo/centos
>> >
>>
>>
>>
>> --
>> Ilyas R. Khasyanov
>> Unix/Linux System Administrator
>> GPG Key ID: 6EC5EB27 (Changed since 2009-05-12)
>
> --
> Dr. Ed Morbius, Chief Scientist /            |
>  Robot Wrangler / Staff Psychologist        | When you seek unlimited power
> Krell Power Systems Unlimited                |                  Go to Krell!
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
Ilyas R. Khasyanov
Unix/Linux System Administrator
GPG Key ID: 6EC5EB27 (Changed since 2009-05-12)