From: Laurent CREPET lct@megrapet.fr

Any ideas ?

Different setup here (compiled latest squid 2.7.STABLE9). Main difference I can see with the rpm's rotatelog is that we use delaycompress...

JD

Le mer 04 mai 2011 12:48:39 CEST, Laurent CREPET a écrit:

[...] However, since:

• my settings have not been modified recently
• logrorate and squid packages have not been updated by CentOS 5.6 (see below)

...an issue with glibc ?

Squid and logrotate work just fine here : $ rpm -q squid logrotate squid-2.6.STABLE21-6.el5 logrotate-3.7.4-9.el5_5.2

$ ls -l /var/log/squid/ total 92624 -rw------- 1 squid squid 43709603 mai 4 13:08 access.log -rw------- 1 squid squid 9424047 mai 1 04:02 access.log.1.gz -rw------- 1 squid squid 9751039 avr 24 04:02 access.log.2.gz -rw------- 1 squid squid 10745924 avr 17 04:02 access.log.3.gz -rw------- 1 squid squid 11373152 avr 10 04:02 access.log.4.gz -rw------- 1 squid squid 9620745 avr 3 04:02 access.log.5.gz -rw-r----- 1 squid squid 2794 mai 4 11:58 cache.log -rw-r----- 1 squid squid 2503 mai 1 04:02 cache.log.1.gz -rw-r----- 1 squid squid 1580 avr 24 04:02 cache.log.2.gz -rw-r----- 1 squid squid 1881 avr 17 04:02 cache.log.3.gz -rw-r----- 1 squid squid 1591 avr 10 04:02 cache.log.4.gz -rw-r----- 1 squid squid 2772 avr 3 04:02 cache.log.5.gz -rw-r--r-- 1 squid squid 511 nov 26 09:13 squid.out

Have you double checked that your problem is not a permission problem, or some SELinux issue ?

You can also try, as root : $ logrotate -d -v -f /etc/logrotate.d/squid # and without -d $ squid -k rotate $ ls -lZ /var/log/squid/ to see if there is any error message.

HTH,

On Wed, 4 May 2011 13:35:37 +0200 Philippe Naudin philippe.naudin@supagro.inra.fr wrote:

Le mer 04 mai 2011 12:48:39 CEST, Laurent CREPET a écrit:

...

[...] However, since:

• my settings have not been modified recently
• logrorate and squid packages have not been updated by CentOS 5.6 (see below)

...an issue with glibc ?

Squid and logrotate work just fine here : $ rpm -q squid logrotate squid-2.6.STABLE21-6.el5 logrotate-3.7.4-9.el5_5.2

$ ls -l /var/log/squid/ total 92624 -rw------- 1 squid squid 43709603 mai 4 13:08 access.log -rw------- 1 squid squid 9424047 mai 1 04:02 access.log.1.gz -rw------- 1 squid squid 9751039 avr 24 04:02 access.log.2.gz -rw------- 1 squid squid 10745924 avr 17 04:02 access.log.3.gz -rw------- 1 squid squid 11373152 avr 10 04:02 access.log.4.gz -rw------- 1 squid squid 9620745 avr 3 04:02 access.log.5.gz -rw-r----- 1 squid squid 2794 mai 4 11:58 cache.log -rw-r----- 1 squid squid 2503 mai 1 04:02 cache.log.1.gz -rw-r----- 1 squid squid 1580 avr 24 04:02 cache.log.2.gz -rw-r----- 1 squid squid 1881 avr 17 04:02 cache.log.3.gz -rw-r----- 1 squid squid 1591 avr 10 04:02 cache.log.4.gz -rw-r----- 1 squid squid 2772 avr 3 04:02 cache.log.5.gz -rw-r--r-- 1 squid squid 511 nov 26 09:13 squid.out

Have you double checked that your problem is not a permission problem, or some SELinux issue ?

SELinux is disabled.

[root@stargate ~]# grep SELINUX /etc/selinux/config # SELINUX= can take one of these three values: SELINUX=disabled # SELINUXTYPE= type of policy in use. Possible values are: SELINUXTYPE=targeted

You can also try, as root : $ logrotate -d -v -f /etc/logrotate.d/squid # and without -d $ squid -k rotate

[root@stargate ~]# /etc/init.d/squid status squid (pid 6173) is running... [root@stargate ~]# logrotate -d -v -f /etc/logrotate.d/squid reading config file /etc/logrotate.d/squid reading config info for /var/log/squid/access.log reading config info for /var/log/squid/cache.log reading config info for /var/log/squid/store.log

Handling 3 logs

rotating pattern: /var/log/squid/access.log forced from command line (5 rotations) empty log files are not rotated, old logs are removed considering log /var/log/squid/access.log log needs rotating rotating log /var/log/squid/access.log, log->rotateCount is 5 renaming /var/log/squid/access.log.5.gz to /var/log/squid/access.log.6.gz (rotatecount 5, logstart 1, i 5), renaming /var/log/squid/access.log.4.gz to /var/log/squid/access.log.5.gz (rotatecount 5, logstart 1, i 4), renaming /var/log/squid/access.log.3.gz to /var/log/squid/access.log.4.gz (rotatecount 5, logstart 1, i 3), renaming /var/log/squid/access.log.2.gz to /var/log/squid/access.log.3.gz (rotatecount 5, logstart 1, i 2), renaming /var/log/squid/access.log.1.gz to /var/log/squid/access.log.2.gz (rotatecount 5, logstart 1, i 1), renaming /var/log/squid/access.log.0.gz to /var/log/squid/access.log.1.gz (rotatecount 5, logstart 1, i 0), copying /var/log/squid/access.log to /var/log/squid/access.log.1 truncating /var/log/squid/access.log compressing log with: /bin/gzip removing old log /var/log/squid/access.log.6.gz

rotating pattern: /var/log/squid/cache.log forced from command line (5 rotations) empty log files are not rotated, old logs are removed considering log /var/log/squid/cache.log log needs rotating rotating log /var/log/squid/cache.log, log->rotateCount is 5 renaming /var/log/squid/cache.log.5.gz to /var/log/squid/cache.log.6.gz (rotatecount 5, logstart 1, i 5), renaming /var/log/squid/cache.log.4.gz to /var/log/squid/cache.log.5.gz (rotatecount 5, logstart 1, i 4), renaming /var/log/squid/cache.log.3.gz to /var/log/squid/cache.log.4.gz (rotatecount 5, logstart 1, i 3), renaming /var/log/squid/cache.log.2.gz to /var/log/squid/cache.log.3.gz (rotatecount 5, logstart 1, i 2), renaming /var/log/squid/cache.log.1.gz to /var/log/squid/cache.log.2.gz (rotatecount 5, logstart 1, i 1), renaming /var/log/squid/cache.log.0.gz to /var/log/squid/cache.log.1.gz (rotatecount 5, logstart 1, i 0), copying /var/log/squid/cache.log to /var/log/squid/cache.log.1 truncating /var/log/squid/cache.log compressing log with: /bin/gzip removing old log /var/log/squid/cache.log.6.gz

rotating pattern: /var/log/squid/store.log forced from command line (5 rotations) empty log files are not rotated, old logs are removed considering log /var/log/squid/store.log log needs rotating rotating log /var/log/squid/store.log, log->rotateCount is 5 renaming /var/log/squid/store.log.5.gz to /var/log/squid/store.log.6.gz (rotatecount 5, logstart 1, i 5), renaming /var/log/squid/store.log.4.gz to /var/log/squid/store.log.5.gz (rotatecount 5, logstart 1, i 4), renaming /var/log/squid/store.log.3.gz to /var/log/squid/store.log.4.gz (rotatecount 5, logstart 1, i 3), renaming /var/log/squid/store.log.2.gz to /var/log/squid/store.log.3.gz (rotatecount 5, logstart 1, i 2), renaming /var/log/squid/store.log.1.gz to /var/log/squid/store.log.2.gz (rotatecount 5, logstart 1, i 1), renaming /var/log/squid/store.log.0.gz to /var/log/squid/store.log.1.gz (rotatecount 5, logstart 1, i 0), copying /var/log/squid/store.log to /var/log/squid/store.log.1 truncating /var/log/squid/store.log running postrotate script running script with arg /var/log/squid/store.log : " /usr/sbin/squid -k rotate " compressing log with: /bin/gzip removing old log /var/log/squid/store.log.6.gz [root@stargate ~]# /etc/init.d/squid status squid (pid 6173) is running... [root@stargate ~]# logrotate -v -f /etc/logrotate.d/squid reading config file /etc/logrotate.d/squid reading config info for /var/log/squid/access.log reading config info for /var/log/squid/cache.log reading config info for /var/log/squid/store.log

Handling 3 logs

rotating pattern: /var/log/squid/access.log forced from command line (5 rotations) empty log files are not rotated, old logs are removed considering log /var/log/squid/access.log log needs rotating rotating log /var/log/squid/access.log, log->rotateCount is 5 renaming /var/log/squid/access.log.5.gz to /var/log/squid/access.log.6.gz (rotatecount 5, logstart 1, i 5), renaming /var/log/squid/access.log.4.gz to /var/log/squid/access.log.5.gz (rotatecount 5, logstart 1, i 4), renaming /var/log/squid/access.log.3.gz to /var/log/squid/access.log.4.gz (rotatecount 5, logstart 1, i 3), renaming /var/log/squid/access.log.2.gz to /var/log/squid/access.log.3.gz (rotatecount 5, logstart 1, i 2), renaming /var/log/squid/access.log.1.gz to /var/log/squid/access.log.2.gz (rotatecount 5, logstart 1, i 1), renaming /var/log/squid/access.log.0.gz to /var/log/squid/access.log.1.gz (rotatecount 5, logstart 1, i 0), old log /var/log/squid/access.log.0.gz does not exist copying /var/log/squid/access.log to /var/log/squid/access.log.1 truncating /var/log/squid/access.log compressing log with: /bin/gzip removing old log /var/log/squid/access.log.6.gz

rotating pattern: /var/log/squid/cache.log forced from command line (5 rotations) empty log files are not rotated, old logs are removed considering log /var/log/squid/cache.log log needs rotating rotating log /var/log/squid/cache.log, log->rotateCount is 5 renaming /var/log/squid/cache.log.5.gz to /var/log/squid/cache.log.6.gz (rotatecount 5, logstart 1, i 5), renaming /var/log/squid/cache.log.4.gz to /var/log/squid/cache.log.5.gz (rotatecount 5, logstart 1, i 4), renaming /var/log/squid/cache.log.3.gz to /var/log/squid/cache.log.4.gz (rotatecount 5, logstart 1, i 3), renaming /var/log/squid/cache.log.2.gz to /var/log/squid/cache.log.3.gz (rotatecount 5, logstart 1, i 2), renaming /var/log/squid/cache.log.1.gz to /var/log/squid/cache.log.2.gz (rotatecount 5, logstart 1, i 1), renaming /var/log/squid/cache.log.0.gz to /var/log/squid/cache.log.1.gz (rotatecount 5, logstart 1, i 0), old log /var/log/squid/cache.log.0.gz does not exist copying /var/log/squid/cache.log to /var/log/squid/cache.log.1 truncating /var/log/squid/cache.log compressing log with: /bin/gzip removing old log /var/log/squid/cache.log.6.gz

rotating pattern: /var/log/squid/store.log forced from command line (5 rotations) empty log files are not rotated, old logs are removed considering log /var/log/squid/store.log log needs rotating rotating log /var/log/squid/store.log, log->rotateCount is 5 renaming /var/log/squid/store.log.5.gz to /var/log/squid/store.log.6.gz (rotatecount 5, logstart 1, i 5), renaming /var/log/squid/store.log.4.gz to /var/log/squid/store.log.5.gz (rotatecount 5, logstart 1, i 4), renaming /var/log/squid/store.log.3.gz to /var/log/squid/store.log.4.gz (rotatecount 5, logstart 1, i 3), renaming /var/log/squid/store.log.2.gz to /var/log/squid/store.log.3.gz (rotatecount 5, logstart 1, i 2), renaming /var/log/squid/store.log.1.gz to /var/log/squid/store.log.2.gz (rotatecount 5, logstart 1, i 1), renaming /var/log/squid/store.log.0.gz to /var/log/squid/store.log.1.gz (rotatecount 5, logstart 1, i 0), old log /var/log/squid/store.log.0.gz does not exist copying /var/log/squid/store.log to /var/log/squid/store.log.1 truncating /var/log/squid/store.log running postrotate script compressing log with: /bin/gzip removing old log /var/log/squid/store.log.6.gz [root@stargate ~]# /etc/init.d/squid status squid (pid 6173) is running...

$ ls -lZ /var/log/squid/ to see if there is any error message.

[root@stargate ~]# cat /var/log/squid/cache.log 2011/05/04 14:37:36| storeDirWriteCleanLogs: Starting... 2011/05/04 14:37:36| 65536 entries written so far. 2011/05/04 14:37:36| 131072 entries written so far. 2011/05/04 14:37:36| 196608 entries written so far. 2011/05/04 14:37:36| 262144 entries written so far. 2011/05/04 14:37:36| 327680 entries written so far. 2011/05/04 14:37:36| 393216 entries written so far. 2011/05/04 14:37:36| 458752 entries written so far. 2011/05/04 14:37:36| 524288 entries written so far. 2011/05/04 14:37:36| 589824 entries written so far. 2011/05/04 14:37:36| 655360 entries written so far. 2011/05/04 14:37:36| 720896 entries written so far. 2011/05/04 14:37:36| 786432 entries written so far. 2011/05/04 14:37:36| 851968 entries written so far. 2011/05/04 14:37:36| 917504 entries written so far. 2011/05/04 14:37:36| 983040 entries written so far. 2011/05/04 14:37:36| 1048576 entries written so far. 2011/05/04 14:37:36| 1114112 entries written so far. 2011/05/04 14:37:36| 1179648 entries written so far. 2011/05/04 14:37:36| Finished. Wrote 1242965 entries. 2011/05/04 14:37:36| Took 0.3 seconds (3692695.5 entries/sec). 2011/05/04 14:37:36| aioSync: flushing pending I/O operations 2011/05/04 14:37:36| aioSync: done 2011/05/04 14:37:36| logfileRotate: /var/log/squid/store.log 2011/05/04 14:37:36| logfileRotate: /var/log/squid/access.log 2011/05/04 14:39:51| storeDirWriteCleanLogs: Starting... 2011/05/04 14:39:51| 65536 entries written so far. 2011/05/04 14:39:51| 131072 entries written so far. 2011/05/04 14:39:51| 196608 entries written so far. 2011/05/04 14:39:51| 262144 entries written so far. 2011/05/04 14:39:51| 327680 entries written so far. 2011/05/04 14:39:51| 393216 entries written so far. 2011/05/04 14:39:51| 458752 entries written so far. 2011/05/04 14:39:51| 524288 entries written so far. 2011/05/04 14:39:51| 589824 entries written so far. 2011/05/04 14:39:51| 655360 entries written so far. 2011/05/04 14:39:51| 720896 entries written so far. 2011/05/04 14:39:51| 786432 entries written so far. 2011/05/04 14:39:51| 851968 entries written so far. 2011/05/04 14:39:51| 917504 entries written so far. 2011/05/04 14:39:51| 983040 entries written so far. 2011/05/04 14:39:51| 1048576 entries written so far. 2011/05/04 14:39:51| 1114112 entries written so far. 2011/05/04 14:39:51| 1179648 entries written so far. 2011/05/04 14:39:51| Finished. Wrote 1242997 entries. 2011/05/04 14:39:51| Took 0.4 seconds (3462811.6 entries/sec). 2011/05/04 14:39:51| aioSync: flushing pending I/O operations 2011/05/04 14:39:51| aioSync: done 2011/05/04 14:39:51| logfileRotate: /var/log/squid/store.log 2011/05/04 14:39:51| logfileRotate: /var/log/squid/access.log

So, logrotating squid logs by hand works. Now, when logrotate is started from /etc/cron.daily/logrotate, it processes more log files. It has consequences on system load and disk space. Here are my last ideas.

Laurent.

On Wed, 4 May 2011 16:33:21 +0200 Philippe Naudin philippe.naudin@supagro.inra.fr wrote:

Not sure it is working : logrotate -d is debug mode, it actually does nothing. You have to try again without -d to check if it works. Don't forget to check the content of /var/log/squid/ (file size, date and time) for the result.

I have run the following commands: logrotate -d -v -f /etc/logrotate.d/squid logrotate -v -f /etc/logrotate.d/squid /usr/sbin/squid -k rotate

After each one, "service squid status" was returning squid's pid. squid's logs were normal.

squid's logs are rotated on a weekly basis by logrorate. squid crashed 2 times, with one week between.

Now, I have switched to daily/rotate 15, and squid did not crash since the change (2 days ago).

Laurent.

On Mon, 9 May 2011 10:10:59 +0200 Laurent CREPET lct@megrapet.fr wrote:

Today, squid crashed again.

So stupid I am ! I did not check for squid in /var/log/messages. Now, I know that Squid does not like to have a temporary filesystem full.

/var was always ok after logrotate completes its work, but at 04:02 sometimes was full.

[root@stargate log]# grep -i squid messages May 9 04:02:11 stargate squid[6173]: logfileWrite: /var/log/squid/store.log: (0) Success May 9 04:02:11 stargate squid[6171]: Squid Parent: child process 6173 exited due to signal 6 May 9 04:02:14 stargate squid[6171]: Squid Parent: child process 20015 started May 9 04:02:21 stargate squid[20015]: logfileWrite: /var/log/squid/store.log: (28) No space left on device May 9 04:02:21 stargate squid[6171]: Squid Parent: child process 20015 exited due to signal 6 May 9 04:02:24 stargate squid[6171]: Squid Parent: child process 20109 started May 9 04:02:31 stargate squid[20109]: logfileWrite: /var/log/squid/store.log: (28) No space left on device May 9 04:02:31 stargate squid[6171]: Squid Parent: child process 20109 exited due to signal 6 May 9 04:02:34 stargate squid[6171]: Squid Parent: child process 20148 started May 9 04:02:35 stargate squid[20148]: logfileWrite: /var/log/squid/store.log: (28) No space left on device May 9 04:02:35 stargate squid[6171]: Squid Parent: child process 20148 exited due to signal 6 May 9 04:02:38 stargate squid[6171]: Squid Parent: child process 20173 started May 9 04:02:42 stargate squid[20173]: logfileWrite: /var/log/squid/store.log: (28) No space left on device May 9 04:02:42 stargate squid[6171]: Squid Parent: child process 20173 exited due to signal 6 May 9 04:02:45 stargate squid[6171]: Squid Parent: child process 20241 started May 9 04:02:45 stargate squid[20241]: logfileWrite: /var/log/squid/store.log: (28) No space left on device May 9 04:02:45 stargate squid[6171]: Squid Parent: child process 20241 exited due to signal 6 May 9 04:02:45 stargate squid[6171]: Exiting due to repeated, frequent failures May 9 09:08:07 stargate squid[31892]: Squid Parent: child process 31894 started [root@stargate log]#

Laurent.

On Mon, 9 May 2011 05:20:48 -0700 (PDT) John Doe jdmls@yahoo.com wrote:

From: Laurent CREPET lct@megrapet.fr

...

Today, squid crashed again.

Maybe ask on the squid mailing list, they might have more insight...

JD

No need, I have plugged my brain today. Check my latest e-mail (squid does not like temporary filesystem full in /var/log/squid) ;-)

No link with 5.6 upgrade at all.

Thanks for your help (all).

Regards, Laurent.

On Monday, May 09, 2011 10:28 PM, Ljubomir Ljubojevic wrote:

Laurent CREPET wrote:

...

On Mon, 9 May 2011 05:20:48 -0700 (PDT) John Doejdmls@yahoo.com wrote:

...

From: Laurent CREPETlct@megrapet.fr

...

Today, squid crashed again.

Maybe ask on the squid mailing list, they might have more insight...

JD

No need, I have plugged my brain today. Check my latest e-mail (squid does not like temporary filesystem full in /var/log/squid) ;-)

No link with 5.6 upgrade at all.

Maybe /var/cache/yum was not cleaned of downloaded rpms after upgrade to 5.6? so /var partition run out of space or helped with out of space issue just enough to put it over the top.

Or maybe it was stuffed with squid reports...

On Tuesday, May 10, 2011 04:51 PM, Brandon Ooi wrote:

On Tue, May 10, 2011 at 1:09 AM, Christopher Chan <christopher.chan@bradbury.edu.hk mailto:christopher.chan@bradbury.edu.hk> wrote:

On Monday, May 09, 2011 10:28 PM, Ljubomir Ljubojevic wrote:
 > Laurent CREPET wrote:
 >> On Mon, 9 May 2011 05:20:48 -0700 (PDT)
 >> John Doe<jdmls@yahoo.com <mailto:jdmls@yahoo.com>>  wrote:
 >>
 >>> From: Laurent CREPET<lct@megrapet.fr <mailto:lct@megrapet.fr>>
 >>>
 >>>> Today, squid crashed  again.
 >>> Maybe ask on the squid mailing list, they might have more
insight...
 >>>
 >>> JD
 >>
 >> No need, I have plugged my brain today. Check my latest e-mail
(squid does not like temporary filesystem full in /var/log/squid) ;-)
 >>
 >> No link with 5.6 upgrade at all.
 >
 > Maybe /var/cache/yum was not cleaned of downloaded rpms after
upgrade to
 > 5.6? so /var partition run out of space or helped with out of space
 > issue just enough to put it over the top.
 >

Or maybe it was stuffed with squid reports...

a whole week of troubleshooting and you didn't check /var/log/messages? comon man.

When I visit servers to make changes, I almost always check /var/log/messages and dmesg for latent errors or other issues that I can proactively solve. This should be sysadmin 101 :P

Nevermind having basic monitoring in place to check current status of various resources or allocating disk space for the sole use of squid or...hey Brandon, what else should be sysadmin 101?

On 05/03/2011 03:20 AM, Laurent CREPET wrote:

On Tue, 3 May 2011 09:46:05 +0200 Laurent CREPET lct@megrapet.fr wrote:

...

I will look at the latest RPMs to check differences on /etc/logrotate.d/squid.

I'm suprised... it seems that both packages have not been modified for a while.

...

[root@stargate ~]# ls -latr /var/log/yum.log* -rw-r--r-- 1 root root 11559 Dec 30 2009 /var/log/yum.log.2 -rw------- 1 root root 27610 Dec 23 11:27 /var/log/yum.log.1 -rw------- 1 root root 11234 May 1 08:45 /var/log/yum.log [root@stargate ~]# [root@stargate ~]# egrep -i "squid|logrotate" !$ egrep -i "squid|logrotate" /var/log/yum.log* /var/log/yum.log.1:May 28 16:19:29 Updated: 7:squid-2.6.STABLE21-6.el5.x86_64 /var/log/yum.log.1:Oct 15 13:07:31 Updated: logrotate-3.7.4-9.el5_5.1.x86_64 /var/log/yum.log.1:Nov 08 09:26:57 Updated: logrotate-3.7.4-9.el5_5.2.x86_64 /var/log/yum.log.2:Dec 06 13:14:54 Installed: 7:squid-2.6.STABLE21-3.el5.x86_64

...

[root@stargate ~]# grep -i centos-release /var/log/yum.log* /var/log/yum.log:Apr 18 15:27:25 Updated: centos-release-notes-5.6-0.x86_64 /var/log/yum.log:Apr 18 15:27:25 Updated: 10:centos-release-5-6.el5.centos.1.x86_64 /var/log/yum.log.1:May 28 16:19:13 Updated: centos-release-notes-5.5-0.x86_64 /var/log/yum.log.1:May 28 16:19:13 Updated: 10:centos-release-5-5.el5.centos.x86_64

Any ideas ?

I do not see anything listed in the upsteam bugzilla concerning squid and EL5.6 ... anyone else have this issue?