One of our systems run Apache 2.0.52 (httpd-2.0.52-12.2.ent.centos4 rpm). The kernel we're running is 2.6.9-11.ELsmp. After a few weeks of uptime, Apache refuses to start with the following error in error.log:
[crit] (28)No space left on device: mod_jk: could not create jk_log_lock
With Apache not running, there're a lot of semaphores left behind by Apache, e.g.:
------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 98304 root 600 1 0x00000000 131073 apache 600 1 0x00000000 163842 apache 600 1 0x00000000 196611 apache 600 1 0x00000000 229380 apache 600 1 0x00000000 786437 root 600 1 ... ... ... /proc/sys/kernel/sem has the following settings: 250 32000 32 128
Is this an Apache issue or is this a kernel issue? Why is Apache not releasing the semaphores?
Thanks for any help.
I've no idea what causes it, but this is how I solved it: http://www.sagonet.com/forums/showthread.php?p=14635#post14635
Someone else know why this happens?
Jonathan
Fong Vang wrote:
One of our systems run Apache 2.0.52 (httpd-2.0.52-12.2.ent.centos4 rpm). The kernel we're running is 2.6.9-11.ELsmp. After a few weeks of uptime, Apache refuses to start with the following error in error.log:
[crit] (28)No space left on device: mod_jk: could not create jk_log_lock
<snip>
On Tue, 2005-10-04 at 17:45 -0700, Fong Vang wrote:
One of our systems run Apache 2.0.52 (httpd-2.0.52-12.2.ent.centos4 rpm). The kernel we're running is 2.6.9-11.ELsmp. After a few weeks of uptime, Apache refuses to start with the following error in error.log:
[crit] (28)No space left on device: mod_jk: could not create jk_log_lockWith Apache not running, there're a lot of semaphores left behind by Apache, e.g.:
------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 98304 root 600 1 0x00000000 131073 apache 600 1 0x00000000 163842 apache 600 1 0x00000000 196611 apache 600 1 0x00000000 229380 apache 600 1 0x00000000 786437 root 600 1 ... ... ... /proc/sys/kernel/sem has the following settings: 250 32000 32 128
Is this an Apache issue or is this a kernel issue? Why is Apache not releasing the semaphores?
---- my guess is that you are out of space on partition containing /var
try command line...
df -h
see what 'No space on device' actually means
Craig
On 10/4/05, Craig White craigwhite@azapple.com wrote:
On Tue, 2005-10-04 at 17:45 -0700, Fong Vang wrote:
One of our systems run Apache 2.0.52 (httpd-2.0.52-12.2.ent.centos4 rpm). The kernel we're running is 2.6.9-11.ELsmp. After a few weeks of uptime, Apache refuses to start with the following error in error.log:
[crit] (28)No space left on device: mod_jk: could not create jk_log_lockWith Apache not running, there're a lot of semaphores left behind by Apache, e.g.:
------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 98304 root 600 1 0x00000000 131073 apache 600 1 0x00000000 163842 apache 600 1 0x00000000 196611 apache 600 1 0x00000000 229380 apache 600 1 0x00000000 786437 root 600 1 ... ... ... /proc/sys/kernel/sem has the following settings: 250 32000 32 128
Is this an Apache issue or is this a kernel issue? Why is Apache not releasing the semaphores?
my guess is that you are out of space on partition containing /var
try command line...
df -h
see what 'No space on device' actually means
Space is not the issue. That's the first thing I checked. It's definitely because the system ran out of semaphores. After I cleared out the semaphores belonging to Apache, it started again.
I need to understand why Apache is not releasing the semaphores on shutdown (or why the system does not release the semaphore if Apache is no longer running).
Craig
-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 10/4/05, Fong Vang sudoyang@gmail.com wrote:
On 10/4/05, Craig White craigwhite@azapple.com wrote:
On Tue, 2005-10-04 at 17:45 -0700, Fong Vang wrote:
One of our systems run Apache 2.0.52 (httpd-2.0.52-12.2.ent.centos4 rpm). The kernel we're running is 2.6.9-11.ELsmp. After a few weeks of uptime, Apache refuses to start with the following error in error.log:
[crit] (28)No space left on device: mod_jk: could not create jk_log_lockWith Apache not running, there're a lot of semaphores left behind by Apache, e.g.:
------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 98304 root 600 1 0x00000000 131073 apache 600 1 0x00000000 163842 apache 600 1 0x00000000 196611 apache 600 1 0x00000000 229380 apache 600 1 0x00000000 786437 root 600 1
It's not an apache issue exactly. We had a similar issue with a third party authentication system. Somthing in it doesn't clean up after itself like it should. I'd recommend looking at the java code, or looking at what your webserver is running that's not stock. It could be bad something in java that's being done outside of apache executable, but as the apache user. Our vendor released an upgrade of the auth system that seems to have fixed our issue, so hopefully there's a fix for yours as well.
-- Jim Perrin System Administrator - UIT Ft Gordon & US Army Signal Center
Fong Vang wrote:
(or why the system does not release the semaphore if Apache is no longer running).
Because IPC resources (semaphores, message queues and shared memory segments) are not tied to a process, an process simply has access to them (and can request for resource to be created or destroyed). It is perfectly OK for a process to create any IPC resource, exit, and then some other process started at a later time to pick it up. If something is allocating IPC resources and not cleaning up behind itself, than it is a bug in application (where application != process). IPCs are outside of that nice box called process, so you should think outside of the box too.
On 10/4/05, Aleksandar Milivojevic alex@milivojevic.org wrote:
Fong Vang wrote:
(or why the system does not release the semaphore if Apache is no longer running).
Because IPC resources (semaphores, message queues and shared memory segments) are not tied to a process, an process simply has access to them (and can request for resource to be created or destroyed). It is perfectly OK for a process to create any IPC resource, exit, and then some other process started at a later time to pick it up. If something is allocating IPC resources and not cleaning up behind itself, than it is a bug in application (where application != process). IPCs are outside of that nice box called process, so you should think outside of the box too.
Makes sense. So looks like there's a semaphore leak. The owner of the semaphore is the "apache" user. Is this most likely Apache or could it be something else?
Is there a way to have the system automatically cleanup?
thanks for the help.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 10/4/05, Aleksandar Milivojevic alex@milivojevic.org wrote:
Fong Vang wrote:
(or why the system does not release the semaphore if Apache is no longer running).
Because IPC resources (semaphores, message queues and shared memory segments) are not tied to a process, an process simply has access to them (and can request for resource to be created or destroyed). It is perfectly OK for a process to create any IPC resource, exit, and then some other process started at a later time to pick it up. If something is allocating IPC resources and not cleaning up behind itself, than it is a bug in application (where application != process). IPCs are outside of that nice box called process, so you should think outside of the box too.
Excellent. Is there a way to tell which process is creating these semaphores? ipcs only lists the user/owner.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Quoting Fong Vang sudoyang@gmail.com:
Excellent. Is there a way to tell which process is creating these semaphores? ipcs only lists the user/owner.
Not really.
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
-
Aleksandar Milivojevic -
Craig White -
Fong Vang -
Jim Perrin -
Jonathan