[CentOS] Server unresponsive until reboot, memory exhausted

Tue Dec 28 16:49:43 UTC 2010
Nico Kadel-Garcia <nkadel at gmail.com>

On Tue, Dec 28, 2010 at 9:59 AM, james <james at wintercastle.net> wrote:
> I'm having an issue with an apache web server running the latest CentOS5
> kernel (this issue is not new to the kernel). After a few days/weeks of
> running the server will become unresponsive and will require a physical
> reboot in order to come back online. The system is so unresponsive when the
> issue occurs that login at console is not even possible.

Do you have everything *else* updated? And what kind of web service
are you running?

There's a lot of third party freeware and commercial tools that was
not written with any kind of resource management in mind, and which
may require a simple web server restart on a regular baris to free
memory. (MusicBrainz: I remember porting MusicBrainz.....)

> I have atop installed and have looked back before the crash to see what
> happened process wise and I can see the http starts using a lot of memory
> and CPU usage. The vmcommit jumps from 1.8 GB to 4.8GB in a matter of a few
> minutes. The VSIZE of the httpd process jumps from 8.1GB to 36.9GB. So
> apache is doing something -- but how can I get historical data for this? I

Look in /var/log/http/*.

> also see that paging is very active, probably why the server is
> unresponsive. I have looked through the apache logs and system logs and
> there is nothing obvious that is consuming all that memory. I know of the
> server-status module for apache but that is only useful if you can get to
> the server during the crash (I can't) and doesn't have any historical data.
>
> The issue occurs seemingly randomly, last time in the middle of the night
> with little or no user traffic.

Do you have a search engine scanning your web server?