I'm having an issue with an apache web server running the latest CentOS5 kernel (this issue is not new to the kernel). After a few days/weeks of running the server will become unresponsive and will require a physical reboot in order to come back online. The system is so unresponsive when the issue occurs that login at console is not even possible.
I have atop installed and have looked back before the crash to see what happened process wise and I can see the http starts using a lot of memory and CPU usage. The vmcommit jumps from 1.8 GB to 4.8GB in a matter of a few minutes. The VSIZE of the httpd process jumps from 8.1GB to 36.9GB. So apache is doing something -- but how can I get historical data for this? I also see that paging is very active, probably why the server is unresponsive. I have looked through the apache logs and system logs and there is nothing obvious that is consuming all that memory. I know of the server-status module for apache but that is only useful if you can get to the server during the crash (I can't) and doesn't have any historical data.
The issue occurs seemingly randomly, last time in the middle of the night with little or no user traffic.
James