Hi,
I have a centos 4.4 box running Apache serving up web pages. One of my customers has a marketing firm that once a week pulls all of the content so that they can do their thing.
When they do this the box stops serving web pages. The load goes to 0.0 and top shows almost no memory free. The box does not swap nor do I get OOM to kick in. Normally the box has about 20-30 httpd processes running but when they are pulling the whole web site I see about 255 httpd processes. If I kill the httpd processes then top shows the load immediately spiking to over 30 and then coming back down to normal.
The box is a 3.2 gig P4 with 2 gigs of memory in it. Normally top shows it has 300-400 megs free, with about 300 megs in buffers.
I can have the marketing company throttle their downloading of the web site but I think that is the wrong answer. I would like to fix the real cause of the problem.
Does anyone know what is going on here or how to troubleshoot this? Is there some kind of tuning I am supposed to be doing to prevent this kind of thing? Do I need more memory?
Any help would be appreciated including pointers to docs on this type of thing. So far I have been unsuccessful in finding the correct google incantation to help explain this.
Regards,