[CentOS] swap memory crash

Thu Sep 4 14:40:20 UTC 2008
nate <centos at linuxpowered.net>

lingu wrote:
> Dear all,
>
>    I am getting below syslog error on my oracle database server running on
> RHEL4U5. During this time my swap shows zero on both available and  free in
> the top command output , result in total system hang. can any one guide me
> how to fix this issue of memory and exactly what  this error indicates??

It indicates that you ran out of memory.

1) add more memory
and/or
2) monitor the system closely to determine what is using all of
the available memory and either configure it not to, kill it, or
don't run it at all.

The system sometimes(depending on kernel version) tries to
kill processes it thinks might be related to the excessive
memory usage, to prevent the total system hang, but it's not
always successful.

I use nagios for threshold alerting, and it has a handy check_swap
monitor, I use it to alert when swap is 5% utilized. For my
systems swap is for emergency purposes only.

If Oracle is the only real application on that box and if this
problem happened suddenly, look at the statspack information
(or whatever it's called these days) and try to track down the
query that caused the spike in memory usage.

I found on a couple of occasions with bad queries in Oracle it
could cause memory usage to suddenly spike. Fortunately the
system had something like 16gigs of swap on an 8 disk 15k RPM
RAID 1+0 that was otherwise not used, so while performance
was really slow, the box didn't go down. I think the system
had 16gigs of ram.

I hope you have at least 16GB of memory, with memory prices
these days it's cheap even to have 32GB. Even if you don't
allocate it all to Oracle it's useful to help protect
against situations like this.

nate