Dear all,
I am getting below syslog error on my oracle database server running on RHEL4U5. During this time my swap shows zero on both available and free in the top command output , result in total system hang. can any one guide me how to fix this issue of memory and exactly what this error indicates??
ERROR FROM SYSLOG
##################################################################################################### Aug 30 15:46:20 crmdb kernel: cpu 7 cold: low 0, high 32, batch 16 Aug 30 15:46:20 crmdb kernel: Aug 30 15:46:20 crmdb kernel: Free pages: 16000kB (2560kB HighMem) Aug 30 15:46:20 crmdb kernel: Active:1433949 inactive:40363 dirty:1 writeback:0 unstable:0 free:4 000 slab:24910 mapped:1405036 pagetables:565097 Aug 30 15:46:20 crmdb kernel: DMA free:12544kB min:16kB low:32kB high:48kB active:0kB inactive:0k B present:16384kB pages_scanned:1878896 all_unreclaimable? yes Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0 Aug 30 15:46:20 crmdb kernel: Normal free:896kB min:928kB low:1856kB high:2784kB active:736kB ina ctive:208kB present:901120kB pages_scanned:1889 all_unreclaimable? yes Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0 Aug 30 15:46:20 crmdb kernel: HighMem free:2560kB min:512kB low:1024kB high:1536kB active:5735060 kB inactive:161244kB present:8257532kB pages_scanned:0 all_unreclaimable? no Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0 Aug 30 15:46:20 crmdb kernel: DMA: 2*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*102 4kB 1*2048kB 2*4096kB = 12544kB Aug 30 15:46:20 crmdb kernel: Normal: 0*4kB 28*8kB 2*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0 *1024kB 0*2048kB 0*4096kB = 896kB Aug 30 15:46:20 crmdb kernel: HighMem: 512*4kB 8*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2560kB Aug 30 15:46:20 crmdb kernel: Swap cache: add 26913978, delete 26700605, find 10635368/14522876, race 848+1364 Aug 30 15:46:20 crmdb kernel: 0 bounce buffer pages Aug 30 15:46:20 crmdb kernel: Free swap: 14593184kB Aug 30 15:46:20 crmdb kernel: 2293759 pages of RAM Aug 30 15:46:20 crmdb kernel: 1867351 pages of HIGHMEM Aug 30 15:46:20 crmdb kernel: 216549 reserved pages Aug 30 15:46:20 crmdb kernel: 44388024 pages shared Aug 30 15:46:20 crmdb kernel: 213522 pages swap cached Aug 30 15:46:20 crmdb kernel: Out of Memory: Killed process 22462 (oracle). ##############################################################################################################
Regards lingu
lingu wrote:
Dear all,
I am getting below syslog error on my oracle database server running on RHEL4U5. During this time my swap shows zero on both available and free in the top command output , result in total system hang. can any one guide me how to fix this issue of memory and exactly what this error indicates??
ERROR FROM SYSLOG
##################################################################################################### Aug 30 15:46:20 crmdb kernel: cpu 7 cold: low 0, high 32, batch 16 Aug 30 15:46:20 crmdb kernel: Aug 30 15:46:20 crmdb kernel: Free pages: 16000kB (2560kB HighMem) Aug 30 15:46:20 crmdb kernel: Active:1433949 inactive:40363 dirty:1 writeback:0 unstable:0 free:4 000 slab:24910 mapped:1405036 pagetables:565097 Aug 30 15:46:20 crmdb kernel: DMA free:12544kB min:16kB low:32kB high:48kB active:0kB inactive:0k B present:16384kB pages_scanned:1878896 all_unreclaimable? yes Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0 Aug 30 15:46:20 crmdb kernel: Normal free:896kB min:928kB low:1856kB high:2784kB active:736kB ina ctive:208kB present:901120kB pages_scanned:1889 all_unreclaimable? yes Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0 Aug 30 15:46:20 crmdb kernel: HighMem free:2560kB min:512kB low:1024kB high:1536kB active:5735060 kB inactive:161244kB present:8257532kB pages_scanned:0 all_unreclaimable? no Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0 Aug 30 15:46:20 crmdb kernel: DMA: 2*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*102 4kB 1*2048kB 2*4096kB = 12544kB Aug 30 15:46:20 crmdb kernel: Normal: 0*4kB 28*8kB 2*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0 *1024kB 0*2048kB 0*4096kB = 896kB Aug 30 15:46:20 crmdb kernel: HighMem: 512*4kB 8*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2560kB Aug 30 15:46:20 crmdb kernel: Swap cache: add 26913978, delete 26700605, find 10635368/14522876, race 848+1364 Aug 30 15:46:20 crmdb kernel: 0 bounce buffer pages Aug 30 15:46:20 crmdb kernel: Free swap: 14593184kB Aug 30 15:46:20 crmdb kernel: 2293759 pages of RAM Aug 30 15:46:20 crmdb kernel: 1867351 pages of HIGHMEM Aug 30 15:46:20 crmdb kernel: 216549 reserved pages Aug 30 15:46:20 crmdb kernel: 44388024 pages shared Aug 30 15:46:20 crmdb kernel: 213522 pages swap cached Aug 30 15:46:20 crmdb kernel: Out of Memory: Killed process 22462 (oracle). ##############################################################################################################
Not sure if this is connected, but I had what looks like a similar problem - see: https://bugzilla.redhat.com/show_bug.cgi?id=193542
The workaround I use is to use the following sysctl setting:
vm.lower_zone_protection = 100
James Pearson
lingu wrote:
Dear all,
I am getting below syslog error on my oracle database server running on RHEL4U5. During this time my swap shows zero on both available and free in the top command output , result in total system hang. can any one guide me how to fix this issue of memory and exactly what this error indicates??
It indicates that you ran out of memory.
1) add more memory and/or 2) monitor the system closely to determine what is using all of the available memory and either configure it not to, kill it, or don't run it at all.
The system sometimes(depending on kernel version) tries to kill processes it thinks might be related to the excessive memory usage, to prevent the total system hang, but it's not always successful.
I use nagios for threshold alerting, and it has a handy check_swap monitor, I use it to alert when swap is 5% utilized. For my systems swap is for emergency purposes only.
If Oracle is the only real application on that box and if this problem happened suddenly, look at the statspack information (or whatever it's called these days) and try to track down the query that caused the spike in memory usage.
I found on a couple of occasions with bad queries in Oracle it could cause memory usage to suddenly spike. Fortunately the system had something like 16gigs of swap on an 8 disk 15k RPM RAID 1+0 that was otherwise not used, so while performance was really slow, the box didn't go down. I think the system had 16gigs of ram.
I hope you have at least 16GB of memory, with memory prices these days it's cheap even to have 32GB. Even if you don't allocate it all to Oracle it's useful to help protect against situations like this.
nate
On Thu, Sep 4, 2008 at 5:53 AM, lingu hicheerup@gmail.com wrote:
Dear all,
I am getting below syslog error on my oracle database server running on RHEL4U5. During this time my swap shows zero on both available and free in the top command output , result in total system hang. can any one guide me how to fix this issue of memory and exactly what this error indicates??
It strikes me that someone running RH4U5 should be able to get technical support from Red Hat, no?...
I know they're virtually identical, but, as others with more weight than I on this list have pointed out, this is for support on CentOS issues for CentOS users on CentOS systems, or am I going too far here?
mhr
MHR wrote:
On Thu, Sep 4, 2008 at 5:53 AM, lingu hicheerup@gmail.com wrote:
Dear all,
I am getting below syslog error on my oracle database server running on RHEL4U5. During this time my swap shows zero on both available and free in the top command output , result in total system hang. can any one guide me how to fix this issue of memory and exactly what this error indicates??
It strikes me that someone running RH4U5 should be able to get technical support from Red Hat, no?...
and, for that matter, someone running Oracle in production should have an Oracle support contract, and should work through them.
John R Pierce wrote:
and, for that matter, someone running Oracle in production should have an Oracle support contract, and should work through them.
Yeah both right, didn't even think about that.. I'd be pretty scared to run an Oracle DB with a newbie system admin. Not knowing what sort of data is in the DB, if it's even a little important I'd strongly suggest the OP getting someone more experienced to manage the system, Not knowing where the OP is I recommend Blue Gecko, a U.S. Oracle/Linux consulting shop based out of Seattle, used them for Oracle DBs I had at my last company.
Oracle is a big complex beast, really powerful, and a real good product if you use it right. When I first started using it I really didn't like it and wanted to get rid of it, but it's grown on me a lot over the years and I've gained a lot of respect for it as a product and the technologies behind it. And while Enterprise edition is really expensive, Standard edition is dirt cheap by comparison and still very capable.
nate
-
James Pearson -
John R Pierce -
lingu -
MHR -
nate