[CentOS] swap memory crash

Thu Sep 4 14:18:49 UTC 2008
James Pearson <james-p at moving-picture.com>

lingu wrote:
> Dear all,
> 
>    I am getting below syslog error on my oracle database server running on
> RHEL4U5. During this time my swap shows zero on both available and  free in
> the top command output , result in total system hang. can any one guide me
> how to fix this issue of memory and exactly what  this error indicates??
> 
> ERROR FROM SYSLOG
> 
> #####################################################################################################
> Aug 30 15:46:20 crmdb kernel: cpu 7 cold: low 0, high 32, batch 16
> Aug 30 15:46:20 crmdb kernel:
> Aug 30 15:46:20 crmdb kernel: Free pages:       16000kB (2560kB HighMem)
> Aug 30 15:46:20 crmdb kernel: Active:1433949 inactive:40363 dirty:1
> writeback:0 unstable:0 free:4
> 000 slab:24910 mapped:1405036 pagetables:565097
> Aug 30 15:46:20 crmdb kernel: DMA free:12544kB min:16kB low:32kB high:48kB
> active:0kB inactive:0k
> B present:16384kB pages_scanned:1878896 all_unreclaimable? yes
> Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0
> Aug 30 15:46:20 crmdb kernel: Normal free:896kB min:928kB low:1856kB
> high:2784kB active:736kB ina
> ctive:208kB present:901120kB pages_scanned:1889 all_unreclaimable? yes
> Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0
> Aug 30 15:46:20 crmdb kernel: HighMem free:2560kB min:512kB low:1024kB
> high:1536kB active:5735060
> kB inactive:161244kB present:8257532kB pages_scanned:0 all_unreclaimable? no
> Aug 30 15:46:20 crmdb kernel: protections[]: 0 0 0
> Aug 30 15:46:20 crmdb kernel: DMA: 2*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
> 2*256kB 0*512kB 1*102
> 4kB 1*2048kB 2*4096kB = 12544kB
> Aug 30 15:46:20 crmdb kernel: Normal: 0*4kB 28*8kB 2*16kB 0*32kB 0*64kB
> 1*128kB 0*256kB 1*512kB 0
> *1024kB 0*2048kB 0*4096kB = 896kB
> Aug 30 15:46:20 crmdb kernel: HighMem: 512*4kB 8*8kB 2*16kB 1*32kB 0*64kB
> 1*128kB 1*256kB 0*512kB
>  0*1024kB 0*2048kB 0*4096kB = 2560kB
> Aug 30 15:46:20 crmdb kernel: Swap cache: add 26913978, delete 26700605,
> find 10635368/14522876, race 848+1364
> Aug 30 15:46:20 crmdb kernel: 0 bounce buffer pages
> Aug 30 15:46:20 crmdb kernel: Free swap:       14593184kB
> Aug 30 15:46:20 crmdb kernel: 2293759 pages of RAM
> Aug 30 15:46:20 crmdb kernel: 1867351 pages of HIGHMEM
> Aug 30 15:46:20 crmdb kernel: 216549 reserved pages
> Aug 30 15:46:20 crmdb kernel: 44388024 pages shared
> Aug 30 15:46:20 crmdb kernel: 213522 pages swap cached
> Aug 30 15:46:20 crmdb kernel: Out of Memory: Killed process 22462 (oracle).
> ##############################################################################################################

Not sure if this is connected, but I had what looks like a similar 
problem - see: <https://bugzilla.redhat.com/show_bug.cgi?id=193542>

The workaround I use is to use the following sysctl setting:

vm.lower_zone_protection = 100

James Pearson