[CentOS] oom-killer.

William Warren hescominsoon at emmanuelcomputerconsulting.com
Wed Dec 14 13:16:46 UTC 2005 

set swappiness to zero.  That will force the machine to use all physical 
ram before swapping for the most part.  Also i have heard that xorg has 
a memory leak.  I do not know if that has been addressed yet and that 
could be the cause of your issue.

J.J.Garcia wrote:
> Hi all,
> 
> Just watching for my first time the 'oom-killer' call in a CentOS 4.2
> workstation. System was running normally but with high load of nice process,
> iow, rosetta and seti, others appart, nautilus, etc... i realized that after a
> night of standby running normally, the next moorning after few manual operations
> from X-session in local console, it started a massive swapping, this time not
> recovered as previously other time, finishing in a oom-killer call.
> 
> Finally inspected the syslog and realize the oom-killer was called to leave the
> system stable having killed firefox-bin, 3 x rosetta, xmms and one nautilus.
> 
> It's a PIII 800 Mhz, 40GB IDE, 512 Mb RAM, 1024 Swap space, up for 4 days.
> 
> I'm not very xperienced in tunning the memmory system nor the swap system, help
> appreciated to avoid this memory 'trashing' situtation (14 virtual desktops
> openned at the same time).
> 
> Is it a normal situation? Is it a correct protecction measure? It's due a low
> bandwitch IDE HD? how can i tune it to avoid this situations? is it a high load
> for this workstation? I think the oom-killer was called correctly to avoid the
> system collapsed, but i think there was a problem in the swapping subsystem or a
> memmory trashing situation to avoid.
> 
> Help and hints appreciated,
> 
> Attached you'll the oom-killer log,
> 
> TIA
> 
> J.J.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> Dec 14 11:48:00 goliathbox kernel: oom-killer: gfp_mask=0xd0
> Dec 14 11:48:00 goliathbox kernel: Mem-info:
> Dec 14 11:48:00 goliathbox kernel: DMA per-cpu:
> Dec 14 11:48:00 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
> Dec 14 11:48:00 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
> Dec 14 11:48:00 goliathbox kernel: Normal per-cpu:
> Dec 14 11:48:00 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
> Dec 14 11:48:00 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
> Dec 14 11:48:00 goliathbox kernel: HighMem per-cpu: empty
> Dec 14 11:48:00 goliathbox kernel:
> Dec 14 11:48:00 goliathbox kernel: Free pages:         676kB (0kB HighMem)
> Dec 14 11:48:00 goliathbox kernel: Active:109642 inactive:10334 dirty:0 writeback:0 unstable:0 free:169 slab:3746 mapped:118866 pagetables:1729
> Dec 14 11:48:00 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:6132kB inactive:6316kB present:16384kB pages_scanned:20658 all_unreclaimable? yes
> Dec 14 11:48:02 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:48:23 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:432436kB inactive:35020kB present:507820kB pages_scanned:1102915 all_unreclaimable? yesDec 14 11:48:23 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:48:23 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
> Dec 14 11:48:23 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:48:23 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
> Dec 14 11:48:23 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
> Dec 14 11:48:23 goliathbox kernel: HighMem: empty
> Dec 14 11:48:23 goliathbox kernel: Swap cache: add 974772, delete 974259, find 292228/382858, race 0+15
> Dec 14 11:48:23 goliathbox kernel: 0 bounce buffer pages
> Dec 14 11:48:23 goliathbox kernel: Free swap:            0kB
> Dec 14 11:48:23 goliathbox kernel: 131051 pages of RAM
> Dec 14 11:48:23 goliathbox kernel: 0 pages of HIGHMEM
> Dec 14 11:48:42 goliathbox kernel: 2433 reserved pages
> Dec 14 11:49:26 goliathbox kernel: 26433 pages shared
> Dec 14 11:49:26 goliathbox kernel: 513 pages swap cached
> Dec 14 11:49:26 goliathbox kernel: Out of Memory: Killed process 13954 (firefox-bin).
> Dec 14 11:49:26 goliathbox kernel: oom-killer: gfp_mask=0x1d2
> Dec 14 11:49:26 goliathbox kernel: Mem-info:
> Dec 14 11:49:26 goliathbox kernel: DMA per-cpu:
> Dec 14 11:49:26 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
> Dec 14 11:49:26 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
> Dec 14 11:49:26 goliathbox kernel: Normal per-cpu:
> Dec 14 11:49:26 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
> Dec 14 11:49:26 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
> Dec 14 11:49:26 goliathbox kernel: HighMem per-cpu: empty
> Dec 14 11:49:26 goliathbox kernel:
> Dec 14 11:49:27 goliathbox kernel: Free pages:         676kB (0kB HighMem)
> Dec 14 11:49:27 goliathbox kernel: Active:1868 inactive:118396 dirty:0 writeback:0 unstable:0 free:169 slab:3738 mapped:118869 pagetables:1729
> Dec 14 11:49:27 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4232kB inactive:7992kB present:16384kB pages_scanned:20064 all_unreclaimable? yes
> Dec 14 11:49:28 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:28 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:3240kB inactive:465592kB present:507820kB pages_scanned:780089 all_unreclaimable? yes
> Dec 14 11:49:28 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:29 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
> Dec 14 11:49:29 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:29 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
> Dec 14 11:49:29 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
> Dec 14 11:49:29 goliathbox kernel: HighMem: empty
> Dec 14 11:49:29 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
> Dec 14 11:49:30 goliathbox kernel: 0 bounce buffer pages
> Dec 14 11:49:30 goliathbox kernel: Free swap:            0kB
> Dec 14 11:49:30 goliathbox kernel: 131051 pages of RAM
> Dec 14 11:49:30 goliathbox kernel: 0 pages of HIGHMEM
> Dec 14 11:49:31 goliathbox kernel: 2433 reserved pages
> Dec 14 11:49:31 goliathbox kernel: 26140 pages shared
> Dec 14 11:49:31 goliathbox kernel: 513 pages swap cached
> Dec 14 11:49:31 goliathbox kernel: Out of Memory: Killed process 15771 (firefox-bin).
> Dec 14 11:49:31 goliathbox kernel: oom-killer: gfp_mask=0xd0
> Dec 14 11:49:31 goliathbox kernel: Mem-info:
> Dec 14 11:49:32 goliathbox kernel: DMA per-cpu:
> Dec 14 11:49:32 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
> Dec 14 11:49:32 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
> Dec 14 11:49:32 goliathbox kernel: Normal per-cpu:
> Dec 14 11:49:32 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
> Dec 14 11:49:32 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
> Dec 14 11:49:32 goliathbox kernel: HighMem per-cpu: empty
> Dec 14 11:49:33 goliathbox kernel:
> Dec 14 11:49:33 goliathbox kernel: Free pages:         676kB (0kB HighMem)
> Dec 14 11:49:33 goliathbox kernel: Active:73675 inactive:46271 dirty:0 writeback:0 unstable:0 free:169 slab:3735 mapped:118871 pagetables:1729
> Dec 14 11:49:33 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4384kB inactive:7968kB present:16384kB pages_scanned:20229 all_unreclaimable? yes
> Dec 14 11:49:33 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:33 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:290316kB inactive:177116kB present:507820kB pages_scanned:556644 all_unreclaimable? yesDec 14 11:49:34 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:34 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
> Dec 14 11:49:34 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:34 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
> Dec 14 11:49:34 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
> Dec 14 11:49:34 goliathbox kernel: HighMem: empty
> Dec 14 11:49:35 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
> Dec 14 11:49:35 goliathbox kernel: 0 bounce buffer pages
> Dec 14 11:49:35 goliathbox kernel: Free swap:            0kB
> Dec 14 11:49:35 goliathbox kernel: 131051 pages of RAM
> Dec 14 11:49:35 goliathbox kernel: 0 pages of HIGHMEM
> Dec 14 11:49:35 goliathbox kernel: 2433 reserved pages
> Dec 14 11:49:36 goliathbox kernel: 26474 pages shared
> Dec 14 11:49:36 goliathbox kernel: 513 pages swap cached
> Dec 14 11:49:36 goliathbox kernel: Out of Memory: Killed process 13973 (rosetta_4.79_i6).
> Dec 14 11:49:36 goliathbox kernel: Out of Memory: Killed process 13972 (rosetta_4.79_i6).
> Dec 14 11:49:36 goliathbox kernel: Out of Memory: Killed process 13974 (rosetta_4.79_i6).
> Dec 14 11:49:36 goliathbox kernel: oom-killer: gfp_mask=0x1d2
> Dec 14 11:49:36 goliathbox kernel: Mem-info:
> Dec 14 11:49:37 goliathbox kernel: DMA per-cpu:
> Dec 14 11:49:38 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
> Dec 14 11:49:38 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
> Dec 14 11:49:39 goliathbox kernel: Normal per-cpu:
> Dec 14 11:49:39 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
> Dec 14 11:49:40 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
> Dec 14 11:49:40 goliathbox kernel: HighMem per-cpu: empty
> Dec 14 11:49:40 goliathbox kernel:
> Dec 14 11:49:41 goliathbox kernel: Free pages:         676kB (0kB HighMem)
> Dec 14 11:49:41 goliathbox kernel: Active:5356 inactive:114646 dirty:0 writeback:0 unstable:0 free:169 slab:3735 mapped:118875 pagetables:1729
> Dec 14 11:49:41 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4412kB inactive:8044kB present:16384kB pages_scanned:20361 all_unreclaimable? yes
> Dec 14 11:49:41 goliathbox crond(pam_unix)[5432]: session opened for user root by (uid=0)
> Dec 14 11:49:41 goliathbox crond(pam_unix)[5434]: session opened for user root by (uid=0)
> Dec 14 11:49:41 goliathbox crond(pam_unix)[5433]: session opened for user sm0ketst by (uid=0)
> Dec 14 11:49:41 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:42 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:17012kB inactive:450540kB present:507820kB pages_scanned:2398502 all_unreclaimable? yesDec 14 11:49:43 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:43 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
> Dec 14 11:49:43 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:44 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
> Dec 14 11:49:44 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
> Dec 14 11:49:44 goliathbox kernel: HighMem: empty
> Dec 14 11:49:44 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
> Dec 14 11:49:45 goliathbox kernel: 0 bounce buffer pages
> Dec 14 11:49:45 goliathbox kernel: Free swap:            0kB
> Dec 14 11:49:46 goliathbox kernel: 131051 pages of RAM
> Dec 14 11:49:46 goliathbox kernel: 0 pages of HIGHMEM
> Dec 14 11:49:46 goliathbox kernel: 2433 reserved pages
> Dec 14 11:49:46 goliathbox kernel: 26419 pages shared
> Dec 14 11:49:47 goliathbox kernel: 513 pages swap cached
> Dec 14 11:49:47 goliathbox kernel: Out of Memory: Killed process 4936 (xmms).
> Dec 14 11:49:47 goliathbox kernel: oom-killer: gfp_mask=0x1d2
> Dec 14 11:49:48 goliathbox kernel: Mem-info:
> Dec 14 11:49:48 goliathbox kernel: DMA per-cpu:
> Dec 14 11:49:48 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
> Dec 14 11:49:49 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
> Dec 14 11:49:49 goliathbox kernel: Normal per-cpu:
> Dec 14 11:49:49 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
> Dec 14 11:49:49 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
> Dec 14 11:49:49 goliathbox kernel: HighMem per-cpu: empty
> Dec 14 11:49:50 goliathbox crond(pam_unix)[5390]: session closed for user root
> Dec 14 11:49:50 goliathbox kernel:
> Dec 14 11:49:50 goliathbox crond(pam_unix)[5432]: session closed for user root
> Dec 14 11:49:50 goliathbox kernel: Free pages:         676kB (0kB HighMem)
> Dec 14 11:49:51 goliathbox kernel: Active:31292 inactive:88792 dirty:0 writeback:0 unstable:0 free:169 slab:3717 mapped:118877 pagetables:1729
> Dec 14 11:49:51 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4412kB inactive:8044kB present:16384kB pages_scanned:20955 all_unreclaimable? yes
> Dec 14 11:49:51 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:51 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:120756kB inactive:347124kB present:507820kB pages_scanned:578655 all_unreclaimable? yesDec 14 11:49:51 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:51 goliathbox crond(pam_unix)[5434]: session closed for user root
> Dec 14 11:49:51 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
> Dec 14 11:49:52 goliathbox crond(pam_unix)[5433]: session closed for user sm0ketst
> Dec 14 11:49:52 goliathbox kernel: protections[]: 0 0 0
> Dec 14 11:49:52 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
> Dec 14 11:49:52 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
> Dec 14 11:49:52 goliathbox kernel: HighMem: empty
> Dec 14 11:49:52 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
> Dec 14 11:49:52 goliathbox kernel: 0 bounce buffer pages
> Dec 14 11:49:53 goliathbox kernel: Free swap:            0kB
> Dec 14 11:49:53 goliathbox kernel: 131051 pages of RAM
> Dec 14 11:49:53 goliathbox kernel: 0 pages of HIGHMEM
> Dec 14 11:49:53 goliathbox kernel: 2433 reserved pages
> Dec 14 11:49:53 goliathbox kernel: 26379 pages shared
> Dec 14 11:49:53 goliathbox kernel: 513 pages swap cached
> Dec 14 11:49:53 goliathbox kernel: Out of Memory: Killed process 5129 (nautilus).
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

-- 
My "Foundation" verse:
Isa 54:17  No weapon that is formed against thee shall prosper; and 
every tongue that shall rise against thee in judgment thou shalt 
condemn. This is the heritage of the servants of the LORD, and their 
righteousness is of me, saith the LORD.

-- carpe ductum -- "Grab the tape"
CDTT (Certified Duct Tape Technician)

Linux user #322099
Machines:
206822
256638
276825
http://counter.li.org/

More information about the CentOS mailing list