14 Dec
2005
14 Dec
'05
11:13 a.m.
Hi all,
Just watching for my first time the 'oom-killer' call in a CentOS 4.2 workstation. System was running normally but with high load of nice process, iow, rosetta and seti, others appart, nautilus, etc... i realized that after a night of standby running normally, the next moorning after few manual operations from X-session in local console, it started a massive swapping, this time not recovered as previously other time, finishing in a oom-killer call.
Finally inspected the syslog and realize the oom-killer was called to leave the system stable having killed firefox-bin, 3 x rosetta, xmms and one nautilus.
It's a PIII 800 Mhz, 40GB IDE, 512 Mb RAM, 1024 Swap space, up for 4 days.
I'm not very xperienced in tunning the memmory system nor the swap system, help appreciated to avoid this memory 'trashing' situtation (14 virtual desktops openned at the same time).
Is it a normal situation? Is it a correct protecction measure? It's due a low bandwitch IDE HD? how can i tune it to avoid this situations? is it a high load for this workstation? I think the oom-killer was called correctly to avoid the system collapsed, but i think there was a problem in the swapping subsystem or a memmory trashing situation to avoid.
Help and hints appreciated,
Attached you'll the oom-killer log,
TIA
J.J.
--
Linux smarteyebox.stigmatedbrain.net 2.6.9-22.0.1.EL i686 GNU/Linux
11:55:01 up 1 day, 12:06, 9 users, load average: 1.20, 1.30, 1.57
---------------------------------------------------------
"Rome wasn't burned in a day. "
Dec 14 11:48:00 goliathbox kernel: oom-killer: gfp_mask=0xd0
Dec 14 11:48:00 goliathbox kernel: Mem-info:
Dec 14 11:48:00 goliathbox kernel: DMA per-cpu:
Dec 14 11:48:00 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 14 11:48:00 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 14 11:48:00 goliathbox kernel: Normal per-cpu:
Dec 14 11:48:00 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 14 11:48:00 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 14 11:48:00 goliathbox kernel: HighMem per-cpu: empty
Dec 14 11:48:00 goliathbox kernel:
Dec 14 11:48:00 goliathbox kernel: Free pages: 676kB (0kB HighMem)
Dec 14 11:48:00 goliathbox kernel: Active:109642 inactive:10334 dirty:0 writeback:0 unstable:0 free:169 slab:3746 mapped:118866 pagetables:1729
Dec 14 11:48:00 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:6132kB inactive:6316kB present:16384kB pages_scanned:20658 all_unreclaimable? yes
Dec 14 11:48:02 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:48:23 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:432436kB inactive:35020kB present:507820kB pages_scanned:1102915 all_unreclaimable? yesDec 14 11:48:23 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:48:23 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 14 11:48:23 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:48:23 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Dec 14 11:48:23 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
Dec 14 11:48:23 goliathbox kernel: HighMem: empty
Dec 14 11:48:23 goliathbox kernel: Swap cache: add 974772, delete 974259, find 292228/382858, race 0+15
Dec 14 11:48:23 goliathbox kernel: 0 bounce buffer pages
Dec 14 11:48:23 goliathbox kernel: Free swap: 0kB
Dec 14 11:48:23 goliathbox kernel: 131051 pages of RAM
Dec 14 11:48:23 goliathbox kernel: 0 pages of HIGHMEM
Dec 14 11:48:42 goliathbox kernel: 2433 reserved pages
Dec 14 11:49:26 goliathbox kernel: 26433 pages shared
Dec 14 11:49:26 goliathbox kernel: 513 pages swap cached
Dec 14 11:49:26 goliathbox kernel: Out of Memory: Killed process 13954 (firefox-bin).
Dec 14 11:49:26 goliathbox kernel: oom-killer: gfp_mask=0x1d2
Dec 14 11:49:26 goliathbox kernel: Mem-info:
Dec 14 11:49:26 goliathbox kernel: DMA per-cpu:
Dec 14 11:49:26 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 14 11:49:26 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 14 11:49:26 goliathbox kernel: Normal per-cpu:
Dec 14 11:49:26 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 14 11:49:26 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 14 11:49:26 goliathbox kernel: HighMem per-cpu: empty
Dec 14 11:49:26 goliathbox kernel:
Dec 14 11:49:27 goliathbox kernel: Free pages: 676kB (0kB HighMem)
Dec 14 11:49:27 goliathbox kernel: Active:1868 inactive:118396 dirty:0 writeback:0 unstable:0 free:169 slab:3738 mapped:118869 pagetables:1729
Dec 14 11:49:27 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4232kB inactive:7992kB present:16384kB pages_scanned:20064 all_unreclaimable? yes
Dec 14 11:49:28 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:28 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:3240kB inactive:465592kB present:507820kB pages_scanned:780089 all_unreclaimable? yes
Dec 14 11:49:28 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:29 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 14 11:49:29 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:29 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Dec 14 11:49:29 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
Dec 14 11:49:29 goliathbox kernel: HighMem: empty
Dec 14 11:49:29 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
Dec 14 11:49:30 goliathbox kernel: 0 bounce buffer pages
Dec 14 11:49:30 goliathbox kernel: Free swap: 0kB
Dec 14 11:49:30 goliathbox kernel: 131051 pages of RAM
Dec 14 11:49:30 goliathbox kernel: 0 pages of HIGHMEM
Dec 14 11:49:31 goliathbox kernel: 2433 reserved pages
Dec 14 11:49:31 goliathbox kernel: 26140 pages shared
Dec 14 11:49:31 goliathbox kernel: 513 pages swap cached
Dec 14 11:49:31 goliathbox kernel: Out of Memory: Killed process 15771 (firefox-bin).
Dec 14 11:49:31 goliathbox kernel: oom-killer: gfp_mask=0xd0
Dec 14 11:49:31 goliathbox kernel: Mem-info:
Dec 14 11:49:32 goliathbox kernel: DMA per-cpu:
Dec 14 11:49:32 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 14 11:49:32 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 14 11:49:32 goliathbox kernel: Normal per-cpu:
Dec 14 11:49:32 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 14 11:49:32 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 14 11:49:32 goliathbox kernel: HighMem per-cpu: empty
Dec 14 11:49:33 goliathbox kernel:
Dec 14 11:49:33 goliathbox kernel: Free pages: 676kB (0kB HighMem)
Dec 14 11:49:33 goliathbox kernel: Active:73675 inactive:46271 dirty:0 writeback:0 unstable:0 free:169 slab:3735 mapped:118871 pagetables:1729
Dec 14 11:49:33 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4384kB inactive:7968kB present:16384kB pages_scanned:20229 all_unreclaimable? yes
Dec 14 11:49:33 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:33 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:290316kB inactive:177116kB present:507820kB pages_scanned:556644 all_unreclaimable? yesDec 14 11:49:34 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:34 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 14 11:49:34 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:34 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Dec 14 11:49:34 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
Dec 14 11:49:34 goliathbox kernel: HighMem: empty
Dec 14 11:49:35 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
Dec 14 11:49:35 goliathbox kernel: 0 bounce buffer pages
Dec 14 11:49:35 goliathbox kernel: Free swap: 0kB
Dec 14 11:49:35 goliathbox kernel: 131051 pages of RAM
Dec 14 11:49:35 goliathbox kernel: 0 pages of HIGHMEM
Dec 14 11:49:35 goliathbox kernel: 2433 reserved pages
Dec 14 11:49:36 goliathbox kernel: 26474 pages shared
Dec 14 11:49:36 goliathbox kernel: 513 pages swap cached
Dec 14 11:49:36 goliathbox kernel: Out of Memory: Killed process 13973 (rosetta_4.79_i6).
Dec 14 11:49:36 goliathbox kernel: Out of Memory: Killed process 13972 (rosetta_4.79_i6).
Dec 14 11:49:36 goliathbox kernel: Out of Memory: Killed process 13974 (rosetta_4.79_i6).
Dec 14 11:49:36 goliathbox kernel: oom-killer: gfp_mask=0x1d2
Dec 14 11:49:36 goliathbox kernel: Mem-info:
Dec 14 11:49:37 goliathbox kernel: DMA per-cpu:
Dec 14 11:49:38 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 14 11:49:38 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 14 11:49:39 goliathbox kernel: Normal per-cpu:
Dec 14 11:49:39 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 14 11:49:40 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 14 11:49:40 goliathbox kernel: HighMem per-cpu: empty
Dec 14 11:49:40 goliathbox kernel:
Dec 14 11:49:41 goliathbox kernel: Free pages: 676kB (0kB HighMem)
Dec 14 11:49:41 goliathbox kernel: Active:5356 inactive:114646 dirty:0 writeback:0 unstable:0 free:169 slab:3735 mapped:118875 pagetables:1729
Dec 14 11:49:41 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4412kB inactive:8044kB present:16384kB pages_scanned:20361 all_unreclaimable? yes
Dec 14 11:49:41 goliathbox crond(pam_unix)[5432]: session opened for user root by (uid=0)
Dec 14 11:49:41 goliathbox crond(pam_unix)[5434]: session opened for user root by (uid=0)
Dec 14 11:49:41 goliathbox crond(pam_unix)[5433]: session opened for user sm0ketst by (uid=0)
Dec 14 11:49:41 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:42 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:17012kB inactive:450540kB present:507820kB pages_scanned:2398502 all_unreclaimable? yesDec 14 11:49:43 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:43 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 14 11:49:43 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:44 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Dec 14 11:49:44 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
Dec 14 11:49:44 goliathbox kernel: HighMem: empty
Dec 14 11:49:44 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
Dec 14 11:49:45 goliathbox kernel: 0 bounce buffer pages
Dec 14 11:49:45 goliathbox kernel: Free swap: 0kB
Dec 14 11:49:46 goliathbox kernel: 131051 pages of RAM
Dec 14 11:49:46 goliathbox kernel: 0 pages of HIGHMEM
Dec 14 11:49:46 goliathbox kernel: 2433 reserved pages
Dec 14 11:49:46 goliathbox kernel: 26419 pages shared
Dec 14 11:49:47 goliathbox kernel: 513 pages swap cached
Dec 14 11:49:47 goliathbox kernel: Out of Memory: Killed process 4936 (xmms).
Dec 14 11:49:47 goliathbox kernel: oom-killer: gfp_mask=0x1d2
Dec 14 11:49:48 goliathbox kernel: Mem-info:
Dec 14 11:49:48 goliathbox kernel: DMA per-cpu:
Dec 14 11:49:48 goliathbox kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 14 11:49:49 goliathbox kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 14 11:49:49 goliathbox kernel: Normal per-cpu:
Dec 14 11:49:49 goliathbox kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 14 11:49:49 goliathbox kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 14 11:49:49 goliathbox kernel: HighMem per-cpu: empty
Dec 14 11:49:50 goliathbox crond(pam_unix)[5390]: session closed for user root
Dec 14 11:49:50 goliathbox kernel:
Dec 14 11:49:50 goliathbox crond(pam_unix)[5432]: session closed for user root
Dec 14 11:49:50 goliathbox kernel: Free pages: 676kB (0kB HighMem)
Dec 14 11:49:51 goliathbox kernel: Active:31292 inactive:88792 dirty:0 writeback:0 unstable:0 free:169 slab:3717 mapped:118877 pagetables:1729
Dec 14 11:49:51 goliathbox kernel: DMA free:20kB min:20kB low:40kB high:60kB active:4412kB inactive:8044kB present:16384kB pages_scanned:20955 all_unreclaimable? yes
Dec 14 11:49:51 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:51 goliathbox kernel: Normal free:656kB min:700kB low:1400kB high:2100kB active:120756kB inactive:347124kB present:507820kB pages_scanned:578655 all_unreclaimable? yesDec 14 11:49:51 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:51 goliathbox crond(pam_unix)[5434]: session closed for user root
Dec 14 11:49:51 goliathbox kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Dec 14 11:49:52 goliathbox crond(pam_unix)[5433]: session closed for user sm0ketst
Dec 14 11:49:52 goliathbox kernel: protections[]: 0 0 0
Dec 14 11:49:52 goliathbox kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Dec 14 11:49:52 goliathbox kernel: Normal: 0*4kB 6*8kB 2*16kB 4*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 656kB
Dec 14 11:49:52 goliathbox kernel: HighMem: empty
Dec 14 11:49:52 goliathbox kernel: Swap cache: add 974851, delete 974338, find 292235/382874, race 0+15
Dec 14 11:49:52 goliathbox kernel: 0 bounce buffer pages
Dec 14 11:49:53 goliathbox kernel: Free swap: 0kB
Dec 14 11:49:53 goliathbox kernel: 131051 pages of RAM
Dec 14 11:49:53 goliathbox kernel: 0 pages of HIGHMEM
Dec 14 11:49:53 goliathbox kernel: 2433 reserved pages
Dec 14 11:49:53 goliathbox kernel: 26379 pages shared
Dec 14 11:49:53 goliathbox kernel: 513 pages swap cached
Dec 14 11:49:53 goliathbox kernel: Out of Memory: Killed process 5129 (nautilus).