I am getting this message quite often lately.
Centos 5.3, AMD dual core 5050e system x64
1 GIG ram, 4 GIG swap
I dont have that much running that the kernel should be cutting out my processes.
Any thoughts?
Jerry
------------
emTotal: 766264 kB MemFree: 583984 kB Buffers: 9572 kB Cached: 31004 kB SwapCached: 36280 kB Active: 20240 kB Inactive: 59780 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 766264 kB LowFree: 583984 kB SwapTotal: 4096564 kB SwapFree: 3932136 kB Dirty: 12 kB Writeback: 0 kB AnonPages: 13184 kB Mapped: 16340 kB Slab: 22428 kB PageTables: 13980 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 4479696 kB Committed_AS: 442124 kB VmallocTotal: 34359738367 kB VmallocUsed: 280856 kB VmallocChunk: 34359457495 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kB
096kB = 3424kB Node 0 Normal: empty Node 0 HighMem: empty 46 pagecache pages Swap cache: add 8934611, delete 8934611, find 7869746/7929974, race 11+11 Free swap = 0kB Total swap = 4096564kB Free swap: 0kB 196064 pages of RAM 13511 reserved pages 5948 pages shared 0 pages swap cached kthread invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
Call Trace: [<ffffffff800c3a6a>] out_of_memory+0x8e/0x2f5 [<ffffffff8000f2eb>] __alloc_pages+0x245/0x2ce [<ffffffff8002303f>] alloc_page_interleave+0x3d/0x74 [<ffffffff8003c061>] __get_free_pages+0xe/0x71 [<ffffffff8001eb82>] copy_process+0xc6/0x15b8 [<ffffffff8009b8bc>] alloc_pid+0x1ee/0x28a [<ffffffff80030cdb>] do_fork+0x69/0x1c1 [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4 [<ffffffff8005df3d>] kernel_thread+0x81/0xeb [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4 [<ffffffff80032282>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 [<ffffffff8009d9a9>] keventd_create_kthread+0x1d/0xc4 [<ffffffff8004d159>] run_workqueue+0x94/0xe4 [<ffffffff800499da>] worker_thread+0x0/0x122 [<ffffffff80049aca>] worker_thread+0xf0/0x122 [<ffffffff8008a4b3>] default_wake_function+0x0/0xe [<ffffffff80032380>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff80032282>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11
Mem-info: Node 0 DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 cpu 1 hot: high 0, batch 1 used:0 cpu 1 cold: high 0, batch 1 used:0 Node 0 DMA32 per-cpu: cpu 0 hot: high 186, batch 31 used:146 cpu 0 cold: high 62, batch 15 used:47 cpu 1 hot: high 186, batch 31 used:18 cpu 1 cold: high 62, batch 15 used:55 Node 0 Normal per-cpu: empty Node 0 HighMem per-cpu: empty Free pages: 5652kB (0kB HighMem) Active:78394 inactive:78664 dirty:0 writeback:0 unstable:0 free:1413 slab:5343 mapped-file:6627 mapped-anon:157109 pagetables:5859 Node 0 DMA free:2160kB min:48kB low:60kB high:72kB active:0kB inactive:0kB present:10724kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 739 739 739 Node 0 DMA32 free:3492kB min:3452kB low:4312kB high:5176kB active:313576kB inactive:314656kB present:757376kB pages_scanned:1226904 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 4*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2160kB Node 0 DMA32: 19*4kB 11*8kB 0*16kB 0*32kB 8*64kB 2*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3492kB Node 0 Normal: empty Node 0 HighMem: empty 45 pagecache pages Swap cache: add 8934620, delete 8934620, find 7869746/7929974, race 11+11 Free swap = 0kB Total swap = 4096564kB Free swap: 0kB 196064 pages of RAM 13511 reserved pages 4950 pages shared 0 pages swap cached
On Wed, Jul 29, 2009 at 4:30 PM, Jerry Geisgeisj@pagestation.com wrote:
I am getting this message quite often lately.
Centos 5.3, AMD dual core 5050e system x64
1 GIG ram, 4 GIG swap
Well that is a lot of swap for a server.. if something really starts using that much swap its going to most likely end up in a bad race where you are swapping out more stuff than you are getting in. For most large memory systems my rule of thumb is no more than 1:1 on swap. Adding to 1 to 3 more GB of memory can fix some of these issues sometimes because you have more room to stuff things in when trying to swap stuff in.
Free swap = 0kB Total swap = 4096564kB Free swap: 0kB
Something is eating up swap. Not sure what it is from the message.. so you are going to need to look in using some sort of script to track whats going on and see if you can find out what is doing it.
Jerry Geis wrote on Wed, 29 Jul 2009 18:30:05 -0400:
Any thoughts?
Doesn't top help in finding out what's eating your RAM?
Kai