I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? Is it about some other memory? highmem? lowmem? stack size?
Best Regards,
Kernel 3.10.24
Dec 27 09:19:05 2013 kernel: : [277622.359064] squid invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0 Dec 27 09:19:05 2013 kernel: : [277622.359069] squid cpuset=/ mems_allowed=0 Dec 27 09:19:05 2013 kernel: : [277622.359074] CPU: 9 PID: 15533 Comm: squid Not tainted 3.10.24-1.lsg #1 Dec 27 09:19:05 2013 kernel: : [277622.359076] Hardware name: Intel Thurley/Greencity, BIOS 080016 10/05/2011 Dec 27 09:19:05 2013 kernel: : [277622.359078] 00000003 e377b280 e03c3c38 c06472d6 e03c3c98 c04d2d96 c0a68f84 e377b580 Dec 27 09:19:05 2013 kernel: : [277622.359089] 000042d0 00000003 00000000 e03c3c64 c04abbda e42bd318 00000000 e03c3cf4 Dec 27 09:19:05 2013 kernel: : [277622.359096] 000042d0 00000001 00000247 00000000 e03c3c94 c04d3d5f 00000001 00000042 Dec 27 09:19:05 2013 kernel: : [277622.359105] Call Trace: Dec 27 09:19:05 2013 kernel: : [277622.359116] [<c06472d6>] dump_stack+0x16/0x20 Dec 27 09:19:05 2013 kernel: : [277622.359121] [<c04d2d96>] dump_header+0x66/0x1c0 Dec 27 09:19:05 2013 kernel: : [277622.359127] [<c04abbda>] ? __delayacct_freepages_end+0x3a/0x40 Dec 27 09:19:05 2013 kernel: : [277622.359131] [<c04d3d5f>] ? zone_watermark_ok+0x2f/0x40 Dec 27 09:19:05 2013 kernel: : [277622.359135] [<c04d2f27>] check_panic_on_oom+0x37/0x60 Dec 27 09:19:05 2013 kernel: : [277622.359138] [<c04d36d2>] out_of_memory+0x92/0x250 Dec 27 09:19:05 2013 kernel: : [277622.359144] [<c04dd1fa>] ? wakeup_kswapd+0xda/0x120 Dec 27 09:19:05 2013 kernel: : [277622.359148] [<c04d6cee>] __alloc_pages_nodemask+0x68e/0x6a0 Dec 27 09:19:05 2013 kernel: : [277622.359155] [<c0801c1e>] sk_page_frag_refill+0x7e/0x120 Dec 27 09:19:05 2013 kernel: : [277622.359160] [<c084b8c7>] tcp_sendmsg+0x387/0xbf0 Dec 27 09:19:05 2013 kernel: : [277622.359166] [<c0469a2f>] ? put_prev_task_fair+0x1f/0x350 Dec 27 09:19:05 2013 kernel: : [277622.359173] [<c0ba7d8b>] ? longrun_init+0x2b/0x30 Dec 27 09:19:05 2013 kernel: : [277622.359177] [<c084b540>] ? tcp_tso_segment+0x380/0x380 Dec 27 09:19:05 2013 kernel: : [277622.359182] [<c086d0da>] inet_sendmsg+0x4a/0xa0 Dec 27 09:19:05 2013 kernel: : [277622.359186] [<c07ff3a6>] sock_aio_write+0x116/0x130 Dec 27 09:19:05 2013 kernel: : [277622.359191] [<c0457acc>] ? hrtimer_try_to_cancel+0x3c/0xb0 Dec 27 09:19:05 2013 kernel: : [277622.359197] [<c050b208>] do_sync_write+0x68/0xa0 Dec 27 09:19:05 2013 kernel: : [277622.359202] [<c050caa0>] vfs_write+0x190/0x1b0 Dec 27 09:19:05 2013 kernel: : [277622.359206] [<c050cbb3>] SyS_write+0x53/0x80 Dec 27 09:19:05 2013 kernel: : [277622.359211] [<c08f72ba>] sysenter_do_call+0x12/0x22 Dec 27 09:19:05 2013 kernel: : [277622.359213] Mem-Info: Dec 27 09:19:05 2013 kernel: : [277622.359215] DMA per-cpu: Dec 27 09:19:05 2013 kernel: : [277622.359218] CPU 0: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359220] CPU 1: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359222] CPU 2: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359224] CPU 3: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359226] CPU 4: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359228] CPU 5: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359230] CPU 6: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359232] CPU 7: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359234] CPU 8: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359236] CPU 9: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359238] CPU 10: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359240] CPU 11: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359242] CPU 12: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359244] CPU 13: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359246] CPU 14: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359248] CPU 15: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359250] CPU 16: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359253] CPU 17: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359255] CPU 18: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359258] CPU 19: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359260] CPU 20: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359262] CPU 21: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359264] CPU 22: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359266] CPU 23: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359268] Normal per-cpu: Dec 27 09:19:05 2013 kernel: : [277622.359270] CPU 0: hi: 186, btch: 31 usd: 34 Dec 27 09:19:05 2013 kernel: : [277622.359272] CPU 1: hi: 186, btch: 31 usd: 72 Dec 27 09:19:05 2013 kernel: : [277622.359274] CPU 2: hi: 186, btch: 31 usd: 40 Dec 27 09:19:05 2013 kernel: : [277622.359276] CPU 3: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359279] CPU 4: hi: 186, btch: 31 usd: 39 Dec 27 09:19:05 2013 kernel: : [277622.359281] CPU 5: hi: 186, btch: 31 usd: 49 Dec 27 09:19:05 2013 kernel: : [277622.359283] CPU 6: hi: 186, btch: 31 usd: 50 Dec 27 09:19:05 2013 kernel: : [277622.359285] CPU 7: hi: 186, btch: 31 usd: 25 Dec 27 09:19:05 2013 kernel: : [277622.359286] CPU 8: hi: 186, btch: 31 usd: 42 Dec 27 09:19:05 2013 kernel: : [277622.359289] CPU 9: hi: 186, btch: 31 usd: 39 Dec 27 09:19:05 2013 kernel: : [277622.359290] CPU 10: hi: 186, btch: 31 usd: 155 Dec 27 09:19:05 2013 kernel: : [277622.359293] CPU 11: hi: 186, btch: 31 usd: 56 Dec 27 09:19:05 2013 kernel: : [277622.359295] CPU 12: hi: 186, btch: 31 usd: 2 Dec 27 09:19:05 2013 kernel: : [277622.359297] CPU 13: hi: 186, btch: 31 usd: 162 Dec 27 09:19:05 2013 kernel: : [277622.359299] CPU 14: hi: 186, btch: 31 usd: 67 Dec 27 09:19:05 2013 kernel: : [277622.359301] CPU 15: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359303] CPU 16: hi: 186, btch: 31 usd: 68 Dec 27 09:19:05 2013 kernel: : [277622.359305] CPU 17: hi: 186, btch: 31 usd: 38 Dec 27 09:19:05 2013 kernel: : [277622.359307] CPU 18: hi: 186, btch: 31 usd: 56 Dec 27 09:19:05 2013 kernel: : [277622.359308] CPU 19: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359310] CPU 20: hi: 186, btch: 31 usd: 54 Dec 27 09:19:05 2013 kernel: : [277622.359312] CPU 21: hi: 186, btch: 31 usd: 35 Dec 27 09:19:05 2013 kernel: : [277622.359314] CPU 22: hi: 186, btch: 31 usd: 2 Dec 27 09:19:05 2013 kernel: : [277622.359316] CPU 23: hi: 186, btch: 31 usd: 60 Dec 27 09:19:05 2013 kernel: : [277622.359318] HighMem per-cpu: Dec 27 09:19:05 2013 kernel: : [277622.359320] CPU 0: hi: 186, btch: 31 usd: 32 Dec 27 09:19:05 2013 kernel: : [277622.359322] CPU 1: hi: 186, btch: 31 usd: 52 Dec 27 09:19:05 2013 kernel: : [277622.359324] CPU 2: hi: 186, btch: 31 usd: 9 Dec 27 09:19:05 2013 kernel: : [277622.359326] CPU 3: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359328] CPU 4: hi: 186, btch: 31 usd: 125 Dec 27 09:19:05 2013 kernel: : [277622.359330] CPU 5: hi: 186, btch: 31 usd: 116 Dec 27 09:19:05 2013 kernel: : [277622.359332] CPU 6: hi: 186, btch: 31 usd: 126 Dec 27 09:19:05 2013 kernel: : [277622.359333] CPU 7: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359336] CPU 8: hi: 186, btch: 31 usd: 79 Dec 27 09:19:05 2013 kernel: : [277622.359338] CPU 9: hi: 186, btch: 31 usd: 34 Dec 27 09:19:05 2013 kernel: : [277622.359340] CPU 10: hi: 186, btch: 31 usd: 111 Dec 27 09:19:05 2013 kernel: : [277622.359341] CPU 11: hi: 186, btch: 31 usd: 144 Dec 27 09:19:05 2013 kernel: : [277622.359343] CPU 12: hi: 186, btch: 31 usd: 15 Dec 27 09:19:05 2013 kernel: : [277622.359345] CPU 13: hi: 186, btch: 31 usd: 166 Dec 27 09:19:05 2013 kernel: : [277622.359347] CPU 14: hi: 186, btch: 31 usd: 185 Dec 27 09:19:05 2013 kernel: : [277622.359349] CPU 15: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359351] CPU 16: hi: 186, btch: 31 usd: 58 Dec 27 09:19:05 2013 kernel: : [277622.359353] CPU 17: hi: 186, btch: 31 usd: 122 Dec 27 09:19:05 2013 kernel: : [277622.359356] CPU 18: hi: 186, btch: 31 usd: 170 Dec 27 09:19:05 2013 kernel: : [277622.359358] CPU 19: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359360] CPU 20: hi: 186, btch: 31 usd: 30 Dec 27 09:19:05 2013 kernel: : [277622.359362] CPU 21: hi: 186, btch: 31 usd: 33 Dec 27 09:19:05 2013 kernel: : [277622.359364] CPU 22: hi: 186, btch: 31 usd: 28 Dec 27 09:19:05 2013 kernel: : [277622.359366] CPU 23: hi: 186, btch: 31 usd: 44 Dec 27 09:19:05 2013 kernel: : [277622.359371] active_anon:658515 inactive_anon:54399 isolated_anon:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] active_file:1172176 inactive_file:323606 isolated_file:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] unevictable:0 dirty:0 writeback:0 unstable:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] free:6911872 slab_reclaimable:29430 slab_unreclaimable:34726 Dec 27 09:19:05 2013 kernel: : [277622.359371] mapped:45784 shmem:9850 pagetables:107714 bounce:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] free_cma:0 Dec 27 09:19:05 2013 kernel: : [277622.359382] DMA free:2332kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15968kB managed:6960kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:288kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Dec 27 09:19:05 2013 kernel: : [277622.359384] lowmem_reserve[]: 0 573 36539 36539 Dec 27 09:19:05 2013 kernel: : [277622.359393] Normal free:114488kB min:3044kB low:3804kB high:4564kB active_anon:0kB inactive_anon:0kB active_file:252kB inactive_file:256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:894968kB managed:587540kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:117712kB slab_unreclaimable:138616kB kernel_stack:11976kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:982 all_unreclaimable? yes Dec 27 09:19:05 2013 kernel: : [277622.359395] lowmem_reserve[]: 0 0 287725 287725 Dec 27 09:19:05 2013 kernel: : [277622.359404] HighMem free:27530668kB min:512kB low:48272kB high:96036kB active_anon:2634060kB inactive_anon:217596kB active_file:4688452kB inactive_file:1294168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:36828872kB managed:36828872kB mlocked:0kB dirty:0kB writeback:0kB mapped:183132kB shmem:39400kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:430856kB unstable:0kB bounce:367564104kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Dec 27 09:19:05 2013 kernel: : [277622.359406] lowmem_reserve[]: 0 0 0 0 Dec 27 09:19:05 2013 kernel: : [277622.359410] DMA: 3*4kB (U) 2*8kB (U) 4*16kB (U) 5*32kB (U) 2*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 2428kB Dec 27 09:19:05 2013 kernel: : [277622.359422] Normal: 5360*4kB (UEM) 3667*8kB (UEM) 3964*16kB (UEMR) 13*32kB (MR) 0*64kB 1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115000kB Dec 27 09:19:05 2013 kernel: : [277622.359435] HighMem: 6672*4kB (M) 74585*8kB (UM) 40828*16kB (UM) 17275*32kB (UM) 3314*64kB (UM) 1126*128kB (UM) 992*256kB (UM) 585*512kB (UM) 225*1024kB (UM) 78*2048kB (UMR) 5957*4096kB (UM) = 27529128kB Dec 27 09:19:05 2013 kernel: : [277622.359452] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Dec 27 09:19:05 2013 kernel: : [277622.359454] 1505509 total pagecache pages Dec 27 09:19:05 2013 kernel: : [277622.359457] 4 pages in swap cache Dec 27 09:19:05 2013 kernel: : [277622.359459] Swap cache stats: add 13, delete 9, find 0/0 Dec 27 09:19:05 2013 kernel: : [277622.359460] Free swap = 35318812kB Dec 27 09:19:05 2013 kernel: : [277622.359462] Total swap = 35318864kB Dec 27 09:19:05 2013 kernel: : [277622.450529] 9699327 pages RAM Dec 27 09:19:05 2013 kernel: : [277622.450532] 9471490 pages HighMem Dec 27 09:19:05 2013 kernel: : [277622.450533] 342749 pages reserved Dec 27 09:19:05 2013 kernel: : [277622.450534] 2864256 pages shared Dec 27 09:19:05 2013 kernel: : [277622.450535] 1501243 pages non-shared Dec 27 09:19:05 2013 kernel: : [277622.450538] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
Dec 27 09:19:05 2013 kernel: : [277622.450538]
# cat /proc/meminfo MemTotal: 37426312 kB MemFree: 28328992 kB Buffers: 94728 kB Cached: 6216068 kB SwapCached: 0 kB Active: 6958572 kB Inactive: 1815380 kB Active(anon): 2329152 kB Inactive(anon): 170252 kB Active(file): 4629420 kB Inactive(file): 1645128 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 36828872 kB HighFree: 28076144 kB LowTotal: 597440 kB LowFree: 252848 kB SwapTotal: 35318864 kB SwapFree: 35318860 kB Dirty: 0 kB Writeback: 8 kB AnonPages: 2463512 kB Mapped: 162296 kB Shmem: 36332 kB Slab: 208676 kB SReclaimable: 120872 kB SUnreclaim: 87804 kB KernelStack: 6320 kB PageTables: 42280 kB NFS_Unstable: 0 kB Bounce: 124 kB WritebackTmp: 0 kB CommitLimit: 54032020 kB Committed_AS: 3191916 kB VmallocTotal: 122880 kB VmallocUsed: 27088 kB VmallocChunk: 29312 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 10232 kB DirectMap2M: 901120 kB
sysctl:
vm.oom_dump_tasks = 0 vm.oom_kill_allocating_task = 1 vm.panic_on_oom = 1
vm.admin_reserve_kbytes = 8192 vm.block_dump = 0 vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.drop_caches = 0 vm.highmem_is_dirtyable = 0 vm.hugepages_treat_as_movable = 0 vm.hugetlb_shm_group = 0 vm.laptop_mode = 0 vm.legacy_va_layout = 0 vm.lowmem_reserve_ratio = 256 32 32 vm.max_map_count = 65530 vm.min_free_kbytes = 3084 vm.mmap_min_addr = 4096 vm.nr_hugepages = 0 vm.nr_overcommit_hugepages = 0 vm.nr_pdflush_threads = 0 vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.percpu_pagelist_fraction = 0 vm.scan_unevictable_pages = 0 vm.stat_interval = 1 vm.swappiness = 30 vm.user_reserve_kbytes = 131072 vm.vdso_enabled = 1 vm.vfs_cache_pressure = 100
# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 292370 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 36728 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 292370 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Try tuning squid memory settings? Are you running latest kernel on centos? 30.12.2013 10.03 kirjoitti "Oguz Yilmaz" oguzyilmazlist@gmail.com:
I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? Is it about some other memory? highmem? lowmem? stack size?
Best Regards,
Kernel 3.10.24
Dec 27 09:19:05 2013 kernel: : [277622.359064] squid invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0 Dec 27 09:19:05 2013 kernel: : [277622.359069] squid cpuset=/ mems_allowed=0 Dec 27 09:19:05 2013 kernel: : [277622.359074] CPU: 9 PID: 15533 Comm: squid Not tainted 3.10.24-1.lsg #1 Dec 27 09:19:05 2013 kernel: : [277622.359076] Hardware name: Intel Thurley/Greencity, BIOS 080016 10/05/2011 Dec 27 09:19:05 2013 kernel: : [277622.359078] 00000003 e377b280 e03c3c38 c06472d6 e03c3c98 c04d2d96 c0a68f84 e377b580 Dec 27 09:19:05 2013 kernel: : [277622.359089] 000042d0 00000003 00000000 e03c3c64 c04abbda e42bd318 00000000 e03c3cf4 Dec 27 09:19:05 2013 kernel: : [277622.359096] 000042d0 00000001 00000247 00000000 e03c3c94 c04d3d5f 00000001 00000042 Dec 27 09:19:05 2013 kernel: : [277622.359105] Call Trace: Dec 27 09:19:05 2013 kernel: : [277622.359116] [<c06472d6>] dump_stack+0x16/0x20 Dec 27 09:19:05 2013 kernel: : [277622.359121] [<c04d2d96>] dump_header+0x66/0x1c0 Dec 27 09:19:05 2013 kernel: : [277622.359127] [<c04abbda>] ? __delayacct_freepages_end+0x3a/0x40 Dec 27 09:19:05 2013 kernel: : [277622.359131] [<c04d3d5f>] ? zone_watermark_ok+0x2f/0x40 Dec 27 09:19:05 2013 kernel: : [277622.359135] [<c04d2f27>] check_panic_on_oom+0x37/0x60 Dec 27 09:19:05 2013 kernel: : [277622.359138] [<c04d36d2>] out_of_memory+0x92/0x250 Dec 27 09:19:05 2013 kernel: : [277622.359144] [<c04dd1fa>] ? wakeup_kswapd+0xda/0x120 Dec 27 09:19:05 2013 kernel: : [277622.359148] [<c04d6cee>] __alloc_pages_nodemask+0x68e/0x6a0 Dec 27 09:19:05 2013 kernel: : [277622.359155] [<c0801c1e>] sk_page_frag_refill+0x7e/0x120 Dec 27 09:19:05 2013 kernel: : [277622.359160] [<c084b8c7>] tcp_sendmsg+0x387/0xbf0 Dec 27 09:19:05 2013 kernel: : [277622.359166] [<c0469a2f>] ? put_prev_task_fair+0x1f/0x350 Dec 27 09:19:05 2013 kernel: : [277622.359173] [<c0ba7d8b>] ? longrun_init+0x2b/0x30 Dec 27 09:19:05 2013 kernel: : [277622.359177] [<c084b540>] ? tcp_tso_segment+0x380/0x380 Dec 27 09:19:05 2013 kernel: : [277622.359182] [<c086d0da>] inet_sendmsg+0x4a/0xa0 Dec 27 09:19:05 2013 kernel: : [277622.359186] [<c07ff3a6>] sock_aio_write+0x116/0x130 Dec 27 09:19:05 2013 kernel: : [277622.359191] [<c0457acc>] ? hrtimer_try_to_cancel+0x3c/0xb0 Dec 27 09:19:05 2013 kernel: : [277622.359197] [<c050b208>] do_sync_write+0x68/0xa0 Dec 27 09:19:05 2013 kernel: : [277622.359202] [<c050caa0>] vfs_write+0x190/0x1b0 Dec 27 09:19:05 2013 kernel: : [277622.359206] [<c050cbb3>] SyS_write+0x53/0x80 Dec 27 09:19:05 2013 kernel: : [277622.359211] [<c08f72ba>] sysenter_do_call+0x12/0x22 Dec 27 09:19:05 2013 kernel: : [277622.359213] Mem-Info: Dec 27 09:19:05 2013 kernel: : [277622.359215] DMA per-cpu: Dec 27 09:19:05 2013 kernel: : [277622.359218] CPU 0: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359220] CPU 1: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359222] CPU 2: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359224] CPU 3: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359226] CPU 4: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359228] CPU 5: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359230] CPU 6: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359232] CPU 7: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359234] CPU 8: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359236] CPU 9: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359238] CPU 10: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359240] CPU 11: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359242] CPU 12: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359244] CPU 13: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359246] CPU 14: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359248] CPU 15: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359250] CPU 16: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359253] CPU 17: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359255] CPU 18: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359258] CPU 19: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359260] CPU 20: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359262] CPU 21: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359264] CPU 22: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359266] CPU 23: hi: 0, btch: 1 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359268] Normal per-cpu: Dec 27 09:19:05 2013 kernel: : [277622.359270] CPU 0: hi: 186, btch: 31 usd: 34 Dec 27 09:19:05 2013 kernel: : [277622.359272] CPU 1: hi: 186, btch: 31 usd: 72 Dec 27 09:19:05 2013 kernel: : [277622.359274] CPU 2: hi: 186, btch: 31 usd: 40 Dec 27 09:19:05 2013 kernel: : [277622.359276] CPU 3: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359279] CPU 4: hi: 186, btch: 31 usd: 39 Dec 27 09:19:05 2013 kernel: : [277622.359281] CPU 5: hi: 186, btch: 31 usd: 49 Dec 27 09:19:05 2013 kernel: : [277622.359283] CPU 6: hi: 186, btch: 31 usd: 50 Dec 27 09:19:05 2013 kernel: : [277622.359285] CPU 7: hi: 186, btch: 31 usd: 25 Dec 27 09:19:05 2013 kernel: : [277622.359286] CPU 8: hi: 186, btch: 31 usd: 42 Dec 27 09:19:05 2013 kernel: : [277622.359289] CPU 9: hi: 186, btch: 31 usd: 39 Dec 27 09:19:05 2013 kernel: : [277622.359290] CPU 10: hi: 186, btch: 31 usd: 155 Dec 27 09:19:05 2013 kernel: : [277622.359293] CPU 11: hi: 186, btch: 31 usd: 56 Dec 27 09:19:05 2013 kernel: : [277622.359295] CPU 12: hi: 186, btch: 31 usd: 2 Dec 27 09:19:05 2013 kernel: : [277622.359297] CPU 13: hi: 186, btch: 31 usd: 162 Dec 27 09:19:05 2013 kernel: : [277622.359299] CPU 14: hi: 186, btch: 31 usd: 67 Dec 27 09:19:05 2013 kernel: : [277622.359301] CPU 15: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359303] CPU 16: hi: 186, btch: 31 usd: 68 Dec 27 09:19:05 2013 kernel: : [277622.359305] CPU 17: hi: 186, btch: 31 usd: 38 Dec 27 09:19:05 2013 kernel: : [277622.359307] CPU 18: hi: 186, btch: 31 usd: 56 Dec 27 09:19:05 2013 kernel: : [277622.359308] CPU 19: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359310] CPU 20: hi: 186, btch: 31 usd: 54 Dec 27 09:19:05 2013 kernel: : [277622.359312] CPU 21: hi: 186, btch: 31 usd: 35 Dec 27 09:19:05 2013 kernel: : [277622.359314] CPU 22: hi: 186, btch: 31 usd: 2 Dec 27 09:19:05 2013 kernel: : [277622.359316] CPU 23: hi: 186, btch: 31 usd: 60 Dec 27 09:19:05 2013 kernel: : [277622.359318] HighMem per-cpu: Dec 27 09:19:05 2013 kernel: : [277622.359320] CPU 0: hi: 186, btch: 31 usd: 32 Dec 27 09:19:05 2013 kernel: : [277622.359322] CPU 1: hi: 186, btch: 31 usd: 52 Dec 27 09:19:05 2013 kernel: : [277622.359324] CPU 2: hi: 186, btch: 31 usd: 9 Dec 27 09:19:05 2013 kernel: : [277622.359326] CPU 3: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359328] CPU 4: hi: 186, btch: 31 usd: 125 Dec 27 09:19:05 2013 kernel: : [277622.359330] CPU 5: hi: 186, btch: 31 usd: 116 Dec 27 09:19:05 2013 kernel: : [277622.359332] CPU 6: hi: 186, btch: 31 usd: 126 Dec 27 09:19:05 2013 kernel: : [277622.359333] CPU 7: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359336] CPU 8: hi: 186, btch: 31 usd: 79 Dec 27 09:19:05 2013 kernel: : [277622.359338] CPU 9: hi: 186, btch: 31 usd: 34 Dec 27 09:19:05 2013 kernel: : [277622.359340] CPU 10: hi: 186, btch: 31 usd: 111 Dec 27 09:19:05 2013 kernel: : [277622.359341] CPU 11: hi: 186, btch: 31 usd: 144 Dec 27 09:19:05 2013 kernel: : [277622.359343] CPU 12: hi: 186, btch: 31 usd: 15 Dec 27 09:19:05 2013 kernel: : [277622.359345] CPU 13: hi: 186, btch: 31 usd: 166 Dec 27 09:19:05 2013 kernel: : [277622.359347] CPU 14: hi: 186, btch: 31 usd: 185 Dec 27 09:19:05 2013 kernel: : [277622.359349] CPU 15: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359351] CPU 16: hi: 186, btch: 31 usd: 58 Dec 27 09:19:05 2013 kernel: : [277622.359353] CPU 17: hi: 186, btch: 31 usd: 122 Dec 27 09:19:05 2013 kernel: : [277622.359356] CPU 18: hi: 186, btch: 31 usd: 170 Dec 27 09:19:05 2013 kernel: : [277622.359358] CPU 19: hi: 186, btch: 31 usd: 0 Dec 27 09:19:05 2013 kernel: : [277622.359360] CPU 20: hi: 186, btch: 31 usd: 30 Dec 27 09:19:05 2013 kernel: : [277622.359362] CPU 21: hi: 186, btch: 31 usd: 33 Dec 27 09:19:05 2013 kernel: : [277622.359364] CPU 22: hi: 186, btch: 31 usd: 28 Dec 27 09:19:05 2013 kernel: : [277622.359366] CPU 23: hi: 186, btch: 31 usd: 44 Dec 27 09:19:05 2013 kernel: : [277622.359371] active_anon:658515 inactive_anon:54399 isolated_anon:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] active_file:1172176 inactive_file:323606 isolated_file:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] unevictable:0 dirty:0 writeback:0 unstable:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] free:6911872 slab_reclaimable:29430 slab_unreclaimable:34726 Dec 27 09:19:05 2013 kernel: : [277622.359371] mapped:45784 shmem:9850 pagetables:107714 bounce:0 Dec 27 09:19:05 2013 kernel: : [277622.359371] free_cma:0 Dec 27 09:19:05 2013 kernel: : [277622.359382] DMA free:2332kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15968kB managed:6960kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:8kB slab_unreclaimable:288kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Dec 27 09:19:05 2013 kernel: : [277622.359384] lowmem_reserve[]: 0 573 36539 36539 Dec 27 09:19:05 2013 kernel: : [277622.359393] Normal free:114488kB min:3044kB low:3804kB high:4564kB active_anon:0kB inactive_anon:0kB active_file:252kB inactive_file:256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:894968kB managed:587540kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:117712kB slab_unreclaimable:138616kB kernel_stack:11976kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:982 all_unreclaimable? yes Dec 27 09:19:05 2013 kernel: : [277622.359395] lowmem_reserve[]: 0 0 287725 287725 Dec 27 09:19:05 2013 kernel: : [277622.359404] HighMem free:27530668kB min:512kB low:48272kB high:96036kB active_anon:2634060kB inactive_anon:217596kB active_file:4688452kB inactive_file:1294168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:36828872kB managed:36828872kB mlocked:0kB dirty:0kB writeback:0kB mapped:183132kB shmem:39400kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:430856kB unstable:0kB bounce:367564104kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Dec 27 09:19:05 2013 kernel: : [277622.359406] lowmem_reserve[]: 0 0 0 0 Dec 27 09:19:05 2013 kernel: : [277622.359410] DMA: 3*4kB (U) 2*8kB (U) 4*16kB (U) 5*32kB (U) 2*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 2428kB Dec 27 09:19:05 2013 kernel: : [277622.359422] Normal: 5360*4kB (UEM) 3667*8kB (UEM) 3964*16kB (UEMR) 13*32kB (MR) 0*64kB 1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115000kB Dec 27 09:19:05 2013 kernel: : [277622.359435] HighMem: 6672*4kB (M) 74585*8kB (UM) 40828*16kB (UM) 17275*32kB (UM) 3314*64kB (UM) 1126*128kB (UM) 992*256kB (UM) 585*512kB (UM) 225*1024kB (UM) 78*2048kB (UMR) 5957*4096kB (UM) = 27529128kB Dec 27 09:19:05 2013 kernel: : [277622.359452] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Dec 27 09:19:05 2013 kernel: : [277622.359454] 1505509 total pagecache pages Dec 27 09:19:05 2013 kernel: : [277622.359457] 4 pages in swap cache Dec 27 09:19:05 2013 kernel: : [277622.359459] Swap cache stats: add 13, delete 9, find 0/0 Dec 27 09:19:05 2013 kernel: : [277622.359460] Free swap = 35318812kB Dec 27 09:19:05 2013 kernel: : [277622.359462] Total swap = 35318864kB Dec 27 09:19:05 2013 kernel: : [277622.450529] 9699327 pages RAM Dec 27 09:19:05 2013 kernel: : [277622.450532] 9471490 pages HighMem Dec 27 09:19:05 2013 kernel: : [277622.450533] 342749 pages reserved Dec 27 09:19:05 2013 kernel: : [277622.450534] 2864256 pages shared Dec 27 09:19:05 2013 kernel: : [277622.450535] 1501243 pages non-shared Dec 27 09:19:05 2013 kernel: : [277622.450538] Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
Dec 27 09:19:05 2013 kernel: : [277622.450538]
# cat /proc/meminfo MemTotal: 37426312 kB MemFree: 28328992 kB Buffers: 94728 kB Cached: 6216068 kB SwapCached: 0 kB Active: 6958572 kB Inactive: 1815380 kB Active(anon): 2329152 kB Inactive(anon): 170252 kB Active(file): 4629420 kB Inactive(file): 1645128 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 36828872 kB HighFree: 28076144 kB LowTotal: 597440 kB LowFree: 252848 kB SwapTotal: 35318864 kB SwapFree: 35318860 kB Dirty: 0 kB Writeback: 8 kB AnonPages: 2463512 kB Mapped: 162296 kB Shmem: 36332 kB Slab: 208676 kB SReclaimable: 120872 kB SUnreclaim: 87804 kB KernelStack: 6320 kB PageTables: 42280 kB NFS_Unstable: 0 kB Bounce: 124 kB WritebackTmp: 0 kB CommitLimit: 54032020 kB Committed_AS: 3191916 kB VmallocTotal: 122880 kB VmallocUsed: 27088 kB VmallocChunk: 29312 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 10232 kB DirectMap2M: 901120 kB
sysctl:
vm.oom_dump_tasks = 0 vm.oom_kill_allocating_task = 1 vm.panic_on_oom = 1
vm.admin_reserve_kbytes = 8192 vm.block_dump = 0 vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.drop_caches = 0 vm.highmem_is_dirtyable = 0 vm.hugepages_treat_as_movable = 0 vm.hugetlb_shm_group = 0 vm.laptop_mode = 0 vm.legacy_va_layout = 0 vm.lowmem_reserve_ratio = 256 32 32 vm.max_map_count = 65530 vm.min_free_kbytes = 3084 vm.mmap_min_addr = 4096 vm.nr_hugepages = 0 vm.nr_overcommit_hugepages = 0 vm.nr_pdflush_threads = 0 vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.percpu_pagelist_fraction = 0 vm.scan_unevictable_pages = 0 vm.stat_interval = 1 vm.swappiness = 30 vm.user_reserve_kbytes = 131072 vm.vdso_enabled = 1 vm.vfs_cache_pressure = 100
# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 292370 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 36728 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 292370 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Oguz Yilmaz said the following on 30/12/2013 09:02:
I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? Is it about some other memory? highmem? lowmem? stack size?
Had a similar issue on some CentOS servers and solved by changing the memory overcommit [1] strategy of Linux kernel.
CentOS via its upstream sets /proc/sys/vm/overcommit_memory [2] to zero, better to set il to 2 and play with /proc/sys/vm/overcommit_ratio (some say that it should be 50, other 80)
Try to put this in /etc/sysctl.conf and see if anithing changes
vm.overcommit_memory = 2 vm.overcommit_ratio = 80
Other useful links: http://www.hskupin.info/2010/06/17/how-to-fix-the-oom-killer-crashe-under-li... http://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6 http://www.dbasquare.com/2012/05/31/mysql-oom-killer-and-everything-related/
Ciao, luigi
[1] http://opsmonkey.blogspot.it/2007/01/linux-memory-overcommit.html [2] http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html
- -- / +--[Luigi Rosa]-- \
It is useless to send armies against ideas. --Georg Brandes
-
Eero Volotinen -
Luigi Rosa -
Oguz Yilmaz