My web & name server runs out of memory from time to time, to the point
where it's completely unresponsive to anything. At that point reset is
the only alternative. (Or, as this is a virtual guest, I just say "virsh
destroy").
But why this happens - I would like to know.
The host in question is a KVM guest, and runs CentOS 6.4.
From "top" (situation now):
Mem: 1361564k total, 1264324k used, 97240k free, 8428k buffers
Swap: 3014648k total, 64852k used, 2949796k free, 358676k cached
At or before the last crash I got a long error message on the console.
"Oom-killer" was called repeatedly by httpd and named.
I paste below the first error message, which is long. There were several
long entries like this. I
- Jussi
[root@ns1 ~]# httpd invoked oom-killer: gfp_mask=0x280da, order=0,
oom_adj=0, oom_score_adj=0
httpd cpuset=/ mems_allowed=0
Pid: 2962, comm: httpd Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Call Trace:
[<ffffffff810cb641>] ? cpuset_print_task_mems_allowed+0x91/0xb0
[<ffffffff8111ce40>] ? dump_header+0x90/0x1b0
[<ffffffff810e930e>] ? __delayacct_freepages_end+0x2e/0x30
[<ffffffff8121d4ec>] ? security_real_capable_noaudit+0x3c/0x70
[<ffffffff8111d2c2>] ? oom_kill_process+0x82/0x2a0
[<ffffffff8111d201>] ? select_bad_process+0xe1/0x120
[<ffffffff8111d700>] ? out_of_memory+0x220/0x3c0
[<ffffffff8112c3dc>] ? __alloc_pages_nodemask+0x8ac/0x8d0
[<ffffffff81160d6a>] ? alloc_pages_vma+0x9a/0x150
[<ffffffff81143f0b>] ? handle_pte_fault+0x76b/0xb50
[<ffffffffa00c60f9>] ? ext4_check_acl+0x29/0x90 [ext4]
[<ffffffff81075887>] ? current_fs_time+0x27/0x30
[<ffffffff8114452a>] ? handle_mm_fault+0x23a/0x310
[<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
[<ffffffff8114aaba>] ? do_mmap_pgoff+0x33a/0x380
[<ffffffff81513bfe>] ? do_page_fault+0x3e/0xa0
[<ffffffff81510fb5>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 89
active_anon:201535 inactive_anon:68173 isolated_anon:3424
active_file:130 inactive_file:284 isolated_file:160
unevictable:0 dirty:0 writeback:245 unstable:0
free:14234 slab_reclaimable:2891 slab_unreclaimable:13218
mapped:239 shmem:14 pagetables:28858 bounce:0
Node 0 DMA free:8252kB min:340kB low:424kB high:508kB active_anon:2168kB
inactive_anon:4624kB active_file:4kB inactive_file:144kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB
dirty:0kB writeback:28kB mapped:8kB shmem:0kB slab_reclaimable:12kB
slab_unreclaimable:128kB kernel_stack:8kB pagetables:376kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:1424 all_unreclaimable? no
lowmem_reserve[]: 0 1956 1956 1956
Node 0 DMA32 free:48684kB min:44712kB low:55888kB high:67068kB
active_anon:803972kB inactive_anon:268068kB active_file:516kB
inactive_file:992kB unevictable:0kB isolated(anon):13696kB
isolated(file):640kB present:2003828kB mlocked:0kB dirty:0kB
writeback:888kB mapped:948kB shmem:56kB slab_reclaimable:11552kB
slab_unreclaimable:52744kB kernel_stack:3344kB pagetables:115056kB
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7552
all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 25*4kB 11*8kB 6*16kB 3*32kB 9*64kB 3*128kB 1*256kB 3*512kB
1*1024kB 2*2048kB 0*4096kB = 8252kB
Node 0 DMA32: 977*4kB 401*8kB 320*16kB 179*32kB 58*64kB 63*128kB
32*256kB 7*512kB 3*1024kB 0*2048kB 1*4096kB = 48684kB
21463 total pagecache pages
20882 pages in swap cache
Swap cache stats: add 1523887, delete 1503005, find 201987/297332
Free swap = 0kB
Total swap = 3014648kB
511996 pages RAM
43605 pages reserved
66036 pages shared
446256 pages non-shared
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[ 449] 0 449 2675 0 0 -17 -1000 udevd
[ 1080] 0 1080 6909 6 0 -17 -1000 auditd
[ 1105] 0 1105 62271 1 0 0 0 rsyslogd
[ 1142] 25 1142 40455 262 0 0 0 named
[ 1184] 0 1184 16563 0 0 -17 -1000 sshd
[ 1195] 0 1195 13036 0 0 0 0 vsftpd
[ 1231] 0 1231 27041 1 0 0 0 mysqld_safe
[ 1333] 27 1333 187111 4825 0 0 0 mysqld
[ 1430] 0 1430 20216 21 0 0 0 master
[ 1438] 89 1438 20236 18 0 0 0 pickup
[ 1439] 89 1439 20279 22 0 0 0 qmgr
[ 1440] 0 1440 79250 368 0 0 0 httpd
[ 1448] 0 1448 29313 34 0 0 0 crond
[ 1459] 48 1459 88738 727 0 0 0 httpd
[ 1460] 48 1460 89244 494 0 0 0 httpd
[ 1461] 48 1461 89729 803 0 0 0 httpd
[ 1462] 48 1462 88987 1055 0 0 0 httpd
[ 1463] 48 1463 89796 2560 0 0 0 httpd
[ 1464] 48 1464 91371 3558 0 0 0 httpd
[ 1465] 48 1465 88596 1283 0 0 0 httpd
[ 1466] 48 1466 90794 1253 0 0 0 httpd
[ 1469] 0 1469 106619 198 0 0 0
fail2ban-server
[ 1497] 0 1497 1015 1 0 0 0 mingetty
[ 1499] 0 1499 1015 1 0 0 0 mingetty
[ 1501] 0 1501 1015 1 0 0 0 mingetty
[ 1503] 0 1503 1015 1 0 0 0 mingetty
[ 1505] 0 1505 1015 1 0 0 0 mingetty
[ 1507] 0 1507 1015 1 0 0 0 mingetty
[ 1512] 0 1512 2674 0 0 -17 -1000 udevd
[ 1513] 0 1513 2674 0 0 -17 -1000 udevd
[ 1514] 0 1514 14249 1 0 0 0 login
[ 1520] 48 1520 89586 1139 0 0 0 httpd
[ 1522] 48 1522 90606 773 0 0 0 httpd
[ 1523] 48 1523 88343 2527 0 0 0 httpd
[ 1524] 0 1524 27075 1 0 0 0 bash
[ 1549] 48 1549 88937 1738 0 0 0 httpd
[ 1550] 48 1550 89721 1736 0 0 0 httpd
[ 1579] 48 1579 91106 600 0 0 0 httpd
[ 1581] 48 1581 88218 1491 0 0 0 httpd
[ 1582] 48 1582 88559 595 0 0 0 httpd
[ 1755] 48 1755 88115 1716 0 0 0 httpd
[ 2376] 48 2376 90860 1318 0 0 0 httpd
[ 2562] 48 2562 89680 2348 0 0 0 httpd
[ 2697] 48 2697 87846 2082 0 0 0 httpd
[ 2709] 48 2709 86589 3201 0 0 0 httpd
[ 2710] 48 2710 86056 2467 0 0 0 httpd
[ 2713] 48 2713 84422 1068 0 0 0 httpd
[ 2714] 48 2714 85544 2688 0 0 0 httpd
[ 2715] 48 2715 84748 1505 0 0 0 httpd
[ 2716] 48 2716 84569 1318 0 0 0 httpd
[ 2721] 48 2721 84815 1718 0 0 0 httpd
[ 2722] 48 2722 84502 1256 0 0 0 httpd
[ 2723] 48 2723 87014 4123 0 0 0 httpd
[ 2724] 48 2724 84502 1246 0 0 0 httpd
[ 2725] 48 2725 84420 1203 0 0 0 httpd
[ 2726] 48 2726 86782 4034 0 0 0 httpd
[ 2727] 48 2727 84502 1262 0 0 0 httpd
[ 2728] 48 2728 86054 2340 0 0 0 httpd
[ 2737] 48 2737 84815 1743 0 0 0 httpd
[ 2738] 48 2738 84748 1639 0 0 0 httpd
[ 2739] 48 2739 86589 3300 0 0 0 httpd
[ 2740] 48 2740 85202 2201 0 0 0 httpd
[ 2741] 48 2741 86054 1742 0 0 0 httpd
[ 2742] 48 2742 87014 4231 0 0 0 httpd
[ 2743] 48 2743 84301 711 0 0 0 httpd
[ 2744] 48 2744 84502 1233 0 0 0 httpd
[ 2745] 48 2745 84815 1614 0 0 0 httpd
[ 2746] 48 2746 84420 1113 0 0 0 httpd
[ 2747] 48 2747 84432 913 0 0 0 httpd
[ 2748] 48 2748 84301 724 0 0 0 httpd
[ 2749] 48 2749 86251 3541 0 0 0 httpd
[ 2750] 48 2750 84301 764 0 0 0 httpd
[ 2751] 48 2751 84422 1004 0 0 0 httpd
[ 2752] 48 2752 84291 588 0 0 0 httpd
[ 2758] 48 2758 84301 687 0 0 0 httpd
[ 2759] 48 2759 84301 725 0 0 0 httpd
[ 2760] 48 2760 84432 890 0 0 0 httpd
[ 2761] 48 2761 84301 788 0 0 0 httpd
[ 2765] 48 2765 84420 930 0 0 0 httpd
[ 2766] 48 2766 84430 868 0 0 0 httpd
[ 2769] 48 2769 84432 946 0 0 0 httpd
[ 2770] 48 2770 87526 8319 0 0 0 httpd
[ 2774] 48 2774 84291 582 0 0 0 httpd
[ 2775] 48 2775 84430 897 0 0 0 httpd
[ 2779] 48 2779 84301 692 0 0 0 httpd
[ 2780] 48 2780 84301 659 0 0 0 httpd
[ 2781] 48 2781 84430 1098 0 0 0 httpd
[ 2782] 48 2782 84420 2692 0 0 0 httpd
[ 2783] 48 2783 84430 823 0 0 0 httpd
[ 2784] 48 2784 84422 1062 0 0 0 httpd
[ 2785] 48 2785 84432 899 0 0 0 httpd
[ 2786] 48 2786 84502 1259 0 0 0 httpd
[ 2787] 48 2787 84301 733 0 0 0 httpd
[ 2788] 48 2788 84434 906 0 0 0 httpd
[ 2789] 48 2789 84432 1000 0 0 0 httpd
[ 2790] 48 2790 87526 8311 0 0 0 httpd
[ 2791] 48 2791 84430 1067 0 0 0 httpd
[ 2792] 48 2792 84504 1259 0 0 0 httpd
[ 2793] 48 2793 84301 749 0 0 0 httpd
[ 2794] 48 2794 84422 935 0 0 0 httpd
[ 2795] 48 2795 84303 740 0 0 0 httpd
[ 2796] 48 2796 84560 1417 0 0 0 httpd
[ 2797] 48 2797 84420 1091 0 0 0 httpd
[ 2798] 48 2798 84420 1023 0 0 0 httpd
[ 2799] 48 2799 84301 678 0 0 0 httpd
[ 2800] 48 2800 84301 643 0 0 0 httpd
[ 2809] 48 2809 84422 1017 0 0 0 httpd
[ 2811] 48 2811 84301 643 0 0 0 httpd
[ 2812] 48 2812 84301 620 0 0 0 httpd
[ 2813] 48 2813 84432 958 0 0 0 httpd
[ 2814] 48 2814 84432 928 0 0 0 httpd
[ 2815] 48 2815 84420 1126 0 0 0 httpd
[ 2816] 48 2816 87014 7940 0 0 0 httpd
[ 2817] 48 2817 84301 725 0 0 0 httpd
[ 2818] 48 2818 84301 695 0 0 0 httpd
[ 2819] 48 2819 84432 800 0 0 0 httpd
[ 2820] 48 2820 84420 1209 0 0 0 httpd
[ 2821] 48 2821 84502 1351 0 0 0 httpd
[ 2822] 48 2822 84815 1684 0 0 0 httpd
[ 2823] 48 2823 84301 702 0 0 0 httpd
[ 2824] 48 2824 84430 1416 0 0 0 httpd
[ 2825] 48 2825 84432 824 0 0 0 httpd
[ 2826] 48 2826 84432 899 0 0 0 httpd
[ 2827] 48 2827 84432 884 0 0 0 httpd
[ 2828] 48 2828 84301 794 0 0 0 httpd
[ 2829] 48 2829 84301 723 0 0 0 httpd
[ 2830] 48 2830 84301 1413 0 0 0 httpd
[ 2831] 48 2831 85331 3528 0 0 0 httpd
[ 2832] 48 2832 84420 3300 0 0 0 httpd
[ 2833] 48 2833 84430 1353 0 0 0 httpd
[ 2834] 48 2834 84432 897 0 0 0 httpd
[ 2835] 48 2835 84291 572 0 0 0 httpd
[ 2836] 48 2836 85460 3295 0 0 0 httpd
[ 2852] 48 2852 83773 4497 0 0 0 httpd
[ 2853] 48 2853 84814 2177 0 0 0 httpd
[ 2857] 48 2857 84420 1140 0 0 0 httpd
[ 2858] 48 2858 84432 925 0 0 0 httpd
[ 2859] 48 2859 84485 1568 0 0 0 httpd
[ 2860] 48 2860 84552 1749 0 0 0 httpd
[ 2861] 48 2861 84291 582 0 0 0 httpd
[ 2862] 48 2862 84420 3627 0 0 0 httpd
[ 2868] 48 2868 84301 804 0 0 0 httpd
[ 2869] 48 2869 84487 1268 0 0 0 httpd
[ 2870] 48 2870 84301 803 0 0 0 httpd
[ 2874] 48 2874 84422 1033 0 0 0 httpd
[ 2875] 48 2875 84301 718 0 0 0 httpd
[ 2876] 48 2876 84485 1125 0 0 0 httpd
[ 2877] 48 2877 84420 924 0 0 0 httpd
[ 2878] 48 2878 84748 2217 0 0 0 httpd
[ 2885] 48 2885 84301 2638 0 0 0 httpd
[ 2886] 48 2886 84420 905 0 0 0 httpd
[ 2887] 48 2887 84420 1558 0 0 0 httpd
[ 2888] 48 2888 84301 697 0 0 0 httpd
[ 2889] 48 2889 84502 1228 0 0 0 httpd
[ 2904] 48 2904 84430 791 0 0 0 httpd
[ 2915] 48 2915 84748 1518 0 0 0 httpd
[ 2916] 48 2916 85331 3691 0 0 0 httpd
[ 2917] 48 2917 84569 1634 0 0 0 httpd
[ 2918] 48 2918 84239 4498 0 0 0 httpd
[ 2919] 48 2919 84502 2969 0 0 0 httpd
[ 2920] 48 2920 86763 4460 0 0 0 httpd
[ 2921] 48 2921 84420 1133 0 0 0 httpd
[ 2922] 48 2922 86381 5779 0 0 0 httpd
[ 2928] 48 2928 84502 2436 0 0 0 httpd
[ 2938] 48 2938 85208 2426 0 0 0 httpd
[ 2939] 48 2939 86055 5438 0 0 0 httpd
[ 2940] 48 2940 84487 4186 0 0 0 httpd
[ 2944] 48 2944 84432 2877 0 0 0 httpd
[ 2945] 48 2945 84291 1446 0 0 0 httpd
[ 2946] 48 2946 84818 3977 0 0 0 httpd
[ 2960] 48 2960 84239 4311 0 0 0 httpd
[ 2961] 48 2961 89636 9983 0 0 0 httpd
[ 2962] 48 2962 87955 8183 0 0 0 httpd
[ 2970] 48 2970 84241 4777 0 0 0 httpd
[ 2973] 48 2973 84239 4759 0 0 0 httpd
[ 2974] 48 2974 80053 1296 0 0 0 httpd
[ 2975] 48 2975 80012 1410 0 0 0 httpd
[ 2976] 48 2976 80185 1606 0 0 0 httpd
[ 2978] 48 2978 79284 414 0 0 0 httpd
[ 2980] 48 2980 80185 1387 0 0 0 httpd
[ 2981] 48 2981 80021 1076 0 0 0 httpd
[ 2982] 48 2982 80181 1383 0 0 0 httpd
[ 2983] 48 2983 80138 1348 0 0 0 httpd
[ 2984] 48 2984 80202 1273 0 0 0 httpd
[ 2985] 48 2985 80010 418 0 0 0 httpd
[ 2986] 48 2986 79990 1409 0 0 0 httpd
[ 2988] 48 2988 80202 1540 0 0 0 httpd
[ 2989] 48 2989 79996 756 0 0 0 httpd
[ 2990] 48 2990 80010 1412 0 0 0 httpd
[ 2991] 48 2991 79986 789 0 0 0 httpd
[ 2992] 48 2992 79284 401 0 0 0 httpd
[ 2993] 48 2993 80189 1562 0 0 0 httpd
[ 2994] 48 2994 80119 1316 0 0 0 httpd
[ 3010] 48 3010 79283 425 0 0 0 httpd
[ 3013] 48 3013 80046 1291 0 0 0 httpd
[ 3016] 48 3016 79250 412 0 0 0 httpd
[ 3018] 48 3018 79283 451 0 0 0 httpd
[ 3022] 0 3022 33480 63 0 0 0 crond
Out of memory: Kill process 1333 (mysqld) score 12 or sacrifice child
Killed process 1333, UID 27, (mysqld) total-vm:748444kB,
anon-rss:19232kB, file-rss:68kB
