Jussi Hirvi wrote:
> My web & name server runs out of memory from time to time, to the point
> where it's completely unresponsive to anything. At that point reset is
> the only alternative. (Or, as this is a virtual guest, I just say "virsh
> destroy").
>
> But why this happens - I would like to know.
>
> The host in question is a KVM guest, and runs CentOS 6.4.
>
That's a *lot* of apache. Is that really correct? Do you really need that
many threads? How heavily is the webserver used?
Also, I see mysql running - does the website use it?
mark
> From "top" (situation now):
> Mem: 1361564k total, 1264324k used, 97240k free, 8428k buffers
> Swap: 3014648k total, 64852k used, 2949796k free, 358676k cached
>
> At or before the last crash I got a long error message on the console.
> "Oom-killer" was called repeatedly by httpd and named.
>
> I paste below the first error message, which is long. There were several
> long entries like this. I
>
> - Jussi
>
> [root at ns1 ~]# httpd invoked oom-killer: gfp_mask=0x280da, order=0,
> oom_adj=0, oom_score_adj=0
> httpd cpuset=/ mems_allowed=0
> Pid: 2962, comm: httpd Not tainted 2.6.32-358.23.2.el6.x86_64 #1
> Call Trace:
> [<ffffffff810cb641>] ? cpuset_print_task_mems_allowed+0x91/0xb0
> [<ffffffff8111ce40>] ? dump_header+0x90/0x1b0
> [<ffffffff810e930e>] ? __delayacct_freepages_end+0x2e/0x30
> [<ffffffff8121d4ec>] ? security_real_capable_noaudit+0x3c/0x70
> [<ffffffff8111d2c2>] ? oom_kill_process+0x82/0x2a0
> [<ffffffff8111d201>] ? select_bad_process+0xe1/0x120
> [<ffffffff8111d700>] ? out_of_memory+0x220/0x3c0
> [<ffffffff8112c3dc>] ? __alloc_pages_nodemask+0x8ac/0x8d0
> [<ffffffff81160d6a>] ? alloc_pages_vma+0x9a/0x150
> [<ffffffff81143f0b>] ? handle_pte_fault+0x76b/0xb50
> [<ffffffffa00c60f9>] ? ext4_check_acl+0x29/0x90 [ext4]
> [<ffffffff81075887>] ? current_fs_time+0x27/0x30
> [<ffffffff8114452a>] ? handle_mm_fault+0x23a/0x310
> [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
> [<ffffffff8114aaba>] ? do_mmap_pgoff+0x33a/0x380
> [<ffffffff81513bfe>] ? do_page_fault+0x3e/0xa0
> [<ffffffff81510fb5>] ? page_fault+0x25/0x30
> Mem-Info:
> Node 0 DMA per-cpu:
> CPU 0: hi: 0, btch: 1 usd: 0
> Node 0 DMA32 per-cpu:
> CPU 0: hi: 186, btch: 31 usd: 89
> active_anon:201535 inactive_anon:68173 isolated_anon:3424
> active_file:130 inactive_file:284 isolated_file:160
> unevictable:0 dirty:0 writeback:245 unstable:0
> free:14234 slab_reclaimable:2891 slab_unreclaimable:13218
> mapped:239 shmem:14 pagetables:28858 bounce:0
> Node 0 DMA free:8252kB min:340kB low:424kB high:508kB active_anon:2168kB
> inactive_anon:4624kB active_file:4kB inactive_file:144kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB
> dirty:0kB writeback:28kB mapped:8kB shmem:0kB slab_reclaimable:12kB
> slab_unreclaimable:128kB kernel_stack:8kB pagetables:376kB unstable:0kB
> bounce:0kB writeback_tmp:0kB pages_scanned:1424 all_unreclaimable? no
> lowmem_reserve[]: 0 1956 1956 1956
> Node 0 DMA32 free:48684kB min:44712kB low:55888kB high:67068kB
> active_anon:803972kB inactive_anon:268068kB active_file:516kB
> inactive_file:992kB unevictable:0kB isolated(anon):13696kB
> isolated(file):640kB present:2003828kB mlocked:0kB dirty:0kB
> writeback:888kB mapped:948kB shmem:56kB slab_reclaimable:11552kB
> slab_unreclaimable:52744kB kernel_stack:3344kB pagetables:115056kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7552
> all_unreclaimable? no
> lowmem_reserve[]: 0 0 0 0
> Node 0 DMA: 25*4kB 11*8kB 6*16kB 3*32kB 9*64kB 3*128kB 1*256kB 3*512kB
> 1*1024kB 2*2048kB 0*4096kB = 8252kB
> Node 0 DMA32: 977*4kB 401*8kB 320*16kB 179*32kB 58*64kB 63*128kB
> 32*256kB 7*512kB 3*1024kB 0*2048kB 1*4096kB = 48684kB
> 21463 total pagecache pages
> 20882 pages in swap cache
> Swap cache stats: add 1523887, delete 1503005, find 201987/297332
> Free swap = 0kB
> Total swap = 3014648kB
> 511996 pages RAM
> 43605 pages reserved
> 66036 pages shared
> 446256 pages non-shared
> [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
> [ 449] 0 449 2675 0 0 -17 -1000 udevd
> [ 1080] 0 1080 6909 6 0 -17 -1000 auditd
> [ 1105] 0 1105 62271 1 0 0 0 rsyslogd
> [ 1142] 25 1142 40455 262 0 0 0 named
> [ 1184] 0 1184 16563 0 0 -17 -1000 sshd
> [ 1195] 0 1195 13036 0 0 0 0 vsftpd
> [ 1231] 0 1231 27041 1 0 0 0
> mysqld_safe
> [ 1333] 27 1333 187111 4825 0 0 0 mysqld
> [ 1430] 0 1430 20216 21 0 0 0 master
> [ 1438] 89 1438 20236 18 0 0 0 pickup
> [ 1439] 89 1439 20279 22 0 0 0 qmgr
> [ 1440] 0 1440 79250 368 0 0 0 httpd
> [ 1448] 0 1448 29313 34 0 0 0 crond
> [ 1459] 48 1459 88738 727 0 0 0 httpd
> [ 1460] 48 1460 89244 494 0 0 0 httpd
> [ 1461] 48 1461 89729 803 0 0 0 httpd
> [ 1462] 48 1462 88987 1055 0 0 0 httpd
> [ 1463] 48 1463 89796 2560 0 0 0 httpd
> [ 1464] 48 1464 91371 3558 0 0 0 httpd
> [ 1465] 48 1465 88596 1283 0 0 0 httpd
> [ 1466] 48 1466 90794 1253 0 0 0 httpd
> [ 1469] 0 1469 106619 198 0 0 0
> fail2ban-server
> [ 1497] 0 1497 1015 1 0 0 0 mingetty
> [ 1499] 0 1499 1015 1 0 0 0 mingetty
> [ 1501] 0 1501 1015 1 0 0 0 mingetty
> [ 1503] 0 1503 1015 1 0 0 0 mingetty
> [ 1505] 0 1505 1015 1 0 0 0 mingetty
> [ 1507] 0 1507 1015 1 0 0 0 mingetty
> [ 1512] 0 1512 2674 0 0 -17 -1000 udevd
> [ 1513] 0 1513 2674 0 0 -17 -1000 udevd
> [ 1514] 0 1514 14249 1 0 0 0 login
> [ 1520] 48 1520 89586 1139 0 0 0 httpd
> [ 1522] 48 1522 90606 773 0 0 0 httpd
> [ 1523] 48 1523 88343 2527 0 0 0 httpd
> [ 1524] 0 1524 27075 1 0 0 0 bash
> [ 1549] 48 1549 88937 1738 0 0 0 httpd
> [ 1550] 48 1550 89721 1736 0 0 0 httpd
> [ 1579] 48 1579 91106 600 0 0 0 httpd
> [ 1581] 48 1581 88218 1491 0 0 0 httpd
> [ 1582] 48 1582 88559 595 0 0 0 httpd
> [ 1755] 48 1755 88115 1716 0 0 0 httpd
> [ 2376] 48 2376 90860 1318 0 0 0 httpd
> [ 2562] 48 2562 89680 2348 0 0 0 httpd
> [ 2697] 48 2697 87846 2082 0 0 0 httpd
> [ 2709] 48 2709 86589 3201 0 0 0 httpd
> [ 2710] 48 2710 86056 2467 0 0 0 httpd
> [ 2713] 48 2713 84422 1068 0 0 0 httpd
> [ 2714] 48 2714 85544 2688 0 0 0 httpd
> [ 2715] 48 2715 84748 1505 0 0 0 httpd
> [ 2716] 48 2716 84569 1318 0 0 0 httpd
> [ 2721] 48 2721 84815 1718 0 0 0 httpd
> [ 2722] 48 2722 84502 1256 0 0 0 httpd
> [ 2723] 48 2723 87014 4123 0 0 0 httpd
> [ 2724] 48 2724 84502 1246 0 0 0 httpd
> [ 2725] 48 2725 84420 1203 0 0 0 httpd
> [ 2726] 48 2726 86782 4034 0 0 0 httpd
> [ 2727] 48 2727 84502 1262 0 0 0 httpd
> [ 2728] 48 2728 86054 2340 0 0 0 httpd
> [ 2737] 48 2737 84815 1743 0 0 0 httpd
> [ 2738] 48 2738 84748 1639 0 0 0 httpd
> [ 2739] 48 2739 86589 3300 0 0 0 httpd
> [ 2740] 48 2740 85202 2201 0 0 0 httpd
> [ 2741] 48 2741 86054 1742 0 0 0 httpd
> [ 2742] 48 2742 87014 4231 0 0 0 httpd
> [ 2743] 48 2743 84301 711 0 0 0 httpd
> [ 2744] 48 2744 84502 1233 0 0 0 httpd
> [ 2745] 48 2745 84815 1614 0 0 0 httpd
> [ 2746] 48 2746 84420 1113 0 0 0 httpd
> [ 2747] 48 2747 84432 913 0 0 0 httpd
> [ 2748] 48 2748 84301 724 0 0 0 httpd
> [ 2749] 48 2749 86251 3541 0 0 0 httpd
> [ 2750] 48 2750 84301 764 0 0 0 httpd
> [ 2751] 48 2751 84422 1004 0 0 0 httpd
> [ 2752] 48 2752 84291 588 0 0 0 httpd
> [ 2758] 48 2758 84301 687 0 0 0 httpd
> [ 2759] 48 2759 84301 725 0 0 0 httpd
> [ 2760] 48 2760 84432 890 0 0 0 httpd
> [ 2761] 48 2761 84301 788 0 0 0 httpd
> [ 2765] 48 2765 84420 930 0 0 0 httpd
> [ 2766] 48 2766 84430 868 0 0 0 httpd
> [ 2769] 48 2769 84432 946 0 0 0 httpd
> [ 2770] 48 2770 87526 8319 0 0 0 httpd
> [ 2774] 48 2774 84291 582 0 0 0 httpd
> [ 2775] 48 2775 84430 897 0 0 0 httpd
> [ 2779] 48 2779 84301 692 0 0 0 httpd
> [ 2780] 48 2780 84301 659 0 0 0 httpd
> [ 2781] 48 2781 84430 1098 0 0 0 httpd
> [ 2782] 48 2782 84420 2692 0 0 0 httpd
> [ 2783] 48 2783 84430 823 0 0 0 httpd
> [ 2784] 48 2784 84422 1062 0 0 0 httpd
> [ 2785] 48 2785 84432 899 0 0 0 httpd
> [ 2786] 48 2786 84502 1259 0 0 0 httpd
> [ 2787] 48 2787 84301 733 0 0 0 httpd
> [ 2788] 48 2788 84434 906 0 0 0 httpd
> [ 2789] 48 2789 84432 1000 0 0 0 httpd
> [ 2790] 48 2790 87526 8311 0 0 0 httpd
> [ 2791] 48 2791 84430 1067 0 0 0 httpd
> [ 2792] 48 2792 84504 1259 0 0 0 httpd
> [ 2793] 48 2793 84301 749 0 0 0 httpd
> [ 2794] 48 2794 84422 935 0 0 0 httpd
> [ 2795] 48 2795 84303 740 0 0 0 httpd
> [ 2796] 48 2796 84560 1417 0 0 0 httpd
> [ 2797] 48 2797 84420 1091 0 0 0 httpd
> [ 2798] 48 2798 84420 1023 0 0 0 httpd
> [ 2799] 48 2799 84301 678 0 0 0 httpd
> [ 2800] 48 2800 84301 643 0 0 0 httpd
> [ 2809] 48 2809 84422 1017 0 0 0 httpd
> [ 2811] 48 2811 84301 643 0 0 0 httpd
> [ 2812] 48 2812 84301 620 0 0 0 httpd
> [ 2813] 48 2813 84432 958 0 0 0 httpd
> [ 2814] 48 2814 84432 928 0 0 0 httpd
> [ 2815] 48 2815 84420 1126 0 0 0 httpd
> [ 2816] 48 2816 87014 7940 0 0 0 httpd
> [ 2817] 48 2817 84301 725 0 0 0 httpd
> [ 2818] 48 2818 84301 695 0 0 0 httpd
> [ 2819] 48 2819 84432 800 0 0 0 httpd
> [ 2820] 48 2820 84420 1209 0 0 0 httpd
> [ 2821] 48 2821 84502 1351 0 0 0 httpd
> [ 2822] 48 2822 84815 1684 0 0 0 httpd
> [ 2823] 48 2823 84301 702 0 0 0 httpd
> [ 2824] 48 2824 84430 1416 0 0 0 httpd
> [ 2825] 48 2825 84432 824 0 0 0 httpd
> [ 2826] 48 2826 84432 899 0 0 0 httpd
> [ 2827] 48 2827 84432 884 0 0 0 httpd
> [ 2828] 48 2828 84301 794 0 0 0 httpd
> [ 2829] 48 2829 84301 723 0 0 0 httpd
> [ 2830] 48 2830 84301 1413 0 0 0 httpd
> [ 2831] 48 2831 85331 3528 0 0 0 httpd
> [ 2832] 48 2832 84420 3300 0 0 0 httpd
> [ 2833] 48 2833 84430 1353 0 0 0 httpd
> [ 2834] 48 2834 84432 897 0 0 0 httpd
> [ 2835] 48 2835 84291 572 0 0 0 httpd
> [ 2836] 48 2836 85460 3295 0 0 0 httpd
> [ 2852] 48 2852 83773 4497 0 0 0 httpd
> [ 2853] 48 2853 84814 2177 0 0 0 httpd
> [ 2857] 48 2857 84420 1140 0 0 0 httpd
> [ 2858] 48 2858 84432 925 0 0 0 httpd
> [ 2859] 48 2859 84485 1568 0 0 0 httpd
> [ 2860] 48 2860 84552 1749 0 0 0 httpd
> [ 2861] 48 2861 84291 582 0 0 0 httpd
> [ 2862] 48 2862 84420 3627 0 0 0 httpd
> [ 2868] 48 2868 84301 804 0 0 0 httpd
> [ 2869] 48 2869 84487 1268 0 0 0 httpd
> [ 2870] 48 2870 84301 803 0 0 0 httpd
> [ 2874] 48 2874 84422 1033 0 0 0 httpd
> [ 2875] 48 2875 84301 718 0 0 0 httpd
> [ 2876] 48 2876 84485 1125 0 0 0 httpd
> [ 2877] 48 2877 84420 924 0 0 0 httpd
> [ 2878] 48 2878 84748 2217 0 0 0 httpd
> [ 2885] 48 2885 84301 2638 0 0 0 httpd
> [ 2886] 48 2886 84420 905 0 0 0 httpd
> [ 2887] 48 2887 84420 1558 0 0 0 httpd
> [ 2888] 48 2888 84301 697 0 0 0 httpd
> [ 2889] 48 2889 84502 1228 0 0 0 httpd
> [ 2904] 48 2904 84430 791 0 0 0 httpd
> [ 2915] 48 2915 84748 1518 0 0 0 httpd
> [ 2916] 48 2916 85331 3691 0 0 0 httpd
> [ 2917] 48 2917 84569 1634 0 0 0 httpd
> [ 2918] 48 2918 84239 4498 0 0 0 httpd
> [ 2919] 48 2919 84502 2969 0 0 0 httpd
> [ 2920] 48 2920 86763 4460 0 0 0 httpd
> [ 2921] 48 2921 84420 1133 0 0 0 httpd
> [ 2922] 48 2922 86381 5779 0 0 0 httpd
> [ 2928] 48 2928 84502 2436 0 0 0 httpd
> [ 2938] 48 2938 85208 2426 0 0 0 httpd
> [ 2939] 48 2939 86055 5438 0 0 0 httpd
> [ 2940] 48 2940 84487 4186 0 0 0 httpd
> [ 2944] 48 2944 84432 2877 0 0 0 httpd
> [ 2945] 48 2945 84291 1446 0 0 0 httpd
> [ 2946] 48 2946 84818 3977 0 0 0 httpd
> [ 2960] 48 2960 84239 4311 0 0 0 httpd
> [ 2961] 48 2961 89636 9983 0 0 0 httpd
> [ 2962] 48 2962 87955 8183 0 0 0 httpd
> [ 2970] 48 2970 84241 4777 0 0 0 httpd
> [ 2973] 48 2973 84239 4759 0 0 0 httpd
> [ 2974] 48 2974 80053 1296 0 0 0 httpd
> [ 2975] 48 2975 80012 1410 0 0 0 httpd
> [ 2976] 48 2976 80185 1606 0 0 0 httpd
> [ 2978] 48 2978 79284 414 0 0 0 httpd
> [ 2980] 48 2980 80185 1387 0 0 0 httpd
> [ 2981] 48 2981 80021 1076 0 0 0 httpd
> [ 2982] 48 2982 80181 1383 0 0 0 httpd
> [ 2983] 48 2983 80138 1348 0 0 0 httpd
> [ 2984] 48 2984 80202 1273 0 0 0 httpd
> [ 2985] 48 2985 80010 418 0 0 0 httpd
> [ 2986] 48 2986 79990 1409 0 0 0 httpd
> [ 2988] 48 2988 80202 1540 0 0 0 httpd
> [ 2989] 48 2989 79996 756 0 0 0 httpd
> [ 2990] 48 2990 80010 1412 0 0 0 httpd
> [ 2991] 48 2991 79986 789 0 0 0 httpd
> [ 2992] 48 2992 79284 401 0 0 0 httpd
> [ 2993] 48 2993 80189 1562 0 0 0 httpd
> [ 2994] 48 2994 80119 1316 0 0 0 httpd
> [ 3010] 48 3010 79283 425 0 0 0 httpd
> [ 3013] 48 3013 80046 1291 0 0 0 httpd
> [ 3016] 48 3016 79250 412 0 0 0 httpd
> [ 3018] 48 3018 79283 451 0 0 0 httpd
> [ 3022] 0 3022 33480 63 0 0 0 crond
> Out of memory: Kill process 1333 (mysqld) score 12 or sacrifice child
> Killed process 1333, UID 27, (mysqld) total-vm:748444kB,
> anon-rss:19232kB, file-rss:68kB
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>