[CentOS] Memory leak - how to investigate

Mon Feb 3 17:58:53 UTC 2014
m.roth at 5-cent.us <m.roth at 5-cent.us>

Jussi Hirvi wrote:
> My web & name server runs out of memory from time to time, to the point
> where it's completely unresponsive to anything. At that point reset is
> the only alternative. (Or, as this is a virtual guest, I just say "virsh
> destroy").
>
> But why this happens - I would like to know.
>
> The host in question is a KVM guest, and runs CentOS 6.4.
>
That's a *lot* of apache. Is that really correct? Do you really need that
many threads? How heavily is the webserver used?

Also, I see mysql running - does the website use it?

       mark

>  From "top" (situation now):
> Mem:   1361564k total,  1264324k used,    97240k free,     8428k buffers
> Swap:  3014648k total,    64852k used,  2949796k free,   358676k cached
>
> At or before the last crash I got a long error message on the console.
> "Oom-killer" was called repeatedly by httpd and named.
>
> I paste below the first error message, which is long. There were several
> long entries like this. I
>
> - Jussi
>
> [root at ns1 ~]# httpd invoked oom-killer: gfp_mask=0x280da, order=0,
> oom_adj=0, oom_score_adj=0
> httpd cpuset=/ mems_allowed=0
> Pid: 2962, comm: httpd Not tainted 2.6.32-358.23.2.el6.x86_64 #1
> Call Trace:
>   [<ffffffff810cb641>] ? cpuset_print_task_mems_allowed+0x91/0xb0
>   [<ffffffff8111ce40>] ? dump_header+0x90/0x1b0
>   [<ffffffff810e930e>] ? __delayacct_freepages_end+0x2e/0x30
>   [<ffffffff8121d4ec>] ? security_real_capable_noaudit+0x3c/0x70
>   [<ffffffff8111d2c2>] ? oom_kill_process+0x82/0x2a0
>   [<ffffffff8111d201>] ? select_bad_process+0xe1/0x120
>   [<ffffffff8111d700>] ? out_of_memory+0x220/0x3c0
>   [<ffffffff8112c3dc>] ? __alloc_pages_nodemask+0x8ac/0x8d0
>   [<ffffffff81160d6a>] ? alloc_pages_vma+0x9a/0x150
>   [<ffffffff81143f0b>] ? handle_pte_fault+0x76b/0xb50
>   [<ffffffffa00c60f9>] ? ext4_check_acl+0x29/0x90 [ext4]
>   [<ffffffff81075887>] ? current_fs_time+0x27/0x30
>   [<ffffffff8114452a>] ? handle_mm_fault+0x23a/0x310
>   [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
>   [<ffffffff8114aaba>] ? do_mmap_pgoff+0x33a/0x380
>   [<ffffffff81513bfe>] ? do_page_fault+0x3e/0xa0
>   [<ffffffff81510fb5>] ? page_fault+0x25/0x30
> Mem-Info:
> Node 0 DMA per-cpu:
> CPU    0: hi:    0, btch:   1 usd:   0
> Node 0 DMA32 per-cpu:
> CPU    0: hi:  186, btch:  31 usd:  89
> active_anon:201535 inactive_anon:68173 isolated_anon:3424
>   active_file:130 inactive_file:284 isolated_file:160
>   unevictable:0 dirty:0 writeback:245 unstable:0
>   free:14234 slab_reclaimable:2891 slab_unreclaimable:13218
>   mapped:239 shmem:14 pagetables:28858 bounce:0
> Node 0 DMA free:8252kB min:340kB low:424kB high:508kB active_anon:2168kB
> inactive_anon:4624kB active_file:4kB inactive_file:144kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB
> dirty:0kB writeback:28kB mapped:8kB shmem:0kB slab_reclaimable:12kB
> slab_unreclaimable:128kB kernel_stack:8kB pagetables:376kB unstable:0kB
> bounce:0kB writeback_tmp:0kB pages_scanned:1424 all_unreclaimable? no
> lowmem_reserve[]: 0 1956 1956 1956
> Node 0 DMA32 free:48684kB min:44712kB low:55888kB high:67068kB
> active_anon:803972kB inactive_anon:268068kB active_file:516kB
> inactive_file:992kB unevictable:0kB isolated(anon):13696kB
> isolated(file):640kB present:2003828kB mlocked:0kB dirty:0kB
> writeback:888kB mapped:948kB shmem:56kB slab_reclaimable:11552kB
> slab_unreclaimable:52744kB kernel_stack:3344kB pagetables:115056kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7552
> all_unreclaimable? no
> lowmem_reserve[]: 0 0 0 0
> Node 0 DMA: 25*4kB 11*8kB 6*16kB 3*32kB 9*64kB 3*128kB 1*256kB 3*512kB
> 1*1024kB 2*2048kB 0*4096kB = 8252kB
> Node 0 DMA32: 977*4kB 401*8kB 320*16kB 179*32kB 58*64kB 63*128kB
> 32*256kB 7*512kB 3*1024kB 0*2048kB 1*4096kB = 48684kB
> 21463 total pagecache pages
> 20882 pages in swap cache
> Swap cache stats: add 1523887, delete 1503005, find 201987/297332
> Free swap  = 0kB
> Total swap = 3014648kB
> 511996 pages RAM
> 43605 pages reserved
> 66036 pages shared
> 446256 pages non-shared
> [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
> [  449]     0   449     2675        0   0     -17         -1000 udevd
> [ 1080]     0  1080     6909        6   0     -17         -1000 auditd
> [ 1105]     0  1105    62271        1   0       0             0 rsyslogd
> [ 1142]    25  1142    40455      262   0       0             0 named
> [ 1184]     0  1184    16563        0   0     -17         -1000 sshd
> [ 1195]     0  1195    13036        0   0       0             0 vsftpd
> [ 1231]     0  1231    27041        1   0       0             0
> mysqld_safe
> [ 1333]    27  1333   187111     4825   0       0             0 mysqld
> [ 1430]     0  1430    20216       21   0       0             0 master
> [ 1438]    89  1438    20236       18   0       0             0 pickup
> [ 1439]    89  1439    20279       22   0       0             0 qmgr
> [ 1440]     0  1440    79250      368   0       0             0 httpd
> [ 1448]     0  1448    29313       34   0       0             0 crond
> [ 1459]    48  1459    88738      727   0       0             0 httpd
> [ 1460]    48  1460    89244      494   0       0             0 httpd
> [ 1461]    48  1461    89729      803   0       0             0 httpd
> [ 1462]    48  1462    88987     1055   0       0             0 httpd
> [ 1463]    48  1463    89796     2560   0       0             0 httpd
> [ 1464]    48  1464    91371     3558   0       0             0 httpd
> [ 1465]    48  1465    88596     1283   0       0             0 httpd
> [ 1466]    48  1466    90794     1253   0       0             0 httpd
> [ 1469]     0  1469   106619      198   0       0             0
> fail2ban-server
> [ 1497]     0  1497     1015        1   0       0             0 mingetty
> [ 1499]     0  1499     1015        1   0       0             0 mingetty
> [ 1501]     0  1501     1015        1   0       0             0 mingetty
> [ 1503]     0  1503     1015        1   0       0             0 mingetty
> [ 1505]     0  1505     1015        1   0       0             0 mingetty
> [ 1507]     0  1507     1015        1   0       0             0 mingetty
> [ 1512]     0  1512     2674        0   0     -17         -1000 udevd
> [ 1513]     0  1513     2674        0   0     -17         -1000 udevd
> [ 1514]     0  1514    14249        1   0       0             0 login
> [ 1520]    48  1520    89586     1139   0       0             0 httpd
> [ 1522]    48  1522    90606      773   0       0             0 httpd
> [ 1523]    48  1523    88343     2527   0       0             0 httpd
> [ 1524]     0  1524    27075        1   0       0             0 bash
> [ 1549]    48  1549    88937     1738   0       0             0 httpd
> [ 1550]    48  1550    89721     1736   0       0             0 httpd
> [ 1579]    48  1579    91106      600   0       0             0 httpd
> [ 1581]    48  1581    88218     1491   0       0             0 httpd
> [ 1582]    48  1582    88559      595   0       0             0 httpd
> [ 1755]    48  1755    88115     1716   0       0             0 httpd
> [ 2376]    48  2376    90860     1318   0       0             0 httpd
> [ 2562]    48  2562    89680     2348   0       0             0 httpd
> [ 2697]    48  2697    87846     2082   0       0             0 httpd
> [ 2709]    48  2709    86589     3201   0       0             0 httpd
> [ 2710]    48  2710    86056     2467   0       0             0 httpd
> [ 2713]    48  2713    84422     1068   0       0             0 httpd
> [ 2714]    48  2714    85544     2688   0       0             0 httpd
> [ 2715]    48  2715    84748     1505   0       0             0 httpd
> [ 2716]    48  2716    84569     1318   0       0             0 httpd
> [ 2721]    48  2721    84815     1718   0       0             0 httpd
> [ 2722]    48  2722    84502     1256   0       0             0 httpd
> [ 2723]    48  2723    87014     4123   0       0             0 httpd
> [ 2724]    48  2724    84502     1246   0       0             0 httpd
> [ 2725]    48  2725    84420     1203   0       0             0 httpd
> [ 2726]    48  2726    86782     4034   0       0             0 httpd
> [ 2727]    48  2727    84502     1262   0       0             0 httpd
> [ 2728]    48  2728    86054     2340   0       0             0 httpd
> [ 2737]    48  2737    84815     1743   0       0             0 httpd
> [ 2738]    48  2738    84748     1639   0       0             0 httpd
> [ 2739]    48  2739    86589     3300   0       0             0 httpd
> [ 2740]    48  2740    85202     2201   0       0             0 httpd
> [ 2741]    48  2741    86054     1742   0       0             0 httpd
> [ 2742]    48  2742    87014     4231   0       0             0 httpd
> [ 2743]    48  2743    84301      711   0       0             0 httpd
> [ 2744]    48  2744    84502     1233   0       0             0 httpd
> [ 2745]    48  2745    84815     1614   0       0             0 httpd
> [ 2746]    48  2746    84420     1113   0       0             0 httpd
> [ 2747]    48  2747    84432      913   0       0             0 httpd
> [ 2748]    48  2748    84301      724   0       0             0 httpd
> [ 2749]    48  2749    86251     3541   0       0             0 httpd
> [ 2750]    48  2750    84301      764   0       0             0 httpd
> [ 2751]    48  2751    84422     1004   0       0             0 httpd
> [ 2752]    48  2752    84291      588   0       0             0 httpd
> [ 2758]    48  2758    84301      687   0       0             0 httpd
> [ 2759]    48  2759    84301      725   0       0             0 httpd
> [ 2760]    48  2760    84432      890   0       0             0 httpd
> [ 2761]    48  2761    84301      788   0       0             0 httpd
> [ 2765]    48  2765    84420      930   0       0             0 httpd
> [ 2766]    48  2766    84430      868   0       0             0 httpd
> [ 2769]    48  2769    84432      946   0       0             0 httpd
> [ 2770]    48  2770    87526     8319   0       0             0 httpd
> [ 2774]    48  2774    84291      582   0       0             0 httpd
> [ 2775]    48  2775    84430      897   0       0             0 httpd
> [ 2779]    48  2779    84301      692   0       0             0 httpd
> [ 2780]    48  2780    84301      659   0       0             0 httpd
> [ 2781]    48  2781    84430     1098   0       0             0 httpd
> [ 2782]    48  2782    84420     2692   0       0             0 httpd
> [ 2783]    48  2783    84430      823   0       0             0 httpd
> [ 2784]    48  2784    84422     1062   0       0             0 httpd
> [ 2785]    48  2785    84432      899   0       0             0 httpd
> [ 2786]    48  2786    84502     1259   0       0             0 httpd
> [ 2787]    48  2787    84301      733   0       0             0 httpd
> [ 2788]    48  2788    84434      906   0       0             0 httpd
> [ 2789]    48  2789    84432     1000   0       0             0 httpd
> [ 2790]    48  2790    87526     8311   0       0             0 httpd
> [ 2791]    48  2791    84430     1067   0       0             0 httpd
> [ 2792]    48  2792    84504     1259   0       0             0 httpd
> [ 2793]    48  2793    84301      749   0       0             0 httpd
> [ 2794]    48  2794    84422      935   0       0             0 httpd
> [ 2795]    48  2795    84303      740   0       0             0 httpd
> [ 2796]    48  2796    84560     1417   0       0             0 httpd
> [ 2797]    48  2797    84420     1091   0       0             0 httpd
> [ 2798]    48  2798    84420     1023   0       0             0 httpd
> [ 2799]    48  2799    84301      678   0       0             0 httpd
> [ 2800]    48  2800    84301      643   0       0             0 httpd
> [ 2809]    48  2809    84422     1017   0       0             0 httpd
> [ 2811]    48  2811    84301      643   0       0             0 httpd
> [ 2812]    48  2812    84301      620   0       0             0 httpd
> [ 2813]    48  2813    84432      958   0       0             0 httpd
> [ 2814]    48  2814    84432      928   0       0             0 httpd
> [ 2815]    48  2815    84420     1126   0       0             0 httpd
> [ 2816]    48  2816    87014     7940   0       0             0 httpd
> [ 2817]    48  2817    84301      725   0       0             0 httpd
> [ 2818]    48  2818    84301      695   0       0             0 httpd
> [ 2819]    48  2819    84432      800   0       0             0 httpd
> [ 2820]    48  2820    84420     1209   0       0             0 httpd
> [ 2821]    48  2821    84502     1351   0       0             0 httpd
> [ 2822]    48  2822    84815     1684   0       0             0 httpd
> [ 2823]    48  2823    84301      702   0       0             0 httpd
> [ 2824]    48  2824    84430     1416   0       0             0 httpd
> [ 2825]    48  2825    84432      824   0       0             0 httpd
> [ 2826]    48  2826    84432      899   0       0             0 httpd
> [ 2827]    48  2827    84432      884   0       0             0 httpd
> [ 2828]    48  2828    84301      794   0       0             0 httpd
> [ 2829]    48  2829    84301      723   0       0             0 httpd
> [ 2830]    48  2830    84301     1413   0       0             0 httpd
> [ 2831]    48  2831    85331     3528   0       0             0 httpd
> [ 2832]    48  2832    84420     3300   0       0             0 httpd
> [ 2833]    48  2833    84430     1353   0       0             0 httpd
> [ 2834]    48  2834    84432      897   0       0             0 httpd
> [ 2835]    48  2835    84291      572   0       0             0 httpd
> [ 2836]    48  2836    85460     3295   0       0             0 httpd
> [ 2852]    48  2852    83773     4497   0       0             0 httpd
> [ 2853]    48  2853    84814     2177   0       0             0 httpd
> [ 2857]    48  2857    84420     1140   0       0             0 httpd
> [ 2858]    48  2858    84432      925   0       0             0 httpd
> [ 2859]    48  2859    84485     1568   0       0             0 httpd
> [ 2860]    48  2860    84552     1749   0       0             0 httpd
> [ 2861]    48  2861    84291      582   0       0             0 httpd
> [ 2862]    48  2862    84420     3627   0       0             0 httpd
> [ 2868]    48  2868    84301      804   0       0             0 httpd
> [ 2869]    48  2869    84487     1268   0       0             0 httpd
> [ 2870]    48  2870    84301      803   0       0             0 httpd
> [ 2874]    48  2874    84422     1033   0       0             0 httpd
> [ 2875]    48  2875    84301      718   0       0             0 httpd
> [ 2876]    48  2876    84485     1125   0       0             0 httpd
> [ 2877]    48  2877    84420      924   0       0             0 httpd
> [ 2878]    48  2878    84748     2217   0       0             0 httpd
> [ 2885]    48  2885    84301     2638   0       0             0 httpd
> [ 2886]    48  2886    84420      905   0       0             0 httpd
> [ 2887]    48  2887    84420     1558   0       0             0 httpd
> [ 2888]    48  2888    84301      697   0       0             0 httpd
> [ 2889]    48  2889    84502     1228   0       0             0 httpd
> [ 2904]    48  2904    84430      791   0       0             0 httpd
> [ 2915]    48  2915    84748     1518   0       0             0 httpd
> [ 2916]    48  2916    85331     3691   0       0             0 httpd
> [ 2917]    48  2917    84569     1634   0       0             0 httpd
> [ 2918]    48  2918    84239     4498   0       0             0 httpd
> [ 2919]    48  2919    84502     2969   0       0             0 httpd
> [ 2920]    48  2920    86763     4460   0       0             0 httpd
> [ 2921]    48  2921    84420     1133   0       0             0 httpd
> [ 2922]    48  2922    86381     5779   0       0             0 httpd
> [ 2928]    48  2928    84502     2436   0       0             0 httpd
> [ 2938]    48  2938    85208     2426   0       0             0 httpd
> [ 2939]    48  2939    86055     5438   0       0             0 httpd
> [ 2940]    48  2940    84487     4186   0       0             0 httpd
> [ 2944]    48  2944    84432     2877   0       0             0 httpd
> [ 2945]    48  2945    84291     1446   0       0             0 httpd
> [ 2946]    48  2946    84818     3977   0       0             0 httpd
> [ 2960]    48  2960    84239     4311   0       0             0 httpd
> [ 2961]    48  2961    89636     9983   0       0             0 httpd
> [ 2962]    48  2962    87955     8183   0       0             0 httpd
> [ 2970]    48  2970    84241     4777   0       0             0 httpd
> [ 2973]    48  2973    84239     4759   0       0             0 httpd
> [ 2974]    48  2974    80053     1296   0       0             0 httpd
> [ 2975]    48  2975    80012     1410   0       0             0 httpd
> [ 2976]    48  2976    80185     1606   0       0             0 httpd
> [ 2978]    48  2978    79284      414   0       0             0 httpd
> [ 2980]    48  2980    80185     1387   0       0             0 httpd
> [ 2981]    48  2981    80021     1076   0       0             0 httpd
> [ 2982]    48  2982    80181     1383   0       0             0 httpd
> [ 2983]    48  2983    80138     1348   0       0             0 httpd
> [ 2984]    48  2984    80202     1273   0       0             0 httpd
> [ 2985]    48  2985    80010      418   0       0             0 httpd
> [ 2986]    48  2986    79990     1409   0       0             0 httpd
> [ 2988]    48  2988    80202     1540   0       0             0 httpd
> [ 2989]    48  2989    79996      756   0       0             0 httpd
> [ 2990]    48  2990    80010     1412   0       0             0 httpd
> [ 2991]    48  2991    79986      789   0       0             0 httpd
> [ 2992]    48  2992    79284      401   0       0             0 httpd
> [ 2993]    48  2993    80189     1562   0       0             0 httpd
> [ 2994]    48  2994    80119     1316   0       0             0 httpd
> [ 3010]    48  3010    79283      425   0       0             0 httpd
> [ 3013]    48  3013    80046     1291   0       0             0 httpd
> [ 3016]    48  3016    79250      412   0       0             0 httpd
> [ 3018]    48  3018    79283      451   0       0             0 httpd
> [ 3022]     0  3022    33480       63   0       0             0 crond
> Out of memory: Kill process 1333 (mysqld) score 12 or sacrifice child
> Killed process 1333, UID 27, (mysqld) total-vm:748444kB,
> anon-rss:19232kB, file-rss:68kB
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>