Hi list,
We are experiencing a memory leak on our SquirrelMail server since the 5.3 update. The server is fully updated, only stock rpms. The httpd processes are eating all the memory and after swapping like hell, the server became unresponsive and we must hard-reboot it. The server is not that much loaded (max 10-15 concurrent users but with tons of mail in their inbox). Configuration's speaking, nothing was changed before and after the update.
Is there someone else experiencing the same thing ? How can I search deeper the origin of the leak ?
Thx, kfx
Details follows:
Screenshot from munin showing the memory usage's graph: http://uppix.net/e/a/7/cd2bdea933d57b0e28bde1f96b854.png
[root@webmail ~]# uname -a Linux webmail 2.6.18-128.1.6.el5xen #1 SMP Wed Apr 1 09:53:14 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@webmail ~]# yum list | grep -e php -e squirrel -e mysql -e httpd | grep installed httpd.x86_64 2.2.3-22.el5.centos installed mysql.x86_64 5.0.45-7.el5 installed mysql-server.x86_64 5.0.45-7.el5 installed php.x86_64 5.1.6-23.el5 installed php-cli.x86_64 5.1.6-23.el5 installed php-common.x86_64 5.1.6-23.el5 installed php-ldap.x86_64 5.1.6-23.el5 installed php-mbstring.x86_64 5.1.6-23.el5 installed php-mysql.x86_64 5.1.6-23.el5 installed php-pdo.x86_64 5.1.6-23.el5 installed php-pear.noarch 1:1.4.9-4.el5.1 installed php-pear-DB.noarch 1.7.13-1.el5.centos installed php-pear-MDB2.noarch 2.4.1-2.el5.centos installed php-pear-MDB2-Driver-mysql.noarch 1.4.1-3.el5.centos installed squirrelmail.noarch 1.4.8-5.el5.centos.3 installed
Dmesg:
Apr 2 17:18:28 s_local@webmail kernel:init invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Call Trace: Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Mem-info: Apr 2 17:18:28 s_local@webmail kernel:DMA per-cpu: Apr 2 17:18:28 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:84 Apr 2 17:18:28 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 17:18:28 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Normal per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Free pages: 2912kB (0kB HighMem) Apr 2 17:18:28 s_local@webmail kernel:Active:73144 inactive:40971 dirty:0 writeback:0 unstable:0 free:728 slab:4076 mapped-file:2 mapped-anon:114411 pagetables:2857 Apr 2 17:18:28 s_local@webmail kernel:DMA free:2912kB min:2916kB low:3644kB high:4372kB active:292576kB inactive:163884kB present:532480kB pages_scanned:1521701 all_unreclaimable? yes Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB Apr 2 17:18:28 s_local@webmail kernel:DMA32: empty Apr 2 17:18:28 s_local@webmail kernel:Normal: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem: empty Apr 2 17:18:28 s_local@webmail kernel:89 pagecache pages Apr 2 17:18:28 s_local@webmail kernel:Swap cache: add 176874, delete 176874, find 114990/119550, race 0+0 Apr 2 17:18:28 s_local@webmail kernel:Free swap = 0kB Apr 2 17:18:28 s_local@webmail kernel:Total swap = 557048kB Apr 2 17:18:28 s_local@webmail kernel:Free swap: 0kB Apr 2 17:18:28 s_local@webmail kernel:133120 pages of RAM Apr 2 17:18:28 s_local@webmail kernel:5993 reserved pages Apr 2 17:18:28 s_local@webmail kernel:4267 pages shared Apr 2 17:18:28 s_local@webmail kernel:0 pages swap cached Apr 2 17:18:28 s_local@webmail kernel:Out of memory: Killed process 1592 (httpd). Apr 2 17:18:28 s_local@webmail kernel:automount invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Call Trace: Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020622a>] hypercall_page+0x22a/0x1000 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020622a>] hypercall_page+0x22a/0x1000 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Mem-info: Apr 2 17:18:28 s_local@webmail kernel:DMA per-cpu: Apr 2 17:18:28 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:88 Apr 2 17:18:28 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 17:18:28 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Normal per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Free pages: 2912kB (0kB HighMem) Apr 2 17:18:28 s_local@webmail kernel:Active:73160 inactive:40987 dirty:0 writeback:0 unstable:0 free:728 slab:4073 mapped-file:2 mapped-anon:114411 pagetables:2857 Apr 2 17:18:28 s_local@webmail kernel:DMA free:2912kB min:2916kB low:3644kB high:4372kB active:292640kB inactive:163948kB present:532480kB pages_scanned:5185565 all_unreclaimable? yes Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB Apr 2 17:18:28 s_local@webmail kernel:DMA32: empty Apr 2 17:18:28 s_local@webmail kernel:Normal: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem: empty Apr 2 17:18:28 s_local@webmail kernel:89 pagecache pages Apr 2 17:18:28 s_local@webmail kernel:Swap cache: add 176874, delete 176874, find 114990/119550, race 0+0 Apr 2 17:18:28 s_local@webmail kernel:Free swap = 0kB Apr 2 17:18:28 s_local@webmail kernel:Total swap = 557048kB Apr 2 17:18:28 s_local@webmail kernel:Free swap: 0kB Apr 2 17:18:28 s_local@webmail kernel:133120 pages of RAM Apr 2 17:18:28 s_local@webmail kernel:5993 reserved pages Apr 2 17:18:28 s_local@webmail kernel:4235 pages shared Apr 2 17:18:28 s_local@webmail kernel:0 pages swap cached Apr 2 17:18:28 s_local@webmail kernel:audispd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Call Trace: Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Mem-info: Apr 2 17:18:28 s_local@webmail kernel:DMA per-cpu: Apr 2 17:18:28 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:98 Apr 2 17:18:28 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 17:18:28 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Normal per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Free pages: 2912kB (0kB HighMem) Apr 2 17:18:28 s_local@webmail kernel:Active:73115 inactive:40968 dirty:0 writeback:0 unstable:0 free:728 slab:4062 mapped-file:2 mapped-anon:114411 pagetables:2857 Apr 2 17:18:28 s_local@webmail kernel:DMA free:2912kB min:2916kB low:3644kB high:4372kB active:292460kB inactive:163872kB present:532480kB pages_scanned:8730229 all_unreclaimable? yes Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB Apr 2 17:18:28 s_local@webmail kernel:DMA32: empty Apr 2 17:18:28 s_local@webmail kernel:Normal: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem: empty Apr 2 17:18:28 s_local@webmail kernel:89 pagecache pages Apr 2 17:18:28 s_local@webmail kernel:Swap cache: add 176874, delete 176874, find 114990/119550, race 0+0 Apr 2 17:18:28 s_local@webmail kernel:Free swap = 0kB Apr 2 17:18:28 s_local@webmail kernel:Total swap = 557048kB Apr 2 17:18:28 s_local@webmail kernel:Free swap: 0kB Apr 2 17:18:28 s_local@webmail kernel:133120 pages of RAM Apr 2 17:18:28 s_local@webmail kernel:5993 reserved pages Apr 2 17:18:28 s_local@webmail kernel:4299 pages shared Apr 2 17:18:28 s_local@webmail kernel:0 pages swap cached Apr 2 17:18:28 s_local@webmail kernel:kthread invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0 Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Call Trace: Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025e012>] cache_alloc_refill+0x269/0x4ba Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020afbc>] kmem_cache_alloc+0x50/0x6d Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8021ef3f>] copy_process+0xa6/0x1a48 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8020afce>] kmem_cache_alloc+0x62/0x6d Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80263a0d>] _spin_lock_irq+0x9/0x14 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80297d10>] alloc_pid+0x26c/0x292 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80231fcf>] do_fork+0x69/0x1c1 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80299dc0>] keventd_create_kthread+0x0/0xc4 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025fac5>] kernel_thread+0x81/0xde Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80299dc0>] keventd_create_kthread+0x0/0xc4 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8026df02>] monotonic_clock+0x35/0x7b Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80233378>] kthread+0x0/0x132 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025fb22>] child_rip+0x0/0x12 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80299ddd>] keventd_create_kthread+0x1d/0xc4 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8022ec93>] __wake_up+0x38/0x4f Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8024ee11>] run_workqueue+0x94/0xe4 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8024b71a>] worker_thread+0x0/0x122 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8024b80a>] worker_thread+0xf0/0x122 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80286daf>] default_wake_function+0x0/0xe Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80233476>] kthread+0xfe/0x132 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025fb2c>] child_rip+0xa/0x12 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff80233378>] kthread+0x0/0x132 Apr 2 17:18:28 s_local@webmail kernel:[<ffffffff8025fb22>] child_rip+0x0/0x12 Apr 2 17:18:28 s_local@webmail kernel: Apr 2 17:18:28 s_local@webmail kernel:Mem-info: Apr 2 17:18:28 s_local@webmail kernel:DMA per-cpu: Apr 2 17:18:28 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:102 Apr 2 17:18:28 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 17:18:28 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Normal per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 17:18:28 s_local@webmail kernel:Free pages: 2912kB (0kB HighMem) Apr 2 17:18:28 s_local@webmail kernel:Active:73053 inactive:40998 dirty:0 writeback:0 unstable:0 free:728 slab:4059 mapped-file:2 mapped-anon:114411 pagetables:2857 Apr 2 17:18:28 s_local@webmail kernel:DMA free:2912kB min:2916kB low:3644kB high:4372kB active:292212kB inactive:163992kB present:532480kB pages_scanned:12263677 all_unreclaimable? yes Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 17:18:28 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 17:18:28 s_local@webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB Apr 2 17:18:28 s_local@webmail kernel:DMA32: empty Apr 2 17:18:28 s_local@webmail kernel:Normal: empty Apr 2 17:18:28 s_local@webmail kernel:HighMem: empty Apr 2 17:18:28 s_local@webmail kernel:89 pagecache pages Apr 2 17:18:28 s_local@webmail kernel:Swap cache: add 176874, delete 176874, find 114990/119550, race 0+0 Apr 2 17:18:28 s_local@webmail kernel:Free swap = 0kB Apr 2 17:18:28 s_local@webmail kernel:Total swap = 557048kB Apr 2 17:18:28 s_local@webmail kernel:Free swap: 0kB Apr 2 17:18:28 s_local@webmail kernel:133120 pages of RAM Apr 2 17:18:28 s_local@webmail kernel:5993 reserved pages Apr 2 17:18:28 s_local@webmail kernel:4331 pages shared Apr 2 17:18:28 s_local@webmail kernel:0 pages swap cached Apr 2 18:05:47 s_local@webmail kernel:init invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 18:05:47 s_local@webmail kernel: Apr 2 18:05:47 s_local@webmail kernel:Call Trace: Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8020622a>] hypercall_page+0x22a/0x1000 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 18:05:47 s_local@webmail kernel: Apr 2 18:05:47 s_local@webmail kernel:Mem-info: Apr 2 18:05:47 s_local@webmail kernel:DMA per-cpu: Apr 2 18:05:47 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:12 Apr 2 18:05:47 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 18:05:47 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:Normal per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:Free pages: 2868kB (0kB HighMem) Apr 2 18:05:47 s_local@webmail kernel:Active:57202 inactive:56889 dirty:0 writeback:0 unstable:0 free:717 slab:4124 mapped-file:2 mapped-anon:114137 pagetables:3184 Apr 2 18:05:47 s_local@webmail kernel:DMA free:2868kB min:2916kB low:3644kB high:4372kB active:228808kB inactive:227556kB present:532480kB pages_scanned:708700 all_unreclaimable? yes Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:DMA: 31*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2868kB Apr 2 18:05:47 s_local@webmail kernel:DMA32: empty Apr 2 18:05:47 s_local@webmail kernel:Normal: empty Apr 2 18:05:47 s_local@webmail kernel:HighMem: empty Apr 2 18:05:47 s_local@webmail kernel:89 pagecache pages Apr 2 18:05:47 s_local@webmail kernel:Swap cache: add 248675, delete 248675, find 135538/146503, race 0+0 Apr 2 18:05:47 s_local@webmail kernel:Free swap = 0kB Apr 2 18:05:47 s_local@webmail kernel:Total swap = 557048kB Apr 2 18:05:47 s_local@webmail kernel:Free swap: 0kB Apr 2 18:05:47 s_local@webmail kernel:133120 pages of RAM Apr 2 18:05:47 s_local@webmail kernel:5993 reserved pages Apr 2 18:05:47 s_local@webmail kernel:4606 pages shared Apr 2 18:05:47 s_local@webmail kernel:0 pages swap cached Apr 2 18:05:47 s_local@webmail kernel:Out of memory: Killed process 21335 (httpd). Apr 2 18:05:47 s_local@webmail kernel:init invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 18:05:47 s_local@webmail kernel: Apr 2 18:05:47 s_local@webmail kernel:Call Trace: Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8020622a>] hypercall_page+0x22a/0x1000 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 18:05:47 s_local@webmail kernel: Apr 2 18:05:47 s_local@webmail kernel:Mem-info: Apr 2 18:05:47 s_local@webmail kernel:DMA per-cpu: Apr 2 18:05:47 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:19 Apr 2 18:05:47 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 18:05:47 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:Normal per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:Free pages: 2868kB (0kB HighMem) Apr 2 18:05:47 s_local@webmail kernel:Active:73107 inactive:40952 dirty:0 writeback:0 unstable:0 free:717 slab:4118 mapped-file:2 mapped-anon:114137 pagetables:3184 Apr 2 18:05:47 s_local@webmail kernel:DMA free:2868kB min:2916kB low:3644kB high:4372kB active:292428kB inactive:163808kB present:532480kB pages_scanned:4108148 all_unreclaimable? yes Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:DMA: 31*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2868kB Apr 2 18:05:47 s_local@webmail kernel:DMA32: empty Apr 2 18:05:47 s_local@webmail kernel:Normal: empty Apr 2 18:05:47 s_local@webmail kernel:HighMem: empty Apr 2 18:05:47 s_local@webmail kernel:83 pagecache pages Apr 2 18:05:47 s_local@webmail kernel:Swap cache: add 248675, delete 248675, find 135538/146503, race 0+0 Apr 2 18:05:47 s_local@webmail kernel:Free swap = 0kB Apr 2 18:05:47 s_local@webmail kernel:Total swap = 557048kB Apr 2 18:05:47 s_local@webmail kernel:Free swap: 0kB Apr 2 18:05:47 s_local@webmail kernel:133120 pages of RAM Apr 2 18:05:47 s_local@webmail kernel:5993 reserved pages Apr 2 18:05:47 s_local@webmail kernel:4638 pages shared Apr 2 18:05:47 s_local@webmail kernel:0 pages swap cached Apr 2 18:05:47 s_local@webmail kernel:init invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 18:05:47 s_local@webmail kernel: Apr 2 18:05:47 s_local@webmail kernel:Call Trace: Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8020622a>] hypercall_page+0x22a/0x1000 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 18:05:47 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 18:05:47 s_local@webmail kernel: Apr 2 18:05:47 s_local@webmail kernel:Mem-info: Apr 2 18:05:47 s_local@webmail kernel:DMA per-cpu: Apr 2 18:05:47 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:26 Apr 2 18:05:47 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:57 Apr 2 18:05:47 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:Normal per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 18:05:47 s_local@webmail kernel:Free pages: 2868kB (0kB HighMem) Apr 2 18:05:47 s_local@webmail kernel:Active:72986 inactive:41009 dirty:0 writeback:0 unstable:0 free:717 slab:4111 mapped-file:2 mapped-anon:114137 pagetables:3184 Apr 2 18:05:47 s_local@webmail kernel:DMA free:2868kB min:2916kB low:3644kB high:4372kB active:291944kB inactive:164036kB present:532480kB pages_scanned:7458532 all_unreclaimable? yes Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:05:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:05:47 s_local@webmail kernel:DMA: 31*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2868kB Apr 2 18:05:47 s_local@webmail kernel:DMA32: empty Apr 2 18:05:47 s_local@webmail kernel:Normal: empty Apr 2 18:05:47 s_local@webmail kernel:HighMem: empty Apr 2 18:05:47 s_local@webmail kernel:83 pagecache pages Apr 2 18:05:47 s_local@webmail kernel:Swap cache: add 248675, delete 248675, find 135538/146503, race 0+0 Apr 2 18:05:47 s_local@webmail kernel:Free swap = 0kB Apr 2 18:05:47 s_local@webmail kernel:Total swap = 557048kB Apr 2 18:05:47 s_local@webmail kernel:Free swap: 0kB Apr 2 18:05:47 s_local@webmail kernel:133120 pages of RAM Apr 2 18:05:47 s_local@webmail kernel:5993 reserved pages Apr 2 18:05:47 s_local@webmail kernel:4702 pages shared Apr 2 18:05:47 s_local@webmail kernel:0 pages swap cached Apr 2 18:14:47 s_local@webmail kernel:httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 18:14:47 s_local@webmail kernel: Apr 2 18:14:47 s_local@webmail kernel:Call Trace: Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff80299fd8>] autoremove_wake_function+0x0/0x2e Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff803164c8>] file_has_perm+0x48/0xa3 Apr 2 18:14:47 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 18:14:47 s_local@webmail kernel: Apr 2 18:14:47 s_local@webmail kernel:Mem-info: Apr 2 18:14:47 s_local@webmail kernel:DMA per-cpu: Apr 2 18:14:47 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:25 Apr 2 18:14:47 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:14 Apr 2 18:14:47 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 18:14:47 s_local@webmail kernel:Normal per-cpu: empty Apr 2 18:14:47 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 18:14:47 s_local@webmail kernel:Free pages: 2876kB (0kB HighMem) Apr 2 18:14:47 s_local@webmail kernel:Active:54916 inactive:59014 dirty:0 writeback:0 unstable:0 free:719 slab:4154 mapped-file:5 mapped-anon:113909 pagetables:3325 Apr 2 18:14:47 s_local@webmail kernel:DMA free:2876kB min:2916kB low:3644kB high:4372kB active:219664kB inactive:236056kB present:532480kB pages_scanned:738327 all_unreclaimable? yes Apr 2 18:14:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:14:47 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:14:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:14:47 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:14:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:14:47 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:14:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:14:47 s_local@webmail kernel:DMA: 1*4kB 11*8kB 0*16kB 3*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2876kB Apr 2 18:14:47 s_local@webmail kernel:DMA32: empty Apr 2 18:14:47 s_local@webmail kernel:Normal: empty Apr 2 18:14:47 s_local@webmail kernel:HighMem: empty Apr 2 18:14:47 s_local@webmail kernel:126 pagecache pages Apr 2 18:14:47 s_local@webmail kernel:Swap cache: add 256186, delete 256186, find 137948/149847, race 0+0 Apr 2 18:14:47 s_local@webmail kernel:Free swap = 0kB Apr 2 18:14:47 s_local@webmail kernel:Total swap = 557048kB Apr 2 18:14:47 s_local@webmail kernel:Free swap: 0kB Apr 2 18:14:47 s_local@webmail kernel:133120 pages of RAM Apr 2 18:14:47 s_local@webmail kernel:5993 reserved pages Apr 2 18:14:47 s_local@webmail kernel:4863 pages shared Apr 2 18:14:47 s_local@webmail kernel:0 pages swap cached Apr 2 18:14:47 s_local@webmail kernel:Out of memory: Killed process 21332 (httpd). Apr 2 18:23:47 s_local@webmail kernel:automount invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Apr 2 18:23:47 s_local@webmail kernel: Apr 2 18:23:47 s_local@webmail kernel:Call Trace: Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff802bc998>] out_of_memory+0x8b/0x203 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff8020f657>] __alloc_pages+0x245/0x2ce Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff8021336e>] __do_page_cache_readahead+0xd0/0x21c Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff80262824>] __wait_on_bit_lock+0x5b/0x66 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff88081d4d>] :dm_mod:dm_any_congested+0x38/0x3f Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff80213c47>] filemap_nopage+0x148/0x322 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff80208db9>] __handle_mm_fault+0x440/0x11f6 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff802639f9>] _spin_lock_irqsave+0x9/0x14 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff802666ef>] do_page_fault+0xf7b/0x12e0 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff802ad687>] audit_syscall_entry+0xf2/0x1a1 Apr 2 18:23:47 s_local@webmail kernel:[<ffffffff8025f82b>] error_exit+0x0/0x6e Apr 2 18:23:47 s_local@webmail kernel: Apr 2 18:23:47 s_local@webmail kernel:Mem-info: Apr 2 18:23:47 s_local@webmail kernel:DMA per-cpu: Apr 2 18:23:47 s_local@webmail kernel:cpu 0 hot: high 186, batch 31 used:30 Apr 2 18:23:47 s_local@webmail kernel:cpu 0 cold: high 62, batch 15 used:14 Apr 2 18:23:47 s_local@webmail kernel:DMA32 per-cpu: empty Apr 2 18:23:47 s_local@webmail kernel:Normal per-cpu: empty Apr 2 18:23:47 s_local@webmail kernel:HighMem per-cpu: empty Apr 2 18:23:47 s_local@webmail kernel:Free pages: 2908kB (0kB HighMem) Apr 2 18:23:47 s_local@webmail kernel:Active:56901 inactive:56963 dirty:0 writeback:0 unstable:0 free:727 slab:4164 mapped-file:6 mapped-anon:113966 pagetables:3306 Apr 2 18:23:47 s_local@webmail kernel:DMA free:2908kB min:2916kB low:3644kB high:4372kB active:227604kB inactive:227852kB present:532480kB pages_scanned:807511 all_unreclaimable? yes Apr 2 18:23:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:23:47 s_local@webmail kernel:DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:23:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:23:47 s_local@webmail kernel:Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:23:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:23:47 s_local@webmail kernel:HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no Apr 2 18:23:47 s_local@webmail kernel:lowmem_reserve[]: 0 0 0 0 Apr 2 18:23:47 s_local@webmail kernel:DMA: 37*4kB 1*8kB 0*16kB 2*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2908kB Apr 2 18:23:47 s_local@webmail kernel:DMA32: empty Apr 2 18:23:47 s_local@webmail kernel:Normal: empty Apr 2 18:23:47 s_local@webmail kernel:HighMem: empty Apr 2 18:23:47 s_local@webmail kernel:147 pagecache pages Apr 2 18:23:47 s_local@webmail kernel:Swap cache: add 267431, delete 267431, find 138909/152168, race 0+0 Apr 2 18:23:47 s_local@webmail kernel:Free swap = 0kB Apr 2 18:23:47 s_local@webmail kernel:Total swap = 557048kB Apr 2 18:23:47 s_local@webmail kernel:Free swap: 0kB Apr 2 18:23:47 s_local@webmail kernel:133120 pages of RAM Apr 2 18:23:47 s_local@webmail kernel:5993 reserved pages Apr 2 18:23:47 s_local@webmail kernel:4965 pages shared Apr 2 18:23:47 s_local@webmail kernel:0 pages swap cached Apr 2 18:23:47 s_local@webmail kernel:Out of memory: Killed process 21336 (httpd).