[CentOS] Memory Leak with stock Squirrelmail, PHP, mysql, apache since 5.3

Fri Apr 3 16:48:16 UTC 2009
kfx <kadafax at gmail.com>

Hi list,

We are experiencing a memory leak on our SquirrelMail server since the 
5.3 update.
The server is fully updated, only stock rpms. The httpd processes are 
eating all the memory and after swapping like hell, the server became 
unresponsive and we must hard-reboot it.
The server is not that much loaded (max 10-15 concurrent users but with 
tons of mail in their inbox). Configuration's speaking, nothing was 
changed before and after the update.

Is there someone else experiencing the same thing ? How can I search 
deeper the origin of the leak ?

Thx,
kfx

Details follows:

Screenshot from munin showing the memory usage's graph:
http://uppix.net/e/a/7/cd2bdea933d57b0e28bde1f96b854.png

[root at webmail ~]# uname -a
Linux webmail 2.6.18-128.1.6.el5xen #1 SMP Wed Apr 1 09:53:14 EDT 2009 
x86_64 x86_64 x86_64 GNU/Linux

[root at webmail ~]# yum list | grep -e php -e squirrel -e mysql -e httpd | 
grep installed
httpd.x86_64                               2.2.3-22.el5.centos         
installed
mysql.x86_64                               5.0.45-7.el5                
installed
mysql-server.x86_64                        5.0.45-7.el5                
installed
php.x86_64                                 5.1.6-23.el5                
installed
php-cli.x86_64                             5.1.6-23.el5                
installed
php-common.x86_64                          5.1.6-23.el5                
installed
php-ldap.x86_64                            5.1.6-23.el5                
installed
php-mbstring.x86_64                        5.1.6-23.el5                
installed
php-mysql.x86_64                           5.1.6-23.el5                
installed
php-pdo.x86_64                             5.1.6-23.el5                
installed
php-pear.noarch                            1:1.4.9-4.el5.1             
installed
php-pear-DB.noarch                         1.7.13-1.el5.centos         
installed
php-pear-MDB2.noarch                       2.4.1-2.el5.centos          
installed
php-pear-MDB2-Driver-mysql.noarch          1.4.1-3.el5.centos          
installed
squirrelmail.noarch                        1.4.8-5.el5.centos.3        
installed


Dmesg:

Apr  2 17:18:28 s_local at webmail kernel:init invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Call Trace:
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Mem-info:
Apr  2 17:18:28 s_local at webmail kernel:DMA per-cpu:
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:84
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 17:18:28 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Free pages:        2912kB (0kB 
HighMem)
Apr  2 17:18:28 s_local at webmail kernel:Active:73144 inactive:40971 
dirty:0 writeback:0 unstable:0 free:728 slab:4076 mapped-file:2 
mapped-anon:114411 pagetables:2857
Apr  2 17:18:28 s_local at webmail kernel:DMA free:2912kB min:2916kB 
low:3644kB high:4372kB active:292576kB inactive:163884kB 
present:532480kB pages_scanned:1521701 all_unreclaimable? yes
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 
1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB
Apr  2 17:18:28 s_local at webmail kernel:DMA32: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem: empty
Apr  2 17:18:28 s_local at webmail kernel:89 pagecache pages
Apr  2 17:18:28 s_local at webmail kernel:Swap cache: add 176874, delete 
176874, find 114990/119550, race 0+0
Apr  2 17:18:28 s_local at webmail kernel:Free swap  = 0kB
Apr  2 17:18:28 s_local at webmail kernel:Total swap = 557048kB
Apr  2 17:18:28 s_local at webmail kernel:Free swap:            0kB
Apr  2 17:18:28 s_local at webmail kernel:133120 pages of RAM
Apr  2 17:18:28 s_local at webmail kernel:5993 reserved pages
Apr  2 17:18:28 s_local at webmail kernel:4267 pages shared
Apr  2 17:18:28 s_local at webmail kernel:0 pages swap cached
Apr  2 17:18:28 s_local at webmail kernel:Out of memory: Killed process 
1592 (httpd).
Apr  2 17:18:28 s_local at webmail kernel:automount invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Call Trace:
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020622a>] 
hypercall_page+0x22a/0x1000
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020622a>] 
hypercall_page+0x22a/0x1000
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Mem-info:
Apr  2 17:18:28 s_local at webmail kernel:DMA per-cpu:
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:88
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 17:18:28 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Free pages:        2912kB (0kB 
HighMem)
Apr  2 17:18:28 s_local at webmail kernel:Active:73160 inactive:40987 
dirty:0 writeback:0 unstable:0 free:728 slab:4073 mapped-file:2 
mapped-anon:114411 pagetables:2857
Apr  2 17:18:28 s_local at webmail kernel:DMA free:2912kB min:2916kB 
low:3644kB high:4372kB active:292640kB inactive:163948kB 
present:532480kB pages_scanned:5185565 all_unreclaimable? yes
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 
1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB
Apr  2 17:18:28 s_local at webmail kernel:DMA32: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem: empty
Apr  2 17:18:28 s_local at webmail kernel:89 pagecache pages
Apr  2 17:18:28 s_local at webmail kernel:Swap cache: add 176874, delete 
176874, find 114990/119550, race 0+0
Apr  2 17:18:28 s_local at webmail kernel:Free swap  = 0kB
Apr  2 17:18:28 s_local at webmail kernel:Total swap = 557048kB
Apr  2 17:18:28 s_local at webmail kernel:Free swap:            0kB
Apr  2 17:18:28 s_local at webmail kernel:133120 pages of RAM
Apr  2 17:18:28 s_local at webmail kernel:5993 reserved pages
Apr  2 17:18:28 s_local at webmail kernel:4235 pages shared
Apr  2 17:18:28 s_local at webmail kernel:0 pages swap cached
Apr  2 17:18:28 s_local at webmail kernel:audispd invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Call Trace:
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Mem-info:
Apr  2 17:18:28 s_local at webmail kernel:DMA per-cpu:
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:98
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 17:18:28 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Free pages:        2912kB (0kB 
HighMem)
Apr  2 17:18:28 s_local at webmail kernel:Active:73115 inactive:40968 
dirty:0 writeback:0 unstable:0 free:728 slab:4062 mapped-file:2 
mapped-anon:114411 pagetables:2857
Apr  2 17:18:28 s_local at webmail kernel:DMA free:2912kB min:2916kB 
low:3644kB high:4372kB active:292460kB inactive:163872kB 
present:532480kB pages_scanned:8730229 all_unreclaimable? yes
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 
1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB
Apr  2 17:18:28 s_local at webmail kernel:DMA32: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem: empty
Apr  2 17:18:28 s_local at webmail kernel:89 pagecache pages
Apr  2 17:18:28 s_local at webmail kernel:Swap cache: add 176874, delete 
176874, find 114990/119550, race 0+0
Apr  2 17:18:28 s_local at webmail kernel:Free swap  = 0kB
Apr  2 17:18:28 s_local at webmail kernel:Total swap = 557048kB
Apr  2 17:18:28 s_local at webmail kernel:Free swap:            0kB
Apr  2 17:18:28 s_local at webmail kernel:133120 pages of RAM
Apr  2 17:18:28 s_local at webmail kernel:5993 reserved pages
Apr  2 17:18:28 s_local at webmail kernel:4299 pages shared
Apr  2 17:18:28 s_local at webmail kernel:0 pages swap cached
Apr  2 17:18:28 s_local at webmail kernel:kthread invoked oom-killer: 
gfp_mask=0xd0, order=0, oomkilladj=0
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Call Trace:
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025e012>] 
cache_alloc_refill+0x269/0x4ba
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020afbc>] 
kmem_cache_alloc+0x50/0x6d
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8021ef3f>] 
copy_process+0xa6/0x1a48
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8020afce>] 
kmem_cache_alloc+0x62/0x6d
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80263a0d>] 
_spin_lock_irq+0x9/0x14
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80297d10>] 
alloc_pid+0x26c/0x292
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80231fcf>] 
do_fork+0x69/0x1c1
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80299dc0>] 
keventd_create_kthread+0x0/0xc4
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025fac5>] 
kernel_thread+0x81/0xde
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80299dc0>] 
keventd_create_kthread+0x0/0xc4
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8026df02>] 
monotonic_clock+0x35/0x7b
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80233378>] 
kthread+0x0/0x132
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025fb22>] 
child_rip+0x0/0x12
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80299ddd>] 
keventd_create_kthread+0x1d/0xc4
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8022ec93>] 
__wake_up+0x38/0x4f
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8024ee11>] 
run_workqueue+0x94/0xe4
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8024b71a>] 
worker_thread+0x0/0x122
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8024b80a>] 
worker_thread+0xf0/0x122
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80286daf>] 
default_wake_function+0x0/0xe
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80233476>] 
kthread+0xfe/0x132
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025fb2c>] 
child_rip+0xa/0x12
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff80233378>] 
kthread+0x0/0x132
Apr  2 17:18:28 s_local at webmail kernel:[<ffffffff8025fb22>] 
child_rip+0x0/0x12
Apr  2 17:18:28 s_local at webmail kernel:
Apr  2 17:18:28 s_local at webmail kernel:Mem-info:
Apr  2 17:18:28 s_local at webmail kernel:DMA per-cpu:
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 
used:102
Apr  2 17:18:28 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 17:18:28 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 17:18:28 s_local at webmail kernel:Free pages:        2912kB (0kB 
HighMem)
Apr  2 17:18:28 s_local at webmail kernel:Active:73053 inactive:40998 
dirty:0 writeback:0 unstable:0 free:728 slab:4059 mapped-file:2 
mapped-anon:114411 pagetables:2857
Apr  2 17:18:28 s_local at webmail kernel:DMA free:2912kB min:2916kB 
low:3644kB high:4372kB active:292212kB inactive:163992kB 
present:532480kB pages_scanned:12263677 all_unreclaimable? yes
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 17:18:28 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 17:18:28 s_local at webmail kernel:DMA: 12*4kB 8*8kB 3*16kB 0*32kB 
1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2912kB
Apr  2 17:18:28 s_local at webmail kernel:DMA32: empty
Apr  2 17:18:28 s_local at webmail kernel:Normal: empty
Apr  2 17:18:28 s_local at webmail kernel:HighMem: empty
Apr  2 17:18:28 s_local at webmail kernel:89 pagecache pages
Apr  2 17:18:28 s_local at webmail kernel:Swap cache: add 176874, delete 
176874, find 114990/119550, race 0+0
Apr  2 17:18:28 s_local at webmail kernel:Free swap  = 0kB
Apr  2 17:18:28 s_local at webmail kernel:Total swap = 557048kB
Apr  2 17:18:28 s_local at webmail kernel:Free swap:            0kB
Apr  2 17:18:28 s_local at webmail kernel:133120 pages of RAM
Apr  2 17:18:28 s_local at webmail kernel:5993 reserved pages
Apr  2 17:18:28 s_local at webmail kernel:4331 pages shared
Apr  2 17:18:28 s_local at webmail kernel:0 pages swap cached
Apr  2 18:05:47 s_local at webmail kernel:init invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 18:05:47 s_local at webmail kernel:
Apr  2 18:05:47 s_local at webmail kernel:Call Trace:
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8020622a>] 
hypercall_page+0x22a/0x1000
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 18:05:47 s_local at webmail kernel:
Apr  2 18:05:47 s_local at webmail kernel:Mem-info:
Apr  2 18:05:47 s_local at webmail kernel:DMA per-cpu:
Apr  2 18:05:47 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:12
Apr  2 18:05:47 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 18:05:47 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:Free pages:        2868kB (0kB 
HighMem)
Apr  2 18:05:47 s_local at webmail kernel:Active:57202 inactive:56889 
dirty:0 writeback:0 unstable:0 free:717 slab:4124 mapped-file:2 
mapped-anon:114137 pagetables:3184
Apr  2 18:05:47 s_local at webmail kernel:DMA free:2868kB min:2916kB 
low:3644kB high:4372kB active:228808kB inactive:227556kB 
present:532480kB pages_scanned:708700 all_unreclaimable? yes
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:DMA: 31*4kB 1*8kB 1*16kB 1*32kB 
0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2868kB
Apr  2 18:05:47 s_local at webmail kernel:DMA32: empty
Apr  2 18:05:47 s_local at webmail kernel:Normal: empty
Apr  2 18:05:47 s_local at webmail kernel:HighMem: empty
Apr  2 18:05:47 s_local at webmail kernel:89 pagecache pages
Apr  2 18:05:47 s_local at webmail kernel:Swap cache: add 248675, delete 
248675, find 135538/146503, race 0+0
Apr  2 18:05:47 s_local at webmail kernel:Free swap  = 0kB
Apr  2 18:05:47 s_local at webmail kernel:Total swap = 557048kB
Apr  2 18:05:47 s_local at webmail kernel:Free swap:            0kB
Apr  2 18:05:47 s_local at webmail kernel:133120 pages of RAM
Apr  2 18:05:47 s_local at webmail kernel:5993 reserved pages
Apr  2 18:05:47 s_local at webmail kernel:4606 pages shared
Apr  2 18:05:47 s_local at webmail kernel:0 pages swap cached
Apr  2 18:05:47 s_local at webmail kernel:Out of memory: Killed process 
21335 (httpd).
Apr  2 18:05:47 s_local at webmail kernel:init invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 18:05:47 s_local at webmail kernel:
Apr  2 18:05:47 s_local at webmail kernel:Call Trace:
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8020622a>] 
hypercall_page+0x22a/0x1000
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 18:05:47 s_local at webmail kernel:
Apr  2 18:05:47 s_local at webmail kernel:Mem-info:
Apr  2 18:05:47 s_local at webmail kernel:DMA per-cpu:
Apr  2 18:05:47 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:19
Apr  2 18:05:47 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 18:05:47 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:Free pages:        2868kB (0kB 
HighMem)
Apr  2 18:05:47 s_local at webmail kernel:Active:73107 inactive:40952 
dirty:0 writeback:0 unstable:0 free:717 slab:4118 mapped-file:2 
mapped-anon:114137 pagetables:3184
Apr  2 18:05:47 s_local at webmail kernel:DMA free:2868kB min:2916kB 
low:3644kB high:4372kB active:292428kB inactive:163808kB 
present:532480kB pages_scanned:4108148 all_unreclaimable? yes
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:DMA: 31*4kB 1*8kB 1*16kB 1*32kB 
0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2868kB
Apr  2 18:05:47 s_local at webmail kernel:DMA32: empty
Apr  2 18:05:47 s_local at webmail kernel:Normal: empty
Apr  2 18:05:47 s_local at webmail kernel:HighMem: empty
Apr  2 18:05:47 s_local at webmail kernel:83 pagecache pages
Apr  2 18:05:47 s_local at webmail kernel:Swap cache: add 248675, delete 
248675, find 135538/146503, race 0+0
Apr  2 18:05:47 s_local at webmail kernel:Free swap  = 0kB
Apr  2 18:05:47 s_local at webmail kernel:Total swap = 557048kB
Apr  2 18:05:47 s_local at webmail kernel:Free swap:            0kB
Apr  2 18:05:47 s_local at webmail kernel:133120 pages of RAM
Apr  2 18:05:47 s_local at webmail kernel:5993 reserved pages
Apr  2 18:05:47 s_local at webmail kernel:4638 pages shared
Apr  2 18:05:47 s_local at webmail kernel:0 pages swap cached
Apr  2 18:05:47 s_local at webmail kernel:init invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 18:05:47 s_local at webmail kernel:
Apr  2 18:05:47 s_local at webmail kernel:Call Trace:
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8020622a>] 
hypercall_page+0x22a/0x1000
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 18:05:47 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 18:05:47 s_local at webmail kernel:
Apr  2 18:05:47 s_local at webmail kernel:Mem-info:
Apr  2 18:05:47 s_local at webmail kernel:DMA per-cpu:
Apr  2 18:05:47 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:26
Apr  2 18:05:47 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:57
Apr  2 18:05:47 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 18:05:47 s_local at webmail kernel:Free pages:        2868kB (0kB 
HighMem)
Apr  2 18:05:47 s_local at webmail kernel:Active:72986 inactive:41009 
dirty:0 writeback:0 unstable:0 free:717 slab:4111 mapped-file:2 
mapped-anon:114137 pagetables:3184
Apr  2 18:05:47 s_local at webmail kernel:DMA free:2868kB min:2916kB 
low:3644kB high:4372kB active:291944kB inactive:164036kB 
present:532480kB pages_scanned:7458532 all_unreclaimable? yes
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:05:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:05:47 s_local at webmail kernel:DMA: 31*4kB 1*8kB 1*16kB 1*32kB 
0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2868kB
Apr  2 18:05:47 s_local at webmail kernel:DMA32: empty
Apr  2 18:05:47 s_local at webmail kernel:Normal: empty
Apr  2 18:05:47 s_local at webmail kernel:HighMem: empty
Apr  2 18:05:47 s_local at webmail kernel:83 pagecache pages
Apr  2 18:05:47 s_local at webmail kernel:Swap cache: add 248675, delete 
248675, find 135538/146503, race 0+0
Apr  2 18:05:47 s_local at webmail kernel:Free swap  = 0kB
Apr  2 18:05:47 s_local at webmail kernel:Total swap = 557048kB
Apr  2 18:05:47 s_local at webmail kernel:Free swap:            0kB
Apr  2 18:05:47 s_local at webmail kernel:133120 pages of RAM
Apr  2 18:05:47 s_local at webmail kernel:5993 reserved pages
Apr  2 18:05:47 s_local at webmail kernel:4702 pages shared
Apr  2 18:05:47 s_local at webmail kernel:0 pages swap cached
Apr  2 18:14:47 s_local at webmail kernel:httpd invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 18:14:47 s_local at webmail kernel:
Apr  2 18:14:47 s_local at webmail kernel:Call Trace:
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff80299fd8>] 
autoremove_wake_function+0x0/0x2e
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff803164c8>] 
file_has_perm+0x48/0xa3
Apr  2 18:14:47 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 18:14:47 s_local at webmail kernel:
Apr  2 18:14:47 s_local at webmail kernel:Mem-info:
Apr  2 18:14:47 s_local at webmail kernel:DMA per-cpu:
Apr  2 18:14:47 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:25
Apr  2 18:14:47 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:14
Apr  2 18:14:47 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 18:14:47 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 18:14:47 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 18:14:47 s_local at webmail kernel:Free pages:        2876kB (0kB 
HighMem)
Apr  2 18:14:47 s_local at webmail kernel:Active:54916 inactive:59014 
dirty:0 writeback:0 unstable:0 free:719 slab:4154 mapped-file:5 
mapped-anon:113909 pagetables:3325
Apr  2 18:14:47 s_local at webmail kernel:DMA free:2876kB min:2916kB 
low:3644kB high:4372kB active:219664kB inactive:236056kB 
present:532480kB pages_scanned:738327 all_unreclaimable? yes
Apr  2 18:14:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:14:47 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:14:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:14:47 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:14:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:14:47 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:14:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:14:47 s_local at webmail kernel:DMA: 1*4kB 11*8kB 0*16kB 3*32kB 
0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2876kB
Apr  2 18:14:47 s_local at webmail kernel:DMA32: empty
Apr  2 18:14:47 s_local at webmail kernel:Normal: empty
Apr  2 18:14:47 s_local at webmail kernel:HighMem: empty
Apr  2 18:14:47 s_local at webmail kernel:126 pagecache pages
Apr  2 18:14:47 s_local at webmail kernel:Swap cache: add 256186, delete 
256186, find 137948/149847, race 0+0
Apr  2 18:14:47 s_local at webmail kernel:Free swap  = 0kB
Apr  2 18:14:47 s_local at webmail kernel:Total swap = 557048kB
Apr  2 18:14:47 s_local at webmail kernel:Free swap:            0kB
Apr  2 18:14:47 s_local at webmail kernel:133120 pages of RAM
Apr  2 18:14:47 s_local at webmail kernel:5993 reserved pages
Apr  2 18:14:47 s_local at webmail kernel:4863 pages shared
Apr  2 18:14:47 s_local at webmail kernel:0 pages swap cached
Apr  2 18:14:47 s_local at webmail kernel:Out of memory: Killed process 
21332 (httpd).
Apr  2 18:23:47 s_local at webmail kernel:automount invoked oom-killer: 
gfp_mask=0x201d2, order=0, oomkilladj=0
Apr  2 18:23:47 s_local at webmail kernel:
Apr  2 18:23:47 s_local at webmail kernel:Call Trace:
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff802bc998>] 
out_of_memory+0x8b/0x203
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff8020f657>] 
__alloc_pages+0x245/0x2ce
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff8021336e>] 
__do_page_cache_readahead+0xd0/0x21c
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff80262824>] 
__wait_on_bit_lock+0x5b/0x66
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff88081d4d>] 
:dm_mod:dm_any_congested+0x38/0x3f
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff80213c47>] 
filemap_nopage+0x148/0x322
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff80208db9>] 
__handle_mm_fault+0x440/0x11f6
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff802639f9>] 
_spin_lock_irqsave+0x9/0x14
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff802666ef>] 
do_page_fault+0xf7b/0x12e0
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff802ad687>] 
audit_syscall_entry+0xf2/0x1a1
Apr  2 18:23:47 s_local at webmail kernel:[<ffffffff8025f82b>] 
error_exit+0x0/0x6e
Apr  2 18:23:47 s_local at webmail kernel:
Apr  2 18:23:47 s_local at webmail kernel:Mem-info:
Apr  2 18:23:47 s_local at webmail kernel:DMA per-cpu:
Apr  2 18:23:47 s_local at webmail kernel:cpu 0 hot: high 186, batch 31 used:30
Apr  2 18:23:47 s_local at webmail kernel:cpu 0 cold: high 62, batch 15 used:14
Apr  2 18:23:47 s_local at webmail kernel:DMA32 per-cpu: empty
Apr  2 18:23:47 s_local at webmail kernel:Normal per-cpu: empty
Apr  2 18:23:47 s_local at webmail kernel:HighMem per-cpu: empty
Apr  2 18:23:47 s_local at webmail kernel:Free pages:        2908kB (0kB 
HighMem)
Apr  2 18:23:47 s_local at webmail kernel:Active:56901 inactive:56963 
dirty:0 writeback:0 unstable:0 free:727 slab:4164 mapped-file:6 
mapped-anon:113966 pagetables:3306
Apr  2 18:23:47 s_local at webmail kernel:DMA free:2908kB min:2916kB 
low:3644kB high:4372kB active:227604kB inactive:227852kB 
present:532480kB pages_scanned:807511 all_unreclaimable? yes
Apr  2 18:23:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:23:47 s_local at webmail kernel:DMA32 free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:23:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:23:47 s_local at webmail kernel:Normal free:0kB min:0kB low:0kB 
high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:23:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:23:47 s_local at webmail kernel:HighMem free:0kB min:128kB 
low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 
all_unreclaimable? no
Apr  2 18:23:47 s_local at webmail kernel:lowmem_reserve[]: 0 0 0 0
Apr  2 18:23:47 s_local at webmail kernel:DMA: 37*4kB 1*8kB 0*16kB 2*32kB 
0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2908kB
Apr  2 18:23:47 s_local at webmail kernel:DMA32: empty
Apr  2 18:23:47 s_local at webmail kernel:Normal: empty
Apr  2 18:23:47 s_local at webmail kernel:HighMem: empty
Apr  2 18:23:47 s_local at webmail kernel:147 pagecache pages
Apr  2 18:23:47 s_local at webmail kernel:Swap cache: add 267431, delete 
267431, find 138909/152168, race 0+0
Apr  2 18:23:47 s_local at webmail kernel:Free swap  = 0kB
Apr  2 18:23:47 s_local at webmail kernel:Total swap = 557048kB
Apr  2 18:23:47 s_local at webmail kernel:Free swap:            0kB
Apr  2 18:23:47 s_local at webmail kernel:133120 pages of RAM
Apr  2 18:23:47 s_local at webmail kernel:5993 reserved pages
Apr  2 18:23:47 s_local at webmail kernel:4965 pages shared
Apr  2 18:23:47 s_local at webmail kernel:0 pages swap cached
Apr  2 18:23:47 s_local at webmail kernel:Out of memory: Killed process 
21336 (httpd).