[CentOS] kernel error -- system crash

Mon Feb 5 15:33:50 UTC 2007
Jon Christensen <jonc21 at gmail.com>

The system stopped responding to ssh but still responded to ping.  The logs
showed this error: (its long)



Feb  4 00:08:08 server1 kernel: gdmgreeter: page allocation failure.
order:0, mode:0x50
Feb  4 00:08:08 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:08 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:08 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:08 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:08 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:08 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:08 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:08 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:08 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:08 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:08 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:08 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:23 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:23 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:23 server1 nrpe[2751]: Error: Could not complete SSL handshake.
5
Feb  4 00:08:23 server1 nrpe[2753]: Error: Could not complete SSL handshake.
5
Feb  4 00:08:23 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:23 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:23 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:23 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:23 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:23 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:23 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:23 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:23 server1 kernel:  [<c031269a>] unix_poll+0x24/0xa3
Feb  4 00:08:23 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:23 server1 kernel:  [<c018127a>] do_pollfd+0x47/0x81
Feb  4 00:08:23 server1 kernel:  [<c01812fd>] do_poll+0x49/0xab
Feb  4 00:08:23 server1 kernel:  [<c018152c>] sys_poll+0x1cd/0x281
Feb  4 00:08:23 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:23 server1 kernel:  [<c0129189>] sys_gettimeofday+0x53/0xac
Feb  4 00:08:23 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:23 server1 kernel: Mem-info:
Feb  4 00:08:23 server1 kernel: DMA per-cpu:
Feb  4 00:08:23 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:23 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:23 server1 kernel: Normal per-cpu:
Feb  4 00:08:23 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:23 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:23 server1 kernel: HighMem per-cpu:
Feb  4 00:08:23 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:23 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:23 server1 kernel:
Feb  4 00:08:23 server1 kernel: Free pages:       49280kB (49280kB HighMem)
Feb  4 00:08:23 server1 kernel: Active:28662 inactive:196240 dirty:60
writeback:0 unstable:0 free:12320 slab:11019 mapped:9351 pagetables:253
Feb  4 00:08:23 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12184kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:23 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:23 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:93152kB inactive:747560kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:23 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:23 server1 kernel: HighMem free:49280kB min:128kB low:256kB
high:384kB active:21460kB inactive:25216kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:23 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:23 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:23 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:23 server1 kernel: HighMem: 1614*4kB 1807*8kB 1105*16kB
290*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
49280kB
Feb  4 00:08:23 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:23 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:23 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:23 server1 kernel: 253936 pages of RAM
Feb  4 00:08:23 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:23 server1 kernel: 3209 reserved pages
Feb  4 00:08:23 server1 kernel: 199769 pages shared
Feb  4 00:08:23 server1 kernel: 0 pages swap cached
Feb  4 00:08:23 server1 kernel: kswapd0: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:23 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:23 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:23 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:23 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:23 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:23 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:23 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:23 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:23 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:23 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:23 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:23 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:23 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:23 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:23 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:23 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:23 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:23 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:23 server1 kernel:  [<c01578cb>] balance_pgdat+0x1b3/0x2cb
Feb  4 00:08:23 server1 kernel:  [<c0157a9c>] kswapd+0xb9/0xbb
Feb  4 00:08:23 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:23 server1 kernel:  [<c0318e06>] ret_from_fork+0x6/0x14
Feb  4 00:08:23 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:23 server1 kernel:  [<c01579e3>] kswapd+0x0/0xbb
Feb  4 00:08:23 server1 kernel:  [<c01041dd>] kernel_thread_helper+0x5/0xb
Feb  4 00:08:23 server1 kernel: Mem-info:
Feb  4 00:08:23 server1 kernel: DMA per-cpu:
Feb  4 00:08:23 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:23 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:23 server1 kernel: Normal per-cpu:
Feb  4 00:08:23 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:23 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:23 server1 kernel: HighMem per-cpu:
Feb  4 00:08:23 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       49280kB (49280kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:28663 inactive:196239 dirty:60
writeback:0 unstable:0 free:12320 slab:11019 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12184kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:93156kB inactive:747556kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:49280kB min:128kB low:256kB
high:384kB active:21460kB inactive:25216kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1614*4kB 1807*8kB 1105*16kB
290*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
49280kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199769 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: nfsd: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af736>] ext3_truncate+0x3d0/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014ce77>]
generic_file_buffered_write+0x172/0x4db
Feb  4 00:08:24 server1 kernel:  [<c01299d0>] current_fs_time+0x44/0x4c
Feb  4 00:08:24 server1 kernel:  [<c014d51d>]
__generic_file_aio_write_nolock+0x33d/0x36b
Feb  4 00:08:24 server1 kernel:  [<c014d64e>]
__generic_file_write_nolock+0x84/0x99
Feb  4 00:08:24 server1 kernel:  [<c011f422>] recalc_task_prio+0x128/0x133
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c014d971>] generic_file_writev+0x54/0xb3
Feb  4 00:08:24 server1 kernel:  [<c014d91d>] generic_file_writev+0x0/0xb3
Feb  4 00:08:24 server1 kernel:  [<c016c605>] do_readv_writev+0x1c5/0x245
Feb  4 00:08:24 server1 kernel:  [<c016c0f2>] do_sync_write+0x0/0xc9
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<c016b335>] __dentry_open+0xca/0x16a
Feb  4 00:08:24 server1 kernel:  [<c016b266>] dentry_open+0x48/0x4d
Feb  4 00:08:24 server1 kernel:  [<c016c703>] vfs_writev+0x3e/0x43
Feb  4 00:08:24 server1 kernel:  [<f8b39600>] nfsd_write+0xeb/0x28f [nfsd]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8b40ee2>] nfsd3_proc_write+0xbf/0xd5
[nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b42f90>]
nfs3svc_decode_writeargs+0x0/0x243 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b35947>] nfsd_dispatch+0xba/0x16f
[nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8ace74c>] svc_process+0x432/0x6e2
[sunrpc]
Feb  4 00:08:24 server1 kernel:  [<f8b355eb>] nfsd+0x2a7/0x549 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b35344>] nfsd+0x0/0x549 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<c01041dd>] kernel_thread_helper+0x5/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       49280kB (49280kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:28663 inactive:196239 dirty:60
writeback:0 unstable:0 free:12320 slab:11019 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12184kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:93156kB inactive:747556kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:49280kB min:128kB low:256kB
high:384kB active:21460kB inactive:25216kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1614*4kB 1807*8kB 1105*16kB
290*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
49280kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199769 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: nrpe: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:24 server1 kernel:  [<c02dd665>] tcp_poll+0x31/0x144
Feb  4 00:08:24 server1 kernel:  [<c0188fc8>] destroy_inode+0x3d/0x4c
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:24 server1 kernel:  [<c0180cca>] do_select+0x25c/0x378
Feb  4 00:08:24 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:24 server1 kernel:  [<c01810d9>] sys_select+0x2e0/0x43a
Feb  4 00:08:24 server1 kernel:  [<c02b520c>] sys_socketcall+0x10c/0x1fb
Feb  4 00:08:24 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       49280kB (49280kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:28664 inactive:196239 dirty:0
writeback:20 unstable:0 free:12320 slab:11019 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:40kB inactive:12184kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:93156kB inactive:747556kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:49280kB min:128kB low:256kB
high:384kB active:21460kB inactive:25216kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1614*4kB 1807*8kB 1105*16kB
290*32kB 20*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
49280kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199768 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: nfsd: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af736>] ext3_truncate+0x3d0/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014ce77>]
generic_file_buffered_write+0x172/0x4db
Feb  4 00:08:24 server1 kernel:  [<c01299d0>] current_fs_time+0x44/0x4c
Feb  4 00:08:24 server1 kernel:  [<c014d51d>]
__generic_file_aio_write_nolock+0x33d/0x36b
Feb  4 00:08:24 server1 kernel:  [<c014d64e>]
__generic_file_write_nolock+0x84/0x99
Feb  4 00:08:24 server1 kernel:  [<c011f422>] recalc_task_prio+0x128/0x133
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c014d971>] generic_file_writev+0x54/0xb3
Feb  4 00:08:24 server1 kernel:  [<c014d91d>] generic_file_writev+0x0/0xb3
Feb  4 00:08:24 server1 kernel:  [<c016c605>] do_readv_writev+0x1c5/0x245
Feb  4 00:08:24 server1 kernel:  [<c016c0f2>] do_sync_write+0x0/0xc9
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<c016b335>] __dentry_open+0xca/0x16a
Feb  4 00:08:24 server1 kernel:  [<c016b266>] dentry_open+0x48/0x4d
Feb  4 00:08:24 server1 kernel:  [<c016c703>] vfs_writev+0x3e/0x43
Feb  4 00:08:24 server1 kernel:  [<f8b39600>] nfsd_write+0xeb/0x28f [nfsd]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8b40ee2>] nfsd3_proc_write+0xbf/0xd5
[nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b42f90>]
nfs3svc_decode_writeargs+0x0/0x243 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b35947>] nfsd_dispatch+0xba/0x16f
[nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8ace74c>] svc_process+0x432/0x6e2
[sunrpc]
Feb  4 00:08:24 server1 kernel:  [<f8b355eb>] nfsd+0x2a7/0x549 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b35344>] nfsd+0x0/0x549 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<c01041dd>] kernel_thread_helper+0x5/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       66820kB (66820kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29068 inactive:191437 dirty:121
writeback:0 unstable:0 free:16705 slab:11020 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12168kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:94776kB inactive:745908kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:66820kB min:128kB low:256kB
high:384kB active:21460kB inactive:7672kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1613*4kB 1804*8kB 1105*16kB
483*32kB 168*64kB 16*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
66820kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199786 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: gdmgreeter: page allocation failure.
order:0, mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:24 server1 kernel:  [<c031269a>] unix_poll+0x24/0xa3
Feb  4 00:08:24 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:24 server1 kernel:  [<c018127a>] do_pollfd+0x47/0x81
Feb  4 00:08:24 server1 kernel:  [<c01812fd>] do_poll+0x49/0xab
Feb  4 00:08:24 server1 kernel:  [<c018152c>] sys_poll+0x1cd/0x281
Feb  4 00:08:24 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:24 server1 kernel:  [<c0129189>] sys_gettimeofday+0x53/0xac
Feb  4 00:08:24 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       66820kB (66820kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29068 inactive:191437 dirty:121
writeback:0 unstable:0 free:16705 slab:11020 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12168kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:94776kB inactive:745908kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:66820kB min:128kB low:256kB
high:384kB active:21460kB inactive:7672kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1613*4kB 1804*8kB 1105*16kB
483*32kB 168*64kB 16*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
66820kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199786 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: syslogd: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:24 server1 kernel:  [<c02ba096>] datagram_poll+0x25/0xcf
Feb  4 00:08:24 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:24 server1 kernel:  [<c0180cca>] do_select+0x25c/0x378
Feb  4 00:08:24 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:24 server1 kernel:  [<c01810d9>] sys_select+0x2e0/0x43a
Feb  4 00:08:24 server1 kernel:  [<c016db6d>] sys_fsync+0xce/0xd6
Feb  4 00:08:24 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       66820kB (66820kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29068 inactive:191437 dirty:121
writeback:0 unstable:0 free:16705 slab:11020 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12168kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:94776kB inactive:745908kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:66820kB min:128kB low:256kB
high:384kB active:21460kB inactive:7672kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1613*4kB 1804*8kB 1105*16kB
483*32kB 168*64kB 16*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
66820kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199786 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: nrpe: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:24 server1 kernel:  [<c02dd665>] tcp_poll+0x31/0x144
Feb  4 00:08:24 server1 kernel:  [<c0188fc8>] destroy_inode+0x3d/0x4c
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:24 server1 kernel:  [<c0180cca>] do_select+0x25c/0x378
Feb  4 00:08:24 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:24 server1 kernel:  [<c01810d9>] sys_select+0x2e0/0x43a
Feb  4 00:08:24 server1 kernel:  [<c02b520c>] sys_socketcall+0x10c/0x1fb
Feb  4 00:08:24 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       66820kB (66820kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29068 inactive:191437 dirty:121
writeback:0 unstable:0 free:16705 slab:11020 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12168kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:94776kB inactive:745908kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:66820kB min:128kB low:256kB
high:384kB active:21460kB inactive:7672kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1613*4kB 1804*8kB 1105*16kB
483*32kB 168*64kB 16*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
66820kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199786 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: kswapd0: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c01578cb>] balance_pgdat+0x1b3/0x2cb
Feb  4 00:08:24 server1 kernel:  [<c0157a9c>] kswapd+0xb9/0xbb
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c0318e06>] ret_from_fork+0x6/0x14
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c01579e3>] kswapd+0x0/0xbb
Feb  4 00:08:24 server1 kernel:  [<c01041dd>] kernel_thread_helper+0x5/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       66820kB (66820kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29068 inactive:191449 dirty:0
writeback:55 unstable:0 free:16705 slab:11020 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:36kB inactive:12188kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:94776kB inactive:745936kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:66820kB min:128kB low:256kB
high:384kB active:21460kB inactive:7672kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1613*4kB 1804*8kB 1105*16kB
483*32kB 168*64kB 16*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
66820kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 199774 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: nrpe: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:24 server1 kernel:  [<c02dd665>] tcp_poll+0x31/0x144
Feb  4 00:08:24 server1 kernel:  [<c0188fc8>] destroy_inode+0x3d/0x4c
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:24 server1 kernel:  [<c0180cca>] do_select+0x25c/0x378
Feb  4 00:08:24 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:24 server1 kernel:  [<c01810d9>] sys_select+0x2e0/0x43a
Feb  4 00:08:24 server1 kernel:  [<c02b520c>] sys_socketcall+0x10c/0x1fb
Feb  4 00:08:24 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       68140kB (68140kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29706 inactive:190479 dirty:144
writeback:0 unstable:0 free:17035 slab:11015 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:40kB inactive:12172kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:97324kB inactive:743392kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:68140kB min:128kB low:256kB
high:384kB active:21460kB inactive:6352kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1657*4kB 1851*8kB 1113*16kB
489*32kB 171*64kB 18*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
68140kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 200567 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: syslogd: page allocation failure. order:0,
mode:0x50
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014b70f>] find_or_create_page+0x39/0x72
Feb  4 00:08:24 server1 kernel:  [<c016eda7>] grow_dev_page+0x2a/0x1eb
Feb  4 00:08:24 server1 kernel:  [<c016f096>] __getblk_slow+0x12e/0x159
Feb  4 00:08:24 server1 kernel:  [<c016f40e>] __getblk+0x3f/0x49
Feb  4 00:08:24 server1 kernel:  [<c016f455>] __bread+0x9/0x22
Feb  4 00:08:24 server1 kernel:  [<f88af239>] ext3_free_branches+0x91/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0180962>] __pollwait+0x2d/0x94
Feb  4 00:08:24 server1 kernel:  [<c02ba096>] datagram_poll+0x25/0xcf
Feb  4 00:08:24 server1 kernel:  [<c02b40f4>] sock_poll+0x12/0x14
Feb  4 00:08:24 server1 kernel:  [<c0180cca>] do_select+0x25c/0x378
Feb  4 00:08:24 server1 kernel:  [<c0180935>] __pollwait+0x0/0x94
Feb  4 00:08:24 server1 kernel:  [<c01810d9>] sys_select+0x2e0/0x43a
Feb  4 00:08:24 server1 kernel:  [<c016db6d>] sys_fsync+0xce/0xd6
Feb  4 00:08:24 server1 kernel:  [<c0318eeb>] syscall_call+0x7/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       68140kB (68140kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29706 inactive:190479 dirty:144
writeback:0 unstable:0 free:17035 slab:11015 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:40kB inactive:12172kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:97324kB inactive:743392kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:68140kB min:128kB low:256kB
high:384kB active:21460kB inactive:6352kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1657*4kB 1851*8kB 1113*16kB
489*32kB 171*64kB 18*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
68140kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 200567 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: kswapd0: page allocation failure. order:0,
mode:0x850
Feb  4 00:08:24 server1 kernel:  [<c014f863>] __alloc_pages+0x28b/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014f88d>] __get_free_pages+0x18/0x24
Feb  4 00:08:24 server1 kernel:  [<c0152e22>] kmem_getpages+0x15/0x94
Feb  4 00:08:24 server1 kernel:  [<c0153adf>] cache_grow+0x107/0x233
Feb  4 00:08:24 server1 kernel:  [<c0153e02>] cache_alloc_refill+0x1f7/0x227
Feb  4 00:08:24 server1 kernel:  [<c0154337>] __kmalloc+0x6b/0x7d
Feb  4 00:08:24 server1 kernel:  [<f886c54c>] __jbd_kmalloc+0x16/0x17 [jbd]
Feb  4 00:08:24 server1 kernel:  [<f8863833>]
journal_get_undo_access+0x58/0x122 [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88aa437>]
ext3_free_blocks_sb+0x17e/0x4d5 [ext3]
Feb  4 00:08:24 server1 kernel:  [<f88aa7bb>] ext3_free_blocks+0x2d/0x84
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af141>] ext3_free_data+0x90/0xf7
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af27e>] ext3_free_branches+0xd6/0x1be
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af702>] ext3_truncate+0x39c/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c01578cb>] balance_pgdat+0x1b3/0x2cb
Feb  4 00:08:24 server1 kernel:  [<c0157a9c>] kswapd+0xb9/0xbb
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c0318e06>] ret_from_fork+0x6/0x14
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c01579e3>] kswapd+0x0/0xbb
Feb  4 00:08:24 server1 kernel:  [<c01041dd>] kernel_thread_helper+0x5/0xb
Feb  4 00:08:24 server1 kernel: Mem-info:
Feb  4 00:08:24 server1 kernel: DMA per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 2, high 6, batch 1
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 2, batch 1
Feb  4 00:08:24 server1 kernel: Normal per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 32, high 96, batch 16
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 32, batch 16
Feb  4 00:08:24 server1 kernel: HighMem per-cpu:
Feb  4 00:08:24 server1 kernel: cpu 0 hot: low 10, high 30, batch 5
Feb  4 00:08:24 server1 kernel: cpu 0 cold: low 0, high 10, batch 5
Feb  4 00:08:24 server1 kernel:
Feb  4 00:08:24 server1 kernel: Free pages:       68140kB (68140kB HighMem)
Feb  4 00:08:24 server1 kernel: Active:29706 inactive:190479 dirty:144
writeback:0 unstable:0 free:17035 slab:11015 mapped:9351 pagetables:253
Feb  4 00:08:24 server1 kernel: DMA free:0kB min:16kB low:32kB high:48kB
active:40kB inactive:12172kB present:16384kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: Normal free:0kB min:936kB low:1872kB
high:2808kB active:97324kB inactive:743392kB present:901120kB
pages_scanned:0 all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: HighMem free:68140kB min:128kB low:256kB
high:384kB active:21460kB inactive:6352kB present:98240kB pages_scanned:0
all_unreclaimable? no
Feb  4 00:08:24 server1 kernel: protections[]: 0 0 0
Feb  4 00:08:24 server1 kernel: DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb  4 00:08:24 server1 kernel: HighMem: 1657*4kB 1851*8kB 1113*16kB
489*32kB 171*64kB 18*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
68140kB
Feb  4 00:08:24 server1 kernel: Swap cache: add 40, delete 40, find 0/0,
race 0+0
Feb  4 00:08:24 server1 kernel: 0 bounce buffer pages
Feb  4 00:08:24 server1 kernel: Free swap:       1052088kB
Feb  4 00:08:24 server1 kernel: 253936 pages of RAM
Feb  4 00:08:24 server1 kernel: 24560 pages of HIGHMEM
Feb  4 00:08:24 server1 kernel: 3209 reserved pages
Feb  4 00:08:24 server1 kernel: 200567 pages shared
Feb  4 00:08:24 server1 kernel: 0 pages swap cached
Feb  4 00:08:24 server1 kernel: journal_get_undo_access: No memory for
committed data
Feb  4 00:08:24 server1 kernel: ext3_free_blocks_sb: aborting transaction:
Out of memory in __ext3_journal_get_undo_access
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_free_blocks_sb: Out of memory
Feb  4 00:08:24 server1 kernel: Aborting journal on device dm-0.
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_truncate: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_free_blocks_sb: Journal has aborted
Feb  4 00:08:24 server1 kernel: ext3_abort called.
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0):
ext3_journal_start_sb: Detected aborted journal
Feb  4 00:08:24 server1 kernel: Remounting filesystem read-only
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
start_transaction: Journal has aborted
Feb  4 00:08:24 server1 last message repeated 8 times
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_free_blocks_sb: Journal has aborted
Feb  4 00:08:24 server1 last message repeated 54 times
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_truncate: Out of memory
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_orphan_del: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_delete_inode: Out of memory
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_orphan_del: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_delete_inode: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_truncate: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_orphan_del: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_delete_inode: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_truncate: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_orphan_del: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_delete_inode: Journal has aborted
Feb  4 00:08:24 server1 kernel: __journal_remove_journal_head: freeing
b_committed_data
Feb  4 00:08:24 server1 last message repeated 65 times
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_free_blocks_sb: Journal has aborted
Feb  4 00:08:24 server1 last message repeated 18 times
Feb  4 00:08:24 server1 kernel: EXT3-fs error (device dm-0) in
ext3_reserve_inode_write: Journal has aborted
Feb  4 00:08:24 server1 kernel: Assertion failure in journal_stop() at
fs/jbd/transaction.c:1344: "transaction->t_updates > 0"
Feb  4 00:08:24 server1 kernel: ------------[ cut here ]------------
Feb  4 00:08:24 server1 kernel: kernel BUG at fs/jbd/transaction.c:1344!
Feb  4 00:08:24 server1 kernel: invalid operand: 0000 [#1]
Feb  4 00:08:24 server1 kernel: Modules linked in: nfsd exportfs lockd
nfs_acl parport_pc lp parport autofs4 i2c_dev i2c_core sunrpc dm_mirror
button battery ac md5 ipv6 ohci_hcd ehci_hcd snd_intel8x0 snd_ac97_codec
snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart
snd_rawmidi snd_seq_device snd soundcore 3c59x mii e1000 floppy ext3 jbd
dm_mod
Feb  4 00:08:24 server1 kernel: CPU:    0
Feb  4 00:08:24 server1 kernel: EIP:    0060:[<f88642b1>]    Not tainted VLI
Feb  4 00:08:24 server1 kernel: EFLAGS: 00010246   (2.6.9-42.0.8.EL)
Feb  4 00:08:24 server1 kernel: EIP is at journal_stop+0x30/0x42f [jbd]
Feb  4 00:08:24 server1 kernel: eax: 00000062   ebx: ffffffe2   ecx:
f886ce4a   edx: f6156b30
Feb  4 00:08:24 server1 kernel: esi: f7c464c0   edi: c18a7200   ebp:
da8082b0   esp: f6156b2c
Feb  4 00:08:24 server1 kernel: ds: 007b   es: 007b   ss: 0068
Feb  4 00:08:24 server1 kernel: Process nfsd (pid: 2947, threadinfo=f6156000
task=c1a0cc50)
Feb  4 00:08:24 server1 kernel: Stack: f886ce4a f886ca96 f886ce0e 00000540
f886cf5c c18a7200 c18a7200 ffffffe2
Feb  4 00:08:24 server1 kernel:        c1ab8400 f88bb06c 00000000 f88b3d06
f6d67438 00000000 f6d67538 f88af7be
Feb  4 00:08:24 server1 kernel:        0000000b 00000000 f6d67618 00000400
f6d67438 da8082b0 f6d67538 07e98504
Feb  4 00:08:24 server1 kernel: Call Trace:
Feb  4 00:08:24 server1 kernel:  [<f88b3d06>] __ext3_journal_stop+0x19/0x34
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88af7be>] ext3_truncate+0x458/0x460
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8862737>] journal_start+0x78/0x9e [jbd]
Feb  4 00:08:24 server1 kernel:  [<f88ad4b5>] ext3_delete_inode+0x60/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<f88ad455>] ext3_delete_inode+0x0/0xaa
[ext3]
Feb  4 00:08:24 server1 kernel:  [<c018adbb>]
generic_delete_inode+0xef/0x229
Feb  4 00:08:24 server1 kernel:  [<c018b1e2>] iput+0x5f/0x61
Feb  4 00:08:24 server1 kernel:  [<c018633c>] prune_dcache+0x50d/0x70c
Feb  4 00:08:24 server1 kernel:  [<c0186df1>] shrink_dcache_memory+0x16/0x2d
Feb  4 00:08:24 server1 kernel:  [<c015628b>] shrink_slab+0xf7/0x14c
Feb  4 00:08:24 server1 kernel:  [<c0157648>] try_to_free_pages+0xb9/0x189
Feb  4 00:08:24 server1 kernel:  [<c014f78d>] __alloc_pages+0x1b5/0x29d
Feb  4 00:08:24 server1 kernel:  [<c014ce77>]
generic_file_buffered_write+0x172/0x4db
Feb  4 00:08:24 server1 kernel:  [<c01299d0>] current_fs_time+0x44/0x4c
Feb  4 00:08:24 server1 kernel:  [<c014d51d>]
__generic_file_aio_write_nolock+0x33d/0x36b
Feb  4 00:08:24 server1 kernel:  [<c014d64e>]
__generic_file_write_nolock+0x84/0x99
Feb  4 00:08:24 server1 kernel:  [<c011f422>] recalc_task_prio+0x128/0x133
Feb  4 00:08:24 server1 kernel:  [<c0121853>]
autoremove_wake_function+0x0/0x2d
Feb  4 00:08:24 server1 kernel:  [<c014d971>] generic_file_writev+0x54/0xb3
Feb  4 00:08:24 server1 kernel:  [<c014d91d>] generic_file_writev+0x0/0xb3
Feb  4 00:08:24 server1 kernel:  [<c016c605>] do_readv_writev+0x1c5/0x245
Feb  4 00:08:24 server1 kernel:  [<c016c0f2>] do_sync_write+0x0/0xc9
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<c016b335>] __dentry_open+0xca/0x16a
Feb  4 00:08:24 server1 kernel:  [<c016b266>] dentry_open+0x48/0x4d
Feb  4 00:08:24 server1 kernel:  [<c016c703>] vfs_writev+0x3e/0x43
Feb  4 00:08:24 server1 kernel:  [<f8b39600>] nfsd_write+0xeb/0x28f [nfsd]
Feb  4 00:08:24 server1 kernel:  [<c0317627>] __cond_resched+0x14/0x3b
Feb  4 00:08:24 server1 kernel:  [<f8b40ee2>] nfsd3_proc_write+0xbf/0xd5
[nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b42f90>]
nfs3svc_decode_writeargs+0x0/0x243 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b35947>] nfsd_dispatch+0xba/0x16f
[nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8ace74c>] svc_process+0x432/0x6e2
[sunrpc]
Feb  4 00:08:24 server1 kernel:  [<f8b355eb>] nfsd+0x2a7/0x549 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<f8b35344>] nfsd+0x0/0x549 [nfsd]
Feb  4 00:08:24 server1 kernel:  [<c01041dd>] kernel_thread_helper+0x5/0xb
Feb  4 00:08:24 server1 kernel: Code: 53 52 52 8b 30 8b 3e 83 7e 50 00 7f 29
68 5c cf 86 f8 68 40 05 00 00 68 0e ce 86 f8 68 96 ca 86 f8 68 4a ce 86 f8
e8 50 06 8c c7 <0f> 0b 40 05 0e ce 86 f8 83 c4 14 b8 00 f0 ff ff 21 e0 8b 00
39
Feb  4 00:08:24 server1 kernel:  <0>Fatal exception: panic in 5 seconds
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos/attachments/20070205/4edb413d/attachment-0004.html>