[CentOS] How to diagnose "Unable to handle kernel paging request" kernel panic?

Sat Apr 28 11:50:32 UTC 2007
Jim King <mlists at microreplay.com>

Does anyone know how to diagnose the error "Unable to handle kernel  
paging request at virtual address xxxxxx"?

I got this error (followed by a panic and a crash) on a server  
running CentOS 4.4 with the 2.6.9-42.0.8.ELsmp kernel.  The machine  
is based on an Intel 975XBX2 motherboard (975X chipset).  I have  
built several of these and they have been extremely stable.  This one  
is an Asterisk PBX.  It was up for over a month before this crash.

Here is the exact kernel build info, from the restart log: "Linux  
version 2.6.9-42.0.8.ELsmp (mockbuild at builder6.centos.org) (gcc  
version 3.4.6 20060404 (Red Hat 3.4.6-3)) #1 SMP Tue Jan 30 12:33:47  
EST 2007"

Does the call trace indicate something to do with the ext3 file  
system?  I am not running RAID on this machine, it is just a single  
drive with LVM.

Any help that the list could be would be really great.

Thanks,

Jim


Here is the /var/log/messages entry:

Apr 25 11:29:25 amd kernel: Unable to handle kernel paging request at  
virtual address 7c4ad43d
Apr 25 11:29:25 amd kernel:  printing eip:
Apr 25 11:29:25 amd kernel: f8892ad2
Apr 25 11:29:25 amd kernel: *pde = 00000000
Apr 25 11:29:25 amd kernel: Oops: 0002 [#1]
Apr 25 11:29:25 amd kernel: SMP
Apr 25 11:29:25 amd kernel: Modules linked in: zttranscode(U)  
wctdm24xxp(U) zaptel(U) md5 ipv6 parport_pc lp parport autofs4 sunrpc  
crc_ccitt button battery ac hw_random sky2 e1000 floppy dm_snapshot  
dm_zero dm_mirror ext3 jbd dm_mod ata_piix libata sd_mod scsi_mod
Apr 25 11:29:25 amd kernel: CPU:    0
Apr 25 11:29:25 amd kernel: EIP:    0060:[<f8892ad2>]    Not tainted VLI
Apr 25 11:29:25 amd kernel: EFLAGS: 00010202   (2.6.9-42.0.8.ELsmp)
Apr 25 11:29:25 amd kernel: EIP is at journal_grab_journal_head 
+0x26/0x42 [jbd]
Apr 25 11:29:25 amd kernel: eax: 1e14ae0f   ebx: cc9e5284   ecx:  
7c4ad439   edx: cc9e5284
Apr 25 11:29:25 amd kernel: esi: c1114300   edi: 00000000   ebp:  
c89e5284   esp: f7cfee24
Apr 25 11:29:25 amd kernel: ds: 007b   es: 007b   ss: 0068
Apr 25 11:29:25 amd kernel: Process kswapd0 (pid: 51,  
threadinfo=f7cfe000 task=f7d276f0)
Apr 25 11:29:25 amd kernel: Stack: f888d9f0 00000000 f7ec5a00  
f88c44d3 000000d0 f3c4f678 f7cfef58 c015d323
Apr 25 11:29:25 amd kernel:        f3c4f678 c1114300 c0149c7f  
00000001 00000001 00000010 00000000 f7cfeec8
Apr 25 11:29:25 amd kernel:        f7cfee64 f7cfee64 00000002  
00000001 c11142c0 c11142e0 c1114140 c1114160
Apr 25 11:29:25 amd kernel: Call Trace:
Apr 25 11:29:25 amd kernel:  [<f888d9f0>] journal_try_to_free_buffers 
+0x5d/0xc5 [jbd]
Apr 25 11:29:25 amd kernel:  [<f88c44d3>] ext3_releasepage+0x0/0x54  
[ext3]
Apr 25 11:29:25 amd kernel:  [<c015d323>] try_to_release_page+0x34/0x46
Apr 25 11:29:25 amd kernel:  [<c0149c7f>] shrink_list+0x293/0x3ed
Apr 25 11:29:25 amd kernel:  [<c0149fb6>] shrink_cache+0x1dd/0x34d
Apr 25 11:29:25 amd kernel:  [<c0144e78>] get_writeback_state+0x30/0x35
Apr 25 11:29:25 amd kernel:  [<c0180c1e>] mb_cache_shrink_fn+0x56/0x131
Apr 25 11:29:25 amd kernel:  [<c0149776>] shrink_slab+0x7e/0x161
Apr 25 11:29:25 amd kernel:  [<c014a674>] shrink_zone+0xa7/0xb6
Apr 25 11:29:25 amd kernel:  [<c014aa6f>] balance_pgdat+0x1c5/0x30e
Apr 25 11:29:25 amd kernel:  [<c02d27c9>] schedule+0x86d/0x8db
Apr 25 11:29:25 amd kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Apr 25 11:29:25 amd kernel:  [<c014ac82>] kswapd+0xca/0xcc
Apr 25 11:29:25 amd kernel:  [<c01204f5>] autoremove_wake_function 
+0x0/0x2d
Apr 25 11:29:25 amd kernel:  [<c02d47f2>] ret_from_fork+0x6/0x14
Apr 25 11:29:25 amd kernel:  [<c01204f5>] autoremove_wake_function 
+0x0/0x2d
Apr 25 11:29:25 amd kernel:  [<c014abb8>] kswapd+0x0/0xcc
Apr 25 11:29:25 amd kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Apr 25 11:29:25 amd kernel: Code: 5b 5e 5f 5d c3 89 c2 31 c9 eb 0b f3  
90 8b 02 a9 00 00 10 00 75 f5 f0 0f ba 2a 14 19 c0 85 c0 75 ec 8b 02  
f6 c4 20 74 06 8b 4a 28 <ff> 41 04 8b 02 a9 00 00 10 00 75 08 0f 0b  
11 02 58 42 89 f8 f0
Apr 25 11:29:25 amd kernel:  <0>Fatal exception: panic in 5 seconds