<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
</style>
</head>
<body class='hmmessage'>
Hello all,<br><br>Came across my first real "problem" in all the years I've been using Linux today and I'm stumped for an answer.<br><br>I've recently built a back-up server on top of Centos 5.3 using a 3Ware 9650SE 4 port SATA card and 4 x 1.5TB drives in a RAID 5 array. I went for the 3Ware card as it has driver support already so any kernel updates won't require me to rebuild the drivers each time. My current kernel is 2.6.18-128.2.1.el5 x86_x64.<br><br>My problems started a few weeks ago with some message issues with the 3Ware card causing it to stop accessing the mounted RAID array. The only way to resolve it was a cold boot of the machine. Then today I log-in to discover the RAIDed partition has been auto un-mounted by XFS itself.<br><br>So at this point I'm not sure where the problem lies. Is it the card that is faulty/problematic or is it something with the XFS file system?<br><br>I'll take the XFS problem first as it's the most recent.<br><br>When I logged in today I found the following in my server logs:<br><br>Sep 7 21:08:19 backup kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff882e356b<br>Sep 7 21:08:19 backup kernel:<br>Sep 7 21:08:19 backup kernel: Call Trace:<br>Sep 7 21:08:19 backup kernel: [<ffffffff882e1c7e>] :xfs:xfs_free_ag_extent+0x19f/0x67f<br>Sep 7 21:08:19 backup kernel: [<ffffffff882e356b>] :xfs:xfs_free_extent+0xa9/0xc9<br>Sep 7 21:08:19 backup kernel: [<ffffffff882f02ad>] :xfs:xfs_bmap_finish+0xf0/0x169<br>Sep 7 21:08:19 backup kernel: [<ffffffff8830de46>] :xfs:xfs_itruncate_finish+0x172/0x2b3<br>Sep 7 21:08:19 backup kernel: [<ffffffff88326f22>] :xfs:xfs_inactive+0x22e/0x821<br>Sep 7 21:08:19 backup kernel: [<ffffffff8832dd66>] :xfs:xfs_validate_fields+0x24/0x4b<br>Sep 7 21:08:19 backup kernel: [<ffffffff8832e361>] :xfs:xfs_vn_unlink+0x3d/0x4f<br>Sep 7 21:08:19 backup kernel: [<ffffffff80063bb7>] mutex_lock+0xd/0x1d<br>Sep 7 21:08:19 backup kernel: [<ffffffff883308fc>] :xfs:xfs_fs_clear_inode+0xa5/0xec<br>Sep 7 21:08:19 backup kernel: [<ffffffff80022861>] clear_inode+0xd2/0x123<br>Sep 7 21:08:19 backup kernel: [<ffffffff8002f19a>] generic_delete_inode+0xde/0x143<br>Sep 7 21:08:19 backup kernel: [<ffffffff8003c1f6>] do_unlinkat+0xd5/0x141<br>Sep 7 21:08:19 backup kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0<br>Sep 7 21:08:19 backup kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0<br>Sep 7 21:08:19 backup kernel:<br>Sep 7 21:08:19 backup kernel: xfs_force_shutdown(sdb1,0x8) called from line 4267 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_bmap.c. Return address = 0xffffffff882f02ea<br>Sep 7 21:08:19 backup kernel: Filesystem "sdb1": Corruption of in-memory data detected. Shutting down filesystem: sdb1<br>Sep 7 21:08:19 backup kernel: Please umount the filesystem, and rectify the problem(s)<br><br>Subsequent recovery was as follows:<br><br>Sep 8 10:59:40 backup kernel: xfs_force_shutdown(sdb1,0x1) called from line 424 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_rw.c. Return address = 0xffffffff883277f1<br>Sep 8 10:59:40 backup kernel: xfs_force_shutdown(sdb1,0x1) called from line 424 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_rw.c. Return address = 0xffffffff883277f1<br>Sep 8 11:00:13 backup kernel: XFS mounting filesystem sdb1<br>Sep 8 11:00:13 backup kernel: Starting XFS recovery on filesystem: sdb1 (logdev: internal)<br>Sep 8 11:00:14 backup kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff882e356b<br>Sep 8 11:00:14 backup kernel:<br>Sep 8 11:00:14 backup kernel: Call Trace:<br>Sep 8 11:00:14 backup kernel: [<ffffffff882e1c7e>] :xfs:xfs_free_ag_extent+0x19f/0x67f<br>Sep 8 11:00:14 backup kernel: [<ffffffff882e356b>] :xfs:xfs_free_extent+0xa9/0xc9<br>Sep 8 11:00:14 backup kernel: [<ffffffff883180e5>] :xfs:xlog_recover_finish+0x15a/0x244<br>Sep 8 11:00:14 backup kernel: [<ffffffff8831bb00>] :xfs:xfs_mountfs+0xa24/0xc30<br>Sep 8 11:00:14 backup kernel: [<ffffffff8000c394>] _atomic_dec_and_lock+0x39/0x57<br>Sep 8 11:00:14 backup kernel: [<ffffffff88321748>] :xfs:xfs_mount+0x762/0x83b<br>Sep 8 11:00:14 backup kernel: [<ffffffff88330c79>] :xfs:xfs_fs_fill_super+0x0/0x1e3<br>Sep 8 11:00:14 backup kernel: [<ffffffff88330cf7>] :xfs:xfs_fs_fill_super+0x7e/0x1e3<br>Sep 8 11:00:14 backup kernel: [<ffffffff8006465d>] __down_write_nested+0x12/0x92<br>Sep 8 11:00:14 backup kernel: [<ffffffff800e72c1>] get_filesystem+0x12/0x3b<br>Sep 8 11:00:14 backup kernel: [<ffffffff800de9b5>] sget+0x365/0x377<br>Sep 8 11:00:14 backup kernel: [<ffffffff800de301>] set_bdev_super+0x0/0xf<br>Sep 8 11:00:14 backup kernel: [<ffffffff800de310>] test_bdev_super+0x0/0xd<br>Sep 8 11:00:14 backup kernel: [<ffffffff800df2c4>] get_sb_bdev+0x10a/0x164<br>Sep 8 11:00:14 backup kernel: [<ffffffff800dec61>] vfs_kern_mount+0x93/0x11a<br>Sep 8 11:00:14 backup kernel: [<ffffffff800ded2a>] do_kern_mount+0x36/0x4d<br>Sep 8 11:00:14 backup kernel: [<ffffffff800e8bfb>] do_mount+0x6a7/0x717<br>Sep 8 11:00:14 backup kernel: [<ffffffff8002c7ef>] mntput_no_expire+0x19/0x89<br>Sep 8 11:00:14 backup kernel: [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5<br>Sep 8 11:00:14 backup kernel: [<ffffffff80066bd2>] do_page_fault+0x4fe/0x830<br>Sep 8 11:00:14 backup kernel: [<ffffffff800076c0>] find_get_page+0x21/0x50<br>Sep 8 11:00:14 backup kernel: [<ffffffff80013445>] filemap_nopage+0x188/0x322<br>Sep 8 11:00:14 backup kernel: [<ffffffff80008b8f>] __handle_mm_fault+0x51d/0xe5c<br>Sep 8 11:00:14 backup kernel: [<ffffffff8002331e>] __user_walk_fd+0x41/0x4c<br>Sep 8 11:00:14 backup kernel: [<ffffffff800c877d>] zone_statistics+0x3e/0x6d<br>Sep 8 11:00:14 backup kernel: [<ffffffff8000f10a>] __alloc_pages+0x65/0x2ce<br>Sep 8 11:00:14 backup kernel: [<ffffffff800e0e13>] sys_readlinkat+0x98/0xa9<br>Sep 8 11:00:14 backup kernel: [<ffffffff8004bf75>] sys_mount+0x8a/0xcd<br>Sep 8 11:00:14 backup kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0<br>Sep 8 11:00:14 backup kernel:<br>Sep 8 11:00:14 backup kernel: Ending XFS recovery on filesystem: sdb1 (logdev: internal)<br>Sep 8 11:03:45 backup kernel: XFS mounting filesystem sdb1<br><br>The 3Ware card issue occured about a month ago but hasn't returned since:<br><br>Aug 9 05:56:32 backup kernel: sd 4:0:0:0: WARNING: (0x06:0x002C): Command (0x8a) timed out, resetting card.<br>Aug 9 05:56:35 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.<br>Aug 9 05:57:37 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x001F): Microcontroller not ready during reset sequence.<br>Aug 9 05:58:19 backup kernel: 3w-9xxx: scsi4: AEN: INFO (0x04:0x005E): Cache synchronization completed:unit=0.<br>Aug 9 05:58:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.<br>Aug 9 05:59:49 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.<br>Aug 9 05:59:49 backup kernel: sd 4:0:0:0: WARNING: (0x06:0x002C): Command (0x28) timed out, resetting card.<br>Aug 9 06:00:19 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:01:19 backup last message repeated 2 times<br>Aug 9 06:01:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:01:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x002B): Controller reset failed during scsi host reset.<br>Aug 9 06:01:49 backup kernel: sd 4:0:0:0: scsi: Device offlined - not ready after error recovery<br>Aug 9 06:01:49 backup last message repeated 6 times<br>Aug 9 06:01:49 backup kernel: sd 4:0:0:0: rejecting I/O to offline device<br>Aug 9 06:01:49 backup last message repeated 7 times<br>Aug 9 06:01:49 backup kernel: sd 4:0:0:0: SCSI error: return code = 0x00010000<br>Aug 9 06:01:49 backup kernel: end_request: I/O error, dev sdb, sector 4394574486<br>Aug 9 06:01:49 backup kernel: sd 4:0:0:0: rejecting I/O to offline device<br>Aug 9 06:01:49 backup last message repeated 2 times<br>Aug 9 06:01:49 backup kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x105efe274 ("xlog_iodone") error 5 buf count 30208<br>Aug 9 06:01:49 backup kernel: xfs_force_shutdown(sdb1,0x2) called from line 956 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_log.c. Return address = 0xffffffff88313780<br>Aug 9 06:01:49 backup kernel: Filesystem "sdb1": Log I/O Error Detected. Shutting down filesystem: sdb1<br>Aug 9 06:01:49 backup kernel: Please umount the filesystem, and rectify the problem(s)<br>Aug 9 06:01:49 backup kernel: sd 4:0:0:0: rejecting I/O to offline device<br>Aug 9 06:01:49 backup kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x1446a8 ("xfs_trans_read_buf") error 5 buf count 4096<br>Aug 9 06:01:49 backup kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x1c16a8 ("xfs_trans_read_buf") error 5 buf count 4096<br>Aug 9 06:02:04 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.<br>Aug 9 06:03:04 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.<br>Aug 9 06:03:34 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:04:04 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:04:19 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.<br>Aug 9 06:05:19 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.<br>Aug 9 06:05:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:06:19 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:06:34 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.<br>Aug 9 06:07:34 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.<br>Aug 9 06:08:04 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:08:34 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.<br>Aug 9 06:08:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.<br><br>repeat last three lines ad infinitum.<br><br>Can anyone shed any light as to what the issue might be? Is it XFS or is it the card? I went with XFS as all the research I did indicated that it gave the best performance for the millions of small files in the backups we run and doesn't have any many issues with hardlinks as EXT3.<br><br>- J<br><br /><hr />Share your memories online with anyone you want <a href='http://www.microsoft.com/middleeast/windows/windowslive/products/photos-share.aspx?tab=1' target='_new'>anyone you want.</a></body>
</html>