Hello all,
Came across my first real "problem" in all the years I've been using Linux today and I'm stumped for an answer.
I've recently built a back-up server on top of Centos 5.3 using a 3Ware 9650SE 4 port SATA card and 4 x 1.5TB drives in a RAID 5 array. I went for the 3Ware card as it has driver support already so any kernel updates won't require me to rebuild the drivers each time. My current kernel is 2.6.18-128.2.1.el5 x86_x64.
My problems started a few weeks ago with some message issues with the 3Ware card causing it to stop accessing the mounted RAID array. The only way to resolve it was a cold boot of the machine. Then today I log-in to discover the RAIDed partition has been auto un-mounted by XFS itself.
So at this point I'm not sure where the problem lies. Is it the card that is faulty/problematic or is it something with the XFS file system?
I'll take the XFS problem first as it's the most recent.
When I logged in today I found the following in my server logs:
Sep 7 21:08:19 backup kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff882e356b
Sep 7 21:08:19 backup kernel:
Sep 7 21:08:19 backup kernel: Call Trace:
Sep 7 21:08:19 backup kernel: [<ffffffff882e1c7e>] :xfs:xfs_free_ag_extent+0x19f/0x67f
Sep 7 21:08:19 backup kernel: [<ffffffff882e356b>] :xfs:xfs_free_extent+0xa9/0xc9
Sep 7 21:08:19 backup kernel: [<ffffffff882f02ad>] :xfs:xfs_bmap_finish+0xf0/0x169
Sep 7 21:08:19 backup kernel: [<ffffffff8830de46>] :xfs:xfs_itruncate_finish+0x172/0x2b3
Sep 7 21:08:19 backup kernel: [<ffffffff88326f22>] :xfs:xfs_inactive+0x22e/0x821
Sep 7 21:08:19 backup kernel: [<ffffffff8832dd66>] :xfs:xfs_validate_fields+0x24/0x4b
Sep 7 21:08:19 backup kernel: [<ffffffff8832e361>] :xfs:xfs_vn_unlink+0x3d/0x4f
Sep 7 21:08:19 backup kernel: [<ffffffff80063bb7>] mutex_lock+0xd/0x1d
Sep 7 21:08:19 backup kernel: [<ffffffff883308fc>] :xfs:xfs_fs_clear_inode+0xa5/0xec
Sep 7 21:08:19 backup kernel: [<ffffffff80022861>] clear_inode+0xd2/0x123
Sep 7 21:08:19 backup kernel: [<ffffffff8002f19a>] generic_delete_inode+0xde/0x143
Sep 7 21:08:19 backup kernel: [<ffffffff8003c1f6>] do_unlinkat+0xd5/0x141
Sep 7 21:08:19 backup kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0
Sep 7 21:08:19 backup kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Sep 7 21:08:19 backup kernel:
Sep 7 21:08:19 backup kernel: xfs_force_shutdown(sdb1,0x8) called from line 4267 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_bmap.c. Return address = 0xffffffff882f02ea
Sep 7 21:08:19 backup kernel: Filesystem "sdb1": Corruption of in-memory data detected. Shutting down filesystem: sdb1
Sep 7 21:08:19 backup kernel: Please umount the filesystem, and rectify the problem(s)
Subsequent recovery was as follows:
Sep 8 10:59:40 backup kernel: xfs_force_shutdown(sdb1,0x1) called from line 424 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_rw.c. Return address = 0xffffffff883277f1
Sep 8 10:59:40 backup kernel: xfs_force_shutdown(sdb1,0x1) called from line 424 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_rw.c. Return address = 0xffffffff883277f1
Sep 8 11:00:13 backup kernel: XFS mounting filesystem sdb1
Sep 8 11:00:13 backup kernel: Starting XFS recovery on filesystem: sdb1 (logdev: internal)
Sep 8 11:00:14 backup kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff882e356b
Sep 8 11:00:14 backup kernel:
Sep 8 11:00:14 backup kernel: Call Trace:
Sep 8 11:00:14 backup kernel: [<ffffffff882e1c7e>] :xfs:xfs_free_ag_extent+0x19f/0x67f
Sep 8 11:00:14 backup kernel: [<ffffffff882e356b>] :xfs:xfs_free_extent+0xa9/0xc9
Sep 8 11:00:14 backup kernel: [<ffffffff883180e5>] :xfs:xlog_recover_finish+0x15a/0x244
Sep 8 11:00:14 backup kernel: [<ffffffff8831bb00>] :xfs:xfs_mountfs+0xa24/0xc30
Sep 8 11:00:14 backup kernel: [<ffffffff8000c394>] _atomic_dec_and_lock+0x39/0x57
Sep 8 11:00:14 backup kernel: [<ffffffff88321748>] :xfs:xfs_mount+0x762/0x83b
Sep 8 11:00:14 backup kernel: [<ffffffff88330c79>] :xfs:xfs_fs_fill_super+0x0/0x1e3
Sep 8 11:00:14 backup kernel: [<ffffffff88330cf7>] :xfs:xfs_fs_fill_super+0x7e/0x1e3
Sep 8 11:00:14 backup kernel: [<ffffffff8006465d>] __down_write_nested+0x12/0x92
Sep 8 11:00:14 backup kernel: [<ffffffff800e72c1>] get_filesystem+0x12/0x3b
Sep 8 11:00:14 backup kernel: [<ffffffff800de9b5>] sget+0x365/0x377
Sep 8 11:00:14 backup kernel: [<ffffffff800de301>] set_bdev_super+0x0/0xf
Sep 8 11:00:14 backup kernel: [<ffffffff800de310>] test_bdev_super+0x0/0xd
Sep 8 11:00:14 backup kernel: [<ffffffff800df2c4>] get_sb_bdev+0x10a/0x164
Sep 8 11:00:14 backup kernel: [<ffffffff800dec61>] vfs_kern_mount+0x93/0x11a
Sep 8 11:00:14 backup kernel: [<ffffffff800ded2a>] do_kern_mount+0x36/0x4d
Sep 8 11:00:14 backup kernel: [<ffffffff800e8bfb>] do_mount+0x6a7/0x717
Sep 8 11:00:14 backup kernel: [<ffffffff8002c7ef>] mntput_no_expire+0x19/0x89
Sep 8 11:00:14 backup kernel: [<ffffffff8000e87a>] link_path_walk+0xd3/0xe5
Sep 8 11:00:14 backup kernel: [<ffffffff80066bd2>] do_page_fault+0x4fe/0x830
Sep 8 11:00:14 backup kernel: [<ffffffff800076c0>] find_get_page+0x21/0x50
Sep 8 11:00:14 backup kernel: [<ffffffff80013445>] filemap_nopage+0x188/0x322
Sep 8 11:00:14 backup kernel: [<ffffffff80008b8f>] __handle_mm_fault+0x51d/0xe5c
Sep 8 11:00:14 backup kernel: [<ffffffff8002331e>] __user_walk_fd+0x41/0x4c
Sep 8 11:00:14 backup kernel: [<ffffffff800c877d>] zone_statistics+0x3e/0x6d
Sep 8 11:00:14 backup kernel: [<ffffffff8000f10a>] __alloc_pages+0x65/0x2ce
Sep 8 11:00:14 backup kernel: [<ffffffff800e0e13>] sys_readlinkat+0x98/0xa9
Sep 8 11:00:14 backup kernel: [<ffffffff8004bf75>] sys_mount+0x8a/0xcd
Sep 8 11:00:14 backup kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Sep 8 11:00:14 backup kernel:
Sep 8 11:00:14 backup kernel: Ending XFS recovery on filesystem: sdb1 (logdev: internal)
Sep 8 11:03:45 backup kernel: XFS mounting filesystem sdb1
The 3Ware card issue occured about a month ago but hasn't returned since:
Aug 9 05:56:32 backup kernel: sd 4:0:0:0: WARNING: (0x06:0x002C): Command (0x8a) timed out, resetting card.
Aug 9 05:56:35 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.
Aug 9 05:57:37 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x001F): Microcontroller not ready during reset sequence.
Aug 9 05:58:19 backup kernel: 3w-9xxx: scsi4: AEN: INFO (0x04:0x005E): Cache synchronization completed:unit=0.
Aug 9 05:58:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.
Aug 9 05:59:49 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.
Aug 9 05:59:49 backup kernel: sd 4:0:0:0: WARNING: (0x06:0x002C): Command (0x28) timed out, resetting card.
Aug 9 06:00:19 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:01:19 backup last message repeated 2 times
Aug 9 06:01:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:01:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x002B): Controller reset failed during scsi host reset.
Aug 9 06:01:49 backup kernel: sd 4:0:0:0: scsi: Device offlined - not ready after error recovery
Aug 9 06:01:49 backup last message repeated 6 times
Aug 9 06:01:49 backup kernel: sd 4:0:0:0: rejecting I/O to offline device
Aug 9 06:01:49 backup last message repeated 7 times
Aug 9 06:01:49 backup kernel: sd 4:0:0:0: SCSI error: return code = 0x00010000
Aug 9 06:01:49 backup kernel: end_request: I/O error, dev sdb, sector 4394574486
Aug 9 06:01:49 backup kernel: sd 4:0:0:0: rejecting I/O to offline device
Aug 9 06:01:49 backup last message repeated 2 times
Aug 9 06:01:49 backup kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x105efe274 ("xlog_iodone") error 5 buf count 30208
Aug 9 06:01:49 backup kernel: xfs_force_shutdown(sdb1,0x2) called from line 956 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_log.c. Return address = 0xffffffff88313780
Aug 9 06:01:49 backup kernel: Filesystem "sdb1": Log I/O Error Detected. Shutting down filesystem: sdb1
Aug 9 06:01:49 backup kernel: Please umount the filesystem, and rectify the problem(s)
Aug 9 06:01:49 backup kernel: sd 4:0:0:0: rejecting I/O to offline device
Aug 9 06:01:49 backup kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x1446a8 ("xfs_trans_read_buf") error 5 buf count 4096
Aug 9 06:01:49 backup kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x1c16a8 ("xfs_trans_read_buf") error 5 buf count 4096
Aug 9 06:02:04 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.
Aug 9 06:03:04 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.
Aug 9 06:03:34 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:04:04 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:04:19 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.
Aug 9 06:05:19 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.
Aug 9 06:05:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:06:19 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:06:34 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.
Aug 9 06:07:34 backup kernel: 3w-9xxx: scsi4: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.
Aug 9 06:08:04 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:08:34 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0036): Response queue (large) empty failed during reset sequence.
Aug 9 06:08:49 backup kernel: 3w-9xxx: scsi4: ERROR: (0x06:0x0010): Microcontroller Error: clearing.
repeat last three lines ad infinitum.
Can anyone shed any light as to what the issue might be? Is it XFS or is it the card? I went with XFS as all the research I did indicated that it gave the best performance for the millions of small files in the backups we run and doesn't have any many issues with hardlinks as EXT3.
- J
_________________________________________________________________
Share your memories online with anyone you want.
http://www.microsoft.com/middleeast/windows/windowslive/products/photos-share.aspx?tab=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos/attachments/20090908/498f307e/attachment-0003.html>