[CentOS] Centos 6.6, apparent xfs corruption

Mon Sep 21 20:41:58 UTC 2015
Nicholas Geovanis <nickgeovanis at gmail.com>

Hi all -
After several months of worry-free operation, we received the following
kernel messages about an xfs filesystem running under CentOS 6.6. The
proximate causes appear to be "Internal error xfs_trans_cancel" and
"Corruption of in-memory data detected.  Shutting down filesystem". The
filesystem is back up, mounted, appears to be working OK underlying a
Splunk datastore. Does anyone have a suggestion on diagnosis or known
problems? Many thanks.....Nick Geo

Sep 18 20:35:15 gries kernel: XFS (dm-2): Internal error xfs_trans_cancel
at line 1948 of file fs/xfs/xfs_trans.c.  Caller 0xffffffffa01f1388
Sep 18 20:35:15 gries kernel:
Sep 18 20:35:15 gries kernel: Pid: 24005, comm: splunkd Not tainted
2.6.32-504.8.1.el6.x86_64 #1
Sep 18 20:35:15 gries kernel: Call Trace:
Sep 18 20:35:15 gries kernel: [<ffffffffa01d57bf>] ?
xfs_error_report+0x3f/0x50 [xfs]
Sep 18 20:35:15 gries kernel: [<ffffffffa01f1388>] ? xfs_rename+0x2d8/0x720
[xfs]
Sep 18 20:35:15 gries kernel: [<ffffffffa01f2e55>] ?
xfs_trans_cancel+0xf5/0x120 [xfs]
Sep 18 20:35:15 gries kernel: [<ffffffffa01f1388>] ? xfs_rename+0x2d8/0x720
[xfs]
Sep 18 20:35:15 gries kernel: [<ffffffff8114eef9>] ? __do_fault+0x469/0x530
Sep 18 20:35:15 gries kernel: [<ffffffffa02050d6>] ?
xfs_vn_rename+0x66/0x70 [xfs]
Sep 18 20:35:15 gries kernel: [<ffffffff8119d149>] ? vfs_rename+0x419/0x480
Sep 18 20:35:15 gries kernel: [<ffffffff8119fab9>] ?
sys_renameat+0x309/0x3a0
Sep 18 20:35:15 gries kernel: [<ffffffff8128c295>] ?
_atomic_dec_and_lock+0x55/0x80
Sep 18 20:35:15 gries kernel: [<ffffffff811b07e0>] ?
mntput_no_expire+0x30/0x110
Sep 18 20:35:15 gries kernel: [<ffffffff810e5c87>] ?
audit_syscall_entry+0x1d7/0x200
Sep 18 20:35:15 gries kernel: [<ffffffff8119fb6b>] ? sys_rename+0x1b/0x20
Sep 18 20:35:15 gries kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
Sep 18 20:35:15 gries kernel: XFS (dm-2): xfs_do_force_shutdown(0x8) called
from line 1949 of file fs/xfs/xfs_trans.c.  Return address =
0xffffffffa01f2e6e
Sep 18 20:35:15 gries kernel: XFS (dm-2): Corruption of in-memory data
detected.  Shutting down filesystem
Sep 18 20:35:15 gries kernel: XFS (dm-2): Please umount the filesystem and
rectify the problem(s)
Sep 18 20:35:27 gries kernel: XFS (dm-2): xfs_log_force: error 5 returned.