Hello Everyone,
Since rebooting my Centos 6.10 Openvz server "daisy" yesterday, I am
getting horrible system performance. /var/log/messages is full of
HDIO_GET_IDENTITY failed for /dev/sdb. The latest entries look like this:
Apr 22 08:51:32 daisy kernel: [141224.655699] CT: 1005: stopped
Apr 22 08:55:04 daisy ata_id[21513]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:00:05 daisy ata_id[21584]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:05:02 daisy ata_id[21644]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:10:01 daisy ata_id[22282]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:11:49 daisy kernel: [142441.721065] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:11:49 daisy kernel: [142441.721083]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:11:49 daisy kernel: [142441.721093] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:11:49 daisy kernel: [142441.721109] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:11:49 daisy kernel: [142441.721115]Â ffff88006654bcb8
0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:11:49 daisy kernel: [142441.721121]Â ffff88000004d238
ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:11:49 daisy kernel: [142441.721125]Â ffff88011a707000
ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:11:49 daisy kernel: [142441.721130] Call Trace:
Apr 22 09:11:49 daisy kernel: [142441.721139]Â [<ffffffff8114f130>] ?
sync_page+0x0/0x50
Apr 22 09:11:49 daisy kernel: [142441.721144]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:11:49 daisy kernel: [142441.721149]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:11:49 daisy kernel: [142441.721155]Â [<ffffffff81067432>] ?
check_preempt_curr+0x82/0xa0
Apr 22 09:11:49 daisy kernel: [142441.721159]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:11:49 daisy kernel: [142441.721162]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:11:49 daisy kernel: [142441.721167]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:11:49 daisy kernel: [142441.721172]Â [<ffffffff811f98d8>]
sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:11:49 daisy kernel: [142441.721176]Â [<ffffffff8114fa6f>] ?
filemap_fdatawait+0x2f/0x40
Apr 22 09:11:49 daisy kernel: [142441.721181]Â [<ffffffff81200f85>]
__sync_filesystem+0x95/0xa0
Apr 22 09:11:49 daisy kernel: [142441.721184]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:11:49 daisy kernel: [142441.721188]Â [<ffffffff812016e5>]
sys_sync+0x155/0x1a0
Apr 22 09:11:49 daisy kernel: [142441.721192]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:13:49 daisy kernel: [142561.721069] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:13:49 daisy kernel: [142561.721087]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:13:49 daisy kernel: [142561.721096] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:13:49 daisy kernel: [142561.721112] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:13:49 daisy kernel: [142561.721118]Â ffff88006654bcb8
0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:13:49 daisy kernel: [142561.721123]Â ffff88000004d238
ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:13:49 daisy kernel: [142561.721128]Â ffff88011a707000
ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:13:49 daisy kernel: [142561.721133] Call Trace:
Apr 22 09:13:49 daisy kernel: [142561.721142]Â [<ffffffff8114f130>] ?
sync_page+0x0/0x50
Apr 22 09:13:49 daisy kernel: [142561.721148]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:13:49 daisy kernel: [142561.721153]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:13:49 daisy kernel: [142561.721158]Â [<ffffffff81067432>] ?
check_preempt_curr+0x82/0xa0
Apr 22 09:13:49 daisy kernel: [142561.721162]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:13:49 daisy kernel: [142561.721166]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:13:49 daisy kernel: [142561.721170]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:13:49 daisy kernel: [142561.721176]Â [<ffffffff811f98d8>]
sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:13:49 daisy kernel: [142561.721180]Â [<ffffffff8114fa6f>] ?
filemap_fdatawait+0x2f/0x40
Apr 22 09:13:49 daisy kernel: [142561.721184]Â [<ffffffff81200f85>]
__sync_filesystem+0x95/0xa0
Apr 22 09:13:49 daisy kernel: [142561.721188]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:13:49 daisy kernel: [142561.721192]Â [<ffffffff812016e5>]
sys_sync+0x155/0x1a0
Apr 22 09:13:49 daisy kernel: [142561.721196]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:15:06 daisy ata_id[22299]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:15:49 daisy kernel: [142681.721085] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:15:49 daisy kernel: [142681.721104]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:15:49 daisy kernel: [142681.721113] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:15:49 daisy kernel: [142681.721129] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:15:49 daisy kernel: [142681.721136]Â ffff88006654bcb8
0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:15:49 daisy kernel: [142681.721141]Â ffff88000004d238
ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:15:49 daisy kernel: [142681.721146]Â ffff88011a707000
ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:15:49 daisy kernel: [142681.721150] Call Trace:
Apr 22 09:15:49 daisy kernel: [142681.721160]Â [<ffffffff8114f130>] ?
sync_page+0x0/0x50
Apr 22 09:15:49 daisy kernel: [142681.721166]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:15:49 daisy kernel: [142681.721172]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:15:49 daisy kernel: [142681.721178]Â [<ffffffff81067432>] ?
check_preempt_curr+0x82/0xa0
Apr 22 09:15:49 daisy kernel: [142681.721182]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:15:49 daisy kernel: [142681.721185]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:15:49 daisy kernel: [142681.721190]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:15:49 daisy kernel: [142681.721196]Â [<ffffffff811f98d8>]
sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:15:49 daisy kernel: [142681.721200]Â [<ffffffff8114fa6f>] ?
filemap_fdatawait+0x2f/0x40
Apr 22 09:15:49 daisy kernel: [142681.721204]Â [<ffffffff81200f85>]
__sync_filesystem+0x95/0xa0
Apr 22 09:15:49 daisy kernel: [142681.721208]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:15:49 daisy kernel: [142681.721212]Â [<ffffffff812016e5>]
sys_sync+0x155/0x1a0
Apr 22 09:15:49 daisy kernel: [142681.721217]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:17:49 daisy kernel: [142801.721064] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:17:49 daisy kernel: [142801.721082]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:17:49 daisy kernel: [142801.721091] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:17:49 daisy kernel: [142801.721107] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:17:49 daisy kernel: [142801.721114]Â ffff88006654bcb8
0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:17:49 daisy kernel: [142801.721119]Â ffff88000004d238
ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:17:49 daisy kernel: [142801.721124]Â ffff88011a707000
ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:17:49 daisy kernel: [142801.721128] Call Trace:
Apr 22 09:17:49 daisy kernel: [142801.721137]Â [<ffffffff8114f130>] ?
sync_page+0x0/0x50
Apr 22 09:17:49 daisy kernel: [142801.721143]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:17:49 daisy kernel: [142801.721149]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:17:49 daisy kernel: [142801.721154]Â [<ffffffff81067432>] ?
check_preempt_curr+0x82/0xa0
Apr 22 09:17:49 daisy kernel: [142801.721158]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:17:49 daisy kernel: [142801.721162]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:17:49 daisy kernel: [142801.721166]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:17:49 daisy kernel: [142801.721172]Â [<ffffffff811f98d8>]
sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:17:49 daisy kernel: [142801.721176]Â [<ffffffff8114fa6f>] ?
filemap_fdatawait+0x2f/0x40
Apr 22 09:17:49 daisy kernel: [142801.721180]Â [<ffffffff81200f85>]
__sync_filesystem+0x95/0xa0
Apr 22 09:17:49 daisy kernel: [142801.721184]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:17:49 daisy kernel: [142801.721188]Â [<ffffffff812016e5>]
sys_sync+0x155/0x1a0
Apr 22 09:17:49 daisy kernel: [142801.721192]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:20:01 daisy ata_id[22405]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:21:49 daisy kernel: [143041.721494] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:21:49 daisy kernel: [143041.721512]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:21:49 daisy kernel: [143041.721522] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:21:49 daisy kernel: [143041.721691] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:21:49 daisy kernel: [143041.721697]Â ffff88006654bcc8
0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:21:49 daisy kernel: [143041.721702]Â ffff880028200000
000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:21:49 daisy kernel: [143041.721706]Â ffff88006654bc68
ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:21:49 daisy kernel: [143041.721711] Call Trace:
Apr 22 09:21:49 daisy kernel: [143041.721720]Â [<ffffffff810098af>] ?
__switch_to+0x16f/0x470
Apr 22 09:21:49 daisy kernel: [143041.721726]Â [<ffffffff8107bbfe>] ?
finish_task_switch+0xce/0x120
Apr 22 09:21:49 daisy kernel: [143041.721730]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:21:49 daisy kernel: [143041.721735]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:21:49 daisy kernel: [143041.721739]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:21:49 daisy kernel: [143041.721743]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:21:49 daisy kernel: [143041.721747]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:21:49 daisy kernel: [143041.721753]Â [<ffffffff811f9773>]
writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:21:49 daisy kernel: [143041.721757]Â [<ffffffff811f9806>]
writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:21:49 daisy kernel: [143041.721762]Â [<ffffffff81200f38>]
__sync_filesystem+0x48/0xa0
Apr 22 09:21:49 daisy kernel: [143041.721765]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:21:49 daisy kernel: [143041.721769]Â [<ffffffff812016d8>]
sys_sync+0x148/0x1a0
Apr 22 09:21:49 daisy kernel: [143041.721773]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:23:49 daisy kernel: [143161.721064] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:23:49 daisy kernel: [143161.721169]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:23:49 daisy kernel: [143161.721259] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:23:49 daisy kernel: [143161.721430] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:23:49 daisy kernel: [143161.721437]Â ffff88006654bcc8
0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:23:49 daisy kernel: [143161.721442]Â ffff880028200000
000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:23:49 daisy kernel: [143161.721447]Â ffff88006654bc68
ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:23:49 daisy kernel: [143161.721451] Call Trace:
Apr 22 09:23:49 daisy kernel: [143161.721460]Â [<ffffffff810098af>] ?
__switch_to+0x16f/0x470
Apr 22 09:23:49 daisy kernel: [143161.721466]Â [<ffffffff8107bbfe>] ?
finish_task_switch+0xce/0x120
Apr 22 09:23:49 daisy kernel: [143161.721470]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:23:49 daisy kernel: [143161.721475]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:23:49 daisy kernel: [143161.721479]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:23:49 daisy kernel: [143161.721483]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:23:49 daisy kernel: [143161.721487]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:23:49 daisy kernel: [143161.721493]Â [<ffffffff811f9773>]
writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:23:49 daisy kernel: [143161.721498]Â [<ffffffff811f9806>]
writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:23:49 daisy kernel: [143161.721502]Â [<ffffffff81200f38>]
__sync_filesystem+0x48/0xa0
Apr 22 09:23:49 daisy kernel: [143161.721506]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:23:49 daisy kernel: [143161.721510]Â [<ffffffff812016d8>]
sys_sync+0x148/0x1a0
Apr 22 09:23:49 daisy kernel: [143161.721514]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:25:02 daisy ata_id[22445]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:25:49 daisy kernel: [143281.721066] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:25:49 daisy kernel: [143281.721159]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:25:49 daisy kernel: [143281.721244] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:25:49 daisy kernel: [143281.721408] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:25:49 daisy kernel: [143281.721415]Â ffff88006654bcc8
0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:25:49 daisy kernel: [143281.721420]Â ffff880028200000
000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:25:49 daisy kernel: [143281.721424]Â ffff88006654bc68
ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:25:49 daisy kernel: [143281.721429] Call Trace:
Apr 22 09:25:49 daisy kernel: [143281.721438]Â [<ffffffff810098af>] ?
__switch_to+0x16f/0x470
Apr 22 09:25:49 daisy kernel: [143281.721444]Â [<ffffffff8107bbfe>] ?
finish_task_switch+0xce/0x120
Apr 22 09:25:49 daisy kernel: [143281.721448]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:25:49 daisy kernel: [143281.721453]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:25:49 daisy kernel: [143281.721457]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:25:49 daisy kernel: [143281.721461]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:25:49 daisy kernel: [143281.721465]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:25:49 daisy kernel: [143281.721471]Â [<ffffffff811f9773>]
writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:25:49 daisy kernel: [143281.721476]Â [<ffffffff811f9806>]
writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:25:49 daisy kernel: [143281.721480]Â [<ffffffff81200f38>]
__sync_filesystem+0x48/0xa0
Apr 22 09:25:49 daisy kernel: [143281.721484]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:25:49 daisy kernel: [143281.721487]Â [<ffffffff812016d8>]
sys_sync+0x148/0x1a0
Apr 22 09:25:49 daisy kernel: [143281.721492]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:27:49 daisy kernel: [143401.721072] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:27:49 daisy kernel: [143401.721165]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:27:49 daisy kernel: [143401.721253] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:27:49 daisy kernel: [143401.721421] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000080
Apr 22 09:27:49 daisy kernel: [143401.721427]Â ffff88006654bcc8
0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:27:49 daisy kernel: [143401.721432]Â ffff880028200000
000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:27:49 daisy kernel: [143401.721436]Â ffff88006654bc68
ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:27:49 daisy kernel: [143401.721441] Call Trace:
Apr 22 09:27:49 daisy kernel: [143401.721450]Â [<ffffffff810098af>] ?
__switch_to+0x16f/0x470
Apr 22 09:27:49 daisy kernel: [143401.721456]Â [<ffffffff8107bbfe>] ?
finish_task_switch+0xce/0x120
Apr 22 09:27:49 daisy kernel: [143401.721460]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:27:49 daisy kernel: [143401.721465]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:27:49 daisy kernel: [143401.721469]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:27:49 daisy kernel: [143401.721473]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:27:49 daisy kernel: [143401.721477]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:27:49 daisy kernel: [143401.721483]Â [<ffffffff811f9773>]
writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:27:49 daisy kernel: [143401.721487]Â [<ffffffff811f9806>]
writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:27:49 daisy kernel: [143401.721492]Â [<ffffffff81200f38>]
__sync_filesystem+0x48/0xa0
Apr 22 09:27:49 daisy kernel: [143401.721495]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:27:49 daisy kernel: [143401.721499]Â [<ffffffff812016d8>]
sys_sync+0x148/0x1a0
Apr 22 09:27:49 daisy kernel: [143401.721503]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:29:49 daisy kernel: [143521.721059] INFO: task hdparm:22246
blocked for more than 120 seconds.
Apr 22 09:29:49 daisy kernel: [143521.721158]Â Â Â Â Â Â Not tainted
2.6.32-042stab142.1 #1
Apr 22 09:29:49 daisy kernel: [143521.721245] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:29:49 daisy kernel: [143521.721415] hdparm       D
ffff88000c778300Â Â Â Â 0 22246Â 20845Â Â Â 0 0x00000084
Apr 22 09:29:49 daisy kernel: [143521.721421]Â ffff88006654bcc8
0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:29:49 daisy kernel: [143521.721426]Â ffff880028200000
000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:29:49 daisy kernel: [143521.721431]Â ffff88006654bc68
ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:29:49 daisy kernel: [143521.721436] Call Trace:
Apr 22 09:29:49 daisy kernel: [143521.721445]Â [<ffffffff810098af>] ?
__switch_to+0x16f/0x470
Apr 22 09:29:49 daisy kernel: [143521.721451]Â [<ffffffff8107bbfe>] ?
finish_task_switch+0xce/0x120
Apr 22 09:29:49 daisy kernel: [143521.721455]Â [<ffffffff8107c851>] ?
update_curr+0xe1/0x1f0
Apr 22 09:29:49 daisy kernel: [143521.721460]Â [<ffffffff81566c55>]
schedule_timeout+0x215/0x2f0
Apr 22 09:29:49 daisy kernel: [143521.721465]Â [<ffffffff815669b4>]
wait_for_completion+0xe4/0x120
Apr 22 09:29:49 daisy kernel: [143521.721469]Â [<ffffffff81071ce0>] ?
default_wake_function+0x0/0x20
Apr 22 09:29:49 daisy kernel: [143521.721473]Â [<ffffffff815694db>] ?
_spin_unlock_bh+0x1b/0x20
Apr 22 09:29:49 daisy kernel: [143521.721479]Â [<ffffffff811f9773>]
writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:29:49 daisy kernel: [143521.721483]Â [<ffffffff811f9806>]
writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:29:49 daisy kernel: [143521.721487]Â [<ffffffff81200f38>]
__sync_filesystem+0x48/0xa0
Apr 22 09:29:49 daisy kernel: [143521.721491]Â [<ffffffff8120151d>]
sync_filesystems+0x30d/0x350
Apr 22 09:29:49 daisy kernel: [143521.721495]Â [<ffffffff812016d8>]
sys_sync+0x148/0x1a0
Apr 22 09:29:49 daisy kernel: [143521.721499]Â [<ffffffff81571424>]
system_call_fastpath+0x22/0x3a
Apr 22 09:30:04 daisy ata_id[22489]: HDIO_GET_IDENTITY failed for '/dev/sdb'
------------------
I tried running hdparm -tT /dev/sda, but after waiting 5+ minutes for
any command output I cancelled it.Â
I am rsyncing the data from this system over to another system now,
clearly something is wrong, but I can't tell what.Â
The system is an older AMD Opteron 180 processor (dual core) 4 GB ram,
RAID controller with RAID 5 set up with 4x 4TB Western Digital Drives.
I rebooted the system day before yesterday, and that's when the timeout
messages started pouring into the log.  Â
when I run tw_cli /c8 show, all four drives say they are ok
[root@daisy cron.daily]# tw_cli /c8 show
Unit UnitType Status        %RCmpl %V/I/M Stripe Size(GB) CacheÂ
AVrfy
------------------------------------------------------------------------------
u0   RAID-5   OK            -      -      256K   11175.8  Ri    ON
VPort Status        Unit Size     Type Phy Encl-Slot   Model
------------------------------------------------------------------------------
p0Â Â Â OKÂ Â Â Â Â Â Â Â Â Â Â Â u0Â Â 3.63 TBÂ Â SATAÂ 0Â Â -Â Â Â Â Â Â Â Â Â Â Â WDC
WD4005FZBX-00K5
p1Â Â Â OKÂ Â Â Â Â Â Â Â Â Â Â Â u0Â Â 3.63 TBÂ Â SATAÂ 1Â Â -Â Â Â Â Â Â Â Â Â Â Â WDC
WD4005FZBX-00K5
p2Â Â Â OKÂ Â Â Â Â Â Â Â Â Â Â Â u0Â Â 3.63 TBÂ Â SATAÂ 2Â Â -Â Â Â Â Â Â Â Â Â Â Â WDC
WD4005FZBX-00K5
p3Â Â Â OKÂ Â Â Â Â Â Â Â Â Â Â Â u0Â Â 3.63 TBÂ Â SATAÂ 3Â Â -Â Â Â Â Â Â Â Â Â Â Â WDC
WD4005FZBX-00K5
Logical Volumes appear active:
[root@daisy cron.daily]# lvscan
 ACTIVE           '/dev/vg_daisy/lv_root' [10.89 TiB] inherit
 ACTIVE           '/dev/vg_daisy/lv_swap' [3.88 GiB] inherit
 ACTIVE           '/dev/vg_daisy/lv_home' [20.00 GiB] inherit
[root@daisy cron.daily]#
[root@daisy cron.daily]# lvmdiskscan
 /dev/ram0            [     16.00 MiB]
 /dev/root            [     10.89 TiB]
 /dev/ram1            [     16.00 MiB]
 /dev/sda1            [      2.82 TiB]
 /dev/vg_daisy/lv_swap [      3.88 GiB]
 /dev/ram2            [     16.00 MiB]
 /dev/vg_daisy/lv_home [     20.00 GiB]
 /dev/ram3            [     16.00 MiB]
 /dev/sda3            [    842.87 GiB]
 /dev/ram4            [     16.00 MiB]
 /dev/ram5            [     16.00 MiB]
 /dev/ram6            [     16.00 MiB]
 /dev/ram7            [     16.00 MiB]
 /dev/ram8            [     16.00 MiB]
 /dev/ram9            [     16.00 MiB]
 /dev/ram10           [     16.00 MiB]
 /dev/ram11           [     16.00 MiB]
 /dev/ram12           [     16.00 MiB]
 /dev/ram13           [     16.00 MiB]
 /dev/ram14           [     16.00 MiB]
 /dev/ram15           [     16.00 MiB]
 /dev/sdb1            [      1.82 TiB] LVM physical volume
 /dev/sdc1            [    500.00 MiB]
 /dev/sdc2            [      4.00 TiB] LVM physical volume
 /dev/sdd1            [      4.00 TiB] LVM physical volume
 /dev/sde1            [      2.91 TiB] LVM physical volume
 3 disks
 19 partitions
 0 LVM physical volume whole disks
 4 LVM physical volumes
[root@daisy cron.daily]#
grub.conf:
[root@daisy grub]# cat grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
#Â Â Â Â Â Â Â Â Â all kernel and initrd paths are relative to /boot/, eg.
#Â Â Â Â Â Â Â Â Â root (hd0,0)
#Â Â Â Â Â Â Â Â Â kernel /vmlinuz-version ro root=/dev/mapper/vg_daisy-lv_root
#Â Â Â Â Â Â Â Â Â initrd /initrd-[generic-]version.img
#boot=/dev/sdb
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title OpenVZ (2.6.32-042stab142.1)
       root (hd0,0)
       kernel /vmlinuz-2.6.32-042stab142.1 ro
root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
       initrd /initramfs-2.6.32-042stab142.1.img
title OpenVZ (2.6.32-042stab141.3)
       root (hd0,0)
       kernel /vmlinuz-2.6.32-042stab141.3 ro
root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
       initrd /initramfs-2.6.32-042stab141.3.img
title OpenVZ (2.6.32-042stab140.4)
       root (hd0,0)
       kernel /vmlinuz-2.6.32-042stab140.4 ro
root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
       initrd /initramfs-2.6.32-042stab140.4.img
title OpenVZ (2.6.32-042stab140.1)
       root (hd0,0)
       kernel /vmlinuz-2.6.32-042stab140.1 ro
root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
       initrd /initramfs-2.6.32-042stab140.1.img
title OpenVZ (2.6.32-042stab139.1)
       root (hd0,0)
       kernel /vmlinuz-2.6.32-042stab139.1 ro
root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
       initrd /initramfs-2.6.32-042stab139.1.img
title CentOS 6 (2.6.32-754.el6.x86_64)
       root (hd0,0)
       kernel /vmlinuz-2.6.32-754.el6.x86_64 ro
root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
       initrd /initramfs-2.6.32-754.el6.x86_64.img
-------------
Top is not showing anything out of the ordinary:
----------
[root@daisy grub]#
top - 09:41:57 up 1 day, 16:04, 3 users, load average: 5.89, 5.83, 5.43
Tasks: 369 total,  1 running, 368 sleeping,  0 stopped,  0 zombie
Cpu(s): 0.2%us, 1.2%sy, 0.0%ni, 25.0%id, 73.5%wa, 0.0%hi, 0.2%si,Â
0.0%st
Mem:  3894628k total, 3861280k used,   33348k free,   95608k buffers
Swap: 4063228k total,   34888k used, 4028340k free, 3139272k cached
   PID USER     PR NI VIRT RES SHR S %CPU %MEM   TIME+ COMMAND
  1266 root     20  0    0   0   0 D 1.0 0.0 12:27.75 flush-253:0
 21041 1153     20  0 3188 1840 1012 D 0.7 0.0  0:00.72 imap
 21599 97       20  0 5160 1940 1568 S 0.7 0.0  0:01.06 imap-login
 22636 root     20  0 15272 1524 964 R 0.7 0.0  0:00.06 top
  1977 root     20  0 2096 644 360 S 0.3 0.0  0:27.92 dovecot
 22528 97       20  0 5160 2044 1672 S 0.3 0.1  0:00.35 imap-login
 22578 1155     20  0 2904 1528 940 D 0.3 0.0  0:00.22 imap
     1 root     20  0 19236 268 136 S 0.0 0.0  0:00.68 init
     2 root     20  0    0   0   0 S 0.0 0.0  0:00.00 kthreadd
     3 root     RT  0    0   0   0 S 0.0 0.0  0:00.04 migration/0
     4 root     20  0    0   0   0 S 0.0 0.0  0:01.88 ksoftirqd/0
     5 root     RT  0    0   0   0 S 0.0 0.0  0:00.00 stopper/0
     6 root     RT  0    0   0   0 S 0.0 0.0  0:00.19 watchdog/0
     7 root     RT  0    0   0   0 S 0.0 0.0  0:00.07 migration/1
     8 root     RT  0    0   0   0 S 0.0 0.0  0:00.00 stopper/1
     9 root     20  0    0   0   0 S 0.0 0.0  0:03.17 ksoftirqd/1
    10 root     RT  0    0   0   0 S 0.0 0.0  0:00.20 watchdog/1
    11 root     20  0    0   0   0 S 0.0 0.0  0:07.23 events/0
    12 root     20  0    0   0   0 S 0.0 0.0  0:08.55 events/1
    13 root     20  0    0   0   0 S 0.0 0.0  0:00.00 events/0
    14 root     20  0    0   0   0 S 0.0 0.0  0:00.00 events/1
    15 root     20  0    0   0   0 S 0.0 0.0  0:00.00 events_long/0
    16 root     20  0    0   0   0 S 0.0 0.0  0:00.00 events_long/1
    17 root     20  0    0   0   0 S 0.0 0.0  0:00.00
events_power_ef
    18 root     20  0    0   0   0 S 0.0 0.0  0:00.00
events_power_ef
    19 root     20  0    0   0   0 S 0.0 0.0  0:00.00 cgroup
    20 root     20  0    0   0   0 S 0.0 0.0  0:00.00 khelper
    21 root     20  0    0   0   0 S 0.0 0.0  0:00.01 netns
    22 root     20  0    0   0   0 S 0.0 0.0  0:00.00 async/mgr
    23 root     20  0    0   0   0 S 0.0 0.0  0:00.00 pm
    24 root     20  0    0   0   0 S 0.0 0.0  0:00.29 sync_supers
------------
This is a company production mail server, and I can't find the solution,
I need help, as soon as someone is able, thank you!