Hi all,
I have updated my Centos 6.5 KVM host to kernel 2.6.32-431.23.3 this morning ... After 2 hours working, the following kernel error appears and all vm guests goes slowly ....
device monif4 entered promiscuous mode monif: port 5(monif4) entering forwarding state INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20 INFO: task cgroup:83 blocked for more than 120 seconds. Not tainted 2.6.32-431.23.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. cgroup D 0000000000000009 0 83 2 0x00000000 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058 Call Trace: [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50 [<ffffffff81094a20>] worker_thread+0x170/0x2a0 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0 [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20
Uhmm , how can I debug this?? Any idea??