[CentOS] Kernel errors after updating

C. L. Martinez

carlopmart at gmail.com
Thu Sep 4 14:19:41 UTC 2014 

Hi all,

 I have updated my Centos 6.5 KVM host to kernel 2.6.32-431.23.3 this
morning ... After 2 hours working, the following kernel error appears
and all vm guests goes slowly ....

device monif4 entered promiscuous mode
monif: port 5(monif4) entering forwarding state
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task cgroup:83 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.23.3.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cgroup        D 0000000000000009     0    83      2 0x00000000
 ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
 ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
 [<ffffffff8152a36e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff810d24d0>] ? do_rebuild_sched_domains+0x0/0x50
 [<ffffffff8152a20b>] mutex_lock+0x2b/0x50
 [<ffffffff810c97b5>] cgroup_lock+0x15/0x20
 [<ffffffff810d24e8>] do_rebuild_sched_domains+0x18/0x50
 [<ffffffff81094a20>] worker_thread+0x170/0x2a0
 [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20

Uhmm , how can I debug this?? Any idea??

More information about the CentOS mailing list