Xen PV DomU running Kernel 4.14.5-1.el7.elrepo.x86_64: xl -v vcpu-set <domU> <val> triggers domU kernel WARNING, then domU becomes unresponsive
Has anyone seen this recently? I couldn't replicate it on: - CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64, kernel-lt-4.4.105-1.el6.elrepo.x86_64 - CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on: - CentOS 6 running 4.14.5-1.el6.elrepo.x86_64 - CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU: - 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64 - 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour: - These commands stall: top ls -l /var/tmp ls -l /tmp - Stuck in D state on the CentOS 7 domU: root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
-----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 start here----- ------------[ cut here ]------------ WARNING: CPU: 4 PID: 60 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x9e/0xc0 Modules linked in: intel_cstate(-) ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack libcrc32c ip6table_filter ip6_tables dm_mod dax xen_netfront crc32_pclmul crct10dif_pclmul ghash_clmulni_intel crc32c_intel pcbc aesni_intel glue_helper crypto_simd cryptd aes_x86_64 coretemp hwmon x86_pkg_temp_thermal sb_edac intel_rapl_perf pcspkr ext4 jbd2 mbcache xen_blkfront CPU: 4 PID: 60 Comm: kworker/4:1H Not tainted 4.14.5-1.el6.elrepo.x86_64 #1 Workqueue: kblockd blk_mq_run_work_fn task: ffff8802711a2780 task.stack: ffffc90041af4000 RIP: e030:__blk_mq_run_hw_queue+0x9e/0xc0 RSP: e02b:ffffc90041af7c48 EFLAGS: 00010202 RAX: 0000000000000001 RBX: ffff88027117fa80 RCX: 0000000000000001 RDX: ffff88026b053ee0 RSI: ffff88027351bca0 RDI: ffff88026b072800 RBP: ffffc90041af7c68 R08: ffffc90041af7eb8 R09: ffff8802711a2810 R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff88026b072800 R13: ffffe8ffffd04d00 R14: 0000000000000000 R15: ffffe8ffffd04d05 FS: 00002b7b7c89b700(0000) GS:ffff880273500000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 000000026d953000 CR4: 0000000000042660 Call Trace: blk_mq_run_work_fn+0x31/0x40 process_one_work+0x174/0x440 ? xen_mc_flush+0xad/0x1b0 ? schedule+0x3a/0xa0 worker_thread+0x6b/0x410 ? default_wake_function+0x12/0x20 ? __wake_up_common+0x84/0x130 ? maybe_create_worker+0x120/0x120 ? schedule+0x3a/0xa0 ? _raw_spin_unlock_irqrestore+0x16/0x20 ? maybe_create_worker+0x120/0x120 kthread+0x111/0x150 ? __kthread_init_worker+0x40/0x40 ret_from_fork+0x25/0x30 Code: 89 df e8 06 2f d9 ff 4c 89 e7 41 89 c5 e8 0b 6e 00 00 44 89 ee 48 89 df e8 20 2f d9 ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 <0f> ff eb aa 4c 89 e7 e8 e6 6d 00 00 48 8b 5d e8 4c 8b 65 f0 4c ---[ end trace fe2aaf4e723042fd ]--- -----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 end here-----
-----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 start here----- [ 116.528885] ------------[ cut here ]------------ [ 116.528894] WARNING: CPU: 3 PID: 38 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 116.528898] Modules linked in: intel_cstate(-) ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 116.528919] CPU: 3 PID: 38 Comm: kworker/3:1H Not tainted 4.14.5-1.el7.elrepo.x86_64 #1 [ 116.529007] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 116.529034] ---[ end trace a7814e3ec9a330c6 ]--- [ 147.424117] ------------[ cut here ]------------ [ 147.424150] WARNING: CPU: 2 PID: 24 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 147.424160] Modules linked in: ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 147.424222] CPU: 2 PID: 24 Comm: kworker/2:0H Tainted: G W 4.14.5-1.el7.elrepo.x86_64 #1 [ 147.424238] Workqueue: kblockd blk_mq_run_work_fn [ 147.424247] task: ffff88007c539840 task.stack: ffffc900403e4000 [ 147.424259] RIP: e030:__blk_mq_run_hw_queue+0x89/0xa0 [ 147.424270] RSP: e02b:ffffc900403e7e30 EFLAGS: 00010202 [ 147.424279] RAX: 0000000000000001 RBX: ffff880003b83800 RCX: ffff88007d11bca0 [ 147.424288] RDX: ffff88007c656c88 RSI: 00000000000000a0 RDI: ffff880003b83800 [ 147.424298] RBP: ffffc900403e7e48 R08: 0000000000000000 R09: 0000000000000000 [ 147.424309] R10: 0000000000007ff0 R11: 00000000000074e5 R12: ffff88007c436900 [ 147.424319] R13: ffff88007d11bc80 R14: ffff88007d121b00 R15: ffff880003b83848 [ 147.424340] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:ffff88007d100000 [ 147.424350] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 147.424359] CR2: 00007f504f19a700 CR3: 0000000079bed000 CR4: 0000000000042660 [ 147.424370] Call Trace: [ 147.424384] blk_mq_run_work_fn+0x2c/0x30 [ 147.424400] process_one_work+0x149/0x360 [ 147.424411] worker_thread+0x4d/0x3e0 [ 147.424421] kthread+0x109/0x140 [ 147.424432] ? rescuer_thread+0x380/0x380 [ 147.424441] ? kthread_park+0x60/0x60 [ 147.424455] ret_from_fork+0x25/0x30 [ 147.424463] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 147.424554] ---[ end trace a7814e3ec9a330c7 ]--- -----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 end here-----
On Mon, Dec 11, 2017 at 4:52 PM, Adi Pircalabu adi@ddns.com.au wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
Adi Pircalabu
Can you test-install 4.15-rcX
to see if the problem persists in the latest kernel?:
http://elrepo.org/people/ajb/devel/kernel-ml/el7/x86_64/RPMS/
Akemi
On 15-12-2017 4:10, Akemi Yagi wrote:
On Mon, Dec 11, 2017 at 4:52 PM, Adi Pircalabu adi@ddns.com.au wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
Adi Pircalabu
Can you test-install 4.15-rcX to see if the problem persists in the latest kernel?:
http://elrepo.org/people/ajb/devel/kernel-ml/el7/x86_64/RPMS/ [1]
Akemi
Thanks for that, tested it on both CentOS 6 and 7 PV domU and I get similar panics:
-----CentOS 6----- [...] dracut: Switching root Welcome to CentOS Starting udev: udev: starting version 147 input: PC Speaker as /devices/platform/pcspkr/input/input0 xen_netfront: Initialising Xen virtual ethernet driver BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 IP: coretemp_cpu_online+0x116/0x190 [coretemp] PGD 7b5c7067 P4D 7b5c7067 PUD 7b5cd067 PMD 0 Oops: 0002 [#1] SMP Modules linked in: coretemp(+) hwmon xen_netfront pcspkr ext4 jbd2 mbcache xen_blkfront dm_mirror dm_region_hash dm_log dm_mod dax CPU: 0 PID: 12 Comm: cpuhp/0 Not tainted 4.15.0-0.rc1.el6.elrepo.x86_64 #1 task: ffff88007c8f43c0 task.stack: ffffc90040390000 RIP: e030:coretemp_cpu_online+0x116/0x190 [coretemp] RSP: e02b:ffffc90040393cd8 EFLAGS: 00010246 RAX: 0000000000000010 RBX: 0000000000000000 RCX: ffff88007c87c248 RDX: 0000000000000000 RSI: ffff880077720c28 RDI: ffff8800069ea020 RBP: ffffc90040393d18 R08: 0000000000000000 R09: ffffc90040393a08 R10: 0000000000000000 R11: 000000000000005f R12: 0000000000000000 R13: ffff8800069ea000 R14: ffff88007f60a040 R15: 0000000000000000 FS: 00007f685ca0a700(0000) GS:ffff88007f600000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000010 CR3: 000000000683a000 CR4: 0000000000042660 Call Trace: ? coretemp_add_core+0x50/0x50 [coretemp] cpuhp_invoke_callback+0xe9/0x700 ? put_prev_task_fair+0x26/0x40 ? __schedule+0x2d0/0x6e0 ? __wake_up_common+0x84/0x130 ? __wake_up_common+0x84/0x130 cpuhp_thread_fun+0xee/0x170 smpboot_thread_fn+0x10c/0x160 ? smpboot_create_threads+0x80/0x80 kthread+0x10a/0x140 ? kthread_probe_data+0x40/0x40 ret_from_fork+0x1f/0x30 Code: 11 15 41 e1 49 89 c5 b8 f4 ff ff ff 4d 85 ed 0f 84 66 ff ff ff 4c 89 ef e8 88 11 41 e1 85 c0 75 6e 48 8b 05 75 17 00 00 4d 63 ff <4e> 89 2c f8 49 81 fd 00 f0 ff ff 44 89 e8 0f 87 3c ff ff ff 49 RIP: coretemp_cpu_online+0x116/0x190 [coretemp] RSP: ffffc90040393cd8 CR2: 0000000000000010 ---[ end trace 8253bafacf228cf2 ]--- -----CentOS 6----- -----CentOS 7----- [...] [ OK ] Found device /dev/xvda2. Activating swap /dev/xvda2... [ 4.998940] alg: No test for pcbc(aes) (pcbc-aes-aesni) [ 5.001054] Adding 1048572k swap on /dev/xvda2. Priority:-2 extents:1 across:1048572k SSFS [ OK ] Activated swap /dev/xvda2. [ OK ] Reached target Swap. [ 5.020760] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 5.020767] IP: coretemp_cpu_online+0xf8/0x1f7 [coretemp] [ 5.020769] PGD 0 P4D 0 [ 5.020771] Oops: 0002 [#1] SMP [ 5.020773] Modules linked in: coretemp(+) crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd pcspkr intel_rapl_perf nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 5.020786] CPU: 0 PID: 12 Comm: cpuhp/0 Not tainted 4.15.0-0.rc3.el7.elrepo.x86_64 #1 [ 5.020789] RIP: e030:coretemp_cpu_online+0xf8/0x1f7 [coretemp] [ 5.020790] RSP: e02b:ffffc90040387e10 EFLAGS: 00010246 [ 5.020793] RAX: 0000000000000010 RBX: ffff8800040d8800 RCX: 0000000000000000 [ 5.020794] RDX: ffff880079761e70 RSI: ffff88007c438cc8 RDI: ffff8800040d8820 [ 5.020796] RBP: ffffc90040387e40 R08: 0000000000000000 R09: ffffffff81f8aff0 [ 5.020798] R10: ffff88007d01f400 R11: 0000000000000000 R12: 0000000000000000 [ 5.020800] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88007d00a020 [ 5.020804] FS: 0000000000000000(0000) GS:ffff88007d000000(0000) knlGS:0000000000000000 [ 5.020806] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.020808] CR2: 0000000000000010 CR3: 0000000004731000 CR4: 0000000000042660 [ 5.020810] Call Trace: [ 5.020814] ? create_core_data+0x5f0/0x5f0 [coretemp] [ 5.020817] cpuhp_invoke_callback+0xae/0x5c0 [ 5.020820] ? __schedule+0x295/0x880 [ 5.020823] cpuhp_thread_fun+0xce/0x170 [ 5.020825] smpboot_thread_fn+0x110/0x160 [ 5.020827] kthread+0x102/0x140 [ 5.020828] ? sort_range+0x30/0x30 [ 5.020831] ? kthread_associate_blkcg+0xa0/0xa0 [ 5.020833] ret_from_fork+0x1f/0x30 [ 5.020834] Code: 21 a0 41 0f b7 f6 e8 38 73 30 e1 48 89 c3 b8 f4 ff ff ff 48 85 db 74 c0 48 89 df e8 d3 69 30 e1 85 c0 75 7c 48 8b 05 40 18 00 00 <4a> 89 1c f0 48 81 fb 00 f0 ff ff 0f 87 e7 00 00 00 49 8b 47 4c [ 5.020852] RIP: coretemp_cpu_online+0xf8/0x1f7 [coretemp] RSP: ffffc90040387e10 [ 5.020854] CR2: 0000000000000010 [ 5.020856] ---[ end trace 9ce91afe6b362317 ]--- [ 5.020858] Kernel panic - not syncing: Fatal exception [ 5.020861] Kernel Offset: disabled -----CentOS 7-----
For CentOS 7 I've also tried booting it using vcpus = 1 and no maxvcpus with the same outcome. Looks like 4.15-rc is a no go for me :)
Cheers,
--- Adi Pircalabu
On 15-12-2017 9:01, Adi Pircalabu wrote:
On 15-12-2017 4:10, Akemi Yagi wrote:
On Mon, Dec 11, 2017 at 4:52 PM, Adi Pircalabu adi@ddns.com.au wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
Adi Pircalabu
Can you test-install 4.15-rcX to see if the problem persists in the latest kernel?:
http://elrepo.org/people/ajb/devel/kernel-ml/el7/x86_64/RPMS/ [1]
Akemi
Thanks for that, tested it on both CentOS 6 and 7 PV domU and I get similar panics:
-----CentOS 6----- [...] dracut: Switching root Welcome to CentOS Starting udev: udev: starting version 147 input: PC Speaker as /devices/platform/pcspkr/input/input0 xen_netfront: Initialising Xen virtual ethernet driver BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 IP: coretemp_cpu_online+0x116/0x190 [coretemp] PGD 7b5c7067 P4D 7b5c7067 PUD 7b5cd067 PMD 0 Oops: 0002 [#1] SMP Modules linked in: coretemp(+) hwmon xen_netfront pcspkr ext4 jbd2 mbcache xen_blkfront dm_mirror dm_region_hash dm_log dm_mod dax CPU: 0 PID: 12 Comm: cpuhp/0 Not tainted 4.15.0-0.rc1.el6.elrepo.x86_64 #1 task: ffff88007c8f43c0 task.stack: ffffc90040390000 RIP: e030:coretemp_cpu_online+0x116/0x190 [coretemp] RSP: e02b:ffffc90040393cd8 EFLAGS: 00010246 RAX: 0000000000000010 RBX: 0000000000000000 RCX: ffff88007c87c248 RDX: 0000000000000000 RSI: ffff880077720c28 RDI: ffff8800069ea020 RBP: ffffc90040393d18 R08: 0000000000000000 R09: ffffc90040393a08 R10: 0000000000000000 R11: 000000000000005f R12: 0000000000000000 R13: ffff8800069ea000 R14: ffff88007f60a040 R15: 0000000000000000 FS: 00007f685ca0a700(0000) GS:ffff88007f600000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000010 CR3: 000000000683a000 CR4: 0000000000042660 Call Trace: ? coretemp_add_core+0x50/0x50 [coretemp] cpuhp_invoke_callback+0xe9/0x700 ? put_prev_task_fair+0x26/0x40 ? __schedule+0x2d0/0x6e0 ? __wake_up_common+0x84/0x130 ? __wake_up_common+0x84/0x130 cpuhp_thread_fun+0xee/0x170 smpboot_thread_fn+0x10c/0x160 ? smpboot_create_threads+0x80/0x80 kthread+0x10a/0x140 ? kthread_probe_data+0x40/0x40 ret_from_fork+0x1f/0x30 Code: 11 15 41 e1 49 89 c5 b8 f4 ff ff ff 4d 85 ed 0f 84 66 ff ff ff 4c 89 ef e8 88 11 41 e1 85 c0 75 6e 48 8b 05 75 17 00 00 4d 63 ff <4e> 89 2c f8 49 81 fd 00 f0 ff ff 44 89 e8 0f 87 3c ff ff ff 49 RIP: coretemp_cpu_online+0x116/0x190 [coretemp] RSP: ffffc90040393cd8 CR2: 0000000000000010 ---[ end trace 8253bafacf228cf2 ]--- -----CentOS 6----- -----CentOS 7----- [...] [ OK ] Found device /dev/xvda2. Activating swap /dev/xvda2... [ 4.998940] alg: No test for pcbc(aes) (pcbc-aes-aesni) [ 5.001054] Adding 1048572k swap on /dev/xvda2. Priority:-2 extents:1 across:1048572k SSFS [ OK ] Activated swap /dev/xvda2. [ OK ] Reached target Swap. [ 5.020760] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 5.020767] IP: coretemp_cpu_online+0xf8/0x1f7 [coretemp] [ 5.020769] PGD 0 P4D 0 [ 5.020771] Oops: 0002 [#1] SMP [ 5.020773] Modules linked in: coretemp(+) crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd pcspkr intel_rapl_perf nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 5.020786] CPU: 0 PID: 12 Comm: cpuhp/0 Not tainted 4.15.0-0.rc3.el7.elrepo.x86_64 #1 [ 5.020789] RIP: e030:coretemp_cpu_online+0xf8/0x1f7 [coretemp] [ 5.020790] RSP: e02b:ffffc90040387e10 EFLAGS: 00010246 [ 5.020793] RAX: 0000000000000010 RBX: ffff8800040d8800 RCX: 0000000000000000 [ 5.020794] RDX: ffff880079761e70 RSI: ffff88007c438cc8 RDI: ffff8800040d8820 [ 5.020796] RBP: ffffc90040387e40 R08: 0000000000000000 R09: ffffffff81f8aff0 [ 5.020798] R10: ffff88007d01f400 R11: 0000000000000000 R12: 0000000000000000 [ 5.020800] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88007d00a020 [ 5.020804] FS: 0000000000000000(0000) GS:ffff88007d000000(0000) knlGS:0000000000000000 [ 5.020806] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.020808] CR2: 0000000000000010 CR3: 0000000004731000 CR4: 0000000000042660 [ 5.020810] Call Trace: [ 5.020814] ? create_core_data+0x5f0/0x5f0 [coretemp] [ 5.020817] cpuhp_invoke_callback+0xae/0x5c0 [ 5.020820] ? __schedule+0x295/0x880 [ 5.020823] cpuhp_thread_fun+0xce/0x170 [ 5.020825] smpboot_thread_fn+0x110/0x160 [ 5.020827] kthread+0x102/0x140 [ 5.020828] ? sort_range+0x30/0x30 [ 5.020831] ? kthread_associate_blkcg+0xa0/0xa0 [ 5.020833] ret_from_fork+0x1f/0x30 [ 5.020834] Code: 21 a0 41 0f b7 f6 e8 38 73 30 e1 48 89 c3 b8 f4 ff ff ff 48 85 db 74 c0 48 89 df e8 d3 69 30 e1 85 c0 75 7c 48 8b 05 40 18 00 00 <4a> 89 1c f0 48 81 fb 00 f0 ff ff 0f 87 e7 00 00 00 49 8b 47 4c [ 5.020852] RIP: coretemp_cpu_online+0xf8/0x1f7 [coretemp] RSP: ffffc90040387e10 [ 5.020854] CR2: 0000000000000010 [ 5.020856] ---[ end trace 9ce91afe6b362317 ]--- [ 5.020858] Kernel panic - not syncing: Fatal exception [ 5.020861] Kernel Offset: disabled -----CentOS 7-----
For CentOS 7 I've also tried booting it using vcpus = 1 and no maxvcpus with the same outcome. Looks like 4.15-rc is a no go for me :)
We might be getting a bit sidetracked here, but also noticed this boot warning on CentOS 6 with 4.14.5-1.el6.elrepo.x86_64:
xen_netfront: Initialising Xen virtual ethernet driver sysfs: cannot create duplicate filename '/devices/platform/coretemp.0/hwmon/hwmon0/temp2_label' ------------[ cut here ]------------ WARNING: CPU: 1 PID: 13 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x63/0x80 Modules linked in: coretemp(+) hwmon xen_netfront pcspkr ext4 jbd2 mbcache xen_blkfront dm_mirror dm_region_hash dm_log dm_mod dax CPU: 1 PID: 13 Comm: cpuhp/1 Not tainted 4.14.5-1.el6.elrepo.x86_64 #1 task: ffff88007c8f8400 task.stack: ffffc90040398000 RIP: e030:sysfs_warn_dup+0x63/0x80 RSP: e02b:ffffc9004039bb18 EFLAGS: 00010296 RAX: 000000000000005f RBX: ffff88000683f000 RCX: ffffffff81e5f308 RDX: 0000000000000001 RSI: 00000000000000e2 RDI: 0000000000000200 RBP: ffffc9004039bb38 R08: 0000000000000000 R09: 0720072007200720 R10: 0720072007200720 R11: 0000000000000000 R12: ffff880006aa8cf8 R13: ffff880006b27180 R14: 0000000000000000 R15: ffff880006b27180 FS: 00007f9f1475d700(0000) GS:ffff88007f680000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000001381000 CR3: 000000007caa3000 CR4: 0000000000042660 Call Trace: sysfs_add_file_mode_ns+0x150/0x190 ? smp_call_function_single+0xc3/0x100 create_files+0x7e/0x1c0 internal_create_group+0x8e/0x130 sysfs_create_group+0x13/0x20 create_core_data+0x29d/0x390 [coretemp] ? coretemp_add_core+0x50/0x50 [coretemp] coretemp_add_core+0x20/0x50 [coretemp] coretemp_cpu_online+0xce/0x190 [coretemp] ? coretemp_add_core+0x50/0x50 [coretemp] cpuhp_invoke_callback+0xe9/0x700 ? __schedule+0x2cd/0x6e0 ? __wake_up_common+0x84/0x130 ? __wake_up_common+0x84/0x130 cpuhp_thread_fun+0xeb/0x170 smpboot_thread_fn+0x10c/0x160 ? smpboot_create_threads+0x80/0x80 kthread+0x111/0x150 ? __kthread_init_worker+0x40/0x40 ret_from_fork+0x25/0x30 Code: 48 89 c3 74 12 b9 00 10 00 00 48 89 c2 31 f6 4c 89 ef e8 61 d1 ff ff 4c 89 e2 48 89 de 48 c7 c7 b8 4c cb 81 31 c0 e8 31 07 e1 ff <0f> ff 48 89 df e8 63 c4 f4 ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d ---[ end trace af2e9ac25142c3a0 ]--- coretemp coretemp.0: Adding Core 1 failed
--- Adi Pircalabu
Adi,
Thanks for your detailed report -- would you mind reposting this to xen-devel@lists.xenproject.org? This looks like a general Xen / Linux bug (not specific to the Virt SIG packages), and there are a lot more eyeballs there.
Thanks, -George
On Tue, Dec 12, 2017 at 12:52 AM, Adi Pircalabu adi@ddns.com.au wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
-----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 start here----- ------------[ cut here ]------------ WARNING: CPU: 4 PID: 60 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x9e/0xc0 Modules linked in: intel_cstate(-) ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack libcrc32c ip6table_filter ip6_tables dm_mod dax xen_netfront crc32_pclmul crct10dif_pclmul ghash_clmulni_intel crc32c_intel pcbc aesni_intel glue_helper crypto_simd cryptd aes_x86_64 coretemp hwmon x86_pkg_temp_thermal sb_edac intel_rapl_perf pcspkr ext4 jbd2 mbcache xen_blkfront CPU: 4 PID: 60 Comm: kworker/4:1H Not tainted 4.14.5-1.el6.elrepo.x86_64 #1 Workqueue: kblockd blk_mq_run_work_fn task: ffff8802711a2780 task.stack: ffffc90041af4000 RIP: e030:__blk_mq_run_hw_queue+0x9e/0xc0 RSP: e02b:ffffc90041af7c48 EFLAGS: 00010202 RAX: 0000000000000001 RBX: ffff88027117fa80 RCX: 0000000000000001 RDX: ffff88026b053ee0 RSI: ffff88027351bca0 RDI: ffff88026b072800 RBP: ffffc90041af7c68 R08: ffffc90041af7eb8 R09: ffff8802711a2810 R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff88026b072800 R13: ffffe8ffffd04d00 R14: 0000000000000000 R15: ffffe8ffffd04d05 FS: 00002b7b7c89b700(0000) GS:ffff880273500000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 000000026d953000 CR4: 0000000000042660 Call Trace: blk_mq_run_work_fn+0x31/0x40 process_one_work+0x174/0x440 ? xen_mc_flush+0xad/0x1b0 ? schedule+0x3a/0xa0 worker_thread+0x6b/0x410 ? default_wake_function+0x12/0x20 ? __wake_up_common+0x84/0x130 ? maybe_create_worker+0x120/0x120 ? schedule+0x3a/0xa0 ? _raw_spin_unlock_irqrestore+0x16/0x20 ? maybe_create_worker+0x120/0x120 kthread+0x111/0x150 ? __kthread_init_worker+0x40/0x40 ret_from_fork+0x25/0x30 Code: 89 df e8 06 2f d9 ff 4c 89 e7 41 89 c5 e8 0b 6e 00 00 44 89 ee 48 89 df e8 20 2f d9 ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 <0f> ff eb aa 4c 89 e7 e8 e6 6d 00 00 48 8b 5d e8 4c 8b 65 f0 4c ---[ end trace fe2aaf4e723042fd ]--- -----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 end here-----
-----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 start here----- [ 116.528885] ------------[ cut here ]------------ [ 116.528894] WARNING: CPU: 3 PID: 38 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 116.528898] Modules linked in: intel_cstate(-) ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 116.528919] CPU: 3 PID: 38 Comm: kworker/3:1H Not tainted 4.14.5-1.el7.elrepo.x86_64 #1 [ 116.529007] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 116.529034] ---[ end trace a7814e3ec9a330c6 ]--- [ 147.424117] ------------[ cut here ]------------ [ 147.424150] WARNING: CPU: 2 PID: 24 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 147.424160] Modules linked in: ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 147.424222] CPU: 2 PID: 24 Comm: kworker/2:0H Tainted: G W 4.14.5-1.el7.elrepo.x86_64 #1 [ 147.424238] Workqueue: kblockd blk_mq_run_work_fn [ 147.424247] task: ffff88007c539840 task.stack: ffffc900403e4000 [ 147.424259] RIP: e030:__blk_mq_run_hw_queue+0x89/0xa0 [ 147.424270] RSP: e02b:ffffc900403e7e30 EFLAGS: 00010202 [ 147.424279] RAX: 0000000000000001 RBX: ffff880003b83800 RCX: ffff88007d11bca0 [ 147.424288] RDX: ffff88007c656c88 RSI: 00000000000000a0 RDI: ffff880003b83800 [ 147.424298] RBP: ffffc900403e7e48 R08: 0000000000000000 R09: 0000000000000000 [ 147.424309] R10: 0000000000007ff0 R11: 00000000000074e5 R12: ffff88007c436900 [ 147.424319] R13: ffff88007d11bc80 R14: ffff88007d121b00 R15: ffff880003b83848 [ 147.424340] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:ffff88007d100000 [ 147.424350] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 147.424359] CR2: 00007f504f19a700 CR3: 0000000079bed000 CR4: 0000000000042660 [ 147.424370] Call Trace: [ 147.424384] blk_mq_run_work_fn+0x2c/0x30 [ 147.424400] process_one_work+0x149/0x360 [ 147.424411] worker_thread+0x4d/0x3e0 [ 147.424421] kthread+0x109/0x140 [ 147.424432] ? rescuer_thread+0x380/0x380 [ 147.424441] ? kthread_park+0x60/0x60 [ 147.424455] ret_from_fork+0x25/0x30 [ 147.424463] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 147.424554] ---[ end trace a7814e3ec9a330c7 ]--- -----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 end here-----
-- Adi Pircalabu _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
There are a couple of xen updates in the 4.9.66 and 4.9.68 kernels:
https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.66 https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.68
Let me build a newer Dom0 kernel and see if that helps.
Thanks, Johnny Hughes
On 12/11/2017 06:52 PM, Adi Pircalabu wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
-----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 start here----- ------------[ cut here ]------------ WARNING: CPU: 4 PID: 60 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x9e/0xc0 Modules linked in: intel_cstate(-) ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack libcrc32c ip6table_filter ip6_tables dm_mod dax xen_netfront crc32_pclmul crct10dif_pclmul ghash_clmulni_intel crc32c_intel pcbc aesni_intel glue_helper crypto_simd cryptd aes_x86_64 coretemp hwmon x86_pkg_temp_thermal sb_edac intel_rapl_perf pcspkr ext4 jbd2 mbcache xen_blkfront CPU: 4 PID: 60 Comm: kworker/4:1H Not tainted 4.14.5-1.el6.elrepo.x86_64 #1 Workqueue: kblockd blk_mq_run_work_fn task: ffff8802711a2780 task.stack: ffffc90041af4000 RIP: e030:__blk_mq_run_hw_queue+0x9e/0xc0 RSP: e02b:ffffc90041af7c48 EFLAGS: 00010202 RAX: 0000000000000001 RBX: ffff88027117fa80 RCX: 0000000000000001 RDX: ffff88026b053ee0 RSI: ffff88027351bca0 RDI: ffff88026b072800 RBP: ffffc90041af7c68 R08: ffffc90041af7eb8 R09: ffff8802711a2810 R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff88026b072800 R13: ffffe8ffffd04d00 R14: 0000000000000000 R15: ffffe8ffffd04d05 FS: 00002b7b7c89b700(0000) GS:ffff880273500000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 000000026d953000 CR4: 0000000000042660 Call Trace: blk_mq_run_work_fn+0x31/0x40 process_one_work+0x174/0x440 ? xen_mc_flush+0xad/0x1b0 ? schedule+0x3a/0xa0 worker_thread+0x6b/0x410 ? default_wake_function+0x12/0x20 ? __wake_up_common+0x84/0x130 ? maybe_create_worker+0x120/0x120 ? schedule+0x3a/0xa0 ? _raw_spin_unlock_irqrestore+0x16/0x20 ? maybe_create_worker+0x120/0x120 kthread+0x111/0x150 ? __kthread_init_worker+0x40/0x40 ret_from_fork+0x25/0x30 Code: 89 df e8 06 2f d9 ff 4c 89 e7 41 89 c5 e8 0b 6e 00 00 44 89 ee 48 89 df e8 20 2f d9 ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 <0f> ff eb aa 4c 89 e7 e8 e6 6d 00 00 48 8b 5d e8 4c 8b 65 f0 4c ---[ end trace fe2aaf4e723042fd ]--- -----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 end here-----
-----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 start here----- [ 116.528885] ------------[ cut here ]------------ [ 116.528894] WARNING: CPU: 3 PID: 38 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 116.528898] Modules linked in: intel_cstate(-) ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 116.528919] CPU: 3 PID: 38 Comm: kworker/3:1H Not tainted 4.14.5-1.el7.elrepo.x86_64 #1 [ 116.529007] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 116.529034] ---[ end trace a7814e3ec9a330c6 ]--- [ 147.424117] ------------[ cut here ]------------ [ 147.424150] WARNING: CPU: 2 PID: 24 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 147.424160] Modules linked in: ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 147.424222] CPU: 2 PID: 24 Comm: kworker/2:0H Tainted: G W 4.14.5-1.el7.elrepo.x86_64 #1 [ 147.424238] Workqueue: kblockd blk_mq_run_work_fn [ 147.424247] task: ffff88007c539840 task.stack: ffffc900403e4000 [ 147.424259] RIP: e030:__blk_mq_run_hw_queue+0x89/0xa0 [ 147.424270] RSP: e02b:ffffc900403e7e30 EFLAGS: 00010202 [ 147.424279] RAX: 0000000000000001 RBX: ffff880003b83800 RCX: ffff88007d11bca0 [ 147.424288] RDX: ffff88007c656c88 RSI: 00000000000000a0 RDI: ffff880003b83800 [ 147.424298] RBP: ffffc900403e7e48 R08: 0000000000000000 R09: 0000000000000000 [ 147.424309] R10: 0000000000007ff0 R11: 00000000000074e5 R12: ffff88007c436900 [ 147.424319] R13: ffff88007d11bc80 R14: ffff88007d121b00 R15: ffff880003b83848 [ 147.424340] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:ffff88007d100000 [ 147.424350] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 147.424359] CR2: 00007f504f19a700 CR3: 0000000079bed000 CR4: 0000000000042660 [ 147.424370] Call Trace: [ 147.424384] blk_mq_run_work_fn+0x2c/0x30 [ 147.424400] process_one_work+0x149/0x360 [ 147.424411] worker_thread+0x4d/0x3e0 [ 147.424421] kthread+0x109/0x140 [ 147.424432] ? rescuer_thread+0x380/0x380 [ 147.424441] ? kthread_park+0x60/0x60 [ 147.424455] ret_from_fork+0x25/0x30 [ 147.424463] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 147.424554] ---[ end trace a7814e3ec9a330c7 ]--- -----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 end here-----
On 12/19/2017 09:12 AM, Johnny Hughes wrote:
There are a couple of xen updates in the 4.9.66 and 4.9.68 kernels:
https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.66 https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.68
Let me build a newer Dom0 kernel and see if that helps.
Thanks, Johnny Hughes
OK .. I have built and pushed to the testing tag the following dom0 kernels:
kernel-4.9.70-29.el7 kernel-4.9.70-29.el6
they will show up in a couple hours here:
https://buildlogs.centos.org/centos/6/virt/x86_64/xen/
https://buildlogs.centos.org/centos/7/virt/x86_64/xen/
On 12/11/2017 06:52 PM, Adi Pircalabu wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
-----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 start here----- ------------[ cut here ]------------ WARNING: CPU: 4 PID: 60 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x9e/0xc0 Modules linked in: intel_cstate(-) ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack libcrc32c ip6table_filter ip6_tables dm_mod dax xen_netfront crc32_pclmul crct10dif_pclmul ghash_clmulni_intel crc32c_intel pcbc aesni_intel glue_helper crypto_simd cryptd aes_x86_64 coretemp hwmon x86_pkg_temp_thermal sb_edac intel_rapl_perf pcspkr ext4 jbd2 mbcache xen_blkfront CPU: 4 PID: 60 Comm: kworker/4:1H Not tainted 4.14.5-1.el6.elrepo.x86_64 #1 Workqueue: kblockd blk_mq_run_work_fn task: ffff8802711a2780 task.stack: ffffc90041af4000 RIP: e030:__blk_mq_run_hw_queue+0x9e/0xc0 RSP: e02b:ffffc90041af7c48 EFLAGS: 00010202 RAX: 0000000000000001 RBX: ffff88027117fa80 RCX: 0000000000000001 RDX: ffff88026b053ee0 RSI: ffff88027351bca0 RDI: ffff88026b072800 RBP: ffffc90041af7c68 R08: ffffc90041af7eb8 R09: ffff8802711a2810 R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff88026b072800 R13: ffffe8ffffd04d00 R14: 0000000000000000 R15: ffffe8ffffd04d05 FS: 00002b7b7c89b700(0000) GS:ffff880273500000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 000000026d953000 CR4: 0000000000042660 Call Trace: blk_mq_run_work_fn+0x31/0x40 process_one_work+0x174/0x440 ? xen_mc_flush+0xad/0x1b0 ? schedule+0x3a/0xa0 worker_thread+0x6b/0x410 ? default_wake_function+0x12/0x20 ? __wake_up_common+0x84/0x130 ? maybe_create_worker+0x120/0x120 ? schedule+0x3a/0xa0 ? _raw_spin_unlock_irqrestore+0x16/0x20 ? maybe_create_worker+0x120/0x120 kthread+0x111/0x150 ? __kthread_init_worker+0x40/0x40 ret_from_fork+0x25/0x30 Code: 89 df e8 06 2f d9 ff 4c 89 e7 41 89 c5 e8 0b 6e 00 00 44 89 ee 48 89 df e8 20 2f d9 ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 <0f> ff eb aa 4c 89 e7 e8 e6 6d 00 00 48 8b 5d e8 4c 8b 65 f0 4c ---[ end trace fe2aaf4e723042fd ]--- -----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 end here-----
-----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 start here----- [ 116.528885] ------------[ cut here ]------------ [ 116.528894] WARNING: CPU: 3 PID: 38 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 116.528898] Modules linked in: intel_cstate(-) ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 116.528919] CPU: 3 PID: 38 Comm: kworker/3:1H Not tainted 4.14.5-1.el7.elrepo.x86_64 #1 [ 116.529007] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 116.529034] ---[ end trace a7814e3ec9a330c6 ]--- [ 147.424117] ------------[ cut here ]------------ [ 147.424150] WARNING: CPU: 2 PID: 24 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 147.424160] Modules linked in: ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 147.424222] CPU: 2 PID: 24 Comm: kworker/2:0H Tainted: G W 4.14.5-1.el7.elrepo.x86_64 #1 [ 147.424238] Workqueue: kblockd blk_mq_run_work_fn [ 147.424247] task: ffff88007c539840 task.stack: ffffc900403e4000 [ 147.424259] RIP: e030:__blk_mq_run_hw_queue+0x89/0xa0 [ 147.424270] RSP: e02b:ffffc900403e7e30 EFLAGS: 00010202 [ 147.424279] RAX: 0000000000000001 RBX: ffff880003b83800 RCX: ffff88007d11bca0 [ 147.424288] RDX: ffff88007c656c88 RSI: 00000000000000a0 RDI: ffff880003b83800 [ 147.424298] RBP: ffffc900403e7e48 R08: 0000000000000000 R09: 0000000000000000 [ 147.424309] R10: 0000000000007ff0 R11: 00000000000074e5 R12: ffff88007c436900 [ 147.424319] R13: ffff88007d11bc80 R14: ffff88007d121b00 R15: ffff880003b83848 [ 147.424340] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:ffff88007d100000 [ 147.424350] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 147.424359] CR2: 00007f504f19a700 CR3: 0000000079bed000 CR4: 0000000000042660 [ 147.424370] Call Trace: [ 147.424384] blk_mq_run_work_fn+0x2c/0x30 [ 147.424400] process_one_work+0x149/0x360 [ 147.424411] worker_thread+0x4d/0x3e0 [ 147.424421] kthread+0x109/0x140 [ 147.424432] ? rescuer_thread+0x380/0x380 [ 147.424441] ? kthread_park+0x60/0x60 [ 147.424455] ret_from_fork+0x25/0x30 [ 147.424463] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 147.424554] ---[ end trace a7814e3ec9a330c7 ]--- -----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 end here-----
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
On 20-12-2017 9:17, Johnny Hughes wrote: [...]
OK .. I have built and pushed to the testing tag the following dom0 kernels:
kernel-4.9.70-29.el7 kernel-4.9.70-29.el6
they will show up in a couple hours here:
Johnny, thanks for that. Unfortunately the bug is still there. Only tested it on C7 for both dom0 and domU.
dom0 xl info: release : 4.9.70-29.el7.x86_64 version : #1 SMP Tue Dec 19 15:25:38 UTC 2017 machine : x86_64 nr_cpus : 8 max_cpu_id : 7 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 3606 hw_caps : bfebfbff:2c100800:00000000:00007f00:77fafbff:00000000:00000121:029c6fbf virt_caps : hvm total_memory : 16275 free_memory : 13020 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 6 xen_extra : .6-8.el7 xen_version : 4.6.6-8.el7 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Tue Dec 12 11:42:15 2017 +0000 git:07e9f39-dirty xen_commandline : placeholder dom0_mem=1024M,max:1024M dom0_max_vcpus=1 dom0_vcpus_pin cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) cc_compile_by : mockbuild cc_compile_domain : centos.org cc_compile_date : Tue Dec 12 12:15:46 UTC 2017 xend_config_format : 4
domU kernel 4.14.7-1.el7.elrepo.x86_64
dom0 stacktrace:
[ 14.881838] ip_set: protocol 6 [ 281.806649] installing Xen timer for CPU 2 [ 281.807377] cpu 2 spinlock event irq 25 [ 281.812667] installing Xen timer for CPU 3 [ 281.813383] cpu 3 spinlock event irq 33 [ 287.710812] ------------[ cut here ]------------ [ 287.710842] WARNING: CPU: 2 PID: 35 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 287.710853] Modules linked in: ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 287.710913] CPU: 2 PID: 35 Comm: kworker/2:1H Not tainted 4.14.7-1.el7.elrepo.x86_64 #1 [ 287.710927] Workqueue: kblockd blk_mq_run_work_fn [ 287.710936] task: ffff88007c6a0000 task.stack: ffffc90040474000 [ 287.710948] RIP: e030:__blk_mq_run_hw_queue+0x89/0xa0 [ 287.710956] RSP: e02b:ffffc90040477e30 EFLAGS: 00010202 [ 287.710968] RAX: 0000000000000001 RBX: ffff880003aa5400 RCX: ffff88007d11bca0 [ 287.710977] RDX: ffff88007c656d98 RSI: 00000000000000a0 RDI: ffff880003aa5400 [ 287.710986] RBP: ffffc90040477e48 R08: 0000000000000000 R09: 0000000000000000 [ 287.710996] R10: 0000000000007ff0 R11: 000000000000018e R12: ffff88007c570000 [ 287.711005] R13: ffff88007d11bc80 R14: ffff88007d121b00 R15: ffff880003aa5448 [ 287.711025] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:ffff88007d100000 [ 287.711036] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 287.711045] CR2: 00007f8750908000 CR3: 00000000796a4000 CR4: 0000000000042660 [ 287.711057] Call Trace: [ 287.711071] blk_mq_run_work_fn+0x2c/0x30 [ 287.711086] process_one_work+0x149/0x360 [ 287.711098] worker_thread+0x4d/0x3e0 [ 287.711108] kthread+0x109/0x140 [ 287.711119] ? rescuer_thread+0x380/0x380 [ 287.711128] ? kthread_park+0x60/0x60 [ 287.711140] ret_from_fork+0x25/0x30 [ 287.711148] Code: 00 e8 4c e8 45 00 4c 89 e7 e8 34 4a d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 4e 4a d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 287.711235] ---[ end trace 7b31b11d076677d1 ]---
However I've just found domU is recoverable by reverting to the previous number of vcpus when it's blocked.
--- Adi Pircalabu
On 12/11/2017 06:52 PM, Adi Pircalabu wrote:
Has anyone seen this recently? I couldn't replicate it on:
- CentOS 6 running kernel-2.6.32-696.16.1.el6.x86_64,
kernel-lt-4.4.105-1.el6.elrepo.x86_64
- CentOS 7 running 4.9.67-1.el7.centos.x86_64
But I can replicate it consistently running "xl -v vcpu-set <domU> <val>" on:
- CentOS 6 running 4.14.5-1.el6.elrepo.x86_64
- CentOS 7 running 4.14.5-1.el7.elrepo.x86_64
dom0 versions tested with similar results in the domU:
- 4.6.6-6.el7 on kernel 4.9.63-29.el7.x86_64
- 4.6.3-15.el6 on kernel 4.9.37-29.el6.x86_64
Noticed behaviour:
- These commands stall:
top ls -l /var/tmp ls -l /tmp
- Stuck in D state on the CentOS 7 domU:
root 5 0.0 0.0 0 0 ? D 11:20 0:00 [kworker/u8:0] root 316 0.0 0.0 0 0 ? D 11:20 0:00 [jbd2/xvda1-8] root 1145 0.0 0.2 116636 4776 ? Ds 11:20 0:00 -bash root 1289 0.0 0.1 25852 2420 ? Ds 11:35 0:00 /usr/bin/systemd-tmpfiles --clean root 1290 0.0 0.1 125248 2696 pts/1 D+ 11:44 0:00 ls --color=auto -l /tmp/ root 1293 0.0 0.1 125248 2568 pts/2 D+ 11:44 0:00 ls --color=auto -l /var/tmp root 1296 0.0 0.2 116636 4908 pts/3 Ds+ 11:44 0:00 -bash root 1358 0.0 0.1 125248 2612 pts/4 D+ 11:47 0:00 ls --color=auto -l /var/tmp
At a first glance it appears the issue is in 4.14.5 kernel. Stack traces follow:
-----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 start here----- ------------[ cut here ]------------ WARNING: CPU: 4 PID: 60 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x9e/0xc0 Modules linked in: intel_cstate(-) ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack libcrc32c ip6table_filter ip6_tables dm_mod dax xen_netfront crc32_pclmul crct10dif_pclmul ghash_clmulni_intel crc32c_intel pcbc aesni_intel glue_helper crypto_simd cryptd aes_x86_64 coretemp hwmon x86_pkg_temp_thermal sb_edac intel_rapl_perf pcspkr ext4 jbd2 mbcache xen_blkfront CPU: 4 PID: 60 Comm: kworker/4:1H Not tainted 4.14.5-1.el6.elrepo.x86_64 #1 Workqueue: kblockd blk_mq_run_work_fn task: ffff8802711a2780 task.stack: ffffc90041af4000 RIP: e030:__blk_mq_run_hw_queue+0x9e/0xc0 RSP: e02b:ffffc90041af7c48 EFLAGS: 00010202 RAX: 0000000000000001 RBX: ffff88027117fa80 RCX: 0000000000000001 RDX: ffff88026b053ee0 RSI: ffff88027351bca0 RDI: ffff88026b072800 RBP: ffffc90041af7c68 R08: ffffc90041af7eb8 R09: ffff8802711a2810 R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff88026b072800 R13: ffffe8ffffd04d00 R14: 0000000000000000 R15: ffffe8ffffd04d05 FS: 00002b7b7c89b700(0000) GS:ffff880273500000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600400 CR3: 000000026d953000 CR4: 0000000000042660 Call Trace: blk_mq_run_work_fn+0x31/0x40 process_one_work+0x174/0x440 ? xen_mc_flush+0xad/0x1b0 ? schedule+0x3a/0xa0 worker_thread+0x6b/0x410 ? default_wake_function+0x12/0x20 ? __wake_up_common+0x84/0x130 ? maybe_create_worker+0x120/0x120 ? schedule+0x3a/0xa0 ? _raw_spin_unlock_irqrestore+0x16/0x20 ? maybe_create_worker+0x120/0x120 kthread+0x111/0x150 ? __kthread_init_worker+0x40/0x40 ret_from_fork+0x25/0x30 Code: 89 df e8 06 2f d9 ff 4c 89 e7 41 89 c5 e8 0b 6e 00 00 44 89 ee 48 89 df e8 20 2f d9 ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 <0f> ff eb aa 4c 89 e7 e8 e6 6d 00 00 48 8b 5d e8 4c 8b 65 f0 4c ---[ end trace fe2aaf4e723042fd ]--- -----CentOS 6 kernel-ml-4.14.5-1.el6.elrepo.x86_64 end here-----
-----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 start here----- [ 116.528885] ------------[ cut here ]------------ [ 116.528894] WARNING: CPU: 3 PID: 38 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 116.528898] Modules linked in: intel_cstate(-) ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 116.528919] CPU: 3 PID: 38 Comm: kworker/3:1H Not tainted 4.14.5-1.el7.elrepo.x86_64 #1 [ 116.529007] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 116.529034] ---[ end trace a7814e3ec9a330c6 ]--- [ 147.424117] ------------[ cut here ]------------ [ 147.424150] WARNING: CPU: 2 PID: 24 at block/blk-mq.c:1144 __blk_mq_run_hw_queue+0x89/0xa0 [ 147.424160] Modules linked in: ip_set_hash_ip ip_set nfnetlink x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 147.424222] CPU: 2 PID: 24 Comm: kworker/2:0H Tainted: G W 4.14.5-1.el7.elrepo.x86_64 #1 [ 147.424238] Workqueue: kblockd blk_mq_run_work_fn [ 147.424247] task: ffff88007c539840 task.stack: ffffc900403e4000 [ 147.424259] RIP: e030:__blk_mq_run_hw_queue+0x89/0xa0 [ 147.424270] RSP: e02b:ffffc900403e7e30 EFLAGS: 00010202 [ 147.424279] RAX: 0000000000000001 RBX: ffff880003b83800 RCX: ffff88007d11bca0 [ 147.424288] RDX: ffff88007c656c88 RSI: 00000000000000a0 RDI: ffff880003b83800 [ 147.424298] RBP: ffffc900403e7e48 R08: 0000000000000000 R09: 0000000000000000 [ 147.424309] R10: 0000000000007ff0 R11: 00000000000074e5 R12: ffff88007c436900 [ 147.424319] R13: ffff88007d11bc80 R14: ffff88007d121b00 R15: ffff880003b83848 [ 147.424340] FS: 0000000000000000(0000) GS:ffff88007d100000(0000) knlGS:ffff88007d100000 [ 147.424350] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 147.424359] CR2: 00007f504f19a700 CR3: 0000000079bed000 CR4: 0000000000042660 [ 147.424370] Call Trace: [ 147.424384] blk_mq_run_work_fn+0x2c/0x30 [ 147.424400] process_one_work+0x149/0x360 [ 147.424411] worker_thread+0x4d/0x3e0 [ 147.424421] kthread+0x109/0x140 [ 147.424432] ? rescuer_thread+0x380/0x380 [ 147.424441] ? kthread_park+0x60/0x60 [ 147.424455] ret_from_fork+0x25/0x30 [ 147.424463] Code: 00 e8 7c c5 45 00 4c 89 e7 e8 14 4b d7 ff 48 89 df 41 89 c5 e8 19 66 00 00 44 89 ee 4c 89 e7 e8 2e 4b d7 ff 5b 41 5c 41 5d 5d c3 <0f> ff eb b4 48 89 df e8 fb 65 00 00 5b 41 5c 41 5d 5d c3 0f ff [ 147.424554] ---[ end trace a7814e3ec9a330c7 ]--- -----CentOS 7 kernel-ml-4.14.5-1.el7.elrepo.x86_64 end here-----
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
-
Adi Pircalabu -
Akemi Yagi -
George Dunlap -
Johnny Hughes