<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Since moving from 4.4 to 4.6, I’ve been seeing an increasing number of stability issues on our hypervisors. I’m not clear if there’s a singular root cause here, or if I’m dealing with multiple bugs…<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>One of the more common ones I’ve seen, is a VM on shutdown will remain in the null state and a kernel bug is thrown:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>xen001 log # xl list<o:p></o:p></p><p class=MsoNormal>Name ID Mem VCPUs State Time(s)<o:p></o:p></p><p class=MsoNormal>Domain-0 0 6144 24 r----- 6639.7<o:p></o:p></p><p class=MsoNormal>(null) 3 0 1 --pscd 36.3<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>[89920.839074] BUG: unable to handle kernel paging request at ffff88020ee9a000<o:p></o:p></p><p class=MsoNormal>[89920.839546] IP: [<ffffffff81430922>] __memcpy+0x12/0x20<o:p></o:p></p><p class=MsoNormal>[89920.839933] PGD 2008067 <o:p></o:p></p><p class=MsoNormal>[89920.840022] PUD 17f43f067 <o:p></o:p></p><p class=MsoNormal>[89920.840390] PMD 1e0976067 <o:p></o:p></p><p class=MsoNormal>[89920.840469] PTE 0<o:p></o:p></p><p class=MsoNormal>[89920.840833] <o:p></o:p></p><p class=MsoNormal>[89920.841123] Oops: 0000 [#1] SMP<o:p></o:p></p><p class=MsoNormal>[89920.841417] Modules linked in: ebt_ip ebtable_filter ebtables arptable_filter arp_tables bridge xen_pciback xen_gntalloc nfsd auth_rpcgss nfsv3 nfs_acl nfs fscache lockd sunrpc grace 8021q mrp garp stp llc bonding xen_acpi_processor blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd dcdbas fjes pcspkr ipmi_devintf ipmi_si ipmi_msghandler joydev i2c_i801 i2c_smbus lpc_ich shpchp mei_me mei ioatdma ixgbe mdio igb dca ptp pps_core uas usb_storage wmi ttm<o:p></o:p></p><p class=MsoNormal>[89920.847080] CPU: 4 PID: 1471 Comm: loop6 Not tainted 4.9.58-29.el6.x86_64 #1<o:p></o:p></p><p class=MsoNormal>[89920.847381] Hardware name: Dell Inc. PowerEdge C6220/03C9JJ, BIOS 2.7.1 03/04/2015<o:p></o:p></p><p class=MsoNormal>[89920.847893] task: ffff8801b75e0700 task.stack: ffffc900460e0000<o:p></o:p></p><p class=MsoNormal>[89920.848192] RIP: e030:[<ffffffff81430922>] [<ffffffff81430922>] __memcpy+0x12/0x20<o:p></o:p></p><p class=MsoNormal>[89920.848783] RSP: e02b:ffffc900460e3b20 EFLAGS: 00010246<o:p></o:p></p><p class=MsoNormal>[89920.849081] RAX: ffff88018916d000 RBX: ffff8801b75e0700 RCX: 0000000000000200<o:p></o:p></p><p class=MsoNormal>[89920.849384] RDX: 0000000000000000 RSI: ffff88020ee9a000 RDI: ffff88018916d000<o:p></o:p></p><p class=MsoNormal>[89920.849686] RBP: ffffc900460e3b38 R08: ffff88011da9fcf8 R09: 0000000000000002<o:p></o:p></p><p class=MsoNormal>[89920.849989] R10: ffff88019535bddc R11: ffffea0006245b5c R12: 0000000000001000<o:p></o:p></p><p class=MsoNormal>[89920.850294] R13: ffff88018916e000 R14: 0000000000001000 R15: ffffc900460e3b68<o:p></o:p></p><p class=MsoNormal>[89920.850605] FS: 00007fb865c30700(0000) GS:ffff880204b00000(0000) knlGS:0000000000000000<o:p></o:p></p><p class=MsoNormal>[89920.851118] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033<o:p></o:p></p><p class=MsoNormal>[89920.851418] CR2: ffff88020ee9a000 CR3: 00000001ef03b000 CR4: 0000000000042660<o:p></o:p></p><p class=MsoNormal>[89920.851720] Stack:<o:p></o:p></p><p class=MsoNormal>[89920.852009] ffffffff814375ca ffffc900460e3b38 ffffc900460e3d08 ffffc900460e3bb8<o:p></o:p></p><p class=MsoNormal>[89920.852821] ffffffff814381c5 ffffc900460e3b68 ffffc900460e3d08 0000000000001000<o:p></o:p></p><p class=MsoNormal>[89920.853633] ffffc900460e3d88 0000000000000000 0000000000001000 ffffea0000000000<o:p></o:p></p><p class=MsoNormal>[89920.854445] Call Trace:<o:p></o:p></p><p class=MsoNormal>[89920.854741] [<ffffffff814375ca>] ? memcpy_from_page+0x3a/0x70<o:p></o:p></p><p class=MsoNormal>[89920.855043] [<ffffffff814381c5>] iov_iter_copy_from_user_atomic+0x265/0x290<o:p></o:p></p><p class=MsoNormal>[89920.855354] [<ffffffff811cf633>] generic_perform_write+0xf3/0x1d0<o:p></o:p></p><p class=MsoNormal>[89920.855673] [<ffffffff8101e39a>] ? xen_load_tls+0xaa/0x160<o:p></o:p></p><p class=MsoNormal>[89920.855992] [<ffffffffc025cf2b>] nfs_file_write+0xdb/0x200 [nfs]<o:p></o:p></p><p class=MsoNormal>[89920.856297] [<ffffffff81269062>] vfs_iter_write+0xa2/0xf0<o:p></o:p></p><p class=MsoNormal>[89920.856599] [<ffffffff815fa365>] lo_write_bvec+0x65/0x100<o:p></o:p></p><p class=MsoNormal>[89920.856899] [<ffffffff815fc375>] do_req_filebacked+0x195/0x300<o:p></o:p></p><p class=MsoNormal>[89920.857202] [<ffffffff815fc53b>] loop_queue_work+0x5b/0x80<o:p></o:p></p><p class=MsoNormal>[89920.857505] [<ffffffff810c6898>] kthread_worker_fn+0x98/0x1b0<o:p></o:p></p><p class=MsoNormal>[89920.857808] [<ffffffff818d9dca>] ? schedule+0x3a/0xa0<o:p></o:p></p><p class=MsoNormal>[89920.858108] [<ffffffff818ddbb6>] ? _raw_spin_unlock_irqrestore+0x16/0x20<o:p></o:p></p><p class=MsoNormal>[89920.858411] [<ffffffff810c6800>] ? kthread_probe_data+0x40/0x40<o:p></o:p></p><p class=MsoNormal>[89920.858713] [<ffffffff810c63f5>] kthread+0xe5/0x100<o:p></o:p></p><p class=MsoNormal>[89920.859014] [<ffffffff810c6310>] ? __kthread_init_worker+0x40/0x40<o:p></o:p></p><p class=MsoNormal>[89920.859317] [<ffffffff818de2d5>] ret_from_fork+0x25/0x30<o:p></o:p></p><p class=MsoNormal>[89920.859615] Code: 81 f3 00 00 00 00 e9 1e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 66 90 66 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 <o:p></o:p></p><p class=MsoNormal>[89920.864410] RIP [<ffffffff81430922>] __memcpy+0x12/0x20<o:p></o:p></p><p class=MsoNormal>[89920.864749] RSP <ffffc900460e3b20><o:p></o:p></p><p class=MsoNormal>[89920.865021] CR2: ffff88020ee9a000<o:p></o:p></p><p class=MsoNormal>[89920.865294] ---[ end trace b77d2ce5646284d1 ]---<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Wondering if anyone has advice on how to troubleshoot the above, or might have some insight into that the issue could be? This hypervisor was only up for a day, had almost no VMs running on it since boot, I booted a single windows test VM which BSOD’ed and then this happened.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This is on xen 4.6.6-4.el6 with 4.9.58-29.el6.x86_64. I see these issues across a wide number of systems with from both Dell and Supermicro, although we run the same Intel x540 10gb nic’s in each system with the same netapp nfs backend storage.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Cheers,<o:p></o:p></p><p class=MsoNormal>Nathan<o:p></o:p></p></div></body></html>