<div dir="ltr">Enabled the logging on both compute and baremetal. Nothing strange in logs:<div><br></div><div>on baremetal :</div><div><div><div>Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808</div><div>Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > 24714903, stop ksm</div></div></div><div><br></div><div>on compute:</div><div><div>Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936</div><div>Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > 20592492, stop ksm</div></div><div><br></div><div>and the compute node is again at 100% CPU utilization.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">În mar., 16 aug. 2016 la 15:26, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com">bderzhavets@hotmail.com</a>> a scris:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>I would enable ksmtuned logging ,if it has been done verify logs<br>
</p>
<br>
<br>
<div style="color:rgb(49,55,57)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"></font></div></div></div></div><div dir="ltr"><div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="color:rgb(49,55,57)"><div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> <a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
</font></div></div></div></div><div dir="ltr"><div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="color:rgb(49,55,57)"><div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Sent:</b> Tuesday, August 16, 2016 7:16 AM</font></div></div></div></div><div dir="ltr"><div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="color:rgb(49,55,57)"><div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font></div></div></div></div><div dir="ltr"><div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif"><div style="color:rgb(49,55,57)">
<div>
<div dir="ltr">Yes. It is on both baremetal and compute node.</div>
<br>
<div class="gmail_quote">
<div dir="ltr">În mar., 16 aug. 2016 la 13:37, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com" target="_blank">bderzhavets@hotmail.com</a>> a scris:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Is KSM enabled on your Compute Nodes ( presuming CentOS 7.2 on bare metal ) ?<br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"></font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Sent:</b> Tuesday, August 16, 2016 5:25 AM</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">Running the compute node for several days simply triggers it.</div>
<br>
<div class="gmail_quote">
<div dir="ltr">În mar., 16 aug. 2016 la 12:12, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com" target="_blank">bderzhavets@hotmail.com</a>> a scris:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Sorry,</p>
<p>How you trigger the problem ?</p>
<p>B.<br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"></font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Sent:</b> Tuesday, August 16, 2016 3:28 AM</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><font face="monospace">Hello,</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">The issue reproduced again and it doesn't look like a swap problem. Some details:</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">on the baremetal, from top:<br>
</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">top - 08:08:52 up 5 days, 16:43, 3 users, load average: 36.19, 36.05, 36.05</font></div>
<div><font face="monospace">Tasks: 493 total, 1 running, 492 sleeping, 0 stopped, 0 zombie</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">%Cpu(s): 3.5 us, 87.9 sy, 0.0 ni, 8.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st</font></div>
<div><font face="monospace">KiB Mem : 12357451+total, 14296000 free, 65634428 used, 43644088 buff/cache</font></div>
<div><font face="monospace">KiB Swap: 4194300 total, 4073868 free, 120432 used. 56953888 avail Mem</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">19158 qemu 20 0 0.098t 0.041t 10476 S 3650 35.6 13048:24 qemu-kvm</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">The compute node has 36 CPUs and the usage is now 100%. There are more than 50 GB of memory still available on the baremetal. The swap is barely used, 120 MB.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">On compute node, from top:</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">top - 05:11:58 up 1 day, 15:08, 2 users, load average: 40.46, 40.49, 40.74</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">%Cpu(s): 99.1 us, 0.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.1 si, 0.1 st</font></div>
<div><font face="monospace">KiB Mem : 10296246+total, 78079936 free, 23671360 used, 1211160 buff/cache</font></div>
<div><font face="monospace">KiB Swap: 0 total, 0 free, 0 used. 78939968 avail Mem</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</font></div>
<div><font face="monospace"> 6032 qemu 20 0 10.601g 1.272g 12964 S 400.0 1.3 588:40.39 qemu-kvm</font></div>
<div><font face="monospace"> 5673 qemu 20 0 10.602g 1.006g 13020 S 399.7 1.0 1161:47 qemu-kvm</font></div>
<div><font face="monospace"> 5998 qemu 20 0 10.601g 1.192g 13028 S 367.9 1.2 1544:30 qemu-kvm</font></div>
<div><font face="monospace"> 5951 qemu 20 0 10.601g 1.246g 13020 S 348.3 1.3 1547:38 qemu-kvm</font></div>
<div><font face="monospace"> 5750 qemu 20 0 10.599g 990136 13060 S 339.1 1.0 1152:25 qemu-kvm</font></div>
<div><font face="monospace"> 5752 qemu 20 0 10.598g 1.426g 13040 S 313.9 1.5 663:13.65 qemu-kvm</font></div>
<div><font face="monospace"><span style="line-height:18px">....</span></font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">There are more than 70 GB of memory available on the compute node. All VMs are using 100% their CPUs and they are not accessible anymore.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Laurentiu </font></div>
<font face="monospace"><br>
</font>
<div class="gmail_quote">
<div dir="ltr"><font face="monospace">În dum., 14 aug. 2016 la 21:44, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com" target="_blank">bderzhavets@hotmail.com</a>> a scris:<br>
</font></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<p><font face="monospace"><br>
</font></p>
<font face="monospace"><br>
<br>
</font>
<div style="color:rgb(0,0,0)"><font face="monospace">
<hr style="display:inline-block;width:98%">
</font>
<div dir="ltr"><font face="monospace"><font style="font-size:11pt" color="#000000"><b>From:</b>
<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
<b>Sent:</b> Sunday, August 14, 2016 10:17 AM<br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font> </font>
<div><font face="monospace"> </font></div>
</div>
<div>
<div dir="ltr"></div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr"><font face="monospace">More details on the subject: </font>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">I suppose it is a nested KVM issue because it raised after I enabled the nested KVM feature. Without it, anyway, the second level VMs are unusable in terms of performance.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">I am using CentOS 7 with:</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">kernel: 3.10.0-327.22.2.el7.x86_64</font></div>
<div><font face="monospace">qemu-kvm:1.5.3-105.el7_2.4</font></div>
<div><font face="monospace">libvirt:1.2.17-13.el7_2.5</font></div>
<div><font face="monospace"><br>
</font></div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><font face="monospace">on both the baremetal and the compute VM.<br>
<br>
</font></div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><font face="monospace"><b>Please, post</b><br>
<br>
1) # virsh dumpxml VM-L1 ( where on L1 level you expect nested KVM to appear)<br>
2) Login into VM-L1 and run :-<br>
# lsmod | grep kvm<br>
3) I need outputs from VM-L1 ( in case it is Compute Node )<br>
<br>
# cat /etc/nova/nova.conf | grep virt_type<br>
# <span>cat /etc/nova/nova.conf | grep cpu_mode</span><br>
<br>
Boris.<br>
<br>
<br>
<br>
<br>
</font></div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">The only workaround now is to shutdown the compute VM and start it back from baremetal with virsh start.</font></div>
<div><font face="monospace">A simple restart of the compute node doesn't help. It looks like the qemu-kvm process corresponding to the compute VM is the problem.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Laurentiu</font></div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0)">
<div><font face="monospace"><br>
</font>
<div class="gmail_quote">
<div dir="ltr"><font face="monospace">În dum., 14 aug. 2016 la 00:19, Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>> a scris:<br>
</font></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><font face="monospace">Hello, </font>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">I have an OpenStack setup in virtual environment on CentOS 7.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">The baremetal has <b>nested KVM</b> enabled and 1 compute node as a VM.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Inside the compute node I have multiple VMs running.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">After about every 3 days the VMs get inaccessible and the compute node reports high CPU usage. The qemu-kvm process for each VM inside the compute node reports full CPU usage.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Please help me with some hints to debug this issue.</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Thanks,</font></div>
<div><font face="monospace">Laurentiu</font></div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0)">
<div></div>
</div>
</div>
</div>
<font face="monospace">_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a></font><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a><br>
</blockquote>
</div>
</div>
</div></div></div>
_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a><br>
</blockquote></div>