<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>I would try to put dmesg on Cron running once per hour and attempt to analyze</p>
<p>logs captured. <br>
</p>
<br>
<br>
<div style="color: rgb(49, 55, 57);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> centos-virt-bounces@centos.org <centos-virt-bounces@centos.org> on behalf of Laurentiu Soica <laurentiu@soica.ro><br>
<b>Sent:</b> Monday, August 15, 2016 2:15 AM<br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font>
<div> </div>
</div>
<div>
<div dir="ltr">The CPUs are IvyBridge microarchitecture, Xeon E5-2670 v2.
<div><br>
</div>
<div>I could try lowering the vCPUs but I doubt it would help. Please note that the same VMs are running just fine (with a load of 2 out of an acceptable 36 on the compute node) for about 3 days after a restart.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Īn lun., 15 aug. 2016 la 08:42, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com">bderzhavets@hotmail.com</a>> a scris:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>I would attempt to decrease number of VCPUS allocated to cloud VMs.</p>
<p>Say try 4 => 2 . My guess there is not enough VCPUs to run OS itself. <br>
</p>
<p>I also guess CPU model << Haswell. Please , confirm ( or not) if possible.<br>
</p>
<p>Since Haswell was launched via my experience Intel Xeons based on this kernel (or latter kernels ) behaves much better then SandyBridge or IvyBridge based.<br>
</p>
<br>
Boris.<br>
<br>
<div style="color:rgb(49,55,57)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"></font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(49,55,57)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(49,55,57)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Sent:</b> Monday, August 15, 2016 1:15 AM</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(49,55,57)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(49,55,57)">
<div>
<div dir="ltr">Hello Borins,
<div><br>
</div>
<div>1. So, in about three days after a reboot (this happened several times already) the compute node reports high CPU usage. It has 36 vCPUs and it reports a load higher than 40. Usually the load is about 2 or 3.</div>
<div>The VMs qemu-kvm processes reports 100% CPU usage (for a VM with 4 CPU it reports almost 400%, for one with 1 CPU it reports almost 100%). The VMs are not accessible anymore through SSH.</div>
<div><br>
</div>
<div>2. The baremetal has 2 CPUs, each with 10 cores and HT activated so it reports 40 CPUs.</div>
<div>It has 128 GB RAM out of which 100 GB are for the compute node.</div>
<div><br>
</div>
<div>I have 15 VMs running inside compute. They are summing up 40 vCPUs and 92 GB RAM.</div>
<div><br>
</div>
<div>There are no swap devices installed on the compute node so the reported SwapTotal is 0 KB.</div>
<div><br>
</div>
<div>I'll check is the memory on the compute gets exhausted as soon as the problem reproduces again (in about 2 days) but for now there are more than 80 GB available.</div>
<div><br>
</div>
<div>Note that a reboot of the compute node doesn't fix the problem. Only a shutdown of the compute and a virsh start on it works.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Laurentiu</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Īn dum., 14 aug. 2016 la 23:27, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com" target="_blank">bderzhavets@hotmail.com</a>> a scris:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Reports posted look good for me. Config should provide the best available performance</p>
<p>for cloud VM (L2) on Compute Node. <br>
</p>
<p><br>
</p>
<p> 1. Please, remind me what goes wrong from your standpoint ?</p>
<p> 2. Which CPU is installed on Compute Node && how much RAM ?<br>
</p>
<p> Actually , my concern is :-</p>
<p> Number_of_ Cloud_VMs versus Number_CPU_Cores ( not threads)</p>
<p> Please, check `top` report in regards of swap area size.<br>
</p>
<p><br>
</p>
<p>Thanks.</p>
<p>Boris.<br>
</p>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"></font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Sent:</b> Sunday, August 14, 2016 3:06 PM</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font></div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><font face="monospace">Hello,</font></div>
<div><font face="monospace"><br>
</font></div>
<font face="monospace">1. <domain type='kvm' id='6'></font>
<div><font face="monospace"> <name>baremetalbrbm_1</name></font></div>
<div><font face="monospace"> <uuid>534e9b54-5e4c-4acb-adcf-793f841551a7</uuid></font></div>
<div><font face="monospace"> <memory unit='KiB'>104857600</memory></font></div>
<div><font face="monospace"> <currentMemory unit='KiB'>104857600</currentMemory></font></div>
<div><font face="monospace"> <vcpu placement='static'>36</vcpu></font></div>
<div><font face="monospace"> <resource></font></div>
<div><font face="monospace"> <partition>/machine</partition></font></div>
<div><font face="monospace"> </resource></font></div>
<div><font face="monospace"> <os></font></div>
<div><font face="monospace"> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type></font></div>
<div><font face="monospace"> <boot dev='hd'/></font></div>
<div><font face="monospace"> <bootmenu enable='no'/></font></div>
<div><font face="monospace"> </os></font></div>
<div><font face="monospace"> <features></font></div>
<div><font face="monospace"> <acpi/></font></div>
<div><font face="monospace"> <apic/></font></div>
<div><font face="monospace"> <pae/></font></div>
<div><font face="monospace"> </features></font></div>
<div><font face="monospace"> <cpu mode='host-passthrough'/></font></div>
<div><font face="monospace"> <clock offset='utc'/></font></div>
<div><font face="monospace"> <on_poweroff>destroy</on_poweroff></font></div>
<div><font face="monospace"> <on_reboot>restart</on_reboot></font></div>
<div><font face="monospace"> <on_crash>restart</on_crash></font></div>
<div><font face="monospace"> <devices></font></div>
<div><font face="monospace"> <emulator>/usr/libexec/qemu-kvm</emulator></font></div>
<div><font face="monospace"> <disk type='file' device='disk'></font></div>
<div><font face="monospace"> <driver name='qemu' type='qcow2' cache='unsafe'/></font></div>
<div><font face="monospace"> <source file='/var/lib/libvirt/images/baremetalbrbm_1.qcow2'/></font></div>
<div><font face="monospace"> <backingStore/></font></div>
<div><font face="monospace"> <target dev='sda' bus='sata'/></font></div>
<div><font face="monospace"> <alias name='sata0-0-0'/></font></div>
<div><font face="monospace"> <address type='drive' controller='0' bus='0' target='0' unit='0'/></font></div>
<div><font face="monospace"> </disk></font></div>
<div><font face="monospace"> <controller type='scsi' index='0' model='virtio-scsi'></font></div>
<div><font face="monospace"> <alias name='scsi0'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/></font></div>
<div><font face="monospace"> </controller></font></div>
<div><font face="monospace"> <controller type='usb' index='0'></font></div>
<div><font face="monospace"> <alias name='usb'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/></font></div>
<div><font face="monospace"> </controller></font></div>
<div><font face="monospace"> <controller type='pci' index='0' model='pci-root'></font></div>
<div><font face="monospace"> <alias name='pci.0'/></font></div>
<div><font face="monospace"> </controller></font></div>
<div><font face="monospace"> <controller type='sata' index='0'></font></div>
<div><font face="monospace"> <alias name='sata0'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/></font></div>
<div><font face="monospace"> </controller></font></div>
<div><font face="monospace"> <interface type='bridge'></font></div>
<div><font face="monospace"> <mac address='00:f1:15:20:c5:46'/></font></div>
<div><font face="monospace"> <source network='brbm' bridge='brbm'/></font></div>
<div><font face="monospace"> <virtualport type='openvswitch'></font></div>
<div><font face="monospace"> <parameters interfaceid='654ad04f-fa0a-41dd-9d30-b84e702462fe'/></font></div>
<div><font face="monospace"> </virtualport></font></div>
<div><font face="monospace"> <target dev='vnet5'/></font></div>
<div><font face="monospace"> <model type='virtio'/></font></div>
<div><font face="monospace"> <alias name='net0'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></font></div>
<div><font face="monospace"> </interface></font></div>
<div><font face="monospace"> <interface type='bridge'></font></div>
<div><font face="monospace"> <mac address='52:54:00:d3:c9:24'/></font></div>
<div><font face="monospace"> <source bridge='br57'/></font></div>
<div><font face="monospace"> <target dev='vnet6'/></font></div>
<div><font face="monospace"> <model type='rtl8139'/></font></div>
<div><font face="monospace"> <alias name='net1'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/></font></div>
<div><font face="monospace"> </interface></font></div>
<div><font face="monospace"> <serial type='pty'></font></div>
<div><font face="monospace"> <source path='/dev/pts/3'/></font></div>
<div><font face="monospace"> <target port='0'/></font></div>
<div><font face="monospace"> <alias name='serial0'/></font></div>
<div><font face="monospace"> </serial></font></div>
<div><font face="monospace"> <console type='pty' tty='/dev/pts/3'></font></div>
<div><font face="monospace"> <source path='/dev/pts/3'/></font></div>
<div><font face="monospace"> <target type='serial' port='0'/></font></div>
<div><font face="monospace"> <alias name='serial0'/></font></div>
<div><font face="monospace"> </console></font></div>
<div><font face="monospace"> <input type='mouse' bus='ps2'/></font></div>
<div><font face="monospace"> <input type='keyboard' bus='ps2'/></font></div>
<div><font face="monospace"> <graphics type='vnc' port='5903' autoport='yes' listen='127.0.0.1'></font></div>
<div><font face="monospace"> <listen type='address' address='127.0.0.1'/></font></div>
<div><font face="monospace"> </graphics></font></div>
<div><font face="monospace"> <video></font></div>
<div><font face="monospace"> <model type='cirrus' vram='16384' heads='1'/></font></div>
<div><font face="monospace"> <alias name='video0'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/></font></div>
<div><font face="monospace"> </video></font></div>
<div><font face="monospace"> <memballoon model='virtio'></font></div>
<div><font face="monospace"> <alias name='balloon0'/></font></div>
<div><font face="monospace"> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/></font></div>
<div><font face="monospace"> </memballoon></font></div>
<div><font face="monospace"> </devices></font></div>
<div><font face="monospace"></domain></font></div>
<div><font face="monospace"><br>
</font></div>
<font face="monospace">2. </font>
<div>
<div><font face="monospace">[root@overcloud-novacompute-0 ~]# lsmod | grep kvm</font></div>
<div><font face="monospace">kvm_intel 162153 70</font></div>
<div><font face="monospace">kvm 525409 1 kvm_intel</font></div>
<div><font face="monospace"><br>
</font></div>
<div>
<div><font face="monospace">[root@overcloud-novacompute-0 ~]# cat /etc/nova/nova.conf | grep virt_type|grep -v '^#'</font></div>
<div><font face="monospace">virt_type=kvm</font></div>
</div>
<div><font face="monospace"><br>
</font></div>
<div>
<div><font face="monospace">[root@overcloud-novacompute-0 ~]# cat /etc/nova/nova.conf | grep cpu_mode|grep -v '^#'</font></div>
<div><font face="monospace">cpu_mode=host-passthrough</font></div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Laurentiu</div>
<div><br>
</div>
<div class="gmail_quote">
<div dir="ltr">Īn dum., 14 aug. 2016 la 21:44, Boris Derzhavets <<a href="mailto:bderzhavets@hotmail.com" target="_blank">bderzhavets@hotmail.com</a>> a scris:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<p><br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b>
<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a> <<a href="mailto:centos-virt-bounces@centos.org" target="_blank">centos-virt-bounces@centos.org</a>> on behalf of Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>><br>
<b>Sent:</b> Sunday, August 14, 2016 10:17 AM<br>
<b>To:</b> Discussion about the virtualization on CentOS<br>
<b>Subject:</b> Re: [CentOS-virt] Nested KVM issue</font>
<div> </div>
</div>
<div>
<div dir="ltr"></div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">More details on the subject:
<div><br>
</div>
<div>I suppose it is a nested KVM issue because it raised after I enabled the nested KVM feature. Without it, anyway, the second level VMs are unusable in terms of performance.</div>
<div><br>
</div>
<div>I am using CentOS 7 with:</div>
<div><br>
</div>
<div>kernel: 3.10.0-327.22.2.el7.x86_64</div>
<div>qemu-kvm:1.5.3-105.el7_2.4</div>
<div>libvirt:1.2.17-13.el7_2.5</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div>on both the baremetal and the compute VM.<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><b>Please, post</b><br>
<br>
1) # virsh dumpxml VM-L1 ( where on L1 level you expect nested KVM to appear)<br>
2) Login into VM-L1 and run :-<br>
# lsmod | grep kvm<br>
3) I need outputs from VM-L1 ( in case it is Compute Node )<br>
<br>
# cat /etc/nova/nova.conf | grep virt_type<br>
# <span>cat /etc/nova/nova.conf | grep cpu_mode</span><br>
<br>
Boris.<br>
<br>
<br>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div>
<div dir="ltr">
<div><br>
</div>
<div>The only workaround now is to shutdown the compute VM and start it back from baremetal with virsh start.</div>
<div>A simple restart of the compute node doesn't help. It looks like the qemu-kvm process corresponding to the compute VM is the problem.</div>
<div><br>
</div>
<div>Laurentiu</div>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div><br>
<div class="gmail_quote">
<div dir="ltr">Īn dum., 14 aug. 2016 la 00:19, Laurentiu Soica <<a href="mailto:laurentiu@soica.ro" target="_blank">laurentiu@soica.ro</a>> a scris:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">Hello,
<div><br>
</div>
<div>I have an OpenStack setup in virtual environment on CentOS 7.</div>
<div><br>
</div>
<div>The baremetal has <b>nested KVM</b> enabled and 1 compute node as a VM.</div>
<div><br>
</div>
<div>Inside the compute node I have multiple VMs running.</div>
<div><br>
</div>
<div>After about every 3 days the VMs get inaccessible and the compute node reports high CPU usage. The qemu-kvm process for each VM inside the compute node reports full CPU usage.</div>
<div><br>
</div>
<div>Please help me with some hints to debug this issue.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Laurentiu</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<div dir="ltr">
<div style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0)">
<div></div>
</div>
</div>
</div>
_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
CentOS-virt mailing list<br>
<a href="mailto:CentOS-virt@centos.org" target="_blank">CentOS-virt@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-virt" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-virt</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>