Live Migration and LibVirt CPU Mode - virt

6 Jul 2017


      Hi All,
First time mailing here, I hope someone can help. We’re running a OpenStack Newton environment on top of CentOS-7.3, LibVirt and Qemu-KVM-EV. We are encountered an issue live migrating a VM between 2 hosts with different CPUs and LibVirt throws the following error:
libvirtError: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, f16c, rdrand, fsgsbase, tsc_adjust, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, rdseed, adx, smap, xsaveopt, abm, 3dnowprefetch; try using 'Broadwell-noTSX' CPU model.
LibVirt appears to be comparing the source and destination host CPUs, instead of guest VM and destination host CPUs. The VM is configured with a common set CPU features that are compatible with both (see below) and can run on both hosts without issue. The hosts have Westmere and Broadwell CPUs (See host CPU capabilities below). The VM can be booted on the Westmere host, successfully live migrated to the Broadwell host, but then throws the above error when we attempt to live migrating back to the Westmere host.
Due to how our Westmere hosts were configured in OpenStack Nova, the VMs CPU is configured with "cpu mode='host-model'", which fail to migrate. BUT if the VMs CPU is configured with "cpu mode='custom' match='exact'" it can be successfully migrate both ways. It appears that when a VMs CPU is configured with "cpu mode='host-model'" the guests CPU is ignored and the host CPUs are compared instead. Which I believe is incorrect behaviour, is this a known issue and/or have any workarounds.
The VMs will be configure to use "cpu mode='custom'" with a Westmere model on their next reboot, but rebooting all VMs prior to migration is something we really want to avoid.
There has already been some discussion on this issue on the OpenStack mailing list: http://lists.openstack.org/pipermail/openstack/2017-July/thread.html#45139
The following package versions are being used;
libvirt-daemon-kvm.x86_642.0.0-10.el7_3.4
qemu-kvm-ev.x86_6410:2.6.0-28.el7_3.3.1
kernel.x86_643.10.0-514.16.1.el7
The Qemu-KVM processes CPU parameters of the running VMs;
-cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp
Guest CPU Configuration that fails to migrate;
<cpu mode='host-model'>
    <model fallback='allow'>Westmere</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='1' threads='1'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='pclmuldq'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='rdtscp'/>
  </cpu>
Guest CPU Configuration that successfully migrates;
<cpu mode='custom' match='exact'>
    <model fallback='allow'>Westmere</model>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
Westmere Hosts CPU capabilities;
<cpu>
      <arch>x86_64</arch>
      <model>Westmere</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='pclmuldq'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='pcid'/>
      <feature name='dca'/>
      <feature name='arat'/>
      <feature name='pdpe1gb'/>
      <feature name='rdtscp'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size=
Broadwell Host CPU Capabilities;
<cpu>
      <arch>x86_64</arch>
      <model>Broadwell</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='12' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='dca'/>
      <feature name='osxsave'/>
      <feature name='f16c'/>
      <feature name='rdrand'/>
      <feature name='arat'/>
      <feature name='tsc_adjust'/>
      <feature name='cmt'/>
      <feature name='xsaveopt'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='pdpe1gb'/>
      <feature name='abm'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>
Regards,
Oisin
Oisin O'Malley
Systems Engineer
Iocane Pty Ltd
763 South Road
Black Forest SA 5035
Office:+61 (8) 8413 1010
Fax:+61 (8) 8231 2050
Email:oisin.omalley@iocane.com.au
Web:www.iocane.com.au
Better for business