[CentOS-devel] 6.4 CR: oVirt 3.1 breaks with missing cpu features after update to CentOS 6.4 (6.3 + CR)

Mon Mar 4 17:03:12 UTC 2013
Patrick Hurrelmann <patrick.hurrelmann at lobster.de>

Hi list,

I tested the upcoming CentOS 6.4 release with my lab installation of
oVirt 3.1 and it fails to play well.

Background: freshly installed CentOS 6.3 host in a Nehalem CPU-type
Cluster with 2 other hosts. Storage is iSCSI. Datacenter and Cluster are
both version 3.1. oVirt 3.1 was installed via Dreyou's repo.

In CentOS 6.3 all is fine and the following rpms are installed:

libvirt.x86_64                0.9.10-21.el6_3.8
libvirt-client.x86_64         0.9.10-21.el6_3.8
libvirt-lock-sanlock.x86_64   0.9.10-21.el6_3.8
libvirt-python.x86_64         0.9.10-21.el6_3.8
vdsm.x86_64                   4.10.0-0.46.15.el6
vdsm-cli.noarch               4.10.0-0.46.15.el6
vdsm-python.x86_64            4.10.0-0.46.15.el6
vdsm-xmlrpc.noarch            4.10.0-0.46.15.el6
qemu-kvm.x86_64               2:0.12.1.2-2.295.el6_3.10


uname -a
Linux vh-test1.mydomain.com 2.6.32-279.22.1.el6.x86_64 #1 SMP Wed Feb 6
03:10:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

virsh cpu capabilities on 6.3:
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>

and corresponding cpu features from vdsClient:

   cpuCores = 4
   cpuFlags = fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,
              cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,
              tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,
              pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,
              dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,
              pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dts,tpr_shadow,vnmi,
              flexpriority,ept,vpid,model_Conroe,model_Penryn,
              model_Nehalem
   cpuModel = Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
   cpuSockets = 1
   cpuSpeed = 2394.132


So the system was updated to 6.4 using the continuous release repo.

Installed rpms after update to 6.4 (6.3 + CR):

libvirt.x86_64                0.10.2-18.el6
libvirt-client.x86_64         0.10.2-18.el6
libvirt-lock-sanlock.x86_64   0.10.2-18.el6
libvirt-python.x86_64         0.10.2-18.el6
vdsm.x86_64                   4.10.0-0.46.15.el6
vdsm-cli.noarch               4.10.0-0.46.15.el6
vdsm-python.x86_64            4.10.0-0.46.15.el6
vdsm-xmlrpc.noarch            4.10.0-0.46.15.el6
qemu-kvm.x86_64               2:0.12.1.2-2.355.el6_4_4.1


uname -a
Linux vh-test1.mydomain.com 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 27
06:06:45 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

virsh capabilities on 6.4:
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>

and corresponding cpu features from vdsClient:

   cpuCores = 4
   cpuFlags = fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,
              cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,
              tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,
              pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,
              dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,
              pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dts,tpr_shadow,vnmi,
              flexpriority,ept,vpid,model_coreduo,model_Conroe
   cpuModel = Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
   cpuSockets = 1
   cpuSpeed = 2394.098

Full outputs of virsh capabilities and vdsCaps are attached. The only
difference I can see is that 6.4 exposes one additional cpu flags (sep)
and this seems to break the cpu recognition of vdsm.

Anyone has some hints on how to resolve or debug this further? What more
information can I provide to help?

Best regards
Patrick
-- 
Lobster LOGsuite GmbH, Münchner Straße 15a, D-82319 Starnberg

HRB 178831, Amtsgericht München
Geschäftsführer: Dr. Martin Fischer, Rolf Henrich
-------------- next part --------------
vdsClient -s 0 getVdsCaps
	HBAInventory = {'iSCSI': [{'InitiatorName': 'iqn.2012-09.com.mydomain:vh-test1'}], 'FC': []}
	ISCSIInitiatorName = iqn.2012-09.com.mydomain:vh-test1
	bondings = {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'USERCTL': 'no', 'MTU': '1500', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=802.3ad xmit_hash_policy=layer3+4', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'slaves': ['eth1', 'eth0'], 'hwaddr': '68:05:ca:01:bc:0d'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}
	clusterLevels = ['3.0', '3.1']
	cpuCores = 4
	cpuFlags = fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Conroe,model_Penryn,model_Nehalem
	cpuModel = Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
	cpuSockets = 1
	cpuSpeed = 2394.132
	emulatedMachines = ['rhel6.3.0', 'pc', 'rhel6.2.0', 'rhel6.1.0', 'rhel6.0.0', 'rhel5.5.0', 'rhel5.4.4', 'rhel5.4.0']
	guestOverhead = 65
	hooks = {}
	kvmEnabled = true
	lastClient = 80.242.147.190
	lastClientIface = ovirtmgmt
	management_ip = 
	memSize = 15939
	netConfigDirty = False
	networks = {'ovirtmgmt': {'addr': '192.168.245.18', 'cfg': {'USERCTL': 'no', 'IPADDR': '192.168.245.18', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '4', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.245.11'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '192.168.245.11', 'ports': ['bond0']}}
	nics = {'eth1': {'permhwaddr': 'B8:AC:6F:99:50:F6', 'addr': '', 'mtu': '1500', 'netmask': '', 'hwaddr': '68:05:ca:01:bc:0d', 'speed': 1000}, 'eth0': {'permhwaddr': '68:05:CA:01:BC:0D', 'addr': '', 'mtu': '1500', 'netmask': '', 'hwaddr': '68:05:ca:01:bc:0d', 'speed': 1000}}
	operatingSystem = {'release': '3.el6.centos.9', 'version': '6', 'name': 'RHEL'}
	packages2 = {'kernel': {'release': '279.22.1.el6.x86_64', 'buildtime': 1360116646.0, 'version': '2.6.32'}, 'spice-server': {'release': '10.el6', 'buildtime': '1340375889', 'version': '0.10.1'}, 'vdsm': {'release': '0.46.15.el6', 'buildtime': '1358926393', 'version': '4.10.0'}, 'qemu-kvm': {'release': '2.295.el6_3.10', 'buildtime': '1355820509', 'version': '0.12.1.2'}, 'libvirt': {'release': '21.el6_3.8', 'buildtime': '1359401311', 'version': '0.9.10'}, 'qemu-img': {'release': '2.295.el6_3.10', 'buildtime': '1355820509', 'version': '0.12.1.2'}}
	reservedMem = 321
	software_revision = 0.46
	software_version = 4.10
	supportedProtocols = ['2.2', '2.3']
	supportedRHEVMs = ['3.0', '3.1']
	uuid = 4C4C4544-0057-4210-8033-B1C04F5A344A_68:05:ca:01:bc:0d
	version_name = Snow Man
	vlans = {'bond0.100': {'netmask': '', 'iface': 'bond0', 'addr': '', 'mtu': '1500'}}
	vmTypes = ['kvm']
-------------- next part --------------
vdsClient -s 0 getVdsCaps
	HBAInventory = {'iSCSI': [{'InitiatorName': 'iqn.2012-09.com.mydomain:vh-test1'}], 'FC': []}
	ISCSIInitiatorName = iqn.2012-09.com.mydomain:vh-test1
	bondings = {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'USERCTL': 'no', 'MTU': '1500', 'BOOTPROTO': 'none', 'BONDING_OPTS': 'miimon=100 mode=802.3ad xmit_hash_policy=layer3+4', 'DEVICE': 'bond0', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'slaves': ['eth1', 'eth0'], 'hwaddr': '68:05:ca:01:bc:0d'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}
	clusterLevels = ['3.0', '3.1']
	cpuCores = 4
	cpuFlags = fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,popcnt,lahf_lm,ida,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_coreduo,model_Conroe
	cpuModel = Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz
	cpuSockets = 1
	cpuSpeed = 2394.098
	emulatedMachines = ['rhel6.4.0', 'pc', 'rhel6.3.0', 'rhel6.2.0', 'rhel6.1.0', 'rhel6.0.0', 'rhel5.5.0', 'rhel5.4.4', 'rhel5.4.0']
	guestOverhead = 65
	hooks = {}
	kvmEnabled = true
	lastClient = 80.242.147.190
	lastClientIface = ovirtmgmt
	management_ip = 
	memSize = 15938
	netConfigDirty = False
	networks = {'ovirtmgmt': {'addr': '192.168.245.18', 'cfg': {'USERCTL': 'no', 'IPADDR': '192.168.245.18', 'ONBOOT': 'yes', 'MTU': '1500', 'DELAY': '4', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'GATEWAY': '192.168.245.11'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '192.168.245.11', 'ports': ['bond0']}}
	nics = {'eth1': {'permhwaddr': 'B8:AC:6F:99:50:F6', 'addr': '', 'mtu': '1500', 'netmask': '', 'hwaddr': '68:05:ca:01:bc:0d', 'speed': 1000}, 'eth0': {'permhwaddr': '68:05:CA:01:BC:0D', 'addr': '', 'mtu': '1500', 'netmask': '', 'hwaddr': '68:05:ca:01:bc:0d', 'speed': 1000}}
	operatingSystem = {'release': '3.el6.centos.9', 'version': '6', 'name': 'RHEL'}
	packages2 = {'kernel': {'release': '358.0.1.el6.x86_64', 'buildtime': 1361941605.0, 'version': '2.6.32'}, 'spice-server': {'release': '12.el6', 'buildtime': '1361511255', 'version': '0.12.0'}, 'vdsm': {'release': '0.46.15.el6', 'buildtime': '1358926393', 'version': '4.10.0'}, 'qemu-kvm': {'release': '2.355.el6_4_4.1', 'buildtime': '1361521425', 'version': '0.12.1.2'}, 'libvirt': {'release': '18.el6', 'buildtime': '1361515221', 'version': '0.10.2'}, 'qemu-img': {'release': '2.355.el6_4_4.1', 'buildtime': '1361521425', 'version': '0.12.1.2'}}
	reservedMem = 321
	software_revision = 0.46
	software_version = 4.10
	supportedProtocols = ['2.2', '2.3']
	supportedRHEVMs = ['3.0', '3.1']
	uuid = 4C4C4544-0057-4210-8033-B1C04F5A344A_68:05:ca:01:bc:0d
	version_name = Snow Man
	vlans = {'bond0.100': {'netmask': '', 'iface': 'bond0', 'addr': '', 'mtu': '1500'}}
	vmTypes = ['kvm']
-------------- next part --------------
virsh -r capabilities
<capabilities>

  <host>
    <uuid>44454c4c-5700-1042-8033-b1c04f5a344a</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_disk/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <cpus num='4'>
            <cpu id='0'/>
            <cpu id='1'/>
            <cpu id='2'/>
            <cpu id='3'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.3.0</machine>
      <machine canonical='rhel6.3.0'>pc</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <pae/>
      <nonpae/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.3.0</machine>
      <machine canonical='rhel6.3.0'>pc</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>
-------------- next part --------------
virsh -r capabilities
<capabilities>

  <host>
    <uuid>44454c4c-5700-1042-8033-b1c04f5a344a</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Nehalem</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <feature name='rdtscp'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_disk/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <cpus num='4'>
            <cpu id='0'/>
            <cpu id='1'/>
            <cpu id='2'/>
            <cpu id='3'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.4.0</machine>
      <machine canonical='rhel6.4.0'>pc</machine>
      <machine>rhel6.3.0</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.4.0</machine>
      <machine canonical='rhel6.4.0'>pc</machine>
      <machine>rhel6.3.0</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>