virt July 2017

virt@lists.centos.org

16 participants
9 discussions

Crash in CentOS 7 kernel-3.10.0-514.16.1.el7.x86_64 in Xen PV mode
by Sarah Newman 24 Oct '17

24 Oct '17

I experienced a bug that is likely the same as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350373 . Commit b7dd0e350e0bd4c0fddcc9b8958342700b00b168 , which is supposed to fix it, doesn't appear in this kernel and doesn't apply cleanly either. Is there any point in trying to backport the patch? The backtrace is as follows: [ 32.304666] ------------[ cut here ]------------ [ 32.304679] kernel BUG at arch/x86/kernel/paravirt.c:252! [ 32.304683] invalid opcode: 0000 [#1] SMP [ 32.304687] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter intel_powerclamp coretemp pcspkr ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 32.304734] CPU: 0 PID: 3901 Comm: dracut Not tainted 3.10.0-514.16.1.el7.x86_64 #1 [ 32.304739] task: ffff880002598000 ti: ffff88001b728000 task.ti: ffff88001b728000 [ 32.304743] RIP: e030:[<ffffffff8167eb81>] [<ffffffff8167eb81>] enter_lazy.part.0+0x4/0x6 [ 32.304755] RSP: e02b:ffff88001f803aa8 EFLAGS: 00010002 [ 32.304758] RAX: 0000000000000001 RBX: ffff88001eacd640 RCX: 00003ffffffff000 [ 32.304761] RDX: ffff880000000640 RSI: ffffc900000c8000 RDI: 0000000000000001 [ 32.304765] RBP: ffff88001f803aa8 R08: ffff88001f803b78 R09: ffffffff813d50f9 [ 32.304771] R10: ffff88001e801e00 R11: ffffea0000093dc0 R12: ffffc900000c9000 [ 32.304777] R13: ffffc900000c8000 R14: 0000000000000000 R15: ffff88001d150340 [ 32.304787] FS: 00007f64425b0740(0000) GS:ffff88001f800000(0000) knlGS:0000000000000000 [ 32.304796] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 32.304801] CR2: 00000000006de2c8 CR3: 000000001b405000 CR4: 0000000000002660 [ 32.304807] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 32.304813] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 32.304818] Stack: [ 32.304823] ffff88001f803ab8 ffffffff81061857 ffff88001f803b60 ffffffff811b1fe2 [ 32.304833] ffffc900000c8fff ffffc900000c9000 ffffffff819bac90 ffffc900000c8fff [ 32.304843] ffffc900000c9000 ffff88001eacb000 ffffffff810206b0 0000000000000000 [ 32.304854] Call Trace: [ 32.304858] <IRQ> [ 32.304861] [<ffffffff81061857>] paravirt_enter_lazy_mmu+0x27/0x30 [ 32.304879] [<ffffffff811b1fe2>] apply_to_page_range+0x282/0x460 [ 32.304888] [<ffffffff810206b0>] ? map_pte_fn+0x60/0x60 [ 32.304894] [<ffffffff810207fb>] arch_gnttab_map_status+0x3b/0x70 [ 32.304904] [<ffffffff813d5176>] gnttab_map_frames_v2+0xd6/0x150 [ 32.304910] [<ffffffff813d5291>] gnttab_map+0xa1/0x140 [ 32.304917] [<ffffffff813d5430>] get_free_entries+0x100/0x2e0 [ 32.304923] [<ffffffff813d56d5>] gnttab_alloc_grant_references+0x15/0x30 [ 32.304933] [<ffffffffa000bd4f>] do_blkif_request+0x6bf/0x8a0 [xen_blkfront] [ 32.304945] [<ffffffff812eb0e2>] ? __freed_request+0x92/0xa0 [ 32.304951] [<ffffffff812eb6e3>] __blk_run_queue+0x33/0x40 [ 32.304957] [<ffffffff812eb719>] blk_start_queue+0x29/0x40 [ 32.304964] [<ffffffffa000bf51>] kick_pending_request_queues+0x21/0x30 [xen_blkfront] [ 32.304975] [<ffffffffa000c6ce>] blkif_interrupt+0x76e/0x820 [xen_blkfront] [ 32.304986] [<ffffffff811dcc8b>] ? kmem_cache_free+0x1bb/0x1f0 [ 32.304995] [<ffffffff8113079e>] handle_irq_event_percpu+0x3e/0x1e0 [ 32.305003] [<ffffffff8113097d>] handle_irq_event+0x3d/0x60 [ 32.305004] [<ffffffff81133647>] handle_edge_irq+0x77/0x130 [ 32.305004] [<ffffffff813d6217>] __xen_evtchn_do_upcall+0x227/0x350 [ 32.305004] [<ffffffff813d83c3>] xen_evtchn_do_upcall+0x33/0x50 [ 32.305004] [<ffffffff81698c7e>] xen_do_hypervisor_callback+0x1e/0x30 [ 32.305004] <EOI> [ 32.305004] [<ffffffff811af916>] ? copy_pte_range+0x2b6/0x5a0 [ 32.305004] [<ffffffff811af8e6>] ? copy_pte_range+0x286/0x5a0 [ 32.305004] [<ffffffff811b24d2>] ? copy_page_range+0x312/0x490 [ 32.305004] [<ffffffff81083012>] ? dup_mm+0x362/0x680 [ 32.305004] [<ffffffff810847ae>] ? copy_process+0x144e/0x1960 [ 32.305004] [<ffffffff81084e71>] ? do_fork+0x91/0x2c0 [ 32.305004] [<ffffffff81085126>] ? SyS_clone+0x16/0x20 [ 32.305004] [<ffffffff816974d9>] ? stub_clone+0x69/0x90 [ 32.305004] [<ffffffff81697189>] ? system_call_fastpath+0x16/0x1b [ 32.305004] Code: 20 e9 2f ff ff ff 44 89 fa 44 89 ee 48 c7 c7 10 45 8c 81 31 c0 e8 9d 14 00 00 58 5a 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5 <0f> 0b 66 66 66 66 90 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 [ 32.305004] RIP [<ffffffff8167eb81>] enter_lazy.part.0+0x4/0x6 [ 32.305004] RSP <ffff88001f803aa8> [ 32.305004] ---[ end trace 49f67f0d85e1ac69 ]--- [ 32.305004] Kernel panic - not syncing: Fatal exception in interrupt Thanks, Sarah

3 6

New xen-4.6.6 rpms for CentOS-6 and CentOS-7
by Johnny Hughes 13 Sep '17

13 Sep '17

There are new xen-4.6.6 rpms for CentOS-6 and CentOS-7 in the testing repository on buildlogs.centos.org. Can we please get some testing of these RPMs and feedback to this list. Thanks, Johnny Hughes

4 5

Copying CentOS AMIs for EBS encryption
by Andrew Hodgson 08 Aug '17

08 Aug '17

Hi, I am trying to copy the latest CentOS image to my own AWS account so that I can enable EBS volume encryption. It currently looks like the option when sharing the image is not enabled to allow direct copying of the snapshots, and I am wondering if this could be looked at? I can start an instance with this AMI, then create a new snapshot for the purpose of enabling EBS volume encryption, but I want to keep the image as clean as possible, and am worried if I run the image then some scripts will get ran that will customise the image with keys etc. If there is a better way to do this then happy to discuss. Thanks, Andrew.

2 2

AWS EC2 - CentOS 6 + 7 AMIs for new g3.* instance types?
by Stephan Koledin 27 Jul '17

27 Jul '17

Hello- Does anyone one this list maintain the official CentOS 6 + 7 AMIs in AWS? If so, could you please enable those images for use with the new g3.* instance types? If this list is the wrong place for this request, please point me in the right direction. Thanks! -Stephan

4 4

kernel-4.9.37-29.el7 (and el6)
by Johnny Hughes 24 Jul '17

24 Jul '17

Are the testing kernels (kernel-4.9.37-29.el7 and kernel-4.9.37-29.el6, with the one config file change) working for everyone: (turn off: CONFIG_IO_STRICT_DEVMEM) If we don't hear any negative comments by Wednesday July 19th, 2017 then we are going to push those to updates as they solve iscsi issues with some hardware and don't seem to impact anything else based on limited testing. BTW .. to test, edit the xen repo config file in /etc/yum.repos.d/ and turn on the testing repository .. or yum --enablerepo=centos-virt-xen-testing upgrade kernel* Thanks, Johnny Hughes

6 18

OVS+DPDK Problem
by Daniel Petrescu 20 Jul '17

20 Jul '17

Hi All, First time mailing here. I have installed on a CentOS 7.0 KVM (with DPDK and OVS) one Deep Packet Inspection VM. I have one channel and some virtual traffic generator. The traffic is lost between dpdk vhostuser and the DPI VM. The setup is attached. Any suggestions or ideas? Regarding the OVS+DPDK configuration, the following configuration is already made: - SELINUX is disabled - QEMU 2.9.0 was downloaded from sources - All Linux packages were updated for QEMU 2.9.0 - DPDK source was downloaded - Paths were set - epel repo was installed - DPDK 16.11 was installed - OVS 2.7.0 was installed with DPDK option - Hugepages VM is set - NICs are configured for DPDK - Permissions are set for DPDK Regarding the OVS+DPDK startup steps, the following configuration is in place: B. OVS+DPDK startup and configuration 1. Set and verify the memory hugepages mount -t hugetlbfs nodev /mnt/huge echo 64 > /sys/devices/system/node/node0/hugepages/hugepages- 1048576kB/nr_hugepages 2. Setting the driver permissions modprobe vfio-pci /usr/bin/chmod a+x /dev/vfio /usr/bin/chmod 0666 /dev/vfio/* 3. Configure the PATHS to DPDK and OVS database cd dpdk-stable-16.11.1/ export DPDK_DIR=$PWD export PATH=$PATH:/usr/local/share/openvswitch/scripts export DB_SOCK=/usr/local/var/run/openvswitch/db.sock 4. Re-initializing OVS rm /usr/local/etc/openvswitch/conf.db mkdir -p /usr/local/etc/openvswitch mkdir -p /usr/local/var/run/openvswitch ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema 5. Starting OVS + DPDK ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=" 1024,1024" ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start 6. Cheking driver mapping for NICs driverctl -v list-devices | grep -i net $DPDK_DIR/tools/dpdk-devbind.py --status 7. Adding OVS+DPDK bridge and ports ovs-vsctl add-br ch1int_dpdk -- set bridge ch1int_dpdk datapath_type=netdev ovs-vsctl add-br ch1ext_dpdk -- set bridge ch1ext_dpdk datapath_type=netdev ovs-vsctl add-port ch1int_dpdk nic810 -- set Interface nic810 type=dpdk options:dpdk-devargs=0000:81:00.0 ovs-vsctl add-port ch1ext_dpdk nic811 -- set Interface nic811 type=dpdk options:dpdk-devargs=0000:81:00.1 8. Adding OVS (no DPDK) bridge and ports ovs-vsctl add-br br0 ovs-vsctl add-port br0 myportnameone 9. Adding VM ports to OVS (or OVS+DPDK) bridge #<interface type='bridge'> # <mac address='52:54:00:cb:57:af'/> # <source bridge='br0'/> # <virtualport type='openvswitch'/> # ... #</interface> 10. Adding dpdkvhostuser to the ovs switch - these are used for the socket created between the VM and the OVS bridge ovs-vsctl add-port ch1int_dpdk ch1int_dvhu -- set Interface ch1int_dvhu type=dpdkvhostuser ovs-vsctl add-port ch1ext_dpdk ch1ext_dvhu -- set Interface ch1ext_dvhu type=dpdkvhostuser 11. DPI deployment virt-install --connect qemu:///system --name=DPI --memory=16384 --vcpus=16 --os-type=linux --os-variant=virtio26 --disk path=/path/image.qcow2,format=qcow2,bus=virtio,cache=none --network bridge=admin,model=e1000 --nographics --noautoconsole --import 12. Adding vhostuser ports to guest VM - for this the XML file for the virtual machine found at etc/libvirt/qemu/<VM_name>.xml was edited <interface type='vhostuser'> <mac address='52:54:00:2f:e8:de'/> <source type='unix' path='/usr/local/var/run/openvswitch/ch1int_dvhu' mode='client'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='52:54:00:b9:40:d0'/> <source type='unix' path='/usr/local/var/run/openvswitch/ch1ext_dvhu' mode='client'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface> Thanks, Daniel

1 0

4.9.34-29 EL6 and EL7 kernels
by Johnny Hughes 13 Jul '17

13 Jul '17

OK Guys and Gals, We have just kicked off a build of the 4.9.34-29 xen kernel on the Community Build Service for both EL6 and EL7. This kernel has some important things in it. 1. It has a patch for the stack guard issue (CVE-2017-1000364). Also the patch for Xen Security fix XSA-216. See these upstream changelogs for ALL kernel commits from 4.9.31 to 4.9.34: https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.32 https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.33 https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.34 2. It has a fix for the xen-netback rate limit queues (http://www.spinics.net/lists/netdev/msg441578.html) (Thanks very much to Kevin Stange and Jean-Louis Dupond for finding a fix for this issue) 3. It now contains debuginfo files (thanks to Sarah Newman for this amazing addition!). ===================== The kernels are currently building, it will likely take 4 to 6 hours for them to make their way to the applicable testing repos: CentOS-7: https://buildlogs.centos.org/centos/7/virt/x86_64/xen-46/ or CentOS-6: https://buildlogs.centos.org/centos/6/virt/x86_64/xen-44/ https://buildlogs.centos.org/centos/6/virt/x86_64/xen-46/ ===================== If you actually need the debuginfo files, they will be here (also in 4 or so hours): CentOS-7: http://cbs.centos.org/repos/virt7-xen-common-testing/x86_64/debug/ CentOS-6: http://cbs.centos.org/repos/virt6-xen-common-testing/x86_64/debug/ ===================== Let's do some real testing of these and see if we can solve the issues people are having .. this should already solve server and the debuginfo files should help solve others. I think Kevin and Jean-Louis also know how to solve the Windows BSOD when run on a Xen 4.9.x dom0 server .. they can comment here. Hopefully whoever maintains the newer Windows drivers will fix the ACPI / APIC issues and no work around will be necessary soon(ish). I also want to personally (and publicly) thank Kevin, Jean-Louis, and Sarah for the substantial work and the community collaboration. This is EXACTLY what we need for the Virt SIG .. people solving problems and doing github pull requests so we can, together as a group, get these projects working much better. If someone else wants to help us with the Kernel .. or Xen .. here are our github branches: https://github.com/CentOS-virt7/xen-kernel https://github.com/CentOS-virt7/xen Fork them .. work on them .. talk to us on #centos-virt in freenode or on this mailing list .. then once we are all happy, you can submit a pull request and we'll get your issues fixed. Thanks, Johnny Hughes

2 2

Live Migration and LibVirt CPU Mode
by Oisin O'Malley 06 Jul '17

06 Jul '17

Hi All, First time mailing here, I hope someone can help. We’re running a OpenStack Newton environment on top of CentOS-7.3, LibVirt and Qemu-KVM-EV. We are encountered an issue live migrating a VM between 2 hosts with different CPUs and LibVirt throws the following error: libvirtError: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, f16c, rdrand, fsgsbase, tsc_adjust, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, rdseed, adx, smap, xsaveopt, abm, 3dnowprefetch; try using 'Broadwell-noTSX' CPU model. LibVirt appears to be comparing the source and destination host CPUs, instead of guest VM and destination host CPUs. The VM is configured with a common set CPU features that are compatible with both (see below) and can run on both hosts without issue. The hosts have Westmere and Broadwell CPUs (See host CPU capabilities below). The VM can be booted on the Westmere host, successfully live migrated to the Broadwell host, but then throws the above error when we attempt to live migrating back to the Westmere host. Due to how our Westmere hosts were configured in OpenStack Nova, the VMs CPU is configured with "cpu mode='host-model'", which fail to migrate. BUT if the VMs CPU is configured with "cpu mode='custom' match='exact'" it can be successfully migrate both ways. It appears that when a VMs CPU is configured with "cpu mode='host-model'" the guests CPU is ignored and the host CPUs are compared instead. Which I believe is incorrect behaviour, is this a known issue and/or have any workarounds. The VMs will be configure to use "cpu mode='custom'" with a Westmere model on their next reboot, but rebooting all VMs prior to migration is something we really want to avoid. There has already been some discussion on this issue on the OpenStack mailing list: http://lists.openstack.org/pipermail/openstack/2017-July/thread.html#45139 The following package versions are being used; libvirt-daemon-kvm.x86_642.0.0-10.el7_3.4 qemu-kvm-ev.x86_6410:2.6.0-28.el7_3.3.1 kernel.x86_643.10.0-514.16.1.el7 The Qemu-KVM processes CPU parameters of the running VMs; -cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp Guest CPU Configuration that fails to migrate; <cpu mode='host-model'> <model fallback='allow'>Westmere</model> <vendor>Intel</vendor> <topology sockets='1' cores='1' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='ds'/> <feature policy='require' name='acpi'/> <feature policy='require' name='ss'/> <feature policy='require' name='ht'/> <feature policy='require' name='tm'/> <feature policy='require' name='pbe'/> <feature policy='require' name='pclmuldq'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='monitor'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='vmx'/> <feature policy='require' name='smx'/> <feature policy='require' name='est'/> <feature policy='require' name='tm2'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pcid'/> <feature policy='require' name='dca'/> <feature policy='require' name='arat'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='rdtscp'/> </cpu> Guest CPU Configuration that successfully migrates; <cpu mode='custom' match='exact'> <model fallback='allow'>Westmere</model> <topology sockets='1' cores='1' threads='1'/> </cpu> Westmere Hosts CPU capabilities; <cpu> <arch>x86_64</arch> <model>Westmere</model> <vendor>Intel</vendor> <topology sockets='1' cores='4' threads='2'/> <feature name='vme'/> <feature name='ds'/> <feature name='acpi'/> <feature name='ss'/> <feature name='ht'/> <feature name='tm'/> <feature name='pbe'/> <feature name='pclmuldq'/> <feature name='dtes64'/> <feature name='monitor'/> <feature name='ds_cpl'/> <feature name='vmx'/> <feature name='smx'/> <feature name='est'/> <feature name='tm2'/> <feature name='xtpr'/> <feature name='pdcm'/> <feature name='pcid'/> <feature name='dca'/> <feature name='arat'/> <feature name='pdpe1gb'/> <feature name='rdtscp'/> <feature name='invtsc'/> <pages unit='KiB' size='4'/> <pages unit='KiB' size='2048'/> <pages unit='KiB' size= Broadwell Host CPU Capabilities; <cpu> <arch>x86_64</arch> <model>Broadwell</model> <vendor>Intel</vendor> <topology sockets='1' cores='12' threads='2'/> <feature name='vme'/> <feature name='ds'/> <feature name='acpi'/> <feature name='ss'/> <feature name='ht'/> <feature name='tm'/> <feature name='pbe'/> <feature name='dtes64'/> <feature name='monitor'/> <feature name='ds_cpl'/> <feature name='vmx'/> <feature name='smx'/> <feature name='est'/> <feature name='tm2'/> <feature name='xtpr'/> <feature name='pdcm'/> <feature name='dca'/> <feature name='osxsave'/> <feature name='f16c'/> <feature name='rdrand'/> <feature name='arat'/> <feature name='tsc_adjust'/> <feature name='cmt'/> <feature name='xsaveopt'/> <feature name='mbm_total'/> <feature name='mbm_local'/> <feature name='pdpe1gb'/> <feature name='abm'/> <feature name='invtsc'/> <pages unit='KiB' size='4'/> <pages unit='KiB' size='2048'/> <pages unit='KiB' size='1048576'/> </cpu> Regards, Oisin Oisin O'Malley Systems Engineer Iocane Pty Ltd 763 South Road Black Forest SA 5035 Office:+61 (8) 8413 1010 Fax:+61 (8) 8231 2050 Email:oisin.omalley@iocane.com.au Web:www.iocane.com.au Better for business

3 2

Lightweight alternative to virt-manager
by C. L. Martinez 04 Jul '17

04 Jul '17

Hi all, Anyone knows any lightweight alternative to virt-manager that works with linux, Windows and macos? Searching I have found ovirt only but it is too heavy. Thanks

4 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

virt July 2017