virt February 2017

virt@lists.centos.org

18 participants
13 discussions

NIC Stability Problems Under Xen 4.4 / CentOS 6 / Linux 3.18
by Kevin Stange 29 Mar '17

29 Mar '17

I have three different types of CentOS 6 Xen 4.4 based hypervisors (by hardware) that are experiencing stability issues which I haven't been able to track down. All three types seem to be having issues with NIC and/or PCIe. In most cases, the issues are unrecoverable and require a hard boot to resolve. All have Intel NICs. Often the systems will remain stable for days or weeks, then suddenly encounter one of these issues. I have yet to tie the error to any specific action on the systems and can't reproduce it reliably. - Supermicro X8DT3, Dual Xeon E5620, 2x 82575EB NICs, 2x 82576 NICs Kernel messages upon failure: pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018 pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Transaction Layer, id=0018(Receiver ID) pcieport 0000:00:03.0: device [8086:340a] error status/mask=00002000/00001001 pcieport 0000:00:03.0: [13] Advisory Non-Fatal pcieport 0000:00:03.0: Error of this Agent(0018) is reported first igb 0000:04:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0400(Receiver ID) igb 0000:04:00.0: device [8086:10a7] error status/mask=00002001/00002000 igb 0000:04:00.0: [ 0] Receiver Error (First) igb 0000:04:00.1: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0401(Receiver ID) igb 0000:04:00.1: device [8086:10a7] error status/mask=00002001/00002000 igb 0000:04:00.1: [ 0] Receiver Error (First) This spams to the console continuously until hard booting. - Supermicro X9DRD-iF/LF, Dual Xeon E5-2630, 2x I350, 2x 82575EB igb 0000:82:00.0: Detected Tx Unit Hang Tx Queue <1> TDH <43> TDT <50> next_to_use <50> next_to_clean <43> buffer_info[next_to_clean] time_stamp <12e6bc0b6> next_to_watch <ffff880006aa7440> jiffies <12e6bc8dc> desc.status <1c8210> This spams to the console continuously until hard booting. - Supermicro X9DRT, Dual Xeon E5-2650, 2x I350, 2x 82571EB e1000e 0000:04:00.0 eth2: Detected Hardware Unit Hang: TDH <ff> TDT <33> next_to_use <33> next_to_clean <fd> buffer_info[next_to_clean]: time_stamp <138230862> next_to_watch <ff> jiffies <138231ac0> next_to_watch.status <0> MAC Status <80383> PHY Status <792d> PHY 1000BASE-T Status <3c00> PHY Extended Status <3000> PCI Status <10> This type of system, the NIC automatically recovers and I don't need to reboot. So far I tried using pcie_aspm=off to see that would help, but it appears that the 3.18 kernel turns off ASPM by default on these due to probing the BIOS. Stability issues were not resolved by the changes. On the latter system type I also turned off all offloading setting. It appears the stability increased slightly but it didn't fully resolve the problem. I haven't adjusted offload settings on the first two server types yet. I suspect this problem is related to the 3.18 kernel used by the virt SIG, as we had these running Xen on CentOS 5's kernel with no issues for years, and systems of these types used elsewhere in our facility are stable under CentOS 6's standard kernel. This affects more than one server of each type, so I don't believe it is a hardware failure, or else it's a hardware design flaw. Has anyone experienced similar issues with this configuration, and if so, does anyone have tips on how to resolve the issues? -- Kevin Stange Chief Technology Officer Steadfast | Managed Infrastructure, Datacenter and Cloud Services 800 S Wells, Suite 190 | Chicago, IL 60607 312.602.2689 X203 | Fax: 312.602.2688 kevin(a)steadfast.net | www.steadfast.net

8 35

xen 4.7 or 4.8
by Christoph 21 Mar '17

21 Mar '17

Hi are there somewhere pkgs with xen 4.7 or 4.8 for centos7? -- ---------- Greetz Christoph

3 2

NUMA placement failed, performance might be affected
by Chris Wik 01 Mar '17

01 Mar '17

I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64 After rebooting, the following notice is printed immediately upon xl create'ing a domain: libxl: notice: libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance might be affected Indeed performance is significantly degraded. This issue didn't happen on the older kernel. Any ideas where it came from or how it can be fixed? # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz stepping : 2 microcode : 0x36 cpu MHz : 3500.090 cache size : 15360 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt bugs : bogomips : 7000.18 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: # xl info host : ****.localdomain release : 3.18.44-20.el7.x86_64 version : #1 SMP Tue Oct 25 11:42:52 UTC 2016 machine : x86_64 nr_cpus : 1 max_cpu_id : 15 nr_nodes : 2 cores_per_socket : 1 threads_per_core : 1 cpu_mhz : 3500 hw_caps : bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab virt_caps : hvm hvm_directio total_memory : 130958 free_memory : 28003 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 6 xen_extra : .3-7.el7 xen_version : 4.6.3-7.el7 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty xen_commandline : placeholder dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) cc_compile_by : mockbuild cc_compile_domain : centos.org cc_compile_date : Wed Feb 15 12:18:43 UTC 2017 xend_config_format : 4 Thanks in advance, Chris -- Chris Wik Anu Internet Services www.anu.net | www.cwik.ch

2 2

New 4.9.11-22 kernels and linux-firmware packages to test in xen-testing for CentOS-6 and CentOS-7
by Johnny Hughes 27 Feb '17

27 Feb '17

I have pushed some new kernels and linux-firmware packages to the xen-testing repos for both CentOS-6 and CentOS-7. The CentOS-7 packages may take a few hours to make it to the repo. The CentOS-6 repo also contains a newer xfsprogs as that is required with newer kernels (https://bugzilla.redhat.com/show_bug.cgi?id=1314605). This kernel (and supplementary xfsprogs, linux-firmware) are scheduled to be the replacements for the 3.18.x LTS tree that has gone EOL from Kernel.org. We can maintain the 4.9 LTS tree until January 2019 (https://www.kernel.org/category/releases.html). We have added in blktap2 support to this kernel, please test that functionality if you need it, it seems to be working to me. We really need people to test this kernel and associated packages thoroughly so we can find and fix issues before release. Please post any issues or questions on this list.

3 4

Running graphical applications from CentOS headless vm
by C. L. Martinez 26 Feb '17

26 Feb '17

Hi all, I have installed a CentOS7 vm in my home server with all graphical tools installed: Gnome, Chrome, Tor Borwser, etc. My idea is to run these graphical applications from two MacOSX desktops. What I am looking for is something similar like Microsoft RDP services that supports copy and paste between client and server, sound, clipboard, etc ... I have seen a possible solution using xrdp: http://www.xrdp.org. But exists some other solution?? Thanks. -- Greetings, C. L. Martinez

4 3

Problem loading the VM after reboots.
by Subscriber 25 Feb '17

25 Feb '17

I had a strange problem. After restarting the VM can not always run. The last line in the console (virsh console): init: Re-executing /sbin/init Please stand by while rebooting the system... Restarting system. machine restart At this time, one of the CPU core is loaded 100% (qemu-kvm process, top command) %Cpu7 : 41.8 us, 57.6 sy, 0.0 ni, 0.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st # free total used free shared buff/cache available Mem: 8009692 377744 1164076 9392 6467872 7299800 Swap: 2097148 0 2097148 This behavior only when the virtual machine CentOS 6.8 and HV CentOS 7.3.1116. Unsuccessful restart may occur after 2 successful, and maybe after 5 successful, and maybe on the first reboot. No clear pattern of a problem I have not seen. Tested different VM. Here's how behaved virtual machine if its reboot through virsh reboot: virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # destroy test1 Domain test1 destroyed virsh # start test1 Domain test1 started virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # destroy test1 Domain test1 destroyed virsh # start test1 Domain test1 started virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # reboot test1 Domain test1 is being rebooted virsh # destroy test1 Domain test1 destroyed If the VM shutdown and then start, the problem does not occur. The problem does not occur if the VM CentOS 7.3.1611 (10 successful reboots of 10) or if the VM Windows Server 2008 R2 (10 successful reboots of 10). If the VM CentOS 6.8 and HV CentOS 6.8 the problem does not occur. First HV is: CentOS 7.3.1611 with 3.10.0-514.6.2.el7.x86_64 and 3.10.0-514.6.1.el7.x86_64 qemu-kvm-ev.x86_64 10:2.6.0-28.el7_3.3.1 libvirt.x86_64 2.0.0-10.el7_3.4 Kernel module: kvm_intel 170181 kvm 554609 Second HV is: CentOS 7.3.1611 with 3.10.0-514.6.2.el7.x86_64 qemu-kvm.x86_64 10:1.5.3-126.el7_3.3 libvirt.x86_64 2.0.0-10.el7_3.4 Kernel module: kvm_intel 170181 kvm 554609

1 0

grub2 for pv
by Christoph 21 Feb '17

21 Feb '17

Hi could you add pv grub2 images to the packages? I've tried it following this howto: https://blog.xenproject.org/2015/01/07/using-grub-2-as-a-bootloader-for-xen… and it works really good... It would be nice to have it in the rpm packages. -- ---------- Greetz Christoph

1 0

Docker container scanner on CentOS?
by Rafał Radecki 21 Feb '17

21 Feb '17

Hi All. I am currently searching for a decent image/container/registry scanner. I would like to be able to check images for CVE, at the moment I am using rhel/centos/ubuntu/debian based images. I tried on CentOS7: - openscap (oscap-docker): needs atomic for installation, allows scanning of rhel based images only; - atomic: allows scanning of rhel based images only; - clair: usable in theory for rhel/centos/ubuntu/debian images but in practice I encountered problems with analyze-local-images and hyperclair "cli" tools and API does not allow automatization; - banyan collector/dockscan/drydock: seem to be stale or not enough mature to be considered; - nessus: seems to be an overkill for my usecase. I am now looking into: - aqua (commercial); - twistlock (commercial); - blackduck docker scanner (commercial). Can you share info about what you are using to scan docker images? Any proposals for my usecase? Thanks! BR, Rafal.

1 0

Network Problem
by Günther J. Niederwimmer 20 Feb '17

20 Feb '17

Hello List, I have two Server with CentOS 7.(3) and KVM, on both a have the Problem with the network ? I have configured 4 NIC on the slave KVM, the most time only 3 NIC is starting sometime only 2 NIC ? two nic are configured as "hostdev" the other two NIC are configures from two bridge (Host) the default KVM Bridge and a created "br0" over a phys NIC. The most time this br0 -> eth1 is lost in the KVM Client ? after a restart mostly all is OK. Have any a idea why ? -- mit freundlichen Grüssen / best regards Günther J. Niederwimmer

1 0

Xen updates in the Testing Repo for XSA-207 and XSA-208
by Johnny Hughes 18 Feb '17

18 Feb '17

There are xen rpms in the testing repos for XSA 207 and 208 in the testing repos (xen-4.4.4-18.el6, xen-4.6.3-7.el6, xen-4.6.3-7.el7). You can enable the applicable centos-virt-xen-testing repo in your /etc/yum.repos.d/CentOS-Xen.repo file. Please report positive and negative tests to this list so we can promote the updates to the main repos. Thanks, Johnny Hughes

2 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

virt February 2017