virt

virt@lists.centos.org

1612 discussions

Problem with Xen4CentOS
by Thomas Weyergraf 18 Nov '14

18 Nov '14

Hi folks, we (the company i am working for) are running several dozens of virtualisation servers using CentOS 6 + Xen4CentOS as the virtualisation infrastructure. With the latest versions of all packages installed [1], we see failures in live-migration of stock-CentOS6 HVM guests, leaving a "Domain-unnamed" on the source host, while the migrated guest runs fine on the target host. Domain-0 0 2048 64 r----- 6791.5 Domain-Unnamed 1 4099 4 --ps-- 94.8 The failure is not consistently reproducable, some guests (of the same type) live-migrate just fine, until eventually some seemingly random guest fails, leaving a "Domain-unnamed" Zombie. That Domain-unnamed causes several problems: - The memory allocated to Domain-unnamed remains blocked, thus creating a veritable 'memory-leak' to the host - The DomU causing Domain-unnamed cannot be restarted on the host, as xm thinks it's already running I have tried various things to get rid of Domain-unnamed, all without success - multiple xm destroy - restart xend - delete everything regarding Dommain-unnamed in xenstore with xenstore-rm. The removal is successful, but the domain remains. Restarting xend after the deletion restores Domain-unnamed in xenstore So far, the only way to get rid of Domain-unnamed is a virt-host reboot. As these hosts are all quad-socket opteron 6272 machines with 256gig ram running dozens of guests, this is highly impractical. I have seen this behaviour using xen 4.2.5. The previous 4.2.4 versions did not show this problem, however we did not use live-migration extensively prior to that. Before switching to Xen4CentOS, we used to build our own Xen 4.2.2 based on a git repo, published by Karanbir Singh. We had several issues with that version, but never observed a "Domain-unnamed". Any idea how to resolve this issue would be highly appreciated, as working live-migration is crucial to us. Regards, Thomas Weyergraf Some notes on our config: 1. We still use xm/xend for various reasons ---- 2. Our grub-config for the virtx-hosts is as follows: ---- default=0 timeout=5 #splashimage=(hd0,0)/grub/splash.xpm.gz #hiddenmenu title CentOS (xen-4.2.5-37.el6.gz vmlinuz-3.10.56-11.el6.centos.alt.x86_64) root (hd0,0) kernel /xen-4.2.5-37.el6.gz iommu=1 console=vga,com1 com1=115200,8n1 vga=text-80x25 dom0_mem=2048M,max:2048M module /vmlinuz-3.10.56-11.el6.centos.alt.x86_64 ro xencons=hvc0 console=hvc0 root=/dev/fravirtx68/root rd_NO_LUKS LANG=en_US.UTF-8 KEYBOARDTYPE=pc KEYTABLE=de-latin1-nodeadkeys rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=fravirtx68/root rd_NO_DM module /initramfs-3.10.56-11.el6.centos.alt.x86_64.img title CentOS (vmlinuz-3.10.56-11.el6.centos.alt.x86_64) root (hd0,0) kernel /vmlinuz-3.10.56-11.el6.centos.alt.x86_64 ro root=/dev/fravirtx68/root rd_NO_LUKS LANG=en_US.UTF-8 KEYBOARDTYPE=pc KEYTABLE=de-latin1-nodeadkeys rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=fravirtx68/root rd_NO_DM module /initramfs-3.10.56-11.el6.centos.alt.x86_64.img 3. A typical guest-config looks like: ---- name = "fraappmgmt05t.test.fra.net-m.internal" uuid = "3778a443-9194-4c46-adff-211d7fcc24da" memory = "4096" vcpus = 4 kernel = "hvmloader" builder = 'hvm' disk = [ 'phy:/dev/disk/by-path/ip-192.168.240.7:3260-iscsi-iqn.1992-08.com.netapp:navfiler21-lun-56,xvda,w', 'phy:/dev/disk/by-path/ip-192.168.240.7:3260-iscsi-iqn.1992-08.com.netapp:navfiler21-lun-57,xvdb,w', ] vif = [ 'mac=00:16:3e:fa:15:4a,bridge=xenbr11' ] device_model = 'qemu-dm' serial='pty' xen_platform_pci=1 on_poweroff = "destroy" on_crash = "restart" 4. The xend.log excerpt of the migration process from the source host: ---- [2014-11-06 22:53:01 13499] DEBUG (XendDomainInfo:1795) Storing domain details: {'console/port': '7', 'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'cpu/2/availability': 'online', 'vm': '/vm/f5139575-984b-4c28-b470-efc042ba2703', 'domid': '1', 'store/port': '6', 'console/type': 'ioemu', 'cpu/0/availability': 'online', 'memory/target': '4194304', 'control/platform-feature-multiprocessor-suspend': '1', 'store/ring-ref': '1044476', 'cpu/1/availability': 'online', 'control/platform-feature-xs_reset_watches': '1', 'image/suspend-cancel': '1', 'name': 'migrating-fraapppeccon06.fra.net-m.internal'} [2014-11-06 22:53:01 13499] INFO (XendCheckpoint:423) xc_save: failed to get the suspend evtchn port [2014-11-06 22:53:01 13499] INFO (XendCheckpoint:423) [2014-11-06 22:53:34 13499] DEBUG (XendCheckpoint:394) suspend [2014-11-06 22:53:34 13499] DEBUG (XendCheckpoint:127) In saveInputHandler suspend [2014-11-06 22:53:34 13499] DEBUG (XendCheckpoint:129) Suspending 1 ... [2014-11-06 22:53:34 13499] DEBUG (XendDomainInfo:524) XendDomainInfo.shutdown(suspend) [2014-11-06 22:53:34 13499] DEBUG (XendDomainInfo:1882) XendDomainInfo.handleShutdownWatch [2014-11-06 22:53:34 13499] DEBUG (XendDomainInfo:1882) XendDomainInfo.handleShutdownWatch [2014-11-06 22:53:34 13499] INFO (XendDomainInfo:2079) Domain has shutdown: name=migrating-fraapppeccon06.fra.net-m.internal id=1 reason=suspend. [2014-11-06 22:53:34 13499] INFO (XendCheckpoint:135) Domain 1 suspended. [2014-11-06 22:53:35 13499] INFO (image:542) signalDeviceModel:restore dm state to running [2014-11-06 22:53:35 13499] DEBUG (XendCheckpoint:144) Written done [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:3077) XendDomainInfo.destroy: domid=1 [2014-11-06 22:53:35 13499] ERROR (XendDomainInfo:3091) XendDomainInfo.destroy: domain destruction failed. Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 3086, in destroy xc.domain_destroy(self.domid) Error: (16, 'Device or resource busy') [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:2402) Destroying device model [2014-11-06 22:53:35 13499] INFO (image:619) migrating-fraapppeccon06.fra.net-m.internal device model terminated [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:2409) Releasing devices [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:2415) Removing vif/0 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:2415) Removing console/0 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:2415) Removing vbd/51712 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:2415) Removing vbd/51728 [2014-11-06 22:53:35 13499] DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728 [2014-11-06 22:53:36 13499] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 26 2 0 0 5

2 3

Reminder: Virt SIG meeting on #centos-devel tomorrow, 2pm GMT
by George Dunlap 17 Nov '14

17 Nov '14

Just a reminder everyone, we'll be having our biweekly Virt SIG meeting Tuesday at 2pm GMT, on IRC, channel #centos-devel on freenode. If you have any particular items you'd like to put on the agenda, please let me know. Lars will be on a plane to China at that point, so I'll be chairing the meeting. -George

2 1

Re: [CentOS-virt] Official AWS AMIs: eu-central-1 support?
by Paul Schyska 14 Nov '14

14 Nov '14

Are the build scripts for the AMIs available somewhere? At least we could build them ourselves. Alternativly, maybe someone of the cloud group can build an AMI and share it as public AMI? We want to use Centos 7 too, and a migrated Image of mine refuses to start in eu-central-1. Thanks, Paul Schyska

1 0

Official AWS AMIs: eu-central-1 support?
by Axel S 13 Nov '14

13 Nov '14

Hello all, and thank you very much for the Centos 7 HVM AMI [1], greatly appreciated! This AMI does not work on the newly launched AWS Frankfurt region (eu-central-1). Any plans for fixing that? According to the AWS announcement [2], "The Region supports all sizes of T2, M3, C3, R3, and I2 instances". (We'll be running the CentOS 7 HVM AMI going forward, but looks like none of the other AMIs works on eu-central-1 either.) [1] https://aws.amazon.com/marketplace/pp/B00O7WM7QW [2] http://aws.amazon.com/blogs/aws/aws-region-germany/ Best, Axel

2 2

Xen Project Test Day for 4.5 RC2 on Thursday, Nov 13
by Russell Pavlicek 11 Nov '14

11 Nov '14

Since the folks on this list work with integrating Xen Project on CentOS, we'd like you to be aware of the upcoming Xen Project 4.5 RC2 Test Day. This Thursday, November 13, is our second Test Day for the 4.5 release cycle. Release Candidate 2 is now available for assessment. Information about testing this release can be found here: http://wiki.xenproject.org/wiki/Xen_4.5_RC2_test_instructions To learn more about Test Days, check out: http://wiki.xenproject.org/wiki/Xen_Project_Test_Days Russ Pavlicek Xen Project Evangelist, Citrix Systems Home Office: +1-301-829-5327 Mobile: +1-240-397-0199 UK VoIP: +44 1223 852 894

1 0

Test new xen, centos-release-xen and kernel for auto grub update on kernel install
by Johnny Hughes 22 Oct '14

22 Oct '14

The new process for auto updates of grub upon kernel install is in the xen4centos testing repo. In order to test these as updates to an existing system, you can do this: 1. download the test repo file: http://dev.centos.org/centos/6/xen-c6-RC1/xen-c6-RC1.repo 2. Put it in /etc/yum.repos.d/ 3. Issue this command: yum --disablerepo=Xen4CentOS upgrade xen\* centos-release-xen (that should install all the new files required to make the kernel update work automatically) 4. Review the new file /etc/sysconfig/xen-kernel ... if there is any other items you want on the 'kernel /xen.gz' line, you would edit the file. For example, I like to add 'com1=115200,8n1 console=com1' to the end of that line so I can use consoles in virsh. So I would change the line: XEN_KERNEL_MBARGS="--mbargs=dom0_mem=1024M,max:1024M loglvl=all guest_loglvl=all" to XEN_KERNEL_MBARGS="--mbargs=dom0_mem=1024M,max:1024M loglvl=all guest_loglvl=all com1=115200,8n1 console=com1" (both of those should be one line if they wrap) This is also where you can assign more or less memory to the dom0. etc. This article has some other possible parameters for the xen.gz line: http://www.informit.com/articles/article.aspx?p=1187966&seqNum=4 5. Make sure this is set as well in /etc/sysconfig/xen-kernel: BOOT_XEN_AS_DEFAULT="yes" 6. issue this command to upgrade the kernel: yum --disablerepo=Xen4CentOS upgrade kernel-\* 7. This should upgrade your kernel and it should also automatically update your /boot/grub/grub.conf to boot for dom0 use. If, for some reason, you want to manually upgrade the grub.conf file yourself, set the BOOT_XEN_AS_DEFAULT to "no" instead of yes. 8. If you want to install the xen kernel in other places other than xen dom0 machines, just make sure not to install the xen package, and grub.conf will get standard and not xen kernel boot entries. ==================== Once ypu have done your updates, please delete /etc/yum.repos.d/xen-c6-RC1.repo =================== Please report positive and negative experiences here so we can move this into the CentOS-6 xen4 repository (or fix things that are broken). Thanks, Johnny HUghes

3 6

Reminder: Virt SIG meeting on IRC today, 2pm BST
by George Dunlap 21 Oct '14

21 Oct '14

Just a reminder to everyone, we'll be having our bi-weekly meeting on IRC this week, #centos-devel, at 2pm BST. Feel free to e-mail any agenda items you'd like to have covered. One thing I'd like to discuss a bit this week is blktap2 -- at the moment we're using a 2-year-old snapshot of XenServer's blktap2 fork (aka "blktap2.5"). At some point the whole project needs to figure out a better way to deal with this; so I'd like to ask what CentOS needs to get from blktap, so we can come up with the best solution. -George

1 0

Domain Controllers virtualized KVM
by Andry Michaelidou 16 Oct '14

16 Oct '14

Hello to you all! We are implementing here at the University KVM virtualization for our servers and services and i was wondering if anyone virtualized domain cotrollers to KVM. Does anyone done this before? Any advice? -- Andry Michaelidou Papa | IT Systems Administrator|Department of Computer Science| University of Cyprus Tel: +357.22.892734 | Fax: +357.22.8927231 | http://www.cs.ucy.ac.cy

7 9

virsh list hangs / guests not starting automatically
by Matt Garman 14 Oct '14

14 Oct '14

I followed the wiki[1] to create a KVM virtual machine using bridged network on CentOS 6.5. It seemed to work fine on initial setup. (FWIW I'm trying to run a MythBuntu guest.) However, after a reboot, it doesn't auto-start the VMs. Shortly after boot, if I go into "virsh", then do a "list", it just hangs. Likewise, if I go into "virt manager", it just hangs with the message "connecting". Kernel version is: 2.6.32-431.29.2.el6.x86_64 Relevant package versions: libvirt.x86_64 0.10.2-29.el6_5.12 libvirt-client.x86_64 0.10.2-29.el6_5.12 libvirt-python.x86_64 0.10.2-29.el6_5.12 python-virtinst.noarch 0.600.0-18.el6 virt-manager.x86_64 0.9.0-19.el6 virt-top.x86_64 1.0.4-3.15.el6 virt-viewer.x86_64 0.5.6-8.el6_5.3 qemu-img.x86_64 2:0.12.1.2-2.415.el6_5.14 qemu-kvm.x86_64 2:0.12.1.2-2.415.el6_5.14 CPU is a Xeon E3-1230v3. I have the virtualization setting enabled in the BIOS. I googled on this, and saw a bunch of talk about two years ago regarding issues with the libvirt packages having a deadlock bug. But I think the versions of the relevant packages that I have installed are new enough to have fixes for that. I also happened across an earlier post to this list[2], where it seemed someone was having a similar problem. I was previously attempting to use balance-rr and 802.3ad bonding modes on my host. However, I just changed to using "active-backup" and the problem remains. I have in /etc/libvirt/libvirtd.conf the following three lines (the rest is stock, i.e. all comments): log_level = 2 log_filters="" log_outputs="1:file:/var/log/libvirt/libvirt.log" Below I posted the contents of the libvirt log file after doing a "service libvirt start". Anyone ever fought this before? Thanks! [1] http://wiki.centos.org/HowTos/KVM [2] http://lists.centos.org/pipermail/centos-virt/2014-March/003722.html /var/log/libvirt/libvirt.log output: 2014-10-14 16:47:11.150+0000: 4657: info : libvirt version: 0.10.2, package: 29.el6_5.12 (CentOS BuildSystem <http://bugs.centos.org>, 2014-09-01-13:44:02, c6b8.bsys.dev.centos.org) 2014-10-14 16:47:11.150+0000: 4657: info : virNetlinkEventServiceStart:517 : starting netlink event service with protocol 0 2014-10-14 16:47:11.151+0000: 4657: info : virNetlinkEventServiceStart:517 : starting netlink event service with protocol 15 2014-10-14 16:47:11.154+0000: 4668: info : dnsmasqCapsSetFromBuffer:667 : dnsmasq version is 2.48, --bind-dynamic is NOT present, SO_BINDTODEVICE is NOT in use 2014-10-14 16:47:11.157+0000: 4668: info : networkReloadIptablesRules:1925 : Reloading iptables rules 2014-10-14 16:47:11.157+0000: 4668: info : networkRefreshDaemons:1287 : Refreshing network daemons 2014-10-14 16:47:11.278+0000: 4668: info : networkStartNetwork:2422 : Starting up network 'default' 2014-10-14 16:47:11.290+0000: 4668: info : virStorageBackendVolOpenCheckMode:1085 : Skipping special dir '.' 2014-10-14 16:47:11.290+0000: 4668: info : virStorageBackendVolOpenCheckMode:1085 : Skipping special dir '..' 2014-10-14 16:47:11.352+0000: 4668: info : qemudStartup:754 : Unable to create cgroup for driver: No such device or address 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 : Configured cgroup controller 'cpu' 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 : Configured cgroup controller 'cpuacct' 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 : Configured cgroup controller 'cpuset' 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 : Configured cgroup controller 'memory' 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 : Configured cgroup controller 'devices' 2014-10-14 16:47:11.353+0000: 4668: info : qemudLoadDriverConfig:411 : Configured cgroup controller 'blkio' 2014-10-14 16:47:11.509+0000: 4668: info : virDomainLoadAllConfigs:14696 : Scanning for configs in /var/run/libvirt/qemu 2014-10-14 16:47:11.527+0000: 4668: info : virDomainLoadAllConfigs:14696 : Scanning for configs in /etc/libvirt/qemu 2014-10-14 16:47:11.527+0000: 4668: info : virDomainLoadAllConfigs:14718 : Loading config file 'mythbuntu.xml' 2014-10-14 16:47:11.529+0000: 4668: info : qemuDomainSnapshotLoad:484 : Scanning for snapshots for domain mythbuntu in /var/lib/libvirt/qemu/snapshot/mythbuntu

1 1

problem with xen virtulization on centos 6.5
by moslem mosadegh 13 Oct '14

13 Oct '14

I have a .img file and it's config xml file that i downloaded from a website. I installed xen 4.2 on my centos 6.5. when i want to create a new domain for my .img file with this xml file with this command: "virsh create wiki.xml". that give me an error message:"unkonw os type xen". anyone know what's the problem?

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

virt