virt March 2010

virt@lists.centos.org

30 participants
22 discussions

reboot guest on panic
by Norman Gaywood 16 Mar '10

16 Mar '10

I have a guest that keeps crashing and want to automatically reboot it when it crashes. See: xen PV guest kernel 2.6.32 processes lock up in D state https://bugzilla.redhat.com/show_bug.cgi?id=550724 if you want to look at the details on the crashing. Anyway, I boot the guest with the kernel command line parameter: hung_task_panic=1 I have kernel.panic = 15 in the guest /etc/sysctl.conf In the guest config file in dom0 I have: on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" The guest manages to panic when it detects the hung tasks and I get this on the guest console: Kernel panic - not syncing: hung_task: blocked tasks Rebooting in 15 seconds.. However, it never restarts. It just hangs around until I do a xm destroy <guest> xm create <guest> Have I missed something? -- Norman Gaywood, Computer Systems Officer University of New England, Armidale, NSW 2351, Australia ngaywood(a)une.edu.au Phone: +61 (0)2 6773 3337 http://mcs.une.edu.au/~norm Fax: +61 (0)2 6773 3312 Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html

5 13

Re: [CentOS-virt] reboot guest on panic
by Christopher G. Stach II 15 Mar '10

15 Mar '10

----- "Norman Gaywood" <ngaywood(a)une.edu.au> wrote: > Well yes, seems I've hit trouble this time. Was expecting some trouble > with the desktop apps and was happy to deal with that. I was not > expecting significant kernel trouble however. It's Fedora. It's *always* broken. > Rebuilding a modern desktop distro with an older kernel is a lot of > work. It's not that bad. The only things you really need to match up are glibc and the kernel. Beyond that, most things are reasonably source compatible with a slightly older glibc. I don't think you would need to go that far, though. What do you need in Fedora that you can't get in the almost equally substandard desktop experience from CentOS or any other enterprise distribution? (I use CentOS as my primary desktop.) > That's what distributions are for right? Gentoo? :) -- Christopher G. Stach II http://ldsys.net/~cgs/

1 0

Re: [CentOS-virt] gpl pv drivers and Centos 5.4
by F M 14 Mar '10

14 Mar '10

TX to you both of you for the info. I will test these drivers tomorrow. Regards, F Regards,----- Original Message ----- From: "Pasi Kärkkäinen" <pasik(a)iki.fi> To: "Discussion about the virtualization on CentOS" <centos-virt(a)centos.org> Sent: Sunday, March 14, 2010 9:12:42 AM Subject: Re: [CentOS-virt] gpl pv drivers and Centos 5.4 On Sat, Mar 13, 2010 at 07:17:58AM -0500, F M wrote: > Hello, > I am using Windows XP/2003 and 2008 under Xen. What is the correct version of gpl pv drivers for Windows under Centos 5.4 Dom0 ? rpm -q xen said 3.0.3.x but xen info said 3.1.x > Xen hypervisor in EL5 is version 3.1.2 + a lot of patches from Redhat and from newer Xen versions. Xen tools in EL5 (xm/xend etc) are based on version 3.0.3 + patches. -- Pasi _______________________________________________ CentOS-virt mailing list CentOS-virt(a)centos.org http://lists.centos.org/mailman/listinfo/centos-virt

1 0

gpl pv drivers and Centos 5.4
by F M 14 Mar '10

14 Mar '10

Hello, I am using Windows XP/2003 and 2008 under Xen. What is the correct version of gpl pv drivers for Windows under Centos 5.4 Dom0 ? rpm -q xen said 3.0.3.x but xen info said 3.1.x Regards,

3 2

Re: [CentOS-virt] reboot guest on panic
by Christopher G. Stach II 14 Mar '10

14 Mar '10

----- "Norman Gaywood" <ngaywood(a)une.edu.au> wrote: > > If your guest's state is corrupted, you can't rely on its behavior. > For that reason, you should set up a watchdog and only rely on the > panic behavior as a preliminary measure. > > I figured that I had setup a watchdog with the hung_task_panic=1 guest > parameter. Is there another way of setting up a watchdog in this case? Internal watchdogs are seldom as effective as you need them to be. I meant an external one. There was talk in the past about a Xen watchdog driver, but I don't know where that ended up. (I use existing code that works for me, so I haven't kept up.) There are various ways to set one up. You can do it on your own with a couple of scripts and xenstore, if you're so inclined. If not, there are a lot of existing examples and projects that will handle it for you. This one that uses QEMU and unfortunately only KVM is decent and should give you enough ideas on how to implement one: http://rwmj.wordpress.com/2010/03/03/what-is-a-watchdog/ -- Christopher G. Stach II http://ldsys.net/~cgs/

1 0

unable to get domain status from libvirt & KVM
by Tom Georgoulias 12 Mar '10

12 Mar '10

I have a python script that monitors the VMs on physical host servers running Xen, but the script doesn't work properly on a server I just built with KVM. The script runs as a non-root user (same on all servers) and simply gathers some details on the status and names of the domains running on the host. Both Xen and KVM servers are running the same version of libvirt (libvirt-0.6.3-20.1.el5_4) and have the same, default /etc/libvirt/libvirtd.conf config file. To troubleshoot, I've been running python interactively. Here's how my Xen servers behave: $ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) >>> domains = conn.listDomainsID() >>> print domains [0, 3, 15, 16, 21, 24, 26, 30, 32, 36, 38, 41, 43, 45, 47, 49, 51, 55, 63, 67] >>> When I try the same thing on the KVM server: $ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) 14:33:07.303: error : No vport operation path found for host0 14:33:07.320: error : No vport operation path found for host4 14:33:07.325: error : No vport operation path found for host3 14:33:07.367: error : No vport operation path found for host1 14:33:07.368: error : No vport operation path found for host2 >>> domains = conn.listDomainsID() >>> print domains [] >>> (The vport stuff is weird, but I found this posting that suggests its harmless (http://www.mail-archive.com/libvir-list@redhat.com/msg17477.html) so I'm ignoring it.) However, when logged in as root on the KVM server, it works just like my Xen servers: # python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) >>> domains = conn.listDomainsID() >>> print domains [1] Again, on the KVM server, plain old "virsh list" with the debug level set to "2" $ export LIBVIRT_DEBUG=2 $ virsh list 14:21:06.532: error : No vport operation path found for host0 14:21:06.550: error : No vport operation path found for host4 14:21:06.555: error : No vport operation path found for host3 14:21:06.598: error : No vport operation path found for host1 14:21:06.599: error : No vport operation path found for host2 14:21:06.615: info : No security driver available Id Name State ---------------------------------- Permissions in /var/run/libvirt: # ls -ld /var/run/libvirt/* srwx------ 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock srwxrwxrwx 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock-ro drwxr-xr-x 2 root root 4096 Jan 21 14:38 /var/run/libvirt/network drwxr-xr-x 2 root root 4096 Oct 20 18:50 /var/run/libvirt/qemu Can someone provide some tips on what else I can check, if this might be a bug, or point out any mistakes that I might've made? Any help is appreciated. Thanks, Tom

3 11

Logrotate/cron and major I/O contention with KVM.
by Steven Ellis 11 Mar '10

11 Mar '10

Is anyone else having major I/O peaks due to logrotate or other jobs running simultaneously across multiple guests. I have one KVM server running Centos 5.4 with local disk that is seriously suffering as most of the guests rotate their syslog at the same time. Looking at the KVM server I'm seeing 11:00:01 PM CPU %user %nice %system %iowait %steal %idle 03:40:01 AM all 0.07 0.00 2.74 0.93 0.00 96.26 03:50:01 AM all 0.07 0.00 1.17 1.18 0.00 97.58 04:00:01 AM all 0.08 0.00 1.51 0.82 0.00 97.59 04:10:02 AM all 0.53 0.03 15.31 51.61 0.00 32.53 04:20:01 AM all 0.28 0.12 4.12 22.21 0.00 73.27 04:30:01 AM all 0.07 0.00 0.80 1.21 0.00 97.92 04:40:01 AM all 0.07 0.00 2.60 1.81 0.00 95.52 04:50:01 AM all 0.08 0.00 0.79 1.44 0.00 97.69 On one of the guests running Centos 4.6 the impact is so bad I get DMA timeout errors in the syslog, and occasional kernel panics. Mar 11 04:05:04 localhost kernel: hda: dma_timer_expiry: dma status == 0x21 Mar 11 04:05:14 localhost kernel: hda: DMA timeout error Mar 11 04:05:14 localhost kernel: hda: dma timeout error: status=0x50 { DriveReady SeekComplete } Mar 11 04:05:14 localhost kernel: Mar 11 04:05:14 localhost kernel: ide: failed opcode was: unknown Mar 11 04:05:59 localhost kernel: hda: dma_timer_expiry: dma status == 0x21 Mar 11 04:06:14 localhost kernel: hda: DMA timeout error Mar 11 04:06:14 localhost kernel: hda: dma timeout error: status=0x50 { DriveReady SeekComplete } One reference I've found is at * http://lonesysadmin.net/linux-virtual-machine-tuning-guide/ This suggests avoiding running scheduled jobs simultaneously across guests, and suggests using a random sleep. Does anyone else have suggestions on reducing the impact of cron/logrotate. -- *Steven Ellis - Director of Worldwide Engineering,* *Bulletin.Net Inc* - http://www.bulletin.net/

4 3

Re: [CentOS-virt] Introducing ConVirt 2.0
by Christopher G. Stach II 10 Mar '10

10 Mar '10

----- "Pasi Kärkkäinen" <pasik(a)iki.fi> wrote: > On Sat, Mar 06, 2010 at 09:04:20AM -0500, Kanwar Ranbir Sandhu wrote: > > Why would one use ConVirt instead of the management tools included in > > RHEL and/or CentOS? What's the difference? > > RHEL/CentOS doesn't provide web-based management.. or even easy > multi-host / cluster management of virtualization nodes. > > -- Pasi Are there any *good* reasons? (Since I really hate commercials, I feel compelled to present my contrarian viewpoint.) ConVirt addresses a pretty small portion of the virtualization landscape, and it consists of only a few significant parts: 1. Do what other free and open tools already do. 2. Slap a web interface on it! 3. Spam lists. 4. Rope in suckers. The suggestion that a web interface is a value add to an infrastructure issue is at least insulting. You could attempt to slap a web interface on a fuel injection system (or maybe at least give access to the magic a la MegaSquirt), but a bunch of assholes are still going to blow something up. It's not going to give any admin worth his or her salt a boner because it's not readily scriptable and it amounts to candy for retards. Secondly, everything else that it does is already there. If you can't do it, you shouldn't be touching the machines. The tool may or may not address some vanilla installations (if there ever was one), but if you need something like that, you are probably better off with EC2 or at least letting someone else handle it. -- Christopher G. Stach II http://ldsys.net/~cgs/

5 5

Re: [CentOS-virt] [fedora-virt] Thoughts on storage infrastructure for small scale HA virtual machine deployments
by Dennis J. 07 Mar '10

07 Mar '10

On 03/02/2010 04:51 AM, Ask Bjørn Hansen wrote: > > On Mar 1, 2010, at 18:56, Dennis J. wrote: > >> The question that bugs me is how I can get enough bandwidth between the >> hosts and the storage to provide the VMs with reasonable I/O performance. >> If all the 40 VMs start copying files at the same time that would mean that >> the bandwidth share for each VM would be tiny. > > It really depends on the specific workloads. In my experience it's generally the number of IOs per second rather than the bandwidth that's the limiting factor. > > We have a bunch of 4-disk boxes with md raid10 and we generally run out of disk IO before we run out of memory (~24-48GB) or CPU (dual quad core 2.26GHz or some such). That's very similar to what we are experiencing. The primary Problem for me is how to deal with the bottleneck of a shared storage setup. The most simple setup is a 2-system criss-cross setup where the two hosts also serve as halves for a DRBD cluster. The advantage of this approach is that it's a cheap solution, that only a part of the storage-traffic has to go over the network between the machines and that the network only hast to handle the sorage-traffic of the VMs of those two machines. The disadvantage of that approach is that you have to keep 50% of potential server capacity free in case of a failure of the twin node. That's quite a lot of wasted capacity. To reduce that problem you can increase the number of hosts to let say for an example four which would reduce the spare capacity needed to 33% on each system but then you really need to separate the storage from the hosts an now you have a bottleneck on the storage end. Increase the number of hosts to 8 and you get even less wasted capacity but also increase the pressure on the storage bottleneck a lot. Since I'm new to the whole SAN aspect I'm currently just looking at all the options that are out there and basically wonder how the big boys are handling this who have hundreds if not thousand of VMs running and need to be able to deal with physical failures too. That is why I find the sheepdog project so interesting because it seems to address this particular problem in a way that would provide almost linear scalability without actually using a SAN at all (well, at least not in the traditional sense of the word). Regards, Dennis

1 0

Thoughts on storage infrastructure for small scale HA virtual machine deployments
by Dennis J. 07 Mar '10

07 Mar '10

Hi, up until now I've always deployed VMs with their storage located directly on the host system but as the number of VMs grows and the hardware becomes more powerful and can handle more virtual machines I'm concerned about a failure of the host taking down too many VMs in one go. As a result I'm now looking at moving to an infrastructure that uses shared storage instead so I can live-migrate VMs or restart them quickly on another host if the one they are running on dies. The problem is that I'm not sure how to go about this bandwidth-wise. What I'm aiming for as a starting point is a 3-4 host cluster with about 10 VMs on each host and a 2 system DRBD based cluster as a redundant storage backend. The question that bugs me is how I can get enough bandwidth between the hosts and the storage to provide the VMs with reasonable I/O performance. If all the 40 VMs start copying files at the same time that would mean that the bandwidth share for each VM would be tiny. Granted this is a worst case scenario and that's why I want to ask if someone in here has experience with such a setup, can give recommendations or comment on alternative setups? Would I maybe get away with 4 bonded gbit ethernet ports? Would I require fiber channel or 10gbit infrastructure? Regards, Dennis PS: The sheepdog project (http://www.osrg.net/sheepdog/) looks interesting in that regard but apparently still is far from production-ready.

6 6

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

virt March 2010