March 2010 - virt - lists.centos.org

RHEL 5.5 Xen fixes
by Pasi Kärkkäinen 15 May '10

15 May '10

Hello, RHEL 5.5 has been released, and it contains many Xen related fixes. Here's the most important of the them.. fixing a dom0 caching bug which could cause domU disk corruption when the domU disk was accessed from dom0 after the domU was shutdown. Most people noticed this bug when pygrub used wrong (cached) information. "pygrub uses cached and eventually outdated grub.conf, kernel and initrd": https://bugzilla.redhat.com/show_bug.cgi?id=466681 And some more: "Xen hypervisor doesn't mask xsave feature from the guest; Fedora 11 PV domU kernel crashes": https://bugzilla.redhat.com/show_bug.cgi?id=524719 "[RHEL-5 Xen]: F-11 Xen 64-bit domU cannot be started with > 2047MB of memory": https://bugzilla.redhat.com/show_bug.cgi?id=502826 " Boot hang when installing HVM DomU": https://bugzilla.redhat.com/show_bug.cgi?id=524052 "[RHEL5 Xen]: PV guest crash on poweroff": https://bugzilla.redhat.com/show_bug.cgi?id=540811 "[RHEL5 Xen]: Cpu frequency scaling is broken on Intel": https://bugzilla.redhat.com/show_bug.cgi?id=553324 RHEL 5.5 kernel changelog: http://rhn.redhat.com/errata/RHSA-2010-0178.html Work-in-progress (Not yet in 5.5): "Grub2 guest support for RHEL5 Xen pygrub": https://bugzilla.redhat.com/show_bug.cgi?id=577511 -- Pasi

6 10

Re: [CentOS-virt] Xen HVM domU won't start since updates to 2.6.18-164.15.1 kernels
by Christopher G. Stach II 12 Apr '10

12 Apr '10

----- "Aaron Clark" <ophidian(a)ophidian.homeip.net> wrote: > qemu: could not open serial device 'none' http://os-drive.com/files/docbook/xen-faq.html#serial_console_hvm Does the LV show that it's open (or ever changes state) when the VM attempts to boot? Did you try taking a snapshot of the LV and mucking around with the xen configuration to see if anything changes? Did you update anything else on the machine, like perhaps the BIOS? Can you mount the VM's filesystem? Is its partition table okay? How about the boot loader? -- Christopher G. Stach II http://ldsys.net/~cgs/

4 5

Re: [CentOS-virt] Xen HVM domU won't start since updates to 2.6.18-164.15.1 kernels
by Christopher G. Stach II 30 Mar '10

30 Mar '10

----- "Aaron Clark" <ophidian(a)ophidian.homeip.net> wrote: > What steps should I take next to debug this? Downgrade back to where you were bit by bit (or just guess that it's dom0's kernel) and see when it starts working again. What does the busted guest say when it tries to boot while you watch its [serial] console? Does the dom0 ever unpause the guest? (It should say in the logs.) -- Christopher G. Stach II http://ldsys.net/~cgs/

4 9

Matthias Lorch ist außer Haus.
by matthias.lorch＠kzvrlp.de 29 Mar '10

29 Mar '10

Ich werde ab 29.03.2010 nicht im Büro sein. Ich kehre zurück am 06.04.2010. Bitte wenden Sie sich in dringenden Fällen an Herrn Roth, 0261 / 9120 - 258 Vielen Dank Gruß M.Lorch

1 0

Xen HVM domU won't start since updates to 2.6.18-164.15.1 kernels
by Aaron Clark 28 Mar '10

28 Mar '10

I have a dom0 machine running CentOS 5.4 with all the latest updates using Xen as my hypervisor. I am using Xen in part because this machine was set up prior to KVM being included in RHEL, and in part because KVM's network bridging configuration is not nearly as simple as Xen's. The dom0 machine is a headless Mac Mini and I do all of my VM management via virsh from the command line. I have two hvm domU's: * A web server running CentOS 5.4 * A mail server running Gentoo Both VM's are backed by LV's on the dom0 but do not use LVM in the domU. Both have virtually identical libvirt configurations (differing by expected things like name, UUID, NIC MAC, VNC port, etc). The web server domU (WSdomU hereafter) does not start since applying the most recent kernel update (kernel-xen-2.6.18-164.15.1.el5.x86_64 and kernel-2.6.18-164.15.1.el5.x86_64 for the dom0 and WSdomU respectively). By 'not start' I mean it appears to be running but it does not use an CPU cycles, does not bring up a graphical console, and does not respond on the network. The WSdomU is listed as no state rather than the normal running or blocked in xentop. The mail server domU starts fine and functions normally. Here are the steps I have taken so far that did not solve the problem: * Reboot the dom0 to see if things come up on their own * Attempted to connect to the WSdomU's graphical (VNC) console from the dom0 * Check xen dmesg on dom0 * Check xend logs (a cursory viewing did not show anything blatant; specific suggestions of things to look for would be appreciated) * Shutdown the mail server domU and attempt to start the WSdomU * Use kpartx to access the partitions of the domU o Tried switching grub to use the previous kernel * Check the SELinux labels on backing LV's (they're the same) * Set SELinux to permissive and attempt to start the WSdomU * Use virsh edit to try tweaking the WSdomU config * virsh undefine, reboot, virsh define the WSdomU config * Use fdisk on the LV to ensure it has the correct partition layout * dd the WSdomU LV to an .img file, copy it to my Fedora desktop and run it under KVM (works fine) * dd the .img file to a new LV and create a new libvirt config xml (fails to start) Sample config used to attempt a reconfiguration: <domain type='xen'> <name>Webserver</name> <os> <type>hvm</type> <loader>/usr/lib/xen/boot/hvmloader</loader> <boot dev='hd'/> </os> <memory>262144</memory> <vcpu>1</vcpu> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <features> <acpi/> <apic/> <pae/> </features> <devices> <emulator>/usr/lib64/xen/bin/qemu-dm</emulator> <interface type='bridge'> <source bridge='xenbr0'/> <script path='vif-bridge'/> </interface> <disk type='block'> <source dev='/dev/mapper/SystemsVG-Webserver'/> <target dev='hda'/> </disk> <graphics type='vnc' /> </devices> </domain> What steps should I take next to debug this? Aaron -- "The goblins are in charge of maintenance? Why not just set it on fire now and call it a day?" --Whip Tongue, Viashino Technician

1 0

PVM Ubuntu and xconsole error with XEN+CentOS
by Fernando Castillo Ortiz 24 Mar '10

24 Mar '10

Hi, I tell you my problem: I've installed a Ubuntu's HVM guest in CentOS with XEN, then I convert my HVM to PVM based in this tutto : http://www.asplund.nu/xencluster/xen-convert-hvm-to-pvm.html After this modification everythings it runs fine but I have a problem with xconsole, I don't see anything when I run the command "xm console ID" , I only see the boot proccess and the last line that I can see it's : * Running local boot scripts (/etc/rc.local) [ OK ] I attached you my config file PS: Sorry for my English -- Fernando Castillo Ortiz Área de Operaciones Emergya Consultoría Tel. +34 951 210 400 +34 663 366 796 Fax. +34 954 51 64 73 http://www.emergya.es

1 0

Silly question about KVM
by Scot P. Floess 18 Mar '10

18 Mar '10

I was wondering, if I do not have hardware that natively supports full virtualization...and I choose to use KVM, will my VMs be running in some form of chip emulation (and therefore terribly slow). To date, I've been using Xen and am very comfortable with it. I have some fears that later whenever Xen is dropped - I'll have to consider KVM. Also, will Xen be carried forward should Xen be dropped from RHEL? -- Scot P. Floess 27 Lake Royale Louisburg, NC 27549 252-478-8087 (Home) 919-890-8117 (Work) Chief Architect JPlate http://sourceforge.net/projects/jplate Chief Architect JavaPIM http://sourceforge.net/projects/javapim Architect Keros http://sourceforge.net/projects/keros

4 6

Re: [CentOS-virt] moving from Xen to KVM
by Christopher G. Stach II 18 Mar '10

18 Mar '10

----- "Dave Augustus" <davea(a)ingraftedsoftware.com> wrote: > I finally realized that when running Xen and in Dom0, Xen hides the > AMD-V in /proc/cpuinfo Really? dom0: flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc pni monitor cx16 lahf_lm cmp_legacy svm cr8_legacy guest: flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc pni monitor cx16 lahf_lm cmp_legacy svm cr8_legacy > (...more reasons to move to KVM) I have a hunch that you are going to do whatever you are told to do and that you really have no idea what is going on. Can we get more "Hey, I'm told KVM is cool so I'm migrating all of my production servers to it because the free OS of my choice may not support it in a year or two after I leave my current suckjob position and I hear, since I don't read mainline kernel mailing lists religiously and keep only a superficial idea in my head about kernel development, since I really have no idea what 'kernel development' means, that KVM is the only Jesus-approved way of doing things" threads? None of you seem to have have a head for any of this, it seems. Maybe you're just working for Billy's Interwebs-r-Us and keeping Sally's Nail Shop going strong, but the last time I checked, the "ent" in CentOS stood for "enterprise". Here are a few tips: 1. Fuck KVM. 2. Stick with Xen because there is quite a lot of time until 5 is EOL'd and if you haven't noticed, it's actually a mature technology. 3. Figure out what you are going to do with Sally's Nail Shop in the meantime. If you have time to fuck everything up in your environment with KVM, you can probably save 20% or more optimizing your environment and even more with proper capital investments and training. 4. Figure out how either a) non-critical/enterprise services would ever be served by KVM's features or lack thereof, or b) you are going to CYA when you can't guarantee an SLA. 5. Spend more time on KVM dev lists instead of posting here and annoying others with your butthurt KVM-won't-work posts, as it's not even supported upstream. -- Christopher G. Stach II http://ldsys.net/~cgs/

6 6

What kernel params to use with KVM hosts??
by carlopmart 18 Mar '10

18 Mar '10

Hi all, I order to reach maximum performance on my centos kvm hosts I have use these params: - On /etc/grub.conf: kernel /vmlinuz-2.6.18-164.11.1.el5 ro root=LABEL=/ elevator=deadline quiet - On sysctl.conf # Special network params net.core.rmem_default = 8388608 net.core.wmem_default = 8388608 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 1048576 4194304 16777216 net.ipv4.tcp_wmem = 1048576 4194304 16777216 # Disable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 # Virtual machines special kernel params vm.swappiness = 0 Do I need to configure something more?? Any tips?? Thanks. -- CL Martinez carlopmart {at} gmail {d0t} com

2 3

get xen name from guest
by Ryan Pugatch 17 Mar '10

17 Mar '10

Hello, Is it possible to get the name that was set in a VM's xen config file from the guest machine? I'm interested in having my kickstart launch a script that can configure hostname, but to do that it would need to get the name set in the config. Thanks, Ryan

5 7

reboot guest on panic
by Norman Gaywood 16 Mar '10

16 Mar '10

I have a guest that keeps crashing and want to automatically reboot it when it crashes. See: xen PV guest kernel 2.6.32 processes lock up in D state https://bugzilla.redhat.com/show_bug.cgi?id=550724 if you want to look at the details on the crashing. Anyway, I boot the guest with the kernel command line parameter: hung_task_panic=1 I have kernel.panic = 15 in the guest /etc/sysctl.conf In the guest config file in dom0 I have: on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" The guest manages to panic when it detects the hung tasks and I get this on the guest console: Kernel panic - not syncing: hung_task: blocked tasks Rebooting in 15 seconds.. However, it never restarts. It just hangs around until I do a xm destroy <guest> xm create <guest> Have I missed something? -- Norman Gaywood, Computer Systems Officer University of New England, Armidale, NSW 2351, Australia ngaywood(a)une.edu.au Phone: +61 (0)2 6773 3337 http://mcs.une.edu.au/~norm Fax: +61 (0)2 6773 3312 Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html

5 13

Re: [CentOS-virt] reboot guest on panic
by Christopher G. Stach II 15 Mar '10

15 Mar '10

----- "Norman Gaywood" <ngaywood(a)une.edu.au> wrote: > Well yes, seems I've hit trouble this time. Was expecting some trouble > with the desktop apps and was happy to deal with that. I was not > expecting significant kernel trouble however. It's Fedora. It's *always* broken. > Rebuilding a modern desktop distro with an older kernel is a lot of > work. It's not that bad. The only things you really need to match up are glibc and the kernel. Beyond that, most things are reasonably source compatible with a slightly older glibc. I don't think you would need to go that far, though. What do you need in Fedora that you can't get in the almost equally substandard desktop experience from CentOS or any other enterprise distribution? (I use CentOS as my primary desktop.) > That's what distributions are for right? Gentoo? :) -- Christopher G. Stach II http://ldsys.net/~cgs/

1 0

Re: [CentOS-virt] gpl pv drivers and Centos 5.4
by F M 14 Mar '10

14 Mar '10

TX to you both of you for the info. I will test these drivers tomorrow. Regards, F Regards,----- Original Message ----- From: "Pasi Kärkkäinen" <pasik(a)iki.fi> To: "Discussion about the virtualization on CentOS" <centos-virt(a)centos.org> Sent: Sunday, March 14, 2010 9:12:42 AM Subject: Re: [CentOS-virt] gpl pv drivers and Centos 5.4 On Sat, Mar 13, 2010 at 07:17:58AM -0500, F M wrote: > Hello, > I am using Windows XP/2003 and 2008 under Xen. What is the correct version of gpl pv drivers for Windows under Centos 5.4 Dom0 ? rpm -q xen said 3.0.3.x but xen info said 3.1.x > Xen hypervisor in EL5 is version 3.1.2 + a lot of patches from Redhat and from newer Xen versions. Xen tools in EL5 (xm/xend etc) are based on version 3.0.3 + patches. -- Pasi _______________________________________________ CentOS-virt mailing list CentOS-virt(a)centos.org http://lists.centos.org/mailman/listinfo/centos-virt

1 0

gpl pv drivers and Centos 5.4
by F M 14 Mar '10

14 Mar '10

Hello, I am using Windows XP/2003 and 2008 under Xen. What is the correct version of gpl pv drivers for Windows under Centos 5.4 Dom0 ? rpm -q xen said 3.0.3.x but xen info said 3.1.x Regards,

3 2

Re: [CentOS-virt] reboot guest on panic
by Christopher G. Stach II 14 Mar '10

14 Mar '10

----- "Norman Gaywood" <ngaywood(a)une.edu.au> wrote: > > If your guest's state is corrupted, you can't rely on its behavior. > For that reason, you should set up a watchdog and only rely on the > panic behavior as a preliminary measure. > > I figured that I had setup a watchdog with the hung_task_panic=1 guest > parameter. Is there another way of setting up a watchdog in this case? Internal watchdogs are seldom as effective as you need them to be. I meant an external one. There was talk in the past about a Xen watchdog driver, but I don't know where that ended up. (I use existing code that works for me, so I haven't kept up.) There are various ways to set one up. You can do it on your own with a couple of scripts and xenstore, if you're so inclined. If not, there are a lot of existing examples and projects that will handle it for you. This one that uses QEMU and unfortunately only KVM is decent and should give you enough ideas on how to implement one: http://rwmj.wordpress.com/2010/03/03/what-is-a-watchdog/ -- Christopher G. Stach II http://ldsys.net/~cgs/

1 0

unable to get domain status from libvirt & KVM
by Tom Georgoulias 12 Mar '10

12 Mar '10

I have a python script that monitors the VMs on physical host servers running Xen, but the script doesn't work properly on a server I just built with KVM. The script runs as a non-root user (same on all servers) and simply gathers some details on the status and names of the domains running on the host. Both Xen and KVM servers are running the same version of libvirt (libvirt-0.6.3-20.1.el5_4) and have the same, default /etc/libvirt/libvirtd.conf config file. To troubleshoot, I've been running python interactively. Here's how my Xen servers behave: $ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) >>> domains = conn.listDomainsID() >>> print domains [0, 3, 15, 16, 21, 24, 26, 30, 32, 36, 38, 41, 43, 45, 47, 49, 51, 55, 63, 67] >>> When I try the same thing on the KVM server: $ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) 14:33:07.303: error : No vport operation path found for host0 14:33:07.320: error : No vport operation path found for host4 14:33:07.325: error : No vport operation path found for host3 14:33:07.367: error : No vport operation path found for host1 14:33:07.368: error : No vport operation path found for host2 >>> domains = conn.listDomainsID() >>> print domains [] >>> (The vport stuff is weird, but I found this posting that suggests its harmless (http://www.mail-archive.com/libvir-list@redhat.com/msg17477.html) so I'm ignoring it.) However, when logged in as root on the KVM server, it works just like my Xen servers: # python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) >>> domains = conn.listDomainsID() >>> print domains [1] Again, on the KVM server, plain old "virsh list" with the debug level set to "2" $ export LIBVIRT_DEBUG=2 $ virsh list 14:21:06.532: error : No vport operation path found for host0 14:21:06.550: error : No vport operation path found for host4 14:21:06.555: error : No vport operation path found for host3 14:21:06.598: error : No vport operation path found for host1 14:21:06.599: error : No vport operation path found for host2 14:21:06.615: info : No security driver available Id Name State ---------------------------------- Permissions in /var/run/libvirt: # ls -ld /var/run/libvirt/* srwx------ 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock srwxrwxrwx 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock-ro drwxr-xr-x 2 root root 4096 Jan 21 14:38 /var/run/libvirt/network drwxr-xr-x 2 root root 4096 Oct 20 18:50 /var/run/libvirt/qemu Can someone provide some tips on what else I can check, if this might be a bug, or point out any mistakes that I might've made? Any help is appreciated. Thanks, Tom

3 11

Logrotate/cron and major I/O contention with KVM.
by Steven Ellis 11 Mar '10

11 Mar '10

Is anyone else having major I/O peaks due to logrotate or other jobs running simultaneously across multiple guests. I have one KVM server running Centos 5.4 with local disk that is seriously suffering as most of the guests rotate their syslog at the same time. Looking at the KVM server I'm seeing 11:00:01 PM CPU %user %nice %system %iowait %steal %idle 03:40:01 AM all 0.07 0.00 2.74 0.93 0.00 96.26 03:50:01 AM all 0.07 0.00 1.17 1.18 0.00 97.58 04:00:01 AM all 0.08 0.00 1.51 0.82 0.00 97.59 04:10:02 AM all 0.53 0.03 15.31 51.61 0.00 32.53 04:20:01 AM all 0.28 0.12 4.12 22.21 0.00 73.27 04:30:01 AM all 0.07 0.00 0.80 1.21 0.00 97.92 04:40:01 AM all 0.07 0.00 2.60 1.81 0.00 95.52 04:50:01 AM all 0.08 0.00 0.79 1.44 0.00 97.69 On one of the guests running Centos 4.6 the impact is so bad I get DMA timeout errors in the syslog, and occasional kernel panics. Mar 11 04:05:04 localhost kernel: hda: dma_timer_expiry: dma status == 0x21 Mar 11 04:05:14 localhost kernel: hda: DMA timeout error Mar 11 04:05:14 localhost kernel: hda: dma timeout error: status=0x50 { DriveReady SeekComplete } Mar 11 04:05:14 localhost kernel: Mar 11 04:05:14 localhost kernel: ide: failed opcode was: unknown Mar 11 04:05:59 localhost kernel: hda: dma_timer_expiry: dma status == 0x21 Mar 11 04:06:14 localhost kernel: hda: DMA timeout error Mar 11 04:06:14 localhost kernel: hda: dma timeout error: status=0x50 { DriveReady SeekComplete } One reference I've found is at * http://lonesysadmin.net/linux-virtual-machine-tuning-guide/ This suggests avoiding running scheduled jobs simultaneously across guests, and suggests using a random sleep. Does anyone else have suggestions on reducing the impact of cron/logrotate. -- *Steven Ellis - Director of Worldwide Engineering,* *Bulletin.Net Inc* - http://www.bulletin.net/

4 3

Re: [CentOS-virt] Introducing ConVirt 2.0
by Christopher G. Stach II 10 Mar '10

10 Mar '10

----- "Pasi Kärkkäinen" <pasik(a)iki.fi> wrote: > On Sat, Mar 06, 2010 at 09:04:20AM -0500, Kanwar Ranbir Sandhu wrote: > > Why would one use ConVirt instead of the management tools included in > > RHEL and/or CentOS? What's the difference? > > RHEL/CentOS doesn't provide web-based management.. or even easy > multi-host / cluster management of virtualization nodes. > > -- Pasi Are there any *good* reasons? (Since I really hate commercials, I feel compelled to present my contrarian viewpoint.) ConVirt addresses a pretty small portion of the virtualization landscape, and it consists of only a few significant parts: 1. Do what other free and open tools already do. 2. Slap a web interface on it! 3. Spam lists. 4. Rope in suckers. The suggestion that a web interface is a value add to an infrastructure issue is at least insulting. You could attempt to slap a web interface on a fuel injection system (or maybe at least give access to the magic a la MegaSquirt), but a bunch of assholes are still going to blow something up. It's not going to give any admin worth his or her salt a boner because it's not readily scriptable and it amounts to candy for retards. Secondly, everything else that it does is already there. If you can't do it, you shouldn't be touching the machines. The tool may or may not address some vanilla installations (if there ever was one), but if you need something like that, you are probably better off with EC2 or at least letting someone else handle it. -- Christopher G. Stach II http://ldsys.net/~cgs/

5 5

Re: [CentOS-virt] [fedora-virt] Thoughts on storage infrastructure for small scale HA virtual machine deployments
by Dennis J. 07 Mar '10

07 Mar '10

On 03/02/2010 04:51 AM, Ask Bjørn Hansen wrote: > > On Mar 1, 2010, at 18:56, Dennis J. wrote: > >> The question that bugs me is how I can get enough bandwidth between the >> hosts and the storage to provide the VMs with reasonable I/O performance. >> If all the 40 VMs start copying files at the same time that would mean that >> the bandwidth share for each VM would be tiny. > > It really depends on the specific workloads. In my experience it's generally the number of IOs per second rather than the bandwidth that's the limiting factor. > > We have a bunch of 4-disk boxes with md raid10 and we generally run out of disk IO before we run out of memory (~24-48GB) or CPU (dual quad core 2.26GHz or some such). That's very similar to what we are experiencing. The primary Problem for me is how to deal with the bottleneck of a shared storage setup. The most simple setup is a 2-system criss-cross setup where the two hosts also serve as halves for a DRBD cluster. The advantage of this approach is that it's a cheap solution, that only a part of the storage-traffic has to go over the network between the machines and that the network only hast to handle the sorage-traffic of the VMs of those two machines. The disadvantage of that approach is that you have to keep 50% of potential server capacity free in case of a failure of the twin node. That's quite a lot of wasted capacity. To reduce that problem you can increase the number of hosts to let say for an example four which would reduce the spare capacity needed to 33% on each system but then you really need to separate the storage from the hosts an now you have a bottleneck on the storage end. Increase the number of hosts to 8 and you get even less wasted capacity but also increase the pressure on the storage bottleneck a lot. Since I'm new to the whole SAN aspect I'm currently just looking at all the options that are out there and basically wonder how the big boys are handling this who have hundreds if not thousand of VMs running and need to be able to deal with physical failures too. That is why I find the sheepdog project so interesting because it seems to address this particular problem in a way that would provide almost linear scalability without actually using a SAN at all (well, at least not in the traditional sense of the word). Regards, Dennis

1 0

Thoughts on storage infrastructure for small scale HA virtual machine deployments
by Dennis J. 07 Mar '10

07 Mar '10

Hi, up until now I've always deployed VMs with their storage located directly on the host system but as the number of VMs grows and the hardware becomes more powerful and can handle more virtual machines I'm concerned about a failure of the host taking down too many VMs in one go. As a result I'm now looking at moving to an infrastructure that uses shared storage instead so I can live-migrate VMs or restart them quickly on another host if the one they are running on dies. The problem is that I'm not sure how to go about this bandwidth-wise. What I'm aiming for as a starting point is a 3-4 host cluster with about 10 VMs on each host and a 2 system DRBD based cluster as a redundant storage backend. The question that bugs me is how I can get enough bandwidth between the hosts and the storage to provide the VMs with reasonable I/O performance. If all the 40 VMs start copying files at the same time that would mean that the bandwidth share for each VM would be tiny. Granted this is a worst case scenario and that's why I want to ask if someone in here has experience with such a setup, can give recommendations or comment on alternative setups? Would I maybe get away with 4 bonded gbit ethernet ports? Would I require fiber channel or 10gbit infrastructure? Regards, Dennis PS: The sheepdog project (http://www.osrg.net/sheepdog/) looks interesting in that regard but apparently still is far from production-ready.

6 6

Fedora 12 2.6.31.5-127.fc12 domU on CentOS 5.4 2.6.18-164.6.1.el5xen fails to boot
by Charles J Gruener 06 Mar '10

06 Mar '10

Doing a kickstart install of Fedora 12 results in a non-bootable image. >From a virt-maanger instance, the error produced when trying to run the domU is: Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/engine.py", line 498, in run_domain vm.startup() File "/usr/share/virt-manager/virtManager/domain.py", line 573, in startup self.vm.create() File "/usr/lib64/python2.4/site-packages/libvirt.py", line 287, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err "Error creating domain: Boot loader didn't return any data!") If you drop to the command line to see what pygrub gets back, you get: # pygrub /var/lib/xen/images/fedora12.img Traceback (most recent call last): File "/usr/bin/pygrub", line 677, in ? fs = fsimage.open(file, get_fs_offset(file)) IOError: [Errno 95] Operation not supported The kickstart worked perfectly in Fedora 11 on a CentOS 5.4 dom0. I've tried investigating all sorts of grub options and even tried copying kernel and initramfs files out from the domU to dom0 and booting from those with no luck. The only thing that does work is if I install Fedora 11 and then perform an upgrade to Fedora 12. I really don't want to have to deploy machines this way. Is anyone else experiencing this issue? I can provide any further information requested. Basically, to repeat my error, install CentOS 5.4 with Virtualization, then try to install a Fedora 12 domU. The installation finishes without error but you can't boot. The systems are all x86_64. Thanks. Charles

9 31

Re: [CentOS-virt] [CentOS] Very unresponsive, sometimes stalling domU (5.4, x86_64)
by Pasi Kärkkäinen 02 Mar '10

02 Mar '10

On Tue, Mar 02, 2010 at 09:30:50AM +0100, Timo Schoeler wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi list, > > please forgive cross posting, but I cannot specify the problem enough to > say whether list it fits perfectly, so I'll ask on both. > > I have some machines based with following specs (see at the end of the > email). > > They run CentOS 5.4 x86_64 with the latest patches applied, Xen-enabled > and should host one or more domUs. I put the domUs' storage on LVM, as I > learnt ages ago (what never caused any problems) and is way faster than > using file-based 'images'. > > However, there's something special about these machines: They have the > new WD EARS series drives, which use 4K sector sizes. So, I booted a > rescue system and used fdisk to start at sector 64 instead of 63 (long > story made short: Due to overhead causing the drive to do much more, > inefficient writes when starting at sector 63, the performance > collapses; with 'normal' geometry (sector 63), the drive achieves about > 25MiByte/sec writes, with starting at sector 64 partition, it achieves > almost 100MiByte/sec writes): > > [root@server2 ~]# fdisk -ul /dev/sda > > Disk /dev/sda: 1000.2 GB, 1000204886016 bytes > 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors > Units = sectors of 1 * 512 = 512 bytes > > Device Boot Start End Blocks Id System > /dev/sda1 * 64 2097223 1048580 fd Linux raid > autodetect > Partition 1 does not end on cylinder boundary. > /dev/sda2 2097224 18876487 8389632 82 Linux swap / Solaris > /dev/sda3 18876488 1953525167 967324340 fd Linux raid > autodetect > > On top of those (two per machine) WD EARS HDs there's ``md'' providing > two RAID1, /boot and LVM, as well as swap per HD (i.e. non-RAIDed). LVM > provides the / partition as well as LVs for Xen domUs. > > I have about 60 machines running that style and never had any problems. > They run like a charm. On these machines, however, domUs are *very* > slow, have a steady (!) load of about two -- 50% stating in 'wait' -- > and all operations take ages, e.g. a ``yum update'' with the recently > released updates. > > Now, can that be due to 4K issues I didn't see, nestet now in LVM? > > Help is very appreciated. > Maybe the default LVM alignment is wrong for these drives.. did you check/verify that? See: http://thunk.org/tytso/blog/2009/02/20/aligning-filesystems-to-an-ssds-eras… Especially the "--metadatasize" option. -- Pasi

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

virt March 2010