Hi all.
I am currently building a small test cloud based on Eucalyptus 2.0.3 and CentOS 5.8 x64. I have a choice which hypervisor to use: KVM or XEN. KVM is the default in CentOS 6 but I have read also many good things (for example PV guest machines, isolation between Dom0 and DomU) about XEN.
Key factors from my opint of view are: - stability (which one runs more smoothly on CentOS?) - performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or without pv drivers)) - security
Could you share your experience in these areas?
Best regards, Rafal Radecki.
On 04/20/2012 01:59 PM, Rafał Radecki wrote:
I am currently building a small test cloud base..
...
Could you share your experience in these areas?
try the centos-virt list ? Lots of people there ( including people who write a lot of the code behind some of these things! )
On 4/20/2012 8:59 AM, Rafał Radecki wrote:
Hi all.
I am currently building a small test cloud based on Eucalyptus 2.0.3 and CentOS 5.8 x64. I have a choice which hypervisor to use: KVM or XEN. KVM is the default in CentOS 6 but I have read also many good things (for example PV guest machines, isolation between Dom0 and DomU) about XEN.
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
- performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or
without pv drivers))
- security
Could you share your experience in these areas?
Best regards, Rafal Radecki. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Xen all the way. That's just my opinion though.
Why?
2012/4/20 Jonathan Vomacka juvix88@gmail.com
On 4/20/2012 8:59 AM, Rafał Radecki wrote:
Hi all.
I am currently building a small test cloud based on Eucalyptus 2.0.3 and CentOS 5.8 x64. I have a choice which hypervisor to use: KVM or XEN. KVM is the default in CentOS 6 but I have read also many good things (for example PV guest machines, isolation between Dom0 and DomU) about XEN.
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
- performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or
without pv drivers))
- security
Could you share your experience in these areas?
Best regards, Rafal Radecki. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Xen all the way. That's just my opinion though. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi,
KVM if used as it is will show very poor performance on CentOS5. To achieve better results you need to update kernel to at least 2.6.32 and compile newer versions of libvirt and qemu. On CentOS6 all is fine with KVM right out of the box.
Never used XEN so cannot compare.
Dmitry Cherkasov
2012/4/20 Rafał Radecki radecki.rafal@gmail.com:
Hi all.
I am currently building a small test cloud based on Eucalyptus 2.0.3 and CentOS 5.8 x64. I have a choice which hypervisor to use: KVM or XEN. KVM is the default in CentOS 6 but I have read also many good things (for example PV guest machines, isolation between Dom0 and DomU) about XEN.
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
- performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or
without pv drivers))
- security
Could you share your experience in these areas?
Best regards, Rafal Radecki. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 04/20/2012 04:23 PM, Dmitry Cherkasov wrote:
On CentOS6 all is fine with KVM right out of the box.
Never used XEN so cannot compare.
Same here. I would add some LXC pins for quick ehanced chroot, depending on the use case.
I think the OP should provide more details: What is benchmarked (Network? HD?...)
Hi,
On Mon, Apr 23, 2012 at 6:29 PM, Mihamina Rakotomandimby mihamina@rktmb.org wrote:
I would add some LXC pins for quick ehanced chroot, depending on the use case.
LXC sounds interesting: are there any yum repositries / RPMs / tutorials for CentOS available?
I've been quite happy with Xen under CentOS5. For CentOS6 the situation is a bit more problematic, as RH switched to KVM and left Xen behind.
Best, Peter
On Mon, 23 Apr 2012, Peter Peltonen wrote:
I've been quite happy with Xen under CentOS5. For CentOS6 the situation is a bit more problematic, as RH switched to KVM and left Xen behind.
I used Xen for about four or five years before switching to KVM. I like KVM better in every way, and for my fork-heavy workloads, the performance is a lot better than Xen. It is also much easier to use and is in my experience more stable.
Steve
On Apr 23, 2012, at 1:02 PM, Steve Thompson wrote:
On Mon, 23 Apr 2012, Peter Peltonen wrote:
I've been quite happy with Xen under CentOS5. For CentOS6 the situation is a bit more problematic, as RH switched to KVM and left Xen behind.
I used Xen for about four or five years before switching to KVM. I like KVM better in every way, and for my fork-heavy workloads, the performance is a lot better than Xen. It is also much easier to use and is in my experience more stable.
Steve
I also prefer KVM over Xen, mainly I don;t have to do anything special when maintaining the env.
But I haven't notice an improvement over Xen.
I really like the fact that the guest OS has a stock kernel, etc..
- aurf
Hi,
On Mon, Apr 23, 2012 at 10:54 PM, aurfalien aurfalien@gmail.com wrote:
I also prefer KVM over Xen, mainly I don;t have to do anything special when maintaining the env.
But I haven't notice an improvement over Xen.
I really like the fact that the guest OS has a stock kernel, etc..
I do not quite see how Xen requires one to do something special for maintenance? With pygrub you can use the stock kernel with your Xen domUs just fine. I have not seen any issues with stability either, but then again I am running mostly just web and mail servers without really high traffic.
But if KVM would offer improvements for performance over Xen, I should perhaps try it out, as sometimes when doing backups and other things that require a lot of disk I/O a better performance could be wished for...
Regards, Peter
On Apr 23, 2012, at 4:01 PM, Peter Peltonen wrote:
Hi,
On Mon, Apr 23, 2012 at 10:54 PM, aurfalien aurfalien@gmail.com wrote:
I also prefer KVM over Xen, mainly I don;t have to do anything special when maintaining the env.
But I haven't notice an improvement over Xen.
I really like the fact that the guest OS has a stock kernel, etc..
I do not quite see how Xen requires one to do something special for maintenance?
Regarding Centos 6 there are some extra things to install.
Even when I deviated from the included version of Xen in 5, I had to pay special attention.
As for stock kernels, you mean HVMs right?
I was speaking more about PVMs which is faster and more flexible then HVMs.
I never had any issues with Xen other then VGA and USB pass through.
But Xen ran well for me.
As for convenience, I'm into KVM now, very cool features with pass throughs, graphics etc...
- aurf
On Mon, Apr 23, 2012 at 11:11 PM, aurfalien aurfalien@gmail.com wrote:
As for stock kernels, you mean HVMs right?
I was speaking more about PVMs which is faster and more flexible then HVMs.
No, with pygrub you can run a stock kernel on a PVM domU: http://wiki.xensource.com/xenwiki/PyGrub
I never had any issues with Xen other then VGA and USB pass through.
But Xen ran well for me.
As for convenience, I'm into KVM now, very cool features with pass throughs, graphics etc...
USB pass through has worked fine for me under Xen. Never had the need for graphics for my servers. For desktops I've been happy with Parallels and VirtualBox.
But from comments it sounds like KVM is maturing and I should perhaps give it a try.
Regards, Peter
On 04/23/2012 10:11 PM, aurfalien wrote:
On Apr 23, 2012, at 4:01 PM, Peter Peltonen wrote:
Hi,
On Mon, Apr 23, 2012 at 10:54 PM, aurfalien aurfalien@gmail.com wrote:
I also prefer KVM over Xen, mainly I don;t have to do anything special when maintaining the env.
But I haven't notice an improvement over Xen.
I really like the fact that the guest OS has a stock kernel, etc..
I do not quite see how Xen requires one to do something special for maintenance?
Regarding Centos 6 there are some extra things to install.
Even when I deviated from the included version of Xen in 5, I had to pay special attention.
As for stock kernels, you mean HVMs right?
I was speaking more about PVMs which is faster and more flexible then HVMs.
The PVM/HVM distinction isn't really that relevant any more on modern hardware and modern hypervisors since most of the overhead is eliminated with hardware features (Nested Page Tables, etc.) and special guest drivers.
Regards, Dennis
On 04/23/12 5:12 PM, Dennis Jacobfeuerborn wrote:
The PVM/HVM distinction isn't really that relevant any more on modern hardware and modern hypervisors since most of the overhead is eliminated with hardware features (Nested Page Tables, etc.) and special guest drivers.
"special guest drivers" is pretty much what paravirtualization is about.
On 04/24/2012 03:08 AM, John R Pierce wrote:
On 04/23/12 5:12 PM, Dennis Jacobfeuerborn wrote:
The PVM/HVM distinction isn't really that relevant any more on modern hardware and modern hypervisors since most of the overhead is eliminated with hardware features (Nested Page Tables, etc.) and special guest drivers.
"special guest drivers" is pretty much what paravirtualization is about.
Exactly, but only since CPU got hardware extensions for virtualization. Before that the CPU could also be paravirtualized and that made a significant difference in performance.
With that advantage gone though the old distinction between a "PVM guest" and "HVM guest" doesn't really matter that much any more (virt-manager asks you which of the two you want to install for example). Now you only have a "guest" that may or may not run certain paravirtualized drivers.
Regards, Dennis
On Apr 23, 2012, at 4:01 PM, Peter Peltonen wrote:
Hi,
On Mon, Apr 23, 2012 at 10:54 PM, aurfalien aurfalien@gmail.com wrote:
I also prefer KVM over Xen, mainly I don;t have to do anything special when maintaining the env.
But I haven't notice an improvement over Xen.
I really like the fact that the guest OS has a stock kernel, etc..
I do not quite see how Xen requires one to do something special for maintenance? With pygrub you can use the stock kernel with your Xen domUs just fine. I have not seen any issues with stability either, but then again I am running mostly just web and mail servers without really high traffic.
But if KVM would offer improvements for performance over Xen, I should perhaps try it out, as sometimes when doing backups and other things that require a lot of disk I/O a better performance could be wished for...
Forgot to add there there are some cool options for increasing disk IO.
Load up KVM and check it out. I'm pretty happy with it.
- aurf
On Mon, Apr 23, 2012 at 11:01:12PM +0300, Peter Peltonen wrote:
Hi,
On Mon, Apr 23, 2012 at 10:54 PM, aurfalien aurfalien@gmail.com wrote:
I also prefer KVM over Xen, mainly I don;t have to do anything special when maintaining the env.
But I haven't notice an improvement over Xen.
I really like the fact that the guest OS has a stock kernel, etc..
I do not quite see how Xen requires one to do something special for maintenance? With pygrub you can use the stock kernel with your Xen domUs just fine. I have not seen any issues with stability either, but then again I am running mostly just web and mail servers without really high traffic.
But if KVM would offer improvements for performance over Xen, I should perhaps try it out, as sometimes when doing backups and other things that require a lot of disk I/O a better performance could be wished for...
Disk performance is usually mainly limited by the number of physical disk spindles, and the raid level, and not so much about virtualization.
Anyway some Xen PV vs. Xen PVHVM vs. KVM benchmarks from XenSummit 2011: http://xen.org/files/xensummit_santaclara11/aug3/6_StefanoS_PVHVM.pdf
-- Pasi
On Mon, Apr 23, 2012 at 01:02:03PM -0400, Steve Thompson wrote:
On Mon, 23 Apr 2012, Peter Peltonen wrote:
I've been quite happy with Xen under CentOS5. For CentOS6 the situation is a bit more problematic, as RH switched to KVM and left Xen behind.
I used Xen for about four or five years before switching to KVM. I like KVM better in every way, and for my fork-heavy workloads, the performance is a lot better than Xen. It is also much easier to use and is in my experience more stable.
with "fork performance" I assume you're comparing Xen PV to KVM ?
Yes, PV has disadvantage (per design) for that workload, since the hypervisor needs to check and verify each new process page table, and that has some performance hit.
For good "fork performance" you can use Xen HVM VMs, which will perform well for that workload, and won't have the mentioned performance hit.
And of course with Xen HVM VMs you should use the Xen PVHVM drivers so the disk/net IO paths are optimized and bypassing all the emulation.
CentOS5 and CentOS6 do have Xen PVHVM drivers.
-- Pasi
On Sun, 6 May 2012, Pasi Kärkkäinen wrote:
with "fork performance" I assume you're comparing Xen PV to KVM ? Yes, PV has disadvantage (per design) for that workload, since the hypervisor needs to check and verify each new process page table, and that has some performance hit. For good "fork performance" you can use Xen HVM VMs, which will perform well for that workload, and won't have the mentioned performance hit.
I used both PV and HVM VMs. I don't have the details to hand at the moment, but KVM was superior to both. PV drivers where applicable. I have been running KVM for about 15 months now, with 30 VM's on one host and 38 VM's on another. It has been solid; no problems, but unfortunately I had problems with Xen.
Steve
On Sat, May 12, 2012 at 02:38:34PM -0400, Steve Thompson wrote:
On Sun, 6 May 2012, Pasi Kärkkäinen wrote:
with "fork performance" I assume you're comparing Xen PV to KVM ? Yes, PV has disadvantage (per design) for that workload, since the hypervisor needs to check and verify each new process page table, and that has some performance hit. For good "fork performance" you can use Xen HVM VMs, which will perform well for that workload, and won't have the mentioned performance hit.
I used both PV and HVM VMs. I don't have the details to hand at the moment, but KVM was superior to both. PV drivers where applicable. I have been running KVM for about 15 months now, with 30 VM's on one host and 38 VM's on another. It has been solid; no problems, but unfortunately I had problems with Xen.
And Xen has been rock solid on my production systems. So it depends :)
-- Pasi
On 04/23/2012 06:44 PM, Peter Peltonen wrote:
I would add some LXC pins for quick ehanced chroot, depending on the use case
LXC sounds interesting: are there any yum repositries / RPMs / tutorials for CentOS available?
You dont need rpms: the libvirt directly use the LXC API. A tutorial: http://goo.gl/kQOxm
All you need is to - setup the XML - define a VM from it - start the VM
No more, for the "basic" example. Really fun. Of course, when you need a custom environment, you'll need to read further: but it's still fun :-)
On 04/24/2012 11:58 AM, Mihamina Rakotomandimby wrote:
On 04/23/2012 06:44 PM, Peter Peltonen wrote:
I would add some LXC pins for quick ehanced chroot, depending on the use case
LXC sounds interesting: are there any yum repositries / RPMs / tutorials for CentOS available?
You dont need rpms: the libvirt directly use the LXC API. A tutorial: http://goo.gl/kQOxm
All you need is to
- setup the XML
- define a VM from it
- start the VM
No more, for the "basic" example. Really fun. Of course, when you need a custom environment, you'll need to read further: but it's still fun :-)
Remember that currently LXC isn't very secure (as mentioned in the tutorial link) so you probably don't want to use it for important stuff in a production environment.
Regards, Dennis
On 04/24/2012 10:58 AM, Mihamina Rakotomandimby wrote:
LXC sounds interesting: are there any yum repositries / RPMs / tutorials for CentOS available?
You dont need rpms: the libvirt directly use the LXC API. A tutorial: http://goo.gl/kQOxm
there are some limitations with libvirt/lxc at the moment - eg. needing to build the root images outside of libvirt.
A late reply, but hopefully a useful set of feedback for the archives:
On 04/20/2012 05:59 AM, Rafał Radecki wrote:
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
I found that xenconsoled could frequently crash in Xen dom0, and that guests would be unable to reboot until it was fixed. I also found that paravirt CentOS domUs would not boot if they were updated before the dom0. In short, Xen paravirt was very fragile and troublesome. I never tested Xen with hardware virtualization.
I have had no such problems with KVM. In my experience KVM is much more stable than Xen paravirtualization. Xen HVM probably would suffer at least some of the same problems.
- performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or
without pv drivers))
PV drivers will make some difference, but the biggest performance difference you'll see is probably the difference between file-backed VMs and LVM-backed VMs. File-backed VMs are extremely slow. Whichever system you choose, use LVMs as the backing for your guests.
- security
There have been bugs that allow guests to escalate privileges and access host resources, but they're relatively few. I don't think there's a significant difference between the two in this area.
Overall I advise the use of KVM. It should be more stable, and has the advantage of Red Hat support.
On 05/12/2012 12:46 AM, Gordon Messmer wrote:
A late reply, but hopefully a useful set of feedback for the archives:
Well let me share my experience as well.
On 04/20/2012 05:59 AM, Rafał Radecki wrote:
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
I found that xenconsoled could frequently crash in Xen dom0, and that guests would be unable to reboot until it was fixed. I also found that paravirt CentOS domUs would not boot if they were updated before the dom0. In short, Xen paravirt was very fragile and troublesome. I never tested Xen with hardware virtualization.
I have had no such problems with KVM. In my experience KVM is much more stable than Xen paravirtualization. Xen HVM probably would suffer at least some of the same problems.
I have some machine that were very unstable under load (max uptime some weeks, then a crash). They were running CentOS5 with XEN kernel. First I thought it was hardware related, but once a non-Xen kernel was loaded and I migrated the VMs to KVM the machines are rock solid. I must say I still have two machines running Xen and they have no problem the last year. So it's probably also related to the specified hardware configuration.
Theo
On 05/12/2012 12:46 AM, Gordon Messmer wrote:
A late reply, but hopefully a useful set of feedback for the archives:
On 04/20/2012 05:59 AM, Rafał Radecki wrote:
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
I found that xenconsoled could frequently crash in Xen dom0, and that guests would be unable to reboot until it was fixed. I also found that paravirt CentOS domUs would not boot if they were updated before the dom0. In short, Xen paravirt was very fragile and troublesome. I never tested Xen with hardware virtualization.
I have had no such problems with KVM. In my experience KVM is much more stable than Xen paravirtualization. Xen HVM probably would suffer at least some of the same problems.
- performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or
without pv drivers))
PV drivers will make some difference, but the biggest performance difference you'll see is probably the difference between file-backed VMs and LVM-backed VMs. File-backed VMs are extremely slow. Whichever system you choose, use LVMs as the backing for your guests.
- security
There have been bugs that allow guests to escalate privileges and access host resources, but they're relatively few. I don't think there's a significant difference between the two in this area.
sVirt mitigates this danger somewhat on the host side so even if you run into such an issue it is very hard to utilize such an export.
Regards, Dennis
On Fri, May 11, 2012 at 03:46:43PM -0700, Gordon Messmer wrote:
A late reply, but hopefully a useful set of feedback for the archives:
On 04/20/2012 05:59 AM, Rafał Radecki wrote:
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
I found that xenconsoled could frequently crash in Xen dom0, and that guests would be unable to reboot until it was fixed. I also found that paravirt CentOS domUs would not boot if they were updated before the dom0. In short, Xen paravirt was very fragile and troublesome. I never tested Xen with hardware virtualization.
This particular problem was fixed some time ago, it hasn't happened to my (many) dom0s in more than a year.
The RHEL5 Xen dom0 was garbage until 5.3 or so. To the point where I'd compile my own and deal with the pain of using a non-rhel kernel with a rehl userland.
Stability has improved vastly.
- performance (XEN PV/HVM(with or without pv drivers) vs KVM HVM(with or
without pv drivers))
PV drivers will make some difference, but the biggest performance difference you'll see is probably the difference between file-backed VMs and LVM-backed VMs. File-backed VMs are extremely slow. Whichever system you choose, use LVMs as the backing for your guests.
My experience has been that using qemu for disk has something of a multiplier effect; e.g. it makes slow spinning disk noticably slower. The paravirtualized drivers help immensely in that regard.
(how are the paravirt drivers in KVM these days? I have a server full of kvm guests running some ancient version of ubuntu I will be moving to RHEL6 shortly.)
On 05/16/2012 02:47 PM, Luke S. Crawford wrote:
(how are the paravirt drivers in KVM these days? I have a server full of kvm guests running some ancient version of ubuntu I will be moving to RHEL6 shortly.)
Since RHEL guests have the virtio block drivers built-in, I never get around to benchmarking them against non-virtio block devices.
On Fri, May 11, 2012 at 03:46:43PM -0700, Gordon Messmer wrote:
A late reply, but hopefully a useful set of feedback for the archives:
On 04/20/2012 05:59 AM, Rafa?? Radecki wrote:
Key factors from my opint of view are:
- stability (which one runs more smoothly on CentOS?)
I found that xenconsoled could frequently crash in Xen dom0, and that guests would be unable to reboot until it was fixed. I also found that paravirt CentOS domUs would not boot if they were updated before the dom0.
This was a problem in RHEL5/CentOS5 Xen. It was fixed in upstream Xen years ago. I think it was fixed finally in RHEL5/CentOS5 Xen in 5.7 or 5.8.
The workaround was to simply kill+restart xenconsoled. No reboot required. Also I think the xenconsoled bug only happened on 32bit hosts.
In short, Xen paravirt was very fragile and troublesome. I never tested Xen with hardware virtualization.
Xen PV has been rock solid for me :)
I have had no such problems with KVM. In my experience KVM is much more stable than Xen paravirtualization. Xen HVM probably would suffer at least some of the same problems.
You should compare Xen HVM with KVM, and you said you haven't been running Xen HVM.
There have been bugs that allow guests to escalate privileges and access host resources, but they're relatively few. I don't think there's a significant difference between the two in this area.
Overall I advise the use of KVM. It should be more stable, and has the advantage of Red Hat support.
Xen is supported by Red Hat support in RHEL5.
-- Pasi
On 06/04/2012 11:36 PM, Pasi Kärkkäinen wrote:
Xen PV has been rock solid for me :)
Maybe, if we ignore the fact that you seem to be familiar with the problem of xenconsoled failing and preventing guests from booting.
Xen is supported by Red Hat support in RHEL5.
Yes, and RHEL5 will be supported for several years. However, there does not appear to be a plan to support Xen in the future, after RHEL5 expires. It would be irrational to invest time and money into training on Xen with no expectation that those skills will remain valuable in the future.
On Tue, Jun 05, 2012 at 01:50:30PM -0700, Gordon Messmer wrote:
On 06/04/2012 11:36 PM, Pasi Kärkkäinen wrote:
Xen PV has been rock solid for me :)
Maybe, if we ignore the fact that you seem to be familiar with the problem of xenconsoled failing and preventing guests from booting.
The el5 xenconsoled bug:
1) Affected only 32bit (i686) hosts running el5 Xen 2) It was fixed years ago on upstream Xen 3) It is already fixed in rhel5/centos5 4) It didn't affect 64bit (x86_64) hosts running el5 Xen 5) It was easy to workaround by killing+restarting xenconsoled
Xen is supported by Red Hat support in RHEL5.
Yes, and RHEL5 will be supported for several years. However, there does not appear to be a plan to support Xen in the future, after RHEL5 expires. It would be irrational to invest time and money into training on Xen with no expectation that those skills will remain valuable in the future.
There are other Xen-based virtualization solutions out there aswell with full support.
-- Pasi
On 07/08/2012 08:47 AM, Pasi Kärkkäinen wrote:
There are other Xen-based virtualization solutions out there aswell with full support.
That's true, but I'm guessing that a lot of people on this list are here specifically because they're not paying for support.
Whether that's true or not, if you're deploying virtualization *on CentOS*, KVM is really the only rational choice. Xen hosting support is no longer a part of new releases.
-
aurfalien -
Dennis Jacobfeuerborn -
Dmitry Cherkasov -
Gordon Messmer -
John R Pierce -
Jonathan Vomacka -
Karanbir Singh -
Luke S. Crawford -
Mihamina Rakotomandimby -
Pasi Kärkkäinen -
Peter Peltonen -
Rafał Radecki -
Steve Thompson -
Theo Band