On Tue, Oct 26, 2010 at 4:15 PM, Eric Searcy emsearcy@gmail.com wrote:
On Oct 25, 2010, at 8:14 AM, Todd Deshane wrote:
I was also going to mention that we should look at scalability and performance isolation. Some references and previous studies here:
http://todddeshane.net/research/Xen_versus_KVM_20080623.pdf http://clarkson.edu/~jnm/publications/isolation_ExpCS_FINALSUBMISSION.pdf http://clarkson.edu/~jnm/publications/freenix04-clark.pdf
I only got as far as the top one. One concern: the nestled comment "We believe that KVM may have performed better than Xen in terms of I/O due to disk caching" makes me skeptical of the value of the results if this wasn't taken into consideration (in other words I think it is a much bigger problem than the aforementioned comment gives credit to, such that it ought to be at least addressed in the concluding remarks) ... for instance if my VM load-outs use all but ~384M of total memory (that being the amount I leave to the host, most of it used) then there's not going to be much extra RAM for memory cache/buffers with on the host side (depending greatly on what vm.swappiness camp you are in). Based on the author's result output [1] (since the VM parameters aren't given in the paper), as relates to a disk-intensive test they in effect gave 2G potential caching to Xen but ~4G to KVM. Based at least on the amount of free memory on my Xen/KVM hosts I don't think this "host memor y cache bias" can be assumed to be a bonus trait that would normally be present for KVM. (And of course a cache bias would be even more noticeable in the 256MB Phoronix test and in the 4x128M isolation tests [2] ...)
[1] http://web2.clarkson.edu/projects/virtualization/benchvm/results/performance... [2] http://web2.clarkson.edu/projects/virtualization/benchvm/results/isolation/x...
BTW, I do realize you're pointing out that we should look at scalability and isolation, and here I am just giving critical feedback on a 3 year old paper ... yes you're right those are important! I just want to make sure the tests are fair ;-)
These are old tests now and not necessarily perfect, but they were Xen and KVM on the same kernel. KVM very early on and not necessarily in its best light. The Xen dom0/domU kernel was also not the best light for Xen. The point was to try to compare them on the same kernel. Xen and KVM on the same kernel has happened in the form of OpenSUSE's forward port of the Xen dom0 kernel with KVM that is already in mainline. The pv_ops kernel is not fully mainline, but is getting close. Some distros now have Xen dom0 kernels based on the pv_ops kernel, which could also run KVM.
In any case, some updated numbers would be very welcome. And yes, taking scalability, performance isolation, and other factors into account is important.
I have been involved in quite a few performance studies over the years and I will try to give advice and help as I can.
On Tue, Oct 26, 2010 at 2:38 PM, Grant McWilliams grantmasterflash@gmail.com wrote:
So you've already done a lot of this then Todd? Should we just be making a standardized test out of your work?
At Clarkson, we tried to build a standardized tool called benchvm as a standard, but it has been stuck in an alpha state for awhile http://code.google.com/p/benchvm/
I can dig up the rejected academic paper on the benchvm if people are interested, feel free to email me privately.
I wonder if using things like puppet, the phoronix test suite, etc. are a simpler way to go? I guess it all depends on how general a benchmarking tool is needed.
Thanks, Todd