This may not be the best place to ask, but I was prompted by a question about guest cores on KVM.
We currently use VMWare Server (v1.0) on CentOS5.
It supports up to two virtual CPUs, but not very well, as I understand it. VMWare Server 2.0 might do better at supporting the same maximum of 2 CPUs, but if my research is correct, they both use what is called "strict co-scheduling". Which means, if a two virtual-CPU VM is waiting for a time slice on the physical host, there needs to be two physical CPUs available to be allocated. This is because amount of time scheduled to each virtual CPU has to stay equal, else assumptions in the virtualised software may become false and things start going wrong.
The problem with this is that it is obviously more of a constraint to have two physical CPUs available. Therefore adding virtual CPUs to a co-scheduled VM can actually make performance worse if the physical CPUs are under any sort of contention. Performance degrades rapidly under load - which is exactly what you don't want for a webserver. Therefore advice seems to be to avoid using multi-virtual CPU VMs. [1]
VMWare ESX uses something supposedly better, called relaxed co-scheduling. [2]. We're not likely to use that any time soon, since we are using 1&1 root servers running CentOS.
I did try and research the algorithm(s) used by KVM. Apparently it doesn't use co-scheduling, but possibly something called the "completely fair scheduler", but I'm no an expert in VM scheduling and I didn't manage to discover what this means, nor specifically what the implications are for performance were in terms of scaling up the number of virtual CPUs on KVM. (I understand that the number of virtual CPUs supported by KVM is be quite high.)
Anyway, I've been asked how to scale a VM up utilise more - say four or more - physical CPUs. Is KVM better at this than VMware Server, or does the same basic problem persist?
Thanks,
Nick
1. http://communities.vmware.com/thread/169323
http://cs.gmu.edu/~hfoxwell/cs671projects/southern_v12n.pdf
"The SMP scheduling algorithms used by ESX server and by Xen have a low overhead for single-threaded compute inten- sive workloads. However, they do not scale as well for multi-threaded workloads on a system with overcommit- ted CPU resources."
2. http://communities.vmware.com/thread/157849 http://communities.vmware.com/docs/DOC-4960 http://communities.vmware.com/docs/DOC-5501
2010/11/19 Nick oinksocket@letterboxes.org:
The problem with this is that it is obviously more of a constraint to have two physical CPUs available. Therefore adding virtual CPUs to a co-scheduled VM can actually make performance worse if the physical CPUs are under any sort of contention. Performance degrades rapidly under load - which is exactly what you don't want for a webserver. Therefore advice seems to be to avoid using multi-virtual CPU VMs. [1]
As long as you keep the number of CPUs for *each* VM - equal to or lower than - the number of physical cores, your performance should not suffer with KVM. If you assign more CPUs to one VM than you have available cores (a core can also be a hyperthreaded core), you will for sure run into performance problems.
I did try and research the algorithm(s) used by KVM. Apparently it doesn't use co-scheduling, but possibly something called the "completely fair scheduler",
The good thing about KVM compared to other virtualization solutions, is that KVM doesn't try to reinvent the wheel. It leaves scheduling to the Linux kernel, so whatever your Linux system is setup to use, KVM will use that. You can choose to run CFS (Completely Fair Scheduler), deadline, BFS, or whatever scheduler you prefer. As long as Linux uses it, KVM will use it.
but I'm no an expert in VM scheduling and I didn't manage to discover what this means, nor specifically what the implications are for performance were in terms of scaling up the number of virtual CPUs on KVM. (I understand that the number of virtual CPUs supported by KVM is be quite high.)
If you want a really competent answer to your question, send it to the kvm-devel list...they don't mind user questions like this, and as long as you're asking a relevant question, there's a good chance of getting a highly competent answer :)
Best regards Kenni
Thanks for the quick reply.
On 18/11/10 23:45, Kenni Lund wrote:
The good thing about KVM compared to other virtualization solutions, is that KVM doesn't try to reinvent the wheel. It leaves scheduling to the Linux kernel, so whatever your Linux system is setup to use, KVM will use that. You can choose to run CFS (Completely Fair Scheduler), deadline, BFS, or whatever scheduler you prefer. As long as Linux uses it, KVM will use it.
Is this process scheduling we're talking about here? Hmm, and I wonder how I find out what scheduling algorithm my kernel is using.
So, this sounds feasible... except if virtual CPUs must have time shared equally as the VMWare co-scheduling explanations imply. A scheduler for threads / processes presumably wouldn't guarantee such a thing?
Maybe I'll go and investigate the KVM list....
Thanks
N
2010/11/19 Nick oinksocket@letterboxes.org:
Thanks for the quick reply.
On 18/11/10 23:45, Kenni Lund wrote:
The good thing about KVM compared to other virtualization solutions, is that KVM doesn't try to reinvent the wheel. It leaves scheduling to the Linux kernel, so whatever your Linux system is setup to use, KVM will use that. You can choose to run CFS (Completely Fair Scheduler), deadline, BFS, or whatever scheduler you prefer. As long as Linux uses it, KVM will use it.
Is this process scheduling we're talking about here?
Yep, a virtual CPU is just a process on the host Linux system.
So, this sounds feasible... except if virtual CPUs must have time shared equally as the VMWare co-scheduling explanations imply. A scheduler for threads / processes presumably wouldn't guarantee such a thing?
I think cgroups is the solution, if you want to guarantee resources to some guests. I haven't tested it with KVM, but perhaps "nice" and "ionice" can be useful as well...the guests are just Linux processes after all.
Maybe I'll go and investigate the KVM list....
:)
Best regards Kenni
On 19/11/10 00:11, Kenni Lund wrote:
I think cgroups is the solution, if you want to guarantee resources to some guests. I haven't tested it with KVM, but perhaps "nice" and "ionice" can be useful as well...the guests are just Linux processes after all.
Just to clarify, it isn't that *I* want to guarantee equal time sharing, it's that the entire premise of co-scheduling implies that the guest VM's OS may malfunction if the system doesn't guarantee it, or some approximation of it.
To quote from http://communities.vmware.com/docs/DOC-4960:
Without coscheduling, the VCPUs associated with an SMP VM would be scheduled independently, breaking the guest's assumptions regarding uniform progress. We use the term "skew" to refer to the difference in execution rates between two or more VCPUs associated with an SMP VM.
Inter-VCPU skew violates the assumptions of guest software. Non-trivial skew can result in severe performance problems, and may even induce failures when the guest expects inter-VCPU operations to complete quickly. Let's first consider the performance implications of skew. Guest OS kernels typically use spin locks for interprocessor synchronization. If the VCPU currently holding a lock is descheduled, then the other VCPUs in the same VM will waste time busy-waiting until the lock is released. Similar performance problems can also occur in multi-threaded user-mode applications, which may also synchronize using locks or barriers. Unequal VCPU progress will also confuse the guest OS cpu scheduler, which attempts to balance load across VCPUs.
N
-
Kenni Lund -
Nick