[CentOS-virt] [fedora-virt] Thoughts on storage infrastructure for small scale HA virtual machine deployments

Sun Mar 7 13:58:14 UTC 2010
Dennis J. <dennisml at conversis.de>

On 03/02/2010 04:51 AM, Ask Bjørn Hansen wrote:
> On Mar 1, 2010, at 18:56, Dennis J. wrote:
>> The question that bugs me is how I can get enough bandwidth between the
>> hosts and the storage to provide the VMs with reasonable I/O performance.
>> If all the 40 VMs start copying files at the same time that would mean that
>> the bandwidth share for each VM would be tiny.
> It really depends on the specific workloads.  In my experience it's generally the number of IOs per second rather than the bandwidth that's the limiting factor.
> We have a bunch of 4-disk boxes with md raid10 and we generally run out of disk IO before we run out of memory (~24-48GB) or CPU (dual quad core 2.26GHz or some such).

That's very similar to what we are experiencing. The primary Problem for me 
is how to deal with the bottleneck of a shared storage setup. The most 
simple setup is a 2-system criss-cross setup where the two hosts also serve 
as halves for a DRBD cluster. The advantage of this approach is that it's a 
cheap solution, that only a part of the storage-traffic has to go over the 
network between the machines and that the network only hast to handle the 
sorage-traffic of the VMs of those two machines.
The disadvantage of that approach is that you have to keep 50% of potential 
server capacity free in case of a failure of the twin node. That's quite a 
lot of wasted capacity.
To reduce that problem you can increase the number of hosts to let say for 
an example four which would reduce the spare capacity needed to 33% on each 
system but then you really need to separate the storage from the hosts an 
now you have a bottleneck on the storage end. Increase the number of hosts 
to 8 and you get even less wasted capacity but also increase the pressure 
on the storage bottleneck a lot.
Since I'm new to the whole SAN aspect I'm currently just looking at all the 
options that are out there and basically wonder how the big boys are 
handling this who have hundreds if not thousand of VMs running and need to 
be able to deal with physical failures too.
That is why I find the sheepdog project so interesting because it seems to 
address this particular problem in a way that would provide almost linear 
scalability without actually using a SAN at all (well, at least not in the 
traditional sense of the word).