On Fri, May 27, 2011 at 2:33 PM, James B. Byrne byrnejb@harte-lyne.ca wrote:

I have been working off and on with Xen and KVM on a couple of test hosts for that past year or so and while now everything seems to function as expected, more or less, I find myself asking the question: Why?

We run our own servers at our own sites for our own purposes.  We do not, with a few small exceptions, host alien domains.  So, while rapidly provisioning or dynamically expanding a client's vm might be very attractive to a public hosting provider that is not our business model at all.

Why would a small company not in the public hosting business choose to employ VM technology?  What are the benefits over operating several individual small form factor servers or blades instead?  I am curious because what I can find on the net respecting VM use cases, outside of for public providers or application testing, seems to me mostly puff and smoke.

This might be considered OT but since CentOS is what we use it seems to me best that I ask here to start.

I would have to say that the usecase really has to do with what you are doing with servers.

For us when our facility was built ~6 years ago they bought racks and racks of servers from Dell. There were racks with 1U dual CPU boxes and then racks with 2U dual CPU and then the racks with 4U Quad CPU boxes with varying amounts of ram and drive space in them. That was the way to have resources available to spin up a new box for someone in a timely manner rather than waiting a few weeks for one to be delivered... but at year 2 it became obvious that many of the boxes were not being used or they were really underutilized as the software didn't need that much power but it was a Windows App that wanted to have the box to itself. So virtualization came into play. Now instead of 10 racks of random servers of varying specs we have 1 rack with some really beefy servers and then a couple racks with individual boxes for specific applications that were to resource intense to make sense to virtualize.

The advantages we have been able to realize is the ability to spin up a 2 processor 1gb VM for someone to start their project and then if it gets off the ground and they need more we can bump it up to what they need by editing a configuration rather than having to add memory to a box or worse yet have to migrate it from one box to another. This has resulted in much better utilization of the hardware that we have. Many servers don't need lots of resources they just need to have the resources they have all to themselves or atleast think that they do. We have many servers that only see a spike in workload periodicaly during the week/month when a researcher has collected the data they need to work on. The rest of the time it sits there waiting.

The other real advantage for us is the DR solutions that virtualizing gives us. Previously DR ment having a stack of machines at the DR site waiting to be recovered to and deciding what servers were important enough that we needed a box with their name on it and what could wait until we could call Dell and get a new one shipped out. Now we just have a smaller VMware cluster at the other end running some hot DR servers and ready to bring up the replicated LUNs from our primary site in case of a failure. We can bring those VMs back up in a short time instead of having to do baremetal installs then restore from backups. Going foward we will be scaling so that we can share our DR at the remote site with the group that is there as their primary and they will be able to use our cluster here as their DR allowing us to consolidate the number of hosts needed. That is something that just isn't possiable with physical boxes.

I can't say that you don't take a monitary hit to buy the shared storage, networking, and software required to support a Virtual world but I personaly see it as worth it.

On the other hand there are those systems that just make no sense to virtualize. We run a HPC cluster with hundreds of nodes. There is no reason to virtualize it as we are already trying to squeeze every bit of performance out of the boxes and adding the hypervisor would defeat that and we don't need/want more than one OS install running on a physical box.

So really look at your workload and your business case. Virtualization isn't the end all be all silver bullet that will make everything in IT perfect. It is just another tool in your toolbox that can help in some cases but much as showing up with a hammer and finding a screw you might be better off going back to get a screw driver than trying to just pound it in.

On 5/27/2011 1:33 PM, James B. Byrne wrote:

I have been working off and on with Xen and KVM on a couple of test hosts for that past year or so and while now everything seems to function as expected, more or less, I find myself asking the question: Why?

We run our own servers at our own sites for our own purposes. We do not, with a few small exceptions, host alien domains. So, while rapidly provisioning or dynamically expanding a client's vm might be very attractive to a public hosting provider that is not our business model at all.

Why would a small company not in the public hosting business choose to employ VM technology? What are the benefits over operating several individual small form factor servers or blades instead? I am curious because what I can find on the net respecting VM use cases, outside of for public providers or application testing, seems to me mostly puff and smoke.

It is fairly difficult to avoid applications that require specific OS versions or conflict with certain other applications, especially as things evolve over time. This means you are likely to end up with machines (and their backups) dedicated to specific legacy apps even though you could otherwise consolidate them on newer/faster hardware and reduce the maintenance/power/space requirements. Putting them on VMs lets you separate the physical resource concerns from the applications you support. And in some cases you might set up backup/failover instances as VMs even where the normally-live host is a physical machine.

This might be considered OT but since CentOS is what we use it seems to me best that I ask here to start.

Aside from consolidating things that don't want to be consolidated (a somewhat odd concept for an OS that should theoretically be able to have multiple versions of most libraries and applications running at once but yum and rpm won't like it), there is also the issue of moving to different hardware if an older machine crashes. While Centos is pretty good about detecting hardware during an install, if you want to restore your backup onto something different you'll need to know as much as anaconda does to make it work, where a VM image will work the same way regardless of the physical hardware.

But taking the other side of the argument, here are two scenarios where I *wouldn't* use virtualization (one could certainly enumerate more):

1. A production DB server or server cluster.

2. I've had services where I needed to maximize uptime. One option I tried were VMs and being able to move the VM back and forth between hosts. That might cover hardware failure, but I'd still take outages when I needed to upgrade software in the VM. Moving to a traditional HA solution on physical hardware means the outages are now measured in seconds instead of minutes, and most of the time are undetectable by the users.

I also tried services where the HA nodes are themselves VMs, but was less than impressed with operational stability.

When CentOS 6 comes out, though, I'll be interested to see how (2) behaves when it comes time to do a rolling upgrade from CentOS 5 (bring a node down, install and reconfigure C6 from scratch, rejoin the cluster, have C6 take over the services, then upgrade the other node).

Thank god for test environments. And backups.

Devin

From: James B. Byrne byrnejb@harte-lyne.ca

I have been working off and on with Xen and KVM on a couple of test hosts for that past year or so and while now everything seems to function as expected, more or less, I find myself asking the question: Why?

I saw a company architecture presentation where they had paired servers with 2 VMs each + DRBD + heartbeat... So, on each server there is an active VM and a 'backup' VM, crossed. If one server goes down, the 'backup' VM on the second server would be activated.

JD