On 06/21/2012 01:29 PM, Philip Durbin wrote: > To allow for live migration between hypervisors, I've been using NFS for shared storage of the disk images for each of my virtual machines. Live migration works great, but I'm concerned about performance as I put more and more virtual machines on this infrastructure. The Red Hat docs warn that NFS won't scale in this situation and that iSCSI is preferred. > > I'm confused about how to effectively use iSCSI with KVM, however. libvirt can create new disk images all by itself in a storage pool backed by NFS, like I'm using, but libvirt can not create new disk images in a storage pool backed by iSCSI on its own. One must manually create the LUN on the iSCSI storage each time one wants to provision a virtual machine. I like how easy it is to deploy new virtual machines on NFS; I just define the system in Cobbler and kickstart it with koan. > > I think my solution to the problem of how to scale shared storage may be OpenStack, which promises this as a feature of Swift. Then, perhaps, I'll be able to leave NFS behind. > > I'd be happy to hear more stories of how to scale shared storage while continuing to allow for live migration. > AFAIK you cannot use Swift storage as a Nova volume backend. Also in order to make Swift scale you need at least a couple of nodes. You might want to take a look at ceph.com The offer an object store that can be attached as a block device (like iScsi) but KVM also contains a driver that can directly talk to the storage. Then there is CephFS which is basically a posix filesystem on top of the object store that has some neat features and would be a closer replacement to NFS. Another thing to look at is http://www.osrg.net/sheepdog/ This is very similar to ceph's object storage approach. Some large scale benchmarks (1000 nodes) can be found here: http://sheepdog.taobao.org/people/zituan/sheepdog1k.html Then there is http://www.gluster.org/ This is probably the most mature solution but I'm not sure if the architecture will be able to compete against the other solutions in the long run. Regards, Dennis