Hi all!
I've been experimenting with Xen on CentOS 5 and RHAS 5 for a while now with mixed success I thought I'd describe my latest challenge. I'll describe this from memory since all the equipment is at work and not contactable from here.
I think I've described this config to the list before but here it is again:
I have 2 x HP DL585 servers each with 4 Dual core Opterons (non-vmx) and 16GB RAM configured as Xen servers. These run CentOS 5.1 with the latest updates applied. These system both attach to an iSCSI target which is an HP DL385 running ietd and serving SAN based storage.
Everything runs fine if I do no migration. I was having a "soft lockup" problem which has been solved by installing the latest test kernel from redhat. They've adjusted a timer as described in this report (comment 5): https://bugzilla.redhat.com/show_bug.cgi?id=250994
Anyway things are pretty stable now but... If I do a series of migrations, live or not, from one server to the other eventually I will get the process to fail. This could take up to an hour with the migrations happening every 5 minutes. It can also happen first try. The message in the xend.log file says that it is unable to find the device number for the virtualised storage i.e. sda. In my configuration I have dom0 connecting to the LUNs used by the VMs to the domU's are not doing iSCSI. I'm passing the "/dev/disks/by-path/iscsixxx:sda1" info in via the xen config file.
If I mount and unmount the same LUN's filesystem to the dom0 over and over again it works every time so there's no fundamental problem with the iSCSI connection itself. Each server is using 3 gige interfaces: One for normal LAN access, a dedicated network for the iSCSI and a crossover cable between the third gige interfaces on the servers for the migration channel.
I have a xentop running on both dom0's and can tell that its failed when the vm appears in the target but the memory used by the vm doesn't start incrementing. The end result is that the VM is hung and has to be destroyed and re-created. The error happens immediately after the failed migration is initiated.
I've tried doing a read of the first few blocks of the LUN on the target immediately before initiating the migration which is successful but makes no difference.
I realize this is pretty sketchy information but I was wondering if others were seeing similar problems. The ability to do reliable migration is basically the prime motivation for us wanting to do virtualization at all. Otherwise we'd just run the required services on the main machine.
Anyway, any help would be appreciated.
Brett