[CentOS] Cannot remove lvs associated with deleted vm guests

Fri Jan 27 17:50:45 UTC 2012
James B. Byrne <byrnejb at harte-lyne.ca>

At the beginning of January I encountered a problem where
several vm guests on a single host somehow managed to see
the the virtual disks assigned to other guests on the same
hosts.  I was unable to resolve this situation and
shutdown the affected guests after creating new guest
instances and moving the services and data off the
corrupted guests.

I have since removed these guests via virt-manager but all
attempts to remove from the host the logical volumes
associated with the former VirtIO disks fail.  The volumes
are considered open by lvremove and nothing I have tried
can get them to close for removal.  The --force option has
no effect on this situation.

# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
  Can't remove open logical volume "lv_vm_base"

# dmsetup info -c vg_vhost01-lv_vm_base
Name                  Maj Min Stat Open Targ Event  UUID
vg_vhost01-lv_vm_base 253   5 L--w    2    1      0
LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52

# dmsetup remove vg_vhost01-lv_vm_base
device-mapper: remove ioctl failed: Device or resource busy
Command failed


There are several bugs filed on similar issues and udev is
sometimes identified as the culprit.  If I kill the udev
daemon with  T=`pidof -x udevd`; kill $T and rerun the
lvremove -f command then I see this change in behaviour:

# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
  Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
not
/dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
  Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_basep2 not
/dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
  Can't remove open logical volume "lv_vm_base"

I need to get this system stable and return the lost disk
space to the storage pool.  Does anyone have any
suggestions as to how to proceed?

If I cannot solve this using the available system commands
then prudence dictates that I have to re-install the
server OS and rebuild all of the vm guests. As these
guests have been laboriously transferred from other hosts
during the past month this is a task I would rather not
have to do.

Any help is gratefully accepted.

-- 
***          E-Mail is NOT a SECURE channel          ***
James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3