At the beginning of January I encountered a problem where several vm guests on a single host somehow managed to see the the virtual disks assigned to other guests on the same hosts. I was unable to resolve this situation and shutdown the affected guests after creating new guest instances and moving the services and data off the corrupted guests.
I have since removed these guests via virt-manager but all attempts to remove from the host the logical volumes associated with the former VirtIO disks fail. The volumes are considered open by lvremove and nothing I have tried can get them to close for removal. The --force option has no effect on this situation.
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base Can't remove open logical volume "lv_vm_base"
# dmsetup info -c vg_vhost01-lv_vm_base Name Maj Min Stat Open Targ Event UUID vg_vhost01-lv_vm_base 253 5 L--w 2 1 0 LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
# dmsetup remove vg_vhost01-lv_vm_base device-mapper: remove ioctl failed: Device or resource busy Command failed
There are several bugs filed on similar issues and udev is sometimes identified as the culprit. If I kill the udev daemon with T=`pidof -x udevd`; kill $T and rerun the lvremove -f command then I see this change in behaviour:
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ: using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2 not /dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2 Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ: using /dev/mapper/vg_vhost01-lv_vm_basep2 not /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2 Can't remove open logical volume "lv_vm_base"
I need to get this system stable and return the lost disk space to the storage pool. Does anyone have any suggestions as to how to proceed?
If I cannot solve this using the available system commands then prudence dictates that I have to re-install the server OS and rebuild all of the vm guests. As these guests have been laboriously transferred from other hosts during the past month this is a task I would rather not have to do.
Any help is gratefully accepted.
Hi,
Its ages since I came across this problem so my memory is a little hazey but something is obviously holding on to the lv so you'll need to figure out whats holding it and kill that. The open value returned by "dmsetup info" is how many things have the device file open), it looks to be 2 at the moment and you cant "dmsetup remove" untill its zero (lvm might be one of them I cant remember)
A few things worth trying are
1) make sure its not mounted anywhere! 2) if multipathd is running try stopping that.
3) either lsof or fuser on the device file may be able to tell you which process has it open 4) something vm related might not have let go properly, are there any deamons/processes etc still running?
5) reboot, the sledgehammer aproach to killing off processes!!
6) you could try hitting it with dmsetup again, you need to suspend the device first using "dmsetup suspend" which *may* persuade the holding process to let go, if it does reduce the open count you'll need to "dmsetup resume" and then suspend again untill open reaches zero when "dmsetup remove" should work. I'd try and avoid this option if you can, you're messing beneath the lvm layer and it may not like that, should be ok but...
if none of that is possible/works you could try asking on the lvm-linux list.
Reinstalling really shouldn't be necesary.
chris
----- Original Message -----
From: James B. Byrne byrnejb@harte-lyne.ca To: centos-virt@centos.org; centos@centos.org Cc: Sent: Friday, 27 January 2012, 17:50 Subject: [CentOS] Cannot remove lvs associated with deleted vm guests
At the beginning of January I encountered a problem where several vm guests on a single host somehow managed to see the the virtual disks assigned to other guests on the same hosts. I was unable to resolve this situation and shutdown the affected guests after creating new guest instances and moving the services and data off the corrupted guests.
I have since removed these guests via virt-manager but all attempts to remove from the host the logical volumes associated with the former VirtIO disks fail. The volumes are considered open by lvremove and nothing I have tried can get them to close for removal. The --force option has no effect on this situation.
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base Can't remove open logical volume "lv_vm_base"
# dmsetup info -c vg_vhost01-lv_vm_base Name Maj Min Stat Open Targ Event UUID vg_vhost01-lv_vm_base 253 5 L--w 2 1 0 LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
# dmsetup remove vg_vhost01-lv_vm_base device-mapper: remove ioctl failed: Device or resource busy Command failed
There are several bugs filed on similar issues and udev is sometimes identified as the culprit. If I kill the udev daemon with T=`pidof -x udevd`; kill $T and rerun the lvremove -f command then I see this change in behaviour:
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ: using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2 not /dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2 Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ: using /dev/mapper/vg_vhost01-lv_vm_basep2 not /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2 Can't remove open logical volume "lv_vm_base"
I need to get this system stable and return the lost disk space to the storage pool. Does anyone have any suggestions as to how to proceed?
If I cannot solve this using the available system commands then prudence dictates that I have to re-install the server OS and rebuild all of the vm guests. As these guests have been laboriously transferred from other hosts during the past month this is a task I would rather not have to do.
Any help is gratefully accepted.
-- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB@Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Pardon the top-post and cross-post, but this apparently should have gone to centos-virt@centos.org
Phil
chris procter wrote on 01/27/2012 03:29 PM:
Hi,
Its ages since I came across this problem so my memory is a little hazey but something is obviously holding on to the lv so you'll need to figure out whats holding it and kill that. The open value returned by "dmsetup info" is how many things have the device file open), it looks to be 2 at the moment and you cant "dmsetup remove" untill its zero (lvm might be one of them I cant remember)
A few things worth trying are
make sure its not mounted anywhere!
if multipathd is running try stopping that.
either lsof or fuser on the device file may be able to tell you which process has it open
something vm related might not have let go properly, are there any deamons/processes etc still running?
reboot, the sledgehammer aproach to killing off processes!!
you could try hitting it with dmsetup again, you need to suspend the device first using "dmsetup suspend" which *may* persuade the holding process to let go, if it does reduce the open count you'll need to "dmsetup resume" and then suspend again untill open reaches zero when "dmsetup remove" should work. I'd try and avoid this option if you can, you're messing beneath the lvm layer and it may not like that, should be ok but...
if none of that is possible/works you could try asking on the lvm-linux list.
Reinstalling really shouldn't be necesary.
chris
----- Original Message -----
From: James B. Byrnebyrnejb@harte-lyne.ca To: centos-virt@centos.org; centos@centos.org Cc: Sent: Friday, 27 January 2012, 17:50 Subject: [CentOS] Cannot remove lvs associated with deleted vm guests
At the beginning of January I encountered a problem where several vm guests on a single host somehow managed to see the the virtual disks assigned to other guests on the same hosts. I was unable to resolve this situation and shutdown the affected guests after creating new guest instances and moving the services and data off the corrupted guests.
I have since removed these guests via virt-manager but all attempts to remove from the host the logical volumes associated with the former VirtIO disks fail. The volumes are considered open by lvremove and nothing I have tried can get them to close for removal. The --force option has no effect on this situation.
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base Can't remove open logical volume "lv_vm_base"
# dmsetup info -c vg_vhost01-lv_vm_base Name Maj Min Stat Open Targ Event UUID vg_vhost01-lv_vm_base 253 5 L--w 2 1 0 LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
# dmsetup remove vg_vhost01-lv_vm_base device-mapper: remove ioctl failed: Device or resource busy Command failed
There are several bugs filed on similar issues and udev is sometimes identified as the culprit. If I kill the udev daemon with T=`pidof -x udevd`; kill $T and rerun the lvremove -f command then I see this change in behaviour:
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ: using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2 not /dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2 Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ: using /dev/mapper/vg_vhost01-lv_vm_basep2 not /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2 Can't remove open logical volume "lv_vm_base"
I need to get this system stable and return the lost disk space to the storage pool. Does anyone have any suggestions as to how to proceed?
If I cannot solve this using the available system commands then prudence dictates that I have to re-install the server OS and rebuild all of the vm guests. As these guests have been laboriously transferred from other hosts during the past month this is a task I would rather not have to do.
Any help is gratefully accepted.
-- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB@Harte-Lyne.ca Harte& Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
chris procter -
James B. Byrne -
Phil Schaffner