Pardon the top-post and cross-post, but this apparently should have gone
to centos-virt(a)centos.org
Phil
chris procter wrote on 01/27/2012 03:29 PM:
> Hi,
>
> Its ages since I came across this problem so my memory is a little hazey but something is obviously holding on to the lv so you'll need to figure out whats holding it and kill that. The open value returned by "dmsetup info" is how many things have the device file open), it looks to be 2 at the moment and you cant "dmsetup remove" untill its zero (lvm might be one of them I cant remember)
>
>
> A few things worth trying are
>
>
> 1) make sure its not mounted anywhere!
> 2) if multipathd is running try stopping that.
>
> 3) either lsof or fuser on the device file may be able to tell you which process has it open
> 4) something vm related might not have let go properly, are there any deamons/processes etc still running?
>
> 5) reboot, the sledgehammer aproach to killing off processes!!
>
> 6) you could try hitting it with dmsetup again, you need to suspend the device first using "dmsetup suspend" which *may* persuade the holding process to let go, if it does reduce the open count you'll need to "dmsetup resume" and then suspend again untill open reaches zero when "dmsetup remove" should work. I'd try and avoid this option if you can, you're messing beneath the lvm layer and it may not like that, should be ok but...
>
>
> if none of that is possible/works you could try asking on the lvm-linux list.
>
>
> Reinstalling really shouldn't be necesary.
>
>
> chris
>
>
>
>
>
> ----- Original Message -----
>> From: James B. Byrne<byrnejb(a)harte-lyne.ca>
>> To: centos-virt(a)centos.org; centos(a)centos.org
>> Cc:
>> Sent: Friday, 27 January 2012, 17:50
>> Subject: [CentOS] Cannot remove lvs associated with deleted vm guests
>>
>> At the beginning of January I encountered a problem where
>> several vm guests on a single host somehow managed to see
>> the the virtual disks assigned to other guests on the same
>> hosts. I was unable to resolve this situation and
>> shutdown the affected guests after creating new guest
>> instances and moving the services and data off the
>> corrupted guests.
>>
>> I have since removed these guests via virt-manager but all
>> attempts to remove from the host the logical volumes
>> associated with the former VirtIO disks fail. The volumes
>> are considered open by lvremove and nothing I have tried
>> can get them to close for removal. The --force option has
>> no effect on this situation.
>>
>> # /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
>> Can't remove open logical volume "lv_vm_base"
>>
>> # dmsetup info -c vg_vhost01-lv_vm_base
>> Name Maj Min Stat Open Targ Event UUID
>> vg_vhost01-lv_vm_base 253 5 L--w 2 1 0
>> LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
>>
>> # dmsetup remove vg_vhost01-lv_vm_base
>> device-mapper: remove ioctl failed: Device or resource busy
>> Command failed
>>
>>
>> There are several bugs filed on similar issues and udev is
>> sometimes identified as the culprit. If I kill the udev
>> daemon with T=`pidof -x udevd`; kill $T and rerun the
>> lvremove -f command then I see this change in behaviour:
>>
>> # /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
>> Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
>> using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
>> not
>> /dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
>> Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
>> using /dev/mapper/vg_vhost01-lv_vm_basep2 not
>> /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
>> Can't remove open logical volume "lv_vm_base"
>>
>> I need to get this system stable and return the lost disk
>> space to the storage pool. Does anyone have any
>> suggestions as to how to proceed?
>>
>> If I cannot solve this using the available system commands
>> then prudence dictates that I have to re-install the
>> server OS and rebuild all of the vm guests. As these
>> guests have been laboriously transferred from other hosts
>> during the past month this is a task I would rather not
>> have to do.
>>
>> Any help is gratefully accepted.
>>
>> --
>> *** E-Mail is NOT a SECURE channel ***
>> James B. Byrne mailto:ByrneJB@Harte-Lyne.ca
>> Harte& Lyne Limited http://www.harte-lyne.ca
>> 9 Brockley Drive vox: +1 905 561 1241
>> Hamilton, Ontario fax: +1 905 561 0757
>> Canada L8E 3C3
>>
>> _______________________________________________
>> CentOS mailing list
>> CentOS(a)centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
> _______________________________________________
> CentOS mailing list
> CentOS(a)centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
At the beginning of January I encountered a problem where
several vm guests on a single host somehow managed to see
the the virtual disks assigned to other guests on the same
hosts. I was unable to resolve this situation and
shutdown the affected guests after creating new guest
instances and moving the services and data off the
corrupted guests.
I have since removed these guests via virt-manager but all
attempts to remove from the host the logical volumes
associated with the former VirtIO disks fail. The volumes
are considered open by lvremove and nothing I have tried
can get them to close for removal. The --force option has
no effect on this situation.
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
Can't remove open logical volume "lv_vm_base"
# dmsetup info -c vg_vhost01-lv_vm_base
Name Maj Min Stat Open Targ Event UUID
vg_vhost01-lv_vm_base 253 5 L--w 2 1 0
LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
# dmsetup remove vg_vhost01-lv_vm_base
device-mapper: remove ioctl failed: Device or resource busy
Command failed
There are several bugs filed on similar issues and udev is
sometimes identified as the culprit. If I kill the udev
daemon with T=`pidof -x udevd`; kill $T and rerun the
lvremove -f command then I see this change in behaviour:
# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
not
/dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_basep2 not
/dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
Can't remove open logical volume "lv_vm_base"
I need to get this system stable and return the lost disk
space to the storage pool. Does anyone have any
suggestions as to how to proceed?
If I cannot solve this using the available system commands
then prudence dictates that I have to re-install the
server OS and rebuild all of the vm guests. As these
guests have been laboriously transferred from other hosts
during the past month this is a task I would rather not
have to do.
Any help is gratefully accepted.
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
If I log on to the newly cloned guest and I try and rename
the vg used by that instance I see this:
[root@vm-centos-6 ~]# vgrename vg_vm_centos_6 vg_vm_renamed
Couldn't find device with uuid
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De.
Cannot change VG vg_vm_centos_6 while PVs are missing.
Consider vgreduce --removemissing.
[root@vm-centos-6 ~]# vgreduce vg_vm_centos_6 --removemissing
Couldn't find device with uuid
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De.
WARNING: Partial LV lv_pgsql needs to be repaired or
removed.
WARNING: Partial LV lv_backups needs to be repaired or
removed.
WARNING: There are still partial LVs in VG vg_vm_centos_6.
To remove them unconditionally use: vgreduce
--removemissing --force.
Proceeding to remove empty missing PVs.
The xmldump for this guest instance contains this:
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/vg_vhost01/lv_vm_test-vg-rename'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' unit='0'/>
</disk>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x1'/>
The /dev entries for this vm guest are these:
[root@vm-centos-6 ~]# ll /dev/vg_vm_centos_6/*
lrwxrwxrwx. 1 root root 7 Jan 16 15:36
/dev/vg_vm_centos_6/lv_root -> ../dm-0
lrwxrwxrwx. 1 root root 7 Jan 16 15:36
/dev/vg_vm_centos_6/lv_swap -> ../dm-1
Searching for the missing uuid I find these files:
# find /etc -print | xargs grep
'umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De'
/etc/lvm/archive/vg_vm_centos_6_00005-1429183950.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00002-807578735.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00004-1687335328.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00003-650133889.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/archive/vg_vm_centos_6_00006-1907108135.vg:
id = "umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
/etc/lvm/backup/vg_vm_centos_6: id =
"umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De"
Looking in the files of /etc/lvm/archive created by the
vgrename command shows a complete mess of lvs from
different vm guests non of which have anything in common
with the test vm guest, other than they all share the same
volume group name.
Evidently, cloning vm instances for the purpose of setting
up a new vm guest to run with other vms cloned from the
same prototype is not a good idea,
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
We are in the process of migrating several stand-alone
server hosts onto a CentOS-6 kvm virtual host. We also
use Webmin to administer our hosts. All of the guests,
without exception, have been cloned brom a prototype guest
using virt-manager. All of the additional VirtIO disks
assigned to some of the guests have been added through
virt-manager as well.
Recently I have encountered a situation with the
development copy of Webmin where I can see an additional
VirtIO disk assigned to one guest instance, but not yet
added to the lvg of that guest, as an unknown device on
different guest. Further, I can see an lv assigned to one
guest instance as an lv on a difference guest.
The webmin author is aware of this problem and is
assiduously trying to determine its cause. My difficulty
is answering the question: How is it even possible for an
application running under a httpd service on one guest to
see anything at all besides the VirtIO storage assigned to
that guest?
Has anyone else encountered this anomaly?
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
On Mon, January 16, 2012 13:05, James B. Byrne wrote:
> How is it even possible for an
> application running under a httpd service on one guest to
> see anything at all besides the VirtIO storage assigned to
> that guest?
>
> Has anyone else encountered this anomaly?
>
I just cloned a guest instance. The clone prototype was
set up with a single VirtIO disk of 8Gbs, divided into a
500 Mb boot and a ~7.1Gb root partition. The root
partition was entirely assigned to the basic vg and two lv
were created, one for swap and one for the actual root
partition.
When the guest was cloned there was only one VirtIO disk
of 8 Gb assigned to it and this was cloned and given a new
name.
When I look at the newly cloned guest instance with
pvdisplay this is what I see:
# pvdisplay
Couldn't find device with uuid
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De.
--- Physical volume ---
PV Name /dev/vda2
VG Name vg_vm_centos_6
PV Size 7.32 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 1874
Free PE 0
Allocated PE 1874
PV UUID
djM23m-6Yeb-BQ2x-gPh9-ORMt-dX2i-Ou9xBQ
--- Physical volume ---
PV Name unknown device
VG Name vg_vm_centos_6
PV Size 31.25 GiB / not usable 3.97 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 7999
Free PE 0
Allocated PE 7999
PV UUID
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De
When I look at it using vgdisplay then this is what I see:
# vgdisplay
Couldn't find device with uuid
umrIn6-Np0c-NC4Z-MuUo-5TBj-IKRE-XBU0De.
--- Volume group ---
VG Name vg_vm_centos_6
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 10
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 2
Max PV 0
Cur PV 2
Act PV 1
VG Size 38.57 GiB
PE Size 4.00 MiB
Total PE 9873
Alloc PE / Size 9873 / 38.57 GiB
Free PE / Size 0 / 0
VG UUID
qa6jwq-5gTp-6mMH-IWl9-OrEK-HjWc-pbaFsa
What is going on and how do I fix this? The size of the
ghost pv (31Gb) is showing up as the size of the clone's
vg whereas the pv for the cloned instance is only 8Gb.
I am only using virt-manager to manage disk storage for
these guests and I have no idea why or how this mismash is
happening. Any ideas?
--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
Hi,
I have installed CentOS 6.2 x86_64 and KVM/Qemu and I have some troubles
with network.
The first case:
-----------------
The configuration is:
- CentOS 6.2 (hardware machine) has one Ethernet interface with static
IP address (eth0).
- Virtual machine has bridge network interface with eth0 interface on
CentOS 6.2. This bridge is created using wizard integrated with
virt-manager.
- Bridge interface on virtual machine receives IP addres from DHCP
server. IP address is received well.
- All firewalls and SELinux are closed.
Virtual machine has no any connection with CentOS 6.2 hardware machine,
and CentOS 6.2 has no any connection with virtual machine. But every
others machines have all connection with both virtual and CentOS 6.2
machine.
The question: Why?
The second case:
----------------------
The configuration is:
- On CentOS 6.2 (hardware machine) I created bridhe (br0) with eth0
hardware interface. Two situations:
+ I force two different IP addresses for eth0 and br0 interfaces -
there no any connection with this server. So I don't test this case.
+ I forced one IP addres for both eth0 and br0 interfaces. ifconfig
shows that eth0 has no IP address, br0 has IP address. So, I test this
case.
- I use Red Hat documentation:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html-single/Vi…)
- On virtual machine I created bridge connection with br0 interface on
CentOS 6.2.
- All machines have all connections.
- All firewalls and SELinux are closed.
NFS server on CentOS 6.2 (hardware machine) doesn't serve exported file
system.
The questions:
1. Is it proper that eth0 has no IP addres and bridge (br0) interface
has?
2. What about my NFS? NFS server worked well before starting
virtualization.
Who can me help with my troubles?
Regards,
Radek
I'm trying to set up a xen guest on a machine that already has a few others
running (not set up by me). It fails with the following error:
+ virt-install -r 512 -n host5 -f /dev/vol0/lvol5 --nographics -p --os-type=linux --os-variant=rhel5 --accelerate -l http://bla -x ks=http://bla/ks.cfg -m MA:CA:DD:RE:SS
Traceback (most recent call last):
File "/usr/bin/virt-install", line 31, in ?
import libvirt
File "/usr/lib64/python2.4/site-packages/libvirt.py", line 12, in ?
import cygvirtmod as libvirtmod
ImportError: No module named cygvirtmod
This is a standard CentOS 5.4 x86_64 install, and I have not been able to
find a solution on the web.
I'm running CentOS 6 x86_64 domU on CentOS 5 x86_64 dom0
The host is running 2.6.18-238.19.1.el5xen
The guest is 2.6.32-220.2.1.el6.x86_64
'xm list' on the host shows 512mb memory allocated.
'free -m' in the guest shows 224mb memory.
'xm mem-set 21 1024' (21 being the domU ID), 'xm list' now shows 1024mb, but in the guest I now see 736mb.
Anyone else seeing this? Any ideas on how to fix it?
Chris
--
Chris Wik
Anu Internet Services Ltd
www.cwik.ch | www.anu.net