[CentOS-virt] libvirtd hang on CentOS6 after latest updates
Karel Hendrych
k+centosvirt at karlos.cz
Thu May 24 08:47:07 UTC 2018
Bump. Folks, any ideas?
Cheers
Karel
On 22.5.2018 11:33, Karel Hendrych wrote:
> Hi, I am seeing frequent libvirtd hangs (clients not responding) after
> last CentOS6-Xen update :
>
> libvirt-libs-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64
> libgcc-4.4.7-18.el6_9.2.x86_64
> 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64
> libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64
> 10:centos-release-xen-common-8-4.el6.x86_64
> xen-licenses-4.6.6-12.el6.x86_64
> xen-libs-4.6.6-12.el6.x86_64
> libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64
> libstdc++-4.4.7-18.el6_9.2.x86_64
> libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64
> libvirt-client-4.1.0-2.xen46.el6.x86_64
> linux-firmware-20171215-82.git2451bb22.el6.noarch
> 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64
> 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64
> libvirt-4.1.0-2.xen46.el6.x86_64
> 10:centos-release-xen-46-8-4.el6.x86_64
> 10:centos-release-xen-44-8-4.el6.x86_64
> tzdata-2018e-3.el6.noarch
> libgomp-4.4.7-18.el6_9.2.x86_64
> kernel-4.9.86-30.el6.x86_64
> xen-hypervisor-4.6.6-12.el6.x86_64
> xen-runtime-4.6.6-12.el6.x86_64
> xen-4.6.6-12.el6.x86_64
> libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64
>
> Remedy is to kill -9 libvirtd and start again. Issue can be replicated
> within few domU starts. Usually libvirtd hangs when domU is bringing up
> xen drivers or something around udev, like:
>
> xen_netfront: Initialising Xen virtual ethernet driver
>
> I've been looking into libvirtd strace and debug logs, so far most
> suspicious in libvirtd debug log is this:
>
> libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7'
> libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6'
> libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4'
> libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5'
> libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2'
> libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3'
> libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6'
> libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5'
> libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4'
> libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7'
> libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2'
> libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug :
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3'
>
> I could not get rid of this by reducing amount of driver queues (not
> sure if that applies to PV)
>
> Is someone out there seeing similar issues? Anyone perhaps interested in
> reviewing full debug log / strace ?
>
> Cheers
> Karel
>
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> https://lists.centos.org/mailman/listinfo/centos-virt
More information about the CentOS-virt
mailing list