[CentOS-virt] libvirtd hang on CentOS6 after latest updates

Bump. Folks, any ideas?

Cheers
Karel

On 22.5.2018 11:33, Karel Hendrych wrote:
> Hi, I am seeing frequent libvirtd hangs (clients not responding) after 
> last CentOS6-Xen update :
> 
> libvirt-libs-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64
> libgcc-4.4.7-18.el6_9.2.x86_64
> 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64
> libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64
> 10:centos-release-xen-common-8-4.el6.x86_64
> xen-licenses-4.6.6-12.el6.x86_64
> xen-libs-4.6.6-12.el6.x86_64
> libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64
> libstdc++-4.4.7-18.el6_9.2.x86_64
> libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64
> libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64
> libvirt-client-4.1.0-2.xen46.el6.x86_64
> linux-firmware-20171215-82.git2451bb22.el6.noarch
> 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64
> 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64
> libvirt-4.1.0-2.xen46.el6.x86_64
> 10:centos-release-xen-46-8-4.el6.x86_64
> 10:centos-release-xen-44-8-4.el6.x86_64
> tzdata-2018e-3.el6.noarch
> libgomp-4.4.7-18.el6_9.2.x86_64
> kernel-4.9.86-30.el6.x86_64
> xen-hypervisor-4.6.6-12.el6.x86_64
> xen-runtime-4.6.6-12.el6.x86_64
> xen-4.6.6-12.el6.x86_64
> libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64
> 
> Remedy is to kill -9 libvirtd and start again. Issue can be replicated 
> within few domU starts. Usually libvirtd hangs when domU is bringing up 
> xen drivers or something around udev, like:
> 
> xen_netfront: Initialising Xen virtual ethernet driver
> 
> I've been looking into libvirtd strace and debug logs, so far most 
> suspicious in libvirtd debug log is this:
> 
> libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7'
> libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6'
> libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4'
> libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5'
> libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2'
> libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3'
> libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6'
> libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5'
> libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4'
> libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7'
> libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2'
> libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug : 
> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev 
> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3'
> 
> I could not get rid of this by reducing amount of driver queues (not 
> sure if that applies to PV)
> 
> Is someone out there seeing similar issues? Anyone perhaps interested in 
> reviewing full debug log / strace ?
> 
> Cheers
> Karel
> 
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> https://lists.centos.org/mailman/listinfo/centos-virt