Hi, I am seeing frequent libvirtd hangs (clients not responding) after last CentOS6-Xen update :
libvirt-libs-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64 libgcc-4.4.7-18.el6_9.2.x86_64 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64 libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-common-8-4.el6.x86_64 xen-licenses-4.6.6-12.el6.x86_64 xen-libs-4.6.6-12.el6.x86_64 libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64 libstdc++-4.4.7-18.el6_9.2.x86_64 libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64 libvirt-client-4.1.0-2.xen46.el6.x86_64 linux-firmware-20171215-82.git2451bb22.el6.noarch 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64 libvirt-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-46-8-4.el6.x86_64 10:centos-release-xen-44-8-4.el6.x86_64 tzdata-2018e-3.el6.noarch libgomp-4.4.7-18.el6_9.2.x86_64 kernel-4.9.86-30.el6.x86_64 xen-hypervisor-4.6.6-12.el6.x86_64 xen-runtime-4.6.6-12.el6.x86_64 xen-4.6.6-12.el6.x86_64 libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64
Remedy is to kill -9 libvirtd and start again. Issue can be replicated within few domU starts. Usually libvirtd hangs when domU is bringing up xen drivers or something around udev, like:
xen_netfront: Initialising Xen virtual ethernet driver
I've been looking into libvirtd strace and debug logs, so far most suspicious in libvirtd debug log is this:
libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3'
I could not get rid of this by reducing amount of driver queues (not sure if that applies to PV)
Is someone out there seeing similar issues? Anyone perhaps interested in reviewing full debug log / strace ?
Cheers Karel
Bump. Folks, any ideas?
Cheers Karel
On 22.5.2018 11:33, Karel Hendrych wrote:
Hi, I am seeing frequent libvirtd hangs (clients not responding) after last CentOS6-Xen update :
libvirt-libs-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64 libgcc-4.4.7-18.el6_9.2.x86_64 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64 libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-common-8-4.el6.x86_64 xen-licenses-4.6.6-12.el6.x86_64 xen-libs-4.6.6-12.el6.x86_64 libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64 libstdc++-4.4.7-18.el6_9.2.x86_64 libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64 libvirt-client-4.1.0-2.xen46.el6.x86_64 linux-firmware-20171215-82.git2451bb22.el6.noarch 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64 libvirt-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-46-8-4.el6.x86_64 10:centos-release-xen-44-8-4.el6.x86_64 tzdata-2018e-3.el6.noarch libgomp-4.4.7-18.el6_9.2.x86_64 kernel-4.9.86-30.el6.x86_64 xen-hypervisor-4.6.6-12.el6.x86_64 xen-runtime-4.6.6-12.el6.x86_64 xen-4.6.6-12.el6.x86_64 libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64
Remedy is to kill -9 libvirtd and start again. Issue can be replicated within few domU starts. Usually libvirtd hangs when domU is bringing up xen drivers or something around udev, like:
xen_netfront: Initialising Xen virtual ethernet driver
I've been looking into libvirtd strace and debug logs, so far most suspicious in libvirtd debug log is this:
libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3'
I could not get rid of this by reducing amount of driver queues (not sure if that applies to PV)
Is someone out there seeing similar issues? Anyone perhaps interested in reviewing full debug log / strace ?
Cheers Karel
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Karel,
Thanks for the detailed report. Would you mind re-posting this to the xen-devel mailing list?
Thanks, -Georeg
On Thu, May 24, 2018 at 9:47 AM, Karel Hendrych k+centosvirt@karlos.cz wrote:
Bump. Folks, any ideas?
Cheers Karel
On 22.5.2018 11:33, Karel Hendrych wrote:
Hi, I am seeing frequent libvirtd hangs (clients not responding) after last CentOS6-Xen update :
libvirt-libs-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64 libgcc-4.4.7-18.el6_9.2.x86_64 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64 libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-common-8-4.el6.x86_64 xen-licenses-4.6.6-12.el6.x86_64 xen-libs-4.6.6-12.el6.x86_64 libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64 libstdc++-4.4.7-18.el6_9.2.x86_64 libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64 libvirt-client-4.1.0-2.xen46.el6.x86_64 linux-firmware-20171215-82.git2451bb22.el6.noarch 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64 libvirt-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-46-8-4.el6.x86_64 10:centos-release-xen-44-8-4.el6.x86_64 tzdata-2018e-3.el6.noarch libgomp-4.4.7-18.el6_9.2.x86_64 kernel-4.9.86-30.el6.x86_64 xen-hypervisor-4.6.6-12.el6.x86_64 xen-runtime-4.6.6-12.el6.x86_64 xen-4.6.6-12.el6.x86_64 libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64
Remedy is to kill -9 libvirtd and start again. Issue can be replicated within few domU starts. Usually libvirtd hangs when domU is bringing up xen drivers or something around udev, like:
xen_netfront: Initialising Xen virtual ethernet driver
I've been looking into libvirtd strace and debug logs, so far most suspicious in libvirtd debug log is this:
libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3'
I could not get rid of this by reducing amount of driver queues (not sure if that applies to PV)
Is someone out there seeing similar issues? Anyone perhaps interested in reviewing full debug log / strace ?
Cheers Karel
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
-
George Dunlap -
Karel Hendrych