[CentOS-virt] Failing to start or create VM, cannot connect to hypervisor host

Sat Jun 16 21:14:04 UTC 2012
jboyce at meridianenv.com <jboyce at meridianenv.com>

Greetings -

I shutdown one of my Centos 6.2 VMs for some offline maintenance and am
now unable to get it to restart.  I am also unable to create and start a
new VM.  The host system is Centos 6.2, fully up to date.  I have been
searching Google for two days and have not been successful in getting a VM
to start.  I have restarted libvirtd, but did not want to shutdown my
other two running VMs and reboot my host system, unless I felt confident
that it would solve the problem.  The other two VMs on the box are still
running fine, but I am also unable to connect to a console in VirtManger
with them.  I was able to connect to all these VM consoles from the host
system just a week or so ago.  The only changes that I had recently made
to the host system was to add some restrictions in my /etc/hosts.allow and
hosts.deny files and installed Fail2ban.  I have reversed all of the
changes to my hosts.allow/deny files and have stopped the Fail2ban
service.  Still unable to start or create a VM.  I am at the end of my
rope here and am hoping someone will be able to direct me with where I
should be looking next.  My error and log messages are listed below. 
Thanks.

Jeff




Trying to start the VM using VirtManager at the console of the host
results in the following error.

error:  Failed to start domain SequoiaVM
error:  internal error process exited while connecting to monitor: char
device redirected to /dev/pts/7
inet_listen_opts:  bind(ipv4, 127.0.0.1, 5902) : Cannot assign requested
address
inet_listen_opts:  FAILED


Each attempt at a restart results in the following set of messages in
/var/log/messages

Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state
Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
/etc/qemu/krb5.tab: No such file or directory
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
NETDEV_UNREGISTER vnet2


Log messages in /var/log/libvirt/libvirtd.log are similar to the console

2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 :
End of file while reading data: Input/output error
2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 :
failed to connect to monitor socket: No such process
2012-06-16 20:15:23.672+0000: 32176: error :
qemuProcessWaitForMonitor:1289 : internal error process exited while
connecting to monitor: char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address
inet_listen_opts: FAILED


Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar

2012-06-16 13:45:04.572: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot order=dc,menu=on -drive
file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
-drive
file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2
-drive
file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3
-drive
file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4
-drive
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa
char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address
inet_listen_opts: FAILED
2012-06-16 13:45:04.872: shutting down


I attempted to create a new VM (TempSequoiaVM) via VirtManager which also
failed to be created and started.  These are the messages in
/var/log/libvirt/qemu/TempSequoiaVM.log

2012-06-16 13:15:23.168: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot
-no-shutdown -drive
file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address
inet_listen_opts: FAILED
2012-06-16 13:15:23.673: shutting down