Re: [CentOS-virt] Failing to start or create VM, cannot connect to hypervisor host - virt

18 Jun 2012

      Responding to the daily digest again.
Well, I finally pinpointed the problem and fixed it.  But to answer you 
curiosity question, the host is running a hardware raid 5 (Dell T610 with a 
PERC H700).
My problem was a result of a change made to my network configuration almost 
2 weeks ago.  When I brought the network backup, I just ifup'ed the em1 and 
br0 interfaces and everything was working.  At least everything that was 
obvious at the time.  In reality, I didn't have a loopback interface 
working, and I hadn't needed to look any any of my guests through 
VirtManager since then to notice the problem.  As soon as I issued a service 
network restart to the host system, everything came back normal and I was 
able to connect to a console on my guest systems.  It took hours of tracking 
down false leads from the log files, and learning how to interpret others 
before I got this pinpointed.  I am now very thankful for keeping fairly 
detailed notes on all the changes to my system, as cross checking the logs 
against my notes helped my pinpoint the problem.
Jeff
...

Message: 2
Date: Sun, 17 Jun 2012 11:19:41 -0700
From: aurfalien aurfalien@gmail.com
Subject: Re: [CentOS-virt] Failing to start or create VM, cannot
connect to hypervisor host
To: Discussion about the virtualization on CentOS
centos-virt@centos.org
Message-ID: B7DCC3C3-C847-4E18-B016-4FBA20F12815@gmail.com
Content-Type: text/plain; charset=us-ascii
Curious, is this system (the host) on a software raid?

aurf

On Jun 17, 2012, at 10:35 AM, jboyce@meridianenv.com wrote:
...
Sorry for the mess on the message threading; responding from the daily
digest.
Thanks for the suggestion.  I'll research any option that may provide a
lead to a solution right now.  I have not tried to touch
/etc/qemu/krb5.tab.  After a little research on what this file is for, I
don't see that it would help me since I have never set up the system to
use kerberos.  I don't even have an /etc/qemu directory on my host.  I
would suspect that part of the error is a side effect of something else
that is the main cause.  Please keep coming with the suggestions though.
I am working on a plan B to make the data files available to the staff on
Monday morning.
Jeff
...
Message: 1
Date: Sat, 16 Jun 2012 14:14:04 -0700 (PDT)
From: jboyce@meridianenv.com
Subject: [CentOS-virt] Failing to start or create VM, cannot connect
to hypervisor host To: centos-virt@centos.org
Message-ID:
21752.184.157.241.237.1339881244.squirrel@www.meridianenv.com
Content-Type: text/plain;charset=iso-8859-1
Greetings -
I shutdown one of my Centos 6.2 VMs for some offline maintenance and am
now unable to get it to restart.  I am also unable to create and start a
new VM.  The host system is Centos 6.2, fully up to date.  I have been
searching Google for two days and have not been successful in getting a
VM
to start.  I have restarted libvirtd, but did not want to shutdown my 
other
two running VMs and reboot my host system, unless I felt confident that 
it
would solve the problem.  The other two VMs on the box are still running
fine, but I am also unable to connect to a console in VirtManger with
them.  I was able to connect to all these VM consoles from the host 
system
just a week or so ago.  The only changes that I had recently made to the
host system was to add some restrictions in my /etc/hosts.allow and
hosts.deny files and installed Fail2ban.  I have reversed all of the
changes to my hosts.allow/deny files and have stopped the Fail2ban
service.  Still unable to start or create a VM.  I am at the end of my
rope here and am hoping someone will be able to direct me with where I
should be looking next.  My error and log messages are listed below.
Thanks.
Jeff
Trying to start the VM using VirtManager at the console of the host
results in the following error.
error:  Failed to start domain SequoiaVM
error:  internal error process exited while connecting to monitor: char
device redirected to /dev/pts/7 inet_listen_opts:  bind(ipv4, 127.0.0.1,
5902) : Cannot assign requested
address inet_listen_opts:  FAILED
Each attempt at a restart results in the following set of messages in
/var/log/messages
Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state
Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
/etc/qemu/krb5.tab: No such file or directory
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
NETDEV_UNREGISTER vnet2
Log messages in /var/log/libvirt/libvirtd.log are similar to the console
2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 :
End of file while reading data: Input/output error
2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 :
failed to connect to monitor socket: No such process 2012-06-16
20:15:23.672+0000: 32176: error :
qemuProcessWaitForMonitor:1289 : internal error process exited while
connecting to monitor: char device redirected to /dev/pts/7
inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested
address inet_listen_opts: FAILED
Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar
2012-06-16 13:45:04.572: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,
nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot order=dc,menu=on -drive
file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,c
ache=none,aio=native -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
tio-disk0 -drive
file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,c
ache=none,aio=native -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=vir
tio-disk1 -drive
file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,ca
che=none,aio=native -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=vir
tio-disk2 -drive
file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,c
ache=none,aio=native -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=vir
tio-disk3 -drive
file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format=
raw,cache=none,aio=native -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=vir
tio-disk4 -drive
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,add
r=0x3 -chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device 
usb-tablet,id=input0
-vnc 127.0.0.1:2 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa char device redirected
to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
requested address inet_listen_opts: FAILED
2012-06-16 13:45:04.872: shutting down
I attempted to create a new VM (TempSequoiaVM) via VirtManager which 
also
failed to be created and started.  These are the messages in
/var/log/libvirt/qemu/TempSequoiaVM.log
2012-06-16 13:15:23.168: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,se
rver,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc
-no-reboot
-no-shutdown -drive
file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,c
ache=none,aio=native -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
tio-disk0,bootindex=2 -drive
file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,add
r=0x3 -chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device 
usb-tablet,id=input0
-vnc 127.0.0.1:2 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device redirected
to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
requested address inet_listen_opts: FAILED
2012-06-16 13:15:23.673: shutting down