[CentOS-virt] Failing to start or create VM, cannot connect to hypervisor host

[Jeff Boyce]

Responding to the daily digest again.

Well, I finally pinpointed the problem and fixed it.  But to answer you 
curiosity question, the host is running a hardware raid 5 (Dell T610 with a 
PERC H700).

My problem was a result of a change made to my network configuration almost 
2 weeks ago.  When I brought the network backup, I just ifup'ed the em1 and 
br0 interfaces and everything was working.  At least everything that was 
obvious at the time.  In reality, I didn't have a loopback interface 
working, and I hadn't needed to look any any of my guests through 
VirtManager since then to notice the problem.  As soon as I issued a service 
network restart to the host system, everything came back normal and I was 
able to connect to a console on my guest systems.  It took hours of tracking 
down false leads from the log files, and learning how to interpret others 
before I got this pinpointed.  I am now very thankful for keeping fairly 
detailed notes on all the changes to my system, as cross checking the logs 
against my notes helped my pinpoint the problem.

Jeff


> ------------------------------
>
> Message: 2
> Date: Sun, 17 Jun 2012 11:19:41 -0700
> From: aurfalien <aurfalien at gmail.com>
> Subject: Re: [CentOS-virt] Failing to start or create VM, cannot
> connect to hypervisor host
> To: Discussion about the virtualization on CentOS
> <centos-virt at centos.org>
> Message-ID: <B7DCC3C3-C847-4E18-B016-4FBA20F12815 at gmail.com>
> Content-Type: text/plain; charset=us-ascii
>
> Curious, is this system (the host) on a software raid?
>
> - aurf
> On Jun 17, 2012, at 10:35 AM, jboyce at meridianenv.com wrote:
>
>> Sorry for the mess on the message threading; responding from the daily
>> digest.
>>
>> Thanks for the suggestion.  I'll research any option that may provide a
>> lead to a solution right now.  I have not tried to touch
>> /etc/qemu/krb5.tab.  After a little research on what this file is for, I
>> don't see that it would help me since I have never set up the system to
>> use kerberos.  I don't even have an /etc/qemu directory on my host.  I
>> would suspect that part of the error is a side effect of something else
>> that is the main cause.  Please keep coming with the suggestions though.
>> I am working on a plan B to make the data files available to the staff on
>> Monday morning.
>>
>> Jeff
>>
>>
>>
>>> Message: 1
>>> Date: Sat, 16 Jun 2012 14:14:04 -0700 (PDT)
>>> From: jboyce at meridianenv.com
>>> Subject: [CentOS-virt] Failing to start or create VM, cannot connect
>>> to hypervisor host To: centos-virt at centos.org
>>> Message-ID:
>>> <21752.184.157.241.237.1339881244.squirrel at www.meridianenv.com>
>>> Content-Type: text/plain;charset=iso-8859-1
>>>
>>>
>>> Greetings -
>>>
>>>
>>> I shutdown one of my Centos 6.2 VMs for some offline maintenance and am
>>> now unable to get it to restart.  I am also unable to create and start a
>>> new VM.  The host system is Centos 6.2, fully up to date.  I have been
>>> searching Google for two days and have not been successful in getting a
>>> VM
>>> to start.  I have restarted libvirtd, but did not want to shutdown my 
>>> other
>>> two running VMs and reboot my host system, unless I felt confident that 
>>> it
>>> would solve the problem.  The other two VMs on the box are still running
>>> fine, but I am also unable to connect to a console in VirtManger with
>>> them.  I was able to connect to all these VM consoles from the host 
>>> system
>>> just a week or so ago.  The only changes that I had recently made to the
>>> host system was to add some restrictions in my /etc/hosts.allow and
>>> hosts.deny files and installed Fail2ban.  I have reversed all of the
>>> changes to my hosts.allow/deny files and have stopped the Fail2ban
>>> service.  Still unable to start or create a VM.  I am at the end of my
>>> rope here and am hoping someone will be able to direct me with where I
>>> should be looking next.  My error and log messages are listed below.
>>> Thanks.
>>>
>>>
>>> Jeff
>>>
>>>
>>>
>>>
>>>
>>> Trying to start the VM using VirtManager at the console of the host
>>> results in the following error.
>>>
>>> error:  Failed to start domain SequoiaVM
>>> error:  internal error process exited while connecting to monitor: char
>>> device redirected to /dev/pts/7 inet_listen_opts:  bind(ipv4, 127.0.0.1,
>>> 5902) : Cannot assign requested
>>> address inet_listen_opts:  FAILED
>>>
>>>
>>>
>>> Each attempt at a restart results in the following set of messages in
>>> /var/log/messages
>>>
>>>
>>> Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode
>>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state
>>> Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file:
>>> /etc/qemu/krb5.tab: No such file or directory
>>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
>>> Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode
>>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state
>>> Jun 16 13:45:04 earth kernel: libfcoe_device_notification:
>>> NETDEV_UNREGISTER vnet2
>>>
>>>
>>>
>>> Log messages in /var/log/libvirt/libvirtd.log are similar to the console
>>>
>>>
>>> 2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 :
>>> End of file while reading data: Input/output error
>>> 2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 :
>>> failed to connect to monitor socket: No such process 2012-06-16
>>> 20:15:23.672+0000: 32176: error :
>>> qemuProcessWaitForMonitor:1289 : internal error process exited while
>>> connecting to monitor: char device redirected to /dev/pts/7
>>> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested
>>> address inet_listen_opts: FAILED
>>>
>>>
>>>
>>> Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar
>>>
>>>
>>> 2012-06-16 13:45:04.572: starting up
>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
>>> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
>>> 4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid
>>> 1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,
>>> nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>>> -no-shutdown -boot order=dc,menu=on -drive
>>> file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,c
>>> ache=none,aio=native -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
>>> tio-disk0 -drive
>>> file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,c
>>> ache=none,aio=native -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=vir
>>> tio-disk1 -drive
>>> file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,ca
>>> che=none,aio=native -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=vir
>>> tio-disk2 -drive
>>> file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,c
>>> ache=none,aio=native -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=vir
>>> tio-disk3 -drive
>>> file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format=
>>> raw,cache=none,aio=native -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=vir
>>> tio-disk4 -drive
>>> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>>> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,add
>>> r=0x3 -chardev pty,id=charserial0 -device
>>> isa-serial,chardev=charserial0,id=serial0 -usb -device 
>>> usb-tablet,id=input0
>>> -vnc 127.0.0.1:2 -vga cirrus -device
>>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa char device redirected
>>> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
>>> requested address inet_listen_opts: FAILED
>>> 2012-06-16 13:45:04.872: shutting down
>>>
>>>
>>>
>>> I attempted to create a new VM (TempSequoiaVM) via VirtManager which 
>>> also
>>> failed to be created and started.  These are the messages in
>>> /var/log/libvirt/qemu/TempSequoiaVM.log
>>>
>>>
>>> 2012-06-16 13:15:23.168: starting up
>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
>>> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp
>>> 2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid
>>> ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,se
>>> rver,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc 
>>> base=utc
>>> -no-reboot
>>> -no-shutdown -drive
>>> file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,c
>>> ache=none,aio=native -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir
>>> tio-disk0,bootindex=2 -drive
>>> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>>> -device
>>> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
>>> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device
>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,add
>>> r=0x3 -chardev pty,id=charserial0 -device
>>> isa-serial,chardev=charserial0,id=serial0 -usb -device 
>>> usb-tablet,id=input0
>>> -vnc 127.0.0.1:2 -vga cirrus -device
>>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device redirected
>>> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign
>>> requested address inet_listen_opts: FAILED
>>> 2012-06-16 13:15:23.673: shutting down
>>>