Responding to the daily digest again. Well, I finally pinpointed the problem and fixed it. But to answer you curiosity question, the host is running a hardware raid 5 (Dell T610 with a PERC H700). My problem was a result of a change made to my network configuration almost 2 weeks ago. When I brought the network backup, I just ifup'ed the em1 and br0 interfaces and everything was working. At least everything that was obvious at the time. In reality, I didn't have a loopback interface working, and I hadn't needed to look any any of my guests through VirtManager since then to notice the problem. As soon as I issued a service network restart to the host system, everything came back normal and I was able to connect to a console on my guest systems. It took hours of tracking down false leads from the log files, and learning how to interpret others before I got this pinpointed. I am now very thankful for keeping fairly detailed notes on all the changes to my system, as cross checking the logs against my notes helped my pinpoint the problem. Jeff > ------------------------------ > > Message: 2 > Date: Sun, 17 Jun 2012 11:19:41 -0700 > From: aurfalien <aurfalien at gmail.com> > Subject: Re: [CentOS-virt] Failing to start or create VM, cannot > connect to hypervisor host > To: Discussion about the virtualization on CentOS > <centos-virt at centos.org> > Message-ID: <B7DCC3C3-C847-4E18-B016-4FBA20F12815 at gmail.com> > Content-Type: text/plain; charset=us-ascii > > Curious, is this system (the host) on a software raid? > > - aurf > On Jun 17, 2012, at 10:35 AM, jboyce at meridianenv.com wrote: > >> Sorry for the mess on the message threading; responding from the daily >> digest. >> >> Thanks for the suggestion. I'll research any option that may provide a >> lead to a solution right now. I have not tried to touch >> /etc/qemu/krb5.tab. After a little research on what this file is for, I >> don't see that it would help me since I have never set up the system to >> use kerberos. I don't even have an /etc/qemu directory on my host. I >> would suspect that part of the error is a side effect of something else >> that is the main cause. Please keep coming with the suggestions though. >> I am working on a plan B to make the data files available to the staff on >> Monday morning. >> >> Jeff >> >> >> >>> Message: 1 >>> Date: Sat, 16 Jun 2012 14:14:04 -0700 (PDT) >>> From: jboyce at meridianenv.com >>> Subject: [CentOS-virt] Failing to start or create VM, cannot connect >>> to hypervisor host To: centos-virt at centos.org >>> Message-ID: >>> <21752.184.157.241.237.1339881244.squirrel at www.meridianenv.com> >>> Content-Type: text/plain;charset=iso-8859-1 >>> >>> >>> Greetings - >>> >>> >>> I shutdown one of my Centos 6.2 VMs for some offline maintenance and am >>> now unable to get it to restart. I am also unable to create and start a >>> new VM. The host system is Centos 6.2, fully up to date. I have been >>> searching Google for two days and have not been successful in getting a >>> VM >>> to start. I have restarted libvirtd, but did not want to shutdown my >>> other >>> two running VMs and reboot my host system, unless I felt confident that >>> it >>> would solve the problem. The other two VMs on the box are still running >>> fine, but I am also unable to connect to a console in VirtManger with >>> them. I was able to connect to all these VM consoles from the host >>> system >>> just a week or so ago. The only changes that I had recently made to the >>> host system was to add some restrictions in my /etc/hosts.allow and >>> hosts.deny files and installed Fail2ban. I have reversed all of the >>> changes to my hosts.allow/deny files and have stopped the Fail2ban >>> service. Still unable to start or create a VM. I am at the end of my >>> rope here and am hoping someone will be able to direct me with where I >>> should be looking next. My error and log messages are listed below. >>> Thanks. >>> >>> >>> Jeff >>> >>> >>> >>> >>> >>> Trying to start the VM using VirtManager at the console of the host >>> results in the following error. >>> >>> error: Failed to start domain SequoiaVM >>> error: internal error process exited while connecting to monitor: char >>> device redirected to /dev/pts/7 inet_listen_opts: bind(ipv4, 127.0.0.1, >>> 5902) : Cannot assign requested >>> address inet_listen_opts: FAILED >>> >>> >>> >>> Each attempt at a restart results in the following set of messages in >>> /var/log/messages >>> >>> >>> Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode >>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state >>> Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file: >>> /etc/qemu/krb5.tab: No such file or directory >>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state >>> Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode >>> Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state >>> Jun 16 13:45:04 earth kernel: libfcoe_device_notification: >>> NETDEV_UNREGISTER vnet2 >>> >>> >>> >>> Log messages in /var/log/libvirt/libvirtd.log are similar to the console >>> >>> >>> 2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 : >>> End of file while reading data: Input/output error >>> 2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 : >>> failed to connect to monitor socket: No such process 2012-06-16 >>> 20:15:23.672+0000: 32176: error : >>> qemuProcessWaitForMonitor:1289 : internal error process exited while >>> connecting to monitor: char device redirected to /dev/pts/7 >>> inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested >>> address inet_listen_opts: FAILED >>> >>> >>> >>> Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar >>> >>> >>> 2012-06-16 13:45:04.572: starting up >>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none >>> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp >>> 4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid >>> 1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev >>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server, >>> nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc >>> -no-shutdown -boot order=dc,menu=on -drive >>> file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,c >>> ache=none,aio=native -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir >>> tio-disk0 -drive >>> file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,c >>> ache=none,aio=native -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=vir >>> tio-disk1 -drive >>> file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,ca >>> che=none,aio=native -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=vir >>> tio-disk2 -drive >>> file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,c >>> ache=none,aio=native -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=vir >>> tio-disk3 -drive >>> file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format= >>> raw,cache=none,aio=native -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=vir >>> tio-disk4 -drive >>> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw >>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 >>> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device >>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,add >>> r=0x3 -chardev pty,id=charserial0 -device >>> isa-serial,chardev=charserial0,id=serial0 -usb -device >>> usb-tablet,id=input0 >>> -vnc 127.0.0.1:2 -vga cirrus -device >>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device >>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device >>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa char device redirected >>> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign >>> requested address inet_listen_opts: FAILED >>> 2012-06-16 13:45:04.872: shutting down >>> >>> >>> >>> I attempted to create a new VM (TempSequoiaVM) via VirtManager which >>> also >>> failed to be created and started. These are the messages in >>> /var/log/libvirt/qemu/TempSequoiaVM.log >>> >>> >>> 2012-06-16 13:15:23.168: starting up >>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none >>> /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp >>> 2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid >>> ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev >>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,se >>> rver,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc >>> base=utc >>> -no-reboot >>> -no-shutdown -drive >>> file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,c >>> ache=none,aio=native -device >>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=vir >>> tio-disk0,bootindex=2 -drive >>> file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw >>> -device >>> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 >>> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device >>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,add >>> r=0x3 -chardev pty,id=charserial0 -device >>> isa-serial,chardev=charserial0,id=serial0 -usb -device >>> usb-tablet,id=input0 >>> -vnc 127.0.0.1:2 -vga cirrus -device >>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device >>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device >>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device redirected >>> to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign >>> requested address inet_listen_opts: FAILED >>> 2012-06-16 13:15:23.673: shutting down >>>