Greetings -
I shutdown one of my Centos 6.2 VMs for some offline maintenance and am now unable to get it to restart. I am also unable to create and start a new VM. The host system is Centos 6.2, fully up to date. I have been searching Google for two days and have not been successful in getting a VM to start. I have restarted libvirtd, but did not want to shutdown my other two running VMs and reboot my host system, unless I felt confident that it would solve the problem. The other two VMs on the box are still running fine, but I am also unable to connect to a console in VirtManger with them. I was able to connect to all these VM consoles from the host system just a week or so ago. The only changes that I had recently made to the host system was to add some restrictions in my /etc/hosts.allow and hosts.deny files and installed Fail2ban. I have reversed all of the changes to my hosts.allow/deny files and have stopped the Fail2ban service. Still unable to start or create a VM. I am at the end of my rope here and am hoping someone will be able to direct me with where I should be looking next. My error and log messages are listed below. Thanks.
Jeff
Trying to start the VM using VirtManager at the console of the host results in the following error.
error: Failed to start domain SequoiaVM error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/7 inet_listen_opts: bind(ipv4, 127.0.0.1, 5902) : Cannot assign requested address inet_listen_opts: FAILED
Each attempt at a restart results in the following set of messages in /var/log/messages
Jun 16 13:45:04 earth kernel: device vnet2 entered promiscuous mode Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering learning state Jun 16 13:45:04 earth qemu-kvm: Could not find keytab file: /etc/qemu/krb5.tab: No such file or directory Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state Jun 16 13:45:04 earth kernel: device vnet2 left promiscuous mode Jun 16 13:45:04 earth kernel: br0: port 4(vnet2) entering disabled state Jun 16 13:45:04 earth kernel: libfcoe_device_notification: NETDEV_UNREGISTER vnet2
Log messages in /var/log/libvirt/libvirtd.log are similar to the console
2012-06-16 19:56:39.807+0000: 32175: error : virNetSocketReadWire:911 : End of file while reading data: Input/output error 2012-06-16 20:15:23.672+0000: 32176: error : qemuMonitorOpenUnix:290 : failed to connect to monitor socket: No such process 2012-06-16 20:15:23.672+0000: 32176: error : qemuProcessWaitForMonitor:1289 : internal error process exited while connecting to monitor: char device redirected to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address inet_listen_opts: FAILED
Log messages in /var/log/libvirt/qemu/SequoiaVM.log are similar
2012-06-16 13:45:04.572: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp 4,sockets=4,cores=1,threads=1 -name SequoiaVM -uuid 1ca3a14b-cd4b-f371-b3fe-1cd518480c09 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/SequoiaVM.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=dc,menu=on -drive file=/dev/vg_mei/lv_sequoiaroot,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/vg_mei/lv_sequoiaswap,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/vg_mei/lv_sequoiavar,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/dev/vg_mei/lv_sequoiahome,if=none,id=drive-virtio-disk3,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3 -drive file=/dev/vg_mei/lv_sequoiaecosystem,if=none,id=drive-virtio-disk4,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4 -drive file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9d:b2:49,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa char device redirected to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address inet_listen_opts: FAILED 2012-06-16 13:45:04.872: shutting down
I attempted to create a new VM (TempSequoiaVM) via VirtManager which also failed to be created and started. These are the messages in /var/log/libvirt/qemu/TempSequoiaVM.log
2012-06-16 13:15:23.168: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -name TempSequoiaVM -uuid ef6c99c7-7fea-e24d-447c-cf98d6ffbec3 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/TempSequoiaVM.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -no-shutdown -drive file=/dev/vg_mei/lv_TempSequoia,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/dev/sr0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:59:9e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 char device redirected to /dev/pts/7 inet_listen_opts: bind(ipv4,127.0.0.1,5902): Cannot assign requested address inet_listen_opts: FAILED 2012-06-16 13:15:23.673: shutting down