I have a python script that monitors the VMs on physical host servers running Xen, but the script doesn't work properly on a server I just built with KVM. The script runs as a non-root user (same on all servers) and simply gathers some details on the status and names of the domains running on the host.
Both Xen and KVM servers are running the same version of libvirt (libvirt-0.6.3-20.1.el5_4) and have the same, default /etc/libvirt/libvirtd.conf config file.
To troubleshoot, I've been running python interactively. Here's how my Xen servers behave:
$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import libvirt conn = libvirt.openReadOnly(None) domains = conn.listDomainsID() print domains
[0, 3, 15, 16, 21, 24, 26, 30, 32, 36, 38, 41, 43, 45, 47, 49, 51, 55, 63, 67]
When I try the same thing on the KVM server:
$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import libvirt conn = libvirt.openReadOnly(None)
14:33:07.303: error : No vport operation path found for host0 14:33:07.320: error : No vport operation path found for host4 14:33:07.325: error : No vport operation path found for host3 14:33:07.367: error : No vport operation path found for host1 14:33:07.368: error : No vport operation path found for host2
domains = conn.listDomainsID() print domains
[]
(The vport stuff is weird, but I found this posting that suggests its harmless (http://www.mail-archive.com/libvir-list@redhat.com/msg17477.html) so I'm ignoring it.)
However, when logged in as root on the KVM server, it works just like my Xen servers:
# python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import libvirt conn = libvirt.openReadOnly(None) domains = conn.listDomainsID() print domains
[1]
Again, on the KVM server, plain old "virsh list" with the debug level set to "2"
$ export LIBVIRT_DEBUG=2 $ virsh list 14:21:06.532: error : No vport operation path found for host0 14:21:06.550: error : No vport operation path found for host4 14:21:06.555: error : No vport operation path found for host3 14:21:06.598: error : No vport operation path found for host1 14:21:06.599: error : No vport operation path found for host2 14:21:06.615: info : No security driver available Id Name State ----------------------------------
Permissions in /var/run/libvirt:
# ls -ld /var/run/libvirt/* srwx------ 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock srwxrwxrwx 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock-ro drwxr-xr-x 2 root root 4096 Jan 21 14:38 /var/run/libvirt/network drwxr-xr-x 2 root root 4096 Oct 20 18:50 /var/run/libvirt/qemu
Can someone provide some tips on what else I can check, if this might be a bug, or point out any mistakes that I might've made? Any help is appreciated.
Thanks, Tom
----- "Tom Georgoulias" tomg@mcclatchyinteractive.com wrote:
Permissions in /var/run/libvirt:
# ls -ld /var/run/libvirt/* srwx------ 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock srwxrwxrwx 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock-ro drwxr-xr-x 2 root root 4096 Jan 21 14:38 /var/run/libvirt/network drwxr-xr-x 2 root root 4096 Oct 20 18:50 /var/run/libvirt/qemu
Can someone provide some tips on what else I can check, if this might be a bug, or point out any mistakes that I might've made? Any help is appreciated.
SELinux?
On 03/11/2010 09:35 AM, Christopher G. Stach II wrote:
----- "Tom Georgoulias"tomg@mcclatchyinteractive.com wrote:
Permissions in /var/run/libvirt:
# ls -ld /var/run/libvirt/* srwx------ 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock srwxrwxrwx 1 root root 0 Feb 5 08:53 /var/run/libvirt/libvirt-sock-ro drwxr-xr-x 2 root root 4096 Jan 21 14:38 /var/run/libvirt/network drwxr-xr-x 2 root root 4096 Oct 20 18:50 /var/run/libvirt/qemu
Can someone provide some tips on what else I can check, if this might be a bug, or point out any mistakes that I might've made? Any help is appreciated.
SELinux?
It's disabled:
# selinuxenabled # echo $? 1
# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - SELinux is fully disabled. SELINUX=disabled # SELINUXTYPE= type of policy in use. Possible values are: # targeted - Only targeted network daemons are protected. # strict - Full SELinux protection. SELINUXTYPE=targeted
Hi Tom,
As someone who suggested to you that you should try this mailing list, I would like to quote some of the discussions that have taken place in the main mailing list ( http://lists.centos.org/pipermail/centos/2010-March/091654.html ).
$ export LIBVIRT_DEBUG=2 $ virsh list 14:21:06.532: error : No vport operation path found for host0 14:21:06.550: error : No vport operation path found for host4 14:21:06.555: error : No vport operation path found for host3 14:21:06.598: error : No vport operation path found for host1 14:21:06.599: error : No vport operation path found for host2 14:21:06.615: info : No security driver available Id Name State
This is empty because, as a non-root user, you are looking at qemu:///session instead of qemu:///system. To be able to access the latter, you would need to do some tweaking.
I wrote: [quote] Look into /etc/libvirt/libvirtd.conf and check out the section "UNIX socket access controls" and make appropriate adjustment. [ I created group 'libvirt' , added myself to the group, and uncommented the line "unix_sock_group = "libvirt"".] Then adjust also the permission bits of the directories and files in /var/run/libvirt to allow access to the group libvirt.
With some luck, you should be able to run the virsh command (for example):
virsh -c qemu:///system list --all [/quote]
Then you wrote: [quote] I read about that on libvirt.org but chose not to make any changes since the Xen server already works with the same config I have on the KVM server. I understood libvirt to be a layer that lets one compatible tool work with many different hypervisors, so I didn't think I'd need to change my libvirt config to work with KVM if it already works with Xen. That might be a bad assumption, though, and I'm not wedded to it. :) [/quote]
This is understandable. However, xen and kvm are different in certain places. For instance, /var/run/libvirt/qemu is unique to kvm, so making changes to this should not affect your xen configuration. <- someone please confirm this.
I wrote: [quote] Also, if you create a guest with the -c qemu:///session option, that would allow non-root user to connect to it. [/quote]
My understanding is that with the -c qemu:///session option, the guest will be set up in that user's directory. Therefore, no root privilege is required.
Comments/help from people who are familiar with kvm welcome. :)
Akemi
On 03/11/2010 11:24 AM, Akemi Yagi wrote:
Look into /etc/libvirt/libvirtd.conf and check out the section "UNIX socket access controls" and make appropriate adjustment. [ I created group 'libvirt' , added myself to the group, and uncommented the line "unix_sock_group = "libvirt"".] Then adjust also the permission bits of the directories and files in /var/run/libvirt to allow access to the group libvirt.
Here are the results of a test where I just made unix_sock_group the same group as an unprivileged test user:
libvirtd.conf changes:
unix_sock_group = "testu" unix_sock_ro_perms = "0777"
[root@kvm ~]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] [root@kvm ~]# ls -l /var/run/libvirt total 16 srwx------ 1 root testu 0 Mar 11 15:03 libvirt-sock srwxrwxrwx 1 root testu 0 Mar 11 15:03 libvirt-sock-ro drwxr-xr-x 2 root root 4096 Mar 8 13:05 network drwxr-xr-x 2 root root 4096 Mar 11 15:00 qemu
(no changes to qemu. Should I expect some?)
With some luck, you should be able to run the virsh command (for example):
virsh -c qemu:///system list --all
As test user "testu":
[testu@kvm ~]$ virsh -c qemu:///system list --all error: unable to connect to '/var/run/libvirt/libvirt-sock': Permission denied error: failed to connect to the hypervisor [testu@kvm ~]$ virsh -c qemu:///session list --all 15:04:05.167: error : No vport operation path found for host0 15:04:05.186: error : No vport operation path found for host4 15:04:05.192: error : No vport operation path found for host3 15:04:05.240: error : No vport operation path found for host1 15:04:05.240: error : No vport operation path found for host2 Id Name State ----------------------------------
[testu@kvm ~]$
Doesn't seem like a socket access issue, the perms for the libvirt-sock-ro are wide open.
Tom
On Thu, Mar 11, 2010 at 12:19 PM, Tom Georgoulias tomg@mcclatchyinteractive.com wrote:
As test user "testu":
[testu@kvm ~]$ virsh -c qemu:///system list --all error: unable to connect to '/var/run/libvirt/libvirt-sock': Permission denied error: failed to connect to the hypervisor [testu@kvm ~]$ virsh -c qemu:///session list --all 15:04:05.167: error : No vport operation path found for host0 15:04:05.186: error : No vport operation path found for host4 15:04:05.192: error : No vport operation path found for host3 15:04:05.240: error : No vport operation path found for host1 15:04:05.240: error : No vport operation path found for host2 Id Name State
[testu@kvm ~]$
Doesn't seem like a socket access issue, the perms for the libvirt-sock-ro are wide open.
It says "unable to connect to '/var/run/libvirt/libvirt-sock'", not to 'libvirt-sock-ro'. Try manually changing it to look like:
srwxrwx--- 1 root testu 0 Mar 11 15:03 libvirt-sock
[This is how mine is configured]
Akemi
On 03/11/2010 03:32 PM, Akemi Yagi wrote:
On Thu, Mar 11, 2010 at 12:19 PM, Tom Georgoulias tomg@mcclatchyinteractive.com wrote:
As test user "testu":
[testu@kvm ~]$ virsh -c qemu:///system list --all error: unable to connect to '/var/run/libvirt/libvirt-sock': Permission denied error: failed to connect to the hypervisor [testu@kvm ~]$ virsh -c qemu:///session list --all 15:04:05.167: error : No vport operation path found for host0 15:04:05.186: error : No vport operation path found for host4 15:04:05.192: error : No vport operation path found for host3 15:04:05.240: error : No vport operation path found for host1 15:04:05.240: error : No vport operation path found for host2 Id Name State
[testu@kvm ~]$
Doesn't seem like a socket access issue, the perms for the libvirt-sock-ro are wide open.
It says "unable to connect to '/var/run/libvirt/libvirt-sock'", not to 'libvirt-sock-ro'. Try manually changing it to look like:
srwxrwx--- 1 root testu 0 Mar 11 15:03 libvirt-sock
[This is how mine is configured]
But isn't that socket used for full domain management? I'm just using a read-only view in the python script ("conn = libvirt.openReadOnly(None)" ), which I thought was using /var/run/libvirt/libvirt-sock-ro.
Anyway, in the interest of an experiment, here's the results of changing group permissions to libvirt-sock:
[root@kvm libvirt]# service libvirtd restart Stopping libvirtd daemon: [ OK ] Starting libvirtd daemon: [ OK ] [root@kvm libvirt]# ls -l total 16 srwxrwx--- 1 root testu 0 Mar 11 15:03 libvirt-sock srwxrwxrwx 1 root testu 0 Mar 11 15:03 libvirt-sock-ro drwxr-xr-x 2 root root 4096 Mar 8 13:05 network drwxr-xr-x 2 root root 4096 Mar 11 15:37 qemu
[testu@kvm ~]$ virsh -c qemu:///system list --all Id Name State ---------------------------------- 8 changed running 12 changed2 running - changed3 shut off
Back to my script:
$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import libvirt conn = libvirt.openReadOnly(None)
15:43:38.001: error : No vport operation path found for host0 15:43:38.020: error : No vport operation path found for host4 15:43:38.026: error : No vport operation path found for host3 15:43:38.069: error : No vport operation path found for host1 15:43:38.069: error : No vport operation path found for host2
domains = conn.listDomainsID() print domains
[]
So the virsh command works as expected, but not when I use python.
Tom
On Thu, Mar 11, 2010 at 12:54 PM, Tom Georgoulias tomg@mcclatchyinteractive.com wrote:
So the virsh command works as expected, but not when I use python.
Because the user running the python script is not part of the group 'testu' ?
Akemi
On 03/11/2010 04:46 PM, Akemi Yagi wrote:
On Thu, Mar 11, 2010 at 12:54 PM, Tom Georgoulias tomg@mcclatchyinteractive.com wrote:
So the virsh command works as expected, but not when I use python.
Because the user running the python script is not part of the group 'testu' ?
Sorry, I should've been more clear about the user running the python script in my previous post. The script runs as the testu user and is in the testu group. I repeated the experiment this morning just to be sure:
[testu@kvm ~]$ ls -l /var/run/libvirt total 16 srwxrwx--- 1 root testu 0 Mar 12 07:45 libvirt-sock srwxrwxrwx 1 root testu 0 Mar 12 07:45 libvirt-sock-ro drwxr-xr-x 2 root root 4096 Mar 8 13:05 network drwxr-xr-x 2 root root 4096 Mar 11 16:17 qemu
[testu@kvm ~]$ virsh -c qemu:///system list --all Id Name State ---------------------------------- 8 changed running 13 changed2 running 14 changed3 running
[testu@kvm ~]$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import libvirt conn = libvirt.openReadOnly(None)
07:45:56.893: error : No vport operation path found for host0 07:45:56.912: error : No vport operation path found for host4 07:45:56.919: error : No vport operation path found for host3 07:45:56.967: error : No vport operation path found for host1 07:45:56.968: error : No vport operation path found for host2
domains = conn.listDomainsID() print domains
[]
On Fri, Mar 12, 2010 at 4:53 AM, Tom Georgoulias tomg@mcclatchyinteractive.com wrote:
[testu@kvm ~]$ virsh -c qemu:///system list --all Id Name State
8 changed running 13 changed2 running 14 changed3 running
[testu@kvm ~]$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly(None) 07:45:56.893: error : No vport operation path found for host0 07:45:56.912: error : No vport operation path found for host4 07:45:56.919: error : No vport operation path found for host3 07:45:56.967: error : No vport operation path found for host1 07:45:56.968: error : No vport operation path found for host2 >>> domains = conn.listDomainsID() >>> print domains []
In the case of virsh, you have a chance to add the -c qemu:///system option (which is required when running the command as non-root). So, the question is if this option can be somehow added in python.
Akemi
On 03/12/2010 11:16 AM, Akemi Yagi wrote:
In the case of virsh, you have a chance to add the -c qemu:///system option (which is required when running the command as non-root). So, the question is if this option can be somehow added in python.
I posted this question on the libvirt-users list and here's the fix: Instead of using: conn = libvirt.openReadOnly(None)
I need to use: conn = libvirt.openReadOnly("qemu:///system")
sh-3.2$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import libvirt conn = libvirt.openReadOnly("qemu:///system") domains = conn.listDomainsID() print domains
[14, 8, 13]
That wass the only change needed to make this work on the KVM server.
Thank you very much for your help with this, I appreciate it!
Tom
On Fri, Mar 12, 2010 at 8:26 AM, Tom Georgoulias tomg@mcclatchyinteractive.com wrote:
On 03/12/2010 11:16 AM, Akemi Yagi wrote:
In the case of virsh, you have a chance to add the -c qemu:///system option (which is required when running the command as non-root). So, the question is if this option can be somehow added in python.
I posted this question on the libvirt-users list and here's the fix: Instead of using: conn = libvirt.openReadOnly(None)
I need to use: conn = libvirt.openReadOnly("qemu:///system")
sh-3.2$ python Python 2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import libvirt >>> conn = libvirt.openReadOnly("qemu:///system") >>> domains = conn.listDomainsID() >>> print domains [14, 8, 13] >>>
That wass the only change needed to make this work on the KVM server.
Thank you very much for your help with this, I appreciate it!
Tom
You are welcome. Glad to hear you've got all sorted out. I am learning, too.
Akemi
-
Akemi Yagi -
Christopher G. Stach II -
Tom Georgoulias