Nico Kadel-Garcia nkadel@gmail.com writes:

On Sun, Jun 1, 2014 at 8:45 PM, lee lee@yun.yagibdah.de wrote:

...

Hi,

what is the proposed way to create domU guests on centos 6.5? At first

[...]

...

Then I followed redhat documentation which suggests to use virt-manager --- which doesn`t work because servers don`t have GUIs. So I finally managed to create a guest with virt-install. I can start and stop the guest (which is also running centos), though I don`t think this is the right way to create one.

So how exactly are you supposed to create guests?

Servers *can* have GUI's. Even if you don't want to install the full Gnome/KDE/display manager toolkits, it's possible to set up enough to run X based applications form another host.

Yes, they /can/, and IMO a server shouldn`t have and shouldn`t need one.

And virt-manager can be run from a client, with authenticated access to the libvirt server, though I've generally not done that.

Yes, I tried that and it didn`t work.

If you don't want to bother with that, you'll need to learn 'virsh', which is the actual tool that libvirt uses to do almost everything.

That`s what I`m using now. Is virsh what centos users are supposed to use? The documentation on the xen project wiki seems to indicate that xen users are supposed to use xl.

[...]

...

Do I have to set up shorewall (or the like) on dom0 to be able to handle network access for guests? Would I need to create a bridge for every guest to be able to handle them separately for firewalling purposes because otherwise packets circumvent firewall rules by directly going over the bridge? If so, why are bridges needed?

You need to pick. One approach is to set up a bridged connection with one VM, with a second localized VLAN connection,

How do you make such a localised connection?

In this case, the VM with the firewall needs to have access to an ethernet interface to do pppoe, a second interface to the LAN and to a third one for a DMZ. The VM with the firewall will be on the same physical host as the VMs in the DMZ. There will also be VMs in the LAN on the same physical host.

This seems to require three bridges, and the firewall VM would need access to all of them, unless the dom0 is doing some of the routing. How do you enable access to several bridges for a VM?

ATM, I`m trying to understand how this networking stuff with bridges basically works.

and run shorewall or other firewalls on that VM to manage connections to the rest of the VM's. This leaves your bandwidth trapped at the capacity of that firewall VM, but it's not an uncommon soluiton, especially when running complex firewalls and/or proxies in small environments.

Hm, I never thought of that. What kind of limit do I have to imagine? For external connections, the highest bandwidth is 1GB LAN. So a limiting factor for internal connections is probably much more important, like a VM accessing a database running in another VM. The database will probably be so small that it can be kept mostly in memory, so disk access won`t slow things down much.

How much information does the host system have available about the VMs for scheduling purposes? Like when there is a VM for firewalling/routing, a VM with a database and another VM with an application accessing the database, all three VMs need resources. Delay any of them or give them resources at the wrong moment, and in the end performance will be diminished.

Suppose each of the three VMs has one CPU assigned, and more CPUs are physically available while other VMs happen to be idle. So three CPUs are busy and another five CPUs remain idle so that they are available just in case one of the idle VMs needs to do something? Or will they be used to speed up the running VMs?

Are there situations in which overcommitment of CPUs is advisable? Like when I have 8 CPUs and 4 VMs, I could assign 2 CPUs to each VM. But when I can expect that 95% or so of the time 2 VMs will be idle when 2 others are running, won`t I be better off to assign 4 CPUs to each VM? Or is this done automatically, and the number of CPUs specified for a VM is only a minimum?

Whether you need bridges then depends on where your firewall is. If it lives on another host on your network, yes, your guests need bridges. If it's on a VM with two connections, as I described above, it's potentially much easier to set up on a single firewall VM. But migrating the firewall among multiple VM servers means establishing, and maintaining, a multiple VM server internal network, and if doing that, *THOSE* might mandate bridges.

Moving is another thing I haven`t thought about yet. It`s not a requirement in this case, though it would be a good option to have.

How do you deal with memory overcommitment? Suppose I set up a VM that does the firewalling and routing. Memory requirements for this are low.

I also want to use squid (2.7), with a fairly large cache and url rewriting (which hasn`t been ported to 3.x yet). That requires memory (and file I/O).

I also need a file server.

So what do I do? Use one VM for firewall, one for squid, one for a file server that also provides squid with its cache? Or run the file server directly on the host? Or run firewall and squid in the same VM and give this VM some more memory? Or don`t use the file server to supply squid and keep the cache in the same VM squid runs in, perhaps giving it even more memory?

I could also have the firewall in it`s own VM (which might be a good idea due to security) and use another VM for the routing and to run squid, an MTA and other basic services which are accessible from the LAN only. Hm, this actually makes the most sense to me. But then, what do I do with the web server?

What is good practise?

...

I would understand doing things like adding those guests that are visible to the LAN only to the same bridge to have them all reachable likewise. When doing that, it would seem to make sense to use a different subnet for guests in the DMZ.

It Depends(tm).

Depends on? ;)

...

All the documentation tells you many different things, none of them work and it`s totally confusing. Is there any /good/ documentation somewhere?

I suggest what you need to accomplish first. Do you have, or want to build, firewalls? Are you isolating DMZ hosts or public facing webservers that need heightened isolation and security?

Well, I don`t really want to build firewalls; having one is merely a requirement, and being able to do some traffic shaping can be nice. I`m running a web server which is reachable from the outside (on a non-default port, so it`s not exactly public) and needs to be isolated, and I`m running an MTA. I`m thinking about adding IMAPs so clients can get their mail through that.

How isolated does a web server need to be? I`m thinking of adding some game servers in their own VM later on, too. But CPU and memory are limited resources ...

Since I need to start somewhere, what needs to be accomplished is probably something like this:

dom0 use that as file server directly, or use a VM as file server?

dom1 pppoe, firewall

dom2 dns, router, squid, MTA, IMAPs, perhaps DHCP

dom3 webserver game server(s) mumble server

dom4 internal use (like distcc, a multi-user X server for clients to connect to, experimentation ...) mysql server

The server has 2x4 CPUs and 8GB of RAM. Unless something speaks against it, I`d prefer to use dom0 as file server because it seems easier to set up, with direct access to the storage volume.

As to resources:

|------+------------------------------+-----------+------------| | dom | RAM | CPUs | scheduling | |------+------------------------------+-----------+------------| | dom0 | how do I assign memory here? | | ++ | | dom1 | 512--768 MB | 1 CPU | - | | dom2 | 2GB? | 1 CPUs | - | | dom3 | 2GB? | 1--3 CPUs | = | | dom4 | 4GB? | 3--5 CPUs | = | |------+------------------------------+-----------+------------| | 5 | 9GB | 8 CPUs | == | |------+------------------------------+-----------+------------|

The file server would run on dom0. What should I set for dom0_mem? 1--4GB?

Dom4 doesn`t need to be up all the time. From what I`ve been reading, you can overcommit memory and it`s a bad idea to overcommit CPUs.

It it better to give VMs less memory (to some point) and let them use their swap files, or is it better to give them a bit more and overcommit in total (to some point) so that dom0 may swap?

It also seems that scheduling means that VMs /can/ get more time when they need it and don`t get it when not. How does that interfere with CPU assignments?

The VMs only need to sustain a very low minimum throughput because usage will be more like short-term spikes on one or another. Would it make sense to overcommit CPUs in this szenario?

Does this make sense or should I do things differently?

George Dunlap dunlapg@umich.edu writes:

On Mon, Jun 2, 2014 at 1:45 AM, lee lee@yun.yagibdah.de wrote:

...

Hi,

what is the proposed way to create domU guests on centos 6.5? At first I tried to follow the documentation on the xen project website which recommends using xl. I created a config file and ended up with getting a message that the kernel is not bootable when trying to create a guest. I also had to stop some daemon (xend?) because it said that xl isn`t compatible with it and the daemon must be stopped first.

I understand how frustrating it can be to be dealing with old / inaccurate documentation. But I'm not sure how we're supposed to help you if you don't give any details about what you did and exactly how it failed.

I was merely trying to create a VM on a centos host, using xen. Hence my question what "the centos way" of doing this (without a GUI) is.

By trial and error, I found that creating them with virsh-install works. Yet it seems to me as if virsh is an additional layer which makes dealing with VMs a lot easier at the cost of increased resource usage. My intention was to avoid this and to create VMs "the xen way" --- which apparently doesn`t work with centos.

If the instructions you followed don't work, then either:

• You misunderstood something / accidentally left out a step /

mistyped something from the documentation.

• There's a bug in the CentOS implementation of Xen that needs to be fixed
• There's a bug in the documentation that needs to be fixed

There seem to be at least three different ways to create VMs, two of which have been used at different times with xen, while virsh tries to give users a common interface which isn`t exactly needed when you want to stick to xen and therefore do things "the xen way".

But then, I guess every distribution has their own way to make things more complicated, so a common interface like virsh seems to be the most efficient way.

If you describe which bit of documentation on the Xen website you tried to follow, what you were trying to do, and what happened, then we can figure out which of those it is and address the issue.

Well, I didn`t really care about the documentation that didn`t work. Like I said, using xl didn`t work because it only says the kernel can`t be booted; xm seems to be deprecated (yet still used with centos), and virsh-install works.

The xend thing is an unfortunate -- as a project we're trying to move away from it, but there are still a large number of CentOS users who use it. Trying to make both new users and old users happy is a bit of a hard balancing act.

There could be different packages in centos, one set featuring xend, the other featuring whatever replaces it.

I`ve come to like virsh because it`s pretty straightforward and actually works. From that point of view, it doesn`t matter to me whether xend is used or xl or xm or something else. It only matters when I`m concerned about what might be more efficient or have other advantages relevant to me.

Since xend and xm are appartenly deprecated and will or have been replaced, I don`t see much point in learning them and figuring out how they might be advantageous. I rather learn to use what is current and may be used in the future.

George Dunlap dunlapg@umich.edu writes:

...

On Wed, Jun 4, 2014 at 3:12 AM, lee lee@yun.yagibdah.de wrote:

...

George Dunlap dunlapg@umich.edu writes:

I was merely trying to create a VM on a centos host, using xen. Hence my question what "the centos way" of doing this (without a GUI) is.

By trial and error, I found that creating them with virsh-install works. Yet it seems to me as if virsh is an additional layer which makes dealing with VMs a lot easier at the cost of increased resource usage. My intention was to avoid this and to create VMs "the xen way" --- which apparently doesn`t work with centos.

The Xen project itself doesn't have anything to build disk images; just as KVM itself doesn't have anything to build disk images. It leaves that to higher-level tools.

Creating a VM and getting the network to work are the biggest problems I`m having with this. Even with virt-install, you cannot use a centos minimal CD ISO because it`ll say that something is missing. With the full installer DVD ISO you get to the point where the installer asks you where the installation media is --- and then you`re screwed because you *must* have network access from within the VM at that point.

The installer lets you set up network access, but only once. If you don`t get it right the first time, you have to start over.

Nobody tells you how to get the network access to work. Suppose the dom0 has an IP of 192.168.10.10 and you tell the installer to use 192.168.10.20. That seems to make sense, but there`s no way to get network access from within the VM with that. I still don`t understand why and why this is so awfully difficult.

You have to use something like 192.168.20.20 for the VM and make sure routing on dom0 is set up accordingly. You have to use an http server or nfs on dom0 to make the installation media available available to the VM. If the http/nfs server is not on dmo0, you have to set up forwarding to make it reachable for the VM. You have to make sure to disable the rudimentary firewall dom0 has to be able to access it from within the VM.

You have to figure out that the installer doesn`t want the installation media, which is the ISO file. No, you have to mount the ISO, copy everything in it to a directory and then export that directory via nfs so that finally, if you got network access to work, the installer can proceed.

Then you need to figure out how to get the bridge set up when starting the system. I created an ifcfg file for it and am told that bringing up that interface is delayed because it doesn`t seem to exist.

There doesn`t seem to be any documentation about the file apparently used by xend to configure the VM. Documentation tells you to use virsh edit <domU name>, and with that, I`m seeing my changes *not* being applied even after restarting the VM. I`d edit the file directly (somewhere under /etc/xend/domains or so), but without documentation about what`s in it, I rather avoid that.

How do you set a memory range (min--max) for the VM with virsh? There doesn`t seem to be a way to do that, and even if there was, how do I make it so that changes through virsh edit are applied?

I know some people use virsh to install the guest, and then use the xl command-line to manage it after that; but I haven't tested that.

It seemed to me that xend and xl are incompatible.

It would be good to have recommendations on the wiki for a simple, standardized way to create guests in CentOS.

Yes, that`s exactly what I`m asking about all the time: The centos way of creating guests.

As it is, you have to piece together hundreds of bits of documentation and by trial and error somehow figure out some way that gets you what you want.

...

...

If you describe which bit of documentation on the Xen website you tried to follow, what you were trying to do, and what happened, then we can figure out which of those it is and address the issue.

Well, I didn`t really care about the documentation that didn`t work. Like I said, using xl didn`t work because it only says the kernel can`t be booted; xm seems to be deprecated (yet still used with centos), and virsh-install works.

*I* care about the documentation that didn't work, so that other people don't trip over the same thing. :-) If you've walked this path and become frustrated, there are probably a dozen other people who have also walked it and just not said anything.

I`d gladly tell you which documentation it was if I could remember. Some of the documentation on the xen wiki seems to be outdated or a mess because it evolved over time and how things are supposed to be done has changed.

Almost all documentation recommends you to create a file to keep the VM in. I was planning to do that until I found some other documentation telling me that it`s a bad idea because file access like that is relatively slow and you`re much better off using LVM volumes instead. Besides being faster, they have other advantages as well, so I`m doing it that way.

Once I figured out how to have VMs started automatically, I need to find out how to give a VM access to a physical interface to do pppoe over it and how to give it access to two more virtual interfaces ...

Is there a good documentation that explains this network stuff with VMs? I still don`t really have a clue how to do that.

On 06/06/2014 06:23 AM, lee wrote:

George Dunlap dunlapg@umich.edu writes:

...

*I* care about the documentation that didn't work, so that other people don't trip over the same thing. :-) If you've walked this path and become frustrated, there are probably a dozen other people who have also walked it and just not said anything.

To give an example:

https://www.centos.org/docs/5/html/5.1/Deployment_Guide/s1-networkscripts-st...

[root@charon ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE="eth0" BOOTPROTO=none DNS1="192.168.178.20" IPADDR="192.168.1.1" NETMASK="255.255.255.255" NM_CONTROLLED=no ONBOOT="yes" TYPE="Ethernet" UUID="1b645d25-9f66-4335-ba0b-939cdd9f553f"

[root@charon ~]# cat /etc/sysconfig/network-scripts/route-eth0 192.168.178.0/24 via 192.168.1.1 eth0

syntax error. you need a "dev" before eth0. which actually is useless to append anyway because the interface is already specified in the file name

default 192.168.178.200 dev eth0

syntax error. you miss a "via" before the IP of the gateway

[root@charon ~]# service network restart Shutting down interface eth0: [ OK ] Shutting down loopback interface: [ OK ] Bringing up loopback interface: [ OK ] Bringing up interface eth0: Determining if ip address 192.168.1.1 is already in use for device eth0... Error: either "to" is duplicate, or "eth0" is a garbage.

explained by the syntax error from the first line in route-eth0

Error: either "to" is duplicate, or "192.168.178.200" is a garbage.

explained by the syntax error from the second line in route-eth0

[ OK ] [root@charon ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [root@charon ~]#

explained by the fact that your route-eth0 file doesn't contain any correct line

On 06/07/2014 03:03 AM, lee wrote:

Manuel Wolfshant wolfy@nobugconsulting.ro writes: [..] You're right, I overlooked the 'dev'.

Why doesn't the error message simply say "syntax error" and perhaps even points out that "dev" might be missing?

Because the program "ip" is pretty cryptic, despite being way more powerful than most people know

There is no 'to' in the configuration file anywhere, and saying that "eth0" might be "a garbage" isn't helpful in any way. This isn't any better than failing silently or just printing "error".

Each line of the route-eth* file(s) is passed ad-literam to ip route commands so all error messages that you see come from ip. Examine /etc/sysconfig/network-scripts/ifup-routes for details. In your case, you should test the content of route-eth* by using: ip route add $EachLineOneByOne

I would like to make a bug report about this so that the useless error message may be changed. But what package should the report refer to?

You could file a RFE either against iproute which actually triggers the messages you've seen or against initscripts (ifup-routes is provided by it - you could ask for a better parser ).