Hi all,
following the discussion on about documentation, I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs. Am starting this with a Xen hat, but the discussion should not be specific to this. There are a number of options, but all have some trade-offs
== #1 virt-install ==
Advantages: similar to KVM
Disadvantages: may cause weird issues / confusion with people switching back to xl. The core issue is that with the current version of xen and libvirt, this only works with xm (when xl is used, this can create some undefined behavior). However as we have seen in some recent threads on this list, people tend to mix which can cause problems.
== #2 xen-tools ==
Advantages: Very flexible. Many other distros use xen-tools, so we have lots of beginners docs that just need to be tweaked
Disadvantages: needs porting/packaging for CentOS. Does not work for kvm. Says "xen". (Maybe that's an advantage.) We know that xen-tools works with Fedora (see http://blog.xen.org/index.php/2013/01/24/using-xen-tools-on-fedora/), so the porting effort may be small
Unknowns: What would be needed to make it work for CentOS
== #3 virt-builder (http://libguestfs.org/virt-builder.1.html) ==
Advantages: supports KVM, Xen and other VM inages. Seems easy to use. - if so, it would avoid xm / xl confusion.
Unknowns: Not sure at which level virt-builder integrates with Xen and other hypervisors. It seems to operate at disk image format (similar to xen-tools) . I don't know whether virt-builder is restricted to some hypervisors in RHEL7.
Disadvantages: may need porting/packaging for CentOS. It appears as if it will be in RHEL7, so it may just appear with CentOS 7. If not, some porting work may need to be done.
== #4 Cloud Image from Cloud Image SIG == We could rely on pre-built cloud images from the Cloud Images SIG. People could just download the cloud image once it's done and customize it, rather than installing / building their own.
Advantages: seems easy
Disadvantages: coordination with Cloud Images SIG. May not be flexible enough
I just wanted to start a discussion about this and ask for input. This topic which has come up a number of times in SIG meetings as a facgtor influencing libvirt and other package versions.
Regards Lars
On 10/06/14 12:21 PM, Lars Kurth wrote:
Hi all,
following the discussion on about documentation, I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs. Am starting this with a Xen hat, but the discussion should not be specific to this. There are a number of options, but all have some trade-offs
== #1 virt-install ==
Advantages: similar to KVM
Disadvantages: may cause weird issues / confusion with people switching back to xl. The core issue is that with the current version of xen and libvirt, this only works with xm (when xl is used, this can create some undefined behavior). However as we have seen in some recent threads on this list, people tend to mix which can cause problems.
== #2 xen-tools ==
Advantages: Very flexible. Many other distros use xen-tools, so we have lots of beginners docs that just need to be tweaked
Disadvantages: needs porting/packaging for CentOS. Does not work for kvm. Says "xen". (Maybe that's an advantage.) We know that xen-tools works with Fedora (see http://blog.xen.org/index.php/2013/01/24/using-xen-tools-on-fedora/), so the porting effort may be small
Unknowns: What would be needed to make it work for CentOS
== #3 virt-builder (http://libguestfs.org/virt-builder.1.html) ==
Advantages: supports KVM, Xen and other VM inages. Seems easy to use.
- if so, it would avoid xm / xl confusion.
Unknowns: Not sure at which level virt-builder integrates with Xen and other hypervisors. It seems to operate at disk image format (similar to xen-tools) . I don't know whether virt-builder is restricted to some hypervisors in RHEL7.
Disadvantages: may need porting/packaging for CentOS. It appears as if it will be in RHEL7, so it may just appear with CentOS 7. If not, some porting work may need to be done.
== #4 Cloud Image from Cloud Image SIG == We could rely on pre-built cloud images from the Cloud Images SIG. People could just download the cloud image once it's done and customize it, rather than installing / building their own.
Advantages: seems easy
Disadvantages: coordination with Cloud Images SIG. May not be flexible enough
I just wanted to start a discussion about this and ask for input. This topic which has come up a number of times in SIG meetings as a facgtor influencing libvirt and other package versions.
Regards Lars
I would recommend the default be the libvirtd tools, as it tries to be hypervisor-agnostic, so it's most portable. Also, it provides a good amount of flexibility for various install type, where images/docker is more restricted. Again, speaking about defaults/generalities.
I would, of course, point to the other tools so that users know what exists and what might work best for their use case.
Lars Kurth lars.kurth@xen.org writes:
Hi all,
following the discussion on about documentation, I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs. Am starting this with a Xen hat, but the discussion should not be specific to this. There are a number of options, but all have some trade-offs
Let me wear the hat of the user. The major hurdles were network setup, installing something in a vm, and the chaotic state the documentation is in.
I knew before I started that network setup would be a PITA because years ago, I set up a VM for someone who didn't have a 64bit system to compile a 64bit version of some software. The network setup being so ridiculously difficult has kept me from touching VMs ever again for years. It's just too difficult and not worth the effort unless you're really forced to do it.
As a user, I'm used to get an ISO of an installer or of a life system, put that into a DVD drive or write it to an USB stick and to boot from that to do the installation. Why can't I do that with xen?
== #1 virt-install == == #2 xen-tools == == #3 virt-builder (http://libguestfs.org/virt-builder.1.html) == == #4 Cloud Image from Cloud Image SIG ==
I wouldn't want #4. I want to be able to just use the installer of whatever distribution I'm installing in a VM and a simple way of saying "run this in a VM and let me install". Or perhaps I merely want to run a life system in a VM to try it out, with network access. What easy way is there to do that?
I haven't heared of #3.
#2 and #1 lead to total confusion. Xen doesn't have any way to use an installer ISO to install a VM. The documentation is confusing because it lacks clarity and refers to many versions of xen at the same time which suggest to do things differently so that you don't know how you are supposed to do something. There is no documentation for centos about how to set up a VM.
#2 and #1 use different ways and different formats of configuration files, and apparently #1 doesn't support everything #2 does and brings about an undesirable overhead while apparently being a good choice for users who also use other things than xen and don't want to learn different ways of managing VMs.
When I use xen, I do not need #1, and it seems to make more sense to me to stick with #2.
#2 needs clarity of documentation. I've got an installer ISO. The documentation needs to tell me straightforwardly what I need to do to get a VM running with this installer ISO. I don't care about #1--4 at that point.
It's that simple. More details can be learned over time.
Insofar the documentation needs to deal with different versions of xen, just split the documentation entirely. When I use version X, I do not want to read the documentation for versions A, B, C or Z. That's confusing and I have to ask myself all the time which part of what I'm reading might apply to the software I'm using, and that makes reading the documentation very painful.
It's like you buy a new TV and it comes with 50000 pages of documentation because the manufacturer of the TV mentions everything for every model of TV they ever manufactured, rather than documenting no more than the TV you actually bought. Trying to figure out how to switch to a different channel or how to increase the volume takes at least two days each.
For example: http://wiki.xen.org/wiki/Credit_Scheduler:
"It is now the default scheduler in the xen-unstable trunk."
So is it the default in the version I'm using or what?
"The SEDF and BVT schedulers are still optionally available but the plan of record is for them to be phased out and eventually removed."
What are those and have they been removed in the version I'm using? But then, it probably doesn't matter what they are because they might have been removed anyway. So what is this information supposed to tell me??
"A domain with a weight of 512 will get twice as much CPU as a domain with a weight of 256 on a contended host. Legal weights range from 1 to 65535 and the default is 256."
Where would I change this default? What happens when I set a weight of 100 or of 33? I can only guess that it might be rounded up or down to something like 128 or 64 or 32 or 64. Or perhaps actually floating point math is employed to compute the relations of the different weights? Probably not ...
"Schedule Rate Limiting (added in Xen 4.2)"
That doesn't belong there. It needs to go into the documentation for that particular version of xen.
"Timeslice (added in Xen 4.2) [...] # xl sched-credit -t [n]"
dito
"Usage
The xm sched-credit command may be used to tune the per VM guest scheduler parameters."
So what am I supposed to use? xm or xl? Or something else, maybe virsh?
I always admire the documentation exim has ... As to your original question:
I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs.
I'd suggest to start with splitting the documentation into different versions, one version of documentation for each version of xen. Some way to get a VM running may then be discovered and improved upon.
What actually is the way to do it? I found something that works, and it allows me to install Debian squeeze, from which then I upgrade to wheezy. For anything else, I don't know.
On gio, 2014-06-12 at 07:17 +0200, lee wrote:
Lars Kurth lars.kurth@xen.org writes:
Let me wear the hat of the user. The major hurdles were network setup, installing something in a vm, and the chaotic state the documentation is in.
Wow... "chaotic state" :-O
Don't get me wrong. I know everything can always be improved, and documentation is --for free software in general, which well includes Xen-- often in a position of needing _a_lot_ of improvement.
All that being said, it would be helpful if you could be a bit more specific and help us actually improve it. I think you've said, in another thread, that you requested editor access to the wiki, but don't know what status such request is in (provided I'm not mistaking you with someone else, in which case, sorry! :-P). Can you double check, so that we can see how to move forward and allow you to act instead of only complaining. :-D
As a user, I'm used to get an ISO of an installer or of a life system, put that into a DVD drive or write it to an USB stick and to boot from that to do the installation. Why can't I do that with xen?
Wait, what? Who said you can't do it? Of course you can! In fact, if you tried and it failed, please, report the bug within the appropriate channel, because a bug is what that is, nothing less! :-O
However, I think we're talking about something different here...
== #1 virt-install == == #2 xen-tools == == #3 virt-builder (http://libguestfs.org/virt-builder.1.html) == == #4 Cloud Image from Cloud Image SIG ==
I wouldn't want #4. I want to be able to just use the installer of whatever distribution I'm installing in a VM and a simple way of saying "run this in a VM and let me install". Or perhaps I merely want to run a life system in a VM to try it out, with network access. What easy way is there to do that?
I'm not sure I got what you mean with 'I merely want to run a life system in a VM to try it out'. If you mean trying out Xen in a VM, then of course you need a nested-virt capable platform. If you have one, here's the instructions of how to create a Xen liveCD. I'm afraid both the instructions and the liveCD are Fedora based, but it should not be a big deal to bias them toward CentOS:
http://lists.xen.org/archives/html/xen-users/2014-06/msg00065.html
I also don't know much of #3, but at least #1 and #2, are both meant at doing exactly what you're saying: "install this in a VM", they only happen not to require install DVDs. I understand that DVDs are your preferred method, but that doesn't make it the one _everyone_ prefer!
AFAIUI, when talking about this kind of provisioning, what we're interested is something a lot more automated and, as a consequence, a lot less interactive than an installer.
For example: http://wiki.xen.org/wiki/Credit_Scheduler:
Ok, now we're talking! That's what I meant when asking to 'be specific'! :-)
"It is now the default scheduler in the xen-unstable trunk."
So is it the default in the version I'm using or what?
Good point, I'll fix this.
"The SEDF and BVT schedulers are still optionally available but the plan of record is for them to be phased out and eventually removed."
What are those and have they been removed in the version I'm using?
Ditto.
"Schedule Rate Limiting (added in Xen 4.2)"
That doesn't belong there. It needs to go into the documentation for that particular version of xen.
"Timeslice (added in Xen 4.2) [...] # xl sched-credit -t [n]"
dito
I don't agree. I think they do belnog. Of course, that does not mean we can't link to them from a 'Xen 4.2' page.
I always admire the documentation exim has ... As to your original question:
I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs.
I'd suggest to start with splitting the documentation into different versions, one version of documentation for each version of xen. Some way to get a VM running may then be discovered and improved upon.
Personally, I don't think that would be the optimal split, or something we need to do right now (which does not mean we should not improve docs in other ways). However, I think you're way off the 'original question' which, again, was about provisioning.
Regards, Dario
Dario Faggioli raistlin@linux.it writes:
On gio, 2014-06-12 at 07:17 +0200, lee wrote:
Lars Kurth lars.kurth@xen.org writes:
Let me wear the hat of the user. The major hurdles were network setup, installing something in a vm, and the chaotic state the documentation is in.
Wow... "chaotic state" :-O
Don't get me wrong. I know everything can always be improved, and documentation is --for free software in general, which well includes Xen-- often in a position of needing _a_lot_ of improvement.
All that being said, it would be helpful if you could be a bit more specific and help us actually improve it.
It's really difficult to be any specific because I have to piece together information from whatever sources are available through search engines and by trial and error.
Perhaps take practical questions and try to answer them from the documentation you can find, for example:
I have an installer for distribution X. How do I install the distribution in a VM?
How do I make sure that a particular network interface I have in dom0 will always show up as a particular network interface in domU?
What is an xvda device, what are the impacts of using one instead of, for example, /dev/sda, and how are the devices named when I have several of them because after installing a VM, I want to give it access to another logical volume: xvda1, xvda2, etc., or xvdb, xvdc, or something else?
Why does it say "I need more memory ..." when trying to create a guest with xm? Can't I do memory overcommittment? My dom0 has plenty of swap space.
What are the effects of overcommitting CPUs? Should I rather limit the VMs tightly or give them CPUs so that they can be used as needed instead of being left idle?
Do I need to load a particular module in a VM to be able to give it more memory with xm? What happens when I give a VM more memory and then take it away after a while?
How do I make it so that memory is assigned to VMs dynamically up to their maximum allowed amounts, depending on how much memory a VM currently happens to need?
Why gives "xenpm get-cpufreq-states" no output?
Why do I get an entry "(XEN) physdev.c:168: dom0: wrong map_pirq type 3" in xm dmesg, and what is it supposed to tell me?
When you assume you're new to using xen and try to answer questions like this from whatever documentation you can find, you'll probably see what I mean :)
I think you've said, in another thread, that you requested editor access to the wiki, but don't know what status such request is in (provided I'm not mistaking you with someone else, in which case, sorry! :-P). Can you double check, so that we can see how to move forward and allow you to act instead of only complaining. :-D
I think the request has been successful. I'm still working on getting everything set up and didn't get to look into editing the wiki yet.
As a user, I'm used to get an ISO of an installer or of a life system, put that into a DVD drive or write it to an USB stick and to boot from that to do the installation. Why can't I do that with xen?
Wait, what? Who said you can't do it? Of course you can! In fact, if you tried and it failed, please, report the bug within the appropriate channel, because a bug is what that is, nothing less! :-O
Well, how do you do that? IIRC, I followed http://wiki.xen.org/wiki/Debian_Guest_Installation_Using_Debian_Installer, and it installs squeeze instead of wheezy. Apparently you require a special kernel and initrd.img to install something in a VM --- so how do you install something else?
However, I think we're talking about something different here...
== #1 virt-install == == #2 xen-tools == == #3 virt-builder (http://libguestfs.org/virt-builder.1.html) == == #4 Cloud Image from Cloud Image SIG ==
I wouldn't want #4. I want to be able to just use the installer of whatever distribution I'm installing in a VM and a simple way of saying "run this in a VM and let me install". Or perhaps I merely want to run a life system in a VM to try it out, with network access. What easy way is there to do that?
I'm not sure I got what you mean with 'I merely want to run a life system in a VM to try it out'. If you mean trying out Xen in a VM, then of course you need a nested-virt capable platform.
No, I was assuming that I have a dom0 running and decide, for example, that I want to try out the live DVD of some Fedora spin in another VM.
If you have one, here's the instructions of how to create a Xen liveCD. I'm afraid both the instructions and the liveCD are Fedora based, but it should not be a big deal to bias them toward CentOS:
http://lists.xen.org/archives/html/xen-users/2014-06/msg00065.html
Oh I've seen this page --- it wasn't was I was looking for, though.
I also don't know much of #3, but at least #1 and #2, are both meant at doing exactly what you're saying: "install this in a VM", they only happen not to require install DVDs. I understand that DVDs are your preferred method, but that doesn't make it the one _everyone_ prefer!
An installer is merely what I find or have when I want to install a distribution. Suppose I'd want to try out Mint. I'd look at their website and download one of the versions. I'd burn a DVD or CD or write it to an USB stick and boot from that.
With xen, what do I do instead? It seems I need a special version to get a VM running with it. Where do I get this special version?
AFAIUI, when talking about this kind of provisioning, what we're interested is something a lot more automated and, as a consequence, a lot less interactive than an installer.
Wouldn't that depend a lot on what you're trying to do? Some people suggest to do a basic install and then to copy that.
I always admire the documentation exim has ... As to your original question:
I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs.
I'd suggest to start with splitting the documentation into different versions, one version of documentation for each version of xen. Some way to get a VM running may then be discovered and improved upon.
Personally, I don't think that would be the optimal split, or something we need to do right now (which does not mean we should not improve docs in other ways).
The documentation would be much easier to read without having to filter it for the version of xen that is being used. Why not just copy the current documentation when the next version is released and remove the cruft? Then you can simply say "it goes like this" instead of having to say "it might go like this but only if you're using that version, and perhaps this way will be removed, but you might be using something that could be deprecated in the next version, so don't use this but there are five other ways to do it ...".
For things removed just say 'removed, see feature X', and for new things, mark them as 'new in the current version'. This helps users of previous versions to catch up and saves new users from filtering the cruft.
However, I think you're way off the 'original question' which, again, was about provisioning.
Hm, yes, I didn't understand at first that this a different question, sorry.
On 06/12/2014 05:17 PM, lee wrote:
I knew before I started that network setup would be a PITA because years ago, I set up a VM for someone who didn't have a 64bit system to compile a 64bit version of some software. The network setup being so ridiculously difficult has kept me from touching VMs ever again for years. It's just too difficult and not worth the effort unless you're really forced to do it.
Networking can be confusing until it "clicks" then it all seems to fall into place, there is a certain amount of understanding of the overall picture of how bridge networking (or other types of networking) works that once you get that understanding it becomes a lot simpler. One thing that probably confuses you is that there are certain parts of the networking that are done by the dom0 distro, certain parts are done by Xen scripts and front and backend drivers, and some parts are done by the domu operating system. Understanding how those parts fit together and how different distros do the same thing but perhaps with different configs helps a lot to getting a grasp on the whole.
Obviously better documentation would help with this, but like anything of this nature there is a learning curve that you simply have to get over before you can really become proficient.
As a user, I'm used to get an ISO of an installer or of a life system, put that into a DVD drive or write it to an USB stick and to boot from that to do the installation. Why can't I do that with xen?
You can do that with Xen, KVM, and most other types of virtualization. The idea is that you need to block-attach the disk (or ISO file if you prefer not to use a physical disk) to the VM and then it can be mounted and used as install media (or any other type of media for that matter). This can be done from teh domain config file or from the xm (or xl) command line.
Another quick note specifically for you. I have noticed in past messages that you seem to be having some confusion with xm vs xl commands and some explanation about that is probably in order to clear up that confusion. xm and xl are nearly identical commands to use, but xm is the "old" way and xl the "new" way. Basically put, xm interfaces with a daemon running in the dom0 (xend) which in turn interfaces with the hypervisor, while xl uses libraries to interface directly with the hypervisor and so eliminates the need for xend. You will see documentation refer to one or the other, sometimes almost interchangeably, because the two tools have almost identical usage, but what you should be aware of is that you really need to pick one and stick with it, Xen does not work well if you try to use xm for some commands and xl for others. Since you're new I would recommend xl because xm is deprecated in newer versions of Xen, so if you want to future-proof your knowledge xl is the way to go. Also if you're using xl then you should not be running the xend daemon, this is easily accomplished by using chkconfig and service to turn xend off.
Peter
Peter peter@pajamian.dhs.org writes:
On 06/12/2014 05:17 PM, lee wrote:
I knew before I started that network setup would be a PITA because years ago, I set up a VM for someone who didn't have a 64bit system to compile a 64bit version of some software. The network setup being so ridiculously difficult has kept me from touching VMs ever again for years. It's just too difficult and not worth the effort unless you're really forced to do it.
Networking can be confusing until it "clicks" then it all seems to fall into place, there is a certain amount of understanding of the overall picture of how bridge networking (or other types of networking) works that once you get that understanding it becomes a lot simpler.
I still don't really understand it. For example, why does the dom0 loose network connectivity when you add the physical interface through which it is connected to the network to a bridge? The bridge supposedly connects networks indiscriminately, and it is illogical that the connection goes away when you connect a network to it.
One thing that probably confuses you is that there are certain parts of the networking that are done by the dom0 distro, certain parts are done by Xen scripts and front and backend drivers, and some parts are done by the domu operating system. Understanding how those parts fit together and how different distros do the same thing but perhaps with different configs helps a lot to getting a grasp on the whole.
Obviously better documentation would help with this, but like anything of this nature there is a learning curve that you simply have to get over before you can really become proficient.
Well, yes, learning is one of the reasons for setting up what I'm still working on atm. The networking remains a black box, though. It now works the way I want it to, yet it is too complicated to understand without solid documentation. It's not a matter of there being a learning curve but a matter of needing something to learn from because you can't learn it out of thin air or by trial and error.
As a user, I'm used to get an ISO of an installer or of a life system, put that into a DVD drive or write it to an USB stick and to boot from that to do the installation. Why can't I do that with xen?
You can do that with Xen, KVM, and most other types of virtualization. The idea is that you need to block-attach the disk (or ISO file if you prefer not to use a physical disk) to the VM and then it can be mounted and used as install media (or any other type of media for that matter). This can be done from teh domain config file or from the xm (or xl) command line.
Is somewhere documented how to do that? At some point I understood that once the VM is somehow started, whatever it started from has become inaccessible from inside the VM. Since the centos installer features using something that is reachable through NFS or HTTP, I tried that, but since it was almost impossible to get network access from within a VM, that didn't work so well.
Another quick note specifically for you. I have noticed in past messages that you seem to be having some confusion with xm vs xl commands and some explanation about that is probably in order to clear up that confusion.
That's a result of reading documentation. The documentation isn't clear about what to use how, and when virsh is mixed in because it finally allows you to run an installer within a VM, it's even more confusing. Since the documentation randomly refers to various versions of xen and tells you to use xl in one case and xm in another, and perhaps virsh in yet another case, you can only try what the documentation says and see what happens. That's why I said that the documentation is chaotic.
xm and xl are nearly identical commands to use, but xm is the "old" way and xl the "new" way. Basically put, xm interfaces with a daemon running in the dom0 (xend) which in turn interfaces with the hypervisor, while xl uses libraries to interface directly with the hypervisor and so eliminates the need for xend.
How do I switch to the new way without breaking things? I'm running everything on Debian now, and they're using xm. Centos does, too.
You will see documentation refer to one or the other, sometimes almost interchangeably, because the two tools have almost identical usage, but what you should be aware of is that you really need to pick one and stick with it, Xen does not work well if you try to use xm for some commands and xl for others.
Someone new to xen and reading the documentation doesn't know this, and the documentation confuses them. It tells them things that don't work when they try them, preventing them from making any progress.
Since you're new I would recommend xl because xm is deprecated in newer versions of Xen, so if you want to future-proof your knowledge xl is the way to go. Also if you're using xl then you should not be running the xend daemon, this is easily accomplished by using chkconfig and service to turn xend off.
So you're saying I can just stop xend? What about the settings in /etc/xen/xend-config.sxp, where/how do I apply those?
Debian comes with xen 4.1, so I should be able to upgrade to using xl.
On Fri, Jun 13, 2014 at 8:42 AM, lee lee@yun.yagibdah.de wrote:
Peter peter@pajamian.dhs.org writes:
On 06/12/2014 05:17 PM, lee wrote:
I knew before I started that network setup would be a PITA because years ago, I set up a VM for someone who didn't have a 64bit system to compile a 64bit version of some software. The network setup being so ridiculously difficult has kept me from touching VMs ever again for years. It's just too difficult and not worth the effort unless you're really forced to do it.
Networking can be confusing until it "clicks" then it all seems to fall into place, there is a certain amount of understanding of the overall picture of how bridge networking (or other types of networking) works that once you get that understanding it becomes a lot simpler.
I still don't really understand it. For example, why does the dom0 loose network connectivity when you add the physical interface through which it is connected to the network to a bridge? The bridge supposedly connects networks indiscriminately, and it is illogical that the connection goes away when you connect a network to it.
Knowledge of what a network bridge would help you clarify that question. [2] When you hook an interface to a Linux bridge the interface cannot be used for anything else [0]. That is why IP addresses are instead placed on the bridge interface [1].
I prefer to segregate management and VM physical interfaces. One interface is strictly used for managing the virtualization node and the other(s) are for VMs. Those others could be hooked to different Linux bridges (cabled to your physical switch) or bonded together. I do not have (nor need) IP addresses on bridges with my set up.
You might find more in-depth explanations online, but this is what I fielded quickly.
[0] http://www.microhowto.info/howto/bridge_traffic_between_two_or_more_ethernet... [1] http://xmodulo.com/2013/04/how-to-configure-linux-bridge-interface.html [2] http://iomem.com/archives/4-Ethernet-Bridges-under-Linux.html
SilverTip257 silvertip257@gmail.com writes:
On Fri, Jun 13, 2014 at 8:42 AM, lee lee@yun.yagibdah.de wrote:
Peter peter@pajamian.dhs.org writes:
On 06/12/2014 05:17 PM, lee wrote:
I knew before I started that network setup would be a PITA because years ago, I set up a VM for someone who didn't have a 64bit system to compile a 64bit version of some software. The network setup being so ridiculously difficult has kept me from touching VMs ever again for years. It's just too difficult and not worth the effort unless you're really forced to do it.
Networking can be confusing until it "clicks" then it all seems to fall into place, there is a certain amount of understanding of the overall picture of how bridge networking (or other types of networking) works that once you get that understanding it becomes a lot simpler.
I still don't really understand it. For example, why does the dom0 loose network connectivity when you add the physical interface through which it is connected to the network to a bridge? The bridge supposedly connects networks indiscriminately, and it is illogical that the connection goes away when you connect a network to it.
Knowledge of what a network bridge would help you clarify that question. [2] When you hook an interface to a Linux bridge the interface cannot be used for anything else [0]. That is why IP addresses are instead placed on the bridge interface [1].
I never came across information like that. I was trying to get network access for VMs to work, and that is just way too difficult.
I prefer to segregate management and VM physical interfaces. One interface is strictly used for managing the virtualization node and the other(s) are for VMs. Those others could be hooked to different Linux bridges (cabled to your physical switch) or bonded together. I do not have (nor need) IP addresses on bridges with my set up.
But when you attach them to bridges and don't have IP addresses on the bridges, then they are unreachable.
This is wrong.
A bridge does not need an ip address itself to pass traffic from interface A to interface C, so it could look like this:
hw interface a <---> bridge b <---> vm interface c 192.168.1.3 none 192.168.1.4
Keep in mind that bridging takes place on layer 2: https://en.wikipedia.org/wiki/Bridging_%28networking%29
HTH
Am 14.06.2014 08:41, schrieb lee:
But when you attach them to bridges and don't have IP addresses on the bridges, then they are unreachable.
On 06/16/2014 08:55 PM, Sven Kieske wrote:
A bridge does not need an ip address itself to pass traffic from interface A to interface C,
Correct, but...
so it could look like this:
hw interface a <---> bridge b <---> vm interface c 192.168.1.3 none 192.168.1.4
Wrong. In the above configuration the IP address 192.168.1.3 won't be recognized. That IP needs to be on the bridge in order to work on the host. This is what the OP was referring to with this statement:
Am 14.06.2014 08:41, schrieb lee:
But when you attach them to bridges and don't have IP addresses on the bridges, then they are unreachable.
Peter
On 06/11/2014 04:21 AM, Lars Kurth wrote:
Hi all,
following the discussion on about documentation, I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs. Am starting this with a Xen hat, but the discussion should not be specific to this. There are a number of options, but all have some trade-offs
== #1 virt-install == == #2 xen-tools == == #3 virt-builder (http://libguestfs.org/virt-builder.1.html) == == #4 Cloud Image from Cloud Image SIG ==
This is not a complete list of the ways you can install a VM either. My personal preference is to manually create the filesystem for the VM and then install the OS core with yum. Then after tweaking some config files you can start up the VM and finish installing whatever else you want with yum as well.
While I don't think that this should be the recommended install method it might be worth mentioning and even giving a wiki page with some instructions on how to do it this way. Doing an install like this is actually very good for a newbie because you "get your hands dirty" and get a really good understanding of how yum works and the internals of the distro.
Peter
On gio, 2014-06-12 at 22:34 +1200, Peter wrote:
This is not a complete list of the ways you can install a VM either. My personal preference is to manually create the filesystem for the VM and then install the OS core with yum. Then after tweaking some config files you can start up the VM and finish installing whatever else you want with yum as well.
While I don't think that this should be the recommended install method it might be worth mentioning and even giving a wiki page with some instructions on how to do it this way. Doing an install like this is actually very good for a newbie because you "get your hands dirty" and get a really good understanding of how yum works and the internals of the distro.
Indeed it does!
If you're up for it, Xen wiki will be glad to host it! :-P
We've got both CentOS and Fedora categories on the Wiki, both hosting various pages, guides, instructions, etc., but none of them (I think) goes through the procedure you mentioned:
http://wiki.xen.org/wiki/Category:CentOS http://wiki.xen.org/wiki/Category:Fedora
Feel free to add it! :-)
Regards, Dario
On Thu, Jun 12, 2014 at 11:47 AM, Dario Faggioli raistlin@linux.it wrote:
On gio, 2014-06-12 at 22:34 +1200, Peter wrote:
This is not a complete list of the ways you can install a VM either. My personal preference is to manually create the filesystem for the VM and then install the OS core with yum. Then after tweaking some config files you can start up the VM and finish installing whatever else you want with yum as well.
While I don't think that this should be the recommended install method it might be worth mentioning and even giving a wiki page with some instructions on how to do it this way. Doing an install like this is actually very good for a newbie because you "get your hands dirty" and get a really good understanding of how yum works and the internals of the distro.
@Peter, Would you mind sharing the process? (on the CentOS or Xen wikis OR share a link to a write-up online?)
This piques my curiosity and I'd like to know more. ;-) Thanks!
Indeed it does!
If you're up for it, Xen wiki will be glad to host it! :-P
We've got both CentOS and Fedora categories on the Wiki, both hosting various pages, guides, instructions, etc., but none of them (I think) goes through the procedure you mentioned:
http://wiki.xen.org/wiki/Category:CentOS http://wiki.xen.org/wiki/Category:Fedora
Feel free to add it! :-)
+1 :-D
Regards, Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 06/13/2014 03:47 AM, Dario Faggioli wrote:
If you're up for it, Xen wiki will be glad to host it! :-P
I would love to do a writeup on this, but my time is extremely limited right now. I'll see what I can do.
Peter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 06/13/2014 08:30 AM, Peter wrote:
On 06/13/2014 03:47 AM, Dario Faggioli wrote:
If you're up for it, Xen wiki will be glad to host it! :-P
I would love to do a writeup on this, but my time is extremely limited right now. I'll see what I can do.
Actually, what I'll probably do is wait for CentOS 7 to drop and then do a writeup based on that. I'll be wanting to set up a VM from it anyways and so then I'll kill two birds with one stone.
Peter
The world needs documentation bounties.
On Thu, Jun 12, 2014 at 4:30 PM, Peter peter@pajamian.dhs.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 06/13/2014 03:47 AM, Dario Faggioli wrote:
If you're up for it, Xen wiki will be glad to host it! :-P
I would love to do a writeup on this, but my time is extremely limited right now. I'll see what I can do.
Peter -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTmg3YAAoJEAUijw0EjkDvHKoH/RTd1myCecNTYALvO8l7nRo/ pDjImHlWaT1N5RUODIkZ66EeGF1BJYPaGMrmmjR9R7GGycUkp1eBG3kkGBc1tyMi Hhd9ZjQUvc5IH2BPh2ik8tCom4d6V2+KMEr1ZpYXfYCMi92HaB23xo52x5PKO4pc 4+PJSBe4Dq/UuBtHHyIbRL9WYXLlCybLPVAyQt65pKNyJduwt6M2yJgeRLFB4Iz+ clQ47+roC7UGEpbo6pxjEnU76/WYwZRFefbiBrl4i3Fpl6sWlwBuPH1zyYup3Cwa 0NSJcB6q0rulOcU24OQSWB3O2SpGXOn2WjcmmZQpBZphviMBPOUJeOSAHGU4e6Q= =xjTL -----END PGP SIGNATURE----- _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On mar, 2014-06-10 at 17:21 +0100, Lars Kurth wrote:
Hi all,
Hi!
== #1 virt-install ==
Advantages: similar to KVM
Disadvantages: may cause weird issues / confusion with people switching back to xl. The core issue is that with the current version of xen and libvirt, this only works with xm (when xl is used, this can create some undefined behavior). However as we have seen in some recent threads on this list, people tend to mix which can cause problems.
When you're saying "this only works with xm (when xl is used, this can create some undefined behavior", that is because of the particular version of libvirt (and Xen) is available in CentOS?
I'm asking because, with current enough versions of both, situation should not be that bad, at least not as far as virt-install and VM provisioning are concerned... :-O
Personally, I'm not sure whether this is the best solution, but I see it like something that should not be left in a 'non-working' state.
Parhaps, a disadvantage I see in going for it as the default and recommended mechanism is that you really need: - a recent version of libvit to start with - a way to update libvirt quite often, to take advantage of new features being added
== #2 xen-tools ==
Advantages: Very flexible. Many other distros use xen-tools, so we have lots of beginners docs that just need to be tweaked
Disadvantages: needs porting/packaging for CentOS. Does not work for kvm. Says "xen". (Maybe that's an advantage.) We know that xen-tools works with Fedora (see http://blog.xen.org/index.php/2013/01/24/using-xen-tools-on-fedora/), so the porting effort may be small
Yep, and I went beyond that, and produced some spec files. This should be the latest one, although it's for version 4.3.1 of xen-tools, so no, not the latest possible or the tip of the repo: http://dariof.fedorapeople.org/SPECS/xen-tools.spec
My main problem with xen-tools is that it is kind of debian (and derivatives) biased, when it comes to building the guest file system. I.e., last time I tried, provisioning a Fedora or CentOS guest was possible, but not as straightforward as creating a debian one. That is not xen-tool's fault, is the fact there were no good alternatives to debootstrap.
However, it looks like rinse (one of said alternatives to debootstrap) is getting some steam again: https://gitorious.org/rinse
so that may actually be worth a try.
Also, I recently saw this in xen-tools-discuss mailing list:
"anouncement : Xen-tools update for centos 6.5" http://xen-tools.org/pipermail/xen-tools-discuss/2014-May/001053.html
Even if only for the title, it looks like something to investigate! :-P
Unknowns: What would be needed to make it work for CentOS
Talking a bit with Axel (xen-tools' maintainer) at the time, the main issue in packaging subsequent versions of xen-tools for Fedora was going to be some dependencies from a bunch of perl module he wrote, and packaged himself for Debian, as separate packages. Nothing terrible, but that would mean making the RPMs for those modules as well.
TBH, it is a while back, and I'm not sure what happened, whether went that route. This link, from the xen-tools-discuss email mentioned above: https://github.com/remsnet/xen-tools-rpmbuild
seems to have (more) updated spec files. I haven't tried them, though.
== #3 virt-builder (http://libguestfs.org/virt-builder.1.html) ==
Advantages: supports KVM, Xen and other VM inages. Seems easy to use.
- if so, it would avoid xm / xl confusion.
Unknowns: Not sure at which level virt-builder integrates with Xen and other hypervisors. It seems to operate at disk image format (similar to xen-tools) . I don't know whether virt-builder is restricted to some hypervisors in RHEL7.
Disadvantages: may need porting/packaging for CentOS. It appears as if it will be in RHEL7, so it may just appear with CentOS 7. If not, some porting work may need to be done.
Not commenting on this as I don't know it.
All that being said, and FWIW, I think I'd prefer #1, provided a recent enough version of Xen and Libvirt could be used.
Regards, Dario
On Tue, 2014-06-10 at 17:21 +0100, Lars Kurth wrote:
Hi all,
following the discussion on about documentation, I was wondering whether we need to look at a standard way in which we recommend how to provision images for VMs. Am starting this with a Xen hat, but the discussion should not be specific to this. There are a number of options, but all have some trade-offs
== #1 virt-install ==
Advantages: similar to KVM
Disadvantages: may cause weird issues / confusion with people switching back to xl. The core issue is that with the current version of xen and libvirt, this only works with xm (when xl is used, this can create some undefined behavior). However as we have seen in some recent threads on this list, people tend to mix which can cause problems. ...
I've chosen the virt-install method on CentOS 5 precisely because it is like KVM. I was hoping it would fulfill the promise of being hypervisor agnostic. I'm hoping it continues to be available on future versions of CentOS with Xen.
Though it is a waste of resources, I make all my virtual machines, Linux and MS Windows alike, fully virtualized. I can then move any of the VM's with the same virt-install --import or virsh dumpxml/edit/virsh define process.
When moving a VM, usually the only thing I have to do outside of virt-install/virt-manager is add <acpi/>, <apic/> or <pae/>, which can be done with virsh edit. I don't know why some of my virtual servers need them and other don't but I have higher priority things to think about.
I'm the only technical support person and I don't work 24/7. The graphical interface of virt-manager makes it possible for non-tech people to see what is running and see consoles to restart any misbehaving VM's (usually MS Windows VM's).
I have completely eliminated my use of xm.
On 06/10/2014 05:21 PM, Lars Kurth wrote:
== #4 Cloud Image from Cloud Image SIG == We could rely on pre-built cloud images from the Cloud Images SIG. People could just download the cloud image once it's done and customize it, rather than installing / building their own.
Advantages: seems easy
Disadvantages: coordination with Cloud Images SIG. May not be flexible enough
We ship a test/devel grade CentOS-6-x86_64-pv image ( well, its a qcow2 image, should work for pvhvm as well, the fstab is label driven ).[1]
The biggest problem in doing pre-baked images is the instance metadata. We need to find an easy way to get network settings into the instance and the root password ( or key ), and finally - in some cases, console redirection/setup, but i dont think the console is a deal breaker or a big deal. The network and access credentials however are.
In a typical cloud environ this info would come from the cloud controller's metadata service; on a typical virtualised setup though this becomes an issue ( and isnt really Xen specific ).
We could work around this by making some assumptions, we could 'own' dnsmasq and ensure that either libvirt is running and doing dhcp, otherwise we do the dhcp with some sane defaults, or we setup a script to 'instantiate image', which asks how the user wants to setup the instance ( pvhvm, hvm, pv ), the root password or key to use, and the network settings ( and if this is run on the dom0, we could even ask what bridge or device to connect with as well as the settings ).[2]
Ofcourse, having these images pushed from here mean that clouds or virtualised environs that have metadata services are able to just-use the image as is, not needing any more tooling etc. And we can easily push monthly image updates and when things like heartbleed come around, there is a single place we need to update.
- KB [1]: http://cloud.centos.org/centos/6/devel/
[2] might need to pull in all of libguestfs to make the changes, which in turn has its own challenges if run inside a virtualised environ.
Karanbir Singh mail-lists@karan.org writes:
On 06/10/2014 05:21 PM, Lars Kurth wrote:
== #4 Cloud Image from Cloud Image SIG == We could rely on pre-built cloud images from the Cloud Images SIG. People could just download the cloud image once it's done and customize it, rather than installing / building their own.
Advantages: seems easy
Disadvantages: coordination with Cloud Images SIG. May not be flexible enough
We ship a test/devel grade CentOS-6-x86_64-pv image ( well, its a qcow2 image, should work for pvhvm as well, the fstab is label driven ).[1]
The biggest problem in doing pre-baked images is the instance metadata. We need to find an easy way to get network settings into the instance and the root password ( or key ), and finally - in some cases, console redirection/setup, but i dont think the console is a deal breaker or a big deal. The network and access credentials however are.
In a typical cloud environ this info would come from the cloud controller's metadata service; on a typical virtualised setup though this becomes an issue ( and isnt really Xen specific ).
We could work around this by making some assumptions, we could 'own' dnsmasq and ensure that either libvirt is running and doing dhcp, otherwise we do the dhcp with some sane defaults, or we setup a script to 'instantiate image', which asks how the user wants to setup the instance ( pvhvm, hvm, pv ), the root password or key to use, and the network settings ( and if this is run on the dom0, we could even ask what bridge or device to connect with as well as the settings ).[2]
Ofcourse, having these images pushed from here mean that clouds or virtualised environs that have metadata services are able to just-use the image as is, not needing any more tooling etc. And we can easily push monthly image updates and when things like heartbleed come around, there is a single place we need to update.
Wouldn't you still need to configure the services running in each VM?
On Tue, Jun 17, 2014 at 6:11 AM, lee lee@yun.yagibdah.de wrote:
Karanbir Singh mail-lists@karan.org writes:
Ofcourse, having these images pushed from here mean that clouds or virtualised environs that have metadata services are able to just-use the image as is, not needing any more tooling etc. And we can easily push monthly image updates and when things like heartbleed come around, there is a single place we need to update.
Wouldn't you still need to configure the services running in each VM?
Yes.
I believe the idea here is to pre-configure as much as possible and complete the remaining configuration via network connectivity (ie: SSH).
SilverTip257 silvertip257@gmail.com writes:
On Tue, Jun 17, 2014 at 6:11 AM, lee lee@yun.yagibdah.de wrote:
Karanbir Singh mail-lists@karan.org writes:
Ofcourse, having these images pushed from here mean that clouds or virtualised environs that have metadata services are able to just-use the image as is, not needing any more tooling etc. And we can easily push monthly image updates and when things like heartbleed come around, there is a single place we need to update.
Wouldn't you still need to configure the services running in each VM?
Yes.
I believe the idea here is to pre-configure as much as possible and complete the remaining configuration via network connectivity (ie: SSH).
How about some sort of package management that lets you define and configure the VM?
This is currently done from "the inside", i. e. when the VM is running, with whatever installer and package manager a distribution comes with.
Why not do it from "the outside", i. e. before the VM even exists, creating it in the process? Define some sort of API so that the same "VM creator tool" could be used with different distributions.
Am 17.06.2014 19:31, schrieb lee:
How about some sort of package management that lets you define and configure the VM?
This is currently done from "the inside", i. e. when the VM is running, with whatever installer and package manager a distribution comes with.
Why not do it from "the outside", i. e. before the VM even exists, creating it in the process? Define some sort of API so that the same "VM creator tool" could be used with different distributions.
I believe you are talking about a bootstrapped minimal installation and than managing this via a config management tool like puppet, chef etc. ?
Nevertheless I think this gets really off topic and out of scope.
Sven Kieske S.Kieske@mittwald.de writes:
Am 17.06.2014 19:31, schrieb lee:
How about some sort of package management that lets you define and configure the VM?
This is currently done from "the inside", i. e. when the VM is running, with whatever installer and package manager a distribution comes with.
Why not do it from "the outside", i. e. before the VM even exists, creating it in the process? Define some sort of API so that the same "VM creator tool" could be used with different distributions.
I believe you are talking about a bootstrapped minimal installation and than managing this via a config management tool like puppet, chef etc. ?
No, I'm talking about an "installer" which can be used to create and to configure a VM image. The resulting VM image would have all needed packages installed and be fully configured.
Nevertheless I think this gets really off topic and out of scope.
Does "provisioning VM images" not involve their creation?
Am 18.06.2014 23:33, schrieb lee:
No, I'm talking about an "installer" which can be used to create and to configure a VM image. The resulting VM image would have all needed packages installed and be fully configured.
Yes, this is e.g. something you can do with saltstack salt-virt or similar tools (especially the "fully configured" part). but you could also use virt-builder/-installer + cloud-init or various other tools.
Does "provisioning VM images" not involve their creation?
No, not necessary.
On Mon, Jun 16, 2014 at 11:54 AM, Karanbir Singh mail-lists@karan.org wrote:
On 06/10/2014 05:21 PM, Lars Kurth wrote:
== #4 Cloud Image from Cloud Image SIG == We could rely on pre-built cloud images from the Cloud Images SIG. People could just download the cloud image once it's done and customize it, rather than installing / building their own.
Advantages: seems easy
Disadvantages: coordination with Cloud Images SIG. May not be flexible enough
We ship a test/devel grade CentOS-6-x86_64-pv image ( well, its a qcow2 image, should work for pvhvm as well, the fstab is label driven ).[1]
The biggest problem in doing pre-baked images is the instance metadata. We need to find an easy way to get network settings into the instance and the root password ( or key ), and finally - in some cases, console redirection/setup, but i dont think the console is a deal breaker or a big deal. The network and access credentials however are.
In a typical cloud environ this info would come from the cloud controller's metadata service; on a typical virtualised setup though this becomes an issue ( and isnt really Xen specific ).
We could work around this by making some assumptions, we could 'own' dnsmasq and ensure that either libvirt is running and doing dhcp, otherwise we do the dhcp with some sane defaults, or we setup a script to 'instantiate image', which asks how the user wants to setup the instance ( pvhvm, hvm, pv ), the root password or key to use, and the network settings ( and if this is run on the dom0, we could even ask what bridge or device to connect with as well as the settings ).[2]
Ofcourse, having these images pushed from here mean that clouds or virtualised environs that have metadata services are able to just-use the image as is, not needing any more tooling etc. And we can easily push monthly image updates and when things like heartbleed come around, there is a single place we need to update.
- KB
[2] might need to pull in all of libguestfs to make the changes, which in turn has its own challenges if run inside a virtualised environ.
I didn't follow this -- virt-builder seems to run fine in dom0 with qemu, albeit a bit slow. Upstream is open to having patches for Xen bindings for the utility VM.
Or is there something else I'm missing?
-George
On Mon, Jun 16, 2014 at 11:54 AM, Karanbir Singh mail-lists@karan.org wrote:
On 06/10/2014 05:21 PM, Lars Kurth wrote:
== #4 Cloud Image from Cloud Image SIG == We could rely on pre-built cloud images from the Cloud Images SIG. People could just download the cloud image once it's done and customize it, rather than installing / building their own.
Advantages: seems easy
Disadvantages: coordination with Cloud Images SIG. May not be flexible enough
We ship a test/devel grade CentOS-6-x86_64-pv image ( well, its a qcow2 image, should work for pvhvm as well, the fstab is label driven ).[1]
The biggest problem in doing pre-baked images is the instance metadata. We need to find an easy way to get network settings into the instance and the root password ( or key ), and finally - in some cases, console redirection/setup, but i dont think the console is a deal breaker or a big deal. The network and access credentials however are.
In a typical cloud environ this info would come from the cloud controller's metadata service; on a typical virtualised setup though this becomes an issue ( and isnt really Xen specific ).
A couple of notes on virt-builder:
* When I tried it, all of the images booted on Xen out-of-the-box except Fedora 19 and 20; I reported this to the list and the developers were very responsive, so now everything works. (The exception might be RHEL7-rc, but I think should be gone now.) * However, libguestfs has a massive dependency tree in RHEL7 for some reason; KB said that a "yum install libguestfs" wanted to install 178 additional RPMs. A minimal C6 install is only 143 RPMs, so this more than doubles it. That makes this a bit less desirable as the primary method of getting a system up and running quickly.
So it looks like we might want to recommend three potential paths we could recommend people to explore:
1) For basic CentOS VMs, use a CentOS-provided cloud image, with our custom metadata tweaking script.
2) For more versatile image set-up and manipulation (including other operating systems), use virt-builder.
3) If you're thinking about using libvirt anyway, use virt-install and install from installation media.
Does that sound reasonable?
#1 still has some issues we need to work out -- particularly with sorting out the metadata. I'll start a separate thread to talk about that.
-George
I just dived a little in virt-builder and here are my findings:
1. It's awesome!
Am 18.06.2014 12:46, schrieb George Dunlap:
So it looks like we might want to recommend three potential paths we could recommend people to explore:
- For basic CentOS VMs, use a CentOS-provided cloud image, with our
custom metadata tweaking script.
2. Why create a custom script if maybe virt-builder can handle it? I don't know if you can point virt-builder to other images than those on libguestfs.org, but this should be possible, include an centos cloud-image on libguestfs.org and just use virt-builder
- For more versatile image set-up and manipulation (including other
operating systems), use virt-builder.
- If you're thinking about using libvirt anyway, use virt-install and
install from installation media.
3. as a fallback if virt-builder is not available or you don't got the images from libguestfs.org cached or you don't have direct internet access: ok, for other purposes I guess virt-builder should be enough?
Does that sound reasonable?
If networking is already configured when a VM boots, how about a kernel parameter for a configuration server? If it is configured to grab an executable file, it would be very flexible.
We use kickstart files to build our CentOS VM's and physical boxes and then pull configuration changes from a server at regular intervals. I'm working on extending our existing internal intranetto keep history and to allow a single change to propagate to all relevant servers. Also would allow fail2ban to share it's banned IP's. Since this is specific to our internal systems, it can't easily be exported, but most of it is pretty straight forward. It could almost be integrated into an existing CMS. Think WebMin but in a LISA (Large Installation System Administration) environment.
A similar technique could be used for cloud machines.
On Wed, Jun 18, 2014 at 2:48 PM, Sven Kieske S.Kieske@mittwald.de wrote:
I just dived a little in virt-builder and here are my findings:
- It's awesome!
Am 18.06.2014 12:46, schrieb George Dunlap:
So it looks like we might want to recommend three potential paths we could recommend people to explore:
- For basic CentOS VMs, use a CentOS-provided cloud image, with our
custom metadata tweaking script.
- Why create a custom script if maybe virt-builder can handle it? I don't know if you can point virt-builder to other images than
those on libguestfs.org, but this should be possible, include an centos cloud-image on libguestfs.org and just use virt-builder
You can point virt-builder to another repo; and in fact, the guys on the libguestfs development list said that them hosting their own template repo was meant to be a temporary measure, and that long-term they'd like to see distros hosting their own templates.
But as to "why", did you miss that "yum install libguestfs" wants to bring in an *additional* 178 packages, more than doubling the size of a base installation? :-)
I think for a lot of people that will be fine; but I agree with KB, that we should also have a really lightweight way for people to just get VMs up and running.
-George
On Thu, Jun 19, 2014 at 7:39 AM, George Dunlap dunlapg@umich.edu wrote:
On Wed, Jun 18, 2014 at 2:48 PM, Sven Kieske S.Kieske@mittwald.de wrote:
I just dived a little in virt-builder and here are my findings:
- It's awesome!
Am 18.06.2014 12:46, schrieb George Dunlap:
So it looks like we might want to recommend three potential paths we could recommend people to explore:
- For basic CentOS VMs, use a CentOS-provided cloud image, with our
custom metadata tweaking script.
- Why create a custom script if maybe virt-builder can handle it? I don't know if you can point virt-builder to other images than
those on libguestfs.org, but this should be possible, include an centos cloud-image on libguestfs.org and just use virt-builder
You can point virt-builder to another repo; and in fact, the guys on the libguestfs development list said that them hosting their own template repo was meant to be a temporary measure, and that long-term they'd like to see distros hosting their own templates.
But as to "why", did you miss that "yum install libguestfs" wants to bring in an *additional* 178 packages, more than doubling the size of a base installation? :-)
Building stripped OS images is an art form. In fact, if you want to get *really* dirty, don't use disk images. Use a local chroot cage to install an OS image from an initial minimal tarball, and use tools like "kickstart" to pre-partition the disk and run theset steps..
1) Configure disk. a( Use parted in CentOS 6, or fdisk in CentOS 5 to pre-align partitions on the 4096 byte block boundaries. All VM's that might run on NetApp or other 4096 byte block drive back end storage needs this, and better safe than sorry. 2) Obtain tarball, for example by downloading it into the swap partition if your installer doesn't have a local copy on a DVD image or local NFS share. 3) Configure /etc/fstab in the chroot cage. (Don't even put one in the tarball, it's not needed!) 4) Configure /etc/syscontig/network-scripts/ifcfg-eth0 in the tarball, as needed, for network layout. a) Pay attention to /etc/udev.d/rules/70-persistent-net-rules, which will reset your network device numbering. b) It's usually safest to just delete it for VM and server images. 5) Run "grub-install" for the relevant bood disk, from inside the chroot cage, to make it bootable. 6: Clear the installation tarball and run "mkswap" on the swap partition.
I don't know why most VM environments don't use a separate disk image for swap, anyway, burying it inside your LVM setup makes it much more onerous to efficiently cut free or resize without disturbing the rest of your system. And it makes a thin-provisioned disk automatically much bulkier, instead just leave out swap from published OS disk images. But that's a very separate issue.
Now, when I first did this sort of thing, it was to shrink a corporate Red Hat 5.2 build onto an installable CD. (Not RHEL. .2 Red Hat 5.2) The original engineer had been using dd'd and gzipped disk images, of different sizes for different disk drives, and trying to put them on the same CD, and always starting over by zeroing the disk and putting new images on it to make any changes, then dd'ing and re-compressing the disk image. Including *swap*!!!!! The same sort of wasted disk, written but uncleared blocks, swap space, and other issues exist for VM images, and a lot of the resulting image size is entirely unneeded for OS installations. Getting the original OS image tarball can be pesky: I had to install an OS, reboot with a rescue CD, and tarball the contents of the disk partitions. I got it down so small it fit on a "buisness card" CD.
But these days, guess what? "mock" already does most of this work for me!!! Take a good look at the stored initialization tarballs for "mock", and they have *exactly* this sort of content. They have some extra components for RPM building, but you can enter the chroot cage and strip those right out, along with unnecessary log files, by staying at run level 1 while in the tarball. I applaud the "mock" authors for their work creating well defined workable chroot cages, and take other advantage of them for testing components.
The initial setup takes some knowledge, and it's no longer just "plug in this disk image and be done", you need a working tool like a PXE environment and a selection of pre-configured kickstart files with '%pre" scripts, or an installer CD or DVD or USB image with a kickstart script set up, to make it efficient But it scales *really well*. I helped do 13,000 systems in one month this way, back in 2002, and the technique has proven invaluable for prototyping new, completely defined, completely reviewable OS images ever since.
-
Dario Faggioli -
Digimer -
Ed Heron -
George Dunlap -
Karanbir Singh -
Lars Kurth -
lee -
Nico Kadel-Garcia -
NightLightHosts Admin -
Peter -
SilverTip257 -
Sven Kieske