[CentOS-virt] Preferred method of provisioning VM images

On Thu, Jun 19, 2014 at 7:39 AM, George Dunlap <dunlapg at umich.edu> wrote:
> On Wed, Jun 18, 2014 at 2:48 PM, Sven Kieske <S.Kieske at mittwald.de> wrote:
>> I just dived a little in virt-builder and here are my findings:
>>
>> 1. It's awesome!
>>
>> Am 18.06.2014 12:46, schrieb George Dunlap:
>>> So it looks like we might want to recommend three potential paths we
>>> could recommend people to explore:
>>>
>>> 1) For basic CentOS VMs, use a CentOS-provided cloud image, with our
>>> custom metadata tweaking script.
>>
>> 2. Why create a custom script if maybe virt-builder can handle it?
>>    I don't know if you can point virt-builder to other images than
>> those on libguestfs.org, but this should be possible, include an centos
>> cloud-image on libguestfs.org and just use virt-builder
>
> You can point virt-builder to another repo; and in fact, the guys on
> the libguestfs development list said that them hosting their own
> template repo was meant to be a temporary measure, and that long-term
> they'd like to see distros hosting their own templates.
>
> But as to "why", did you miss that "yum install libguestfs" wants to
> bring in an *additional* 178 packages, more than doubling the size of
> a base installation? :-)

Building stripped OS images is an art form. In fact, if you want to
get *really* dirty, don't use disk images. Use a local chroot cage to
install an OS image from an initial minimal tarball, and use tools
like "kickstart" to pre-partition the disk and run theset steps..

1) Configure disk.
     a( Use parted in CentOS 6, or fdisk in CentOS 5 to pre-align
partitions on the 4096 byte block boundaries. All VM's that might run
on NetApp or other 4096 byte block drive back end storage needs this,
and better safe than sorry.
2) Obtain tarball, for example by downloading it into the swap
partition if your installer doesn't have a local copy on a DVD image
or local NFS share.
3) Configure /etc/fstab in the chroot cage. (Don't even put one in the
tarball, it's not needed!)
4) Configure /etc/syscontig/network-scripts/ifcfg-eth0 in the tarball,
as needed, for network layout.
    a) Pay attention to /etc/udev.d/rules/70-persistent-net-rules,
which will reset your network device numbering.
    b) It's usually safest to just delete it for VM and server images.
5) Run "grub-install" for the relevant bood disk, from inside the
chroot cage, to make it bootable.
6: Clear the installation tarball and run "mkswap" on the swap partition.

I don't know why most VM environments don't use a separate disk image
for swap, anyway, burying it inside your LVM setup makes it much more
onerous to efficiently cut free or resize without disturbing the rest
of your system. And it makes a thin-provisioned disk automatically
much bulkier, instead just leave out swap from published OS disk
images. But that's a very separate issue.

Now, when I first did this sort of thing, it was to shrink a corporate
Red Hat 5.2 build onto an installable CD. (Not RHEL. .2 Red Hat 5.2)
The original engineer had been using dd'd and gzipped disk images, of
different sizes for different disk drives, and trying to put them on
the same CD, and always starting over by zeroing the disk and putting
new images on it to make any changes, then dd'ing and re-compressing
the disk image. Including *swap*!!!!!  The same sort of wasted disk,
written but uncleared blocks, swap space, and other issues exist for
VM images, and a lot of the resulting image size is entirely unneeded
for OS installations. Getting the original OS image tarball can be
pesky: I had to install an OS, reboot with a rescue CD, and tarball
the contents of the disk partitions. I got it down so small it fit on
a "buisness card" CD.

But these days, guess what? "mock" already does most of this work for
me!!! Take a good look at the stored initialization tarballs for
"mock", and they have *exactly* this sort of content. They have some
extra components for RPM building, but you can enter the chroot cage
and strip those right out, along with unnecessary log files, by
staying at run level 1 while in the tarball. I applaud the "mock"
authors for their work creating well defined workable chroot cages,
and take other advantage of them for testing components.

The initial setup takes some knowledge, and it's no longer just "plug
in this disk image and be done", you need a working tool like a PXE
environment and a selection of pre-configured kickstart files with
'%pre" scripts, or an installer CD or DVD or USB image with a
kickstart script set up, to make it efficient But it scales *really
well*. I helped do 13,000 systems in one month this way, back in 2002,
and the technique has proven invaluable for prototyping new,
completely defined, completely reviewable OS images ever since.