Hi everyone,
I'm working at Scaleway on the multiarch images system, I'm @moul on IRC and talked with @Arrfab and @Evolution (thanks a lot, guys).
---
Context: Today, we are using Docker as a build-step to generate our baremetal servers rootfs, what we call, the "Scaleway images".
Here is an example of "1 click app" image:
- Public Hub: https://hub.scaleway.com/wordpress.html - DevHub (aggregate APIs): http://devhub.scaleway.com/#/images/wordpress:latest - Source (GitHub): https://github.com/scaleway-community/scaleway-wordpress/ - Docker Hub: https://hub.docker.com/r/scaleway/wordpress/ - Build logs: https://travis-ci.org/scaleway/qa/builds/98515979
We are quite happy today with this method and are succesfully running plenty Linux distributions (Ubuntu, openSUSE, Gentoo, Fedora, Debian, Arch Linux and Alpine Linux).
Actually we also have a proof-of-concept of CentOS image, but based on an old development CubieTruck snapshot with no public mirrors.
Here is how I built this POC image (I wasn't aware of virt-tar-out): https://github.com/armbuild/centos-img/blob/master/buildcontainers.sh
---
We will soon release new server models with new architectures, our next priorities are: `arm64` and `x86_64`, but we plan to support more architectures if we feel enough people will need it so the development worth it.
I started to work on "multiarch" images (one image per architecture). Here are some images:
- https://github.com/multiarch/alpine -> built with `apk-static` in a chroot - https://github.com/multiarch/busybox -> static binaries extracted from latest busybox-static.deb - https://github.com/multiarch/ubuntu-debootstrap -> qemu-debootstrap - https://github.com/multiarch/debian-debootstrap -> same as above - https://github.com/multiarch/ubuntu-core -> based on released rootfs tarballs at https://partner-images.canonical.com/core/wily/current/
All those images also contains an `x86_64` `qemu-${ARCH}-static` so you can run them on an `x86_64` host with `binfmt-support` available. Look at the README.md files for usage and examples.
I began to work on the Centos image https://github.com/multiarch/centos,
I successfully built the `armhf` and `arm64` images by `virt-tar-out`ing the CubieTruck released image, i.e.: http://mirror.centos.org/altarch/7.2.1511/isos/armhfp/
I also successfully built the `x86_64` and `i386` images using kickstart:
livemedia-creator --arch=i386 --make-disk \ --iso=/tmp/boot7-i386.iso \ --ks=$(pwd)/centos-7.ks \ --image-name=centos-7-base-i386
based on https://github.com/CentOS/sig-cloud-instance-build/tree/livemedia_builds/doc...
For now I failed to build the `ppc64` nor `ppc64le` images, but that's not a priority.
I have now an issue, my `armhf` and `x86_64` images are not the same, the ones based on real boards .img.xz embed about 300M of kernel images, dtb, kernel modules, firmware (I don't want them) + a lot more apps that we can find by default in the <200Mb `x86_64` Docker image.
I would like to find a solution to build "same" images for all the targets and avoid as much as possible the need of diffing the outputs of `yum list installed`.
My favorite solution would be to use kickstart files, which are very cool to put on GitHub. But the non `x86_64` architectures lack of a `boot.iso` file packing `isolinux`.
Is it possible to add those `boot.iso` files on the mirror or tell me how to build them ? Do you see alternative ways to build identical images ?
-- Manfred Touron - @moul CTO @ scaleway.com
On 14/01/2016 16:29, Manfred Touron wrote:
For now I failed to build the `ppc64` nor `ppc64le` images, but that's not a > priority.
Manfred, If I am correct, docker is requiring golang packages which are not yet available for Centos 7.2 on ppc64/ppc64le.
On 01/14/2016 10:13 AM, Thierry Fauck@linux.vnet.ibm.com wrote:
On 14/01/2016 16:29, Manfred Touron wrote:
For now I failed to build the `ppc64` nor `ppc64le` images, but that's not a > priority.
Manfred, If I am correct, docker is requiring golang packages which are not yet available for Centos 7.2 on ppc64/ppc64le.
They aren't providing them for AArch64 either, but for the AArch64 build, we're using golang 1.5.x pulled from fedora to meet the dependency. It's not ideal, but it works.
Yes you are,
However, Docker build system supports two kind of image manipulation:
- `FROM`, `COPY/ADD`, `ENV/LABEL`, `EXPOSE/VOLUME`, `CMD/ENTRYPOINT`, `ONBUILD`, `WORKDIR/USER`: direct filesystem manipulation from the Docker binary, it does not care about the architecture of the rootfs, you can even build an image without any binary inside
- `RUN yum install …` calls -> `/bin/sh -ec ‘yum install …’` relies on LXC, you need to run this on a host of the same architecture of the target, but, you are using the same Kernel, so `binfmt-support` is available
Here is an example of `Dockerfile` that can be built on any architecture and will always produce an x86_64 image: https://github.com/CentOS/sig-cloud-instance-images/tree/CentOS-7/docker and here an armhf image that can be built on any architecture: https://github.com/armbuild/alpine/tree/master/edge
If we achieve to get a tarball, i.e. using `livemedia-creator` we can build everything on an `x86_64` host.
If we still need to RUN some commands, we need an host of the target architecture with Docker or `qemu-XXX-static` + `binfmt-support`.
-- Manfred Touron
On 14 Jan 2016, at 17:13, Thierry Fauck@linux.vnet.ibm.com thierry@linux.vnet.ibm.com wrote:
On 14/01/2016 16:29, Manfred Touron wrote:
For now I failed to build the `ppc64` nor `ppc64le` images, but that's not a priority.
Manfred, If I am correct, docker is requiring golang packages which are not yet available for Centos 7.2 on ppc64/ppc64le.
-- Thierry Fauck @ linux.vnet.ibm
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 01/14/2016 10:13 AM, Thierry Fauck@linux.vnet.ibm.com wrote:
On 14/01/2016 16:29, Manfred Touron wrote:
For now I failed to build the `ppc64` nor `ppc64le` images, but that's not a > priority.
Manfred, If I am correct, docker is requiring golang packages which are not yet available for Centos 7.2 on ppc64/ppc64le.
That is true, however, gccgo might be an option for ppc64/ppc64le. Ubuntu uses gccgo to provide Docker packages for their ppc64le distro.
On 01/14/2016 01:22 PM, Jason Furmanek wrote:
On 01/14/2016 10:13 AM, Thierry Fauck@linux.vnet.ibm.com wrote:
On 14/01/2016 16:29, Manfred Touron wrote:
For now I failed to build the `ppc64` nor `ppc64le` images, but
that's not a > priority. Manfred, If I am correct, docker is requiring golang packages which are not yet available for Centos 7.2 on ppc64/ppc64le.
That is true, however, gccgo might be an option for ppc64/ppc64le. Ubuntu uses gccgo to provide Docker packages for their ppc64le distro.
gccgo requires gcc 4.9.x at minimum and for more recent docker versions you'd need gcc 5.1.1 or 5.2.x. Since the distro gcc is 4.8.3, that's not really workable.
In theory, golang 1.5 has experimental support for ppc64 and ppc64le. We can certainly try to bootstrap golang from x86_64 to ppc64le the same way we did for aarch64.
On 01/14/2016 04:38 PM, Jim Perrin wrote:
gccgo requires gcc 4.9.x at minimum and for more recent docker versions you'd need gcc 5.1.1 or 5.2.x. Since the distro gcc is 4.8.3, that's not really workable.
In theory, golang 1.5 has experimental support for ppc64 and ppc64le. We can certainly try to bootstrap golang from x86_64 to ppc64le the same way we did for aarch64.
So, using a variation of what I did for arm64, the Dockerfile below will create a bootstrap build of golang for ppc64le. You can then run the container and copy out the tarball. This gets you past the first step. ============================ FROM centos:7
RUN yum install -y glibc-static git gcc RUN git clone https://go.googlesource.com/go /go # build bootstrap version RUN cd /go && \ git checkout go1.4.2 && \ cd src && \ GOROOT=/go GOROOT_FINAL=/opt/go1.4.2 \ ./make.bash && \ mkdir -p /opt/go1.4.2 && \ cd /go && \ cp -pav * /opt/go1.4.2/
RUN cd /go && \ rm -rf * && git reset --hard HEAD && \ git checkout go1.5.3 && \ cd src && \ GOROOT=/go \ GOHOSTARCH=amd64 \ GOARCH=ppc64le \ GOROOT_BOOTSTRAP=/opt/go1.4.2 \ GOROOT_FINAL=/opt/go1.5-ppc64le \ ./make.bash && \ mkdir -p /opt/go1.5-ppc64le && \ cd /go && \ cp -pav * /opt/go1.5-ppc64le/
RUN tar czf /go1.5-ppc64le.tar.gz /opt/go1.5-ppc64le ==============================
Extract the tarball on your ppc64le system someplace out of the way, and run the go binary to be sure it works.
From here you can pull down the fedora src.rpm for golang, and build it
manually with rpmbuild. This is a scratch copy, and you'll need to alter the spec file to reference your instance of go in /opt or wherever, and strip buildrequires for golang as needed.
Once this is successful, you *should* be able to use the resulting rpm via lookaside repo in mock to build the unmodified src.rpm.
If you've made it this far and you're still sober, reward yourself with a nice glass of scotch for a job well done. Isn't bootstrapping fun!?!?
Hi again!
I’mI’m not sure we are talking about the same thing here, though I agree that having Docker on CentOS is a great thing and I can help on this, this is not what I’m trying to achieve.
Right now, I’m just trying to build a Scaleway image on a host system already having Docker, I don’t need Docker inside CentOS to get a Scaleway image. My problem is that I want the ARM CentOS Scaleway image to be the same as the x86_64 CentOS Scaleway image, and for this I’d need the equivalent of the boot.iso file for ARMHF; Or another method with the same result. Do you have any hints in that direction?
— Manfred Touron - @moul VSRE.info compliant
On 14 Jan 2016, at 17:13, Thierry Fauck@linux.vnet.ibm.com thierry@linux.vnet.ibm.com wrote:
On 14/01/2016 16:29, Manfred Touron wrote:
For now I failed to build the `ppc64` nor `ppc64le` images, but that's not a priority.
Manfred, If I am correct, docker is requiring golang packages which are not yet available for Centos 7.2 on ppc64/ppc64le.
-- Thierry Fauck @ linux.vnet.ibm
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 27/01/16 15:38, Manfred Touron wrote:
My problem is that I want the ARM CentOS Scaleway image to be the same as the x86_64 CentOS Scaleway image, and for this I’d need the equivalent of the boot.iso file for ARMHF; Or another method with the same result. Do you have any hints in that direction?
I believe that is the crux of the matter - Armhfp is not installer driven, but image driven. There are no installer images created for that platform, hence no boot.iso or DVD images etc.
Regards,
On 27 Jan 2016, at 16:43, Karanbir Singh mail-lists@karan.org wrote:
I believe that is the crux of the matter - Armhfp is not installer driven, but image driven. There are no installer images created for that platform, hence no boot.iso or DVD images etc.
Ok, that makes sense.
Is this possible to do something like “debootstrap" or alpine linux' “apk” do, i.e:
mkdir rootfs cp yum-static rootfs chroot rootfs ./yum-static install base
?
— Manfred Touron - @moul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 27/01/16 20:17, Manfred Touron wrote:
On 27 Jan 2016, at 16:43, Karanbir Singh <mail-lists@karan.org mailto:mail-lists@karan.org> wrote:
I believe that is the crux of the matter - Armhfp is not installer driven, but image driven. There are no installer images created for that platform, hence no boot.iso or DVD images etc.
Ok, that makes sense.
Is this possible to do something like “debootstrap" or alpine linux' “apk” do, i.e:
mkdir rootfs cp yum-static rootfs chroot rootfs ./yum-static install base
more or less, yes, and that's how the actual armhfp .img files were created (yum --installroot=) As said before on irc, have a look at https://github.com/mndar/rbf (and for example https://github.com/arrfab/rbf/blob/master/templates/centos-cubietruck.xml)
BTW, if you just need the RootFS, why not using one of the generated images, and get what you need from that image ?
- -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab
On 28/01/16 06:34, Fabian Arrotin wrote:
On 27/01/16 20:17, Manfred Touron wrote:
On 27 Jan 2016, at 16:43, Karanbir Singh <mail-lists@karan.org mailto:mail-lists@karan.org> wrote:
I believe that is the crux of the matter - Armhfp is not installer driven, but image driven. There are no installer images created for that platform, hence no boot.iso or DVD images etc.
Ok, that makes sense.
Is this possible to do something like “debootstrap" or alpine linux' “apk” do, i.e:
mkdir rootfs cp yum-static rootfs chroot rootfs ./yum-static install base
more or less, yes, and that's how the actual armhfp .img files were created (yum --installroot=) As said before on irc, have a look at https://github.com/mndar/rbf (and for example https://github.com/arrfab/rbf/blob/master/templates/centos-cubietruck.xml)
BTW, if you just need the RootFS, why not using one of the generated images, and get what you need from that image ?
If you want, we can also just run the image builds for you - if you let us know what the specific requirements are - we already do that for many vendor clouds / vps setups etc
On 28 Jan 2016, at 09:13, Karanbir Singh mail-lists@karan.org wrote: If you want, we can also just run the image builds for you - if you let us know what the specific requirements are - we already do that for many vendor clouds / vps setups etc
Ok for me, basically I need the minimal amount of packages (base + yum), no drivers, no kernel, no firmwares, no dts, nothing boot related.
I need one image per supported architectures: - x86_64 - i386 - aarch64 - armhfp - ppc64
Is this something possible ?
— Manfred Touron - @moul
On 28/01/16 11:58, Manfred Touron wrote:
On 28 Jan 2016, at 09:13, Karanbir Singh <mail-lists@karan.org mailto:mail-lists@karan.org> wrote: If you want, we can also just run the image builds for you - if you let us know what the specific requirements are - we already do that for many vendor clouds / vps setups etc
Ok for me, basically I need the minimal amount of packages (base + yum), no drivers, no kernel, no firmwares, no dts, nothing boot related.
I need one image per supported architectures:
- x86_64
- i386
- aarch64
- armhfp
- ppc64
Is this something possible ?
yes.
when you say minimal, does that still include cloud-init or how do users get access to the instances ( eg. over ssh etc ) when it runs ?
On 30 Jan 2016, at 13:10, Karanbir Singh mail-lists@karan.org wrote:
when you say minimal, does that still include cloud-init or how do users get access to the instances ( eg. over ssh etc ) when it runs ?
I don’t need cloud-init, nor SSH for these images,
I will install all the required packages in a second time by myself, so I can create a very minimal image usable in Docker without cloud-init, SSH, etc; And one more complete image with all the required packages to run a baremetal Scaleway server
— Manfred Touron - @moul VSRE.info compliant
On Saturday 30 January 2016 12:10:10 Karanbir Singh wrote:
On 28/01/16 11:58, Manfred Touron wrote:
On 28 Jan 2016, at 09:13, Karanbir Singh <mail-lists@karan.org mailto:mail-lists@karan.org> wrote: If you want, we can also just run the image builds for you - if you let us know what the specific requirements are - we already do that for many vendor clouds / vps setups etc
Ok for me, basically I need the minimal amount of packages (base + yum), no drivers, no kernel, no firmwares, no dts, nothing boot related.
I need one image per supported architectures:
- x86_64
- i386
- aarch64
- armhfp
- ppc64
Is this something possible ?
yes.
when you say minimal, does that still include cloud-init or how do users get access to the instances ( eg. over ssh etc ) when it runs ?
Hello,
I have created a centos 7 32 bits based docker image. You can find the docker file and image build process in https://github.com/nunofernandes/docker-centos7-32bits
The kickstart is based on https://github.com/CentOS/sig-cloud-instance-build/tree/master/docker (removing systemdcontainer and i found that is not necessary). Because the original kickstart doesn't include cloud-init package i choose not to include it either.. If you think that it's a "plus" tell me and i'll gladly add it.
Hope it helps to have an "oficial" centos 32 bits release.
Best regards, Nuno Fernandes
On 28 Jan 2016, at 07:34, Fabian Arrotin arrfab@centos.org wrote:
more or less, yes, and that's how the actual armhfp .img files were created (yum --installroot=)
I will try to play with this, thanks
As said before on irc, have a look at https://github.com/mndar/rbf https://github.com/mndar/rbf(and for example https://github.com/arrfab/rbf/blob/master/templates/centos-cubietruck.xml https://github.com/arrfab/rbf/blob/master/templates/centos-cubietruck.xml)
Does rbf works for non-arm images ? Can you provide an x86_64 example on the repository ? rbf sounds really really good if it is not limited to the arm architecture.
BTW, if you just need the RootFS, why not using one of the generated images, and get what you need from that image ?
Actually it works, but it results in doing:
- x86_64, i386 -> build an image from its kickstart + boot.iso disk == a minimal ready to use image, easy to read on GitHub (perfect) - arm, mips, powerpc, arm64 -> import a disk image, then yum remove everything not needed, export the image and reimport it to avoid non-effective file removals due to Docker copy-on-write layering system.
Basically it will work, but it is harder to maintain, something like maintaining a `yum remove $(diff <(yum list x86_64 images) <(yum list arm image)).` The image build procedure will also be more obscure.
This solution is my fallback if we can’t achieve to find a generic way to build CentOS on all architectures.
Manfred Touron
-
Fabian Arrotin -
Jason Furmanek -
Jim Perrin -
Karanbir Singh -
Manfred Touron -
Nuno Fernandes -
Thierry Fauck＠linux.vnet.ibm.com