Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
Thanks for any insights?
Simon
[Resent this message to the list without aa GPG signature, since my signature blows the message up past the 50 KB mark... meh.]
* On 12/5/20 8:15 PM, Simon Matter wrote:
[...] We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
I cannot speak for the CentOS project, but I'm doing something along those lines, albeit in a Debian-context with sbuild chroots (which are static, compared to mock's chroots, so getting this stuff working with mock could be painful).
Essentially, you can leverage the binfmt_misc Linux kernel feature inn tandem with qemu-static-user binaries. The Fedora package name for that is qemu-user-static, I believe it's the same for CentOS.
Basically, what you have to do is to make sure that BINFMT_MISC is enabled in the kernel (the default for any non-ancient version) and to copy the static user binary (typically /usr/bin/qemu-${arch}-static on the host system) to /usr/bin/qemu-${arch} into the chroot. Boom, suddenly you can just execute any native ${arch} binary within that chroot.
Note that this method has shortcomings. Qemu's emulation might be buggy and incomplete at times (for instance, I saw aptitude crashing due to a qemu bug, while it worked fine natively). It's also quite slow compared to native code execution and it might even be slower than a fully-emulated qemu-powered VM due to the additional overhead of having to spawn an ad-hoc VM for every new foreign-arch process.
Also, you'd better periodically update the qemu user static binary. Naturally this won't be done automatically.
In general, such a setup is doable. It has its drawbacks, but should mostly work.
The CentOS project doesn't have to work with such trickery. AFAIK, they just run native-arch builders, even for architectures such as aarch64.
Mihai
On Sat, Dec 5, 2020 at 2:35 PM Mihai Moldovan ionic@ionic.de wrote:
[Resent this message to the list without aa GPG signature, since my signature blows the message up past the 50 KB mark... meh.]
- On 12/5/20 8:15 PM, Simon Matter wrote:
[...] We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
I cannot speak for the CentOS project, but I'm doing something along those lines, albeit in a Debian-context with sbuild chroots (which are static, compared to mock's chroots, so getting this stuff working with mock could be painful).
Essentially, you can leverage the binfmt_misc Linux kernel feature inn tandem with qemu-static-user binaries. The Fedora package name for that is qemu-user-static, I believe it's the same for CentOS.
I'd use "mock" on an arm64 architecture host. If I lack spare hardware, I'd grab an AWS account and run mock in a CentOS 8 AMI. The mock provided for Amazon Linux is, I'm afraid, a bit out of date and tricky to manage due to "createrepo" versus "createrepo_c" dependencies.
On Sat, Dec 5, 2020 at 2:35 PM Mihai Moldovan ionic@ionic.de wrote:
[Resent this message to the list without aa GPG signature, since my signature blows the message up past the 50 KB mark... meh.]
- On 12/5/20 8:15 PM, Simon Matter wrote:
[...] We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance
it
and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the
new
arch?
How does the CentOS project do it, run ARM64 hardware or using
emulation?
Or to ask differently, is it an option to run the build system
emulated
with QEMU?
I cannot speak for the CentOS project, but I'm doing something along those lines, albeit in a Debian-context with sbuild chroots (which are static, compared to mock's chroots, so getting this stuff working with mock could be painful).
Essentially, you can leverage the binfmt_misc Linux kernel feature inn tandem with qemu-static-user binaries. The Fedora package name for that is qemu-user-static, I believe it's the same for CentOS.
I'd use "mock" on an arm64 architecture host. If I lack spare hardware, I'd grab an AWS account and run mock in a CentOS 8 AMI. The mock provided for Amazon Linux is, I'm afraid, a bit out of date and tricky to manage due to "createrepo" versus "createrepo_c" dependencies.
Thanks for all who answered so far. We're not using mock but use our own build system to build for a number of distributions. It still uses an older python based createrepo which I hoped would also work for the aarch64 packages. Will it work at all for aarch64 RPMs?
Will running a whole ARM64 VM going to be stable enough, I mean one with QEMU?
Thanks, Simon
On Sat, Dec 5, 2020 at 3:33 PM Simon Matter simon.matter@invoca.ch wrote:
On Sat, Dec 5, 2020 at 2:35 PM Mihai Moldovan ionic@ionic.de wrote:
[Resent this message to the list without aa GPG signature, since my signature blows the message up past the 50 KB mark... meh.]
- On 12/5/20 8:15 PM, Simon Matter wrote:
[...] We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance
it
and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the
new
arch?
How does the CentOS project do it, run ARM64 hardware or using
emulation?
Or to ask differently, is it an option to run the build system
emulated
with QEMU?
I cannot speak for the CentOS project, but I'm doing something along those lines, albeit in a Debian-context with sbuild chroots (which are static, compared to mock's chroots, so getting this stuff working with mock could be painful).
Essentially, you can leverage the binfmt_misc Linux kernel feature inn tandem with qemu-static-user binaries. The Fedora package name for that is qemu-user-static, I believe it's the same for CentOS.
I'd use "mock" on an arm64 architecture host. If I lack spare hardware, I'd grab an AWS account and run mock in a CentOS 8 AMI. The mock provided for Amazon Linux is, I'm afraid, a bit out of date and tricky to manage due to "createrepo" versus "createrepo_c" dependencies.
Thanks for all who answered so far. We're not using mock but use our own build system to build for a number of distributions. It still uses an older python based createrepo which I hoped would also work for the aarch64 packages. Will it work at all for aarch64 RPMs?
Will running a whole ARM64 VM going to be stable enough, I mean one with QEMU?
It should be fine, but I would recommend moving to createrepo_c since it's faster and creates more correct repodata.
If you're doing it by hand (that is, not with Mock), you need to be using DNF 2.6.2 or newer, as the --forcearch flag was introduced then.
The steps for doing it manually is documented by Mageia here: https://wiki.mageia.org/en/Using_DNF#Setting_up_a_container_for_a_non-native...
On Sat, Dec 5, 2020 at 2:15 PM Simon Matter simon.matter@invoca.ch wrote:
Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
Thanks for any insights?
The Mock tool from Fedora EPEL supports automatically running emulation for foreign architectures for building packages if qemu-user-static is installed. That makes building AArch64 packages from x86_64 or vice versa pretty simple.
To build a package for aarch64 from x86_64 from a fresh system using Mock:
$ sudo dnf install mock qemu-user-static $ sudo usermod -a -G mock $USER $ newgrp mock $ mock -r epel-8-aarch64 </path/to/package.src.rpm>
-- 真実はいつも一つ！/ Always, there's only one truth!
* On 12/5/20 8:39 PM, Neal Gompa wrote:
The Mock tool from Fedora EPEL supports automatically running emulation for foreign architectures for building packages if qemu-user-static is installed.
Heh, not even slightly painful after all, but with support built-in directly! Thanks for chiming in. :)
Mihai
On Sat, 5 Dec 2020 at 14:15, Simon Matter simon.matter@invoca.ch wrote:
Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
CentOS and Fedora use ARM64 hardware to do their builds. I think at one point it was being done with emulation but it was slower than expected.. [if you have only one thing it is probably not a large issue but if you have a chain of items to build continuously...] For hardware that works well in an 'enterprise' environment, I can recommend the Ampere systems that we are using in Fedora.
Thanks for any insights?
Simon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Sat, 5 Dec 2020 at 14:15, Simon Matter simon.matter@invoca.ch wrote:
Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
CentOS and Fedora use ARM64 hardware to do their builds. I think at one point it was being done with emulation but it was slower than expected.. [if you have only one thing it is probably not a large issue but if you have a chain of items to build continuously...] For hardware that works well in an 'enterprise' environment, I can recommend the Ampere systems that we are using in Fedora.
I only know the Gigabyte ARM servers, is it what you're using then?
For what we need they are a bit overkill then, because a dedicated Raspi4 may also do it for us.
Simon
Thanks for any insights?
Simon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Stephen J Smoogen. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 5/12/20 17:24, Simon Matter wrote:
On Sat, 5 Dec 2020 at 14:15, Simon Matter simon.matter@invoca.ch wrote:
Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
CentOS and Fedora use ARM64 hardware to do their builds. I think at one point it was being done with emulation but it was slower than expected.. [if you have only one thing it is probably not a large issue but if you have a chain of items to build continuously...] For hardware that works well in an 'enterprise' environment, I can recommend the Ampere systems that we are using in Fedora.
I only know the Gigabyte ARM servers, is it what you're using then?
I'm using a Solidrun Honeycomb, which is somewhere in the middle, It is not a server, but definitely more than a raspi.
For what we need they are a bit overkill then, because a dedicated Raspi4 may also do it for us.
Simon
Thanks for any insights?
Simon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Stephen J Smoogen. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Pablo
On Sat, 5 Dec 2020 at 15:24, Simon Matter simon.matter@invoca.ch wrote:
On Sat, 5 Dec 2020 at 14:15, Simon Matter simon.matter@invoca.ch
wrote:
Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
CentOS and Fedora use ARM64 hardware to do their builds. I think at one point it was being done with emulation but it was slower than expected.. [if you have only one thing it is probably not a large issue but if you have a chain of items to build continuously...] For hardware that works well in an 'enterprise' environment, I can recommend the Ampere systems that we are using in Fedora.
I only know the Gigabyte ARM servers, is it what you're using then?
For what we need they are a bit overkill then, because a dedicated Raspi4 may also do it for us.
https://amperecomputing.com/wp-content/uploads/2019/01/Lenovo_ThinkSystem_HR...
Simon
Thanks for any insights?
Simon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Stephen J Smoogen. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 05/12/2020 20:15, Simon Matter wrote:
Hi,
We are thinking about adding ARM64 based devices to our systems.
As we are using CentOS almost everywhere and have quite a number of inhouse RPM packages in our company repository, we'd like to enhance it and add ARM64 alias aarch64 to it.
What I'm still wondering is how to run the build environment for the new arch?
How does the CentOS project do it, run ARM64 hardware or using emulation? Or to ask differently, is it an option to run the build system emulated with QEMU?
Thanks for any insights?
Simon
Easy answer : we don't use emulation but build on target architecture[s] (same for IBM power ppc64 and ppc64le) For aarch64, we have a mix of ThunderX/ThunderX2 and Ampere servers. For armhfp, we run kvm armhfp guests on top of Ampere servers , so aarch64 hypervisors booting armhfp/armv7hl (worth knowing that Ampere let you do that, while it's *impossible* to run 32bits arm on ThunderX/ThunderX2)
If you don't have access to real aarch64 hardware and just want to use such nodes to build/compile/test, I'd suggest using some Cloud options. For example, we have official CentOS 7 and 8 aarch64 AMI images that can run natively on AWS EC2 aarch64 nodes
Emulation/translation for different arch (or cross-compiling) is nothing but asking for troubles further down the road ....
-
Fabian Arrotin -
Mihai Moldovan -
Neal Gompa -
Nico Kadel-Garcia -
Pablo Sebastián Greco -
Simon Matter -
Stephen John Smoogen