[Arm-dev] Work in progress ARM v7 port

Tue Feb 17 18:48:38 UTC 2015

Hello all.  Recently I've been having a bit of fun trying to port CentOS 
7 to ARMv7, and I thought I'd share info on what I've been doing with 
the list.

As a long-time Red Hat family distribution user, I've been interested in 
a CentOS build for ARM since the release of the originally Raspberry 
Pi.  However, it's only with the release of RHEL 7 that a source code 
base that largely works on ARM has been available to CentOS [1].  The 
announcement and release of the new, faster, quad-core ARMv7-based 
Raspberry Pi 2 sounded like an ideal reason to have a go at making a 
CentOS-like RHEL rebuild for ARM, so I ordered a couple of Pi 2s, and 
started having a think about how I'd go about building EL7.

Having been on the periphery of the CentOS project for quite a while, I 
knew how CentOS 7 was built: the RHEL 7 sources were rebuilt multiple 
times; first against the public RHEL 7 beta, then against the results of 
this build, then against then those results, and so on.  Ideally, a 
similar process could be used to built EL7 for ARM, but of course there 
has never been a RHEL 7 release for ARM of any sort.  Fortunately, 
Fedora 19, which RHEL 7 was forked from, does have a public (and 
functional) ARM release.  So an EL7 ARM port can be bootstrapped from 
the Fedora binary RPMs.

Of course, before I could start rebuilding CentOS sources, I'd need an 
OS image to rebuild on.  Fortunately, the Raspberry Pi's bootloader 
system is pretty simple: the Pi's firmware looks for specifically named 
files in the VFAT-formatted first partition of the SD card.  The Fedora 
19 Minimal image has an ext4 /boot partition which is easily 
reformatted, and populated with the contents of /boot from the Pi 2 
Rasbian image, and the kernel modules to go with the Rasbian-shipped 
kernel copied to the ext4 / filesystem.  A quick (not really) dd of the 
image to an SD card, and I had a booting Pi 2.  The next step was to 
test out running mock to build Fedora 19 chroots ready to build packages 
in.  Testing showed that it took the Pi roughly 8m30s to create a basic 
F19 mock chroot on the SD card.

At the same time as starting out working with the Pi 2, a friend pointed 
me at the ODROID C1 [2], a Pi-like small ARM board.  The C1 has a couple 
of advantages over even the Pi 2: it has a somewhat faster CPU [3], but, 
more importantly, it has on-SoC gigabit ethernet [4].  The ODROID is 
designed and manufactured by Hardkernel in South Korea. The ODROID is a 
bit more expensive (£33 from a UK supplier [5] versus the £25 Pi 2), but 
still quite affordable. Realising that building RPMs on a Pi on the SD 
card or over the network would be very slow, I ordered a C1 to try.

I was really rather pleased with the C1 when it arrived.  After trying 
and failing to modify the Fedora 19 ARM Minimal image with an 
ODROID-specific kernel [6], I ended up grabbing an F21 image made by one 
of the ODROID forum members [7].  The (one of the) nice thing(s) about 
building RPMs in mock is that as long as the host OS can run mock, it 
doesn't really matter what the host OS is, so a Fedora 21 OS for the C1s 
is fine.

Of course, even the ODROID wouldn't be able to make mock chroots very 
quickly on their SD card, but the built-in gigabit ethernet makes 
another option quite sensible: iSCSI.  By mounting a block device [8] 
from a server over iSCSI, the C1 can create an F19 chroot in about 
2m40s.  Having decided the C1 was better for the builder job than the Pi 
2, I borrowed another C1 from my friend, and put an order in for two more.

In the meantime, I made a start on building packages using the C1. The 
sensible starting point was building the C7 versions of the packages 
that go into the basic mock build chroot.  The basic idea here is that 
the sooner that the set of packages used to build all packages is 
replaced with the C7 versions, the closer the OS will be to a "true" C7 
OS [9]. Once this base set was built, the order of the remaining 1000+ 
SRPMs is a bit less critical, and more at the mercy of my random whims 
[10].  I should also note that I'm concentrating on building the CentOS 
7 GA package set, not any of the updates.  Once I have a C7 GA package 
set I'm happy with, then I'll look at all the package updates from GA to 
current.

I was originally planning to knock up a script or two to wrap mock and 
do createrepo runs.  Instead I decided to dust off Plague, the old build 
system Seth Vidal and co wrote for Fedora Extras.  Plague takes care of 
managing a queue of SRPMs to distribute to multiple builders, running 
builds, collecting resulting logs and RPMs, and making yum repos.  
Plague is a tad old, and needed a few code changes to support ARM 
architectures.  It's also a bit... tempremental at times, as the central 
manager process has issues spotting when builders come and go, but 
overall it's done quite well.

So what's the current status of the rebuild effort?  Of the 2481 SRPMs 
in CentOS 7.0.1406 (the GA C7 release), 1125 don't need to be rebuilt 
for ARM, either because they're entirely noarch packages [11], or 
because they're listed as either "exclusivearch", which means they 
should only be built on a specific list of architectures [12].  Of the 
remaining 1356 packages, 943 have been successfully rebuilt, 62 have 
failed to build, and 351 haven't been attempted yet.  Thus far only two 
RPMs have needed to be modified: centos-release had to be altered to not 
conflict with the Fedora 19 systemd (both packages contain 
/usr/lib/systemd/system-preset/90-default.preset), but is now replaced 
with the stock C7 version, and elfutils has had its test suite disabled 
[13].  Very much on the positive side, after a 36 hour build, the third 
build attempt of libreoffice completed successfully [14].

The other thing that's worth saying is that I've not yet looked at any 
ARM-specific "special" stuff, like an ARM kernel build, or any 
bootloader support.  That's potentially hard stuff, and will take some 
thinking about [15].

I'm planning to put the various hacky scripts I'm using into a github 
repo or similar, and I'll be posting status updates in the future with 
how I get on.

Cheers!

[1] RHEL6's Fedora 12/13-derived codebase largely predates the Fedora 
ARM effort, whereas the Fedora 19 base of RHEL 7 had an actively 
maintained ARM secondary architecture; fixes to Fedora packages for ARM 
were incorporated into Fedora proper, and RHEL 7 inherited these fixes.

[2] 
http://www.hardkernel.com/main/products/prdt_info.php?g_code=G141578608433

[3] Four 1.5GHz cores versus the Pi 2's four 900MHz cores.

[4] Contrasting with the Pi's fast ethernet adapter wired to the onboard 
USB hub.

[5] http://www.lilliputdirect.com/odroid-c1-quad-core-computer

[6] Not helped by me not having bought a serial cable, so I couldn't 
watch the bootloader messages.

[7] http://forum.odroid.com/viewtopic.php?f=114&t=8872

[8] SSD backed, using a Samsung EVO 850 SATA SSD.

[9] To the point where it might save an entire OS rebuild cycle, which 
will be a not inconsiderate amount of effort.

[10] For example, one night I queued up all SRPMs with names starting 
with "lib"; unfortunately I forgot that that would include libreoffice...

[11] So we can use the C7 builds unmodified.

[12] Packages in this group are generally platform-specific packages 
like efibootmgr or dmidecode which make no sense on ARMv7 platforms.

[13] Due to a single failing test that wasn't run in the Fedora 19 
elfutils build.

[14] The first died when the C1 ran out of RAM, which made me add 
iSCSI-backed swap to all the builders, while the second ended when I 
found that the libreoffice build requires more than 16GB of space; on 
the bright side I found out that the iSCSI LUNs can be resized entirely 
online.

[15] Apologies for all the footnotes, I like them.

-- 
HJ