[Arm-dev] a plague farm ?

Fri Apr 24 16:23:53 UTC 2015
Gordan Bobic <gordan at redsleeve.org>

On 2015-04-24 17:06, Troy Dawson wrote:
> On Thu, Apr 23, 2015 at 10:26 AM, Gordan Bobic <gordan at redsleeve.org>
> wrote:
> 
>> On 2015-04-23 16:08, Karanbir Singh wrote:
>> On 04/23/2015 02:38 PM, Troy Dawson wrote:
>> 
>> On Fri, Apr 17, 2015 at 11:07 AM, Karanbir Singh
>> <mail-lists at karan.org
>> <mailto:mail-lists at karan.org>> wrote:
>> 
>> On 17/04/15 14:38, Mandar Joshi wrote:
>>>> I can work with Online/Scaleway to build the first CentOS
>> image for
>>>> their systems.
>>> I would like to join the CentOS ARM effort officially.
>> 
>> nothing official here :) people doing the work are in, people
>> hanging
>> around talking and hand waving are not.
>> 
>> This doesn't make any sense.
>> So, if people want to join the CentOS arm effort, they cannot,
>> unless
>> they are already part of the CentOS arm effort?
>> How can people change from "hand waving" to actually doing work?
>> 
>> as you can see, there is no real 'CentOS Arm Effort' - whatever is
>> happening is being done by people on their own, on their own
>> hardware
>> using the centos sources from git.centos.org [1] or srpms.
>> 
>> hoping to fix that with a plague setup, across some nodes, that
>> people
>> can ask for access to and then pool in the work being done.
> 
>  Having been doing exactly this kind of thing for the past 3-4 years
> with
>  the RedSleeve project, IMO the most important part of this kind of an
> effort,
>  if it is to be a community effort that scales, is coordination.
> 
>  Ultimately, anyone can take a F19 image and packages and write a 20
>  line bash script to rebuild the EL7 src.rpms using mock, then do it
>  again based on the packages that fell out of the first stage, and
>  then doing it again based on the package that fall out of the second
>  stage just to make sure. That just takes CPU time, and situation
>  today is massively less bad than it was about 4 years ago when I
> first
>  started working on it. For example, back the RedSleeve builders were
>  all Sheeva/Guru/Dream Plus, with 512MB of RAM and running on an NFS
>  rootfs.
> 
>  Today you can get a single 8-core ARM board with 4GB of RAM, which
>  will outright outperform my entire old build farm on it's own, and
>  avoid the complications of running on NFS by having a SATA port
>  (running NFS root on the build directory caused some package self
>  tests to fail, at least back on EL6).
> 
>  The part that is human time consuming is fixing all the packages
>  that refuse/fail to build. On EL6 there was over a hundred. On EL7,
>  I'm told it's less bad, but I'm not so sure - having tried to build
>  an i686 EL7 build, there seems to be a lot of spurious build
> breakages
>  so it isn't a plain, simple, out of the box "just works" process.
>  Having some way to distribute that effort is the difficult part to
>  coordinate. Raising tickets is cumbersome, and some issues often
>  disappear on a subsequent rebuild (and some things fail in stage 2
>  even though they build in stage 1 based off Fedora packages).
> 
>  If anyone has a sensible method to automate such a process, I would
>  love to hear about it. Otherwise it takes somebody to take charge
>  and effectively perform such coordination as almost a full time job
>  until the initial complete release build is rolled out (after that
>  the effort reduces dramatically).
> 
> I wish I could say I had a nice way.  It seems to be 90% of the work
> on 10% of the packages.

Yes, that's about right.

> For the most part, doing the build loop like you said works for alot
> of the packages.  The majority of the packages that needed tweaking
> are the ones that RHEL7 has marked as x86_64 only.

Since RHEL src rpms are no longer available, CentOS is about as
upstream as it gets for the rest of us. And in CentOS, IIRC from
the other day, there are a total of 2 packages that are marked as
x86-64 only (as in not including i686). I don't know how many are
marked as x86* only.

> I will give this tip.  Don't start on Fedora 20 for your build repo,
> or even use it partway through.  I did that on my arm build and was
> delighted that so many package built.  Then after a month I started
> checking dependencies on them.  Hardly anything would install because
> of the crazy dependencies. I think libpng was the biggest issue.  It
> wanted the version of libpng in Fedora 20, which was newer than the
> version in RHEL7.  Had to rebuild the whole thing again.

I seem to recall that Jacco bootstrapped stage 1 with F18, because
that is the last Fedora with a soft-float image available and we are
targetting RedSleeve at armv5tel. That seems to have worked out quite
well.

I am a few days' testing and some de-branding away from announcing
RS 7 alpha release for public consumption, but if you want to take
it for a spin right now you can find everything you need on the RS
mirrors.

AFAIK there is no intention for CentOS to support armv5tel, only
armv7hl.

My biggest current frustration is that there seems to be no
official effort of any kind to actually get an ARM build released
and made available. There were noises about it back in EL6 days, and
nothing happened (RedSleeve came about because I needed it and I
felt that those claiming it would be too difficult needed putting
in their place and those making  noise but not producing anything
publicly available needed some motivating competition.

Sadly, here we are with EL7 and while the nay-sayers have piped
down, there is nothing at all visible and available in terms of
a wider concerted effort. In other words, very little has changed.

Gordan