[CentOS-devel] Updates from today

Fri Mar 11 11:28:15 UTC 2011
Lamar Owen <lowen at pari.edu>

[While this message is fairly long, I did take the time to cut it down some from its original form, which was over 200 lines longer than the current form.  I'm sure glad I took touch-typing in school where they still used manual Remington-Rand typewriters....]

[Also, I had this in the last paragraph, but decided to top-post this piece: While those building C6 haven't given 'scheduled' updates, I want to thank them for the updates they have provided in reply to questions in these threads on these subjects, and for there continued work on the builds. ]

On Thursday, March 10, 2011 09:30:27 pm Trent Johnson wrote:
> What are the differences (if any) between the build procedures
> documented by Scientific Linux at the url below?
> https://www.scientificlinux.org/distributions/6x/build/problembyrpm

SL uses koji, except in those cases where the koji environment's mock produces the wrong result (as documented in that page above for the perl-Sys-Virt package, just picking the first one that wouldn't build at all in koji).  The SL started the koji 'adventure' way back in June, according to the SL archives, so it hasn't been a quick process for them, either.  CentOS does not use koji as documented on this list a while back.

In my investigation of doing my own rebuild, I looked at koji.  Koji does a fantastic job of managing a full distribution build where there are lots of contributors who are actually developing source RPM packaging; that was, after all, its design goal for the Fedora project.  Koji is overkill for doing a rebuild from someone else's source RPM repo where you are only maintaining a relative handful of the full package set, and where you are building from SRPM rather than out of an SCM (koji uses an SCM on the backend, and rebuilding out of SRPM isn't by default allowed except for koji admins).

Even then it's not a magic bullet for rebuilding from a set of distributed source RPMs, as the contents of the binary RPM repository koji uses to populate the mock buildroots is still not known (I'm talking about upstream's binaries and buildroots); that information is not public (again, from the upstream vendor, not CentOS).  What is known is that the buildroot-populating repository is a mix of packages from various sources; some might channel Frank Herbert and say a melange of sources.  And the building dependencies documented in the source RPMs do not tell the full story in terms of the whole distribution.  

Since the full CentOS package set has not yet successfully met the stringent binary compatibility tests that CentOS uses, I would say the full solution to the buildroot-populating binary repo is not yet known by CentOS, and thus can't yet be documented (it's just a smack difficult to document something you don't know yet).  

And since SL is built using koji, their results don't necessarily translate directly to a non-koji-fied mock builder, and while the page you linked is useful information as a starting point, there are multitudes of details, especially once you get down to just a small number of packages left to successfully build and pass binary verification.

And, to throw another wrinkle, upstream may not have gone GA with a completely consistent package set; that is, where all packages in the package set shared exactly the same buildroot repository, and the distribution includes all packages in the correct versions to rebuild itself; this is known as 'self-hosting.'  

We do know that 'self hosting' is not one of upstream's public goals, and, for that matter, never has been to the best of my knowledge (which is rather incomplete).  It is a Fedora goal (and is periodically tested), but upstream EL is really just loosely pulled from a running snapshot of some Fedora packages across a relatively wide timeline.... and thus there isn't really a rawhide snapshot at any single point in time that completely corresponds to the buildroot-populating binary package repository used by mock, whether in koji or outside of koji, to do the builds.

Many links to many pieces of information have been posted; grep the archives.  Enough information has been posted that anyone here with the requisite experience in package building could bootstrap, using the last known upstream beta binaries, and do their own rebuild of EL6, given the time required to do the work across the full package set.  You might get it done very quickly; but part of the draw of CentOS is the strict binary-linkage-and-requires-and-provides bug-for-bug compatibility, and that is what takes a lot of time to do.

I find it rather interesting that the buildsystem under mock does have an effect when (re)building EL6; the newer version of rpm has to be present on the buildhost for xz compression, for one.  So mock isn't total build isolation, unfortunately.  Its isolation is very good, though, for the actual compilation of package sources; but mock has to use the buildhost's toolchain to actually put together the resulting binary packages, thus a newer RPM than C5's is required, as stated by Johnny upthread.  This is a change from previous information.