On 03/11/2011 05:28 AM, Lamar Owen wrote: > [While this message is fairly long, I did take the time to cut it down some from its original form, which was over 200 lines longer than the current form. I'm sure glad I took touch-typing in school where they still used manual Remington-Rand typewriters....] > > [Also, I had this in the last paragraph, but decided to top-post this piece: While those building C6 haven't given 'scheduled' updates, I want to thank them for the updates they have provided in reply to questions in these threads on these subjects, and for there continued work on the builds. ] > > On Thursday, March 10, 2011 09:30:27 pm Trent Johnson wrote: >> What are the differences (if any) between the build procedures >> documented by Scientific Linux at the url below? >> https://www.scientificlinux.org/distributions/6x/build/problembyrpm > > SL uses koji, except in those cases where the koji environment's mock produces the wrong result (as documented in that page above for the perl-Sys-Virt package, just picking the first one that wouldn't build at all in koji). The SL started the koji 'adventure' way back in June, according to the SL archives, so it hasn't been a quick process for them, either. CentOS does not use koji as documented on this list a while back. > > In my investigation of doing my own rebuild, I looked at koji. Koji does a fantastic job of managing a full distribution build where there are lots of contributors who are actually developing source RPM packaging; that was, after all, its design goal for the Fedora project. Koji is overkill for doing a rebuild from someone else's source RPM repo where you are only maintaining a relative handful of the full package set, and where you are building from SRPM rather than out of an SCM (koji uses an SCM on the backend, and rebuilding out of SRPM isn't by default allowed except for koji admins). Exactly ... koji is a wonderful thing for "maintaining 100,000 packages from source" ... it is NOT so great for building someone else's SRPMS where you are not maintaining the sources (though as you say, koji admins can push SRPMS in). We are not using koji because of this reason. I have set up a koji system, built some packages in it and we have decided that it is not what we want to use. But regardless of whether you are using koji, plague, or just manually building via mock, it is mock that does the heavy lifting. > Even then it's not a magic bullet for rebuilding from a set of distributed source RPMs, as the contents of the binary RPM repository koji uses to populate the mock buildroots is still not known (I'm talking about upstream's binaries and buildroots); that information is not public (again, from the upstream vendor, not CentOS). What is known is that the buildroot-populating repository is a mix of packages from various sources; some might channel Frank Herbert and say a melange of sources. And the building dependencies documented in the source RPMs do not tell the full story in terms of the whole distribution. And since we have not completed the "check binaries" phase yet, we do not know the exact iteration of what packages we need as our "binary repos" yet. We have available to use the RHEL6B2 and Fedora 12 binaries, and the CentOS 6 binaries that we have already built as a staged repo to choose from. > Since the full CentOS package set has not yet successfully met the stringent binary compatibility tests that CentOS uses, I would say the full solution to the buildroot-populating binary repo is not yet known by CentOS, and thus can't yet be documented (it's just a smack difficult to document something you don't know yet). This is also exactly correct ... we do not yet have the full build, so we are not sure exactly what it takes and we have certainly not tried to rebuild it on itself. > And since SL is built using koji, their results don't necessarily translate directly to a non-koji-fied mock builder, and while the page you linked is useful information as a starting point, there are multitudes of details, especially once you get down to just a small number of packages left to successfully build and pass binary verification. > > And, to throw another wrinkle, upstream may not have gone GA with a completely consistent package set; that is, where all packages in the package set shared exactly the same buildroot repository, and the distribution includes all packages in the correct versions to rebuild itself; this is known as 'self-hosting.' > > We do know that 'self hosting' is not one of upstream's public goals, and, for that matter, never has been to the best of my knowledge (which is rather incomplete). It is a Fedora goal (and is periodically tested), but upstream EL is really just loosely pulled from a running snapshot of some Fedora packages across a relatively wide timeline.... and thus there isn't really a rawhide snapshot at any single point in time that completely corresponds to the buildroot-populating binary package repository used by mock, whether in koji or outside of koji, to do the builds. And this version (EL6) is less "complete" than most because it is much harder to get all the upstream RPMS for testing, etc. There is an "optional tree" that does not seem to be available on any ISOs, but that is part of the upstream distribution. This total change in the way upstream distributes their packages (many items in the distro, but not on the ISO media) is something else we are going to need to wrap our hands around ... especially for doing comparisons. > Many links to many pieces of information have been posted; grep the archives. Enough information has been posted that anyone here with the requisite experience in package building could bootstrap, using the last known upstream beta binaries, and do their own rebuild of EL6, given the time required to do the work across the full package set. You might get it done very quickly; but part of the draw of CentOS is the strict binary-linkage-and-requires-and-provides bug-for-bug compatibility, and that is what takes a lot of time to do. > > I find it rather interesting that the buildsystem under mock does have an effect when (re)building EL6; the newer version of rpm has to be present on the buildhost for xz compression, for one. So mock isn't total build isolation, unfortunately. Its isolation is very good, though, for the actual compilation of package sources; but mock has to use the buildhost's toolchain to actually put together the resulting binary packages, thus a newer RPM than C5's is required, as stated by Johnny upthread. This is a change from previous information. Right, one would have to either roll in a newer version of RPM into the build host (that can handle the xz compression of the SRPMS) ... or you can write a script to rebuild the SRPMS without xz compression (an equally valid approach that has also been used by us). You would just need to do this to set them up to build on c5: rpm -Uvh --nomd5 $SRPMS; rpm -bs <path_to_specfile>/<name>.spec Another approach is to build on RHEL6B2 or Fedora 12, or RHEL6 proper if you want to do that. Regardless of the approach, the SRPMS and RPMS from within the build root (not the buildhost OS) will have xz compression. We will also rebuild the packages again on CentOS6 when it is available and those will be the packages we release. We will continue to build CentOS-4 and CentOS-5 packages on a CentOS-5 buildhost machine and CentOS-6 will be built on a CentOS-6 buildhost machine (once CentOS-6 is done). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20110311/df92d65f/attachment-0007.sig>