On Wed, Oct 2, 2019 at 12:13 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 2 Oct 2019 at 11:07, Thomas Oulevey thomas.oulevey@cern.ch wrote:
Hi Folks,
We are going to announce soon a plan to upgrade cbs.centos.org and associated builders to be able to build standard C8 RPM packages, without interfering too much with SIGs ongoing work.
is this plan discussed on Mondays CBS meetings? I don't see it logged in
https://www.centos.org/minutes/2019/September/
We didn't manage to have a meeting on Monday.
To conclude, we will send an update beginning of next week with more information, a schedule and a list of points that need to be sorted
out
with the community.
I'll try to give a summary of my investigation. I'll decouple two problems : the koji upgrade and having a working CentOS 8 target/buildroot in cbs.centos.org with CentOS 7 builders.
1/ koji upgrade
The first step is to upgrade our version of koji to match the version mbox uses. At the same time we move away from puppet to ansible (thanks Arrfab for all the work! [1] & [2]) and upgrade the base operating system from CentOS 6 to CentOS 7.
At the same time the builders (where mock runs) need to run specific version of certain tools for C8 buildroots :
distribution-gpg-keys.noarch 1.21-1.el7 kernel.x86_64 4.19.34-300.el7
mock.noarch 1.3.4-1.el7_build.rhel8 python-perf.x86_64 4.19.34-300.el7 python2-distro.noarch 1.0.3-1.el7.rhel8
- Status: Update works and db schema can be easily upgraded. We have few
more test to validate that the previous mentioned RPM updates don't break anything for C6 & C7 targets. Next step is to announce a downtime and execute the upgrade plan.
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
So the mergerepo process is how koji pulls packages in from external repos. It has 2 major methods (simple and bare?) which are used to determine what is going to be 'merged' into what the builder mock can see. Simple mode is the default and koji will usually pick something it built over something it didn't (which is why if an EPEL package is built with the same name it may trump a RHEL package with a bigger EVR). This works ok for non-modular repositories but modules are 'virtual' repositories.. so you need extra data to say that repo1 is chosen over repo2. koji simple just sees one large repo and says ok there are 2 perl-CGI with the same NEV? choose one whether it works or not.
Koji has _three_ modes for merging external repos: koji (the default), simple, and bare.
The original Koji method tries to make repo merging behave more like Koji inheritance. It groups the external rpms by srpm name and performs the merge at that level, either taking or omitting each entire build. This is done with the --koji option to mergerepos. It also does things like honor koji's block list, prioritizes internal koji builds over external ones, and gives priority to earlier external repos in the list. It also writes a pkgorigins file so that Koji can figure out which repo the rpms originally came from.
The simple mode was added last year and as the name implies does much less. It honors Koji's block list and creates a pkgorigins file. Other than that, it's pretty much just a straight mergerepo.
The bare mode is actually even simpler. It performs /usr/bin/mergerepo_c --pkgorigins --all. So it writes pkgorigins, but does not honor the block list.
bare mode adds some sophistication in that it asks dnf to choose what packages it will pull into the build root. This sort of fixes some items but adds others in that dnf doesn't also know which module you wanted so it can sometimes skip one or something else.
Bare mode is not sophisticated. It is simpler than simple mode.
Bare mode was added for modularity, because they needed much more flexibility in their external repo content. The real work for modularity is passed on to mergerepo_c, which is expected to be module aware (Koji itself is not).
Side note: none of this is optimal. MBS was developed outside of Koji and very few Koji changes were made to enable it. Better Koji/MBS integration will come eventually.
Enter grobisplitter. What this tool does is break up the release from virtual repositories into real repositories. You then split out the packages into repos, and then choose the ones you want in your build root. Then have koji use bare mode to pull in the packages so dnf can figure out some things simple doesn't. [It has been a while and my notes just say FIRE FIRE FIRE BURN IT ALL DOWN, IA IA IA..]
Is this overly complicated just to make a package? I think it is probably but it is similar to 1st/2nd generation automatic transmissions. It is trying to make things simpler for allowing users choices they wanted but couldn't get.. but it is also choosing all the wrong gears if you know damn well you want 2nd to go up this hill.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
- Status: Investigating possible solutions.
-- Thomas Oulevey
[1] : https://github.com/CentOS/ansible-role-kojihub [2] : https://github.com/CentOS/ansible-role-kojid [3] : https://github.com/fedora-modularity/GrobiSplitter [4] : https://koji.mbox.centos.org/koji/taginfo?tagID=3 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Stephen J Smoogen. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel