Hi Folks,
We are going to announce soon a plan to upgrade cbs.centos.org and associated builders to be able to build standard C8 RPM packages, without interfering too much with SIGs ongoing work.
is this plan discussed on Mondays CBS meetings? I don't see it logged in https://www.centos.org/minutes/2019/September/
We didn't manage to have a meeting on Monday.
To conclude, we will send an update beginning of next week with more information, a schedule and a list of points that need to be sorted out with the community.
I'll try to give a summary of my investigation. I'll decouple two problems : the koji upgrade and having a working CentOS 8 target/buildroot in cbs.centos.org with CentOS 7 builders.
1/ koji upgrade
The first step is to upgrade our version of koji to match the version mbox uses. At the same time we move away from puppet to ansible (thanks Arrfab for all the work! [1] & [2]) and upgrade the base operating system from CentOS 6 to CentOS 7.
At the same time the builders (where mock runs) need to run specific version of certain tools for C8 buildroots :
distribution-gpg-keys.noarch 1.21-1.el7 kernel.x86_64 4.19.34-300.el7
mock.noarch 1.3.4-1.el7_build.rhel8 python-perf.x86_64 4.19.34-300.el7 python2-distro.noarch 1.0.3-1.el7.rhel8
* Status: Update works and db schema can be easily upgraded. We have few more test to validate that the previous mentioned RPM updates don't break anything for C6 & C7 targets. Next step is to announce a downtime and execute the upgrade plan.
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
* Status: Investigating possible solutions.
On Wed, 2 Oct 2019 at 11:07, Thomas Oulevey thomas.oulevey@cern.ch wrote:
Hi Folks,
We are going to announce soon a plan to upgrade cbs.centos.org and associated builders to be able to build standard C8 RPM packages, without interfering too much with SIGs ongoing work.
is this plan discussed on Mondays CBS meetings? I don't see it logged in https://www.centos.org/minutes/2019/September/
We didn't manage to have a meeting on Monday.
To conclude, we will send an update beginning of next week with more information, a schedule and a list of points that need to be sorted out with the community.
I'll try to give a summary of my investigation. I'll decouple two problems : the koji upgrade and having a working CentOS 8 target/buildroot in cbs.centos.org with CentOS 7 builders.
1/ koji upgrade
The first step is to upgrade our version of koji to match the version mbox uses. At the same time we move away from puppet to ansible (thanks Arrfab for all the work! [1] & [2]) and upgrade the base operating system from CentOS 6 to CentOS 7.
At the same time the builders (where mock runs) need to run specific version of certain tools for C8 buildroots :
distribution-gpg-keys.noarch 1.21-1.el7 kernel.x86_64 4.19.34-300.el7
mock.noarch 1.3.4-1.el7_build.rhel8 python-perf.x86_64 4.19.34-300.el7 python2-distro.noarch 1.0.3-1.el7.rhel8
- Status: Update works and db schema can be easily upgraded. We have few
more test to validate that the previous mentioned RPM updates don't break anything for C6 & C7 targets. Next step is to announce a downtime and execute the upgrade plan.
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
So the mergerepo process is how koji pulls packages in from external repos. It has 2 major methods (simple and bare?) which are used to determine what is going to be 'merged' into what the builder mock can see. Simple mode is the default and koji will usually pick something it built over something it didn't (which is why if an EPEL package is built with the same name it may trump a RHEL package with a bigger EVR). This works ok for non-modular repositories but modules are 'virtual' repositories.. so you need extra data to say that repo1 is chosen over repo2. koji simple just sees one large repo and says ok there are 2 perl-CGI with the same NEV? choose one whether it works or not.
bare mode adds some sophistication in that it asks dnf to choose what packages it will pull into the build root. This sort of fixes some items but adds others in that dnf doesn't also know which module you wanted so it can sometimes skip one or something else.
Enter grobisplitter. What this tool does is break up the release from virtual repositories into real repositories. You then split out the packages into repos, and then choose the ones you want in your build root. Then have koji use bare mode to pull in the packages so dnf can figure out some things simple doesn't. [It has been a while and my notes just say FIRE FIRE FIRE BURN IT ALL DOWN, IA IA IA..]
Is this overly complicated just to make a package? I think it is probably but it is similar to 1st/2nd generation automatic transmissions. It is trying to make things simpler for allowing users choices they wanted but couldn't get.. but it is also choosing all the wrong gears if you know damn well you want 2nd to go up this hill.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
- Status: Investigating possible solutions.
-- Thomas Oulevey
[1] : https://github.com/CentOS/ansible-role-kojihub [2] : https://github.com/CentOS/ansible-role-kojid [3] : https://github.com/fedora-modularity/GrobiSplitter [4] : https://koji.mbox.centos.org/koji/taginfo?tagID=3 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Wed, Oct 2, 2019 at 12:13 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 2 Oct 2019 at 11:07, Thomas Oulevey thomas.oulevey@cern.ch wrote:
Hi Folks,
We are going to announce soon a plan to upgrade cbs.centos.org and associated builders to be able to build standard C8 RPM packages, without interfering too much with SIGs ongoing work.
is this plan discussed on Mondays CBS meetings? I don't see it logged in
https://www.centos.org/minutes/2019/September/
We didn't manage to have a meeting on Monday.
To conclude, we will send an update beginning of next week with more information, a schedule and a list of points that need to be sorted
out
with the community.
I'll try to give a summary of my investigation. I'll decouple two problems : the koji upgrade and having a working CentOS 8 target/buildroot in cbs.centos.org with CentOS 7 builders.
1/ koji upgrade
The first step is to upgrade our version of koji to match the version mbox uses. At the same time we move away from puppet to ansible (thanks Arrfab for all the work! [1] & [2]) and upgrade the base operating system from CentOS 6 to CentOS 7.
At the same time the builders (where mock runs) need to run specific version of certain tools for C8 buildroots :
distribution-gpg-keys.noarch 1.21-1.el7 kernel.x86_64 4.19.34-300.el7
mock.noarch 1.3.4-1.el7_build.rhel8 python-perf.x86_64 4.19.34-300.el7 python2-distro.noarch 1.0.3-1.el7.rhel8
- Status: Update works and db schema can be easily upgraded. We have few
more test to validate that the previous mentioned RPM updates don't break anything for C6 & C7 targets. Next step is to announce a downtime and execute the upgrade plan.
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
So the mergerepo process is how koji pulls packages in from external repos. It has 2 major methods (simple and bare?) which are used to determine what is going to be 'merged' into what the builder mock can see. Simple mode is the default and koji will usually pick something it built over something it didn't (which is why if an EPEL package is built with the same name it may trump a RHEL package with a bigger EVR). This works ok for non-modular repositories but modules are 'virtual' repositories.. so you need extra data to say that repo1 is chosen over repo2. koji simple just sees one large repo and says ok there are 2 perl-CGI with the same NEV? choose one whether it works or not.
Koji has _three_ modes for merging external repos: koji (the default), simple, and bare.
The original Koji method tries to make repo merging behave more like Koji inheritance. It groups the external rpms by srpm name and performs the merge at that level, either taking or omitting each entire build. This is done with the --koji option to mergerepos. It also does things like honor koji's block list, prioritizes internal koji builds over external ones, and gives priority to earlier external repos in the list. It also writes a pkgorigins file so that Koji can figure out which repo the rpms originally came from.
The simple mode was added last year and as the name implies does much less. It honors Koji's block list and creates a pkgorigins file. Other than that, it's pretty much just a straight mergerepo.
The bare mode is actually even simpler. It performs /usr/bin/mergerepo_c --pkgorigins --all. So it writes pkgorigins, but does not honor the block list.
bare mode adds some sophistication in that it asks dnf to choose what packages it will pull into the build root. This sort of fixes some items but adds others in that dnf doesn't also know which module you wanted so it can sometimes skip one or something else.
Bare mode is not sophisticated. It is simpler than simple mode.
Bare mode was added for modularity, because they needed much more flexibility in their external repo content. The real work for modularity is passed on to mergerepo_c, which is expected to be module aware (Koji itself is not).
Side note: none of this is optimal. MBS was developed outside of Koji and very few Koji changes were made to enable it. Better Koji/MBS integration will come eventually.
Enter grobisplitter. What this tool does is break up the release from virtual repositories into real repositories. You then split out the packages into repos, and then choose the ones you want in your build root. Then have koji use bare mode to pull in the packages so dnf can figure out some things simple doesn't. [It has been a while and my notes just say FIRE FIRE FIRE BURN IT ALL DOWN, IA IA IA..]
Is this overly complicated just to make a package? I think it is probably but it is similar to 1st/2nd generation automatic transmissions. It is trying to make things simpler for allowing users choices they wanted but couldn't get.. but it is also choosing all the wrong gears if you know damn well you want 2nd to go up this hill.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
- Status: Investigating possible solutions.
-- Thomas Oulevey
[1] : https://github.com/CentOS/ansible-role-kojihub [2] : https://github.com/CentOS/ansible-role-kojid [3] : https://github.com/fedora-modularity/GrobiSplitter [4] : https://koji.mbox.centos.org/koji/taginfo?tagID=3 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Stephen J Smoogen. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 02/10/2019 17:07, Thomas Oulevey wrote:
Hi Folks,
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
- Status: Investigating possible solutions.
Hi,
since I keep hearing interest of building SIG packages for CentOS 8, is there an update on this topic?
Matthias
On Thu, Oct 17, 2019, at 11:24, Matthias Runge wrote:
On 02/10/2019 17:07, Thomas Oulevey wrote:
Hi Folks,
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
- Status: Investigating possible solutions.
Hi,
since I keep hearing interest of building SIG packages for CentOS 8, is there an update on this topic?
Matthias _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
There are a lot of things happening behind the scenes that needed to be resolved before we open up the gates and do builds against CentOS 8 or CentOS Stream buildroots. Here are the 2 major items:
- Upgrading koji inCBS: (Fabian and Thomas can correct me if I'm wrong) This should be ready enough to set a date for the upgrade
- Dealing with modular content in the buildroot: We're closer than I expected us to be and I'd like to finish this discussion with a proposal
Over the past few weeks, we haven't been holding the regular CBS meeting since most of us were busy wrangling these items and others. I'd like for us to start the meetings back again on Monday 22-October 14:00 UTC in #centos-meeting on Freenode
At the 22-October meeting, we'll outline the solutions and proposals we came up with and decide on a schedule (if the necessary folks are present). Come join us for that discussion.
-- Brian Stinson
On Thu, Oct 17, 2019, at 11:52, Brian Stinson wrote:
On Thu, Oct 17, 2019, at 11:24, Matthias Runge wrote:
On 02/10/2019 17:07, Thomas Oulevey wrote:
Hi Folks,
2/ CentOS 8 buildroot
When 1/ is done we can create an C8 buildroot without enabling the Appstream repo.
Due to the nature of Appstream, and how koji uses mergerepo (--koji)/updaterepo/mock, more work is needed to get a working buildroot.
I don't fully understand the underlying issue so comments are very welcome to enlighten us !
One of the solution found by Fedora team is to split Appstream per repository and enable only the one needed in a specific buildroot [3].
We are evaluating the tools. And see if we can use koji external-repo or tag inheritance to achieve it (which is mbox solution [4])
However for a community build service, we don't know exactly what people will build and what Appstream packages will be needed in each buildroot (we have 290 targets at this time for C7). So we still need time and new ideas :-)
- Status: Investigating possible solutions.
Hi,
since I keep hearing interest of building SIG packages for CentOS 8, is there an update on this topic?
Matthias _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
There are a lot of things happening behind the scenes that needed to be resolved before we open up the gates and do builds against CentOS 8 or CentOS Stream buildroots. Here are the 2 major items:
- Upgrading koji inCBS: (Fabian and Thomas can correct me if I'm wrong)
This should be ready enough to set a date for the upgrade
- Dealing with modular content in the buildroot: We're closer than I
expected us to be and I'd like to finish this discussion with a proposal
Over the past few weeks, we haven't been holding the regular CBS meeting since most of us were busy wrangling these items and others. I'd like for us to start the meetings back again on Monday 22-October 14:00 UTC in #centos-meeting on Freenode
At the 22-October meeting, we'll outline the solutions and proposals we came up with and decide on a schedule (if the necessary folks are present). Come join us for that discussion.
My apologies, those both should read '21-October' we are indeed meeting at our usual time on *Monday*
-- Brian Stinson _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 17/10/2019 18:52, Brian Stinson wrote:
There are a lot of things happening behind the scenes that needed to be resolved before we open up the gates and do builds against CentOS 8 or CentOS Stream buildroots. Here are the 2 major items:
Upgrading koji inCBS: (Fabian and Thomas can correct me if I'm wrong) This should be ready enough to set a date for the upgrade
Dealing with modular content in the buildroot: We're closer than I expected us to be and I'd like to finish this discussion with a proposal
Over the past few weeks, we haven't been holding the regular CBS meeting since most of us were busy wrangling these items and others. I'd like for us to start the meetings back again on Monday 22-October 14:00 UTC in #centos-meeting on Freenode
At the 22-October meeting, we'll outline the solutions and proposals we came up with and decide on a schedule (if the necessary folks are present). Come join us for that discussion.
Thank you Brian,
this is good to see this moving forward.
Best, Matthias
Hi Folks,
Let me give you an update after this week upgrade.
I'll try to give a summary of my investigation. I'll decouple two problems : the koji upgrade and having a working CentOS 8 target/buildroot in cbs.centos.org with CentOS 7 builders.
1/ koji upgrade
Koji upgrade has been done and we didn't detect any major issues. We upgraded all build service components to use CentOS 7 and additional packages needed for CentOS 8 buildroots have been installed.
Please report bugs at https://bugs.centos.org if you find issues.
2/ CentOS 8 buildroot
As now 1/ is finished we finally have a plan for C8 buildroot in the Community build service.
Brian will generate, in next days, two 'compose' (C8 and C8 Stream) from https://koji.mbox.centos.org/koji/ and we will declare it as an external repo in CBS.
We will validate that those 'compose' allow use to build (most of) SIG package in our test instance and then deploy it to the production one.
Please note that the 'compose' will be (automagically) regenerated on every new updates. On the koji side we will detect those changes and regenerate all affected buildroot tags when there is an update.
In term of impact, we may have to reconfigure some option on the koji side, but it will likely be transparent. We hope to have a working solution in the next two weeks.
I'd like now discuss few additional points.
3/ Namespace for C8 and C8 Stream
Koji allow to have a single NVR for each build, you cannot rebuild a NVR that exist. So to be able to build in parallel for C8 and C8 Stream the same package we will need to use a different disttag.
The proposal is to use ".el8" and ".el8s".
At this point SIGs will have to submit twice the builds to koji, I hope we can automate further this step with a better centos-packager in the beginning of 2020.
Let us know if you have some feedback.
4/ Building from GIT
During the previous upgrade we deployed the changes needed to be able to build from git.centos.org for SIGs (mbox allowed it already). We still have to do some validation but expect this feature to be available soon as well. We will document the process in the wiki when we have a working PoC.
Thanks for reading and let us know if you have questions,
On Tue, Oct 29, 2019 at 11:24:10AM +0100, Thomas Oulevey wrote:
Hi Folks,
Let me give you an update after this week upgrade.
Great, thanks for the details!
Niels
I'll try to give a summary of my investigation. I'll decouple two problems : the koji upgrade and having a working CentOS 8 target/buildroot in cbs.centos.org with CentOS 7 builders.
1/ koji upgrade
Koji upgrade has been done and we didn't detect any major issues. We upgraded all build service components to use CentOS 7 and additional packages needed for CentOS 8 buildroots have been installed.
Please report bugs at https://bugs.centos.org if you find issues.
2/ CentOS 8 buildroot
As now 1/ is finished we finally have a plan for C8 buildroot in the Community build service.
Brian will generate, in next days, two 'compose' (C8 and C8 Stream) from https://koji.mbox.centos.org/koji/ and we will declare it as an external repo in CBS.
We will validate that those 'compose' allow use to build (most of) SIG package in our test instance and then deploy it to the production one.
Please note that the 'compose' will be (automagically) regenerated on every new updates. On the koji side we will detect those changes and regenerate all affected buildroot tags when there is an update.
In term of impact, we may have to reconfigure some option on the koji side, but it will likely be transparent. We hope to have a working solution in the next two weeks.
I'd like now discuss few additional points.
3/ Namespace for C8 and C8 Stream
Koji allow to have a single NVR for each build, you cannot rebuild a NVR that exist. So to be able to build in parallel for C8 and C8 Stream the same package we will need to use a different disttag.
The proposal is to use ".el8" and ".el8s".
At this point SIGs will have to submit twice the builds to koji, I hope we can automate further this step with a better centos-packager in the beginning of 2020.
Let us know if you have some feedback.
4/ Building from GIT
During the previous upgrade we deployed the changes needed to be able to build from git.centos.org for SIGs (mbox allowed it already). We still have to do some validation but expect this feature to be available soon as well. We will document the process in the wiki when we have a working PoC.
Thanks for reading and let us know if you have questions,
Thomas 'alphacc' Oulevey
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-
Brian Stinson -
Matthias Runge -
Mike McLean -
Niels de Vos -
Stephen John Smoogen -
Thomas Oulevey