Hi,
As many of you are aware, we are migrating CentOS Stream RPM workflows to Konflux and we'd like some early community feedback.
For those unfamiliar, Konflux is an open source, cloud-native build system for producing trusted software artefacts - https://konflux-ci.dev/. In our case, it produces the RPM builds that then land in Koji. You can learn more about it and some of the other changes in the past year from the following talk:
CentOS Connect talk: https://www.youtube.com/watch?v=1yhTvL_rk-I - This talk by Troy and me covers what changed in the CentOS Stream pipeline over the past year, including additional Konflux-related updates and direction.
Updated CentOS Stream docs: https://docs.centos.org/centos-stream-docs/build/
Relevant repos related to Konflux and our pipelines: https://gitlab.com/redhat/centos-stream/rpms/osci-internal-test-package - test RPM package used to validate CentOS Stream/Konflux build flow safely. https://gitlab.com/redhat/centos-stream/konflux-tests/rpmbuild-pipeline - CentOS Stream Konflux RPM pipeline test repo used for pipeline integration and iteration.
We've completed the following test build using Konflux: https://kojihub.stream.centos.org/koji/buildinfo?buildID=110177. We'd appreciate you reviewing the build logs and letting us know if you spot any issues or concerns with this new build system.
One important thing to note is Koji is not going away; Konflux is about how we produce builds upstream of Koji, and the aim is for the resulting packages to behave the same as they do today.
Thanks, Sean
On 5/8/26 10:06, Sean Ryan via devel wrote:
Hi,
As many of you are aware, we are migrating CentOS Stream RPM workflows to Konflux and we'd like some early community feedback.
For those unfamiliar, Konflux is an open source, cloud-native build system for producing trusted software artefacts - https://konflux-ci.dev/. In our case, it produces the RPM builds that then land in Koji. You can learn more about it and some of the other changes in the past year from the following talk:
CentOS Connect talk: https://www.youtube.com/watch?v=1yhTvL_rk-I - This talk by Troy and me covers what changed in the CentOS Stream pipeline over the past year, including additional Konflux-related updates and direction.
Updated CentOS Stream docs: https://docs.centos.org/centos-stream-docs/build/
Relevant repos related to Konflux and our pipelines: https://gitlab.com/redhat/centos-stream/rpms/osci-internal-test-package - test RPM package used to validate CentOS Stream/Konflux build flow safely. https://gitlab.com/redhat/centos-stream/konflux-tests/rpmbuild-pipeline - CentOS Stream Konflux RPM pipeline test repo used for pipeline integration and iteration.
We've completed the following test build using Konflux: https://kojihub.stream.centos.org/koji/buildinfo?buildID=110177. We'd appreciate you reviewing the build logs and letting us know if you spot any issues or concerns with this new build system.
One important thing to note is Koji is not going away; Konflux is about how we produce builds upstream of Koji, and the aim is for the resulting packages to behave the same as they do today.
Thanks, Sean
Hi Sean,
Thanks for the transparency and so letting people know changes that will be landing at one point or another (Do you have ETA about when you'd like to switch the rpm build tasks to Konflux instead of kojid ?)
What I do see directly is that in Koji we always tried (following recommendation from koji people) to have the same core/memory ratio, across all architectures but it seems in Konflux (at least in that test build) that it's not the case.
x86_64 and aarch64 are very low spec (2 core, 8G of RAM), which isn't enough to build some core packages, versus ppc64le (32 cores, 123G of RAM) and s390x (16 cores, 62G ram).
I hope you have a way to tune the konflux pipelines to request some specific builder specs (like a "channel" equivalent in Koji workflow) so that some heavy builds wouldn't be failing due to OOM issues, etc ?
Hey Fabian,
We don't have a firm switchover date yet. We want to gather feedback from the community first and make any changes required.
We plan to complete a stress test / mass rebuild to test both RoG (RHEL on Gitlab) and Konflux for Stream 11 on the 19th of May to avoid affecting existing workflows and the current userbase. However this is dependent on getting the code changes in place in time, which we're hopeful but not certain about.
On the builder specs, thats a good catch I wasn't aware of. It looks like Konflux does support tuning the builder specs https://konflux-ci.dev/docs/building/overriding-compute-resources/ I'll look into this more and confirm with the Konflux team.
Sean Ryan Software Engineer seryan@redhat.com
On Fri, 8 May 2026 at 10:16, Fabian Arrotin arrfab@centos.org wrote:
On 5/8/26 10:06, Sean Ryan via devel wrote:
Hi,
As many of you are aware, we are migrating CentOS Stream RPM workflows
to Konflux and we'd like some early community feedback.
For those unfamiliar, Konflux is an open source, cloud-native build
system for producing trusted software artefacts - https://konflux-ci.dev/. In our case, it produces the RPM builds that then land in Koji. You can learn more about it and some of the other changes in the past year from the following talk:
CentOS Connect talk: https://www.youtube.com/watch?v=1yhTvL_rk-I - This talk by Troy and me
covers what changed in the CentOS Stream pipeline over the past year, including additional Konflux-related updates and direction.
Updated CentOS Stream docs: https://docs.centos.org/centos-stream-docs/build/
Relevant repos related to Konflux and our pipelines: https://gitlab.com/redhat/centos-stream/rpms/osci-internal-test-package
- test RPM package used to validate CentOS Stream/Konflux build flow safely.
https://gitlab.com/redhat/centos-stream/konflux-tests/rpmbuild-pipeline
- CentOS Stream Konflux RPM pipeline test repo used for pipeline
integration and iteration.
We've completed the following test build using Konflux:
https://kojihub.stream.centos.org/koji/buildinfo?buildID=110177.
We'd appreciate you reviewing the build logs and letting us know if you
spot any issues or concerns with this new build system.
One important thing to note is Koji is not going away; Konflux is about
how we produce builds upstream of Koji, and the aim is for the resulting packages to behave the same as they do today.
Thanks, Sean
Hi Sean,
Thanks for the transparency and so letting people know changes that will be landing at one point or another (Do you have ETA about when you'd like to switch the rpm build tasks to Konflux instead of kojid ?)
What I do see directly is that in Koji we always tried (following recommendation from koji people) to have the same core/memory ratio, across all architectures but it seems in Konflux (at least in that test build) that it's not the case.
x86_64 and aarch64 are very low spec (2 core, 8G of RAM), which isn't enough to build some core packages, versus ppc64le (32 cores, 123G of RAM) and s390x (16 cores, 62G ram).
I hope you have a way to tune the konflux pipelines to request some specific builder specs (like a "channel" equivalent in Koji workflow) so that some heavy builds wouldn't be failing due to OOM issues, etc ?
-- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org] _______________________________________________ devel mailing list -- devel@lists.centos.org To unsubscribe send an email to devel-leave@lists.centos.org
* Sean Ryan via devel:
We've completed the following test build using Konflux: https://kojihub.stream.centos.org/koji/buildinfo?buildID=110177. We'd appreciate you reviewing the build logs and letting us know if you spot any issues or concerns with this new build system.
Could you share the logs of a failed build, too?
By the way, I get 403 Forbidden for RPM info URLs:
https://kojihub.stream.centos.org/koji/rpminfo?rpmID=2985244
Thanks, Florian
On 5/8/26 12:12, Florian Weimer via devel wrote:
- Sean Ryan via devel:
We've completed the following test build using Konflux: https://kojihub.stream.centos.org/koji/buildinfo?buildID=110177. We'd appreciate you reviewing the build logs and letting us know if you spot any issues or concerns with this new build system.
Could you share the logs of a failed build, too?
By the way, I get 403 Forbidden for RPM info URLs:
https://kojihub.stream.centos.org/koji/rpminfo?rpmID=2985244
Thanks, Florian
Hi Florian,
Due to massive load from AI scrapers putting some time ago the Stream koji on its knees, some koji endpoints are now denied from external view through proxy, but you can view that from the authoritative (internal only) koji web url
Hey Florian,
i don't actually have a "good" example of a failed build. If we are able to do the mass rebuild of the packages using Konflux I'm sure I will and can forward it here as well.
With this change to the RPM builds it doesn't upload the failed builds to Konflux but it will post them and the logs in the merge request of the changes. See for example https://gitlab.com/redhat/centos-stream/rpms/osci-internal-test-package/-/me...
Sean Ryan Software Engineer seryan@redhat.com
On Fri, 8 May 2026 at 11:22, Fabian Arrotin arrfab@centos.org wrote:
On 5/8/26 12:12, Florian Weimer via devel wrote:
- Sean Ryan via devel:
We've completed the following test build using Konflux: https://kojihub.stream.centos.org/koji/buildinfo?buildID=110177. We'd appreciate you reviewing the build logs and letting us know if you spot any issues or concerns with this new build system.
Could you share the logs of a failed build, too?
By the way, I get 403 Forbidden for RPM info URLs:
https://kojihub.stream.centos.org/koji/rpminfo?rpmID=2985244
Thanks, Florian
Hi Florian,
Due to massive load from AI scrapers putting some time ago the Stream koji on its knees, some koji endpoints are now denied from external view through proxy, but you can view that from the authoritative (internal only) koji web url
-- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org] _______________________________________________ devel mailing list -- devel@lists.centos.org To unsubscribe send an email to devel-leave@lists.centos.org
* Sean Ryan via devel:
i don't actually have a "good" example of a failed build. If we are able to do the mass rebuild of the packages using Konflux I'm sure I will and can forward it here as well.
With this change to the RPM builds it doesn't upload the failed builds to Konflux but it will post them and the logs in the merge request of the changes. See for example https://gitlab.com/redhat/centos-stream/rpms/osci-internal-test-package/-/me...
I don't think these pipeline views are publicly accessible. That's why I asked about failed builds—sharing logs for them may need a different mechanism.
Thanks, Florian
Oh you're correct. I'll look into this and see what can be done to make it more accessible.
Cheers, Sean Ryan Software Engineer seryan@redhat.com
On Fri, 8 May 2026 at 18:01, Florian Weimer fweimer@redhat.com wrote:
- Sean Ryan via devel:
i don't actually have a "good" example of a failed build. If we are able to do the mass rebuild of the packages using Konflux I'm sure I will and can forward it here as well.
With this change to the RPM builds it doesn't upload the failed builds to Konflux but it will post them and the logs in the merge request of the changes. See for example
https://gitlab.com/redhat/centos-stream/rpms/osci-internal-test-package/-/me...
I don't think these pipeline views are publicly accessible. That's why I asked about failed builds—sharing logs for them may need a different mechanism.
Thanks, Florian
-
Fabian Arrotin -
Florian Weimer -
Sean Ryan