Sending this ticket for awareness : In the last days, I had time to disable some koji builders (https://cbs.centos.org/koji/hosts) , reinstall/migrate these from el8 to el9, and enable these builders back in the available pool.
It finished today [1] and so all builders, all architectures are now running el9.4 . Only the kojihub infra is still running on el8 but that will be migrated later (kojihub itself doesn't have any impact on build infrastructure).
The main reason why we updated was to ensure we'd be running a newer kernel for the systemd-nspawn isolation, that was causing troubles when running on top of el8 (kernel 4.18) , especially when relying on likcapi calls from within the chroot'ed build env. For more details and workaround (not needed anymore) that we used to allow building CentOS Stream 10 pkgs for SIGs, see [2]
At the same time, what triggered also the reinstall was also the need to re-balance workload across different hardware/chassis [3] : the x86_64 kojid builders are in fact running on some un-warrantied (and older) machines and I wanted to reconfigure similar machines dedicated previously for some other things (including CI / on-premises openshift cluster / etc) to ensure that we'd still have some x86_64 builders running behind https://cbs.centos.org, should we encounter a whole chassis hardware failure .
Same for some small services used by koji builders themselves (like small cache/proxy to retrieve external content, like gitlab.com - now spread across two distinct hardware with ucarp and floating ip address[es] for ha purposes)
You'll see that we have now more x86_64 builders, and that some tasks (like createrepo, rebuildSRPM, etc) are exclusively targeted on these hosts, freeing some cpu cycles from aarch64/ppc64le/s390x builders, that will be only used for specific rpmbuild tasks.
Also worth knowing that I was able (due to decommissioned cores/memory that was used for centos stream 8/centos linux 7) to extend/tune number of cores/available memory on reinstalled aarch64 and ppc64le koji builders, hopefully helping to build faster your SIGs packages for these architectures.
Nothing is required from you, SIGs builders, as the whole migration happened transparently , without any downtime, so just enjoy more compute capacity / redundancy for https://cbs.centos.org (and let's hope that we'll not suffer from hardware failure in the future)
[1] https://pagure.io/centos-infra/issue/1505 [2] https://pagure.io/centos-infra/issue/1474 [3] https://pagure.io/centos-infra/issue/1528