Due to a (announced now) DC move, we'll have to migrate the existing CentOS CBS Koji build infrastructure (https://cbs.centos.org) to new DC and so new hosts
Migration is scheduled for """"Tuesday November 18th, 8:00 am UTC time"""". You can convert to local time with $(date -d '2025-11-18 08:00 UTC')
We don't have a full estimate about the needed time but we'll proceed like this :
- disable all builders on actual CBS infra - wait for some remaining build tasks to finish (ideally you'd stop in advance for that day any automated workflow to not submit builds) - shutdown cbs kojihub in rdu2 - backup/import latest postgresql into new cbs kojihub in rdu3 - synchronize last time kojishare volume from rdu2 => rdu3 (last time it took ~70min) - verify service can start (still all kojid builders disabled) - reconfigure hub to add new aws/rdu3 build hosts - test a simple createrepo (koji regen-repo) task to ensure kojid machines can reach hub and nfs - test a new scratch build from git to test git + lookaside workflow - validate new signing host get notifications on message bus - opening up hub for new requests - builds should proceed as usual
Worth knowing that we'll switch the base profile for the all the builders to match a 4:1 ratio (so 4GB RAM / core) and so on-prem (rdu3) and remote aws builders will all (as starting point) be 16 cores/64GB ram. The s390x builder will remain the same (as itself already remote and so not migrating, just attaching remotely to new hub)
The biggest change will be for ppc64le architecture, as we'll be switching from IBM Power 9 to Power 10, but these new builders will also us the same 16cores/64GB ram profile
Also worth knowing that it's the first (but biggest !) part of the migration. We expect all to work for the SIGs workflow, but some actual machines in RDU2 will then be physically migrated (second week of December, based on actual planning) and then be reinstalled/reconfigured by ansible and so also added to CBS pool and/or other CI needs (like adding back ppc64le as option for duffy.ci.centos.org consumers)
The other (tested) infra change we'll use is a caching server that will speed up access to external repositories (like dl.fedoraproject.org for epel and kojihub.stream.centos.org for centos-buildroots for SIGs that opted in for that option, instead of just building against available and release content set on mirror.stream.centos.org)
I'm working on preparing all that is needed but all the infra is deployed, and just need to be configured with Ansible and then we'll just orchestrate the main switch. We'll keep you informed about any change or status update on that day.
Thanks for your understanding and patience.
On 07/11/2025 16:35, Fabian Arrotin wrote:
Due to a (announced now) DC move, we'll have to migrate the existing CentOS CBS Koji build infrastructure (https://cbs.centos.org) to new DC and so new hosts
Migration is scheduled for """"Tuesday November 18th, 8:00 am UTC time"""". You can convert to local time with $(date -d '2025-11-18 08:00 UTC')
We don't have a full estimate about the needed time but we'll proceed like this :
- disable all builders on actual CBS infra
- wait for some remaining build tasks to finish (ideally you'd stop in
advance for that day any automated workflow to not submit builds)
- shutdown cbs kojihub in rdu2
- backup/import latest postgresql into new cbs kojihub in rdu3
- synchronize last time kojishare volume from rdu2 => rdu3 (last time it
took ~70min)
- verify service can start (still all kojid builders disabled)
- reconfigure hub to add new aws/rdu3 build hosts
- test a simple createrepo (koji regen-repo) task to ensure kojid
machines can reach hub and nfs
- test a new scratch build from git to test git + lookaside workflow
- validate new signing host get notifications on message bus
- opening up hub for new requests
- builds should proceed as usual
Worth knowing that we'll switch the base profile for the all the builders to match a 4:1 ratio (so 4GB RAM / core) and so on-prem (rdu3) and remote aws builders will all (as starting point) be 16 cores/64GB ram. The s390x builder will remain the same (as itself already remote and so not migrating, just attaching remotely to new hub)
The biggest change will be for ppc64le architecture, as we'll be switching from IBM Power 9 to Power 10, but these new builders will also us the same 16cores/64GB ram profile
Also worth knowing that it's the first (but biggest !) part of the migration. We expect all to work for the SIGs workflow, but some actual machines in RDU2 will then be physically migrated (second week of December, based on actual planning) and then be reinstalled/reconfigured by ansible and so also added to CBS pool and/or other CI needs (like adding back ppc64le as option for duffy.ci.centos.org consumers)
The other (tested) infra change we'll use is a caching server that will speed up access to external repositories (like dl.fedoraproject.org for epel and kojihub.stream.centos.org for centos-buildroots for SIGs that opted in for that option, instead of just building against available and release content set on mirror.stream.centos.org)
I'm working on preparing all that is needed but all the infra is deployed, and just need to be configured with Ansible and then we'll just orchestrate the main switch. We'll keep you informed about any change or status update on that day.
Thanks for your understanding and patience.
Status update : everything is ready and waiting at this point, so just to confirm that we'll switch the whole CBS build environment to new RDU3 DC next week.
Have a nice weekend !
On 14/11/2025 17:43, Fabian Arrotin wrote:
On 07/11/2025 16:35, Fabian Arrotin wrote:
Due to a (announced now) DC move, we'll have to migrate the existing CentOS CBS Koji build infrastructure (https://cbs.centos.org) to new DC and so new hosts
Migration is scheduled for """"Tuesday November 18th, 8:00 am UTC time"""". You can convert to local time with $(date -d '2025-11-18 08:00 UTC')
We don't have a full estimate about the needed time but we'll proceed like this :
- disable all builders on actual CBS infra
- wait for some remaining build tasks to finish (ideally you'd stop in
advance for that day any automated workflow to not submit builds)
- shutdown cbs kojihub in rdu2
- backup/import latest postgresql into new cbs kojihub in rdu3
- synchronize last time kojishare volume from rdu2 => rdu3 (last time
it took ~70min)
- verify service can start (still all kojid builders disabled)
- reconfigure hub to add new aws/rdu3 build hosts
- test a simple createrepo (koji regen-repo) task to ensure kojid
machines can reach hub and nfs
- test a new scratch build from git to test git + lookaside workflow
- validate new signing host get notifications on message bus
- opening up hub for new requests
- builds should proceed as usual
Worth knowing that we'll switch the base profile for the all the builders to match a 4:1 ratio (so 4GB RAM / core) and so on-prem (rdu3) and remote aws builders will all (as starting point) be 16 cores/64GB ram. The s390x builder will remain the same (as itself already remote and so not migrating, just attaching remotely to new hub)
The biggest change will be for ppc64le architecture, as we'll be switching from IBM Power 9 to Power 10, but these new builders will also us the same 16cores/64GB ram profile
Also worth knowing that it's the first (but biggest !) part of the migration. We expect all to work for the SIGs workflow, but some actual machines in RDU2 will then be physically migrated (second week of December, based on actual planning) and then be reinstalled/ reconfigured by ansible and so also added to CBS pool and/or other CI needs (like adding back ppc64le as option for duffy.ci.centos.org consumers)
The other (tested) infra change we'll use is a caching server that will speed up access to external repositories (like dl.fedoraproject.org for epel and kojihub.stream.centos.org for centos-buildroots for SIGs that opted in for that option, instead of just building against available and release content set on mirror.stream.centos.org)
I'm working on preparing all that is needed but all the infra is deployed, and just need to be configured with Ansible and then we'll just orchestrate the main switch. We'll keep you informed about any change or status update on that day.
Thanks for your understanding and patience.
Status update : everything is ready and waiting at this point, so just to confirm that we'll switch the whole CBS build environment to new RDU3 DC next week.
Have a nice weekend !
Just to let you know that the whole migration is now done. I tested some builds for all covered architectures and all new builders are attached and picking up tasks. Same for the new signing infra , also migrated to new DC
Should you encounter any issue, feel free to report on Infra tracker (as usual : https://gitlab.com/CentOS/infra/tracker)
-
Fabian Arrotin