CI-users April 2025

ci-users@lists.centos.org

2 participants
2 discussions

CI Infra outage (openshift) : Monday April 28th
by Fabian Arrotin 24 Apr '25

24 Apr '25

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 06:00 to 08:00 UTC on 2025-04-28 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. Detailed version : (see https://pagure.io/centos-infra/issue/1609) We're still running 4.15.x openshift, and we're due to some upgrades in a row to jump to 4.18.x These updates will require some time and so will restart all your pods multiple times. The longest operation is the conversion to OVN-kubernetest config type (instead of deprecated/removed now OpenShiftSDN one) We tested all the intermediate steps on our staging cluster and all pods were coming back, so we're confident that it should still work for you after the upgrade, but at least you're aware of what's coming and why some of your CI jobs would be stopped/restarted by jenkins (if you use jenkins that is) -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CI infra outage (duffy.ci.centos.org) : Wednesday April 23rd
by Fabian Arrotin 24 Apr '25

24 Apr '25

We recently discovered that number of duffy ec2 instances (aws side) is much higher than what Duffy thinks is provisioned so at some point, it lost track of really deployed ec2 and we need to reconcile DB and reality) We just need to : stop duffy delete all duffy ec2 instances clean-up duffy DB restart duffy (and it will reprovision from scratch/zero) Once restarted, you'll be able to resume your ci jobs and requesting duffy nodes Maintenance is scheduled for """"Wednesday April 23rd, 11:30 am UTC time"""". You can convert to local time with $(date -d '2025-04-23 11:30 UTC') So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

2 8

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users April 2025