CI-users

ci-users@lists.centos.org

2 participants
422 discussions

Pool size for metal-ec2-c5n-centos-9s-x86_64
by Pierre-Yves Chibon 21 Oct '25

21 Oct '25

Good Morning! I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'" } } and { "error": { "detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'" } So I check the pools: $ duffy client list-pools { "action": "get", "pools": [ { "name": "virt-ec2-t2-centos-9s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-9s-aarch64", "fill_level": 5 }, { "name": "metal-ec2-c5n-centos-9s-x86_64", "fill_level": 6 }, { "name": "virt-ec2-t2-centos-10s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-10s-aarch64", "fill_level": 2 }, { "name": "metal-ec2-c5n-centos-10s-x86_64", "fill_level": 3 } ] } Which seems fine from here. Do we have monitoring of how busy the nodes are in the pools? Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful) Thanks in advance :) Pierre

2 5

[Infra] - Planned outage : duffy.ci.centos.org (September 15th)
by Fabian Arrotin 15 Sep '25

15 Sep '25

Due to a upcoming DC move, we'll have to move the existing CI Duffy endpoint (aka https://duffy.ci.centos.org) to a new server in different location. Migration is scheduled for """"Monday September 15th, 7:00 am UTC time"""". You can convert to local time with $(date -d '2025-09-15 07:00 UTC') The expected "downtime" is estimated to ~30 minutes , time needed to : - stop services - update public CNAME record to point to new instance (TTL is set to 10min) - backup/restore last postgresql dump - let duffy reprovision pools from scratch As mentioned, while we'll reimport tenants and sessions we'll just drop all ec2 machines and restart from a freshly provisioned infra for duffy nodes (x86_64 and aarch64) So just keep that in mind about some of your jobs that might be stopped so probably better to pause your jobs and wait for whole service to be back online. You can also reach out to either this mail thread or directly on #centos-ci on irc.libera.chat Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CI Infra outage (openshift) : Monday April 28th
by Fabian Arrotin 28 Apr '25

28 Apr '25

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 06:00 to 08:00 UTC on 2025-04-28 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. Detailed version : (see https://pagure.io/centos-infra/issue/1609) We're still running 4.15.x openshift, and we're due to some upgrades in a row to jump to 4.18.x These updates will require some time and so will restart all your pods multiple times. The longest operation is the conversion to OVN-kubernetest config type (instead of deprecated/removed now OpenShiftSDN one) We tested all the intermediate steps on our staging cluster and all pods were coming back, so we're confident that it should still work for you after the upgrade, but at least you're aware of what's coming and why some of your CI jobs would be stopped/restarted by jenkins (if you use jenkins that is) -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 2

CI infra outage (duffy.ci.centos.org) : Wednesday April 23rd
by Fabian Arrotin 24 Apr '25

24 Apr '25

We recently discovered that number of duffy ec2 instances (aws side) is much higher than what Duffy thinks is provisioned so at some point, it lost track of really deployed ec2 and we need to reconcile DB and reality) We just need to : stop duffy delete all duffy ec2 instances clean-up duffy DB restart duffy (and it will reprovision from scratch/zero) Once restarted, you'll be able to resume your ci jobs and requesting duffy nodes Maintenance is scheduled for """"Wednesday April 23rd, 11:30 am UTC time"""". You can convert to local time with $(date -d '2025-04-23 11:30 UTC') So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

2 8

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2025-01-20)
by Fabian Arrotin 16 Jan '25

16 Jan '25

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2025-01-20. During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. PS : We have already tested on staging cluster and it applied successfully. We'd like then to start evaluating migrating to OCP 4.16 branch Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

upgrading aarch64 ec2 instance type request
by Fabian Arrotin 14 Jan '25

14 Jan '25

Hi all CI tenants, We got today a request to provide more powerful aarch64 ec2 instances for duffy (https://pagure.io/centos-infra/issue/1576) While I don't see a problem doing that, there are two ways to implement this : - either just change the ec2 instance type from c6g.2xlarge to c6g.4xlarge or c6g.8xlarge and then let the pool being drained and reprovisioned automatically with newer instance type (no change in current workflow) - or have a new duffy pool with limited number of provisioned instances and using these more powerful ec2 instances. (so tenants willing to use these ones would need to query a different pool name) From a usage perspective, and fact that machines are pre-provisioned, I'd like to implement option 2, so that nothing would impact the actual pool and then propose another pool name for these other aarch64 machines (both for c9s and c10s). While AWS is sponsoring that infra, I'd like to keep "costs" at reasonable level, and so not deploying for all machines that would have an impact on "virtual budget" while tenants wouldn't even use the added vcpus/memory. Ideas, opinions, thoughts ? -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 2

CI infra outage : Thursday November 7th
by Fabian Arrotin 07 Nov '24

07 Nov '24

Hi, As announced on centos-devel list (and reflected in https://pagure.io/centos-infra/issue/1528 and https://pagure.io/centos-infra/issue/1533) I need to move some VMs from old hypervisors (to be decommissioned) to different setup. That means that the following internet facing services (for CI infra) will be temporary offline : - duffy.ci.centos.org - site-to-site vpn tunnel[s] between AWS VPC and internal CI vlan Maintenance is scheduled for """"Thursday November 7th, 7:30 am UTC time"""". You can convert to local time with $(date -d '2024-11-07 07:30 UTC') We already moved some VMs already and so we're confident that it shouldn't take a long time to move/restart VMs on different hardware/hypervisors. It usually takes less than 10 min for VM to be back on correct vlan/bridge. So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CentOS Stream 10 availability for CI tenants (aka duffy.ci.centos.org)
by Fabian Arrotin 21 Oct '24

21 Oct '24

As discussed back and forth with CI tenants using duffy.ci.centos.org (see https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine…) , there was a request to start testing against/for Stream 10 (as it was available already for SIGs building against/for it on https://cbs.centos.org) I had time to work on this ticket (https://pagure.io/centos-infra/issue/1519) so I'm happy to share that you can now request through same duffy api calls CentOS Stream 10 machines that you'll be able to use for your CI tests. Please just keep in mind that Stream 10 packages set is still subject to major changes, but at least this gives you an opportunity to test things in advance. Happy testing ! -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Planned outage , please read (Oct 8th)
by Fabian Arrotin 08 Oct '24

08 Oct '24

Due to a mandatory core switches upgrade in the DC where some services are hosted, we'll lose connectivity between some services/storage servers and some publicly available services will be impacted. Impacted services: - https://git.centos.org - https://cbs.centos.org (builders will be put on hold during maintenance window) - https://lists.centos.org - CentOS mailing lists - CentOS CI env (https://duffy.ci.centos.org api endpoint) Maintenance is scheduled for """"Tuesday September 22nd, 6:00 am UTC time"""". You can convert to local time with $(date -d '2020-09-22 06:00 UTC') The expected/communicated "downtime" was estimated to be 4h, but we'll just announce when all services will be available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 3

Reminder : CentOS 7 and 8s EOL soon (impact on CI workflow)
by Fabian Arrotin 27 Jun '24

27 Jun '24

Hi CI tenants ! Just a reminder that (as you know it) CentOS Stream 8 will go EOL in May and then CentOS Linux 7 itself going EOL in June (so very soon) That would mean that we'll just remove the various pools that provide CentOS 8s and 7 for your tests. Reason is that mirror network itself will be unavailable and so all tests would fail anyway. Happy to discuss here but we'll send reminders about when we'll just remove the various pools in Duffy. I also need to find an option for something else that you *still* all use when using Jenkins on openshift cluster : our cico-workspace image (https://quay.io/repository/centosci/cico-workspace) That one is "problematic" as it's based on upstream openshift jenkins agent (https://github.com/openshift/jenkins/tree/master?tab=readme-ov-file#hosted-…) These ones are based on UBI8 and it seems that there is (still) no momentum to move to UBI9 (don't have direct contact with them though) While our current images were based from the upstream ones, we could just "convert" these to 8s (https://github.com/centosci/images/blob/master/cico-workspace/Dockerfile#L1 and https://github.com/centosci/images/blob/master/cico-workspace/Dockerfile#L8) We need to find a solution but was wondering if just using vault (archived mirror content) would be a good solution or not : tempted myself to answer "no" as that would mean running container that will *never* be updated for security reasons (CentOS Stream 8 going EOL in May) Opinions, thoughts ? -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

3 6

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users