CI-users

ci-users@lists.centos.org

1 participants
424 discussions

Artifacts.ci.centos.org migrated to newer host
by Fabian Arrotin 20 Apr '26

20 Apr '26

Sending this as a FYI : The instance offering the https://artifacts.ci.centos.org was migrated to a new host earlier today (block device attached to new instance so all data was moved to new host directly) In case you'd have an issue with the migrated instance, feel free to reach out (on https://gitlab.com/CentOS/infra/tracker/-/work_items) Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Duffy ppc64le nodes availability
by Fabian Arrotin 27 Mar '26

27 Mar '26

Hi all, Last year, due to DC move, and so machines needed to be reshuffled/reinstalled/moved/recycled/etc, we had to remove the ppc64le architectures from the duffy pools. While it wasn't a problem for most of you CI tenants (using almost exclusively x86_64 - or eventually aarch64), some, like Kmods SIG, were keen on having the ppc64le support added back in, as they also build kernel and kernel modules for ppc64le. It took some time (due to other higher priorities on the list) but we have now the pool ready to be used ! : ``` for i in 9 10 ; do duffy client show-pool virt-kvm-medium-centos-${i}s-ppc64le ; done { "action": "get", "pool": { "name": "virt-kvm-medium-centos-9s-ppc64le", "fill-level": 2, "levels": { "provisioning": 0, "ready": 3, "contextualizing": 0, "deployed": 0, "deprovisioning": 0 } } } { "action": "get", "pool": { "name": "virt-kvm-medium-centos-10s-ppc64le", "fill-level": 2, "levels": { "provisioning": 0, "ready": 2, "contextualizing": 0, "deployed": 0, "deprovisioning": 0 } } } ``` One thing to keep in mind (but it was already the case when we were offering that from previous on-prem DC) : they don't have public ip addresses at all, but once requested from openshift cico workspace, you can still reach these directly. If you want to consume from other environment, (like gitlab runner or else), just use a small duffy x86_64 or aarch64 ec2 instance, as these ones have the needed VPC info and tunnel to get to the provisioned ppc64le instances Let us know if something doesn't work for you but I just tested and it was working fine PS: the ppc64le have smaller specs : - kvm guest - Architecture: ppc64le , CPU(s): 4 , Model name: POWER9 (architected), altivec - Mem : 16G - block device: Virtio 10GB device -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Pool size for metal-ec2-c5n-centos-9s-x86_64
by Pierre-Yves Chibon 21 Oct '25

21 Oct '25

Good Morning! I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'" } } and { "error": { "detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'" } So I check the pools: $ duffy client list-pools { "action": "get", "pools": [ { "name": "virt-ec2-t2-centos-9s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-9s-aarch64", "fill_level": 5 }, { "name": "metal-ec2-c5n-centos-9s-x86_64", "fill_level": 6 }, { "name": "virt-ec2-t2-centos-10s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-10s-aarch64", "fill_level": 2 }, { "name": "metal-ec2-c5n-centos-10s-x86_64", "fill_level": 3 } ] } Which seems fine from here. Do we have monitoring of how busy the nodes are in the pools? Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful) Thanks in advance :) Pierre

2 5

[Infra] - Planned outage : duffy.ci.centos.org (September 15th)
by Fabian Arrotin 15 Sep '25

15 Sep '25

Due to a upcoming DC move, we'll have to move the existing CI Duffy endpoint (aka https://duffy.ci.centos.org) to a new server in different location. Migration is scheduled for """"Monday September 15th, 7:00 am UTC time"""". You can convert to local time with $(date -d '2025-09-15 07:00 UTC') The expected "downtime" is estimated to ~30 minutes , time needed to : - stop services - update public CNAME record to point to new instance (TTL is set to 10min) - backup/restore last postgresql dump - let duffy reprovision pools from scratch As mentioned, while we'll reimport tenants and sessions we'll just drop all ec2 machines and restart from a freshly provisioned infra for duffy nodes (x86_64 and aarch64) So just keep that in mind about some of your jobs that might be stopped so probably better to pause your jobs and wait for whole service to be back online. You can also reach out to either this mail thread or directly on #centos-ci on irc.libera.chat Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CI Infra outage (openshift) : Monday April 28th
by Fabian Arrotin 28 Apr '25

28 Apr '25

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 06:00 to 08:00 UTC on 2025-04-28 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. Detailed version : (see https://pagure.io/centos-infra/issue/1609) We're still running 4.15.x openshift, and we're due to some upgrades in a row to jump to 4.18.x These updates will require some time and so will restart all your pods multiple times. The longest operation is the conversion to OVN-kubernetest config type (instead of deprecated/removed now OpenShiftSDN one) We tested all the intermediate steps on our staging cluster and all pods were coming back, so we're confident that it should still work for you after the upgrade, but at least you're aware of what's coming and why some of your CI jobs would be stopped/restarted by jenkins (if you use jenkins that is) -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 2

CI infra outage (duffy.ci.centos.org) : Wednesday April 23rd
by Fabian Arrotin 24 Apr '25

24 Apr '25

We recently discovered that number of duffy ec2 instances (aws side) is much higher than what Duffy thinks is provisioned so at some point, it lost track of really deployed ec2 and we need to reconcile DB and reality) We just need to : stop duffy delete all duffy ec2 instances clean-up duffy DB restart duffy (and it will reprovision from scratch/zero) Once restarted, you'll be able to resume your ci jobs and requesting duffy nodes Maintenance is scheduled for """"Wednesday April 23rd, 11:30 am UTC time"""". You can convert to local time with $(date -d '2025-04-23 11:30 UTC') So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

2 8

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2025-01-20)
by Fabian Arrotin 16 Jan '25

16 Jan '25

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2025-01-20. During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. PS : We have already tested on staging cluster and it applied successfully. We'd like then to start evaluating migrating to OCP 4.16 branch Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

upgrading aarch64 ec2 instance type request
by Fabian Arrotin 14 Jan '25

14 Jan '25

Hi all CI tenants, We got today a request to provide more powerful aarch64 ec2 instances for duffy (https://pagure.io/centos-infra/issue/1576) While I don't see a problem doing that, there are two ways to implement this : - either just change the ec2 instance type from c6g.2xlarge to c6g.4xlarge or c6g.8xlarge and then let the pool being drained and reprovisioned automatically with newer instance type (no change in current workflow) - or have a new duffy pool with limited number of provisioned instances and using these more powerful ec2 instances. (so tenants willing to use these ones would need to query a different pool name) From a usage perspective, and fact that machines are pre-provisioned, I'd like to implement option 2, so that nothing would impact the actual pool and then propose another pool name for these other aarch64 machines (both for c9s and c10s). While AWS is sponsoring that infra, I'd like to keep "costs" at reasonable level, and so not deploying for all machines that would have an impact on "virtual budget" while tenants wouldn't even use the added vcpus/memory. Ideas, opinions, thoughts ? -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 2

CI infra outage : Thursday November 7th
by Fabian Arrotin 07 Nov '24

07 Nov '24

Hi, As announced on centos-devel list (and reflected in https://pagure.io/centos-infra/issue/1528 and https://pagure.io/centos-infra/issue/1533) I need to move some VMs from old hypervisors (to be decommissioned) to different setup. That means that the following internet facing services (for CI infra) will be temporary offline : - duffy.ci.centos.org - site-to-site vpn tunnel[s] between AWS VPC and internal CI vlan Maintenance is scheduled for """"Thursday November 7th, 7:30 am UTC time"""". You can convert to local time with $(date -d '2024-11-07 07:30 UTC') We already moved some VMs already and so we're confident that it shouldn't take a long time to move/restart VMs on different hardware/hypervisors. It usually takes less than 10 min for VM to be back on correct vlan/bridge. So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CentOS Stream 10 availability for CI tenants (aka duffy.ci.centos.org)
by Fabian Arrotin 21 Oct '24

21 Oct '24

As discussed back and forth with CI tenants using duffy.ci.centos.org (see https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine…) , there was a request to start testing against/for Stream 10 (as it was available already for SIGs building against/for it on https://cbs.centos.org) I had time to work on this ticket (https://pagure.io/centos-infra/issue/1519) so I'm happy to share that you can now request through same duffy api calls CentOS Stream 10 machines that you'll be able to use for your CI tests. Please just keep in mind that Stream 10 packages set is still subject to major changes, but at least this gives you an opportunity to test things in advance. Happy testing ! -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users