CI-users

ci-users@lists.centos.org

1 participants
418 discussions

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2025-01-20)
by Fabian Arrotin 16 Jan '25

16 Jan '25

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2025-01-20. During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. PS : We have already tested on staging cluster and it applied successfully. We'd like then to start evaluating migrating to OCP 4.16 branch Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

upgrading aarch64 ec2 instance type request
by Fabian Arrotin 14 Jan '25

14 Jan '25

Hi all CI tenants, We got today a request to provide more powerful aarch64 ec2 instances for duffy (https://pagure.io/centos-infra/issue/1576) While I don't see a problem doing that, there are two ways to implement this : - either just change the ec2 instance type from c6g.2xlarge to c6g.4xlarge or c6g.8xlarge and then let the pool being drained and reprovisioned automatically with newer instance type (no change in current workflow) - or have a new duffy pool with limited number of provisioned instances and using these more powerful ec2 instances. (so tenants willing to use these ones would need to query a different pool name) From a usage perspective, and fact that machines are pre-provisioned, I'd like to implement option 2, so that nothing would impact the actual pool and then propose another pool name for these other aarch64 machines (both for c9s and c10s). While AWS is sponsoring that infra, I'd like to keep "costs" at reasonable level, and so not deploying for all machines that would have an impact on "virtual budget" while tenants wouldn't even use the added vcpus/memory. Ideas, opinions, thoughts ? -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 2

CI infra outage : Thursday November 7th
by Fabian Arrotin 07 Nov '24

07 Nov '24

Hi, As announced on centos-devel list (and reflected in https://pagure.io/centos-infra/issue/1528 and https://pagure.io/centos-infra/issue/1533), I need to move some VMs from old hypervisors (to be decommissioned) to different setup. That means that the following internet facing services (for CI infra) will be temporary offline : - duffy.ci.centos.org - site-to-site vpn tunnel[s] between AWS VPC and internal CI vlan Maintenance is scheduled for """"Thursday November 7th, 7:30 am UTC time"""". You can convert to local time with $(date -d '2024-11-07 07:30 UTC') We already moved some VMs already and so we're confident that it shouldn't take a long time to move/restart VMs on different hardware/hypervisors. It usually takes less than 10 min for VM to be back on correct vlan/bridge. So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CentOS Stream 10 availability for CI tenants (aka duffy.ci.centos.org)
by Fabian Arrotin 21 Oct '24

21 Oct '24

As discussed back and forth with CI tenants using duffy.ci.centos.org (see https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine…) , there was a request to start testing against/for Stream 10 (as it was available already for SIGs building against/for it on https://cbs.centos.org) I had time to work on this ticket (https://pagure.io/centos-infra/issue/1519) so I'm happy to share that you can now request through same duffy api calls CentOS Stream 10 machines that you'll be able to use for your CI tests. Please just keep in mind that Stream 10 packages set is still subject to major changes, but at least this gives you an opportunity to test things in advance. Happy testing ! -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Planned outage , please read (Oct 8th)
by Fabian Arrotin 08 Oct '24

08 Oct '24

Due to a mandatory core switches upgrade in the DC where some services are hosted, we'll lose connectivity between some services/storage servers and some publicly available services will be impacted. Impacted services: - https://git.centos.org - https://cbs.centos.org (builders will be put on hold during maintenance window) - https://lists.centos.org - CentOS mailing lists - CentOS CI env (https://duffy.ci.centos.org api endpoint) Maintenance is scheduled for """"Tuesday September 22nd, 6:00 am UTC time"""". You can convert to local time with $(date -d '2020-09-22 06:00 UTC') The expected/communicated "downtime" was estimated to be 4h, but we'll just announce when all services will be available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 3

Reminder : CentOS 7 and 8s EOL soon (impact on CI workflow)
by Fabian Arrotin 27 Jun '24

27 Jun '24

Hi CI tenants ! Just a reminder that (as you know it) CentOS Stream 8 will go EOL in May and then CentOS Linux 7 itself going EOL in June (so very soon) That would mean that we'll just remove the various pools that provide CentOS 8s and 7 for your tests. Reason is that mirror network itself will be unavailable and so all tests would fail anyway. Happy to discuss here but we'll send reminders about when we'll just remove the various pools in Duffy. I also need to find an option for something else that you *still* all use when using Jenkins on openshift cluster : our cico-workspace image (https://quay.io/repository/centosci/cico-workspace) That one is "problematic" as it's based on upstream openshift jenkins agent (https://github.com/openshift/jenkins/tree/master?tab=readme-ov-file#hosted-…) These ones are based on UBI8 and it seems that there is (still) no momentum to move to UBI9 (don't have direct contact with them though) While our current images were based from the upstream ones, we could just "convert" these to 8s (https://github.com/centosci/images/blob/master/cico-workspace/Dockerfile#L1 and https://github.com/centosci/images/blob/master/cico-workspace/Dockerfile#L8) We need to find a solution but was wondering if just using vault (archived mirror content) would be a good solution or not : tempted myself to answer "no" as that would mean running container that will *never* be updated for security reasons (CentOS Stream 8 going EOL in May) Opinions, thoughts ? -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

3 6

[action required] Updating Jenkins pods for CI tenants (openshift)
by Fabian Arrotin 19 Jun '24

19 Jun '24

Recently , after the openshift cluster upgrade to ocp 4.15, some tenants asked about lack of update for Jenkins (stuck on version 2.401.1 , EOL LTS version) Long story short (people can read https://pagure.io/centos-infra/issue/1409) : - Openshift (during releases) modified the default jenkins imagestreamtag (https://access.redhat.com/documentation/en-us/openshift_container_platform/…) - deployed jenkins pods were using DeploymentConfig using jenkins:2 imagestreamtag (not up2date) What is required from you , CI tenants : you can just edit (for now) your deploymentconfig like this, and that will automatically trigger a pod redeploy, using latest jenkins lts version from registry ( v2.440.3 when writing this) : ``` oc edit dc/jenkins replace "name: jenkins:2" with "name: jenkins:ocp-upgrade-redeploy" and exit ``` Watch for your logs and then update your plugins Worth knowing that we'll enforce this rule on June 17th, by making (ocp cluster wise in openshift namespace) jenkins:2 an alias to jenkins:ocp-upgrade-redeploy, so redeploying *all* jenkins pods that aren't migrated yet Summary : So if you prefer to do that yourself and watch process, and pause your ci jobs, etc , do it now If you don't do it, it will be done automatically (but in unattended way) on June 17th -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

3 6

CentOS Stream 8 nodes removed from Duffy/CI pools
by Fabian Arrotin 03 Jun '24

03 Jun '24

As announced some time ago, CentOS Stream 8 reached EOL (End Of Life) and so will be archived today. For that reason, CentOS Stream 8 duffy nodes (see https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine…) were removed from configuration and pool cleared out Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2024-05-07)
by Fabian Arrotin 07 May '24

07 May '24

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2024-05-07 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. PS : this is a follow-up to last week update, and this time the upgrade will bring the cluster from 4.14.x to 4.15.x (latest stable released version) Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2024-05-02)
by Fabian Arrotin 02 May '24

02 May '24

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2024-05-02 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users