CI-users

ci-users@lists.centos.org

416 discussions

CI infra outage : Thursday November 7th
by Fabian Arrotin 07 Nov '24

07 Nov '24

Hi, As announced on centos-devel list (and reflected in https://pagure.io/centos-infra/issue/1528 and https://pagure.io/centos-infra/issue/1533), I need to move some VMs from old hypervisors (to be decommissioned) to different setup. That means that the following internet facing services (for CI infra) will be temporary offline : - duffy.ci.centos.org - site-to-site vpn tunnel[s] between AWS VPC and internal CI vlan Maintenance is scheduled for """"Thursday November 7th, 7:30 am UTC time"""". You can convert to local time with $(date -d '2024-11-07 07:30 UTC') We already moved some VMs already and so we're confident that it shouldn't take a long time to move/restart VMs on different hardware/hypervisors. It usually takes less than 10 min for VM to be back on correct vlan/bridge. So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

CentOS Stream 10 availability for CI tenants (aka duffy.ci.centos.org)
by Fabian Arrotin 21 Oct '24

21 Oct '24

As discussed back and forth with CI tenants using duffy.ci.centos.org (see https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine…) , there was a request to start testing against/for Stream 10 (as it was available already for SIGs building against/for it on https://cbs.centos.org) I had time to work on this ticket (https://pagure.io/centos-infra/issue/1519) so I'm happy to share that you can now request through same duffy api calls CentOS Stream 10 machines that you'll be able to use for your CI tests. Please just keep in mind that Stream 10 packages set is still subject to major changes, but at least this gives you an opportunity to test things in advance. Happy testing ! -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Planned outage , please read (Oct 8th)
by Fabian Arrotin 08 Oct '24

08 Oct '24

Due to a mandatory core switches upgrade in the DC where some services are hosted, we'll lose connectivity between some services/storage servers and some publicly available services will be impacted. Impacted services: - https://git.centos.org - https://cbs.centos.org (builders will be put on hold during maintenance window) - https://lists.centos.org - CentOS mailing lists - CentOS CI env (https://duffy.ci.centos.org api endpoint) Maintenance is scheduled for """"Tuesday September 22nd, 6:00 am UTC time"""". You can convert to local time with $(date -d '2020-09-22 06:00 UTC') The expected/communicated "downtime" was estimated to be 4h, but we'll just announce when all services will be available again. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 3

Reminder : CentOS 7 and 8s EOL soon (impact on CI workflow)
by Fabian Arrotin 27 Jun '24

27 Jun '24

Hi CI tenants ! Just a reminder that (as you know it) CentOS Stream 8 will go EOL in May and then CentOS Linux 7 itself going EOL in June (so very soon) That would mean that we'll just remove the various pools that provide CentOS 8s and 7 for your tests. Reason is that mirror network itself will be unavailable and so all tests would fail anyway. Happy to discuss here but we'll send reminders about when we'll just remove the various pools in Duffy. I also need to find an option for something else that you *still* all use when using Jenkins on openshift cluster : our cico-workspace image (https://quay.io/repository/centosci/cico-workspace) That one is "problematic" as it's based on upstream openshift jenkins agent (https://github.com/openshift/jenkins/tree/master?tab=readme-ov-file#hosted-…) These ones are based on UBI8 and it seems that there is (still) no momentum to move to UBI9 (don't have direct contact with them though) While our current images were based from the upstream ones, we could just "convert" these to 8s (https://github.com/centosci/images/blob/master/cico-workspace/Dockerfile#L1 and https://github.com/centosci/images/blob/master/cico-workspace/Dockerfile#L8) We need to find a solution but was wondering if just using vault (archived mirror content) would be a good solution or not : tempted myself to answer "no" as that would mean running container that will *never* be updated for security reasons (CentOS Stream 8 going EOL in May) Opinions, thoughts ? -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

3 6

[action required] Updating Jenkins pods for CI tenants (openshift)
by Fabian Arrotin 19 Jun '24

19 Jun '24

Recently , after the openshift cluster upgrade to ocp 4.15, some tenants asked about lack of update for Jenkins (stuck on version 2.401.1 , EOL LTS version) Long story short (people can read https://pagure.io/centos-infra/issue/1409) : - Openshift (during releases) modified the default jenkins imagestreamtag (https://access.redhat.com/documentation/en-us/openshift_container_platform/…) - deployed jenkins pods were using DeploymentConfig using jenkins:2 imagestreamtag (not up2date) What is required from you , CI tenants : you can just edit (for now) your deploymentconfig like this, and that will automatically trigger a pod redeploy, using latest jenkins lts version from registry ( v2.440.3 when writing this) : ``` oc edit dc/jenkins replace "name: jenkins:2" with "name: jenkins:ocp-upgrade-redeploy" and exit ``` Watch for your logs and then update your plugins Worth knowing that we'll enforce this rule on June 17th, by making (ocp cluster wise in openshift namespace) jenkins:2 an alias to jenkins:ocp-upgrade-redeploy, so redeploying *all* jenkins pods that aren't migrated yet Summary : So if you prefer to do that yourself and watch process, and pause your ci jobs, etc , do it now If you don't do it, it will be done automatically (but in unattended way) on June 17th -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

3 6

CentOS Stream 8 nodes removed from Duffy/CI pools
by Fabian Arrotin 03 Jun '24

03 Jun '24

As announced some time ago, CentOS Stream 8 reached EOL (End Of Life) and so will be archived today. For that reason, CentOS Stream 8 duffy nodes (see https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine…) were removed from configuration and pool cleared out Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2024-05-07)
by Fabian Arrotin 07 May '24

07 May '24

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2024-05-07 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. PS : this is a follow-up to last week update, and this time the upgrade will bring the cluster from 4.14.x to 4.15.x (latest stable released version) Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2024-05-02)
by Fabian Arrotin 02 May '24

02 May '24

Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 07:00 to 08:00 UTC on 2024-05-02 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation (pods restarting on other workers) during this maintenance period. We apologize for any inconvenience that this may cause. Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 1

[Infra] - Planned outage : lists.centos.org (migration to mailman3) : please read
by Fabian Arrotin 02 Apr '24

02 Apr '24

Due to a needed upgrade , we'll have to move the existing CentOS mailman instance (aka https://lists.centos.org) to a new server/host. Migration is scheduled for """"Tuesday April 8th, 7:00 am UTC time"""". You can convert to local time with $(date -d '2024-04-08 07:00 UTC') The expected "downtime" is estimated to ~60 minutes , time needed to : - take last mailman2 backup - reimport / convert mailman2 archives to mailman3 DB - DNS propagation for A/AAAA/MX records Here are also some important information about the mailman2 => mailman3 migration : # Renamed lists Worth knowing that, based on open discussion on the centos-devel list (see whole thread at https://lists.centos.org/pipermail/centos-devel/2024-March/165576.html), existing lists will be *renamed* , so while we'll put aliases for incoming mails, each list member will start receiving list mails from new list name. So start updating your filters if you filter on email address instead of "subject:" Here is the overview of the new lists names : arm-dev at centos.org => arm-dev at lists.centos.org centos at centos.org => discuss at lists.centos.org centos-devel at centos.org => devel at lists.centos.org centos-announce at centos.org => announce at lists.centos.org centos-automotive-sig at centos.org => automotive-sig at lists.centos.org centos-{cz,de,es,fr,nl,pt-br,zh}(a)centos.org => discuss-{cz,de,es,fr,nl,pt-br,zh}(a)lists.centos.org ci-users at centos.org => ci-users at lists.centos.org centos-gsoc: => gsoc at lists.centos.org centos-mirror at centos.org => mirror at lists.centos.org centos-mirror-announce at centos.org => mirror-announce at lists.centos.org centos-newsletter at centos.org => newsletter at lists.centos.org centos-promo at centos.org => promo at lists.centos.org centos-virt at centos.org => virt at lists.centos.org # Authentication Mailman2 had no real concept of authentication so you could just subscribe to one or more lists, and have a password associated with your email address for that/these subscription(s). Mailman3 itself is split into "core" and "webui" components, so when we'll import mailman2 lists/config into mailman3, your existing subscriptions will continue to work *but* not your password. Mailman3 will be configured to support SSO, and so if you already have a FAS/ACO account (https://accounts.centos.org) you'll be able to login directly into new webui and manage your settings/subscriptions *if* your ACO email address of course matches the one you initially subscribed with for lists.centos.org. If that's not the case, either create an ACO/FAS account that will match and you'll be then able to "link" your mailman3 account with FAS and so manage your settings/subscriptions. If you don't want to, there is always the documented process : https://docs.mailman3.org/en/latest/userguide.html#making-a-mailman-account Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Scheduled Upgrade Notification: CentOS CI Production Openshift Cluster (2023-12-11)
by Fabian Arrotin 11 Dec '23

11 Dec '23

Dear CentOS CI User, Please be advised that the CentOS CI Production Openshift cluster is due for scheduled maintenance from 08:00 to 10:00 UTC on 2023-12-11 . During this period necessary updates will be applied to the cluster. The upgrade is not expected to be service affecting, however there might be periods of slight performance degradation during this maintenance period. We apologize for any inconvenience that this may cause. Kind regards, CentOS CI Infrastructure. -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users