CI-users

ci-users@lists.centos.org

1 participants
423 discussions

[unscheduled outage] hardware issue impacting CI services (including https://ci.centos.org)
by Fabian Arrotin 04 Oct '20

04 Oct '20

Yesterday (Saturday) evening we got zabbix notifications that some nodes in CI environment were unreachable. After a quick look, I discovered that it was an embedded network switch in a chassis hosting multiple nodes (including but not limited to jenkins node behind ci.centos.org) that went nuts. I tried a remote "hardware reset" and nodes were back online after ~10min. But this morning (sunday), I see through zabbix that same issue happened again, and in the hour after I already did the "hardware reset", but this time, even that doesn't work anymore. So that means that we have a network switch not working anymore. As that chassis (like almost *all* equipment in CI) *isn't* under warranty, we'll see on monday what can be done and how we give priority to try to dispatch services elsewhere (and that probably means then powering down other services , depending on priority that will be given), but it's easy to understand that we can't even give any ETA at this point. Thanks for your comprehending, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

[infra outage] : nfs storage server hosting PVs for openshift
by Fabian Arrotin 02 Oct '20

02 Oct '20

we had a kernel panic on the storage box used as nfs server for openshift (both okd and ocp) and machine doesn't come back online due to md device refusing to start. machine is now in single-mode to analyze the situation and try to fix it. We'll send more details and progress when possible -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

Infra : scheduled hardware maintenance (Openshift/NFS)
by Fabian Arrotin 01 Oct '20

01 Oct '20

Due to a hardware maintenance that needs to take place on the NFS storage node used by openshift ("legacy" and current one - ocp ), we'll have to shutdown the openshift cluster, and then proceed with hardware maintenance on the NFS server (that itself needs to be powered down, no way to actually do that "online") Migration is scheduled for """"Wednesday September 30th, 12:00 pm UTC time"""". You can convert to local time with $(date -d '2020-09-30 12:00 UTC') The expected "downtime" is estimated to ~60 minutes , time needed to shutdown the machine, install new disks, restart the machine and also do some updates and tuning on the setup. For more informations about this, here are some relevant tickets that were created for the perf issue in openshift and nfs : https://pagure.io/centos-infra/issue/53 https://pagure.io/centos-infra/issue/105 https://pagure.io/centos-infra/issue/85 https://pagure.io/centos-infra/issue/26 <subliminal message> PS : worth noting that while we'll investigate reports on new ocp cluster, we'll probably not spend time investigating in the old/legacy one, that projects are supposed to migrate away from soon, as the legacy openshift setup will disappear soon (see https://pagure.io/centos-infra/issue/16) </subliminal message> Thanks for your comprehending and patience. on behalf of the CI Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

3 4

Infra Pre-Announce : moving CI ssh jump host soon, please read !
by Fabian Arrotin 07 Sep '20

07 Sep '20

Hi, As you noticed recently, we started to refresh the infra used for CentOS CI (not the hardware, still the same, but the software stack and the way to control/manage it). One of the identified nodes still being used and that needs to be converted to the new infra layout is the ssh jumphost (see https://wiki.centos.org/QaWiki/CI/GettingStarted#How_to_use_it) Normally, some of you have switched to OpenShift workload, (including to the new Openshift 4/OCP setup that went live recently) but some Projects are still on the old setup with sometimes a need to reach dedicated/shared VMs acting as Jenkins agent[s], connected to Jenkins behind https://ci.centos.org. We have already provisioned a new VM in the new setup (that can reach the whole CI subnet and VLAN) but here are some points to consider, reason why we wanted to pre-announce long time in advance before we do the real switch) : * New ssh jump host is CentOS 8 based, versus CentOS 6, meaning that if you used ssh-dss key (instead of ssh-rsa), you'll *not* be able to connect through that new host. We already identified such keys and Vipul will try (when it's tied to a real email address for the project) to reach out. But better to announce it here too, so that you have time to ask us to reflect a change (through ticket on https://pagure.io/centos-infra/issues) * Old VM allowed shell access, but it will be disallowed on the new one (there is no need for shell on that intermediate node anyway). Reminder that you can configure your ssh config to directly use ProxyCommand or even now ProxyJump (on recent openssh-client). See https://wiki.centos.org/TipsAndTricks/SshTips/JumpHost) * Because the host has a new sshd_host_key, it will come with a new fingerprint too, so if you have automation and that you don't trust our CA already, the fingerprint for new host will be : [fingerprint] rsa=3072 SHA256:n7y0qZS/FvhjaskOBds3TTKQh5EtgNQ25E7cmTNBATg (RSA) rsa_md5=3072 MD5:9e:83:46:d0:c5:8a:a0:94:50:10:58:9d:af:ca:50:19 (RSA) ecdsa=256 SHA256:ZQacwDsWkKBYL9HJJYwHr94Ny1sMhHMDnz9GiLFb8Uc (ECDSA) ecdsa_md5=256 MD5:dd:24:ea:6a:fd:8b:29:3d:1d:d0:a9:32:8c:b2:ea:62 (ECDSA) As we know that it's August and that some of you are probably on PTO (coming back or leaving soon), after discussion with Vipul , David and myself, we considered that we'll probably go live around beginning of September. Should you have any question around that migration, feel free to reply to this thread (ideally on dedicated ci-users mailing list), or on irc.freenode.net (#centos-ci) On behalf of the CentOS CI infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

2 4

New CentOS Infra/CBS/CI issues/requests tracker, please read
by Fabian Arrotin 05 Aug '20

05 Aug '20

Hi all, As you probably noticed in the last weeks/months, we have a stronger collaboration and synergy with the Fedora infrastructure team. Combining forces and resources help both projects at the same time, as majority of the CentOS contributors are already Fedora contributors and probably the same in reverse. It's not a secret (it was announced through CPE weekly mails on this list) that the CentOS board approved the idea of merging authentication systems in a near future (as an example). This email to let you know that all RFE/issues concerning the following areas should be reported to a new issues tracker : https://pagure.io/centos-infra/issues/ , to adapt the same workflow as the Fedora infra team is already using. ( see https://docs.fedoraproject.org/en-US/cpe/working_with_us/ ) Concerned areas : - https://cbs.centos.org (Community BuildSystem, aka koji) - Special Interest Groups requests (for mirror, resources, etc) - https://ci.centos.org (All CI infra ecosystem) - Everything around CentOS Infra (mirror issues, etc) We have already moved/migrated for example the (opened) tickets that were filed under the "Buildsys" , "CI" and "Infrastructure" categories to the new issues tracker. The idea being to *not* request work to be done through IRC but rather through new infra issues tracker. Imported tickets will be discussed there and worked on (reviewed on a daily basis) after having been prioritized Should you feel a need to discuss this new process, feel free to do so in #centos-devel on irc.freenode.net or on this centos-devel list. Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

Call for migration to new openshift cluster
by Vipul Siddharth 19 Jul '20

19 Jul '20

Hi all, We are done and ready with the new Openshift 4 CI cluster. Now we want to start the second phase of it i.e migrating projects to the new cluster. If you are currently using apps.ci.centos.org (OCP 3.6), please contact us off-list so that we can create a namespace for you in the new OCP 4.4 cluster, so you can start migrating your workloads. It's not a hard deadline but we are hoping to retire the older Openshift 3.6 cluster in the upcoming quarter (3 months) and legacy (ci.centos.org) environment in a couple of months after that. If you are using ci.centos.org, it's a very good time to start working on updating your workload If you have any questions, please reach out to us Thank You and stay safe -- Vipul Siddharth Fedora | CentOS CI Infrastructure Team

3 12

CI Infra maintenance on Monday - 7am UTC for 2 hours
by Vipul Siddharth 13 Jul '20

13 Jul '20

Hi, On early July 13th, 2020 (Monday), CI will be out for ~2 hours for maintenance. During that time, Duffy will not be working which means we won't be handing out nodes in that window. Date: July 13th Estimated outage time: 0700 UTC to 0900 UTC Apologies for any inconveniences -- Vipul Siddharth Fedora | CentOS CI Infrastructure Team

1 1

Can not get node on ci.centos.org
by Katerina Foniok 16 Jun '20

16 Jun '20

Hi guys, we are not able to get nodes on ci.centos.org for quite some time. Our jobs are running on devtools-ci-slave04 <https://ci.centos.org/computer/devtools-ci-slave04> Can somebody please take a look? Thank you :) Katka

2 2

Update on OpenShift cluster upgrade
by Vipul Siddharth 19 May '20

19 May '20

Hi folks, Here is a quick and short update on what CI Infrastructure team has been up to for the last few days. # Done * identified infrastructure suitable for hosting the new ocp4 cluster and curated inventory * configured DNS for the new ocp4 ocp.ci.centos.org cluster * provisioned 2 haproxy load balancers * ansible template for PXE ocp4 nodes # Hiccups we had * Faced issue pulling machine config operator image from quay.io on the ocp4 bootstrap node. We reached out to Jonathan Lebon of Fedora CoreOS related help (Thank you Jonathan for the help). The issue was resolved by recreating the ignition files from scratch. * Faced issue with multiple ethernet interfaces on the RHCOS nodes and DHCP. Resolved by disabling DHCP, and configuring static IPs on a single interface. # In progress * Provisioning OCP4 nodes (Master, Worker, and Bootstrap) # Blockers * RHEL CoreOS does not support RAID on the root device Thank you and Stay Safe / Brian Stinson, David Kirwan, Fabian Arrotin & -- Vipul Siddharth Fedora | CentOS CI Infrastructure Team

2 3

CI state and efforts
by Vipul Siddharth 01 May '20

01 May '20

Greetings everyone This email is to update you all on some of the efforts we are about to put/putting in place within the CentOS CI Infrastructure. Some of the things have been long due and we realize that. For a couple of years, our efforts have mainly been under keeping the fires away and general maintenance (adding new releases, arches, making things in the background a bit sophisticated, etc). Right now we have started working on some things that you would notice and will impact you (in a better way hopefully!) and we will be keeping you updated. * Monitoring - Done * A couple of months ago, All I had was basic ansible scripts to show me data, and that was my way of monitoring things. As one would expect, I often missed things, and email on the list would pop up with `insufficient nodes` or `no space on Jenkins master`. While we tried to be relatively quick on the resolution, we needed a better way. In early February, Fabian (Thank you) set up a monitoring system (added ci.centos.org to CentOS Infra Zabbix instance) which in turn reduced such incidence by a lot. I hope you have not noticed those emails/complaints in the last 2-3 months. We will be adding monitoring to more areas as we go. * OpenShift cluster upgrade - In Progress * https://console.apps.ci.centos.org:8443/ is using openshift version 3.6 which has been outdated for a while. We are upgrading the cluster to OpenShift 4.3 which handles some of the group permissions in a better way. Apart from the shiny new deployment which has a bunch of improvements/features, it will allow us to give admin access to more members easily which in turn will decrease the wait time for new accounts/users (it won't wait on a single person to do all the work). * Deployment of OpenNebula for VM checkouts through cico - Upcoming * We will be deploying OpenNebula and will extend duffy api to support VMs in the check out process. This will be useful for the jobs who don't need bare metals to do their tests and need slightly less resources. This will also enable us to support Fedora as an on-demand offering distro (in vm) that can be used to run your test. Ideally, it would look like `cico node get --type vm` or `cico node get --type vm --distro fedora -- version rawhide`. This needs more grooming and planning before we start planning exact images/distros and ways but I am hopeful that it will make your experience better. We are hoping to update you all with the things going on in the background pretty frequently and the updates will show up as weekly emails `progress of $initiatives_mentioned_above` and "probably" monthly email of some matrices of things that happened in that month (not sure yet). Once we have upgraded the openshift cluster, we will need your help in migrating the jobs from legacy `ci.centos.org` to openshift. You will get a namespace where you can control all things related. This will introduce easier workflow and management for admins and more freedom to you, tenants! If you want to discuss things more, feel free to contact us or catch us at #centos-ci@Freenode Thank you! Be safe -- @siddharthvipul He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat

2 1

← Newer
1
...
7
8
9
10
11
12
13
...
43
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users