CI-users

ci-users@lists.centos.org

424 discussions

devtools-ci-slave04 is offline
by Katerina Foniok 09 Oct '20

09 Oct '20

Hello guys, our jobs on ci.centos.org are pending because the *devtools-ci-slave04* is offline. Can someone take a look, please? One of the affected jobs is here <https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.…> . Thank you! Have a great day, Katka

3 8

[unscheduled outage] hardware issue impacting CI services (including https://ci.centos.org)
by Fabian Arrotin 04 Oct '20

04 Oct '20

Yesterday (Saturday) evening we got zabbix notifications that some nodes in CI environment were unreachable. After a quick look, I discovered that it was an embedded network switch in a chassis hosting multiple nodes (including but not limited to jenkins node behind ci.centos.org) that went nuts. I tried a remote "hardware reset" and nodes were back online after ~10min. But this morning (sunday), I see through zabbix that same issue happened again, and in the hour after I already did the "hardware reset", but this time, even that doesn't work anymore. So that means that we have a network switch not working anymore. As that chassis (like almost *all* equipment in CI) *isn't* under warranty, we'll see on monday what can be done and how we give priority to try to dispatch services elsewhere (and that probably means then powering down other services , depending on priority that will be given), but it's easy to understand that we can't even give any ETA at this point. Thanks for your comprehending, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

[infra outage] : nfs storage server hosting PVs for openshift
by Fabian Arrotin 02 Oct '20

02 Oct '20

we had a kernel panic on the storage box used as nfs server for openshift (both okd and ocp) and machine doesn't come back online due to md device refusing to start. machine is now in single-mode to analyze the situation and try to fix it. We'll send more details and progress when possible -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

Infra : scheduled hardware maintenance (Openshift/NFS)
by Fabian Arrotin 01 Oct '20

01 Oct '20

Due to a hardware maintenance that needs to take place on the NFS storage node used by openshift ("legacy" and current one - ocp ), we'll have to shutdown the openshift cluster, and then proceed with hardware maintenance on the NFS server (that itself needs to be powered down, no way to actually do that "online") Migration is scheduled for """"Wednesday September 30th, 12:00 pm UTC time"""". You can convert to local time with $(date -d '2020-09-30 12:00 UTC') The expected "downtime" is estimated to ~60 minutes , time needed to shutdown the machine, install new disks, restart the machine and also do some updates and tuning on the setup. For more informations about this, here are some relevant tickets that were created for the perf issue in openshift and nfs : https://pagure.io/centos-infra/issue/53 https://pagure.io/centos-infra/issue/105 https://pagure.io/centos-infra/issue/85 https://pagure.io/centos-infra/issue/26 <subliminal message> PS : worth noting that while we'll investigate reports on new ocp cluster, we'll probably not spend time investigating in the old/legacy one, that projects are supposed to migrate away from soon, as the legacy openshift setup will disappear soon (see https://pagure.io/centos-infra/issue/16) </subliminal message> Thanks for your comprehending and patience. on behalf of the CI Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

3 4

Infra Pre-Announce : moving CI ssh jump host soon, please read !
by Fabian Arrotin 07 Sep '20

07 Sep '20

Hi, As you noticed recently, we started to refresh the infra used for CentOS CI (not the hardware, still the same, but the software stack and the way to control/manage it). One of the identified nodes still being used and that needs to be converted to the new infra layout is the ssh jumphost (see https://wiki.centos.org/QaWiki/CI/GettingStarted#How_to_use_it) Normally, some of you have switched to OpenShift workload, (including to the new Openshift 4/OCP setup that went live recently) but some Projects are still on the old setup with sometimes a need to reach dedicated/shared VMs acting as Jenkins agent[s], connected to Jenkins behind https://ci.centos.org. We have already provisioned a new VM in the new setup (that can reach the whole CI subnet and VLAN) but here are some points to consider, reason why we wanted to pre-announce long time in advance before we do the real switch) : * New ssh jump host is CentOS 8 based, versus CentOS 6, meaning that if you used ssh-dss key (instead of ssh-rsa), you'll *not* be able to connect through that new host. We already identified such keys and Vipul will try (when it's tied to a real email address for the project) to reach out. But better to announce it here too, so that you have time to ask us to reflect a change (through ticket on https://pagure.io/centos-infra/issues) * Old VM allowed shell access, but it will be disallowed on the new one (there is no need for shell on that intermediate node anyway). Reminder that you can configure your ssh config to directly use ProxyCommand or even now ProxyJump (on recent openssh-client). See https://wiki.centos.org/TipsAndTricks/SshTips/JumpHost) * Because the host has a new sshd_host_key, it will come with a new fingerprint too, so if you have automation and that you don't trust our CA already, the fingerprint for new host will be : [fingerprint] rsa=3072 SHA256:n7y0qZS/FvhjaskOBds3TTKQh5EtgNQ25E7cmTNBATg (RSA) rsa_md5=3072 MD5:9e:83:46:d0:c5:8a:a0:94:50:10:58:9d:af:ca:50:19 (RSA) ecdsa=256 SHA256:ZQacwDsWkKBYL9HJJYwHr94Ny1sMhHMDnz9GiLFb8Uc (ECDSA) ecdsa_md5=256 MD5:dd:24:ea:6a:fd:8b:29:3d:1d:d0:a9:32:8c:b2:ea:62 (ECDSA) As we know that it's August and that some of you are probably on PTO (coming back or leaving soon), after discussion with Vipul , David and myself, we considered that we'll probably go live around beginning of September. Should you have any question around that migration, feel free to reply to this thread (ideally on dedicated ci-users mailing list), or on irc.freenode.net (#centos-ci) On behalf of the CentOS CI infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

2 4

New CentOS Infra/CBS/CI issues/requests tracker, please read
by Fabian Arrotin 05 Aug '20

05 Aug '20

Hi all, As you probably noticed in the last weeks/months, we have a stronger collaboration and synergy with the Fedora infrastructure team. Combining forces and resources help both projects at the same time, as majority of the CentOS contributors are already Fedora contributors and probably the same in reverse. It's not a secret (it was announced through CPE weekly mails on this list) that the CentOS board approved the idea of merging authentication systems in a near future (as an example). This email to let you know that all RFE/issues concerning the following areas should be reported to a new issues tracker : https://pagure.io/centos-infra/issues/ , to adapt the same workflow as the Fedora infra team is already using. ( see https://docs.fedoraproject.org/en-US/cpe/working_with_us/ ) Concerned areas : - https://cbs.centos.org (Community BuildSystem, aka koji) - Special Interest Groups requests (for mirror, resources, etc) - https://ci.centos.org (All CI infra ecosystem) - Everything around CentOS Infra (mirror issues, etc) We have already moved/migrated for example the (opened) tickets that were filed under the "Buildsys" , "CI" and "Infrastructure" categories to the new issues tracker. The idea being to *not* request work to be done through IRC but rather through new infra issues tracker. Imported tickets will be discussed there and worked on (reviewed on a daily basis) after having been prioritized Should you feel a need to discuss this new process, feel free to do so in #centos-devel on irc.freenode.net or on this centos-devel list. Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

Call for migration to new openshift cluster
by Vipul Siddharth 19 Jul '20

19 Jul '20

Hi all, We are done and ready with the new Openshift 4 CI cluster. Now we want to start the second phase of it i.e migrating projects to the new cluster. If you are currently using apps.ci.centos.org (OCP 3.6), please contact us off-list so that we can create a namespace for you in the new OCP 4.4 cluster, so you can start migrating your workloads. It's not a hard deadline but we are hoping to retire the older Openshift 3.6 cluster in the upcoming quarter (3 months) and legacy (ci.centos.org) environment in a couple of months after that. If you are using ci.centos.org, it's a very good time to start working on updating your workload If you have any questions, please reach out to us Thank You and stay safe -- Vipul Siddharth Fedora | CentOS CI Infrastructure Team

3 12

CI Infra maintenance on Monday - 7am UTC for 2 hours
by Vipul Siddharth 13 Jul '20

13 Jul '20

Hi, On early July 13th, 2020 (Monday), CI will be out for ~2 hours for maintenance. During that time, Duffy will not be working which means we won't be handing out nodes in that window. Date: July 13th Estimated outage time: 0700 UTC to 0900 UTC Apologies for any inconveniences -- Vipul Siddharth Fedora | CentOS CI Infrastructure Team

1 1

Can not get node on ci.centos.org
by Katerina Foniok 16 Jun '20

16 Jun '20

Hi guys, we are not able to get nodes on ci.centos.org for quite some time. Our jobs are running on devtools-ci-slave04 <https://ci.centos.org/computer/devtools-ci-slave04> Can somebody please take a look? Thank you :) Katka

2 2

Update on OpenShift cluster upgrade
by Vipul Siddharth 19 May '20

19 May '20

Hi folks, Here is a quick and short update on what CI Infrastructure team has been up to for the last few days. # Done * identified infrastructure suitable for hosting the new ocp4 cluster and curated inventory * configured DNS for the new ocp4 ocp.ci.centos.org cluster * provisioned 2 haproxy load balancers * ansible template for PXE ocp4 nodes # Hiccups we had * Faced issue pulling machine config operator image from quay.io on the ocp4 bootstrap node. We reached out to Jonathan Lebon of Fedora CoreOS related help (Thank you Jonathan for the help). The issue was resolved by recreating the ignition files from scratch. * Faced issue with multiple ethernet interfaces on the RHCOS nodes and DHCP. Resolved by disabling DHCP, and configuring static IPs on a single interface. # In progress * Provisioning OCP4 nodes (Master, Worker, and Bootstrap) # Blockers * RHEL CoreOS does not support RAID on the root device Thank you and Stay Safe / Brian Stinson, David Kirwan, Fabian Arrotin & -- Vipul Siddharth Fedora | CentOS CI Infrastructure Team

2 3

← Newer
1
...
7
8
9
10
11
12
13
...
43
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users