CI-users

ci-users@lists.centos.org

418 discussions

Changes to CentOS CI: New OpenShift instance and Updates
by Camila Granella 02 Jan '23

02 Jan '23

*TL;DR: new OpenShift instance is up and running for CentOS CI, migrate your workloads by March 31st, 2023. Test your access to both the Duffy API and the new OpenShift instance by the end of 2022. CentOS CI changes approaching the end of Phase 3. * Hello everyone, We are happy to announce that the new OpenShift instance is up and running (ticket [0]). For the tenants that opted in to continue using CentOS CI, you are being required to *migrate your workload to the new OpenShift instance by the end of Q1CY23*, with the final deadline being *March 31st, 2023*. In order to support you through the migration process, the team has created a quick guide that can be found in [1]. We would also like to request you test your access to both the Duffy API and the new OpenShift instance by the *end of 2022*. Please reach out to us in case it is not working. Here is the link to the new OCP cluster [2]. As a reminder, this new OCP cluster is running in AWS, on top of EC2 instances, so coming with different limitations when compared with “on-premises bare-metal OCP deployments”. One notable change is that it is not possible to expose /dev/kvm and so there is no way for tenants to deploy VMs through OpenShift (like with the kubevirt operator or else). Tenants having a need to use VMs can then request some through Duffy API. Finally, here's a friendly reminder of the CentOS CI changes' current status. According to our previous communications [3], we are approaching the end of the last phase of the changes that have been implemented to CentOS CI. *Phase 3 - Decommission* - Legacy/compatibility API deprecated and requests (even for EC2 instances) will no longer be accepted - All tenants that opted in will be using only EC2 for aarch64/x86_64 and on-premise cloud for ppc64le For a full description of the project phases, you can check [3]. Please, feel free to reach out to us in case you have any questions or concerns. Regards, [0] https://pagure.io/centos-infra/issue/969 [1] https://sigs.centos.org/guide/ci/#migration-to-new-ci-instance [2] https://console-openshift-console.apps.ocp.cloud.ci.centos.org/ [3] https://lists.centos.org/pipermail/ci-users/2022-June/004547.html -- Camila Granella Associate Manager, Software Engineering Red Hat <https://www.redhat.com/> @Red Hat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

3 3

Duffy Phase 3 : removing duffy legacy compat api
by Fabian Arrotin 12 Dec '22

12 Dec '22

Hi, As announced in our initial plan (and Camilla sent a reminder last week about that too), we'll just deprecate the internal api endpoint running on duffy (tcp port 8080, mainly for compatibility with the now deprecated cico client). All tenants had enough time (since August) to migrate from old api to new duffy one, as documented on https://sigs.centos.org/guide/ci/#duffy-ephemeral-bare-metalvirtual-machine… So this mail is just a reminder that we'll drop that legacy/compatibility api endpoint today, as announced already Tracking ticket: https://pagure.io/centos-infra/issue/998 Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

Changes on CentOS CI and next steps
by Camila Granella 02 Dec '22

02 Dec '22

*TL;DR: CentOS CI is going hardwareless and if you wish your project remains using it, we need your opt-in by August 2022. There is a Dojo Summer 2022 <https://wiki.centos.org/Events/Dojo/Summer2022> session happening on Thursday, June 17th, that will explain further technical details. * Hello everyone, As many of you know, since the beginning of this year we have been reevaluating the future of CentOS CI, as currently the hardware being used for it is out of warranty. This is due to the fact that CentOS CI came from community donations of hardware which were maintained in a best effort manner by our team. With no warranties, when the physical machine dies we have no means to replace it. Right now though, our hardware, due to a lack of warranty, will not be moved with the upcoming data center changes due to data center requirements to have in warranty hardware for supportability. We decided to take this opportunity to modernize our current infrastructure, pushing it to a hybrid cloud environment. Duffy CI will become the main tool from now on, so that we can support the CI workflow and best practices on cloud and for this reason, the current hardware infra will no longer be available soon. However, as an effort to continuously provide resources and support CI best practices for projects, our team is adapting Duffy CI so that we can maintain most of the characteristics of our current, physical-based offering. At the technical level, what does that mean for you, CI tenants? - A new Duffy API service will replace the existing one: while it will be running in compatibility/legacy mode with the previous version, you will need to adapt your workflow to the new API, but more details below - We will transition to AWS EC2 instances for the aarch64 and x86_64 architectures by default, with a (limited) option to request “metal” instances for projects requiring virtualization for their tests (like KVM/vagrant/etc) - We will keep a (very small) Power9 infra “on-premise” (AWS does not support ppc64le) for the ppc64le tests (available through a dedicated VPN tunnel) - The existing OpenShift cluster will be also decommissioned and a new one (hosted in AWS, so without an option to run kubevirt operator nor VMs) will be then used (you will have to migrate from one to the other) With that being said, tenants can start preparing for the changes to happen with the maximum deadline of the end of December 2022 wherein at this point, Duffy API legacy mode will be removed. You are required to opt-in if you and/or your team want to use Duffy CI. Projects will only be migrated if they reply to this email confirming that they wish to proceed. Worth knowing that not opting in means that your API key will not be migrated and so all your requests to get temporary/ephemeral nodes will be rejected by the new Duffy API. The maximum decommission deadline of the current hardware infrastructure is December 12th, 2022 and the new Duffy CI will go live in August 2022, so please, complete your migration process by the end of CY22. Reminders of deadlines and of the opt-in requirements will be sent monthly, but your confirmation of opt-in is required by August 2022. When approaching December, reminders about deadlines frequency will increase so that we can ensure effective communication throughout the process. Here are the steps in which we will migrate CI Infra: Phase 1 - Deploy Duffy V3 (August 2022) - Deploy in legacy/compatibility mode, so existing tenants (that opted in !) can still request duffy nodes the same way (like with 'python-cicoclient') : no change at tenants side, and exactly same hardware for tests (transparent migration) - New Duffy API endpoint becomes available, and tenants can start adapting their workflows to point to new API (new ‘duffy-cli’ tool coming, with documentation) - Bare metal and VMs options will be available already through the new API (x86_64, aarch64, ppc64le) Phase 2 - Hybrid Cloud (October 2022) - Legacy/compatibility API endpoint will handover EC2 instances instead of local seamicro nodes (VMs vs bare metal) - Bare metal options will be available through the new API only - Legacy seamicro and aarch64/ThunderX hardware are decommissioned - Only remaining "on-premise" option is ppc64le (local cloud) Phase 3 - Decommission (December 2022) - Legacy/compatibility API deprecated and requests (even for EC2 instances) will no longer be accepted - All tenants that opted in will be using only EC2 for aarch64/x86_64 and on-premise cloud for ppc64le OpenShift new deployment planning and timeline To be defined (deadline for planning and timeline: end of June 2022) Do not hesitate to reach out if you have any questions. It is worth knowing that there will be a dedicated session about the Future of CentOS CI infra at the next CentOS Dojo happening on June 17h (check Dojo Summer 2022 <https://wiki.centos.org/Events/Dojo/Summer2022>). That session will be recorded and then available on Youtube but if you have any questions. Feel free to join the CentOS Dojo and reach out to us! Best regards, -- Camila Granella Associate Manager, Software Engineering Red Hat <https://www.redhat.com/> @Red Hat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

9 20

Disabling dnf execution in background on an EC2 bare metal node
by Anoop C S 28 Nov '22

28 Nov '22

Hi, Post migration to new Duffy API we started noticing the following RPM error intermittently on bare metal nodes reserved from EC2 pool: . . . Running transaction check Waiting for process with pid 2881 to finish. Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction RPM: error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch RPM: error: cannot open Packages index using db5 - (-30969) RPM: error: cannot open Packages database in /var/lib/rpm The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Could not run transaction. . . . Especially the "Waiting for process with pid 2881 to finish." indicates some DNF operation in background which may or may not conflict with current DNF execution. With the help of others I got to know about a cloud-init service performing `dnf update` to cope with the very old AMI image with which it is provisioned. Please note that nodes are already marked ready in the pool before `dnf update` kicks in. Therefore I put forward a suggestion to disable `dnf update` as part of cloud-init service such that it does not interfere with other DNF operations done after the node is reserved by a tenant. Please feel free to correct me in any of the details mentioned above and let me know your thoughts. Thanks, Anoop C S.

3 9

Fwd: [CentOS-devel] Changes to CentOS CI: reminder of Phase 3 and update on OpenShift new instance
by Fabian Arrotin 17 Nov '22

17 Nov '22

-------- Forwarded Message -------- Subject: Re: [CentOS-devel] Changes to CentOS CI: reminder of Phase 3 and update on OpenShift new instance Date: Thu, 17 Nov 2022 16:39:48 +0100 From: Fabian Arrotin <arrfab(a)centos.org> Reply-To: The CentOS developers mailing list. <centos-devel(a)centos.org> To: centos-devel(a)centos.org

1 0

Changes to CentOS CI: reminder of Phase 3 and update on OpenShift new instance
by Camila Granella 09 Nov '22

09 Nov '22

Hello everyone, This is a friendly reminder of CentOS CI changes' current status and upcoming action items (check [0]) in addition to a quick update on our new OpenShift instance. According to our previous communications [0], we are approaching the last phase of the changes that have been implemented to CentOS CI. We are currently closing phase 2, which began in October. In December, we will move to *phase 3*: *Phase 3 - Decommission* - Legacy/compatibility API deprecated and requests (even for EC2 instances) will no longer be accepted - All tenants that opted in will be using only EC2 for aarch64/x86_64 and on-premise cloud for ppc64le For a full description of the project phases, you can check [0]. Currently, our Infra & Releng team is working to deploy our new OpenShift instance (AWS) in production after some improvements that have been done in staging. When OpenShift gets fully set up, the team will work to migrate the Pagure project. This will then be documented and passed on to you (users) as a guide in order to *facilitate the workload migration that is expected to be done on your side*. We are expecting this to be completed by the end of the year. We will be sharing further information on this at the beginning of Q1CY2023. A reasonable timeline will be provided so that you have enough bandwidth to work on the migration of your workloads. Please, feel free to reach out to us in case you have any questions or concerns. Regards, [0] [ci-users] Changes on CentOS CI and next steps: https://lists.centos .org/pipermail/ci-users/2022-June/004547.html -- Camila Granella Associate Manager, Software Engineering Red Hat <https://www.redhat.com/> @Red Hat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

1 0

Performance issues on the EC2 T2 nodes
by František Šumšal 23 Oct '22

23 Oct '22

Hey! As the follow-up discussion after the migration to AWS suggested, I took one of our systemd CI jobs (which relies heavily on spawning QEMU VMs) from a metal machine to a T2 VM. This took quite a while and required tuning down the "depth" of some tests (which also means less coverage), but in the end it works quite reliably, and with parallelization the job takes a reasonable amount of time (~95 minutes). However, recently (~three weeks ago) I noticed that the jobs using T2 machines (from the virt-ec2-t2-centos-8s-x86_64 pool) started having issues - the runtime was over 4 hours (instead of the usual ~95 minutes) and several tests kept timing out. In this case the issue fixed itself before I had a chance to debug it further, though. Fast forward to this week, the issue appeared again on Monday, but, again, it fixed itself - this time while I was debugging it. I put some possibly helpful debug checks in place and waited for the next occurrence, which happened to be this Wednesday. Taking a couple of affected machines aside, I checked various things, including all possible logs, I/O measurements, etc. and in the end I noticed that more than 50% of the vCPU time is actually being "stolen" by the hypervisor: # mpstat -P ALL 5 7 Linux 4.18.0-408.el8.x86_64 (n27-29-92.pool.ci.centos.org) 19/10/22 _x86_64_ (8 CPU) ... Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 11.65 0.00 3.17 0.32 1.72 0.30 52.92 0.00 0.00 29.92 Average: 0 12.99 0.00 2.68 0.37 1.69 0.32 59.64 0.00 0.00 22.31 Average: 1 11.00 0.00 3.14 0.34 1.95 0.34 49.16 0.00 0.00 34.06 Average: 2 11.31 0.00 2.72 0.52 1.84 0.41 47.02 0.00 0.00 36.18 Average: 3 11.75 0.00 2.97 0.45 1.96 0.29 52.45 0.00 0.00 30.11 Average: 4 10.88 0.00 3.06 0.19 1.99 0.21 46.12 0.00 0.00 37.55 Average: 5 13.19 0.00 3.30 0.24 1.11 0.22 64.23 0.00 0.00 17.71 Average: 6 11.92 0.00 3.71 0.08 1.60 0.24 58.52 0.00 0.00 23.92 Average: 7 10.22 0.00 3.82 0.32 1.63 0.32 46.71 0.00 0.00 36.99 After some tinkering I managed to reproduce it on "clean" machine as well, with just `stress --cpu 8`, where the results were even worse: # mpstat -P ALL 5 7 Linux 4.18.0-408.el8.x86_64 (n27-34-82.pool.ci.centos.org) 19/10/22 _x86_64_ (8 CPU) Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 19.08 0.00 0.00 0.00 0.27 0.00 80.64 0.00 0.00 0.00 Average: 0 12.75 0.00 0.00 0.00 0.22 0.00 87.03 0.00 0.00 0.00 Average: 1 19.08 0.00 0.00 0.00 0.15 0.00 80.77 0.00 0.00 0.00 Average: 2 22.55 0.00 0.00 0.00 0.24 0.00 77.21 0.00 0.00 0.00 Average: 3 25.94 0.00 0.00 0.00 0.38 0.00 73.68 0.00 0.00 0.00 Average: 4 19.17 0.00 0.00 0.00 0.30 0.00 80.53 0.00 0.00 0.00 Average: 5 21.31 0.00 0.00 0.00 0.39 0.00 78.30 0.00 0.00 0.00 Average: 6 17.37 0.00 0.00 0.00 0.30 0.00 82.33 0.00 0.00 0.00 Average: 7 20.48 0.00 0.00 0.00 0.27 0.00 79.25 0.00 0.00 0.00 This is really unfortunate. As the EC2 VMs don't support nested virtualization, we're "at mercy" of TCG, which is quite CPU intensive. When this rate-limiting kicks in, the CI jobs take more than 4 hours, and that's with timeout protections in place (meaning the CI results are unusable) - without them they would take significantly more and would be most likely killed by the watchdog, which is currently 6 hours (iirc). And since the current project queue limit is 10 parallel jobs, this make the CI part almost infeasible. I originally reported it to the CentOS Infra tracker [0] but was advised to post it here, since this behavior is, for better or worse, expected, or at least that's how the T2 machines are advertised. Is there anything that can be done to mitigate this? The only available solution would be to move the job back to the metal nodes, but that's going against the original issue (and the metal pool is quite limited anyway). Cheers, Frantisek [0] https://pagure.io/centos-infra/issue/950 -- PGP Key ID: 0xFB738CE27B634E4B

3 3

CentOS CI infra changes: retiring ci.centos.org legacy jenkins
by Fabian Arrotin 03 Oct '22

03 Oct '22

as a FYI message : the legacy Jenkins server (initially shared instance between CI tenants) and available behind https://ci.centos.org will be decommissioned and powered off next week. It's not used anymore and was just put online with a notification about shutting it down and pointing to relevant ticket template to ask to be migrated to openshift (2 years ago) As we're also decommissioning other part of the CI infra, we'll also just delete that one from and also remove A record from public DNS (actually pointing to haproxy in front) related ticket: https://pagure.io/centos-infra/issue/931 Kind Regards -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

CentOS CI infra changes: Phase 2 approaching
by Fabian Arrotin 20 Sep '22

20 Sep '22

Hi , Reminder (see https://lists.centos.org/pipermail/ci-users/2022-June/004547.html) that we're approaching October 2022 and so Phase 2 of our move for CentOS CI infra. That means that on Monday October 3rd, we'll remove the seamicro pools and tenants still using the legacy API endpoint (that will also disappear end of this year in Phase 3 !) will get now EC2 instances transparently, instead of Seamicro bare-metal nodes. Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

infra outage : openshift (aka ocp.ci cluster)
by Fabian Arrotin 15 Sep '22

15 Sep '22

We are aware of an issue with the openshift cluster running CI workload. David is investigating the root cause as it's actually stuck between update/upgrade steps. We'll keep you informed on this list for status update[s] -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

← Newer
1
2
3
4
5
6
7
8
...
42
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users