CI-users

ci-users@lists.centos.org

2 participants
422 discussions

Performance issues on the EC2 T2 nodes
by František Šumšal 23 Oct '22

23 Oct '22

Hey! As the follow-up discussion after the migration to AWS suggested, I took one of our systemd CI jobs (which relies heavily on spawning QEMU VMs) from a metal machine to a T2 VM. This took quite a while and required tuning down the "depth" of some tests (which also means less coverage), but in the end it works quite reliably, and with parallelization the job takes a reasonable amount of time (~95 minutes). However, recently (~three weeks ago) I noticed that the jobs using T2 machines (from the virt-ec2-t2-centos-8s-x86_64 pool) started having issues - the runtime was over 4 hours (instead of the usual ~95 minutes) and several tests kept timing out. In this case the issue fixed itself before I had a chance to debug it further, though. Fast forward to this week, the issue appeared again on Monday, but, again, it fixed itself - this time while I was debugging it. I put some possibly helpful debug checks in place and waited for the next occurrence, which happened to be this Wednesday. Taking a couple of affected machines aside, I checked various things, including all possible logs, I/O measurements, etc. and in the end I noticed that more than 50% of the vCPU time is actually being "stolen" by the hypervisor: # mpstat -P ALL 5 7 Linux 4.18.0-408.el8.x86_64 (n27-29-92.pool.ci.centos.org) 19/10/22 _x86_64_ (8 CPU) ... Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 11.65 0.00 3.17 0.32 1.72 0.30 52.92 0.00 0.00 29.92 Average: 0 12.99 0.00 2.68 0.37 1.69 0.32 59.64 0.00 0.00 22.31 Average: 1 11.00 0.00 3.14 0.34 1.95 0.34 49.16 0.00 0.00 34.06 Average: 2 11.31 0.00 2.72 0.52 1.84 0.41 47.02 0.00 0.00 36.18 Average: 3 11.75 0.00 2.97 0.45 1.96 0.29 52.45 0.00 0.00 30.11 Average: 4 10.88 0.00 3.06 0.19 1.99 0.21 46.12 0.00 0.00 37.55 Average: 5 13.19 0.00 3.30 0.24 1.11 0.22 64.23 0.00 0.00 17.71 Average: 6 11.92 0.00 3.71 0.08 1.60 0.24 58.52 0.00 0.00 23.92 Average: 7 10.22 0.00 3.82 0.32 1.63 0.32 46.71 0.00 0.00 36.99 After some tinkering I managed to reproduce it on "clean" machine as well, with just `stress --cpu 8`, where the results were even worse: # mpstat -P ALL 5 7 Linux 4.18.0-408.el8.x86_64 (n27-34-82.pool.ci.centos.org) 19/10/22 _x86_64_ (8 CPU) Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 19.08 0.00 0.00 0.00 0.27 0.00 80.64 0.00 0.00 0.00 Average: 0 12.75 0.00 0.00 0.00 0.22 0.00 87.03 0.00 0.00 0.00 Average: 1 19.08 0.00 0.00 0.00 0.15 0.00 80.77 0.00 0.00 0.00 Average: 2 22.55 0.00 0.00 0.00 0.24 0.00 77.21 0.00 0.00 0.00 Average: 3 25.94 0.00 0.00 0.00 0.38 0.00 73.68 0.00 0.00 0.00 Average: 4 19.17 0.00 0.00 0.00 0.30 0.00 80.53 0.00 0.00 0.00 Average: 5 21.31 0.00 0.00 0.00 0.39 0.00 78.30 0.00 0.00 0.00 Average: 6 17.37 0.00 0.00 0.00 0.30 0.00 82.33 0.00 0.00 0.00 Average: 7 20.48 0.00 0.00 0.00 0.27 0.00 79.25 0.00 0.00 0.00 This is really unfortunate. As the EC2 VMs don't support nested virtualization, we're "at mercy" of TCG, which is quite CPU intensive. When this rate-limiting kicks in, the CI jobs take more than 4 hours, and that's with timeout protections in place (meaning the CI results are unusable) - without them they would take significantly more and would be most likely killed by the watchdog, which is currently 6 hours (iirc). And since the current project queue limit is 10 parallel jobs, this make the CI part almost infeasible. I originally reported it to the CentOS Infra tracker [0] but was advised to post it here, since this behavior is, for better or worse, expected, or at least that's how the T2 machines are advertised. Is there anything that can be done to mitigate this? The only available solution would be to move the job back to the metal nodes, but that's going against the original issue (and the metal pool is quite limited anyway). Cheers, Frantisek [0] https://pagure.io/centos-infra/issue/950 -- PGP Key ID: 0xFB738CE27B634E4B

3 3

CentOS CI infra changes: retiring ci.centos.org legacy jenkins
by Fabian Arrotin 03 Oct '22

03 Oct '22

as a FYI message : the legacy Jenkins server (initially shared instance between CI tenants) and available behind https://ci.centos.org will be decommissioned and powered off next week. It's not used anymore and was just put online with a notification about shutting it down and pointing to relevant ticket template to ask to be migrated to openshift (2 years ago) As we're also decommissioning other part of the CI infra, we'll also just delete that one from and also remove A record from public DNS (actually pointing to haproxy in front) related ticket: https://pagure.io/centos-infra/issue/931 Kind Regards -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

CentOS CI infra changes: Phase 2 approaching
by Fabian Arrotin 20 Sep '22

20 Sep '22

Hi , Reminder (see https://lists.centos.org/pipermail/ci-users/2022-June/004547.html) that we're approaching October 2022 and so Phase 2 of our move for CentOS CI infra. That means that on Monday October 3rd, we'll remove the seamicro pools and tenants still using the legacy API endpoint (that will also disappear end of this year in Phase 3 !) will get now EC2 instances transparently, instead of Seamicro bare-metal nodes. Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

infra outage : openshift (aka ocp.ci cluster)
by Fabian Arrotin 15 Sep '22

15 Sep '22

We are aware of an issue with the openshift cluster running CI workload. David is investigating the root cause as it's actually stuck between update/upgrade steps. We'll keep you informed on this list for status update[s] -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

Scheduled service migration : artifacts.ci.centos.org
by Fabian Arrotin 05 Sep '22

05 Sep '22

As announced some months ago, we are moving all CI infra components to new infra (mostly AWS) WRT Duffy api service itself, phase 1 was completed earlier this month (August) and phase 2 will be announced for October. Another service we need to migrate is https://artifacts.ci.centos.org, which is the next service we'll migrate next week. Migration is scheduled for """"Monday September 5th, 6:00 am UTC time"""". You can convert to local time with $(date -d '2022-09-05 06:00 UTC') The expected "downtime" should be really small, as it's will be just pointing DNS CNAME record to new host, and eventually (see below) last rsync between hosts. Worth knowing that CI tenants using that service in the past were relying on direct rsync access (tcp/873 and available only internally in dedicated CI VLAN) with a rsync secret. Due to the service being now publicly available on AWS, we decided to just disable plain rsync but allow rsync over ssh (or sftp/scp if you want), reusing your existing project keypair. More details available at https://sigs.centos.org/guide/ci/#artifacts-storage Don't forget to update your script for next monday or you'll not be able to push to the new storage server ! PS: while we consider all data "ephemeral" (and so to be discarded as there is also no backup at all for that temporary hosting solution), we can though migrate your existing data from old to new server. For that, please "opt-in" in the existing ticket so that we can keep track of projects we need to migrate. See ticket https://pagure.io/centos-infra/issue/906 Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

Network Issue on CentOS Infra/services
by Fabian Arrotin 29 Aug '22

29 Aug '22

Hi, We are currently suffering from a flapping network connectivity to the main DC where majority of the CentOS Infra is hosted. After some internal discussion, we confirmed that upstream link provider is aware of the issue and they are looking for a fix (but no ETA) Impacted services: - centos ci - git.centos.org - cbs.centos.org - mirror.centos.org (downstream consumer and having issues pulling content) - mirror.stream.centos.org (downstream consumer and having issues pulling content) - buildlogs.centos.org (same reason) We'll post an update when this will be finally resolved -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

Changes to CentOS CI: reminder of Phase 1 and 2
by Camila Granella 24 Aug '22

24 Aug '22

Hello everyone, This is a friendly reminder of the current and upcoming status of CentOS CI changes (check [1]). Projects that opted-in for continuing on CentOS CI have been migrated, and the new Duffy API is available. With that, *phase 0 has been completed*. Regarding *phase 1*, we are still working on a permanent fix for the DB Concurrency issues [2]. Also, as for our OpenShift new deployment, we have a staging environment up and running, and it should be available at the beginning of September 2022. In October 2022 we begin *phase 2* when we will work through the following items (these were also previously communicated in [1]): - legacy/compatibility API endpoint will handover EC2 instances instead of local seamicro nodes (VMs vs bare metal) - bare-metal options will be available through the new API only - legacy seamicro and aarch64/ThunderX hardware are decommissioned - only remaining "on-premises" option is ppc64le (local cloud)Feel free to reach out if you have any questions or concerns The final deadline for decommissioning the old infrastructure (*phase 3*) is *December 2022*. We will be communicating further until then, and meanwhile, reach out to any of us in case you have any questions. Regards, [1] [ci-users] Changes on CentOS CI and next steps: https://lists.centos.org/pipermail/ci-users/2022-June/004547.html [2] DB Concurrency issues: https://github.com/CentOS/duffy/issues/523 -- Camila Granella Associate Manager, Software Engineering Red Hat <https://www.redhat.com/> @Red Hat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

5 15

Duffy API sessions mismatch (not all the time)
by Fabian Arrotin 03 Aug '22

03 Aug '22

hi, We switched last Monday to new Duffy API and while we saw machines being requested (previous seamicro pool but also now VMs from AWS/EC2) and returned, from time to time tenants informed us of transient errors. Based on some troubleshooting, it seems that Duffy api was answering in the same second to either different tenant with same nodes (so nodes being handed over to different tenants) or even same tenant but with different session IDs (but same hostname) Nils (Duffy code author) is busy today looking at a fix and we'll let you know when we'll be able to roll it out. PS : that means that we'll have also to stop Duffy API and proceed with some DB clean-up operations to restart from a clean/fresh situation. That will mean that duffy will consider deployed node as unused and so will start reinstalling these (to start from clean situation). We'll let you know when we'll proceed with that hotfix push -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

Planned outage : Duffy CI infra migration phase 1 on August 1st
by Fabian Arrotin 28 Jul '22

28 Jul '22

As previously announced in June (see https://lists.centos.org/pipermail/ci-users/2022-June/004547.html and also https://www.youtube.com/watch?v=aqgs-3NnRmA) we'll migrate existing Duffy API service to the new Duffy v3, starting with Phase 1 . Migration is scheduled for """"Monday August 1st 7:00 am UTC time"""". You can convert to local time with $(date -d '2022-08-01 07:00 UTC') The expected API downtime will be under the 60 seconds (DNS TTL). Worth knowing: - new duffy api (in legacy mode) has currently less available nodes, but that will be resolved during the day (see below) - we'll have to wait ~6h (default maximum "lease" time for a test node) to be able to let new duffy api reinstall older seamicro nodes and so be available in the new duffy v3 pool (transitioning from old to new duffy pool) - currently running jobs will continue so to work, but when you'll hit new duffy api endpoint (DNS switch) to "return nodes" it will just answer that session doesn't exist (new one) but it's safe to ignore As soon as new duffy API is available you'll already be able to switch your workflow to new the Duffy API and so use the new features. Tenants/users documentation about how to interact with Duffy is available on https://sigs.centos.org/guide/ci/ (see the Duffy part of that documentation) That means that switching to EC2 nodes will be directly possible. During the day we'll tune the duffy nodepool configuration based on usage metrics. IMPORTANT remark : *only* projects/tenants that opted-in in the last 45 days are currently migrated (api and ssh keys), so if you haven't (yet), you can still opt-in, otherwise all your requests for duffy ephemeral nodes will be declined by new duffy api service next monday. In case of troubles, both Pedro (nick phsmoura) and myself (nick arrfab) will be also present in the #centos-ci irc channel on irc.libera.chat the whole day. Thanks for your understanding and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

2 3

CentOS CI changes: opt-in reminder + update on OpenShift new deployment
by Camila Granella 11 Jul '22

11 Jul '22

Hello everyone, This is a friendly reminder to opt-in if you are interested in using CentOS CI for your project after the changes we announced last month (check "[ci-users] Changes on CentOS CI and next steps <https://lists.centos.org/pipermail/ci-users/2022-June/004547.html>"). *Projects will be migrated only if they opt-in*. Please,* opt-in by August 2022*, you can do that by simply replying to this email expressing your interest. Also, we now have a deadline for the OpenShift new deployment as per our last email. It should be up and running at the *beginning of September 2022* . Feel free to reach out if you have any questions or concerns, -- Camila Granella Associate Manager, Software Engineering Community Platform Engineering Red Hat <https://www.redhat.com/> @Red Hat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

2 2

← Newer
1
2
3
4
5
6
7
8
9
...
43
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users