devel August 2020

devel@lists.centos.org

27 participants
22 discussions

CPE Weekly: 2020-08-14
by Aoife Moloney 14 Aug '20

14 Aug '20

--- title: CPE Weekly status email tags: CPE Weekly, email --- # CPE Weekly: 2020-08-14 Background: The Community Platform Engineering group, or CPE for short, is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our wiki page here for more information:https://docs.fedoraproject.org/en-US/cpe/ ## General Project Updates The CPE team are working on the following projects for Quarter 3, which is the months of July, August & September: * Data Centre Move - Final Works * CentOS Stream Phase 3 * Noggin Phase 3 * Packager Workflow Healthcare * Fedora Messaging Schemas Details of the above projects, and of projects currently in progress, done and what projects are in our backlog, can be found on our taiga board per project card: https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null We also have an updated initiative timetable for briefing in new projects to our team & key dates here:https://docs.fedoraproject.org/en-US/cpe/time_tables/ *Note: Initiatives are large pieces of work that require a team of people and weeks/months to complete. Please continue to open tickets in the normal way for bugs, issues, etc. ### CPE Product Owner Office Hours #### #fedora-meeting-1 * Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1 * Next Meeting: 2020-08-20 #### #centos-meeting * Every second Tuesday @ 1500 UTC on #centos-meeting * Next Meeting: 2020-08-18 ### Misc #### Nest With Fedora Note Thank you to everyone who attended Nest With Fedora over the weekend and engaged with us during our talks and social sessions. It was a fantastic event, very well put together and run by the Fedora team and it was both my pleasure and honour to have been a part of what was a fantastic schedule! I cant wait to catch up on talks I missed when they are uploaded, and the CPE team, with thanks to Michal Konecny who put the structure of our piece together, will be submitting a collaborative blog post to the Community blog space in the coming weeks, recapping our experience at Nest this year. Well done Marie Nordin, Matthew Miller and the wider team, and our very own Vipul Siddarth, on what was a very successful and enjoyable event! #### Engagement Email Feedback At the beginning of July I sent an email to devel-announce requesting feedback from the community on changes I and the wider CPE team have made when scheduling projects to work on, in an effort to find the balance between work and life. We are still searching :) However, I got some very good tips that I will definitely be incorporating which I shared at Nest during the CPE AMA Session and in reply to the original mail. The link to the mails are here for full reading https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapr… But the suggestions and actions are here for quicker reference: Continue to communicate regularly on projects & updates - Will do, weekly emails have been a bit more sporadic lately as I have had some time off Less acronyms & abbreviations in comms :) - Sure, that's an easy fix on my end and makes sense Publish team members timezone on docs.fpo/cpe to help define our ‘working hours’ - Will do, I hope to get to this by end of August and they will be reflected on docs.fpo/cpe Publish the workflow diagram to docs.fpo/cpe and add filtered versions that are user specific - Same as above, publishing it on docs.fpo/cpe-initiatives Office Hours on IRC are a useful way to contact team Product Owner - Great to hear, please feel free to stop by when you can/want to. They are on Thursdays @ 1300 UTC on #fedora-meeting-1 and every second Tuesday @ 1500 UTC on #centos-meeting Public tracker for bugs - Our team meets twice a day, every day on IRC to review tickets and issues to work on. They meet @ 0830 UTC on #centos-meeting and again @ 1800 UTC on #fedora-admin. These are public meetings so please feel free to attend. If you would like to offer any further suggestions or feedback on the Community Platform Engineering team, please feel free to complete our survey which is open until August 30th https://fedoraproject.limequery.com/696793?lang=en ## Fedora Updates ### Data Centre Move * Nearly done!!!!!! * Firewalls for staging are going up this week * Last of the hardware has been set up for networking changes * The bringup of Communishift is being made into a dedicated project for work as soon as the team have capacity to do so - this may be in late September as the members of the colo team will (hopefully) take some very well deserved time off work before tackling this one :) ### AAA Replacement * Some of the Noggin team have been enjoying some very well deserved time off work over the last week or two so work has, naturally slowed. * The code is currently being security audited by Patrick Uiterwijk, thank you Patrick! * Next steps will be to successfully deploy Noggin to staging when it has been brought back up - we are estimating this to be late next week * Once Noggin has been deployed to staging, we would love some community feedback on the application and its performance. We will be emailing the infra/devel lists on when to test and how to give us feedback when we are fully deployed in stg. * In the meantime. Please feel free to check out the team kanban board for more information on the features the team are working on and have already completed here https://github.com/orgs/fedora-infra/projects/6 ### Fedora Messaging Schemas * This project is being worked on by the same team as Noggin, so there has been some progress made but not a lot as team members enjoy some holidays * The team have already built a list of applications that require messaging schemas, list can be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit * They also have completed a readme which contains documentation on messaging schemas, a cookie-cutter template to create the schema and a definition of Done for writing a schemas https://github.com/fedora-infra/fedora-messaging-schemas-issues * The board they are working from can be viewed here https://github.com/orgs/fedora-infra/projects/7 ### Packager Workflow Healthcare * Project information: This is an investigative project that aims to look at the entire packager workflow as a single piece of tooling to identify where failures happen, and try to identify when and why packages fail at different points within the workflow. We hope to have two possible outcomes from this project at the end of the quarter (September): * The workflow breaks at X point and we will work on a solution to fix * OR * The workflow works fine, but we will need better monitoring on the pipeline so we will work on a solution for this * The team are using this Monitor-Script https://pagure.io/fedora-ci/monitor-gating/ and are making improvements to it on resiliency/reliability. * They are finishing the investigation phase of the project and are going to document the packager workflow (with graphs I have been promised!) showcasing how the different systems interact with one another * And are working on an outline of the workflow steps (from packager PoV) and systems involved (CPE team PoV), identifying metrics to be measured * The teams work is being tracked here https://teams.fedoraproject.org/project/cpe-cicd/kanban ## CentOS Updates ### CentOS * CentOS-infra team have created a ticket board, check it out here https://pagure.io/centos-infra/boards/CentOS%20CI%20Infra * CBS Koji x86 builders moved to new(er) infrastructure (https://lists.centos.org/pipermail/centos-devel/2020-August/055988.html) * The team also caught up C6, C7 and C8 Linux .. 2 outstanding Bootstrap Modules for C8 Linux (eclipse and the latest rust-toolset). ### CentOS Stream * Not too much to report this week - The team are mostly working on developing utility scripts that will ease the CentOS 8 and CentOS Stream packaging workflow and business as usual updates to CentOS Stream. As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great week! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

Infra Pre-Announce : moving CI ssh jump host soon, please read !
by Fabian Arrotin 13 Aug '20

13 Aug '20

Hi, As you noticed recently, we started to refresh the infra used for CentOS CI (not the hardware, still the same, but the software stack and the way to control/manage it). One of the identified nodes still being used and that needs to be converted to the new infra layout is the ssh jumphost (see https://wiki.centos.org/QaWiki/CI/GettingStarted#How_to_use_it) Normally, some of you have switched to OpenShift workload, (including to the new Openshift 4/OCP setup that went live recently) but some Projects are still on the old setup with sometimes a need to reach dedicated/shared VMs acting as Jenkins agent[s], connected to Jenkins behind https://ci.centos.org. We have already provisioned a new VM in the new setup (that can reach the whole CI subnet and VLAN) but here are some points to consider, reason why we wanted to pre-announce long time in advance before we do the real switch) : * New ssh jump host is CentOS 8 based, versus CentOS 6, meaning that if you used ssh-dss key (instead of ssh-rsa), you'll *not* be able to connect through that new host. We already identified such keys and Vipul will try (when it's tied to a real email address for the project) to reach out. But better to announce it here too, so that you have time to ask us to reflect a change (through ticket on https://pagure.io/centos-infra/issues) * Old VM allowed shell access, but it will be disallowed on the new one (there is no need for shell on that intermediate node anyway). Reminder that you can configure your ssh config to directly use ProxyCommand or even now ProxyJump (on recent openssh-client). See https://wiki.centos.org/TipsAndTricks/SshTips/JumpHost) * Because the host has a new sshd_host_key, it will come with a new fingerprint too, so if you have automation and that you don't trust our CA already, the fingerprint for new host will be : [fingerprint] rsa=3072 SHA256:n7y0qZS/FvhjaskOBds3TTKQh5EtgNQ25E7cmTNBATg (RSA) rsa_md5=3072 MD5:9e:83:46:d0:c5:8a:a0:94:50:10:58:9d:af:ca:50:19 (RSA) ecdsa=256 SHA256:ZQacwDsWkKBYL9HJJYwHr94Ny1sMhHMDnz9GiLFb8Uc (ECDSA) ecdsa_md5=256 MD5:dd:24:ea:6a:fd:8b:29:3d:1d:d0:a9:32:8c:b2:ea:62 (ECDSA) As we know that it's August and that some of you are probably on PTO (coming back or leaving soon), after discussion with Vipul , David and myself, we considered that we'll probably go live around beginning of September. Should you have any question around that migration, feel free to reply to this thread (ideally on dedicated ci-users mailing list), or on irc.freenode.net (#centos-ci) On behalf of the CentOS CI infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

NFV SIG meeting minutes - 2020-08-12
by YATIN KAREL 12 Aug '20

12 Aug '20

============================================= #centos-meeting: NFV SIG meeting - 2020-08-12 ============================================= Meeting started by ykarel at 15:07:53 UTC. The full logs are available athttps://www.centos.org/minutes/2020/August/centos-meeting.2020-08-12-15.0… . Meeting summary --------------- * roll call (ykarel, 15:08:23) * New issue tracking for infra issues (ykarel, 15:12:43) * LINK: https://pagure.io/centos-infra/issues/ (ykarel, 15:12:57) * LINK: https://wiki.centos.org/SIGGuide (ykarel, 15:14:00) * SIG Membership (ykarel, 15:14:15) * All members requested membership(https://bugs.centos.org/view.php?id=17629) have been added to sig-nfv (ykarel, 15:14:31) * LINK: https://git.centos.org/group/sig-nfv (ykarel, 15:14:41) * Distgit branches (ykarel, 15:17:54) * https://bugs.centos.org/view.php?id=17614 is resolved (ykarel, 15:18:42) * Pushed a branch(c8-sig-nfv-openvswitch-2.13) for openvswitch(imported openvswitch2.13-2.13.0-39), ovn(imported ovn2.13-2.13.0-39) (ykarel, 15:19:21) * Built both ovn and openvswitch 2.13 against nfv8-openvswitch-2-el8 tag https://cbs.centos.org/koji/builds?tagID=2120 (ykarel, 15:19:42) * LINK: https://cbs.centos.org/koji/builds?tagID=2120 (ykarel, 15:20:17) * LINK: https://git.centos.org/rpms/ovn/branches (ykarel, 15:21:31) * Packages available in Testing repo:- https://buildlogs.centos.org/centos/8/nfv/x86_64/openvswitch-2/Packages/o/ (ykarel, 15:24:57) * tuned profiles (tfherbert, 15:30:13) * Any other discussion? (tfherbert, 15:42:09) * chair for next meeting (ykarel, 15:46:23) * ACTION: tfherbert to chair next week (ykarel, 15:47:46) * ACTION: correction next meeting not next week. (tfherbert, 15:48:26) Meeting ended at 15:48:34 UTC. Action Items ------------ * tfherbert to chair next week * correction next meeting not next week. Action Items, by person ----------------------- * tfherbert * tfherbert to chair next week * **UNASSIGNED** * correction next meeting not next week. People Present (lines said) --------------------------- * ykarel (64) * tfherbert (31) * dholler (13) * centbot (4) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot _______________________________________________ CentOS-devel mailing list CentOS-devel(a)centos.org https://lists.centos.org/mailman/listinfo/centos-devel

1 0

CBS : builders migration (already done, nothing to worry about)
by Fabian Arrotin 10 Aug '20

10 Aug '20

Hi, On my backlog for quite some time, there was need to migrate existing CBS/koji builders to new(er) hardware/hosts. When I moved items from backlog to the new place, I so imported this one : https://pagure.io/centos-infra/issue/52 This mail is just a "FYI" as nobody should have noticed any service disruption, as each kojid instance was migrated while no job was running, so disabled, migrated, enabled back in the pool and so on. Hopefully you still *should* notice some speed improvements, including for build jobs (newer/faster nodes) and surely because local storage for these builders is SSD backed (instead of old SAS disks on 8y+ nodes). Same rule applies for the distrepo tasks, as we now have multiple nodes in the "createrepo" channel, with access to the koji NFS share now over 10Gbit ethernet (while it' was limited to 1Gbit on older nodes). That was possible because of the previous storage migration that happened last week : https://lists.centos.org/pipermail/centos-devel/2020-August/055981.html Should you encounter any issue, (but we're ran several --scratch builds) , feel free to reply to this email, drop on #centos-devel irc channel on irc.freenode.net, or even better now (really!), create a ticket on https://pagure.io/centos-infra/issues. Happy building ! -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

Re: [CentOS-devel] [CentOS-announce] CentOS Linux, CentOS Stream and the Boot Hole vulnerability
by Leon Fauster 09 Aug '20

09 Aug '20

Am 29.07.20 um 19:38 schrieb Brian Stinson: > We are aware of the Boot Hole vulnerability in grub2 (CVE-2020-1073) and > are working on releasing new packages for CentOS Linux 7, CentOS Linux 8 > and CentOS Stream in response. These should make it out to a mirror near > you shortly. > Should be ? CVE-2020-10713 and CVE-2020-14308 CVE-2020-14309 CVE-2020-14310 CVE-2020-14311 -- Leon

3 3

centos 7.8 and kernel-3.10.0-1127.18.2.el7
by Stefan Puiu 06 Aug '20

06 Aug '20

Hi, We have a CentOS 7 -based (more accurately, CentOS 7 atomic host-based) distro that we build and run on specialized hardware. For the kernel, we pick the CentOS 7 kernel, apply two patches (and a different configuration) and build it. We're mostly following CentOS 7.8 at the moment, and using the 3.10.0-1127.13.1 kernel. A few days ago, I noticed kernel-3.10.0-1127.18.2.el7 was out (and it's labeled as a security update - https://lists.centos.org/pipermail/centos-announce/2020-July/035780.html) so I decided to build it. Our kernel build process involves downloading the src.rpm, patching the spec and then calling rpmbuild. For this latest kernel, our spec patch (which adds our 2 patches) failed to apply - as far as I can tell, the hunks where patches are listed, like this one: == @@ -449,6 +449,8 @@ Patch1000: debrand-single-cpu.patch Patch1001: debrand-rh_taint.patch Patch1002: debrand-rh-i686-cpu.patch +Patch88881: kernel_ixia.patch +Patch88882: at24.patch BuildRoot: %{_tmppath}/kernel-%{KVRA}-root == I checked the new spec file, and the debrand patches are missing. Looking at https://git.centos.org/rpms/kernel/commits/c7 (let me know if that's the right place), I see there's a debranding change (https://git.centos.org/rpms/kernel/c/548139faa91edfe29fc84695da827230de3c5d…) but it simply added a README saying debranding failed. Do I need to wait for a new change fixing this? I see older kernels have a "debranding" commit and then a "Manual CentOS Debranding" commit, is something like that required now as well? Thanks in advance, Stefan.

2 1

installing cbs client on Fedora
by Ken Dreyer 06 Aug '20

06 Aug '20

Hi folks, What is the best way to get /usr/bin/cbs and the corresponding Koji profile config on Fedora? I'm wondering if we should package centos-packager in Fedora. My concern is that pull requests are not merged upstream at https://github.com/bstinsonmhk/centos-packager/pull/3 - Ken

4 8

[Infra] - Planned outage : cbs.centos.org
by Fabian Arrotin 06 Aug '20

06 Aug '20

Due to a hardware issue, we'll have to move the existing NFS share used by cbs.centos.org to a new node. Migration is scheduled for """"Thursday August 5th, 6:30 am UTC time"""". You can convert to local time with $(date -d '2020-08-05 6:30 UTC') The expected "downtime" is estimated to ~30 minutes , time needed to stop services, reflect change for NFS storage, update dns A record , and ansible restarting the whole process. During that downtime, nothing will be processed on the koji builders (nor signed and pushed out to mirror CDN) Thanks for your comprehending and patience. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 2

New CentOS Infra/CBS/CI issues/requests tracker, please read !
by Fabian Arrotin 05 Aug '20

05 Aug '20

Hi all, As you probably noticed in the last weeks/months, we have a stronger collaboration and synergy with the Fedora infrastructure team. Combining forces and resources help both projects at the same time, as majority of the CentOS contributors are already Fedora contributors and probably the same in reverse. It's not a secret (it was announced through CPE weekly mails on this list) that the CentOS board approved the idea of merging authentication systems in a near future (as an example). This email to let you know that all RFE/issues concerning the following areas should be reported to a new issues tracker : https://pagure.io/centos-infra/issues/ , to adapt the same workflow as the Fedora infra team is already using. ( see https://docs.fedoraproject.org/en-US/cpe/working_with_us/ ) Concerned areas : - https://cbs.centos.org (Community BuildSystem, aka koji) - Special Interest Groups requests (for mirror, resources, etc) - https://ci.centos.org (All CI infra ecosystem) - Everything around CentOS Infra (mirror issues, etc) We have already moved/migrated for example the (opened) tickets that were filed under the "Buildsys" , "CI" and "Infrastructure" categories to the new issues tracker. The idea being to *not* request work to be done through IRC but rather through new infra issues tracker. Imported tickets will be discussed there and worked on (reviewed on a daily basis) after having been prioritized Should you feel a need to discuss this new process, feel free to do so in #centos-devel on irc.freenode.net or on this centos-devel list. Kind Regards, -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 0

kernel-4.18.0-193.14.2.el8_2 in git.centos.org
by Antal Nemeš 03 Aug '20

03 Aug '20

Hi all, I need sources for latest Centos 8.2 kernel (kernel-4.18.0-193.14.2.el8_2). Since they are (now customarily) not available on vault.centos.org, I am attempting to build from git.centos.org. However, I am unable to find the exact commit to build from. On https://git.centos.org/rpms/kernel/commits/c8, I see 13.2 and 14.3, but not 14.2 that actually shipped. $git log --format="%C(auto) %h %s" origin/c8 | head -n 20 08e2843 debrand kernel-4.18.0-193.14.3.el8_2 284ac7b import kernel-4.18.0-193.14.3.el8_2 f990a59 Manual CentOS Debranding 6694789 debrand kernel-4.18.0-193.13.2.el8_2 56933c6 import kernel-4.18.0-193.13.2.el8_2 efc1e5b Manual CentOS Debranding afc068a debrand kernel-4.18.0-193.6.3.el8_2 d0c1e45 import kernel-4.18.0-193.6.3.el8_2 9af314c Manual CentOS Debranding b36366f debrand kernel-4.18.0-193.1.2.el8_2 c6227ee import kernel-4.18.0-193.1.2.el8_2 063bbb9 change to centos.pem 05d7b37 Manual CentOS Debranding 9f8b3f1 debrand kernel-4.18.0-193.el8 78ffa6b import kernel-4.18.0-193.el8 89ceb16 Manual CentOS Debranding 47aeda1 debrand kernel-4.18.0-147.8.1.el8_1 e9cecf3 import kernel-4.18.0-147.8.1.el8_1 718e82b Manual CentOS Debranding 223d051 debrand kernel-4.18.0-147.5.1.el8_1 Any help would be appreciated. Thanks, Antal

7 11

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

devel August 2020