Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/)
Week: 20th - 24th June 2022
If you wish to read this in form of a blog post, check the post on the
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update-week-25-2022/
Highlights of the week Infrastructure & Release Engineering Goal of this
Initiative
The purpose of this team is to take care of day-to-day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-06-22.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update Fedora Infra
- Most apps have moved over to the OpenShift4 cluster. Hopefully, the
transition should be finishing up this week.
- Wiki: All upgraded in production and working (thanks Ryan!)
- Resultsdb: All moved over to OpenShift 4 in prod and working (thanks
Leo!)
- Business proceeding as usual
CentOS Infra including CentOS CI
- Kerberos settings switch for git.centos.org (kcm on el8 vs keyring on
el7) for lookaside upload CGI <https://pagure.io/centos-infra/issue/811>
[1]
- Issue on iad2 hosted reference mirror
<https://pagure.io/centos-infra/issue/812>[2] (epel.next and
mirrormanager), all fixed now
- Duffy CI ongoing tasks and deployments (all announced)
- Equinix nodes migration <https://pagure.io/centos-infra/issue/816>[3]
(on their request)
- Business proceeding as usual
Release Engineering
- Compose-tracker updated to f36 in staging, production happening
tomorrow
- Python 3.11 merged to rawhide
- MBS randomly fails to process builds
- Rawhide compose failures recently (syslinux retirement, then python
3.11 merge) all fixed now
- Business proceeding as usual
CentOS Stream Goal of this Initiative
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
- CentOS Stream 8: Manually keeping regular RPMs and module RPMs updated
on the koji.stream server as current updates are composed and released.
CentOS Duffy CI Goal of this Initiative
Duffy is a system within CentOS CI infrastructure allowing tenants to
provision and access machines (physical and/or virtual, of different
architectures and configurations) for the purposes of CI testing.
Development of Duffy is largely finished, we're currently planning and
testing deployment scenarios.
Updates
- Release version 3.2.1
- Docs, docs, docs and a Dojo
Package Automation (Packit Service) Goal of this initiative
Automate RPM packaging of infra apps/packages
Updates
- Mostly business as usual
- Thanks again to all who are reviewing our PRs
- Most of our GitHub critical apps are enabled now or close to being
enabled
Flask-oidc: oauth2client replacement Goal of this initiative
Flask-oidc is a library used across the Fedora infrastructure and is the
client for ipsilon for its authentication. flask-oidc uses oauth2client.
This library is now deprecated and no longer maintained. This will need to
be replaced with authlib.
Updates:
- POC working using authlib, tidying up code to prepare to submit a PR
back to upstream
EPEL Goal of this initiative
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high-quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including a build system, Bugzilla instance, updates manager, mirror
manager and more.
Updates
- This week we have 6442 (+127) packages, from 2882 (+76) source
packages
- Containerd and puppet retired from EPEL7 because of upstream EOL and
multiple CVEs.
- Caddy was updated, fixing 4 CVEs in EPEL9
*Index*
[1] https://pagure.io/centos-infra/issue/811
[2] https://pagure.io/centos-infra/issue/812
[3] https://pagure.io/centos-infra/issue/816
Thanks and Regards,
Akashdeep Dhar (he/him),
Objective Representative, Fedora Council
Red Hat Community Platform Engineering
t0xic0der(a)fedoraproject.org
akashdeep(a)redhat.com
June 2022 Quarterly report submitted by: Jefro Osier-Mixon, Red Hat -
acting chair
_____________________________________
Membership update
This SIG does not have a formal membership process. The mailing list
currently has 94 subscribers representing at least 30 organizations, though
not all subscribers use corporate emails and some are participating as
individuals.
_____________________________________
Releases in the most recent quarter (or most recent release, if none in
that quarter)
The SIG provides a new distribution: Automotive Stream Distribution
(AutoSD), a CentOS Stream derivative designed specifically around the needs
of an automotive OS, and transparently the upstream project for Red Hat's
eventual in-vehicle OS product. AutoSD has been downloaded and used by
several organizations who have commented or asked for help, so we know it
is getting some traction though of course we don't have exact metrics on
usage.
_____________________________________
Health report and general activity narrative.
The SIG has had two public meetings per month, one formal and one informal
"office hours", each with 25-40 attendees, with visible participation from
7-10 separate organizations. This SIG is intended to be a community effort
with contributions and shared benefits from all participants. All formal
meetings are recorded and posted on this page:
https://wiki.centos.org/SpecialInterestGroup/Automotive/Meetings
Several Red Hat employees made the initial contributions to the project as
well as the infrastructure required to build and test it. We now occupy a
gitlab repository building software regularly using CI, with build
instructions provided on the documentation page at
https://sigs.centos.org/automotive/ . Sample images are present and
downloadable along with customization and build instructions.
This is a high-level summary of current activity:
- All work has migrating to a CentOS namespace within GitLab in order to
cement our intention as a community project:
https://gitlab.com/centos/automotive
- CI/CD infrastructure is in place
- Firmware size has been reduced
- Encryption is available via LUKS
- a Linux chroot is being worked on for Android devices
- manifests now enable EFI runtime services
- Lookaside cache structure
- Downloadable images are available: https://autosd.sig.centos.org/
- Documentation has been greatly expanded with both community and corporate
contributions: https://sigs.centos.org/automotive/
- Documentation now includes detailed contribution guidelines:
https://sigs.centos.org/automotive/contributing/contributing-to-auto-sig-re…
- Ongoing discussions within the meetings have centered around supported
hardware and expectations for documentation
- A new meetings page contains recordings of all meetings:
https://wiki.centos.org/SpecialInterestGroup/Automotive/Meetings
- Synchronous communications have moved from IRC to Matrix thanks to some
help from Fedora:
https://app.element.io/#/room/#centos-automotive-sig:fedoraproject.org
_____________________________________
Issues for the board to address, if any
None, keep up the excellent work :)
Jeffrey "Jefro" Osier-Mixon | jefro(a)redhat.com
Red Hat Office of the CTO | Sr. Principal Community Architect, Automotive
Thanks to everybody that attended our online Summer Dojo. We are
holding an in-person Dojo on August 17 at Boston University. This is
the day before the main track of DevConf.US, at the same venue.
https://wiki.centos.org/Events/Dojo/DevConfUS2022
The call for presentations is open. We look forward to talks from the
entire Enterprise Linux ecosystem, from how it's built to how it's
used. We have both 20-minute and 40-minute talk slots.
https://forms.gle/QjGki4whyWTvnD4q9
(If you previously tried to submit a presentation and didn't have
permission, it's because I forgot to change the settings to allow
submissions from outside Red Hat. Sorry. It's fixed now.)
Thanks,
Shaun McCance
CentOS Community Architect
Red Hat Open Source Program Office
Hi everyone,
Is there some sort of specfile tarball or other method to search through all
the CentOS Stream specfiles for a regex? In Fedora, we have a tarball[1] with
all the specfiles that's generated daily, as well as access to Sourcegraph[2].
Is there an equivalent for CentOS Stream/RHEL? Of course, I could manually
extract them from the *-source yum repos' SRPMs or distgit, but that would be
tedious.
[1]: https://src.fedoraproject.org/lookaside/git-seed-latest.tar.xz
[2]: https://sourcegraph.com/search?
q=context:global+r:src.fedoraproject.org+file:.*%5C.spec%24+
%5EName:&patternType=regexp&case=yes
--
Thanks,
Maxwell G (@gotmax23)
Pronouns: He/Him/His
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/)
Week: 13th - 17th June 2022
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update--week-24-2022/
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release (mirrors,
mass branching, new namespaces etc.).
The ARC (which is a subset of the team) investigates possible
initiatives that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-06-15.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Resultsdb almost moved to ocp4 in prod, just a few parts to finish
(Thanks Leo!)
* Ocp4 cluster now on our vpn, so all proxies can reach apps (thanks
darknao!)
* Wiki upgrade looking good in staging, prod to come (thanks ryan!)
* Some more vm’s to f36
* About 50% done moving apps to ocp4.
* Image builder prod move blocked due to firewall issues
### CentOS Infra including CentOS CI
* https://git.centos.org went live last monday
* Trying to resume discussion with RH IT for Stream storage migration
* New CI infra deployment tasks to be ready
* https://lists.centos.org/pipermail/ci-users/2022-June/004547.html
### Release Engineering
* ELN composes were broken over the weekend because of ODCS backend /
front end version mismatch
* Nodejs-sig removed as the default assignee on a bunch of components in BZ
* we have discovered workflow in bodhi that locks update in a weird
state more info https://github.com/fedora-infra/bodhi/issues/4566
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* We imported all RPMs for modules (CentOS Stream 8) to the shared
buildsystem
* All sources imported to GitLab (CentOS Stream 8)
## CentOS Duffy CI
Goal of this Initiative
-----------------------
Duffy is a system within CentOS CI infrastructure allowing tenants to
provision and access machines (physical and/or virtual, of different
architectures and configurations) for the purposes of CI testing.
Development of Duffy is largely finished, we're currently planning and
testing deployment scenarios.
Updates
-------
* Test and polish `duffy client …` experience
* Docs and CentOS Dojo talk prep
## Package Automation (Packit Service)
Goal of this initiative
-----------------------
Automate RPM packaging of infra apps/packages
Updates
-------
* Almost finished, only mirrormanager2 remaining from our critical apps
on Github
* Couple of outliers (fasjson, flask-mod-auth) need downstream repos created
* Datanommer.models manually packaged so datagrepper can be automated
* Noggin now fully automated
## Flask-oidc: oauth2client replacement
Goal of this initiative
-----------------------
Flask-oidc is a library used across the Fedora infrastructure and is the
client for ipsilon for its authentication. flask-oidc uses oauth2client.
This library is now deprecated and no longer maintained. This will need
to be replaced with authlib.
Updates:
--------
* Working [poc
app](https://app-flask-oidc-dev.apps.ocp.stg.fedoraproject.org/oidc/)
which authenticates against noggin/ipa using authlib and OIDC.
* Working on an upstream PR with the working code now.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* This week we have 6315 (+297) packages, from 2806 (+76) source packages
* [XFCE now available in
epel9-testing](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-5fe…
* Removed epel9 packages that were included in rhel9.0
* oniguruma: [backported
fix](https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-a9236c0113)
for CVE-2019-13225 (moderate) from rhel8 to epel7
* drbd: pull request to [switch pacemaker dependency from requires to
recommends](https://src.fedoraproject.org/rpms/drbd/pull-request/2) to
comply with policy
Kindest regards,
CPE Team
Hello,
We are waiting on a change made to libselinux [1] to hit the 9-stream
mirrors as described in [2].
It's rather unclear to me how this happens? I have read [3] but
unfortunately I'm still not groking it.
I know this build has been committed to gitlib, and koji has picked it
up and built it [4]. It has tags
* c9s-build-side-477-stack-gate
* c9s-candidate
* c9s-pending
which [3] sort-of explains -- the package build has passed testing?
>From what I understand, it then goes into a compose visible from [5].
Not sure what "odcs" stands for, but these seem to be daily builds
(?-daily-centos-stream maybe?).
Then there is production/ which has timestamped directories currently:
CentOS-Stream-9-20220531.0/
CentOS-Stream-9-20220606.0/
CentOS-Stream-9-20220607.0/
CentOS-Stream-9-20220613.0/
latest-CentOS-Stream/
Are these production composes weekly, daily, or perhaps ad-hoc
depending on somebody flipping a switch?
Something suggests to me that "latest-CentOS-Stream" should be ~= to
the mirror repos [6]. Is that right?
If I look in CentOS-STream-9-20220613.0 (latest-CentOS-stream,
currently), it indeed has libselinux-3.4-2 that we want. However, if
I sort the mirror by date [7] nothing seems to have updated since
2022-06-06? So, if I'm on the right track here -- what promotes the
"production" compose to the public mirrors?
Thanks for any insight
-i
[1] https://gitlab.com/redhat/centos-stream/rpms/libselinux/-/commit/c68e490c94…
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2094683
[3] https://docs.centos.org/en-US/stream-contrib/quickstart/
[4] https://kojihub.stream.centos.org/koji/buildinfo?buildID=21551
[5] https://composes.stream.centos.org/
[6] http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/
[7] http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/?C=M;O=D
Hi guys.
I filed a bug report with RH's Bugzilla but nothing happened
there thus thought I'd let - if SIGs read this list - you
guys know that Samba 4.16 is broken, hard-crashes.
For better picture - Samba is managed by IPA
-> $ systemctl restart smb.service
Starting Samba SMB Daemon...
[2022/06/08 19:17:32.937351, 0, pid=49122]
../../source3/smbd/server.c:1741(main)
smbd version 4.16.1 started.
Copyright Andrew Tridgell and the Samba Team 1992-2022
[2022/06/08 19:17:32.960756, 0, pid=49122]
../../lib/util/fault.c:172(smb_panic_log)
===============================================================
[2022/06/08 19:17:32.960925, 0, pid=49122]
../../lib/util/fault.c:176(smb_panic_log)
INTERNAL ERROR: Signal 6: Aborted in pid 49122 (4.16.1)
[2022/06/08 19:17:32.960981, 0, pid=49122]
../../lib/util/fault.c:181(smb_panic_log)
If you are running a recent Samba version, and if you
think this problem is not yet fixed in the latest versions,
please consider reporting this bug, see
https://wiki.samba.org/index.php/Bug_Reporting
[2022/06/08 19:17:32.961022, 0, pid=49122]
../../lib/util/fault.c:182(smb_panic_log)
===============================================================
[2022/06/08 19:17:32.961056, 0, pid=49122]
../../lib/util/fault.c:184(smb_panic_log)
PANIC (pid 49122): Signal 6: Aborted in 4.16.1
[2022/06/08 19:17:32.961647, 0, pid=49122]
../../lib/util/fault.c:288(log_stack_trace)
BACKTRACE: 15 stack frames:
#0 /lib64/libsamba-util.so.0(log_stack_trace+0x34)
[0x7f41bbc0c7c4]
#1 /lib64/libsamba-util.so.0(smb_panic+0xd)
[0x7f41bbc0ca1d]
#2 /lib64/libsamba-util.so.0(+0xeab9) [0x7f41bbc0cab9]
#3 /lib64/libpthread.so.0(+0x12ce0) [0x7f41bb83bce0]
#4 /lib64/libc.so.6(gsignal+0x10f) [0x7f41bb2a4a4f]
#5 /lib64/libc.so.6(abort+0x127) [0x7f41bb277db5]
#6 /lib64/libtalloc.so.2(+0x2f40) [0x7f41a34a7f40]
#7 /lib64/libtalloc.so.2(+0x336c) [0x7f41a34a836c]
#8 /usr/lib64/samba/pdb/ipasam.so(+0x11c2f)
[0x7f41a38e7c2f]
#9
/lib64/libsamba-passdb.so.0(make_pdb_method_name+0xb2)
[0x7f41bbdae2f2]
#10 /lib64/libsamba-passdb.so.0(+0x1d604) [0x7f41bbdae604]
#11
/lib64/libsamba-passdb.so.0(initialize_password_db+0x1d)
[0x7f41bbdb096d]
#12 /usr/sbin/smbd(main+0x5d9) [0x55c945bbcf09]
#13 /lib64/libc.so.6(__libc_start_main+0xf3)
[0x7f41bb290ca3]
#14 /usr/sbin/smbd(_start+0x2e) [0x55c945bbe6de]
[2022/06/08 19:17:32.962008, 0, pid=49122]
../../source3/lib/dumpcore.c:318(dump_core)
coredump is handled by helper binary specified at
/proc/sys/kernel/core_pattern
smb.service: Main process exited, code=dumped, status=6/ABRT
smb.service: Failed with result 'core-dump'.
Failed to start Samba SMB Daemon.
many thanks, L.
Hi folks,
I've uploaded the recording of last week's board meeting:
https://youtu.be/qPluJM12AhQ
If you'd like to discuss any issues with the board, we'll have an open
board office hours this Thursday, June 16, at 14:00 UTC.
https://meet.google.com/ppx-hjdj-zrc
Email me directly if you need dial-in info instead.
Thanks,
Shaun
Due to a scheduled pagure upgrade, we'll have to move the existing
Pagure instance (aka https://git.centos.org) to a new node.
Migration is scheduled for """"Monday June 13rd, 7:00 am UTC time"""".
You can convert to local time with $(date -d '2022-06-13 07:00 UTC')
The expected "downtime" is estimated to ~60 minutes , time needed to :
- backup/restore last DB dump
- import and convert DB schema for newer pagure version
- last data (sources in lookaside cache and git repositories) sync
- verify service and switch public IP to new host.
Important note wrt that migration is that we're moving from pagure 5.8
running on CentOS 7 to pagure 5.13 on RHEL 8.
It means it's a new host and so sshd host keys will change (we didn't
want to import older host keys to comply with newer algo)
The new fingerprint will be displayed at usual place
(https://git.centos.org/ssh_info) but it only matters for SIGs users
pushing to specific projects/branches over ssh.
For people pulling through https, nothing changes.
Here are in advance the new fingerprints :
rsa=3072 SHA256:qeSehpwh3X7HI0D/jF7N4qZcergdr9tUCdaZ2EIdiLc (RSA)
rsa_md5=3072 MD5:a9:a1:ba:83:96:71:28:ca:86:19:c0:5d:4f:48:9f:63 (RSA)
ecdsa=256 SHA256:vIRsg5g/t/7ucYP4NKTkcPJdE7CWbFQVInscthHKihU (ECDSA)
ecdsa_md5=256 MD5:8f:40:35:4f:b9:43:60:d9:09:c0:5f:80:52:69:c8:8d (ECDSA)
Also worth knowing that we also present a signed cert for sshd host
keys, so if you already trust our CA
(https://github.com/CentOS/ansible-role-sshd/blob/master/defaults/main.yml#L…)
in your ~/.ssh/known_hosts file, you'll not even have to accept new key
Thanks for your understanding and patience.
on behalf of the Infra team,
--
Fabian Arrotin
The CentOS Project | https://www.centos.org
gpg key: 17F3B7A1 | twitter: @arrfab
Hi all,
In case you missed it, we are having a free online Dojo next week on
Friday, June 17. The schedule is live. It is free, but registration is
required.
https://wiki.centos.org/Events/Dojo/Summer2022
But wait, there's more! We've just announced an in-person Dojo to be
held at Boston University on August 17. This is the day before the main
sessions of DevConf.US. The CFP is open. The Dojo is free, as always,
but we recommend and appreciate registrations to help us plan.
https://wiki.centos.org/Events/Dojo/DevConfUS2022
I hope to see you all online next week, and I'm looking forward to
finally seeing some of you in person in two months.
Thanks,
Shaun McCance
CentOS Community Architect
Red Hat Open Source Program Office