We have recently been made aware that the internal ABI changes due to
LD_AUDIT fixes cause processes launched concurrently during the update
to crash:
glibc: Upgrading to glibc-2.28-209.el8.x86_64 causes segfaults during
concurrent process launch
<https://bugzilla.redhat.com/show_bug.cgi?id=2119304>
In glibc-2.28-211.el8, we have made the ABI changes more compatible with
*older* versions (before -208), but this might cause yet another bumpy
update from -209 or -210 to -211 or later. (The version ranges are
slightly different for s390x; see the bug. ppc64le has a different
issue caused by RPM file writing order, but no crashes.)
CentOS 9 Stream should be less sensitive to these issues due to the
merged libpthread/libdl.
We are putting some checks into place to catch this earlier. In
general, it might not be possible to avoid such issues completely,
particularly on CentOS 8 Stream. The workaround is to do updates in
quasi-single-user mode, with as little running on the system as
possible.
Thanks,
Florian
As part of OpenStack deployments we deploy RabbitMQ. During current
cycle I looked at moving from CentOS Stream 8 to 9.
And RabbitMQ is a problem.
When I boot CentOS Stream 9 system and then use just built 'rabbitmq'
container memory use of "/usr/lib64/erlang/erts-12.3.2.2/bin/beam.smp"
process goes up to 1.6GB ram:
(rabbitmq)[root@kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rabbit@kolla-cs9...
other_system: 1.6233 gb (68.59%)
allocated_unused: 0.5164 gb (21.82%)
binary: 0.1551 gb (6.55%)
code: 0.0356 gb (1.51%)
other_proc: 0.0184 gb (0.78%)
connection_other: 0.0037 gb (0.16%)
other_ets: 0.0036 gb (0.15%)
queue_procs: 0.0021 gb (0.09%)
plugins: 0.0021 gb (0.09%)
connection_readers: 0.0021 gb (0.09%)
atom: 0.0014 gb (0.06%)
mgmt_db: 0.001 gb (0.04%)
metrics: 8.0e-4 gb (0.03%)
connection_writers: 4.0e-4 gb (0.02%)
connection_channels: 4.0e-4 gb (0.02%)
mnesia: 3.0e-4 gb (0.01%)
msg_index: 0.0 gb (0.0%)
quorum_ets: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)
If I boot the same container on Debian host then same process uses 0.2GB
ram:
(rabbitmq)[root@debian /]# rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rabbit@debian...
binary: 0.2787 gb (70.2%)
code: 0.0355 gb (8.93%)
other_system: 0.0255 gb (6.44%)
other_proc: 0.0209 gb (5.26%)
connection_other: 0.0108 gb (2.72%)
mgmt_db: 0.0073 gb (1.84%)
plugins: 0.0048 gb (1.2%)
other_ets: 0.0037 gb (0.93%)
connection_readers: 0.003 gb (0.75%)
queue_procs: 0.0028 gb (0.7%)
atom: 0.0015 gb (0.37%)
metrics: 0.0011 gb (0.28%)
connection_channels: 8.0e-4 gb (0.21%)
mnesia: 4.0e-4 gb (0.1%)
connection_writers: 2.0e-4 gb (0.05%)
msg_index: 0.0 gb (0.01%)
quorum_ets: 0.0 gb (0.01%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
allocated_unused: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)
So I started checking. Maybe it is a problem of Erlang/RabbitMQ packages
provided by RabbitMQ team?
Booted CS9 system and deployed OpenStack using Debian based containers.
Again 1.6GB memory use.
So let build CS9 based containers using Erlang/RabbitMQ from CentOS
Stream 9 "messaging/rabbitmq-38" repository. Again 1.6GB memory use.
I am wondering what is a reason?
And what is proper solution?
For now I only know not-acceptable solution: deploy OpenStack using
Debian or Ubuntu and forget about CentOS Stream 9 and rest of RHEL 9 family.
======================================================
#centos-meeting: CentOS Cloud SIG meeting (2022-08-11)
======================================================
Meeting started by amoralej at 15:01:25 UTC. The full logs are available
athttps://www.centos.org/minutes/2022/August/centos-meeting.2022-08-11-15.0…
.
Meeting summary
---------------
* OKD/SCOS Update (amoralej, 15:04:38)
* OKD (OpenShift community edition) team is working in a new variant
that runs on CentOS Stream 9 CoreOS (amoralej, 15:07:25)
* OKD team is looking for the best way to provide scos images so that
users can use them to run OpenShift community edition or other
things (amoralej, 15:17:35)
* LINK: https://wiki.centos.org/Events/Dojo/DevConfUS2022
(mcraychee, 15:24:22)
* LINK: http://mirror.stream.centos.org/9-stream/NFV/ (aleskandro,
15:38:19)
* LINK: http://mirror.stream.centos.org/9-stream/NFV/ (amoralej,
15:42:09)
* AGREED: SCOS team will join CloudSIG, welcome! (amoralej, 15:45:02)
* ACTION: someone from the CoreOS team should be designated as
CloudSIG co-chair (amoralej, 15:45:53)
* LINK:
https://pbs.twimg.com/profile_images/1455901978194690048/f1_Z6llN_400x400.j…
(lorbus, 15:55:45)
Meeting ended at 16:00:40 UTC.
Action Items
------------
* someone from the CoreOS team should be designated as CloudSIG co-chair
Action Items, by person
-----------------------
* **UNASSIGNED**
* someone from the CoreOS team should be designated as CloudSIG
co-chair
People Present (lines said)
---------------------------
* amoralej (68)
* lorbus (56)
* travier (23)
* spotz84 (22)
* aleskandro (9)
* mcraychee (9)
* centguard (6)
* davdunc[m (2)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: http://wiki.debian.org/MeetBot
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/).
Week: 15th August - 19th August 2022
If you wish to read this in form of a blog post, check the post on
Fedora community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update---week-33-2022/
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release (mirrors,
mass branching, new namespaces etc.).
The ARC (which is a subset of the team) investigates possible
initiatives that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-17.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Mass update/reboot cycle this week (stg/nonoutage done, outage later
today)
* Freeze for f37 beta starts next week
### CentOS Infra including CentOS CI
* Discussion around the CentOS Stream infra hand over
* New tasks for the CI infra migration
* S3 bucket for the Stream CoreOS effort
* Some infra projects moved from gitea to gitlab
### Release Engineering
* Openh264 composes fo f36,37,38 send to cisco
* Package retirement issues after the branching, thanks to human error
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* Face to face meeting in Boston.
* Penultimate parts of Module process sync. Between el8 and el9.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special
Interest Group that creates, maintains, and manages a high quality set
of additional packages for Enterprise Linux, including, but not limited
to, Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL),
Oracle Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will
never conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror
manager and more.
Updates
-------
* EPEL9 is up to 7339 (+106) packages from 3278 (+71) source packages
* Found the bloaty package was uninstallable because of a libre2 soname
fix, rebuilding it fixed the issue.
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users
to create filters on messages sent to (currently) fedmsg and forward
these as notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Frontend
* Use CoreUI components
* Set up i18n
* Initial version of a “New Rule” page
* Authentication integration FE/BE (ongoing)
* Backend: SQLAlchemy integration (ongoing))
Kindest regards,
CPE Team
Hello everyone,
This is a friendly reminder of the current and upcoming status of CentOS CI
changes (check [1]).
Projects that opted-in for continuing on CentOS CI have been migrated, and
the new Duffy API is available. With that, *phase 0 has been completed*.
Regarding *phase 1*, we are still working on a permanent fix for the DB
Concurrency issues [2]. Also, as for our OpenShift new deployment, we have
a staging environment up and running, and it should be available at
the beginning of September 2022.
In October 2022 we begin *phase 2* when we will work through the following
items (these were also previously communicated in [1]):
- legacy/compatibility API endpoint will handover EC2 instances instead
of local seamicro nodes (VMs vs bare metal)
- bare-metal options will be available through the new API only
- legacy seamicro and aarch64/ThunderX hardware are decommissioned
- only remaining "on-premises" option is ppc64le (local cloud)Feel free
to reach out if you have any questions or concerns
The final deadline for decommissioning the old infrastructure (*phase 3*)
is *December 2022*. We will be communicating further until then, and
meanwhile, reach out to any of us in case you have any questions.
Regards,
[1] [ci-users] Changes on CentOS CI and next steps:
https://lists.centos.org/pipermail/ci-users/2022-June/004547.html
[2] DB Concurrency issues: https://github.com/CentOS/duffy/issues/523
--
Camila Granella
Associate Manager, Software Engineering
Red Hat <https://www.redhat.com/>
@Red Hat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://www.redhat.com/>
Hi folks,
I've uploaded the recording of last week's board meeting:
https://youtu.be/C0Xv33ml3Xo
If you'd like to discuss any issues with the board, we'll have an open
board office hours this Thursday, August 18, at 14:00 UTC.
https://meet.google.com/hsn-sher-use
Email me directly if you need dial-in info instead.
This is the day after the Dojo, so many of us will be at DevConf. But
I'll run the office hours anyway. I just don't know what the turnout
will be.
Thanks,
Shaun
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/).
Week: 8th - 12th August 2022
If you wish to read this in form of a blog post, check the post on Fedora
community blog:
https://communityblog.fedoraproject.org/?p=11316
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-10.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Some great nest talks and discussions, check them out on replay!
* Debugging instability in 32bit arm builders again. ;(
* Rebalanced the s390x builders.
* Business as usual
### CentOS Infra including CentOS CI
* koji/cbs.centos.org upgrade complete
* Kick off for Stream infra handover to I&R
* Adapted ansible baseline role to include workaround for Rsync issue on
CentOS Stream 8
* Enabled RHEL9 Buildroots on cbs/koji for SIGs
### Release Engineering
* Mass branching yesterday (f37 split off rawhide, which is now f38)
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
-------
* Git source moved from git.centos.org to gitlab, for c8s modules.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 is up to 7233 (+106) packages from 3207 (+27) source packages
* “State of EPEL” presentation at Nest conference, recording will be posted
to YouTube at a later date
* EPEL Survey has launched and is available through the end of August
https://tinyurl.com/epelsurvey2022
* epel-release has been improved with a recommends on
dnf-command(config-manager) to ensure crb enabling script can work out of
the box
* KDE Plasma updated from 5.23 to 5.24 (LTS release) in epel8-next and
epel9-next
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users to
create filters on messages sent to (currently) fedmsg and forward these as
notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Unit tests/coverage tests on frontend (Vue.js)
* Auth/OIDC work on both frontend and backend
* Initial backend connection via SQLAlchemy/fastAPI
* Basic functionality of connecting to FASJSON
* CI improvements and fixes
Kindest regards,
CPE Team
C8 has
nodejs-10.24.0-1.module_el8.3.0+717+fa496f1d.x86_64
CS8 has
nodejs-10.23.1-1.module_el8.4.0+645+9ce14ba2
should CentOS Stream 8's package version not be at least == C8?
--
Thanks
Leon
Hi,
We got some monitoring alerts that CentOS Stream 9 build infra is
actually unreachable. We have an internal ticket open at Red Hat to
investigate the issue (seems network related)
If you're directly consuming pkgs from that infra (like
https://kojihub.stream.centos.org) it's actually unreachable.
That means that it also impacts downstream buildsystems, like
https://cbs.centos.org, but only if you asked to consume directly pkgs
from teh CentOS Stream 9 buildroot. For SIGs only building against
content from mirror.stream.centos.org, builds should continue to work.
We'll update this thread when we'll have more information.
--
Fabian Arrotin
The CentOS Project | https://www.centos.org
gpg key: 17F3B7A1 | twitter: @arrfab