[CentOS-devel] Balancing the needs around the RHEL platform

Alexander Bokovoy

abokovoy at redhat.com
Mon Dec 28 12:16:09 UTC 2020


On ma, 28 joulu 2020, redbaronbrowser via CentOS-devel wrote:
>On Monday, December 28, 2020 2:01 AM, Alexander Bokovoy <abokovoy at redhat.com> wrote:
>
>> On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
>>
>> > On 12/27/20 6:27 PM, Alexander Bokovoy wrote:
>> >
>> > > The simplest solution for these use cases is to actually report a bug
>> > > against RHEL and/or CentOS Stream and make sure it is reproducible. This
>> > > would be the quickest way to get the issue backed out or fixed in a
>> > > number of days. There are means to remove broken packages from RHEL
>> > > composes and I hope we'd have a way to propagate those 'removals' to
>> > > CentOS Stream.
>> > > This is something worth raising as a feature request if it doesn't exist
>> > > yet.
>> >
>> > Several people explained how bug reports like that tend end up without
>> > solution and even response while Red hat continues to do what they
>> > want... So I would not hold my breath that community is going to jump
>> > through hoops any more. Majority will just migrate to another clone,
>> > leaving developers to play "find the bug" with Red Hat.
>>
>> I am personally not happy with 'old' CentOS because I couldn't and still
>> cannot influence even a tiny bit of what happens with CentOS after a RHEL
>> release. Multiple times my components were broken in CentOS because they
>> weren't built in the right order and weren't rebuilt for months even
>> after users opened CentOS bugs, with zero transparency into what's
>> happens.
>
>That indicates bugs with Red Hat's SPEC files and assumptions made by
>Red Hat about the behavior of their own build system.  The order in
>which packages are built should be reflected in the BuildRequires
>lines.

The particular issue I am alluding to was an order of rebuilds of two
different components, samba and ipa. idm:DL1 module stream build
happened before samba and used older internal samba ABI, making IPA
server components to not load into a newer compiled smbd process. The
order of builds was right in released RHEL version and wrong in CentOS
8.

Could it be forced by bumping BuildRequires in IPA spec file? Yes, it
should have been. This was not deemed a bug high enough to stop RHEL
release at a late stage since the problem was not in RHEL.

Could CentOS module stream be rebuilt when people reported about the
issue? Most likely but there was no rebuild for months, even after I and
others asked about this publicly and in personal communications. We
offered help. The answer from CentOS maintainers was to wait until next
CentOS release where things will fix themselves due to a rebuild of RHEL
content.

With CentOS Stream it is hopefully will be easier to catch and fix in
RHEL itself, by enforcing the right build requirements as a feedback
coming from CentOS Stream users before we reach a stage in RHEL
development where such bugs would be considered 'not important' to
prevent release schedule from slipping.

>> I am one of those people who can fix the bugs raised against CentOS
>> Stream. I would like to see them raised. My QE teams will certainly
>> treat CentOS Stream bugs similar to RHEL bugs as that improves our
>> ability to deliver RHEL releases with less issues. In the end, you are
>> the ones who get the benefit once the fixes are out there.
>>
>> At least in my area of interest other distributions aren't better than
>> 'old' CentOS or RHEL, there is no magic wand that could help fix issues
>> without them being reported and addressed by engineers involved in
>> upstream projects. If your's only contribution is by raising bugs,
>> please continue doing so, it helps, really.
>
>What fix did we get for Bugzilla #1186352?  Was the lack of Stream the
>reason for fixing the bug being refused?  Now that Stream exists, will
>the bug be reopen and will we get the benefit of a real fix?

The bug you refer is on me and I had no time to work on that. Sadly,
still have no time for it. Any help upstream is welcome. For complex
components like LDAP server or its plugins we don't get many
contributions.

>There are several other bugs I can find that was treated similarly by Red Hat.
>
>I don't expect Red Hat to be able to solve every single bug.  I also
>acknowledge there are several cases when openning a bug has resulted in
>fixes for everyone.
>
>However, the assertion that terminating CentOS 8 before 2022 will help
>fix bugs does not pass the smell test.

I don't see where that assertion can be attributed to me. I only compare
models of contribution in the 'old' CentOS and CentOS Stream. Let's get
it clear: CentOS Stream gives opportunity to contribute into RHEL
development with your use cases and bug fixes well before that is late,
right at the time when RHEL engineers have ability to influence the
changes going into a particular release. Ideally, of course, but if
timing is wrong for a specific release (say, a bug is reported after
RHEL beta milestone when things become much harder to argue for), then
such bug report, bug fix, or a feature request can at least be highlited
for a next development cycle. With 'old' CentOS model you have none of
these possibilities for the next development cycle because it is already
late: at the time CentOS release is out, development window might
already be shut down for the next RHEL release, especially for feature
requests.

>What the CentOS Governance Board has done is put a chilling effect on
>adoption of CentOS 8.  The EOL date of CentOS 7 is now a bonus over
>using CentOS 8.  Why would anyone do that?  The lack of transcript to
>the board meeting makes this move even more troubling.  There still is
>no straight answer about why slowing adoption of CentOS 8 is a good
>thing for the community.
>
>There is also no clear answer right now why fragmenting the CentOS
>community is good for the CentOS community.

I would like to get answers to those questions as well.

>But there is one "magic wand" that can help fix issues without a bug
>report which is to empower the community with the resources that should
>come with being the upstream provider.  Right now the Stream kernel has
>four patches which are all for debranding.  The 4.18 kernel tar.xz file
>in the SRPM is 110% the size of a vanilla tar.xz for 4.18.  What is
>that additional 10%?  How many patches are there?  How many are
>security patches?  How many are bug patches?  How many are feature
>patches?  How many patches were added between releases?  How does a
>CentOS Stream contributor roll back individual patches?
>
>If killing CentOS 8 before 2022 is part of an effort to close the
>openness gap and to better address bugs, then let's /*REALLY*/ do that
>with a kernel SRPM.  It is time to make the patch information fully
>open to assist the community in fixing bugs.
>
>But if the kernel SRPM can be left in the horrific state it currently
>is, then stop hitting us with all this empty posturing about the
>so-called positives we get out of killing CentOS 8 early.

These are good questions too. I have no experience working with RHEL
kernel packaging myself, but looking at the changelogs, I wonder how
many individual patches would be there. One thing I noticed in past is
that using tens of thousands of patches in RPM spec file will amount to
tens of thousands individual invocations of /usr/bin/patch command, each
with its own cost. What is a cumulative overhead of that? Sadly, I was
not able to find any performance investigations for GNU Patch. May be
this can be done as a synthetic test using kernel changesets (5.10
development cycle got 16K of non-merge changesets, for example).

-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland



More information about the CentOS-devel mailing list