[CentOS-devel] Balancing the needs around the CentOS platform

On Sat, Dec 19, 2020 at 6:13 PM Mark Mielke <mark.mielke at gmail.com> wrote:
> The problem with this conclusion, is that you are ignoring the reality
> of approximately (in the example of something reaching RHEL X.Y):
>
> 1) RHEL Master (days?) -> RHEL X.N+2 Staging  (weeks to months)-> RHEL
> X.N+1 Staging or Beta (weeks to months) -> RHEL X.N Stable (weeks to
> months) -> CentOS X.N Stable (weeks to months)
>
> And the claim of several people on this is that the following is "no big deal":
>
> 1) RHEL Master (days?) -> CentOS X Stream (days?)
>
> So you can say that CentOS only has one branch - but that is only if
> you ignore what processes and gates existed prior to that branch. I
> think the conclusion that CentOS 7 was just one branch is misleading
> at the very least, and false in the only picture that matters.

I presumed everybody understood this, but perhaps I was in an error.
In order to ensure we are all on the same page here, let's consider a
real-life scenario.

https://access.redhat.com/articles/3078

This shows:

RHEL 8.3 - 4.18.0-240
RHEL 8.2 - 4.18.0-193
RHEL 8.1 - 4.18.0-147

Let's pick RHEL 8.2 as an example. Once the release was branched, it
enters a stabilization and testing period ("staging" period where it
remains private, and then a "beta" period where it might be formally
announced and made available to the public). During this time -
upstream has already moved on with -194, -195, -196, ... eventually
reached -240 for RHEL 8.3.

If any problems are encountered during this stabilization period or
during the life of the minor release, or - if any high priority bug
fix, such as a security fix becamse available upstream and are
important enough to backport, then *only* the changes necessary will
be backported. In the RHEL 8.2 case, this would become 4.18.0-193.x.

In the case of CentOS 8.2, this included:

https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.el8.src.rpm
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.1.2.el8_2.src.rpm
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.6.3.el8_2.src.rpm
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.14.2.el8_2.src.rpm
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.19.1.el8_2.src.rpm
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.28.1.el8_2.src.rpm

Upgrades within the minor release, such as .14.2 -> .19.1, would be
considered "safer", because rather than picking up *all* the upstream
changes - only the changes that were considered important enough, and
safe enough, to backport to the RHEL 8.2 stream, were made available
in RHEL 8.2, and then made available in CentOS 8.2.

With CentOS 8 Stream, none of the above happens any more. Instead, you
will get -241, -242, -243, -244, ... there is no stable version. There
is no baseline to align against or certify against. There is only
"LATEST". Your choice is to pick up "LATEST", or stay behind. No
security patches. No important bug fixes. You must stay exactly where
you are until *all* of your vendors provide support for the new
kernel.

In many cases, such as user space API, the public interfaces make an
attempt to be forwards and backwards compatible. However, in many
cases - this fails. If I build something for EL 8.2, it will
*probably* work in EL 8.3. However, if I build something in EL 8.3,
there is no guarantee it will work in EL 8.2. The kernel is
particularly problematic, because the kernel API is not covered by the
same compatibility criteria, and things like EL 7.5 can introduced
updates to the VFS layer to support OverlayFS, and then break any
systems that hook into the VFS layer, such as in our case for RHEL 7.5
- IBM Rational ClearCase. Consequently, we need to wait for IBM to
produce a version of ClearCase that supports RHEL 7.5, which is only
normally expected 90 days after RHEL 7.5 is released.

This same thing happens all over EL. It's actually difficult for me to
comprehend that people think this is no big deal, but I understand
that for me this is a regular problem, and for many of the people
posting, it seems you may not be fully aware. But, I think I'm
describing a problem that is known to a majority of the users of EL,
but perhaps not to the developers of CentOS, and this may be the
fundamental problem of how everything went wrong.

-- 
Mark Mielke <mark.mielke at gmail.com>