[CentOS-devel] Balancing the needs around the CentOS platform

On Sat, Dec 19, 2020 at 12:50 PM Nico Kadel-Garcia <nkadel at gmail.com> wrote:
> On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller <mattdm at mattdm.org> wrote:
> > On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
> > > 2. Minor release milestones to stabilize branches. We have breakage
> > > with most minor release upgrades, and the stabilization process is an
> > > important method of isolating users from being affected by this. This
> > > is why CentOS 8 Stream is being said "for developers", while RHEL 8
> > > would be "for production". It is being said, because it is a real
> > > thing. If you truly believed minor release milestones were unnecessary
> > > for CentOS 8 Stream, then you would also believe that minor release
> > > milestones were unnecessary for RHEL 8.
> > It's important to note that the CentOS Linux rebuild never actually had
> > this. RHEL minor releases are actually branches, and you can stay at a minor
> > release and still get security updates. For CentOS Linux, a minor release is
> No. RHEL minor releases are more like source control "tags" than
> branches. They have stable installation images one can refer and start
> new deployments from, and ongoing work is branched from that tag. This
> has been the case since the earliest Red Hat, not even RHEL, releases
> since I encountered them back in 1995.

Exactly. Matthew: You are pretending as if RHEL beta releases, and
RHEL stabilization doesn't exist for some important reason,

These gates +/- exist between RHEL Stream and RHEL:

1. Changes first go through designer testing and user testing of some
sort. "Try this fix". This is typically limited to only a few users
who reported the problem.

2. Changes were typically made in the "next" branch, which I believe
was often +2 minor releases. This would see the first QA testing, and
the start of some integration testing. Historically this has been an
internal process to Red Hat, which is a problem as it also limits
exposure of the QA testing to Red Hat, and if any behaviours are
changing, it limits the ability for users to be ready for upstream
changes coming down the pipeline. Changes may sit in this
stabilization branch for weeks to months.

3. Changes would backported into the +1 minor release, and get
periodically rolled into a beta program, that would allow the outside
world the first glimpse of what they might be getting, including
changes documented in release notes. Presumedly Red Hat is doing some
form of QA and certification during this period, although this is also
a bit of a black box. These release could be analyzed and if users
were interested in testing, they could go through the special effort
of testing. As the beta program requires some setup to participate in,
many vendors do not. However, of the people that do try out the beta -
presumedly at least some bugs are found and fixed, before they make it
into a release. Changes may sit in this stabilization branch for weeks
to months.

4. Changes are released into an official minor release, Users receive
release notes that they can analyze for impact. Users can and should
run early adopter programs, to ensure that the large drop of changes
will not impact their workflows. Anybody who arbitrarily does "yum
update" at this point is playing with fire. Red Hat QA processes are
insufficient to detect and prevent every problem that might affect
customers. In many cases, the problems are simply missed, and pass
through the above gates without detection. In many other cases, the
problems are impossible to have been predicted by Red Hat, because Red
Hat does not know how people use RHEL in real-life. For example, Red
Hat cannot run all our product builds before release RHEL, so how are
they supposed to be able to detect if our product builds will break?
This is why there are release notes and minor releases, to provide
this level of insulation.

Where exactly, does CentOS 8 Stream fit into this? I'm thinking you
are cutting at 3 + 4 out of the picture, and then some people on this
list are essentially claiming 3 + 4 provides no value for 95% of the
"users" or 95% of the "installed base" (not sure which at this point,
since the numbers are quite different).

Downstream of RHEL, there are additional gates. Including what Nico is
talking about, and what anybody with a significant deployment will
include, which is internal repositories and internal "stable" branches
of RHEL. We want all of the prior branches and gates, but we know that
mistakes still make it through these gates. So, we analyze every
change that comes out of RHEL, and if it might impact our workflows,
we do testing of our workflows before releasing the packages to our
internal systems. In my case we had a "latest" branch which included
content covered by the above gates, a "testing" branch which includes
content that was being tentatively approved, and a "stable" branch
which includes content that was approved.

If the people in favour of abandoning CentOS 8 for CentOS 8 Stream
effectively remove gates 3 + 4 from this process, you introduce
unacceptable risk to our programs, and this will force us to choose
some other solution. "Us" is a really big "us". I think by numbers,
"us" is almost your entire community. If you abandon "us" as is
described, the consequence of this is that "us" will go somewhere
else. "Us" is the reason why CentOS 8 existed in the first place. Red
Hat and the CentOS board can ignore "us", but they can't dictate to
"us". You can't force us to use this CentOS 8 Stream merely by
claiming it is not really different at all.

I see Facebook mentioned several times as evidence that CentOS 8
Stream is no big deal. I imagine that Facebook has their own
stabilization processes, and they are effectively abandoning *RHEL*. I
need to be clear of what I mean by this. In this scenario where CentOS
8 Stream is upstream of RHEL 8, and CentOS 8 is cancelled, the
situation we are left with is that RHEL is deciding when to cut new
CentOS 8 Stream milestones and stabilize them. There is a lot of
emphasis on the Red Hat QA processes, and how they are very important,
but I just explained that many bugs escape the Red Hat QA process, and
we catch them. In effect, the Red Hat milestones are not necessarily
good ones, and this is where Amazon Linux and others have gone, and it
sounds like Facebook as well. The community can pick a different
process for snapshot of CentOS 8 Stream and stabilization. The
community can run its own beta programs separately from RHEL,
effectively making RHEL obsolete. This may be hard to imagine for
individual users who are accustomed to Red Hat controlling everything
they are allowed to see and do, but it's actually a problem for
companies that are being asked to pay $4M USD annual for RHEL
subscription fees for the right to use a collection of free software.
We can do our own testing and stabilization for $4M USD annually. We
don't need Red Hat. And this is the problem with the Red Hat
subscription model, and the problem with CentOS 8 Stream. Red Hat has
priced itself out of essential markets. This isn't about "if you can't
afford RHEL, you can use CentOS 8 Stream". This is about the value
proposition of RHEL. If the value proposition of RHEL is lower than
the value proposition of doing it ourselves, we would be financially
irresponsible to choose RHEL.

If you don't think a few large companies can do as good a job as Red
Hat at branching, and stabilizing - you are mistaken. This is what we
do for *our* products. We know exactly how to do this. We have build
systems and test systems. As described above, we catch problems that
Red Hat does not catch. I didn't really comment on Mike McGraph's
comment that "we are the best", because I don't really want to go
there - but I would be cautious about claiming words like "best".
There is a pretty significant difference between "we are highly
valuable contributors" and "we are the best".

The reason we align on Red Hat, is because it is beneficial for us to
do so. The QA testing and such is useful, for sure - but more useful
is to have a unified "EL" environment where packages can be deployed
in multiple different "EL" distributions with confidence, and vendor
solutions will generally work across all "EL" distributions. If Red
Hat makes this too difficult too achieve, there is an outcome that has
not been described here: Red Hat will become the vendor lock-in
speciality solution that nobody chooses to use.

Based upon several posts in centos-devel, It has become clear to me
that many CentOS board members, CentOS contributors, and Red Hat
staff, do not fully comprehend the value that is brought to the table
by allowing large companies to consolidate on "EL" as a standard. This
is not "EL Next", this is "EL". It is fine for this value to not be
recognized - but when failure to recognize this value has the
consequence of CentOS/RHEL actually *removing* CentOS as the common
standard around which all EL distributions align, we will simply pick
a new standard. The rest of us can work together just fine, and we
don't need Red Hat.

It will just be a terrible shame. None of us wants to see this. We
want Red Hat to be the unifying force. We want to help Red Hat. But,
if Red Hat doesn't want us around any more - it will be more like a
break up with a first girlfriend. Life goes on.

It's up to you, CentOS/RHEL. You basically have these choices:

1. CentOS 8 stays, and "EL" distributions use CentOS 8 as a common baseline.

2. RHEL 8, including minor releases, stays accessible in source form,
so that the community can build a clone that aligns with RHEL 8 for
the purposes of *standards*, not for the purpose of *cheap*.

3. RHEL 8 hides the source behind a legal framework that makes it
difficult or impossible for "bug-for-bug" compatibility to be
maintained. Either the community reverse engineers this (for example,
OpenJDK 8 is kind of like this), or the community creates its own
baselines and ignores RHEL. RHEL becomes irrelevant.

-- 
Mark Mielke <mark.mielke at gmail.com>