[CentOS-devel] Balancing the needs around the CentOS platform

On Sat, Dec 19, 2020 at 6:05 PM Gordon Messmer <gordon.messmer at gmail.com> wrote:
> On 12/19/20 1:25 PM, Mark Mielke wrote:
> > On Sat, Dec 19, 2020 at 4:15 PM Gordon Messmer <gordon.messmer at gmail.com> wrote:
> >> CentOS point releases are more like source control tags than branches.
> >> If you have only used CentOS and not RHEL, then I can see why you might
> >> be confused about this point.  You are accurately describing CentOS.
> > This is false. I don't think you understand the RHEL release process.
> > Check my other post. Red Hat has internal branching that you were
> > unaware of (apparently), and CentOS is using the later branch, not the
> > earlier branch.
> Sure, but this conversation isn't about the specific mechanics of RHEL's
> branching.  Matthew pointed out that minor releases of RHEL are
> branches, and that CentOS releases have no branches.  Nico made an
> inaccurate analogy to VCS tags and branches.  Matthew is correct,
> though: RHEL major releases have branches and CentOS releases do not.
> (Or, they have just one branch.  Whatever phrasing you prefer.)

I want to make sure we are clear about this, and why the mechanics
matter, especially in the context of "CentOS 8 Stream":

CentOS has a branch, which is provided by Red Hat. This is not a
periodic dump of "RHEL Latest", but it is a periodic import of an RHEL
branch. The importance of this is that what you are seeing on the "c7"
branch (in the case of CentOS 7), is a flattened representation from a
more complex branching strategy. Technically you can call this "one
branch", but this is a very superficial view of what is going on, that
glosses over super important details.

RHEL 7.8 is a separate branch from RHEL 7.9. While RHEL 7.8 was the
latest minor release published, Red Hat was periodically importing
content and de-branding from RHEL 7.8 - not from RHEL Latest. Once
RHEL 7.9 was released, the RHEL 7.9 content was imported onto the
CentOS 7 branch, and de-branded.

Let's look at Firefox ESR for an example of this in action:

https://git.centos.org/rpms/firefox/commits/c7

Scroll down to where el7_8 becomes el7_9, and see these two commits:

812ff3 import firefox-68.12.0-1.el7_8
- https://git.centos.org/rpms/firefox/c/812ff34cfe1f0f157d1af7ba2375167451be8233?branch=c7
e98f2c import firefox-78.3.0-1.el7_9
- https://git.centos.org/rpms/firefox/c/e98f2c4cf130b5dd05e5cb7bb86f0a1cccea78cb?branch=c7

In the first commit, see how the changelog has this entry at the top:

 * Thu Aug 20 2020 Jan Horak <jhorak at redhat.com>
 - Update to 68.12.0 build1

Now check the second commit and see how the above lines are deleted
(and one date is even updated, strangely!) and replaced with:

* Fri Sep 18 2020 Jan Horak <jhorak at redhat.com>
- Update to 78.3.0 build1

* Tue Aug 18 2020 Jan Horak <jhorak at redhat.com> - 78.2.0-3
- Update to 78.2.0 build1

* Fri Jul 24 2020 Jan Horak <jhorak at redhat.com>
- Update to 68.11.0 build1

We don't know what CentOS 8 Stream will look like for real - but my
presumption without any evidence to the contrary, would be that:

According to the CentOS 8 model, Firefox ESR 68.x would have been
maintained until Aug 20, 2020 when the final version of 68.12.0 was
released. Then, the first change after this would have been Firefox
ESR 78.3 on Sep 18, 2020.

According to the CentOS 8 Stream model, Firefox ESR 68.x would have
been maintained until Jul 24, 2020 when the 68.11.0 was released.
Then, the first change after this would have been Firefox ESR 78.2 on
Aug 18, 2020.

I picked a relatively tame example just to demonstrate how the
approach changes what happens, and could change the outcome and
experience for the user. But, in this case, of some important note, is
that Firefox ESR has its own "test" / "stabilization" process. When it
was first released:

https://mail.mozilla.org/pipermail/enterprise/2020-July/002326.html

"Firefox ESR 78.0 is the first release from the ESR78 branch, which is
slated to replace ESR68 in September. We recommend that organizations use
this opportunity to test this new version in their environment, ahead of
the ESR68 End Of Life. Firefox ESR 78.0.1 fixed a last-minute issue found
prior to the 78.0 release."

Firefox ESR 78.1 and Firefox ESR 78.2 had a similar caveat, and also
specified that wide deployment would begin with "78.3.0esr in
September":

https://mail.mozilla.org/pipermail/enterprise/2020-August/002462.html

"This is the final scheduled release of the ESR 68.x branch. With the release
of 78.3.0esr in September, we will begin offering automatic updates from
earlier ESR versions to the 78.x branch. We recommend that organizations use
this opportunity to test this new version in their environment ahead of the
ESR68 End Of Life."

As you can see from the above timeline - if CentOS 7 was run according
to the CentOS 8 Stream model, CentOS would have received the *beta*
version of Firefox ESR 78.2, while if it had followed the CentOS 7 and
CentOS 8 model, CentOS would have received the *production* version of
Firefox ESR 78.3.

This is just one of hundreds of packages with problems like this.
CentOS 8 Stream is not "Enterprise Linux". CentOS 8 Stream is "the
first stage of stabilization for Enterprise Linux". It does not seem
appropriate to use for environments that are risk-averse. This had
been clearly stated by many people in Red Hat and outside Red Hat who
do know what they are talking about. It is just that some people are
of the belief that this makes it ok for "95% of users". But, who are
these 95%? Who are these users who are not risk-averse who choose
CentOS as their distribution? I can't stop you from using CentOS 8
Stream in a production environment without building your own
stabilization process to substitute for what you just lost - but I can
warn you, that it sounds like a terrible idea.

-- 
Mark Mielke <mark.mielke at gmail.com>