[CentOS-devel] Balancing the needs around the CentOS platform

On Mon, Dec 21, 2020 at 2:28 AM Gordon Messmer <gordon.messmer at gmail.com> wrote:
> On 12/20/20 9:32 PM, Mark Mielke wrote:
> > Red Hat never claimed that CentOS Stream will be just as hardened as
> > CentOS. They never claimed it will get the same process. Doing so
> > would make RHEL obsolete, so it's not even an intelligent thing for
> > them to claim. This is more like wishful thinking.
> I think they have, repeatedly.  That's now I interpret their statements:
> "no one is rolling in packages here straight from
> a new Fedora version or from Rawhide.  These changes are point release
> type changes.  That is the type of changes you see from 8.2 to 8.3 or
> 7.8 to 7.9 ... not major changes."
> https://lists.centos.org/pipermail/centos/2020-December/352374.html

Rawhide is a totally different beast. I lived on rawhide for a year
just to learn from it back around 2005. I did learn a lot. It did hurt
a lot. :-)

> "no, CentOS Stream isn't beta. Packages landing in Stream
> have already passed QA and gating."
> https://lists.centos.org/pipermail/centos/2020-December/352383.html

If "Red Hat Enterprise Linux" worth hardening had occurred, we
wouldn't need CentOS 8 Stream. But, let's just see what it looks like
for real... I'm genuinely curious as it may help us decode the secret
sauce, and extrapolate the impacts...

Here is a recently updated package from AppStream:

http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/postgresql-10.15-1.module_el8.4.0+579+22c56897.x86_64.rpm

A new version of PostgreSQL. Great! I'm a fan of PostgreSQL. :-)

According to the date on the mirror directory, it says: 2020-12-15 17:57

That's pretty recent. But, the date it was published shouldn't matter
- surely there will be quite a lot "QA and gating" that occurred,
right?

Build Date  : Tue 15 Dec 2020 12:01:55 PM

Hmm... Built on Dec 15, 2020 @ 12:01:55 PM, published on Dec 15, 2020
@ 17:57 PM. Well, let's check the change log:

* Wed Nov 18 2020 Patrik Novotný <panovotn at redhat.com> - 10.15-1
- Rebase to upstream release 10.15
  Resolves: rhbz#1898213
  Resolves: rhbz#1898341
  Resolves: rhbz#1901567

This one is interesting. In particular, it makes me wonder what is
happening behind the scenes. I've seen two seemingly conflicting
descriptions of what CentOS 8 Stream is:

1. CentOS 8 Stream is where development happens first, allowing the
CentOS community to influence what goes into CentOS 8. This would
imply "LATEST". This suggests that CentOS community is in control of
the direction of RHEL.

2. CentOS 8 Stream is the "next" (N+1?) version of RHEL 8. This might
NOT be "LATEST", but only "LATEST" as shared by Red Hat. Perhaps they
are still maintaining a private N+2? This says that the CentOS
community has only limited influence. They are seeing N+1, but not
seeing N+2.

If the QA was being performed *after* it was built, then this means
the change was queued from some time after November 18, until December
15, before being published within 6 hours (assumes the same time
zones). Less than 6 hours of post-build testing was performed before
releasing it to the public.

If the QA was being performed *before* it was built, then this implies
that the build being published is not the build that Red Hat did QA
on. The build procedure might be the same, but the actual build in
CentOS 8 Stream is *not* the original build that was tested. This
would suggest that a private Red Hat branch exists, possibly the N+2
branch? And this means that the end-to-end testing occurred some time
between Nov 18 and Dec 15, on a private Red Hat branch, hidden from
the CentOS community.

If somebody who actually knows how this works for real could explain,
it would be enlightening? Which of the above conclusions is the
closest to being right?

Now, I picked PostgreSQL above as it is a favourite of mine, and it
happened to be recently updated in AppStream. But, more important to
everybody else, is the kernel:

http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-257.el8.x86_64.rpm

By the date on the mirror, it seems to say it was published on
2020-12-04 00:39. Let's look inside at build date, and the change log:

Build Date  : Thu 03 Dec 2020 05:32:49 PM

This is similar to the PostgreSQL update, as it was built on Dec 3 @
5:32:49 PM, and published to the mirror by Dec 4 @ 00:39, around 7
hours later?

Inside, however, the changelog

* Wed Dec 02 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-257.el8]

Hmmm... The changelog is for Dec 2nd. The build date was Dec 3rd. The
deploy date was Dec 4th.

Maybe this was an urgent change of some sort, and very tiny - so it
was easy to QA?

Oh uh....

* Wed Dec 02 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-257.el8]
- [hv] hv: vmbus: Allow cleanup of VMBUS_CONNECT_CPU if disconnected
(Mohammed Gamal) [1886096]
- [hv] hv: vmbus: Add parsing of VMbus interrupt in ACPI DSDT
(Mohammed Gamal) [1886096]
.. 167 more lines ...

Wow. That's a pretty big change to QA in less than a day. And what
about the prior change?

* Mon Nov 30 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-256.el8]
- [kernel] PM: hibernate: Batch hibernate and resume IO requests
(Lenny Szubowicz) [1868096]
- [net] tunnels: Fix off-by-one in lower MTU bounds for ICMP/ICMPv6
replies (Antoine Tenart) [1895765]
... 124 more lines ...

Another big change, and so is the one after that, on Nov 27!

In fact, if you look at the changes since November 2, the changelog
from 4.18.0-257 to 4.18.0-240 is 6149 lines long.

You just changed my mind!

*YOU* should definitely use CentOS 8 Stream! All of *YOU*. That way,
by the time you test all of these changes, and report all the bugs, it
won't affect *ME*. :-) :-) :-)

> Making a reliable distribution doesn't make RHEL obsolete. There's still
> a lot of value in an OS that is itself semantically versioned, and has
> paid support contracts.

Yes there is. There is value, because it is important. That's what I'm
trying to say. :-)

> There's no evidence to support the idea that CentOS Stream will be
> unreliable because a reliable system would compete with RHEL. And lots
> of evidence to the contrary, including the extent to which Red Hat
> publishes their work in a form usable to rebuild the distribution.

"Unreliable" and "reliable" are extremes. RHEL 8 isn't 100% reliable,
and CentOS 8 Stream isn't 100% unreliable.

However, RHEL 8 will be a lot more reliable if *YOU* test CentOS 8
Stream on *all* of your systems. The more, the better.

Thank you!

-- 
Mark Mielke <mark.mielke at gmail.com>