[CentOS-devel] Balancing the needs around the CentOS platform

On Mon, Dec 21, 2020 at 4:28 AM Mark Mielke <mark.mielke at gmail.com> wrote:
>
> On Mon, Dec 21, 2020 at 2:28 AM Gordon Messmer <gordon.messmer at gmail.com> wrote:
> > On 12/20/20 9:32 PM, Mark Mielke wrote:
> > > Red Hat never claimed that CentOS Stream will be just as hardened as
> > > CentOS. They never claimed it will get the same process. Doing so
> > > would make RHEL obsolete, so it's not even an intelligent thing for
> > > them to claim. This is more like wishful thinking.
> > I think they have, repeatedly.  That's now I interpret their statements:
> > "no one is rolling in packages here straight from
> > a new Fedora version or from Rawhide.  These changes are point release
> > type changes.  That is the type of changes you see from 8.2 to 8.3 or
> > 7.8 to 7.9 ... not major changes."
> > https://lists.centos.org/pipermail/centos/2020-December/352374.html
>
> Rawhide is a totally different beast. I lived on rawhide for a year
> just to learn from it back around 2005. I did learn a lot. It did hurt
> a lot. :-)
>

These days Rawhide isn't *quite* as painful as it was back in 2005. :)

> > "no, CentOS Stream isn't beta. Packages landing in Stream
> > have already passed QA and gating."
> > https://lists.centos.org/pipermail/centos/2020-December/352383.html
>
> If "Red Hat Enterprise Linux" worth hardening had occurred, we
> wouldn't need CentOS 8 Stream. But, let's just see what it looks like
> for real... I'm genuinely curious as it may help us decode the secret
> sauce, and extrapolate the impacts...
>
> Here is a recently updated package from AppStream:
>
> http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/postgresql-10.15-1.module_el8.4.0+579+22c56897.x86_64.rpm
>
> A new version of PostgreSQL. Great! I'm a fan of PostgreSQL. :-)
>
> According to the date on the mirror directory, it says: 2020-12-15 17:57
>
> That's pretty recent. But, the date it was published shouldn't matter
> - surely there will be quite a lot "QA and gating" that occurred,
> right?
>
> Build Date  : Tue 15 Dec 2020 12:01:55 PM
>
> Hmm... Built on Dec 15, 2020 @ 12:01:55 PM, published on Dec 15, 2020
> @ 17:57 PM. Well, let's check the change log:
>
> * Wed Nov 18 2020 Patrik Novotný <panovotn at redhat.com> - 10.15-1
> - Rebase to upstream release 10.15
>   Resolves: rhbz#1898213
>   Resolves: rhbz#1898341
>   Resolves: rhbz#1901567
>
> This one is interesting. In particular, it makes me wonder what is
> happening behind the scenes. I've seen two seemingly conflicting
> descriptions of what CentOS 8 Stream is:
>
> 1. CentOS 8 Stream is where development happens first, allowing the
> CentOS community to influence what goes into CentOS 8. This would
> imply "LATEST". This suggests that CentOS community is in control of
> the direction of RHEL.
>
> 2. CentOS 8 Stream is the "next" (N+1?) version of RHEL 8. This might
> NOT be "LATEST", but only "LATEST" as shared by Red Hat. Perhaps they
> are still maintaining a private N+2? This says that the CentOS
> community has only limited influence. They are seeing N+1, but not
> seeing N+2.
>
> If the QA was being performed *after* it was built, then this means
> the change was queued from some time after November 18, until December
> 15, before being published within 6 hours (assumes the same time
> zones). Less than 6 hours of post-build testing was performed before
> releasing it to the public.
>
> If the QA was being performed *before* it was built, then this implies
> that the build being published is not the build that Red Hat did QA
> on. The build procedure might be the same, but the actual build in
> CentOS 8 Stream is *not* the original build that was tested. This
> would suggest that a private Red Hat branch exists, possibly the N+2
> branch? And this means that the end-to-end testing occurred some time
> between Nov 18 and Dec 15, on a private Red Hat branch, hidden from
> the CentOS community.
>
> If somebody who actually knows how this works for real could explain,
> it would be enlightening? Which of the above conclusions is the
> closest to being right?
>
> Now, I picked PostgreSQL above as it is a favourite of mine, and it
> happened to be recently updated in AppStream. But, more important to
> everybody else, is the kernel:
>
> http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-257.el8.x86_64.rpm
>
> By the date on the mirror, it seems to say it was published on
> 2020-12-04 00:39. Let's look inside at build date, and the change log:
>
> Build Date  : Thu 03 Dec 2020 05:32:49 PM
>
> This is similar to the PostgreSQL update, as it was built on Dec 3 @
> 5:32:49 PM, and published to the mirror by Dec 4 @ 00:39, around 7
> hours later?
>
> Inside, however, the changelog
>
> * Wed Dec 02 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-257.el8]
>
> Hmmm... The changelog is for Dec 2nd. The build date was Dec 3rd. The
> deploy date was Dec 4th.
>
> Maybe this was an urgent change of some sort, and very tiny - so it
> was easy to QA?
>
> Oh uh....
>
> * Wed Dec 02 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-257.el8]
> - [hv] hv: vmbus: Allow cleanup of VMBUS_CONNECT_CPU if disconnected
> (Mohammed Gamal) [1886096]
> - [hv] hv: vmbus: Add parsing of VMbus interrupt in ACPI DSDT
> (Mohammed Gamal) [1886096]
> .. 167 more lines ...
>
> Wow. That's a pretty big change to QA in less than a day. And what
> about the prior change?
>
> * Mon Nov 30 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-256.el8]
> - [kernel] PM: hibernate: Batch hibernate and resume IO requests
> (Lenny Szubowicz) [1868096]
> - [net] tunnels: Fix off-by-one in lower MTU bounds for ICMP/ICMPv6
> replies (Antoine Tenart) [1895765]
> ... 124 more lines ...
>
> Another big change, and so is the one after that, on Nov 27!
>
> In fact, if you look at the changes since November 2, the changelog
> from 4.18.0-257 to 4.18.0-240 is 6149 lines long.
>
> You just changed my mind!
>
> *YOU* should definitely use CentOS 8 Stream! All of *YOU*. That way,
> by the time you test all of these changes, and report all the bugs, it
> won't affect *ME*. :-) :-) :-)
>
> > Making a reliable distribution doesn't make RHEL obsolete. There's still
> > a lot of value in an OS that is itself semantically versioned, and has
> > paid support contracts.
>
> Yes there is. There is value, because it is important. That's what I'm
> trying to say. :-)
>
> > There's no evidence to support the idea that CentOS Stream will be
> > unreliable because a reliable system would compete with RHEL. And lots
> > of evidence to the contrary, including the extent to which Red Hat
> > publishes their work in a form usable to rebuild the distribution.
>
> "Unreliable" and "reliable" are extremes. RHEL 8 isn't 100% reliable,
> and CentOS 8 Stream isn't 100% unreliable.
>
> However, RHEL 8 will be a lot more reliable if *YOU* test CentOS 8
> Stream on *all* of your systems. The more, the better.
>

Judging by this response, I assume you're not familiar with how
distribution testing typically works. At least for the past ten years,
distribution testing of packages as they're updated has been almost
entirely automated. Rather than spending ridiculous amounts of time
having *people* test it, the gates are enforced by scripts, Ansible
playbooks, etc. that execute workflows to validate behavior of a
package. In the last couple of years, Red Hat has been rewriting their
test suite for public consumption and pushing them into Fedora, so
that packages can be validated at the earliest stage with Fedora ELN.

But let's focus on RHEL/CentOS for the moment. At the RHEL/CentOS
Stream level, stuff is pushed to the RHEL internal Git server *first*.
Their internal Git server is wired up with checks to run on commits
being pushed so that tests run on the commit *before* it lands. Once
it passes *those* tests, the commit lands and the package is built in
"Brew" (the RHEL Koji instance). Once the package is built there, a
series of integration and system validation tests are run before the
build is accepted for release. Once *that* happens, it is pushed to
the CentOS Git server (git.centos.org), and then the CentOS Stream
build is kicked off. Once that build is made, it is then run through
the CentOS test suite and validated based on that before pushing it
out for release to CentOS Stream.

With the kernel in particular, the test suite probably validated it
for release in less than half a day on the RHEL side. It was then
pushed on December 3[1], Carl debranded it shortly after[2] and built
on December 4[3], and released later that day after it passed the
CentOS functional tests.

As someone who has also designed such systems in the past, I'd say
that's actually pretty good. There's further work to be done, for
sure, but there always is. It's all about continuous improvement.
That's why I made suggestions about bringing ELRepo and other open
source kernel modules into the CentOS Project as SIGs so that we can
*add* to the kernel automated testing and prevent broken expectations
with things like the kernel ABI that Red Hat maintains for the RHEL
kernel[4][5]. Because if you look at CentOS Stream as the gift and
opportunity that it truly is, then there are amazing things we can
accomplish together in the CentOS Project.


[1]: https://git.centos.org/rpms/kernel/c/cf5e7d39f59af389825122cf70ca8ad664364eca?branch=c8s
[2]: https://git.centos.org/rpms/kernel/c/1af03967e6580e55d84053c181555449e58da90a?branch=c8s
[3]: https://koji.mbox.centos.org/koji/buildinfo?buildID=14937
[4]: https://lists.centos.org/pipermail/centos-devel/2020-December/075593.html
[5]: https://lists.centos.org/pipermail/centos-devel/2020-December/075615.html



--
真実はいつも一つ！/ Always, there's only one truth!