On Mon, Dec 21, 2020 at 2:28 AM Gordon Messmer <gordon.messmer at gmail.com> wrote: > On 12/20/20 9:32 PM, Mark Mielke wrote: > > Red Hat never claimed that CentOS Stream will be just as hardened as > > CentOS. They never claimed it will get the same process. Doing so > > would make RHEL obsolete, so it's not even an intelligent thing for > > them to claim. This is more like wishful thinking. > I think they have, repeatedly. That's now I interpret their statements: > "no one is rolling in packages here straight from > a new Fedora version or from Rawhide. These changes are point release > type changes. That is the type of changes you see from 8.2 to 8.3 or > 7.8 to 7.9 ... not major changes." > https://lists.centos.org/pipermail/centos/2020-December/352374.html Rawhide is a totally different beast. I lived on rawhide for a year just to learn from it back around 2005. I did learn a lot. It did hurt a lot. :-) > "no, CentOS Stream isn't beta. Packages landing in Stream > have already passed QA and gating." > https://lists.centos.org/pipermail/centos/2020-December/352383.html If "Red Hat Enterprise Linux" worth hardening had occurred, we wouldn't need CentOS 8 Stream. But, let's just see what it looks like for real... I'm genuinely curious as it may help us decode the secret sauce, and extrapolate the impacts... Here is a recently updated package from AppStream: http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/postgresql-10.15-1.module_el8.4.0+579+22c56897.x86_64.rpm A new version of PostgreSQL. Great! I'm a fan of PostgreSQL. :-) According to the date on the mirror directory, it says: 2020-12-15 17:57 That's pretty recent. But, the date it was published shouldn't matter - surely there will be quite a lot "QA and gating" that occurred, right? Build Date : Tue 15 Dec 2020 12:01:55 PM Hmm... Built on Dec 15, 2020 @ 12:01:55 PM, published on Dec 15, 2020 @ 17:57 PM. Well, let's check the change log: * Wed Nov 18 2020 Patrik Novotný <panovotn at redhat.com> - 10.15-1 - Rebase to upstream release 10.15 Resolves: rhbz#1898213 Resolves: rhbz#1898341 Resolves: rhbz#1901567 This one is interesting. In particular, it makes me wonder what is happening behind the scenes. I've seen two seemingly conflicting descriptions of what CentOS 8 Stream is: 1. CentOS 8 Stream is where development happens first, allowing the CentOS community to influence what goes into CentOS 8. This would imply "LATEST". This suggests that CentOS community is in control of the direction of RHEL. 2. CentOS 8 Stream is the "next" (N+1?) version of RHEL 8. This might NOT be "LATEST", but only "LATEST" as shared by Red Hat. Perhaps they are still maintaining a private N+2? This says that the CentOS community has only limited influence. They are seeing N+1, but not seeing N+2. If the QA was being performed *after* it was built, then this means the change was queued from some time after November 18, until December 15, before being published within 6 hours (assumes the same time zones). Less than 6 hours of post-build testing was performed before releasing it to the public. If the QA was being performed *before* it was built, then this implies that the build being published is not the build that Red Hat did QA on. The build procedure might be the same, but the actual build in CentOS 8 Stream is *not* the original build that was tested. This would suggest that a private Red Hat branch exists, possibly the N+2 branch? And this means that the end-to-end testing occurred some time between Nov 18 and Dec 15, on a private Red Hat branch, hidden from the CentOS community. If somebody who actually knows how this works for real could explain, it would be enlightening? Which of the above conclusions is the closest to being right? Now, I picked PostgreSQL above as it is a favourite of mine, and it happened to be recently updated in AppStream. But, more important to everybody else, is the kernel: http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4.18.0-257.el8.x86_64.rpm By the date on the mirror, it seems to say it was published on 2020-12-04 00:39. Let's look inside at build date, and the change log: Build Date : Thu 03 Dec 2020 05:32:49 PM This is similar to the PostgreSQL update, as it was built on Dec 3 @ 5:32:49 PM, and published to the mirror by Dec 4 @ 00:39, around 7 hours later? Inside, however, the changelog * Wed Dec 02 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-257.el8] Hmmm... The changelog is for Dec 2nd. The build date was Dec 3rd. The deploy date was Dec 4th. Maybe this was an urgent change of some sort, and very tiny - so it was easy to QA? Oh uh.... * Wed Dec 02 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-257.el8] - [hv] hv: vmbus: Allow cleanup of VMBUS_CONNECT_CPU if disconnected (Mohammed Gamal) [1886096] - [hv] hv: vmbus: Add parsing of VMbus interrupt in ACPI DSDT (Mohammed Gamal) [1886096] .. 167 more lines ... Wow. That's a pretty big change to QA in less than a day. And what about the prior change? * Mon Nov 30 2020 Jan Stancek <jstancek at redhat.com> [4.18.0-256.el8] - [kernel] PM: hibernate: Batch hibernate and resume IO requests (Lenny Szubowicz) [1868096] - [net] tunnels: Fix off-by-one in lower MTU bounds for ICMP/ICMPv6 replies (Antoine Tenart) [1895765] ... 124 more lines ... Another big change, and so is the one after that, on Nov 27! In fact, if you look at the changes since November 2, the changelog from 4.18.0-257 to 4.18.0-240 is 6149 lines long. You just changed my mind! *YOU* should definitely use CentOS 8 Stream! All of *YOU*. That way, by the time you test all of these changes, and report all the bugs, it won't affect *ME*. :-) :-) :-) > Making a reliable distribution doesn't make RHEL obsolete. There's still > a lot of value in an OS that is itself semantically versioned, and has > paid support contracts. Yes there is. There is value, because it is important. That's what I'm trying to say. :-) > There's no evidence to support the idea that CentOS Stream will be > unreliable because a reliable system would compete with RHEL. And lots > of evidence to the contrary, including the extent to which Red Hat > publishes their work in a form usable to rebuild the distribution. "Unreliable" and "reliable" are extremes. RHEL 8 isn't 100% reliable, and CentOS 8 Stream isn't 100% unreliable. However, RHEL 8 will be a lot more reliable if *YOU* test CentOS 8 Stream on *all* of your systems. The more, the better. Thank you! -- Mark Mielke <mark.mielke at gmail.com>