On Sat, Dec 19, 2020 at 12:50 PM Nico Kadel-Garcia <nkadel at gmail.com> wrote: > On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller <mattdm at mattdm.org> wrote: > > On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote: > > > 2. Minor release milestones to stabilize branches. We have breakage > > > with most minor release upgrades, and the stabilization process is an > > > important method of isolating users from being affected by this. This > > > is why CentOS 8 Stream is being said "for developers", while RHEL 8 > > > would be "for production". It is being said, because it is a real > > > thing. If you truly believed minor release milestones were unnecessary > > > for CentOS 8 Stream, then you would also believe that minor release > > > milestones were unnecessary for RHEL 8. > > It's important to note that the CentOS Linux rebuild never actually had > > this. RHEL minor releases are actually branches, and you can stay at a minor > > release and still get security updates. For CentOS Linux, a minor release is > No. RHEL minor releases are more like source control "tags" than > branches. They have stable installation images one can refer and start > new deployments from, and ongoing work is branched from that tag. This > has been the case since the earliest Red Hat, not even RHEL, releases > since I encountered them back in 1995. Exactly. Matthew: You are pretending as if RHEL beta releases, and RHEL stabilization doesn't exist for some important reason, These gates +/- exist between RHEL Stream and RHEL: 1. Changes first go through designer testing and user testing of some sort. "Try this fix". This is typically limited to only a few users who reported the problem. 2. Changes were typically made in the "next" branch, which I believe was often +2 minor releases. This would see the first QA testing, and the start of some integration testing. Historically this has been an internal process to Red Hat, which is a problem as it also limits exposure of the QA testing to Red Hat, and if any behaviours are changing, it limits the ability for users to be ready for upstream changes coming down the pipeline. Changes may sit in this stabilization branch for weeks to months. 3. Changes would backported into the +1 minor release, and get periodically rolled into a beta program, that would allow the outside world the first glimpse of what they might be getting, including changes documented in release notes. Presumedly Red Hat is doing some form of QA and certification during this period, although this is also a bit of a black box. These release could be analyzed and if users were interested in testing, they could go through the special effort of testing. As the beta program requires some setup to participate in, many vendors do not. However, of the people that do try out the beta - presumedly at least some bugs are found and fixed, before they make it into a release. Changes may sit in this stabilization branch for weeks to months. 4. Changes are released into an official minor release, Users receive release notes that they can analyze for impact. Users can and should run early adopter programs, to ensure that the large drop of changes will not impact their workflows. Anybody who arbitrarily does "yum update" at this point is playing with fire. Red Hat QA processes are insufficient to detect and prevent every problem that might affect customers. In many cases, the problems are simply missed, and pass through the above gates without detection. In many other cases, the problems are impossible to have been predicted by Red Hat, because Red Hat does not know how people use RHEL in real-life. For example, Red Hat cannot run all our product builds before release RHEL, so how are they supposed to be able to detect if our product builds will break? This is why there are release notes and minor releases, to provide this level of insulation. Where exactly, does CentOS 8 Stream fit into this? I'm thinking you are cutting at 3 + 4 out of the picture, and then some people on this list are essentially claiming 3 + 4 provides no value for 95% of the "users" or 95% of the "installed base" (not sure which at this point, since the numbers are quite different). Downstream of RHEL, there are additional gates. Including what Nico is talking about, and what anybody with a significant deployment will include, which is internal repositories and internal "stable" branches of RHEL. We want all of the prior branches and gates, but we know that mistakes still make it through these gates. So, we analyze every change that comes out of RHEL, and if it might impact our workflows, we do testing of our workflows before releasing the packages to our internal systems. In my case we had a "latest" branch which included content covered by the above gates, a "testing" branch which includes content that was being tentatively approved, and a "stable" branch which includes content that was approved. If the people in favour of abandoning CentOS 8 for CentOS 8 Stream effectively remove gates 3 + 4 from this process, you introduce unacceptable risk to our programs, and this will force us to choose some other solution. "Us" is a really big "us". I think by numbers, "us" is almost your entire community. If you abandon "us" as is described, the consequence of this is that "us" will go somewhere else. "Us" is the reason why CentOS 8 existed in the first place. Red Hat and the CentOS board can ignore "us", but they can't dictate to "us". You can't force us to use this CentOS 8 Stream merely by claiming it is not really different at all. I see Facebook mentioned several times as evidence that CentOS 8 Stream is no big deal. I imagine that Facebook has their own stabilization processes, and they are effectively abandoning *RHEL*. I need to be clear of what I mean by this. In this scenario where CentOS 8 Stream is upstream of RHEL 8, and CentOS 8 is cancelled, the situation we are left with is that RHEL is deciding when to cut new CentOS 8 Stream milestones and stabilize them. There is a lot of emphasis on the Red Hat QA processes, and how they are very important, but I just explained that many bugs escape the Red Hat QA process, and we catch them. In effect, the Red Hat milestones are not necessarily good ones, and this is where Amazon Linux and others have gone, and it sounds like Facebook as well. The community can pick a different process for snapshot of CentOS 8 Stream and stabilization. The community can run its own beta programs separately from RHEL, effectively making RHEL obsolete. This may be hard to imagine for individual users who are accustomed to Red Hat controlling everything they are allowed to see and do, but it's actually a problem for companies that are being asked to pay $4M USD annual for RHEL subscription fees for the right to use a collection of free software. We can do our own testing and stabilization for $4M USD annually. We don't need Red Hat. And this is the problem with the Red Hat subscription model, and the problem with CentOS 8 Stream. Red Hat has priced itself out of essential markets. This isn't about "if you can't afford RHEL, you can use CentOS 8 Stream". This is about the value proposition of RHEL. If the value proposition of RHEL is lower than the value proposition of doing it ourselves, we would be financially irresponsible to choose RHEL. If you don't think a few large companies can do as good a job as Red Hat at branching, and stabilizing - you are mistaken. This is what we do for *our* products. We know exactly how to do this. We have build systems and test systems. As described above, we catch problems that Red Hat does not catch. I didn't really comment on Mike McGraph's comment that "we are the best", because I don't really want to go there - but I would be cautious about claiming words like "best". There is a pretty significant difference between "we are highly valuable contributors" and "we are the best". The reason we align on Red Hat, is because it is beneficial for us to do so. The QA testing and such is useful, for sure - but more useful is to have a unified "EL" environment where packages can be deployed in multiple different "EL" distributions with confidence, and vendor solutions will generally work across all "EL" distributions. If Red Hat makes this too difficult too achieve, there is an outcome that has not been described here: Red Hat will become the vendor lock-in speciality solution that nobody chooses to use. Based upon several posts in centos-devel, It has become clear to me that many CentOS board members, CentOS contributors, and Red Hat staff, do not fully comprehend the value that is brought to the table by allowing large companies to consolidate on "EL" as a standard. This is not "EL Next", this is "EL". It is fine for this value to not be recognized - but when failure to recognize this value has the consequence of CentOS/RHEL actually *removing* CentOS as the common standard around which all EL distributions align, we will simply pick a new standard. The rest of us can work together just fine, and we don't need Red Hat. It will just be a terrible shame. None of us wants to see this. We want Red Hat to be the unifying force. We want to help Red Hat. But, if Red Hat doesn't want us around any more - it will be more like a break up with a first girlfriend. Life goes on. It's up to you, CentOS/RHEL. You basically have these choices: 1. CentOS 8 stays, and "EL" distributions use CentOS 8 as a common baseline. 2. RHEL 8, including minor releases, stays accessible in source form, so that the community can build a clone that aligns with RHEL 8 for the purposes of *standards*, not for the purpose of *cheap*. 3. RHEL 8 hides the source behind a legal framework that makes it difficult or impossible for "bug-for-bug" compatibility to be maintained. Either the community reverse engineers this (for example, OpenJDK 8 is kind of like this), or the community creates its own baselines and ignores RHEL. RHEL becomes irrelevant. -- Mark Mielke <mark.mielke at gmail.com>