On Mon, Dec 21, 2020 at 2:51 PM Mark Mielke <mark.mielke at gmail.com> wrote: > > On Mon, Dec 21, 2020 at 8:44 AM Neal Gompa <ngompa13 at gmail.com> wrote: > > On Mon, Dec 21, 2020 at 4:28 AM Mark Mielke <mark.mielke at gmail.com> wrote: > > > If somebody who actually knows how this works for real could explain, > > > it would be enlightening? Which of the above conclusions is the > > > closest to being right? > > Judging by this response, I assume you're not familiar with how > > distribution testing typically works. At least for the past ten years, > > distribution testing of packages as they're updated has been almost > > entirely automated. Rather than spending ridiculous amounts of time > > having *people* test it, the gates are enforced by scripts, Ansible > > playbooks, etc. that execute workflows to validate behavior of a > > package. In the last couple of years, Red Hat has been rewriting their > > test suite for public consumption and pushing them into Fedora, so > > that packages can be validated at the earliest stage with Fedora ELN. > > Correct. I was not familiar with how Red Hat does distribution > testing. I suspected, but I did not know. Your information helps fill > in the missing gaps. > > > But let's focus on RHEL/CentOS for the moment. At the RHEL/CentOS > > Stream level, stuff is pushed to the RHEL internal Git server *first*. > > Their internal Git server is wired up with checks to run on commits > > being pushed so that tests run on the commit *before* it lands. Once > > it passes *those* tests, the commit lands and the package is built in > > "Brew" (the RHEL Koji instance). Once the package is built there, a > > series of integration and system validation tests are run before the > > build is accepted for release. Once *that* happens, it is pushed to > > the CentOS Git server (git.centos.org), and then the CentOS Stream > > build is kicked off. Once that build is made, it is then run through > > the CentOS test suite and validated based on that before pushing it > > out for release to CentOS Stream. > > Good. CI/CD in action. > > > With the kernel in particular, the test suite probably validated it > > for release in less than half a day on the RHEL side. It was then > > pushed on December 3[1], Carl debranded it shortly after[2] and built > > on December 4[3], and released later that day after it passed the > > CentOS functional tests. > > Without adding qualitative judgements here, this confirms that fundamentally: > > 1. There is an upstream RHEL-private branch. To reach the upstream > RHEL-private branch, it must pass a set of automated tests. > 2. Once a set of automated integration and system validation tests > pass, which can complete in as little as 12 hours, the change is > published to CentOS 8 Stream. > > > As someone who has also designed such systems in the past, I'd say > > that's actually pretty good. There's further work to be done, for > > sure, but there always is. It's all about continuous improvement. > > That's why I made suggestions about bringing ELRepo and other open > > source kernel modules into the CentOS Project as SIGs so that we can > > *add* to the kernel automated testing and prevent broken expectations > > with things like the kernel ABI that Red Hat maintains for the RHEL > > kernel[4][5]. Because if you look at CentOS Stream as the gift and > > opportunity that it truly is, then there are amazing things we can > > accomplish together in the CentOS Project. > > Yes, it is good. I can easily see why this is of great value to people > who produce content that needs to align with EL, especially to people > such as yourself who understand the value of moving the automated > testing upstream so that Red Hat would not automatically approve a > change that would break your downstream use case, or even better - > that Red Hat would include your downstream code into their upstream > and do the testing for you (automated, to minimize expenses). > > CentOS 8 Stream is a great option for people such as yourself - and > should be a great option for other vendors. Unfortunately, most > vendors are much more lazier than yourself (by historical evidence, > and by their response to questions like "Do you support 7.9?"), so it > isn't clear that you represent most vendors. Please take that as a > compliment, although - if I'm being honest, I think you are doing what > you should be doing, and they should take this as an insult. :-) > Believe me, I know *exactly* what you're talking about. My experience with those (nameless) horrible vendors is why I deliberately didn't do that at $DAYJOB. But I also think those vendors are going to be in for a rude awakening as RHEL continues to move faster and faster to support everything that paying customers are asking of Red Hat. I'm just ahead of the curve because I got burned by a vendor before, and I wouldn't inflict that on my customers if I can help it. :) > The value of CentOS 8 Stream as a new option, was never a question for > me. I am sorry if my responses might have lead to the conclusion that > I don't like CentOS 8 Stream. I *do* like it. > > The problem is that CentOS 8 Stream is being sold as a replacement for > CentOS 8, and phrases like "good enough for 95% of use cases" are > being thrown around. I believe this to be a naive or ingenuous > statement. However, not everybody is doing this. A significant number > of people, including senior leadership from Red Hat, are clearly > saying "CentOS is for development, RHEL is for production". These > people are telling the truth, at least on 2020-12-21. > I'll say this flat-out as an independent user and contributor: CentOS Stream is *more* ready for production than CentOS Linux is. I personally run CentOS Stream 8 for my CentOS boxes because I've seen nothing but continuous improvement. Things *work* better on Stream for me than they did with the last checkpoint. That's why I switched from CentOS Linux 8 to CentOS Stream 8 back in the spring. For some Red Hatters, there's some vested interest from some folks to maintain the confusion, but here's the cold, hard truth: CentOS Stream 8 isn't being tested any less than Red Hat Enterprise Linux. RHEL has no extra gates on top of CentOS Stream, since the RHEL tests happen *before* pushing to CentOS Stream. The main reason for RHEL to have point releases is for easy delivery for certification validation. If you're using CentOS, those certifications do not apply to you anyway. :) And a final note: CentOS has *never* supported staying on a particular point release. It's *always* been a "stream" on the major version. The difference is that we're not doing semi-annual large dumps and just pushing them out as they come. > In some future, where the automated testing performed by a change > lands in CentOS 8 Stream is good enough for production use, then it > may well be "good enough for 95% of use cases". I don't believe it is, > and I think anybody who believes it is today - is not making a correct > assessment, and if they deploy CentOS 8 Stream in production without > providing their own stabilization process, including what amounts to > creating a baseline, and running their own patch program separate from > RHEL/CentOS, they are very likely to have a rude awakening. They will > be stuck in a hell where a change escapes Red Hat's automated testing, > and they will have trouble upgrading or downgrading. Eventually, they > will create their own repository that is locked to a specific > baseline, and they will only permit packages in once it passes there > own internal criteria, and they will then realize that certain > breaking changes are not acceptable in their environment but that the > breaking change is a dependency of some other change they do want. > They will then need to learn to backport their own patches. And so on, > until eventually they are doing a crappy job of re-implementing what > CentOS already was. Probably, they will give up and move to some other > distribution. Probably, it will not be RHEL. > The other way to look at this is that you can add your own circumstances to the test suites and make sure they don't break you after they do it once. :) The way the CentOS Stream project is set up, Red Hat is trying to work *with* the community to make a better project and a better product. I think that's awesome and should be applauded. > Or, maybe all will be fine, and I have no idea what I am talking about. :-) > > But, none of my concerns about dropping CentOS 8, are intended to > imply that CentOS 8 Stream is bad. CentOS 8 Stream is good, and I > totally get why you want it - and why I should want it. > Honestly, I think everything will be fine. This whole thing was *very* poorly handled by Red Hat, but having lived through the RHL->RHEL/Fedora change, I can say it could have gone worse. :) -- 真実はいつも一つ！/ Always, there's only one truth!