On Mon, Dec 21, 2020 at 8:44 AM Neal Gompa <ngompa13 at gmail.com> wrote: > On Mon, Dec 21, 2020 at 4:28 AM Mark Mielke <mark.mielke at gmail.com> wrote: > > If somebody who actually knows how this works for real could explain, > > it would be enlightening? Which of the above conclusions is the > > closest to being right? > Judging by this response, I assume you're not familiar with how > distribution testing typically works. At least for the past ten years, > distribution testing of packages as they're updated has been almost > entirely automated. Rather than spending ridiculous amounts of time > having *people* test it, the gates are enforced by scripts, Ansible > playbooks, etc. that execute workflows to validate behavior of a > package. In the last couple of years, Red Hat has been rewriting their > test suite for public consumption and pushing them into Fedora, so > that packages can be validated at the earliest stage with Fedora ELN. Correct. I was not familiar with how Red Hat does distribution testing. I suspected, but I did not know. Your information helps fill in the missing gaps. > But let's focus on RHEL/CentOS for the moment. At the RHEL/CentOS > Stream level, stuff is pushed to the RHEL internal Git server *first*. > Their internal Git server is wired up with checks to run on commits > being pushed so that tests run on the commit *before* it lands. Once > it passes *those* tests, the commit lands and the package is built in > "Brew" (the RHEL Koji instance). Once the package is built there, a > series of integration and system validation tests are run before the > build is accepted for release. Once *that* happens, it is pushed to > the CentOS Git server (git.centos.org), and then the CentOS Stream > build is kicked off. Once that build is made, it is then run through > the CentOS test suite and validated based on that before pushing it > out for release to CentOS Stream. Good. CI/CD in action. > With the kernel in particular, the test suite probably validated it > for release in less than half a day on the RHEL side. It was then > pushed on December 3[1], Carl debranded it shortly after[2] and built > on December 4[3], and released later that day after it passed the > CentOS functional tests. Without adding qualitative judgements here, this confirms that fundamentally: 1. There is an upstream RHEL-private branch. To reach the upstream RHEL-private branch, it must pass a set of automated tests. 2. Once a set of automated integration and system validation tests pass, which can complete in as little as 12 hours, the change is published to CentOS 8 Stream. > As someone who has also designed such systems in the past, I'd say > that's actually pretty good. There's further work to be done, for > sure, but there always is. It's all about continuous improvement. > That's why I made suggestions about bringing ELRepo and other open > source kernel modules into the CentOS Project as SIGs so that we can > *add* to the kernel automated testing and prevent broken expectations > with things like the kernel ABI that Red Hat maintains for the RHEL > kernel[4][5]. Because if you look at CentOS Stream as the gift and > opportunity that it truly is, then there are amazing things we can > accomplish together in the CentOS Project. Yes, it is good. I can easily see why this is of great value to people who produce content that needs to align with EL, especially to people such as yourself who understand the value of moving the automated testing upstream so that Red Hat would not automatically approve a change that would break your downstream use case, or even better - that Red Hat would include your downstream code into their upstream and do the testing for you (automated, to minimize expenses). CentOS 8 Stream is a great option for people such as yourself - and should be a great option for other vendors. Unfortunately, most vendors are much more lazier than yourself (by historical evidence, and by their response to questions like "Do you support 7.9?"), so it isn't clear that you represent most vendors. Please take that as a compliment, although - if I'm being honest, I think you are doing what you should be doing, and they should take this as an insult. :-) The value of CentOS 8 Stream as a new option, was never a question for me. I am sorry if my responses might have lead to the conclusion that I don't like CentOS 8 Stream. I *do* like it. The problem is that CentOS 8 Stream is being sold as a replacement for CentOS 8, and phrases like "good enough for 95% of use cases" are being thrown around. I believe this to be a naive or ingenuous statement. However, not everybody is doing this. A significant number of people, including senior leadership from Red Hat, are clearly saying "CentOS is for development, RHEL is for production". These people are telling the truth, at least on 2020-12-21. In some future, where the automated testing performed by a change lands in CentOS 8 Stream is good enough for production use, then it may well be "good enough for 95% of use cases". I don't believe it is, and I think anybody who believes it is today - is not making a correct assessment, and if they deploy CentOS 8 Stream in production without providing their own stabilization process, including what amounts to creating a baseline, and running their own patch program separate from RHEL/CentOS, they are very likely to have a rude awakening. They will be stuck in a hell where a change escapes Red Hat's automated testing, and they will have trouble upgrading or downgrading. Eventually, they will create their own repository that is locked to a specific baseline, and they will only permit packages in once it passes there own internal criteria, and they will then realize that certain breaking changes are not acceptable in their environment but that the breaking change is a dependency of some other change they do want. They will then need to learn to backport their own patches. And so on, until eventually they are doing a crappy job of re-implementing what CentOS already was. Probably, they will give up and move to some other distribution. Probably, it will not be RHEL. Or, maybe all will be fine, and I have no idea what I am talking about. :-) But, none of my concerns about dropping CentOS 8, are intended to imply that CentOS 8 Stream is bad. CentOS 8 Stream is good, and I totally get why you want it - and why I should want it. -- Mark Mielke <mark.mielke at gmail.com>