On Thu, Dec 24, 2020 at 5:38 AM Neal Gompa <ngompa13 at gmail.com> wrote: > On Thu, Dec 24, 2020 at 8:22 AM Phil Perry <pperry at elrepo.org> wrote: [snip] > > Further, we have an issue with the Stream installation images which are > > constantly being updates during the latest compose and feature the > > latest Stream kernel - these are unable to use Driver Update Disk images > > (DUDs) which are generally built around the point release GA kernel and > > are likely not compatible with newer Stream kernels. > > I'm a bit more ambitious here: I'd like kernel updates to not be > released *at all* to users unless it's validated alongside kernel > module packages. I suspect this would be problematic for some CentOS Stream use cases, but providing a mechanism (Like a second repo) to flag kernels as good-to-go with a community-defined kernel module compatibility baseline makes sense to me. The automation required seems feasible. I imagine it's a question of somebody doing the work. Seems like a good community building opportunity to me. > > What happens in reality (especially in the first 5 years during the > > active development phase or Stream phase) is that Red Hat branch the > > RHEL kernel at point release time and the 8.3 kernel, for example, stays > > stable for 6 months with only important bug fix and security fixes, but > > no new features whilst the RHEL development kernel branch for 8.4, which > > is now being released to Stream, gets all the big backports that will be > > in the 8.4 kernel, and those backports are what causes breakage of > > symbols that are not on the kABI whitelist but are used in the real > > world by many/most 3rd party drivers. > > Right, but I think Brendan Conoboy has said elsewhere on this list > that they want to expand the kABI whitelist as much as practically > possible based on real-world data. We have a large repository of these > with ELRepo (among others) and we can use that for helping the RHEL > team expand the whitelist so that they don't break so often. I do think CentOS Stream could be one of the feeders for high value symbols. We have to be cognizant that sometimes a symbol can't be added to the list no matter how much we want to, because it would make backporting upstream work impossible. Because of this I think it's better to focus efforts on managing the impact of symbols changing. Work in this area can be beneficial for the providers of kernel modules and for end users at the same time. [snip] > > So I'm absolutely not against it and nor do I want to prevent or stop it > > from happening. Quite the opposite - I am really looking forward to the > > day I can contribute simple fixes to the RHEL kernel rather than having > > to file a bug and wait months/years to see the incorporation of a simple > > upstream fix or have to open a support case and spend months dealing > > people that do not understand the issue. But above all I just want > > people to recognise that this is a *development* system and stop trying > > to tell people that is it a drop in replacement for CentOS Linux because > > it is not. > > In the strictest sense, it obviously is not. But in a very real > practical sense, it absolutely is. Aside from the kernel issues (which > I firmly believe are solvable), people are generally not going to > notice a difference between CentOS Linux 8 and CentOS Stream 8. > > My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the > spring because it was strictly better for production *and* > development. I've been in the process of opportunistically switching > our build targets from CentOS Linux 8 to CentOS Stream 8 most of the > year. With the retirement of CentOS Linux 8, it now becomes more of a > priority, but it was already going to happen. > > And for the couple of third-party kernel modules we use, our build > system already triggers automatic rebuilds when new kernel builds are > released. Heck, I support *Fedora* just fine because of that. So the > kernel thing has been a non-issue for me. However, I recognize that > most people haven't put in the engineering effort *I* have to support > that kind of thing, and Red Hat's kABI promises are intended to make > it so people don't have to. So we need to improve processes around > that to make kernel updates less painful for third party kernel module > maintainers, since that's a huge part of the value of the RHEL/CentOS > ecosystem. > > CentOS Stream 8 may suck a bit for that *now*, but it doesn't have to > remain that way. In the future, it could make both RHEL and CentOS > even better in this regard if we take advantage of this opportunity. This is really well said, Neal. With a little work on these fundamentals I suspect we can make CentOS Stream do for everybody what you've had to do for yourself. -- Brendan Conoboy / Linux Project Lead / Red Hat, Inc.