[CentOS-devel] https://blog.centos.org/2020/12/future-is-centos-stream/

On 10/12/2020 13:31, Neal Gompa wrote:
> On Thu, Dec 10, 2020 at 8:25 AM Phil Perry <pperry at elrepo.org> wrote:
>>
>> On 10/12/2020 12:10, Josh Boyer wrote:
>>> On Wed, Dec 9, 2020 at 3:49 PM Phil Perry <pperry at elrepo.org> wrote:
>>>>
>>>> On 09/12/2020 19:01, Brendan Conoboy wrote:
>>>>> On Wed, Dec 9, 2020 at 10:04 AM Phil Perry <pperry at elrepo.org
>>>>> <mailto:pperry at elrepo.org>> wrote:
>>>>>
>>>>>       On 09/12/2020 17:40, Stef Walter wrote:
>>>>>        > On Wed, Dec 9, 2020 at 6:33 PM David Hrbáč <david-lists at hrbac.cz
>>>>>       <mailto:david-lists at hrbac.cz>
>>>>>        > <mailto:david-lists at hrbac.cz <mailto:david-lists at hrbac.cz>>> wrote:
>>>>>        >
>>>>>        >         I don't use CentOS Stream, I use RHEL. I use RHEL to develop
>>>>>        >         software
>>>>>        >         for RHEL and compatible OS clones, including CentOS. If
>>>>>       Stream
>>>>>        >         retains
>>>>>        >         binary compatibility, and specifically kernel ABI
>>>>>       compatibility,
>>>>>        >         then
>>>>>        >         the users of the software packages we develop can continue to
>>>>>        >         use them.
>>>>>        >         If not, they can't. Simple as that. So please don't push
>>>>>       rolling
>>>>>        >         kernel
>>>>>        >         updates to Stream that break the kernel ABI.
>>>>>        >
>>>>>        >
>>>>>        > Indeed. If any such broken change (eg: that breaks kernel ABI) is
>>>>>       pushed
>>>>>        > to Stream, that is treated as a serious problem by the RHEL
>>>>>       engineering
>>>>>        > teams. We have the necessary process in place to QE test changes
>>>>>       before
>>>>>        > they arrive in CentOS Stream.
>>>>>        >
>>>>>        > I understand this fact alone is not a panacea for all the problems
>>>>>        > people are highlighting. But it does seem to cover your use case.
>>>>>       From a
>>>>>        > regression, stability, ABI, and kernel ABI perspective, it is the
>>>>>       goal
>>>>>        > and focus of many of us in RHEL Engineering for CentOS Stream to
>>>>>       be stable.
>>>>>
>>>>>       Hi Stef,
>>>>>
>>>>>       Thank you for your response. You do realise I'm not just talking about
>>>>>       whitelisted kernel symbols, but the whole kernel ABI?
>>>>>
>>>>>       Whilst the RHEL kernel ABI whitelist is great in principle, in practice
>>>>>       I am yet to find a kernel driver that uses only symbols on the
>>>>>       whitelist. As I said previously, every single driver I maintain broke
>>>>>       between RHEL8.2 and RHEL8.3 due to changes in the kernel ABI.
>>>
>>> Just to touch on this a bit, the reason the RHEL kABI whitelist exists
>>> is because the upstream Linux kernel explicitly does NOT have a kernel
>>> ABI.  Given that you maintain out of tree drivers, I'm sure you know
>>> this but it might not be evident to anyone that doesn't.  So the RHEL
>>> kABI whitelist is there to express what RHEL is committing to keep
>>> stable for a kernel ABI for 10 years while dealing with an upstream
>>> that doesn't care at all.
>>>
>>
>> Hi Josh,
>>
>> Yes, I get that.
>>
>> The issue here is not so much that the kABI, outside of the whitelist,
>> changes, it's that it's not the same between the kernels in RHEL and the
>> kernels in STREAM. The issue is the two products are out of sync and are
>> not the same.
>>
>> Red Hat are trying to sell Stream to the community on the basis it will
>> be an almost like for like drop in replacement for CentOS 8 users once
>> CentOS 8 no longer exists, and at the kernel level that is simply not true.
>>
>> People talk about "3rd party out of tree" drivers like it is a small
>> thing. It is not. It affects a large number of your users. The elrepo
>> project has over 150,000 unique IPs per week hitting our repositories,
>> and somewhere around 1-2 million users per year. We would estimate the
>> vast majority of those are CentOS users. If you fail to keep the kernel
>> ABI in sync between RHEL and Stream, you break Stream for all those users.
>>
>>>>> Yes, the symbol list is helpful, but it's rarely 100%, so we're going to
>>>>> have to iteratively do better.
>>>>>
>>
>> Don't get me wrong, I love the fact RHEL has a stable ABI and have the
>> kABI whitelist, but to be brutally frank it is totally irrelevant when
>> considering 3rd party out of tree drivers. In my 10 plus years
>> experience developing, backporting and packaging such drivers for the
>> RHEL kernel, I can categorically state I have not seen a single driver
>> that *only* uses symbols that are on the whitelist. Further, I would
>> challenge anyone to write even the most simplistic "hello world" kernel
>> driver that uses only whitelisted symbols, let alone a driver that
>> actually interacts with hardware and does something useful.
>>
> 
> I'd like to turn this around and ask: could we leverage that expertise
> and active work as a SIG in CentOS to improve the quality of the RHEL
> kernel ABI whitelist? If I remember rightly, you do work in the ELRepo
> project. Perhaps it would make sense to start pulling ELRepo into
> CentOS as a SIG and continuously integrate it with CentOS Stream?
> 
> By doing so, we could introduce gating mechanisms to prevent kernels
> that break driver packages from being released in the first place. We
> could also get mechanisms in place to do proper Secure Boot signing
> for these driver packages, signed with a key automatically trusted by
> the CentOS kernel and trivially usable by RHEL folks, too.
> 
> I too maintain an out of tree driver for work, and this is something
> that would be valuable to *me* to make my life easier. If you do as
> many as I think you do, I would imagine that this would be even *more*
> valuable to you.
> 
> This is one of the reasons I started getting involved in CentOS Stream
> in the first place, and I think this would be a good thing for you
> too.
> 

Hi Neal,

I'm probably not the right person to ask, as I probably have a very 
narrow view, highly influenced by the work I'm directly involved with, 
as to the role of RHEL's kABI whitelist.

Generally, I will say, that up until now the process has worked really 
well. Red Hat introduced the Driver Update Programme back in the early 
days of RHEL5, and the elrepo project has simply leveraged that 
framework through RHEL5 -> RHEL8. Hopefully that can continue. I really 
see little need to dream up complicated fixes for something which wasn't 
broken.

If we are left with a scenario where it's broken, then I think there are 
far simpler alternatives than trying to manage the ABI of a constantly 
moving target. If RH won't make the RHEL kernels available to Stream 
users, it would be trivial for anyone to rebuild them as part of a SIG, 
centosplus, or as part of an independent 3rd party repo for users that 
need them. After all, there is a lot of expertise in the CentOS 
community at rebuilding RHEL source code.