[CentOS-devel] https://blog.centos.org/2020/12/future-is-centos-stream/

On 10/12/2020 12:10, Josh Boyer wrote:
> On Wed, Dec 9, 2020 at 3:49 PM Phil Perry <pperry at elrepo.org> wrote:
>>
>> On 09/12/2020 19:01, Brendan Conoboy wrote:
>>> On Wed, Dec 9, 2020 at 10:04 AM Phil Perry <pperry at elrepo.org
>>> <mailto:pperry at elrepo.org>> wrote:
>>>
>>>      On 09/12/2020 17:40, Stef Walter wrote:
>>>       > On Wed, Dec 9, 2020 at 6:33 PM David Hrbáč <david-lists at hrbac.cz
>>>      <mailto:david-lists at hrbac.cz>
>>>       > <mailto:david-lists at hrbac.cz <mailto:david-lists at hrbac.cz>>> wrote:
>>>       >
>>>       >         I don't use CentOS Stream, I use RHEL. I use RHEL to develop
>>>       >         software
>>>       >         for RHEL and compatible OS clones, including CentOS. If
>>>      Stream
>>>       >         retains
>>>       >         binary compatibility, and specifically kernel ABI
>>>      compatibility,
>>>       >         then
>>>       >         the users of the software packages we develop can continue to
>>>       >         use them.
>>>       >         If not, they can't. Simple as that. So please don't push
>>>      rolling
>>>       >         kernel
>>>       >         updates to Stream that break the kernel ABI.
>>>       >
>>>       >
>>>       > Indeed. If any such broken change (eg: that breaks kernel ABI) is
>>>      pushed
>>>       > to Stream, that is treated as a serious problem by the RHEL
>>>      engineering
>>>       > teams. We have the necessary process in place to QE test changes
>>>      before
>>>       > they arrive in CentOS Stream.
>>>       >
>>>       > I understand this fact alone is not a panacea for all the problems
>>>       > people are highlighting. But it does seem to cover your use case.
>>>      From a
>>>       > regression, stability, ABI, and kernel ABI perspective, it is the
>>>      goal
>>>       > and focus of many of us in RHEL Engineering for CentOS Stream to
>>>      be stable.
>>>
>>>      Hi Stef,
>>>
>>>      Thank you for your response. You do realise I'm not just talking about
>>>      whitelisted kernel symbols, but the whole kernel ABI?
>>>
>>>      Whilst the RHEL kernel ABI whitelist is great in principle, in practice
>>>      I am yet to find a kernel driver that uses only symbols on the
>>>      whitelist. As I said previously, every single driver I maintain broke
>>>      between RHEL8.2 and RHEL8.3 due to changes in the kernel ABI.
> 
> Just to touch on this a bit, the reason the RHEL kABI whitelist exists
> is because the upstream Linux kernel explicitly does NOT have a kernel
> ABI.  Given that you maintain out of tree drivers, I'm sure you know
> this but it might not be evident to anyone that doesn't.  So the RHEL
> kABI whitelist is there to express what RHEL is committing to keep
> stable for a kernel ABI for 10 years while dealing with an upstream
> that doesn't care at all.
> 

Hi Josh,

Yes, I get that.

The issue here is not so much that the kABI, outside of the whitelist, 
changes, it's that it's not the same between the kernels in RHEL and the 
kernels in STREAM. The issue is the two products are out of sync and are 
not the same.

Red Hat are trying to sell Stream to the community on the basis it will 
be an almost like for like drop in replacement for CentOS 8 users once 
CentOS 8 no longer exists, and at the kernel level that is simply not true.

People talk about "3rd party out of tree" drivers like it is a small 
thing. It is not. It affects a large number of your users. The elrepo 
project has over 150,000 unique IPs per week hitting our repositories, 
and somewhere around 1-2 million users per year. We would estimate the 
vast majority of those are CentOS users. If you fail to keep the kernel 
ABI in sync between RHEL and Stream, you break Stream for all those users.

>>> Yes, the symbol list is helpful, but it's rarely 100%, so we're going to
>>> have to iteratively do better.
>>>

Don't get me wrong, I love the fact RHEL has a stable ABI and have the 
kABI whitelist, but to be brutally frank it is totally irrelevant when 
considering 3rd party out of tree drivers. In my 10 plus years 
experience developing, backporting and packaging such drivers for the 
RHEL kernel, I can categorically state I have not seen a single driver 
that *only* uses symbols that are on the whitelist. Further, I would 
challenge anyone to write even the most simplistic "hello world" kernel 
driver that uses only whitelisted symbols, let alone a driver that 
actually interacts with hardware and does something useful.