[CentOS-devel] [EXT] Balancing the needs around the CentOS platform

On 19/12/2020 21:42, Brendan Conoboy wrote:
> On Sat, Dec 19, 2020 at 1:13 AM Peter Georg
> <peter.georg at physik.uni-regensburg.de> wrote:
>> On 19/12/2020 07.44, Karsten Wade wrote:
>> I agree with you that CentOS Stream might actually be a good way forward
>> and consider (even looking forward to) switching from CentOS Linux to
>> CentOS Stream for the reasons you mentioned.
>> However I disagree on one point: You mention CentOS Stream can cover
>> "95% (or so)" of the current users. As long as 3rd party drivers are not
>> working with CentOS Stream the number will be lower.
>> This topic has already been mentioned several times on this mailing list
>> by various users. So far feedback has been positive here on the list.
>>
>> In my opinion, the only way to guarantee compatibility is by 100% kernel
>> ABI compatibility. The only way I can think of implementing this is by
>> offering an optional "RHEL kernel repository" containing the (updated)
>> kernel of the current RHEL minor release. Feedback to this proposal has
>> been positive so far as well, but no concrete feedback yet.
>> I even suggested this to centos-questions at redhat.com. So far I only got
>> a response that I shall wait till next year for no- and low-cost RHEL
>> offerings. Let's wait and see if I get a useful answer to my proposal.
>>
>> Sorry for mentioning this issue once again here on the mailing list, but
>> I just hope that by mentioning it often enough, the possibility of this
>> issue/proposal being acknowledged goes up.
> 
> Hi Peter,
> 
> Thanks for mentioning it again- I would encourage anybody else who is
> using kernel modules that aren't included in the stock kernel to mail
> centos-questions at redhat.com and let us know about it.
> 
> Just on list I've read that the ongoing ability to use CentOSPlus,
> ELRepo, and a few miscellaneous third party drivers is the difference
> between being able to adopt CentOS Stream or not.  My impression is
> that if CentOS Stream can provide some mechanism by which people will
> be unable to 'dnf update' their way into an unusable system for want
> of a compatible kernel module, that somewhat resolves the issue.  I
> picture a SIG being formed of community members to create this and
> oversee the multiple implementation details that would be necessary
> per use case (EG, don't stop ELRepo, provide something that works
> smoothly with it).
> 

Hi Brendan,

I fear this is not really practicable. 3rd Party kernel driver updates 
are largely delivered for RHEL using Red Hat's Driver Update Programme 
framework that has existed since the early days of RHEL5 [1-4]. It is 
totally dependent upon the relatively stable kABI that exists in RHEL 
between point releases. CentOS Stream does not provide this. You are 
proposing we try banging a square peg into a round hole. As it stands, 
CentOS Stream is great as a CI development system for us to use to 
develop packages for the next point release of RHEL (e.g, 8.4), not for 
us to develop packages intended to be consumed today by users running 
Stream.

The solution is to make regular RHEL kernel updates available in the 
Stream repository, on either an opt-in or opt-out basis, depending upon 
one's intended use case. To suggest that a SIG or other mechanism could 
be implemented to regularly rebuild 3rd party content is simply not 
feasible for a number of reasons. Firstly, it's not simply a matter of 
rebuilding 3rd party drivers against each newer kernel. Many of our own 
drivers will require a code rebase for each rebuild or patches updating 
that no longer apply cleanly. This is not something that can be readily 
automated. Secondly, as I understand it, as we move forward and Red Hat 
kernel development moves towards the CentOS Stream model, we will be 
seeing nightly builds so this work will need to be undertaken on a daily 
basis which again is simply not feasible, nor sustainable. Even if we 
could rebuild our content on a daily basis, by the time we've updated 
our build systems with the latest CentOS Stream kernel, rebased , fixed 
and rebuilt 50+ packages, signed them, put them through QA and released 
them, and our global mirror network has synced, CentOS Stream will have 
likely released the next daily kernel update and we are back to square 
one. Square peg, round hole.

I am encouraged that people within Red Hat are starting to recognise and 
acknowledge some of the challenges we face, but the conversation seems 
to be stuck at the same place and we are not moving forward to discuss 
and implement the very workable solution which has been proposed. Can 
regular RHEL kernels be populated into a Stream repository, either by 
default or as an optional add on? If so, when can we expect this to happen?

Or maybe Red Hat's solution is for any affected CentOS Stream customers 
to email centos-questions at redhat.com and be given free RHEL licenses? It 
would be nice to have some clarity around this.

[1] 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/5.0_release_notes/sect-red_hat_enterprise_linux-release_notes-release_notes_for_ppc-driver_update_program

[2] 
http://people.redhat.com/jcm/kmods/RHEL5.2/docs/RHEL5_DUP_InstallQuickstart.pdf

[3] http://people.redhat.com/jcm/el6/dup/docs/old/pre-release/whitepaper.pdf

[4] https://access.redhat.com/articles/2743651