[CentOS-devel] [EXT] Re: RFC: Stream Kernel SIG Proposal

On 11/02/2021 00:03, Peter Georg wrote:
> 
> 
> On 10/02/2021 22.51, Brian Stinson wrote:
>>
>> On Wed, Feb 10, 2021, at 13:17, redbaronbrowser via CentOS-devel wrote:
>>> On Wednesday, January 20, 2021 9:31 AM, Mike McGrath 
>>> <mmcgrath at redhat.com> wrote:
>>>
>>>> I've confirmed with the team, the git repo is going to be all the 
>>>> normal git patches you would expect (IE: not arbitrarily munged 
>>>> together in some way).  There's one or two more things they're 
>>>> configuring with gitlab and they expect to have an actual repo that 
>>>> you can look at / poke at to validate what I'm saying in a few weeks.
>>>>
>>>>             -Mike
>>>
>>> Mike, is there any status update on individual patches applied to 
>>> Stream's kernel?
>>>
>>> Can we move forward with this SIG so I can publish the information I 
>>> already have?
>>>
>>> What about a Stream Plus kernel?  The directory for Stream still has 
>>> no packages at all.
>>>
>>> Or delivering kpatch live security updates without a reboot?
>>>
>>> I am ready to move forward immediately if the SIG is approved.  Is 
>>> there a reason to wait another "few weeks"?
>>> _______________________________________________
>>> CentOS-devel mailing list
>>> CentOS-devel at centos.org
>>> https://lists.centos.org/mailman/listinfo/centos-devel
>>>
>>
>> I see your questions roughly grouped in 3 categories, correct me if 
>> I've misinterpreted:
>>
>> - How do I get patches into the Stream kernel *now*?
>> Have you submitted any bugs against CentOS Stream on 
>> bugzilla.redhat.com with your patches? I want to make sure, first, 
>> that we have an actual answer because some of them may indeed be 
>> workable for inclusion in RHEL.
>>
>> All contributions for RHEL/Stream 8 (not just the kernel) are flowing 
>> through this bugzilla process for right now.
>>
>> - When can we see the new kernel workflows?
>> I don't have specific date/time when we expect to see the new-way of 
>> doing kernel repos but I would check https://gitlab.com/redhat/ 
>> towards the end of the month. Those will be targeted at RHEL/Stream 9 
>> kernels. I'm asking around about some of the work we're doing to make 
>> the Stream 8 kernel sources more accessible in a format that isn't 
>> just tarballs.
>>
>> - How do I get kernels that are not shipped in-distro?
>> Here's the problem as it exists right now, Kernels typically need 
>> secureboot signatures on them, and we don't have the infrastructure we 
>> need to make those signatures happen in the community build system. 
>> We're talking about ways to enable that directly in CBS, but in the 
>> meantime we plan to take community-based kernels and assist with 
>> getting builds done on infrastructure that can do signing. Note: this 
>> affects other packages like fwupd, grub, etc. We are absolutely open 
>> to continuing to build the Plus kernel. Would it make sense to get 
>> yourself, @toracat for the Plus kernel, and whoever else is interested 
>> in building kernels together on IRC to talk about how to handle 
>> coordination here?
> 
> Please add me to the list of interested people (@pjgeorg in IRC).
> 
> Let me note that I have a different goal that might be added to this SIG 
> and is not part of the four goals listed in the initial proposal:
> 
> 5. Package external kernel modules
> 
> So far (running CentOS Linux) I have been using various different 
> sources for kmod packages (and built some locally). However many of 
> these officially only support RHEL (most prominent on this list: ELRepo. 
> At least according to recent comments by Phil Perry. I haven't seen any 
> comments by other contributors). For the time being I "fix" this locally 
> by using dkms packages if avaible or maintain dkms packages myself for 
> modules that are otherwise only provided as kmod packages. I really 
> dislike doing such a job locally without any open collaboration. Just 
> feels wrong working with open source software without trying to 
> collaborate "upstream". Hence my interest in contributing to the 
> proposed SIG.
> 
> At first I'd suggest starting with drivers included in upstream kernel 
> but disabled by Red Hat. Yes, these are also available by using the 
> CentOS Plus kernel. However I, and maybe other users as well, prefer to 
> run the "vanilla" CentOS Stream kernel and add missing drivers as 
> external kernel modules as required. And of course, ELRepo currently 
> provides exactly this. However most of these can sadly not be used for 
> CentOS Stream. This also applies to building LTS kernel packages for 
> CentOS Stream. As Phil Perry mentioned this is and has already been done 
> by ELRepo for some time. I agree with Matthew Miller here: These things 
> should be connected. On a side note: The Hyperscale SIG seems to be 
> interested in providing LTS kernel(s) for CentOS Stream as well (has 
> been mentioned in their talk at the CentOS Dojo last week).
> 
> Once everything is set up and working for these simpler cases, packages 
> for any other kernel modules that are of use for CentOS Stream users can 
> be added.
> 
> Please let me know what you think about adding this goal to the proposed 
> SIG. Of course I'm especially interested in comments/opinions by people 
> currently involved in building the CentOS Plus kernel and/or ELRepo.
> 
> Thanks!
> 
> Peter
> 

Hi Peter,

Packaging of external kernel modules for Stream presents a dilemma. IMHO 
you first need to decide if you are packaging for Stream as an end user 
product, or if you are packaging for Stream as a development product for 
RHEL.

If it's the latter, then the established procedure is to use 
kABI-tracking kmod packages, and any development work done in Stream 
should do likewise otherwise it may have little value in RHEL.

However, if you are packaging for Stream as an end user product, using 
kABI-tracking kmod packages makes little sense as the kABI is not 
stable. For example, of the elrepo kmods we have tested, around a 
quarter broke on the first Stream kernel update (-257) and by the latest 
update (-277), only 5 packages out of ~50 remain intact.

Therefore, DKMS seems a better technical fit for the constantly changing 
ABI of the Stream kernel. One of the main objections to using DKMS in 
Enterprise Linux was always the need for having compiler and other 
development tools installed on production systems. As Stream is not 
intended for production-ready Enterprise Linux use, but is rather a 
'beta' type development product, DKMS seems a better technical fit in 
these circumstances but such packages are unlikely to find a place in RHEL.

Hence I think the first thing you will need to decide is if you are 
developing for Stream as an end-user product or as an upstream of RHEL, 
and that will largely dictate the route you take. For me, the whole 
point of Stream seems to be as a place to contribute to upstream 
development of RHEL, and as such it is kmods that are exclusively used 
in RHEL.

As a side note, you mentioned the use case of drivers that are disabled 
by Red Hat in the RHEL kernel. What I would like to see is the 
opportunity for the community (a SIG) to maintain these drivers within 
the upstream Stream kernel (negating the need for a Stream-Plus kernel*) 
and then Red Hat can disable them downstream for RHEL if it doesn't wish 
to support them, rather than us (the community) constantly having to 
re-enable (undo) something Red Hat has decided we do not want/need and 
has disabled. But that can't realistically happen until we have a 
totally open git repository for the kernel and all commits whereby Red 
Hat can apply RHEL-specific patches downstream of Stream and what SIGs 
like this are doing.

Phil

[*] Having a Plus kernel in a downstream product such as CentOS Linux 
makes perfect sense, but it makes absolutely no sense where Stream is 
the upstream product. RHEL now becomes the downstream, and they are the 
ones who need a 'Plus' kernel for RHEL, by forking/submitting patches 
against the Stream kernel to remove/disable the bits they don't want in 
RHEL. Moving CentOS-Stream upstream of RHEL essentially flips the 'Plus' 
kernel relationship on it's head.