[CentOS-devel] [EXT] Re: RFC: Stream Kernel SIG Proposal

On 11/02/2021 10.37, Phil Perry wrote:
> On 11/02/2021 00:03, Peter Georg wrote:
>>
>>
>> On 10/02/2021 22.51, Brian Stinson wrote:
>>>
>>> On Wed, Feb 10, 2021, at 13:17, redbaronbrowser via CentOS-devel wrote:
>>>> On Wednesday, January 20, 2021 9:31 AM, Mike McGrath 
>>>> <mmcgrath at redhat.com> wrote:
>>>>
>>>>> I've confirmed with the team, the git repo is going to be all the 
>>>>> normal git patches you would expect (IE: not arbitrarily munged 
>>>>> together in some way).  There's one or two more things they're 
>>>>> configuring with gitlab and they expect to have an actual repo that 
>>>>> you can look at / poke at to validate what I'm saying in a few weeks.
>>>>>
>>>>>             -Mike
>>>>
>>>> Mike, is there any status update on individual patches applied to 
>>>> Stream's kernel?
>>>>
>>>> Can we move forward with this SIG so I can publish the information I 
>>>> already have?
>>>>
>>>> What about a Stream Plus kernel?  The directory for Stream still has 
>>>> no packages at all.
>>>>
>>>> Or delivering kpatch live security updates without a reboot?
>>>>
>>>> I am ready to move forward immediately if the SIG is approved.  Is 
>>>> there a reason to wait another "few weeks"?
>>>> _______________________________________________
>>>> CentOS-devel mailing list
>>>> CentOS-devel at centos.org
>>>> https://lists.centos.org/mailman/listinfo/centos-devel
>>>>
>>>
>>> I see your questions roughly grouped in 3 categories, correct me if 
>>> I've misinterpreted:
>>>
>>> - How do I get patches into the Stream kernel *now*?
>>> Have you submitted any bugs against CentOS Stream on 
>>> bugzilla.redhat.com with your patches? I want to make sure, first, 
>>> that we have an actual answer because some of them may indeed be 
>>> workable for inclusion in RHEL.
>>>
>>> All contributions for RHEL/Stream 8 (not just the kernel) are flowing 
>>> through this bugzilla process for right now.
>>>
>>> - When can we see the new kernel workflows?
>>> I don't have specific date/time when we expect to see the new-way of 
>>> doing kernel repos but I would check https://gitlab.com/redhat/ 
>>> towards the end of the month. Those will be targeted at RHEL/Stream 9 
>>> kernels. I'm asking around about some of the work we're doing to make 
>>> the Stream 8 kernel sources more accessible in a format that isn't 
>>> just tarballs.
>>>
>>> - How do I get kernels that are not shipped in-distro?
>>> Here's the problem as it exists right now, Kernels typically need 
>>> secureboot signatures on them, and we don't have the infrastructure 
>>> we need to make those signatures happen in the community build 
>>> system. We're talking about ways to enable that directly in CBS, but 
>>> in the meantime we plan to take community-based kernels and assist 
>>> with getting builds done on infrastructure that can do signing. Note: 
>>> this affects other packages like fwupd, grub, etc. We are absolutely 
>>> open to continuing to build the Plus kernel. Would it make sense to 
>>> get yourself, @toracat for the Plus kernel, and whoever else is 
>>> interested in building kernels together on IRC to talk about how to 
>>> handle coordination here?
>>
>> Please add me to the list of interested people (@pjgeorg in IRC).
>>
>> Let me note that I have a different goal that might be added to this 
>> SIG and is not part of the four goals listed in the initial proposal:
>>
>> 5. Package external kernel modules
>>
>> So far (running CentOS Linux) I have been using various different 
>> sources for kmod packages (and built some locally). However many of 
>> these officially only support RHEL (most prominent on this list: 
>> ELRepo. At least according to recent comments by Phil Perry. I haven't 
>> seen any comments by other contributors). For the time being I "fix" 
>> this locally by using dkms packages if avaible or maintain dkms 
>> packages myself for modules that are otherwise only provided as kmod 
>> packages. I really dislike doing such a job locally without any open 
>> collaboration. Just feels wrong working with open source software 
>> without trying to collaborate "upstream". Hence my interest in 
>> contributing to the proposed SIG.
>>
>> At first I'd suggest starting with drivers included in upstream kernel 
>> but disabled by Red Hat. Yes, these are also available by using the 
>> CentOS Plus kernel. However I, and maybe other users as well, prefer 
>> to run the "vanilla" CentOS Stream kernel and add missing drivers as 
>> external kernel modules as required. And of course, ELRepo currently 
>> provides exactly this. However most of these can sadly not be used for 
>> CentOS Stream. This also applies to building LTS kernel packages for 
>> CentOS Stream. As Phil Perry mentioned this is and has already been 
>> done by ELRepo for some time. I agree with Matthew Miller here: These 
>> things should be connected. On a side note: The Hyperscale SIG seems 
>> to be interested in providing LTS kernel(s) for CentOS Stream as well 
>> (has been mentioned in their talk at the CentOS Dojo last week).
>>
>> Once everything is set up and working for these simpler cases, 
>> packages for any other kernel modules that are of use for CentOS 
>> Stream users can be added.
>>
>> Please let me know what you think about adding this goal to the 
>> proposed SIG. Of course I'm especially interested in comments/opinions 
>> by people currently involved in building the CentOS Plus kernel and/or 
>> ELRepo.
>>
>> Thanks!
>>
>> Peter
>>
> 
> Hi Peter,

Hi Phil,

Thanks for your response! Please see my comments inline below.

> Packaging of external kernel modules for Stream presents a dilemma. IMHO 
> you first need to decide if you are packaging for Stream as an end user 
> product, or if you are packaging for Stream as a development product for 
> RHEL.

I intend to package for Stream as an end user product.

> If it's the latter, then the established procedure is to use 
> kABI-tracking kmod packages, and any development work done in Stream 
> should do likewise otherwise it may have little value in RHEL.
> 
> However, if you are packaging for Stream as an end user product, using 
> kABI-tracking kmod packages makes little sense as the kABI is not 
> stable. For example, of the elrepo kmods we have tested, around a 
> quarter broke on the first Stream kernel update (-257) and by the latest 
> update (-277), only 5 packages out of ~50 remain intact.

My idea was that doing this packaging within a CentOS Stream SIG 
*should* hopefully allow us to easily re-build for new kernel releases. 
In a perfect world this would be done automatically at some point and 
human interaction would only be required in case of an error. Of course 
this only works as expected in case of errors happening rarely and new 
kernel releases not showing up every day. The current frequency of new 
kernel releases for CentOS Stream is actually not as high as I first 
expected it to be.


> Therefore, DKMS seems a better technical fit for the constantly changing 
> ABI of the Stream kernel. One of the main objections to using DKMS in 
> Enterprise Linux was always the need for having compiler and other 
> development tools installed on production systems. As Stream is not 
> intended for production-ready Enterprise Linux use, but is rather a 
> 'beta' type development product, DKMS seems a better technical fit in 
> these circumstances but such packages are unlikely to find a place in RHEL.

That's the reason why I currently chose to use DKMS for all kernel 
modules I need. I sent an e-mail with similar technical arguments to 
centos-questions last year, arguing that DKMS support should be added to 
CentOS Stream (currently provided by EPEL). However the main objections 
to using DKMS in the first place still apply.


> Hence I think the first thing you will need to decide is if you are 
> developing for Stream as an end-user product or as an upstream of RHEL, 
> and that will largely dictate the route you take. For me, the whole 
> point of Stream seems to be as a place to contribute to upstream 
> development of RHEL, and as such it is kmods that are exclusively used 
> in RHEL.

This really seems to be the main "issue" with CentOS Stream nowadays: 
One group of users is using it as an upstream development of RHEL, the 
other group is using / is planning to use it as an end-user product 
replacing CentOS Linux. Talking with someone about CentOS Stream one 
should always ask this question first.

I'm part of the latter.


> As a side note, you mentioned the use case of drivers that are disabled 
> by Red Hat in the RHEL kernel. What I would like to see is the 
> opportunity for the community (a SIG) to maintain these drivers within 
> the upstream Stream kernel (negating the need for a Stream-Plus kernel*) 
> and then Red Hat can disable them downstream for RHEL if it doesn't wish 
> to support them, rather than us (the community) constantly having to 
> re-enable (undo) something Red Hat has decided we do not want/need and 
> has disabled. But that can't realistically happen until we have a 
> totally open git repository for the kernel and all commits whereby Red 
> Hat can apply RHEL-specific patches downstream of Stream and what SIGs 
> like this are doing.

While this sounds like a reasonable idea to me, I can easily see why 
various stakeholders would disagree: This would complicate switching 
from CentOS Stream to RHEL for users as they might end up with a system 
not working anymore due to missing hardware support. Both CentOS Stream 
and RHEL running a similar kernel with the same config it is very likely 
that you can switch from CentOS Stream to RHEL after simply checking 
that all packages you have installed are available for both (in case of 
having enabled non-default repositories for CentOS Stream and/or RHEL).


Peter

> 
> Phil
> 
> [*] Having a Plus kernel in a downstream product such as CentOS Linux 
> makes perfect sense, but it makes absolutely no sense where Stream is 
> the upstream product. RHEL now becomes the downstream, and they are the 
> ones who need a 'Plus' kernel for RHEL, by forking/submitting patches 
> against the Stream kernel to remove/disable the bits they don't want in 
> RHEL. Moving CentOS-Stream upstream of RHEL essentially flips the 'Plus' 
> kernel relationship on it's head.
> 
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel