[CentOS-devel] [EXT] Re: RFC: Stream Kernel SIG Proposal

Tue May 18 10:43:12 UTC 2021
Peter Georg <peter.georg at physik.uni-regensburg.de>

On 18/02/2021 18.13, Peter Georg wrote:
> On 15/02/2021 15.52, Peter Georg wrote:
>> On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote:
>>> On Saturday, February 13, 2021 2:45 PM, Peter Georg 
>>> <peter.georg at physik.uni-regensburg.de> wrote:
>>>> On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote:
>>>>> There are four main goals of this SIG:
>>>>> First, break down changes in the Stream version 4.18.0 kernel into 
>>>>> individual patches. When possible, document the source the patch is 
>>>>> from and the purpose of the patch. Moving forward also document 
>>>>> which Stream kernel package revision introduced the patch.
>>>>> Second, build CentOS Plus kernel for Stream.
>>>>> Third, package newer LTS kernels during the LTS lifespan. These 
>>>>> would be available as alternative kernels for Stream similar to the 
>>>>> CentOS Plus kernel.
>>>>> Fourth, work on tools for better delivery of live security patches 
>>>>> to be applied via kpatch.
>>>>> The reasoning for this SIG is to improve the openness of kernels 
>>>>> available through Stream.
>>>>> Normally a distribution upstream would have access to which patch 
>>>>> or commit each addition to the kernel belongs to. The patch name or 
>>>>> commit comment would also give information as to the purpose of the 
>>>>> patch. Stream's kernel reports to be 4.18.0 but uses a tar that is 
>>>>> 110% the size of a true 4.18.0 tar. A breakdown of the additional 
>>>>> 10% is not provided. Hence Stream's kernel still functions similar 
>>>>> to a downstream rather than providing what is expected for a 
>>>>> successful upstream project.
>>>>> This SIG would help close the kernel openness gap by documenting 
>>>>> the changes. Stream users would then be in a better position to 
>>>>> track down individual patches that cause regressions or bugs.
>>>>> This SIG would also help smooth the adoption of Stream for users 
>>>>> that depend on CentOS Plus kernel features.
>>>>> Next, this SIG should help improve feedback to the LTS kernel 
>>>>> developers. Tracking LTS kernel changes may also help with 
>>>>> documenting some changes in the primary Stream kernel.
>>>>> Lastly, this SIG should help improve kernel security by promoting 
>>>>> methods to enable kernel security updates even when a reboot of the 
>>>>> system is not possible.
>>>> Replying directly to the head of this topic now as I'm not sure which
>>>> branch it fits best.
>>>> As announced by Brian Stinson, he was available on IRC yesterday 
>>>> (twice)
>>>> to discuss anything related to kernels and this proposal in particular.
>>>> Sadly there have only been a few people to discuss this matter.
>>>> Most of the discussion was indeed about the fifth main goal that I
>>>> proposed in a follow up mail to the original proposal: Package external
>>>> kernel modules
>>>> This does not mean that the other proposed goals are not worth
>>>> investigating, but simply not enough people have shown up on IRC
>>>> yesterday to discuss these.
>>>> Brian also mentioned that there might actually be fewer challenges
>>>> blocking a SIG to build kernel modules in CBS than he initially thought
>>>> (especially if secureboot is not a hard requirement, i.e. can be 
>>>> tackled
>>>> at a later stage). So we might be able to start earlier than expected.
>>>> It seems like starting with packaging external kernel modules is a good
>>>> starting point for the proposed SIG. Hence we concluded to start
>>>> gathering a list of kernel modules people are interested in building 
>>>> for
>>>> CentOS Stream here on the mailing list. Of course this is not a pure
>>>> wish list, but people should also commit to be part of the process /
>>>> SIG. On IRC the following kernel modules have been mentioned / 
>>>> discussed
>>>> as potential candidates to be built by a kernel SIG. I added 
>>>> information
>>>> where these modules can currently be retrieved for RHEL and CentOS 
>>>> Linux
>>>> (only one source for each module). However these are probably not
>>>> compatible with CentOS Stream.
>>>> -   isci (ELRepo, kmod)
>>>> -   mlx4 (ELRepo, kmod)
>>>> -   lustre-client (Lustre upstream, kmod and dkms)
>>>> -   nvidia (nVidia, kmod and dkms)
>>>> -   rocm (AMD, dkms)
>>>> -   wireguard (ELRepo, kmod)
>>>>      Please feel free to extend this list with modules you are 
>>>> interested in.
>>>>      I suggest you also add the source you currently get these 
>>>> modules (for
>>>>      RHEL/CentOS Linux) from if any is available.
>>> I am glad you found IRC to be productive forum to move forward on the 
>>> topic of kernel modules.
>>> It does not seem like the best place to get answers to what I am 
>>> looking for.
>>> Mike McGrath wrote back on Jan 20th the following:
>>>> "I've confirmed with the team, the git repo is going to be all the 
>>>> normal git patches you would expect (IE: not arbitrarily munged 
>>>> together in some way).  There's one or two more things they're 
>>>> configuring with gitlab and they expect to have an actual repo that 
>>>> you can look at / poke at to validate what I'm saying in a few weeks."
>>> It has been "a few weeks" and there has been no update.  There is no 
>>> details on what the one or two things still left to configure are. 
>>> Additionally, there is no status on how the one or two things are 
>>> progressing.
>>> So far, the CentOS governance board has established only *ONE* hard 
>>> deadline for the transition which is the Dec 31st termination of 
>>> CentOS 8 in favor of Stream.  All other deliverables to make that 
>>> transition successful remain vague and unspecified.  I haven't even 
>>> found a public list of all the items Red Hat considers to be 
>>> deliverables as part of this transition.
>>> Brian Stinson's offer to talk on IRC, while helpful in other aspects, 
>>> gave no indication of being able to address this fundamental reality. 
>>> It left me with doubts on if the disappointing follow-through on the 
>>> part of Mike McGrath could be meaningfully discussed via that forum. 
>>> All I was looking for was at least a status update from Mike McGrath 
>>> after a few weeks and even that was not provided.
>>> If you want me to contribute to the discussion of packaging 
>>> additional kernel modules, here is all I have to say right now:
>>> First, documenting the Stream kernel modifications needs to be the 
>>> top priority.  If a kernel crash dump points to code that doesn't 
>>> belong to vanilla 4.18.0, we should already have the resources to 
>>> understand what the code/patch that cause the crash dump is.  That 
>>> includes being able to investigate patches which cause bad 
>>> interactions with other kernel modules we claim we will be supporting.
>>> Second, if Stream is going to release new kernels more frequently 
>>> than CentOS 8 then I agree this should be a goal of the Kernel SIG to 
>>> address.
>>> Third, binary only kernel module under restrictive licensing terms 
>>> should never be promoted/packaged by the Kernel SIG.
>>> Specifically, NVIDIA license mandates security problems for the 
>>> victims that are lured into using their BLOB of a driver.
>>> These drivers have a history of creating security issues including 
>>> CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056.
>>> nVidia mandates it is ILLEGAL to compare the object code differences 
>>> between driver versions making it illegal to meaningfully address 
>>> them through kpatch.  Hence, there will NEVER be live security 
>>> patching of the security issues NVIDIA inflicts upon Stream users.
>>> There are also other issues with the license.  It might be legal to 
>>> perform fuzzing against the NVIDIA driver, but if you find an issue 
>>> then you can't reverse engineer the impacted code to explore if it 
>>> creates a larger issue.  You might believe a problem which causes a 
>>> kernel crash could also lead to a buffer overflow to run arbitrary 
>>> code in kernel space but under the terms of the license you must 
>>> operate with both hands tied behind your back.  Instead, you have to 
>>> submit the issue to NVIDIA and hope the investigate it further to 
>>> determine the full extent of the security issue.  It seems to me they 
>>> have a conflict of interest in setting the timeline to investigate 
>>> and adddress the problem when they have a monopoly on being able to 
>>> look and modify.
>>> The nouveau drivers are provided under superior licensing terms and 
>>> do not inflict any of the draconian terms of the NVIDIA drivers.  It 
>>> is my expectation to be able to someday address security issues with 
>>> nouveau drivers via kpatch.
>> Thanks for your input. I agree that there are licensing issues, 
>> especially concerning the nvidia drivers. That's something we have to 
>> think about, hence I'd suggest to move adding these kernel modules to 
>> a later stage and start with kernel modules without this kind of 
>> issues. Actually the nvidia driver is the only one on the current list 
>> with licensing issues.
>> Peter
>>> Back to the topic of IRC, when Stream and the Kernel SIG are in a 
>>> more mature state, I will feel more comfortable discussing things via 
>>> IRC. Until then using the mailing list remains the best way to 
>>> contact me. Please don't misrepresent that as a lack of interest.
> Responding to my own message as there seems to be low interest.
> @Brian Stinson: Any suggestions on how we could proceed in this matter?
> I'm not sure why there is so little interest in this endeavor. There at 
> least seems to be some demand for kmods to support Hardware not 
> supported by the CentOS Stream kernel (see question about ELRepo on 
> #centos-devel few days ago). Having the recent announcement about EOL of 
> CentOS Linux 8 in mind, I assume a possible explanations (at least for 
> this case) is that (CentOS Linux 7/8) users:
> 1. haven't decided yet what to do once CentOS Linux 7/8 is EOL.
> 2. are planning to move to a free RHEL subscription or an other RHEL 
> downstream rebuild (AlmaLinux, RockyLinux, etc.).
> 3. tried to run CentOS Stream 8 and realized these kmod packages 
> provided by ELRepo and others are not working anymore. These probably 
> stick with CentOS Linux 8 for now and switch
>    a) to an alternative RHEL downstream rebuild (AlmaLinux, RockyLinux, 
> etc.) before EOL (supported by ELRepo and others);
>    b) to CentOS Stream 8 once someone has fixed this issue.
> Just my guess. Anyway, to allow users currently running CentOS Linux 8 
> (i) with kmods provided by ELRepo and others or (ii) running the CentOS 
> Linux Plus kernel to switch to CentOS Stream 8 without changing 
> hardware, one should either provide (i) an alternate source for these 
> kmods or (ii) build CentOS Stream Plus kernel.
> Has anyone heard from Akemi / toracat about plans concerning building 
> the plus kernel for CentOS Stream?
> As I already mentioned in one of my previous emails, I do need some of 
> these kmods locally and hence build them for CentOS Stream anyway.
> I do not mind the extra work (actually it might be lower once all ist 
> set up) to provide the kmods to other CentOS Stream users as well.
> I currently have SPECS ready for:
> - Rebuilds of upstream drivers supported by RHEL with (almost all) 
> deprecated adapters [1] re-enabled: aacraid, be2iscsi, be2net, lpfc, 
> megaraid, mlx4, mpt3sas, qla2xxx, qla4xxx
> - Upstream drivers not supported by RHEL: isci, sky2
> - Wireguard

Short update on this matter:
I have updated the packages listed above (and a few more) locally for 
every kernel release since February: So far there have only been two 
cases that required manual work: i) megaraid patches had to be adapted 
for kernel 4.18.0-301.1 as previously deprecated and disabled adapters 
have been re-enabled. ii) wireguard had to be updated to the latest 
upstream release to work with kernel 4.18.0-301.1 (the first kernel 
release that changed RHEL_MINOR to 5).

In all other cases I have been able to automate rebuilding these kernel 
modules for every newly released kernel version. Hence I see no reason 
why this could not also be done within CBS.

Concerning the CBS infrastructure I assume there are no issues either, 
or at least no issues the Hyperscale SIG doesn't have to deal with as 
well to provide an alternative kernel. Note: I'm currently not signing 
any kmod.

So in case there is any way to move this SIG proposal forward, or some 
other more suitable way (under the umbrella of centos-plus or some other 
SIG?) to provide such kmod packages, please let me know.


> The second group of packages is actually more complicated to maintain 
> than the first. Wireguard and others highly depend on upstream support.
> Is there any reason why one should not be able to build these packages 
> in CBS? Note: The kernel modules are currently not being signed.
> Just let me know how I can move this proposal forward. If there is no 
> interest by anyone, I'll just keep on building the packages I do require 
> in private.
> Best
> Peter
> [1] 
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.9_release_notes/deprecated_functionality#deprecated_adapters 
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel