On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote: > On Saturday, February 13, 2021 2:45 PM, Peter Georg <peter.georg at physik.uni-regensburg.de> wrote: > >> On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote: >> >>> There are four main goals of this SIG: >>> First, break down changes in the Stream version 4.18.0 kernel into individual patches. When possible, document the source the patch is from and the purpose of the patch. Moving forward also document which Stream kernel package revision introduced the patch. >>> Second, build CentOS Plus kernel for Stream. >>> Third, package newer LTS kernels during the LTS lifespan. These would be available as alternative kernels for Stream similar to the CentOS Plus kernel. >>> Fourth, work on tools for better delivery of live security patches to be applied via kpatch. >>> The reasoning for this SIG is to improve the openness of kernels available through Stream. >>> Normally a distribution upstream would have access to which patch or commit each addition to the kernel belongs to. The patch name or commit comment would also give information as to the purpose of the patch. Stream's kernel reports to be 4.18.0 but uses a tar that is 110% the size of a true 4.18.0 tar. A breakdown of the additional 10% is not provided. Hence Stream's kernel still functions similar to a downstream rather than providing what is expected for a successful upstream project. >>> This SIG would help close the kernel openness gap by documenting the changes. Stream users would then be in a better position to track down individual patches that cause regressions or bugs. >>> This SIG would also help smooth the adoption of Stream for users that depend on CentOS Plus kernel features. >>> Next, this SIG should help improve feedback to the LTS kernel developers. Tracking LTS kernel changes may also help with documenting some changes in the primary Stream kernel. >>> Lastly, this SIG should help improve kernel security by promoting methods to enable kernel security updates even when a reboot of the system is not possible. >> >> Replying directly to the head of this topic now as I'm not sure which >> branch it fits best. >> >> As announced by Brian Stinson, he was available on IRC yesterday (twice) >> to discuss anything related to kernels and this proposal in particular. >> Sadly there have only been a few people to discuss this matter. >> >> Most of the discussion was indeed about the fifth main goal that I >> proposed in a follow up mail to the original proposal: Package external >> kernel modules >> >> This does not mean that the other proposed goals are not worth >> investigating, but simply not enough people have shown up on IRC >> yesterday to discuss these. >> >> Brian also mentioned that there might actually be fewer challenges >> blocking a SIG to build kernel modules in CBS than he initially thought >> (especially if secureboot is not a hard requirement, i.e. can be tackled >> at a later stage). So we might be able to start earlier than expected. >> >> It seems like starting with packaging external kernel modules is a good >> starting point for the proposed SIG. Hence we concluded to start >> gathering a list of kernel modules people are interested in building for >> CentOS Stream here on the mailing list. Of course this is not a pure >> wish list, but people should also commit to be part of the process / >> SIG. On IRC the following kernel modules have been mentioned / discussed >> as potential candidates to be built by a kernel SIG. I added information >> where these modules can currently be retrieved for RHEL and CentOS Linux >> (only one source for each module). However these are probably not >> compatible with CentOS Stream. >> >> - isci (ELRepo, kmod) >> - mlx4 (ELRepo, kmod) >> - lustre-client (Lustre upstream, kmod and dkms) >> - nvidia (nVidia, kmod and dkms) >> - rocm (AMD, dkms) >> - wireguard (ELRepo, kmod) >> >> Please feel free to extend this list with modules you are interested in. >> I suggest you also add the source you currently get these modules (for >> RHEL/CentOS Linux) from if any is available. > > I am glad you found IRC to be productive forum to move forward on the topic of kernel modules. > > It does not seem like the best place to get answers to what I am looking for. > > Mike McGrath wrote back on Jan 20th the following: > >> "I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way). There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks." > > It has been "a few weeks" and there has been no update. There is no details on what the one or two things still left to configure are. Additionally, there is no status on how the one or two things are progressing. > > So far, the CentOS governance board has established only *ONE* hard deadline for the transition which is the Dec 31st termination of CentOS 8 in favor of Stream. All other deliverables to make that transition successful remain vague and unspecified. I haven't even found a public list of all the items Red Hat considers to be deliverables as part of this transition. > > Brian Stinson's offer to talk on IRC, while helpful in other aspects, gave no indication of being able to address this fundamental reality. It left me with doubts on if the disappointing follow-through on the part of Mike McGrath could be meaningfully discussed via that forum. All I was looking for was at least a status update from Mike McGrath after a few weeks and even that was not provided. > > If you want me to contribute to the discussion of packaging additional kernel modules, here is all I have to say right now: > > First, documenting the Stream kernel modifications needs to be the top priority. If a kernel crash dump points to code that doesn't belong to vanilla 4.18.0, we should already have the resources to understand what the code/patch that cause the crash dump is. That includes being able to investigate patches which cause bad interactions with other kernel modules we claim we will be supporting. > > Second, if Stream is going to release new kernels more frequently than CentOS 8 then I agree this should be a goal of the Kernel SIG to address. > > Third, binary only kernel module under restrictive licensing terms should never be promoted/packaged by the Kernel SIG. > > Specifically, NVIDIA license mandates security problems for the victims that are lured into using their BLOB of a driver. > > These drivers have a history of creating security issues including CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056. > > nVidia mandates it is ILLEGAL to compare the object code differences between driver versions making it illegal to meaningfully address them through kpatch. Hence, there will NEVER be live security patching of the security issues NVIDIA inflicts upon Stream users. > > There are also other issues with the license. It might be legal to perform fuzzing against the NVIDIA driver, but if you find an issue then you can't reverse engineer the impacted code to explore if it creates a larger issue. You might believe a problem which causes a kernel crash could also lead to a buffer overflow to run arbitrary code in kernel space but under the terms of the license you must operate with both hands tied behind your back. Instead, you have to submit the issue to NVIDIA and hope the investigate it further to determine the full extent of the security issue. It seems to me they have a conflict of interest in setting the timeline to investigate and adddress the problem when they have a monopoly on being able to look and modify. > > The nouveau drivers are provided under superior licensing terms and do not inflict any of the draconian terms of the NVIDIA drivers. It is my expectation to be able to someday address security issues with nouveau drivers via kpatch. Thanks for your input. I agree that there are licensing issues, especially concerning the nvidia drivers. That's something we have to think about, hence I'd suggest to move adding these kernel modules to a later stage and start with kernel modules without this kind of issues. Actually the nvidia driver is the only one on the current list with licensing issues. Peter > Back to the topic of IRC, when Stream and the Kernel SIG are in a more mature state, I will feel more comfortable discussing things via IRC. Until then using the mailing list remains the best way to contact me. Please don't misrepresent that as a lack of interest.