On 15/02/2021 15.52, Peter Georg wrote: > On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote: >> On Saturday, February 13, 2021 2:45 PM, Peter Georg >> <peter.georg at physik.uni-regensburg.de> wrote: >> >>> On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote: >>> >>>> There are four main goals of this SIG: >>>> First, break down changes in the Stream version 4.18.0 kernel into >>>> individual patches. When possible, document the source the patch is >>>> from and the purpose of the patch. Moving forward also document >>>> which Stream kernel package revision introduced the patch. >>>> Second, build CentOS Plus kernel for Stream. >>>> Third, package newer LTS kernels during the LTS lifespan. These >>>> would be available as alternative kernels for Stream similar to the >>>> CentOS Plus kernel. >>>> Fourth, work on tools for better delivery of live security patches >>>> to be applied via kpatch. >>>> The reasoning for this SIG is to improve the openness of kernels >>>> available through Stream. >>>> Normally a distribution upstream would have access to which patch or >>>> commit each addition to the kernel belongs to. The patch name or >>>> commit comment would also give information as to the purpose of the >>>> patch. Stream's kernel reports to be 4.18.0 but uses a tar that is >>>> 110% the size of a true 4.18.0 tar. A breakdown of the additional >>>> 10% is not provided. Hence Stream's kernel still functions similar >>>> to a downstream rather than providing what is expected for a >>>> successful upstream project. >>>> This SIG would help close the kernel openness gap by documenting the >>>> changes. Stream users would then be in a better position to track >>>> down individual patches that cause regressions or bugs. >>>> This SIG would also help smooth the adoption of Stream for users >>>> that depend on CentOS Plus kernel features. >>>> Next, this SIG should help improve feedback to the LTS kernel >>>> developers. Tracking LTS kernel changes may also help with >>>> documenting some changes in the primary Stream kernel. >>>> Lastly, this SIG should help improve kernel security by promoting >>>> methods to enable kernel security updates even when a reboot of the >>>> system is not possible. >>> >>> Replying directly to the head of this topic now as I'm not sure which >>> branch it fits best. >>> >>> As announced by Brian Stinson, he was available on IRC yesterday (twice) >>> to discuss anything related to kernels and this proposal in particular. >>> Sadly there have only been a few people to discuss this matter. >>> >>> Most of the discussion was indeed about the fifth main goal that I >>> proposed in a follow up mail to the original proposal: Package external >>> kernel modules >>> >>> This does not mean that the other proposed goals are not worth >>> investigating, but simply not enough people have shown up on IRC >>> yesterday to discuss these. >>> >>> Brian also mentioned that there might actually be fewer challenges >>> blocking a SIG to build kernel modules in CBS than he initially thought >>> (especially if secureboot is not a hard requirement, i.e. can be tackled >>> at a later stage). So we might be able to start earlier than expected. >>> >>> It seems like starting with packaging external kernel modules is a good >>> starting point for the proposed SIG. Hence we concluded to start >>> gathering a list of kernel modules people are interested in building for >>> CentOS Stream here on the mailing list. Of course this is not a pure >>> wish list, but people should also commit to be part of the process / >>> SIG. On IRC the following kernel modules have been mentioned / discussed >>> as potential candidates to be built by a kernel SIG. I added information >>> where these modules can currently be retrieved for RHEL and CentOS Linux >>> (only one source for each module). However these are probably not >>> compatible with CentOS Stream. >>> >>> - isci (ELRepo, kmod) >>> - mlx4 (ELRepo, kmod) >>> - lustre-client (Lustre upstream, kmod and dkms) >>> - nvidia (nVidia, kmod and dkms) >>> - rocm (AMD, dkms) >>> - wireguard (ELRepo, kmod) >>> >>> Please feel free to extend this list with modules you are >>> interested in. >>> I suggest you also add the source you currently get these >>> modules (for >>> RHEL/CentOS Linux) from if any is available. >> >> I am glad you found IRC to be productive forum to move forward on the >> topic of kernel modules. >> >> It does not seem like the best place to get answers to what I am >> looking for. >> >> Mike McGrath wrote back on Jan 20th the following: >> >>> "I've confirmed with the team, the git repo is going to be all the >>> normal git patches you would expect (IE: not arbitrarily munged >>> together in some way). There's one or two more things they're >>> configuring with gitlab and they expect to have an actual repo that >>> you can look at / poke at to validate what I'm saying in a few weeks." >> >> It has been "a few weeks" and there has been no update. There is no >> details on what the one or two things still left to configure are. >> Additionally, there is no status on how the one or two things are >> progressing. >> >> So far, the CentOS governance board has established only *ONE* hard >> deadline for the transition which is the Dec 31st termination of >> CentOS 8 in favor of Stream. All other deliverables to make that >> transition successful remain vague and unspecified. I haven't even >> found a public list of all the items Red Hat considers to be >> deliverables as part of this transition. >> >> Brian Stinson's offer to talk on IRC, while helpful in other aspects, >> gave no indication of being able to address this fundamental reality. >> It left me with doubts on if the disappointing follow-through on the >> part of Mike McGrath could be meaningfully discussed via that forum. >> All I was looking for was at least a status update from Mike McGrath >> after a few weeks and even that was not provided. >> >> If you want me to contribute to the discussion of packaging additional >> kernel modules, here is all I have to say right now: >> >> First, documenting the Stream kernel modifications needs to be the top >> priority. If a kernel crash dump points to code that doesn't belong >> to vanilla 4.18.0, we should already have the resources to understand >> what the code/patch that cause the crash dump is. That includes being >> able to investigate patches which cause bad interactions with other >> kernel modules we claim we will be supporting. >> >> Second, if Stream is going to release new kernels more frequently than >> CentOS 8 then I agree this should be a goal of the Kernel SIG to address. >> >> Third, binary only kernel module under restrictive licensing terms >> should never be promoted/packaged by the Kernel SIG. >> >> Specifically, NVIDIA license mandates security problems for the >> victims that are lured into using their BLOB of a driver. >> >> These drivers have a history of creating security issues including >> CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056. >> >> nVidia mandates it is ILLEGAL to compare the object code differences >> between driver versions making it illegal to meaningfully address them >> through kpatch. Hence, there will NEVER be live security patching of >> the security issues NVIDIA inflicts upon Stream users. >> >> There are also other issues with the license. It might be legal to >> perform fuzzing against the NVIDIA driver, but if you find an issue >> then you can't reverse engineer the impacted code to explore if it >> creates a larger issue. You might believe a problem which causes a >> kernel crash could also lead to a buffer overflow to run arbitrary >> code in kernel space but under the terms of the license you must >> operate with both hands tied behind your back. Instead, you have to >> submit the issue to NVIDIA and hope the investigate it further to >> determine the full extent of the security issue. It seems to me they >> have a conflict of interest in setting the timeline to investigate and >> adddress the problem when they have a monopoly on being able to look >> and modify. >> >> The nouveau drivers are provided under superior licensing terms and do >> not inflict any of the draconian terms of the NVIDIA drivers. It is >> my expectation to be able to someday address security issues with >> nouveau drivers via kpatch. > > Thanks for your input. I agree that there are licensing issues, > especially concerning the nvidia drivers. That's something we have to > think about, hence I'd suggest to move adding these kernel modules to a > later stage and start with kernel modules without this kind of issues. > Actually the nvidia driver is the only one on the current list with > licensing issues. > > Peter > > >> Back to the topic of IRC, when Stream and the Kernel SIG are in a more >> mature state, I will feel more comfortable discussing things via IRC. >> Until then using the mailing list remains the best way to contact me. >> Please don't misrepresent that as a lack of interest. Responding to my own message as there seems to be low interest. @Brian Stinson: Any suggestions on how we could proceed in this matter? I'm not sure why there is so little interest in this endeavor. There at least seems to be some demand for kmods to support Hardware not supported by the CentOS Stream kernel (see question about ELRepo on #centos-devel few days ago). Having the recent announcement about EOL of CentOS Linux 8 in mind, I assume a possible explanations (at least for this case) is that (CentOS Linux 7/8) users: 1. haven't decided yet what to do once CentOS Linux 7/8 is EOL. 2. are planning to move to a free RHEL subscription or an other RHEL downstream rebuild (AlmaLinux, RockyLinux, etc.). 3. tried to run CentOS Stream 8 and realized these kmod packages provided by ELRepo and others are not working anymore. These probably stick with CentOS Linux 8 for now and switch a) to an alternative RHEL downstream rebuild (AlmaLinux, RockyLinux, etc.) before EOL (supported by ELRepo and others); b) to CentOS Stream 8 once someone has fixed this issue. Just my guess. Anyway, to allow users currently running CentOS Linux 8 (i) with kmods provided by ELRepo and others or (ii) running the CentOS Linux Plus kernel to switch to CentOS Stream 8 without changing hardware, one should either provide (i) an alternate source for these kmods or (ii) build CentOS Stream Plus kernel. Has anyone heard from Akemi / toracat about plans concerning building the plus kernel for CentOS Stream? As I already mentioned in one of my previous emails, I do need some of these kmods locally and hence build them for CentOS Stream anyway. I do not mind the extra work (actually it might be lower once all ist set up) to provide the kmods to other CentOS Stream users as well. I currently have SPECS ready for: - Rebuilds of upstream drivers supported by RHEL with (almost all) deprecated adapters [1] re-enabled: aacraid, be2iscsi, be2net, lpfc, megaraid, mlx4, mpt3sas, qla2xxx, qla4xxx - Upstream drivers not supported by RHEL: isci, sky2 - Wireguard The second group of packages is actually more complicated to maintain than the first. Wireguard and others highly depend on upstream support. Is there any reason why one should not be able to build these packages in CBS? Note: The kernel modules are currently not being signed. Just let me know how I can move this proposal forward. If there is no interest by anyone, I'll just keep on building the packages I do require in private. Best Peter [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.9_release_notes/deprecated_functionality#deprecated_adapters