On 18/02/2021 18.13, Peter Georg wrote: > On 15/02/2021 15.52, Peter Georg wrote: >> On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote: >>> On Saturday, February 13, 2021 2:45 PM, Peter Georg >>> <peter.georg at physik.uni-regensburg.de> wrote: >>> >>>> On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote: >>>> >>>>> There are four main goals of this SIG: >>>>> First, break down changes in the Stream version 4.18.0 kernel into >>>>> individual patches. When possible, document the source the patch is >>>>> from and the purpose of the patch. Moving forward also document >>>>> which Stream kernel package revision introduced the patch. >>>>> Second, build CentOS Plus kernel for Stream. >>>>> Third, package newer LTS kernels during the LTS lifespan. These >>>>> would be available as alternative kernels for Stream similar to the >>>>> CentOS Plus kernel. >>>>> Fourth, work on tools for better delivery of live security patches >>>>> to be applied via kpatch. >>>>> The reasoning for this SIG is to improve the openness of kernels >>>>> available through Stream. >>>>> Normally a distribution upstream would have access to which patch >>>>> or commit each addition to the kernel belongs to. The patch name or >>>>> commit comment would also give information as to the purpose of the >>>>> patch. Stream's kernel reports to be 4.18.0 but uses a tar that is >>>>> 110% the size of a true 4.18.0 tar. A breakdown of the additional >>>>> 10% is not provided. Hence Stream's kernel still functions similar >>>>> to a downstream rather than providing what is expected for a >>>>> successful upstream project. >>>>> This SIG would help close the kernel openness gap by documenting >>>>> the changes. Stream users would then be in a better position to >>>>> track down individual patches that cause regressions or bugs. >>>>> This SIG would also help smooth the adoption of Stream for users >>>>> that depend on CentOS Plus kernel features. >>>>> Next, this SIG should help improve feedback to the LTS kernel >>>>> developers. Tracking LTS kernel changes may also help with >>>>> documenting some changes in the primary Stream kernel. >>>>> Lastly, this SIG should help improve kernel security by promoting >>>>> methods to enable kernel security updates even when a reboot of the >>>>> system is not possible. >>>> >>>> Replying directly to the head of this topic now as I'm not sure which >>>> branch it fits best. >>>> >>>> As announced by Brian Stinson, he was available on IRC yesterday >>>> (twice) >>>> to discuss anything related to kernels and this proposal in particular. >>>> Sadly there have only been a few people to discuss this matter. >>>> >>>> Most of the discussion was indeed about the fifth main goal that I >>>> proposed in a follow up mail to the original proposal: Package external >>>> kernel modules >>>> >>>> This does not mean that the other proposed goals are not worth >>>> investigating, but simply not enough people have shown up on IRC >>>> yesterday to discuss these. >>>> >>>> Brian also mentioned that there might actually be fewer challenges >>>> blocking a SIG to build kernel modules in CBS than he initially thought >>>> (especially if secureboot is not a hard requirement, i.e. can be >>>> tackled >>>> at a later stage). So we might be able to start earlier than expected. >>>> >>>> It seems like starting with packaging external kernel modules is a good >>>> starting point for the proposed SIG. Hence we concluded to start >>>> gathering a list of kernel modules people are interested in building >>>> for >>>> CentOS Stream here on the mailing list. Of course this is not a pure >>>> wish list, but people should also commit to be part of the process / >>>> SIG. On IRC the following kernel modules have been mentioned / >>>> discussed >>>> as potential candidates to be built by a kernel SIG. I added >>>> information >>>> where these modules can currently be retrieved for RHEL and CentOS >>>> Linux >>>> (only one source for each module). However these are probably not >>>> compatible with CentOS Stream. >>>> >>>> - isci (ELRepo, kmod) >>>> - mlx4 (ELRepo, kmod) >>>> - lustre-client (Lustre upstream, kmod and dkms) >>>> - nvidia (nVidia, kmod and dkms) >>>> - rocm (AMD, dkms) >>>> - wireguard (ELRepo, kmod) >>>> >>>> Please feel free to extend this list with modules you are >>>> interested in. >>>> I suggest you also add the source you currently get these >>>> modules (for >>>> RHEL/CentOS Linux) from if any is available. >>> >>> I am glad you found IRC to be productive forum to move forward on the >>> topic of kernel modules. >>> >>> It does not seem like the best place to get answers to what I am >>> looking for. >>> >>> Mike McGrath wrote back on Jan 20th the following: >>> >>>> "I've confirmed with the team, the git repo is going to be all the >>>> normal git patches you would expect (IE: not arbitrarily munged >>>> together in some way). There's one or two more things they're >>>> configuring with gitlab and they expect to have an actual repo that >>>> you can look at / poke at to validate what I'm saying in a few weeks." >>> >>> It has been "a few weeks" and there has been no update. There is no >>> details on what the one or two things still left to configure are. >>> Additionally, there is no status on how the one or two things are >>> progressing. >>> >>> So far, the CentOS governance board has established only *ONE* hard >>> deadline for the transition which is the Dec 31st termination of >>> CentOS 8 in favor of Stream. All other deliverables to make that >>> transition successful remain vague and unspecified. I haven't even >>> found a public list of all the items Red Hat considers to be >>> deliverables as part of this transition. >>> >>> Brian Stinson's offer to talk on IRC, while helpful in other aspects, >>> gave no indication of being able to address this fundamental reality. >>> It left me with doubts on if the disappointing follow-through on the >>> part of Mike McGrath could be meaningfully discussed via that forum. >>> All I was looking for was at least a status update from Mike McGrath >>> after a few weeks and even that was not provided. >>> >>> If you want me to contribute to the discussion of packaging >>> additional kernel modules, here is all I have to say right now: >>> >>> First, documenting the Stream kernel modifications needs to be the >>> top priority. If a kernel crash dump points to code that doesn't >>> belong to vanilla 4.18.0, we should already have the resources to >>> understand what the code/patch that cause the crash dump is. That >>> includes being able to investigate patches which cause bad >>> interactions with other kernel modules we claim we will be supporting. >>> >>> Second, if Stream is going to release new kernels more frequently >>> than CentOS 8 then I agree this should be a goal of the Kernel SIG to >>> address. >>> >>> Third, binary only kernel module under restrictive licensing terms >>> should never be promoted/packaged by the Kernel SIG. >>> >>> Specifically, NVIDIA license mandates security problems for the >>> victims that are lured into using their BLOB of a driver. >>> >>> These drivers have a history of creating security issues including >>> CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056. >>> >>> nVidia mandates it is ILLEGAL to compare the object code differences >>> between driver versions making it illegal to meaningfully address >>> them through kpatch. Hence, there will NEVER be live security >>> patching of the security issues NVIDIA inflicts upon Stream users. >>> >>> There are also other issues with the license. It might be legal to >>> perform fuzzing against the NVIDIA driver, but if you find an issue >>> then you can't reverse engineer the impacted code to explore if it >>> creates a larger issue. You might believe a problem which causes a >>> kernel crash could also lead to a buffer overflow to run arbitrary >>> code in kernel space but under the terms of the license you must >>> operate with both hands tied behind your back. Instead, you have to >>> submit the issue to NVIDIA and hope the investigate it further to >>> determine the full extent of the security issue. It seems to me they >>> have a conflict of interest in setting the timeline to investigate >>> and adddress the problem when they have a monopoly on being able to >>> look and modify. >>> >>> The nouveau drivers are provided under superior licensing terms and >>> do not inflict any of the draconian terms of the NVIDIA drivers. It >>> is my expectation to be able to someday address security issues with >>> nouveau drivers via kpatch. >> >> Thanks for your input. I agree that there are licensing issues, >> especially concerning the nvidia drivers. That's something we have to >> think about, hence I'd suggest to move adding these kernel modules to >> a later stage and start with kernel modules without this kind of >> issues. Actually the nvidia driver is the only one on the current list >> with licensing issues. >> >> Peter >> >> >>> Back to the topic of IRC, when Stream and the Kernel SIG are in a >>> more mature state, I will feel more comfortable discussing things via >>> IRC. Until then using the mailing list remains the best way to >>> contact me. Please don't misrepresent that as a lack of interest. > > Responding to my own message as there seems to be low interest. > > @Brian Stinson: Any suggestions on how we could proceed in this matter? > > I'm not sure why there is so little interest in this endeavor. There at > least seems to be some demand for kmods to support Hardware not > supported by the CentOS Stream kernel (see question about ELRepo on > #centos-devel few days ago). Having the recent announcement about EOL of > CentOS Linux 8 in mind, I assume a possible explanations (at least for > this case) is that (CentOS Linux 7/8) users: > > 1. haven't decided yet what to do once CentOS Linux 7/8 is EOL. > 2. are planning to move to a free RHEL subscription or an other RHEL > downstream rebuild (AlmaLinux, RockyLinux, etc.). > 3. tried to run CentOS Stream 8 and realized these kmod packages > provided by ELRepo and others are not working anymore. These probably > stick with CentOS Linux 8 for now and switch > a) to an alternative RHEL downstream rebuild (AlmaLinux, RockyLinux, > etc.) before EOL (supported by ELRepo and others); > b) to CentOS Stream 8 once someone has fixed this issue. > > Just my guess. Anyway, to allow users currently running CentOS Linux 8 > (i) with kmods provided by ELRepo and others or (ii) running the CentOS > Linux Plus kernel to switch to CentOS Stream 8 without changing > hardware, one should either provide (i) an alternate source for these > kmods or (ii) build CentOS Stream Plus kernel. > Has anyone heard from Akemi / toracat about plans concerning building > the plus kernel for CentOS Stream? > > > As I already mentioned in one of my previous emails, I do need some of > these kmods locally and hence build them for CentOS Stream anyway. > I do not mind the extra work (actually it might be lower once all ist > set up) to provide the kmods to other CentOS Stream users as well. > > I currently have SPECS ready for: > - Rebuilds of upstream drivers supported by RHEL with (almost all) > deprecated adapters [1] re-enabled: aacraid, be2iscsi, be2net, lpfc, > megaraid, mlx4, mpt3sas, qla2xxx, qla4xxx > - Upstream drivers not supported by RHEL: isci, sky2 > - Wireguard Short update on this matter: I have updated the packages listed above (and a few more) locally for every kernel release since February: So far there have only been two cases that required manual work: i) megaraid patches had to be adapted for kernel 4.18.0-301.1 as previously deprecated and disabled adapters have been re-enabled. ii) wireguard had to be updated to the latest upstream release to work with kernel 4.18.0-301.1 (the first kernel release that changed RHEL_MINOR to 5). In all other cases I have been able to automate rebuilding these kernel modules for every newly released kernel version. Hence I see no reason why this could not also be done within CBS. Concerning the CBS infrastructure I assume there are no issues either, or at least no issues the Hyperscale SIG doesn't have to deal with as well to provide an alternative kernel. Note: I'm currently not signing any kmod. So in case there is any way to move this SIG proposal forward, or some other more suitable way (under the umbrella of centos-plus or some other SIG?) to provide such kmod packages, please let me know. Peter > The second group of packages is actually more complicated to maintain > than the first. Wireguard and others highly depend on upstream support. > > > Is there any reason why one should not be able to build these packages > in CBS? Note: The kernel modules are currently not being signed. > > Just let me know how I can move this proposal forward. If there is no > interest by anyone, I'll just keep on building the packages I do require > in private. > > > Best > > Peter > > [1] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.9_release_notes/deprecated_functionality#deprecated_adapters > > _______________________________________________ > CentOS-devel mailing list > CentOS-devel at centos.org > https://lists.centos.org/mailman/listinfo/centos-devel