On 20/01/2021 08:12, Julien Pivotto wrote:

On 20 Jan 08:08, redbaronbrowser via CentOS-devel wrote:

...

There are four main goals of this SIG:

First, break down changes in the Stream version 4.18.0 kernel into individual patches. When possible, document the source the patch is from and the purpose of the patch. Moving forward also document which Stream kernel package revision introduced the patch.

Let's face it that should be the sane technical default for stream, from red hat. They should provide it out of the box. What they do downstream after this is up to them.

Agreed. I makes it very very difficult to contribute to Stream at the kernel level is any meaningful way if there is not a proper git repository (or similar) tracking all commits that have been made over the lifetime of the project.

Red Hat killed CentOS Linux on the basis it's community was simply a community of users and didn't contribute anything. As it stands CentOS Stream is no different as there is still no effective way to contribute. Having a proper git repository is the first step to changing that. Otherwise we are shackled to remaining users who can file bug reports when something breaks but little else.

In reality, Stream is not an open upstream of RHEL, but rather CentOS and RHEL have simply switched positions within a closed downstream process, CentOS Stream now being the first (development) step in that closed process rather than a rebuild of it.

I'm not offering to be a part of this proposed SIG, but am watching the whole topic with much interest.

On Wednesday, January 20, 2021 7:06 AM, Mike McGrath mmcgrath@redhat.com wrote:

On Wed, Jan 20, 2021 at 2:51 AM Phil Perry pperry@elrepo.org wrote:

...

On 20/01/2021 08:12, Julien Pivotto wrote:

...

On 20 Jan 08:08, redbaronbrowser via CentOS-devel wrote:

...

There are four main goals of this SIG:

First, break down changes in the Stream version 4.18.0 kernel into individual patches.  When possible, document the source the patch is from and the purpose of the patch.  Moving forward also document which Stream kernel package revision introduced the patch.

Let's face it that should be the sane technical default for stream, from red hat. They should provide it out of the box. What they do downstream after this is up to them.

Agreed. I makes it very very difficult to contribute to Stream at the kernel level is any meaningful way if there is not a proper git repository (or similar) tracking all commits that have been made over the lifetime of the project.

Red Hat killed CentOS Linux on the basis it's community was simply a community of users and didn't contribute anything. As it stands CentOS Stream is no different as there is still no effective way to contribute. Having a proper git repository is the first step to changing that. Otherwise we are shackled to remaining users who can file bug reports when something breaks but little else.

In reality, Stream is not an open upstream of RHEL, but rather CentOS and RHEL have simply switched positions within a closed downstream process, CentOS Stream now being the first (development) step in that closed process rather than a rebuild of it.

I'm not offering to be a part of this proposed SIG, but am watching the whole topic with much interest.

I'll check with the kernel team and see if we have any published plans or even a test repo we can share so people can see what we're planning.

I'm sorry for the confusion but I want to make clear this SIG does not depend on any changes on the part of Red Hat.  They are welcome to contribute if they can do so in a meaningful way but I don't want this SIG purposal refused on a misconception this is an attempt to change Red Hat's policy not releasing the details of their obfuscared kernel themselves.

You had said on the mailing list on Dec 25th: "We want a robust SIG community, and that includes welcoming things that Red Hat isn't particularly interested in."

I am trying to do exactly what you indicated CentOS is interested in. I brought up the issue with the openness gap with the kernel several time perviously and Red Hat showed they were not particularly interested in addressing it.

I would also like to point out that the types of errors I am getting when trying Stream falls short of Q/A testing required of a Fedora release. There are some bugs am I seeing that Fedora would treat as release blockers and what I am seeing could not pass a Fedora Go / No-Go release meeting.

At the same time, it has been made clear on the mailing list that Red Hat does not consider this to be another Rawhide or Beta release. Given Red Hat consider this production quality while falling short of Fedora standards, it seem clear at this point Red Hat is incompetent in performing Fedora-grade testing and fixing Stream is being left up to the community.

With that in mind, the place I need to start is at the kernel. It has a higher level of priviledge than any other part of Stream. It also is the only part that normally requires a full reboot of the hardware to put an update into actual use.

So, again, Goal #1 is already being worked on. It is just a matter of it is published as part of Stream or as part of a fork of CentOS. If Red Hat is welcoming of things it isn't particularly interested in, then it shouldn't be a problem to work on this as a CentOS SIG.

I was really hoping to get feedback about the other three goals instead of remaining stuck on just one specific goal. Is no one interested in any of the other three goals?

On Wed, Jan 20, 2021 at 7:06 AM Mike McGrath mmcgrath@redhat.com wrote:

On Wed, Jan 20, 2021 at 2:51 AM Phil Perry pperry@elrepo.org wrote:

...

On 20/01/2021 08:12, Julien Pivotto wrote:

...

On 20 Jan 08:08, redbaronbrowser via CentOS-devel wrote:

...

There are four main goals of this SIG:

First, break down changes in the Stream version 4.18.0 kernel into

individual patches. When possible, document the source the patch is from and the purpose of the patch. Moving forward also document which Stream kernel package revision introduced the patch.

...

Let's face it that should be the sane technical default for stream, from red hat. They should provide it out of the box. What they do downstream after this is up to them.

Agreed. I makes it very very difficult to contribute to Stream at the kernel level is any meaningful way if there is not a proper git repository (or similar) tracking all commits that have been made over the lifetime of the project.

Red Hat killed CentOS Linux on the basis it's community was simply a community of users and didn't contribute anything. As it stands CentOS Stream is no different as there is still no effective way to contribute. Having a proper git repository is the first step to changing that. Otherwise we are shackled to remaining users who can file bug reports when something breaks but little else.

In reality, Stream is not an open upstream of RHEL, but rather CentOS and RHEL have simply switched positions within a closed downstream process, CentOS Stream now being the first (development) step in that closed process rather than a rebuild of it.

I'm not offering to be a part of this proposed SIG, but am watching the whole topic with much interest.

I'll check with the kernel team and see if we have any published plans or even a test repo we can share so people can see what we're planning.

I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way). There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks.

-Mike

       -Mike

...

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

On Wed, Feb 10, 2021, at 13:17, redbaronbrowser via CentOS-devel wrote:

On Wednesday, January 20, 2021 9:31 AM, Mike McGrath mmcgrath@redhat.com wrote:

...

I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way). There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks.

       -Mike

Mike, is there any status update on individual patches applied to Stream's kernel?

Can we move forward with this SIG so I can publish the information I already have?

What about a Stream Plus kernel? The directory for Stream still has no packages at all.

Or delivering kpatch live security updates without a reboot?

I am ready to move forward immediately if the SIG is approved. Is there a reason to wait another "few weeks"? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

I see your questions roughly grouped in 3 categories, correct me if I've misinterpreted:

- How do I get patches into the Stream kernel *now*? Have you submitted any bugs against CentOS Stream on bugzilla.redhat.com with your patches? I want to make sure, first, that we have an actual answer because some of them may indeed be workable for inclusion in RHEL.

All contributions for RHEL/Stream 8 (not just the kernel) are flowing through this bugzilla process for right now.

- When can we see the new kernel workflows? I don't have specific date/time when we expect to see the new-way of doing kernel repos but I would check https://gitlab.com/redhat/ towards the end of the month. Those will be targeted at RHEL/Stream 9 kernels. I'm asking around about some of the work we're doing to make the Stream 8 kernel sources more accessible in a format that isn't just tarballs.

- How do I get kernels that are not shipped in-distro? Here's the problem as it exists right now, Kernels typically need secureboot signatures on them, and we don't have the infrastructure we need to make those signatures happen in the community build system. We're talking about ways to enable that directly in CBS, but in the meantime we plan to take community-based kernels and assist with getting builds done on infrastructure that can do signing. Note: this affects other packages like fwupd, grub, etc. We are absolutely open to continuing to build the Plus kernel. Would it make sense to get yourself, @toracat for the Plus kernel, and whoever else is interested in building kernels together on IRC to talk about how to handle coordination here?

With regard to distributing kpatch updates, I'd advise caution here. I get the sense that there may be problems building and testing all of the combinations of kpatches that a SIG might want to ship.

Let me know if there's anything else I can help with.

--Brian

On 11/02/2021 00:03, Peter Georg wrote:

On 10/02/2021 22.51, Brian Stinson wrote:

...

On Wed, Feb 10, 2021, at 13:17, redbaronbrowser via CentOS-devel wrote:

...

On Wednesday, January 20, 2021 9:31 AM, Mike McGrath mmcgrath@redhat.com wrote:

...

I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way).  There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks.

-Mike

Mike, is there any status update on individual patches applied to Stream's kernel?

Can we move forward with this SIG so I can publish the information I already have?

What about a Stream Plus kernel?  The directory for Stream still has no packages at all.

Or delivering kpatch live security updates without a reboot?

I am ready to move forward immediately if the SIG is approved.  Is there a reason to wait another "few weeks"? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

I see your questions roughly grouped in 3 categories, correct me if I've misinterpreted:

• How do I get patches into the Stream kernel *now*?

Have you submitted any bugs against CentOS Stream on bugzilla.redhat.com with your patches? I want to make sure, first, that we have an actual answer because some of them may indeed be workable for inclusion in RHEL.

All contributions for RHEL/Stream 8 (not just the kernel) are flowing through this bugzilla process for right now.

• When can we see the new kernel workflows?

I don't have specific date/time when we expect to see the new-way of doing kernel repos but I would check https://gitlab.com/redhat/ towards the end of the month. Those will be targeted at RHEL/Stream 9 kernels. I'm asking around about some of the work we're doing to make the Stream 8 kernel sources more accessible in a format that isn't just tarballs.

• How do I get kernels that are not shipped in-distro?

Here's the problem as it exists right now, Kernels typically need secureboot signatures on them, and we don't have the infrastructure we need to make those signatures happen in the community build system. We're talking about ways to enable that directly in CBS, but in the meantime we plan to take community-based kernels and assist with getting builds done on infrastructure that can do signing. Note: this affects other packages like fwupd, grub, etc. We are absolutely open to continuing to build the Plus kernel. Would it make sense to get yourself, @toracat for the Plus kernel, and whoever else is interested in building kernels together on IRC to talk about how to handle coordination here?

Please add me to the list of interested people (@pjgeorg in IRC).

Let me note that I have a different goal that might be added to this SIG and is not part of the four goals listed in the initial proposal:

5. Package external kernel modules

So far (running CentOS Linux) I have been using various different sources for kmod packages (and built some locally). However many of these officially only support RHEL (most prominent on this list: ELRepo. At least according to recent comments by Phil Perry. I haven't seen any comments by other contributors). For the time being I "fix" this locally by using dkms packages if avaible or maintain dkms packages myself for modules that are otherwise only provided as kmod packages. I really dislike doing such a job locally without any open collaboration. Just feels wrong working with open source software without trying to collaborate "upstream". Hence my interest in contributing to the proposed SIG.

At first I'd suggest starting with drivers included in upstream kernel but disabled by Red Hat. Yes, these are also available by using the CentOS Plus kernel. However I, and maybe other users as well, prefer to run the "vanilla" CentOS Stream kernel and add missing drivers as external kernel modules as required. And of course, ELRepo currently provides exactly this. However most of these can sadly not be used for CentOS Stream. This also applies to building LTS kernel packages for CentOS Stream. As Phil Perry mentioned this is and has already been done by ELRepo for some time. I agree with Matthew Miller here: These things should be connected. On a side note: The Hyperscale SIG seems to be interested in providing LTS kernel(s) for CentOS Stream as well (has been mentioned in their talk at the CentOS Dojo last week).

Once everything is set up and working for these simpler cases, packages for any other kernel modules that are of use for CentOS Stream users can be added.

Please let me know what you think about adding this goal to the proposed SIG. Of course I'm especially interested in comments/opinions by people currently involved in building the CentOS Plus kernel and/or ELRepo.

Thanks!

Peter

Hi Peter,

Packaging of external kernel modules for Stream presents a dilemma. IMHO you first need to decide if you are packaging for Stream as an end user product, or if you are packaging for Stream as a development product for RHEL.

If it's the latter, then the established procedure is to use kABI-tracking kmod packages, and any development work done in Stream should do likewise otherwise it may have little value in RHEL.

However, if you are packaging for Stream as an end user product, using kABI-tracking kmod packages makes little sense as the kABI is not stable. For example, of the elrepo kmods we have tested, around a quarter broke on the first Stream kernel update (-257) and by the latest update (-277), only 5 packages out of ~50 remain intact.

Therefore, DKMS seems a better technical fit for the constantly changing ABI of the Stream kernel. One of the main objections to using DKMS in Enterprise Linux was always the need for having compiler and other development tools installed on production systems. As Stream is not intended for production-ready Enterprise Linux use, but is rather a 'beta' type development product, DKMS seems a better technical fit in these circumstances but such packages are unlikely to find a place in RHEL.

Hence I think the first thing you will need to decide is if you are developing for Stream as an end-user product or as an upstream of RHEL, and that will largely dictate the route you take. For me, the whole point of Stream seems to be as a place to contribute to upstream development of RHEL, and as such it is kmods that are exclusively used in RHEL.

As a side note, you mentioned the use case of drivers that are disabled by Red Hat in the RHEL kernel. What I would like to see is the opportunity for the community (a SIG) to maintain these drivers within the upstream Stream kernel (negating the need for a Stream-Plus kernel*) and then Red Hat can disable them downstream for RHEL if it doesn't wish to support them, rather than us (the community) constantly having to re-enable (undo) something Red Hat has decided we do not want/need and has disabled. But that can't realistically happen until we have a totally open git repository for the kernel and all commits whereby Red Hat can apply RHEL-specific patches downstream of Stream and what SIGs like this are doing.

Phil

[*] Having a Plus kernel in a downstream product such as CentOS Linux makes perfect sense, but it makes absolutely no sense where Stream is the upstream product. RHEL now becomes the downstream, and they are the ones who need a 'Plus' kernel for RHEL, by forking/submitting patches against the Stream kernel to remove/disable the bits they don't want in RHEL. Moving CentOS-Stream upstream of RHEL essentially flips the 'Plus' kernel relationship on it's head.

On Wednesday, February 10, 2021 3:51 PM, Brian Stinson brian@bstinson.com wrote:

On Wed, Feb 10, 2021, at 13:17, redbaronbrowser via CentOS-devel wrote:

...

On Wednesday, January 20, 2021 9:31 AM, Mike McGrath mmcgrath@redhat.com wrote:

...

I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way). There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks.

-Mike

Mike, is there any status update on individual patches applied to Stream's kernel?

Can we move forward with this SIG so I can publish the information I already have?

What about a Stream Plus kernel? The directory for Stream still has no packages at all.

Or delivering kpatch live security updates without a reboot?

I am ready to move forward immediately if the SIG is approved. Is there a reason to wait another "few weeks"? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

I see your questions roughly grouped in 3 categories, correct me if I've misinterpreted:

• How do I get patches into the Stream kernel *now*?

Have you submitted any bugs against CentOS Stream on bugzilla.redhat.com with your patches? I want to make sure, first, that we have an actual answer because some of them may indeed be workable for inclusion in RHEL.

Ok then. If I am ever talking about inclusion of new patches, I will consider doing that.

Before contributing more patches to the tar-ball mess claiming to be "4.18.0," I would like to work with the Stream community on the information I have so far of code that doesn't exist in a true 4.18.0 kernel. I have a break down of chunks that are related to specific modular patches but not publicly documented as such.

All contributions for RHEL/Stream 8 (not just the kernel) are flowing through this bugzilla process for right now.

Yes, I have already figured out that the designation of Stream being an "upstream" has changed nothing on how others contribute yet. There really is no need to rub it in our faces. I already get it.

To be meaningful members of an "upstream" we need to get things documented like an upstream. I have already been working on the documentation that RH/RHEL can't by policy make available even after the request to wait "a few weeks."

• When can we see the new kernel workflows?

I don't have specific date/time when we expect to see the new-way of doing kernel repos but I would check https://gitlab.com/redhat/ towards the end of the month. Those will be targeted at RHEL/Stream 9 kernels. I'm asking around about some of the work we're doing to make the Stream 8 kernel sources more accessible in a format that isn't just tarballs.

End of February isn't just "a few weeks" after Jan 20th, it is over an entire month.

I also don't care about Stream 9 right now. Stream 8 kernel is in serious need to being better documented. We were told back in December by a number of Red Hat employees we had been given enough information to decide for ourselves what we want to do. We were also told that Stream was to enable us more power to contribute and RH wants to see what SIGs the community would like to create. This is the most glaring area of needed help. There was nothing available reasonably documenting the Stream 8 kernel. There was no Stream 9 kernel available or any ETA given to one coming. I have focused for over a month in documenting what Red Hat fail to.

Mike indicated to just wait "a few weeks" and we would see it available in some official way. It has been a few week and there is nothing. It is not clear if we see something at the beginning of March or if there will be additional reasons it won't be til April. It isn't clear if some git commits lumped multiple unrelated code into the same commit. It isn't even clear if we will ever get a break down of the Stream 8 kernel. But it doesn't matter, I have done the work that needs to be done and am requesting now to have someplace to post it.

• How do I get kernels that are not shipped in-distro?

Here's the problem as it exists right now, Kernels typically need secureboot signatures on them, and we don't have the infrastructure we need to make those signatures happen in the community build system. We're talking about ways to enable that directly in CBS, but in the meantime we plan to take community-based kernels and assist with getting builds done on infrastructure that can do signing. Note: this affects other packages like fwupd, grub, etc. We are absolutely open to continuing to build the Plus kernel. Would it make sense to get yourself, @toracat for the Plus kernel, and whoever else is interested in building kernels together on IRC to talk about how to handle coordination here?

I would like to get started now. If that means we need documentation explaining how to turn off Secure Boot or how to register a self-signed key into Secure Boot, I will do that. But there is no reason this needs to be an all or nothing. Withholding 100% of everything on the progress of Stream 8 Plus until it is ready to ship to general users is not the way for Stream to achieve being more "open" than CentOS 8. It is how Stream 8 is even that much more closed than the CentOS 8 which already has a Plus kernel available.

With regard to distributing kpatch updates, I'd advise caution here. I get the sense that there may be problems building and testing all of the combinations of kpatches that a SIG might want to ship.

I have no interest in contributing to something were the need is the least. I want to make progress on what needs the most work. It is understandable that SIG will be working for months before having anything to ship. But if Stream is the "upstream," why isn't the community involved in the process over those months?

Let me know if there's anything else I can help with.

I don't know. It seem like we don't even have a common ground on what Stream being an upstream even means.

There is no interest on my part of being Steve Jobs and presenting a finished product. Instead it is my hope to be part of the team that works on the "and another things." To me going from the downstream to the upstream puts the type of related SIGs to be under the moto of "release early, release often." You are calling for caution because being an upstream SIG means things will be messy? Let it be messy! Otherwise what is the point?

What this SIG purposal is *NOT* and never will be is advise for people to sit on bugzilla and IRC waiting for the beginning of March to see if it is an ETA to another ETA. It is intended to be messy in February, March, April and all of the other months of 2020 to get progress done completely in the open with others than want to get messy.

If you get over your mysophobia (fear of dirt), then we can better talk about how you can help.

On 11 Feb 01:58, redbaronbrowser via CentOS-devel wrote:

Mike indicated to just wait "a few weeks" and we would see it available in some official way. It has been a few week and there is nothing. It is not clear if we see something at the beginning of March or if there will be additional reasons it won't be til April. It isn't clear if some git commits lumped multiple unrelated code into the same commit. It isn't even clear if we will ever get a break down of the Stream 8 kernel. But it doesn't matter, I have done the work that needs to be done and am requesting now to have someplace to post it.

I understand what you mean, but I do not think that this forum can answer this question. In Stream, "centos" and the centos project seems to only provide the pipeline (and CI scripts). The content of the packages seem to be just thrown over the wall by RHEL team.

I hope RHEL engineers will join the same forums as the CentOS community and really work in the open - then, someone with knowledge of the current state of affairs could answer this question. Until then, I am afraid you won't know a lot.

On Sunday, February 14, 2021 5:52 AM, Julien Pivotto roidelapluie@inuits.eu wrote:

On 11 Feb 01:58, redbaronbrowser via CentOS-devel wrote:

...

Mike indicated to just wait "a few weeks" and we would see it available in some official way. It has been a few week and there is nothing. It is not clear if we see something at the beginning of March or if there will be additional reasons it won't be til April. It isn't clear if some git commits lumped multiple unrelated code into the same commit. It isn't even clear if we will ever get a break down of the Stream 8 kernel. But it doesn't matter, I have done the work that needs to be done and am requesting now to have someplace to post it.

I understand what you mean, but I do not think that this forum can answer this question. In Stream, "centos" and the centos project seems to only provide the pipeline (and CI scripts). The content of the packages seem to be just thrown over the wall by RHEL team.

I hope RHEL engineers will join the same forums as the CentOS community and really work in the open - then, someone with knowledge of the current state of affairs could answer this question. Until then, I am afraid you won't know a lot.

Well, I still feel it is questions that are important to get on the record.

IRC is a transient communication which was already used in the past to communicate concerns about CentOS governance under Red Hat. To claim the community never expressed concerns until recently does not fit reality. However, the claim does highlight the dangers of using a transient communication method.

Also, as Brian Exelbierd has explained the need to kill CentOS 8 in a very blunt way:

"CentOS is a sponsored project, we are the funding agent and we happen to also be a heavy contributor. We have learned that open-source communities do well with independence. We let those governing bodies govern."

The FAQ goes on to explain CentOS 8 is being terminated with no option for a SIG to continue it to force the community to focus on Stream.

But how has it been the failure of the community to focus on Stream?

From September 2019 to November 11, 2020, the community has contributed nothing through gitlab because Red Hat has made it impossible to do so.

After the November 11th meeting including the entire month of November, Red Hat has continued to make it impossible to to contribute using the resources of gitlab.

In December when KW posted his blog post claiming CentOS could now be the Upstream and Red Hat has closed the openness gap, Red Hat still had made it impossible to contribute using gitlab.

The entire month of January, the problem continued. It appears for the entire month of February the problem will continue.

We are now 23% of the way from November 11th to the end of 2021. I'm willing to accept to keep funding we need to pull our own weight. I am not willing to accept we have been given the resources to have independence. I am also not willing to accept we will be given enough time between now and the end of the year to get Stream CD/CI to the point to encourage adoption. It helps to know what is being changes to the 4.18.0 kernel are being performed to best target what CD/CI tests should be written.

To be as blunt as Brain Exelbierd, I believe Stream is being passively sabotaged such that the end result will eventually be that Red Hat will treat it as not worth funding after 2024.

What exactly does Red Hat believe they have made available to us *today* for us to focus on Stream with the independence of being the Upstream?

Or are they willing to admit we currently are functioning with our arms tied behind our back and that maybe weekly updates could smooth the transistion?

Is there any itemized list on bringing Stream to independence and what level of progress has been made on those items?

But getting placating via IRC (again) and having my concerns go into /dev/null just does not work for me. Even if this forum also won't get any answers, at least I haven't made the same mistake twice of failing to get them posted to some form of archive.

On Fri, Feb 19, 2021, at 15:23, Mike McGrath wrote:

On Wed, Feb 10, 2021 at 7:58 PM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:

...

On Wednesday, February 10, 2021 3:51 PM, Brian Stinson brian@bstinson.com wrote:

...

• When can we see the new kernel workflows?

I don't have specific date/time when we expect to see the new-way of doing kernel repos but I would check https://gitlab.com/redhat/ towards the end of the month. Those will be targeted at RHEL/Stream 9 kernels. I'm asking around about some of the work we're doing to make the Stream 8 kernel sources more accessible in a format that isn't just tarballs.

End of February isn't just "a few weeks" after Jan 20th, it is over an entire month.

I also don't care about Stream 9 right now. Stream 8 kernel is in serious need to being better documented. We were told back in December by a number of Red Hat employees we had been given enough information to decide for ourselves what we want to do. We were also told that Stream was to enable us more power to contribute and RH wants to see what SIGs the community would like to create. This is the most glaring area of needed help. There was nothing available reasonably documenting the Stream 8 kernel. There was no Stream 9 kernel available or any ETA given to one coming. I have focused for over a month in documenting what Red Hat fail to.

Mike indicated to just wait "a few weeks" and we would see it available in some official way. It has been a few week and there is nothing. It is not clear if we see something at the beginning of March or if there will be additional reasons it won't be til April. It isn't clear if some git commits lumped multiple unrelated code into the same commit. It isn't even clear if we will ever get a break down of the Stream 8 kernel. But it doesn't matter, I have done the work that needs to be done and am requesting now to have someplace to post it.

Just under a month. I know you don't care about Stream 9, but that's what we're focused on right now. I know the kernel package has been a hot topic so you can see some of our work in progress now.

Stay tuned for a formal announcement on when this will actually be usable, contributable, and consumable. But for now, you can watch us while we get things in place:

https://gitlab.com/redhat/centos-stream/rpms/kernel

       -Mike

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

The commit-by-commit Kernl source repo is here for a preview as well: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9

We have some things left to wire up, but this gives an idea of the different repos involved.

--Brian

On Wed, Jan 20, 2021 at 07:11:25PM +0000, Phil Perry wrote:

...

On this point: the Fedora kernel packages are probably a fine place to start and it would be nice to see this SIG collaborate with the Fedora kernel maintainers and possibly the Fedora Server SIG rather than just being a disconnected downstream.

No need, this already exists both in the form of the current mainline kernel and an LTS offering, and has done so for many years. Assuming such mainline and LTS kernels built on RHEL will install seamlessly on Stream, there seems little point reinventing the wheel.

https://elrepo.org/linux/kernel/el8/x86_64/RPMS/

I would put that a different way: better to connect these things up than to have them in different places.

On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote:

On Saturday, February 13, 2021 2:45 PM, Peter Georg peter.georg@physik.uni-regensburg.de wrote:

...

On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote:

...

There are four main goals of this SIG: First, break down changes in the Stream version 4.18.0 kernel into individual patches. When possible, document the source the patch is from and the purpose of the patch. Moving forward also document which Stream kernel package revision introduced the patch. Second, build CentOS Plus kernel for Stream. Third, package newer LTS kernels during the LTS lifespan. These would be available as alternative kernels for Stream similar to the CentOS Plus kernel. Fourth, work on tools for better delivery of live security patches to be applied via kpatch. The reasoning for this SIG is to improve the openness of kernels available through Stream. Normally a distribution upstream would have access to which patch or commit each addition to the kernel belongs to. The patch name or commit comment would also give information as to the purpose of the patch. Stream's kernel reports to be 4.18.0 but uses a tar that is 110% the size of a true 4.18.0 tar. A breakdown of the additional 10% is not provided. Hence Stream's kernel still functions similar to a downstream rather than providing what is expected for a successful upstream project. This SIG would help close the kernel openness gap by documenting the changes. Stream users would then be in a better position to track down individual patches that cause regressions or bugs. This SIG would also help smooth the adoption of Stream for users that depend on CentOS Plus kernel features. Next, this SIG should help improve feedback to the LTS kernel developers. Tracking LTS kernel changes may also help with documenting some changes in the primary Stream kernel. Lastly, this SIG should help improve kernel security by promoting methods to enable kernel security updates even when a reboot of the system is not possible.

Replying directly to the head of this topic now as I'm not sure which branch it fits best.

As announced by Brian Stinson, he was available on IRC yesterday (twice) to discuss anything related to kernels and this proposal in particular. Sadly there have only been a few people to discuss this matter.

Most of the discussion was indeed about the fifth main goal that I proposed in a follow up mail to the original proposal: Package external kernel modules

This does not mean that the other proposed goals are not worth investigating, but simply not enough people have shown up on IRC yesterday to discuss these.

Brian also mentioned that there might actually be fewer challenges blocking a SIG to build kernel modules in CBS than he initially thought (especially if secureboot is not a hard requirement, i.e. can be tackled at a later stage). So we might be able to start earlier than expected.

It seems like starting with packaging external kernel modules is a good starting point for the proposed SIG. Hence we concluded to start gathering a list of kernel modules people are interested in building for CentOS Stream here on the mailing list. Of course this is not a pure wish list, but people should also commit to be part of the process / SIG. On IRC the following kernel modules have been mentioned / discussed as potential candidates to be built by a kernel SIG. I added information where these modules can currently be retrieved for RHEL and CentOS Linux (only one source for each module). However these are probably not compatible with CentOS Stream.

• isci (ELRepo, kmod)

• mlx4 (ELRepo, kmod)

• lustre-client (Lustre upstream, kmod and dkms)

• nvidia (nVidia, kmod and dkms)

• rocm (AMD, dkms)

• wireguard (ELRepo, kmod)

  Please feel free to extend this list with modules you are interested in. I suggest you also add the source you currently get these modules (for RHEL/CentOS Linux) from if any is available.

I am glad you found IRC to be productive forum to move forward on the topic of kernel modules.

It does not seem like the best place to get answers to what I am looking for.

Mike McGrath wrote back on Jan 20th the following:

...

"I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way). There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks."

It has been "a few weeks" and there has been no update. There is no details on what the one or two things still left to configure are. Additionally, there is no status on how the one or two things are progressing.

So far, the CentOS governance board has established only *ONE* hard deadline for the transition which is the Dec 31st termination of CentOS 8 in favor of Stream. All other deliverables to make that transition successful remain vague and unspecified. I haven't even found a public list of all the items Red Hat considers to be deliverables as part of this transition.

Brian Stinson's offer to talk on IRC, while helpful in other aspects, gave no indication of being able to address this fundamental reality. It left me with doubts on if the disappointing follow-through on the part of Mike McGrath could be meaningfully discussed via that forum. All I was looking for was at least a status update from Mike McGrath after a few weeks and even that was not provided.

If you want me to contribute to the discussion of packaging additional kernel modules, here is all I have to say right now:

First, documenting the Stream kernel modifications needs to be the top priority. If a kernel crash dump points to code that doesn't belong to vanilla 4.18.0, we should already have the resources to understand what the code/patch that cause the crash dump is. That includes being able to investigate patches which cause bad interactions with other kernel modules we claim we will be supporting.

Second, if Stream is going to release new kernels more frequently than CentOS 8 then I agree this should be a goal of the Kernel SIG to address.

Third, binary only kernel module under restrictive licensing terms should never be promoted/packaged by the Kernel SIG.

Specifically, NVIDIA license mandates security problems for the victims that are lured into using their BLOB of a driver.

These drivers have a history of creating security issues including CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056.

nVidia mandates it is ILLEGAL to compare the object code differences between driver versions making it illegal to meaningfully address them through kpatch. Hence, there will NEVER be live security patching of the security issues NVIDIA inflicts upon Stream users.

There are also other issues with the license. It might be legal to perform fuzzing against the NVIDIA driver, but if you find an issue then you can't reverse engineer the impacted code to explore if it creates a larger issue. You might believe a problem which causes a kernel crash could also lead to a buffer overflow to run arbitrary code in kernel space but under the terms of the license you must operate with both hands tied behind your back. Instead, you have to submit the issue to NVIDIA and hope the investigate it further to determine the full extent of the security issue. It seems to me they have a conflict of interest in setting the timeline to investigate and adddress the problem when they have a monopoly on being able to look and modify.

The nouveau drivers are provided under superior licensing terms and do not inflict any of the draconian terms of the NVIDIA drivers. It is my expectation to be able to someday address security issues with nouveau drivers via kpatch.

Thanks for your input. I agree that there are licensing issues, especially concerning the nvidia drivers. That's something we have to think about, hence I'd suggest to move adding these kernel modules to a later stage and start with kernel modules without this kind of issues. Actually the nvidia driver is the only one on the current list with licensing issues.

Peter

Back to the topic of IRC, when Stream and the Kernel SIG are in a more mature state, I will feel more comfortable discussing things via IRC. Until then using the mailing list remains the best way to contact me. Please don't misrepresent that as a lack of interest.

On 15/02/2021 15.52, Peter Georg wrote:

...

On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote:

...

On Saturday, February 13, 2021 2:45 PM, Peter Georg peter.georg@physik.uni-regensburg.de wrote:

...

On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote:

...

There are four main goals of this SIG: First, break down changes in the Stream version 4.18.0 kernel into individual patches. When possible, document the source the patch is from and the purpose of the patch. Moving forward also document which Stream kernel package revision introduced the patch. Second, build CentOS Plus kernel for Stream. Third, package newer LTS kernels during the LTS lifespan. These would be available as alternative kernels for Stream similar to the CentOS Plus kernel. Fourth, work on tools for better delivery of live security patches to be applied via kpatch. The reasoning for this SIG is to improve the openness of kernels available through Stream. Normally a distribution upstream would have access to which patch or commit each addition to the kernel belongs to. The patch name or commit comment would also give information as to the purpose of the patch. Stream's kernel reports to be 4.18.0 but uses a tar that is 110% the size of a true 4.18.0 tar. A breakdown of the additional 10% is not provided. Hence Stream's kernel still functions similar to a downstream rather than providing what is expected for a successful upstream project. This SIG would help close the kernel openness gap by documenting the changes. Stream users would then be in a better position to track down individual patches that cause regressions or bugs. This SIG would also help smooth the adoption of Stream for users that depend on CentOS Plus kernel features. Next, this SIG should help improve feedback to the LTS kernel developers. Tracking LTS kernel changes may also help with documenting some changes in the primary Stream kernel. Lastly, this SIG should help improve kernel security by promoting methods to enable kernel security updates even when a reboot of the system is not possible.

Replying directly to the head of this topic now as I'm not sure which branch it fits best.

As announced by Brian Stinson, he was available on IRC yesterday (twice) to discuss anything related to kernels and this proposal in particular. Sadly there have only been a few people to discuss this matter.

Most of the discussion was indeed about the fifth main goal that I proposed in a follow up mail to the original proposal: Package external kernel modules

This does not mean that the other proposed goals are not worth investigating, but simply not enough people have shown up on IRC yesterday to discuss these.

Brian also mentioned that there might actually be fewer challenges blocking a SIG to build kernel modules in CBS than he initially thought (especially if secureboot is not a hard requirement, i.e. can be tackled at a later stage). So we might be able to start earlier than expected.

It seems like starting with packaging external kernel modules is a good starting point for the proposed SIG. Hence we concluded to start gathering a list of kernel modules people are interested in building for CentOS Stream here on the mailing list. Of course this is not a pure wish list, but people should also commit to be part of the process / SIG. On IRC the following kernel modules have been mentioned / discussed as potential candidates to be built by a kernel SIG. I added information where these modules can currently be retrieved for RHEL and CentOS Linux (only one source for each module). However these are probably not compatible with CentOS Stream.

-   isci (ELRepo, kmod) -   mlx4 (ELRepo, kmod) -   lustre-client (Lustre upstream, kmod and dkms) -   nvidia (nVidia, kmod and dkms) -   rocm (AMD, dkms) -   wireguard (ELRepo, kmod)

Please feel free to extend this list with modules you are interested in.      I suggest you also add the source you currently get these modules (for      RHEL/CentOS Linux) from if any is available.

I am glad you found IRC to be productive forum to move forward on the topic of kernel modules.

It does not seem like the best place to get answers to what I am looking for.

Mike McGrath wrote back on Jan 20th the following:

...

"I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way).  There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks."

It has been "a few weeks" and there has been no update.  There is no details on what the one or two things still left to configure are. Additionally, there is no status on how the one or two things are progressing.

So far, the CentOS governance board has established only *ONE* hard deadline for the transition which is the Dec 31st termination of CentOS 8 in favor of Stream.  All other deliverables to make that transition successful remain vague and unspecified.  I haven't even found a public list of all the items Red Hat considers to be deliverables as part of this transition.

Brian Stinson's offer to talk on IRC, while helpful in other aspects, gave no indication of being able to address this fundamental reality. It left me with doubts on if the disappointing follow-through on the part of Mike McGrath could be meaningfully discussed via that forum. All I was looking for was at least a status update from Mike McGrath after a few weeks and even that was not provided.

If you want me to contribute to the discussion of packaging additional kernel modules, here is all I have to say right now:

First, documenting the Stream kernel modifications needs to be the top priority.  If a kernel crash dump points to code that doesn't belong to vanilla 4.18.0, we should already have the resources to understand what the code/patch that cause the crash dump is.  That includes being able to investigate patches which cause bad interactions with other kernel modules we claim we will be supporting.

Second, if Stream is going to release new kernels more frequently than CentOS 8 then I agree this should be a goal of the Kernel SIG to address.

Third, binary only kernel module under restrictive licensing terms should never be promoted/packaged by the Kernel SIG.

Specifically, NVIDIA license mandates security problems for the victims that are lured into using their BLOB of a driver.

These drivers have a history of creating security issues including CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056.

nVidia mandates it is ILLEGAL to compare the object code differences between driver versions making it illegal to meaningfully address them through kpatch.  Hence, there will NEVER be live security patching of the security issues NVIDIA inflicts upon Stream users.

There are also other issues with the license.  It might be legal to perform fuzzing against the NVIDIA driver, but if you find an issue then you can't reverse engineer the impacted code to explore if it creates a larger issue.  You might believe a problem which causes a kernel crash could also lead to a buffer overflow to run arbitrary code in kernel space but under the terms of the license you must operate with both hands tied behind your back.  Instead, you have to submit the issue to NVIDIA and hope the investigate it further to determine the full extent of the security issue.  It seems to me they have a conflict of interest in setting the timeline to investigate and adddress the problem when they have a monopoly on being able to look and modify.

The nouveau drivers are provided under superior licensing terms and do not inflict any of the draconian terms of the NVIDIA drivers.  It is my expectation to be able to someday address security issues with nouveau drivers via kpatch.

Thanks for your input. I agree that there are licensing issues, especially concerning the nvidia drivers. That's something we have to think about, hence I'd suggest to move adding these kernel modules to a later stage and start with kernel modules without this kind of issues. Actually the nvidia driver is the only one on the current list with licensing issues.

Peter

...

Back to the topic of IRC, when Stream and the Kernel SIG are in a more mature state, I will feel more comfortable discussing things via IRC. Until then using the mailing list remains the best way to contact me. Please don't misrepresent that as a lack of interest.

Responding to my own message as there seems to be low interest.

@Brian Stinson: Any suggestions on how we could proceed in this matter?

I'm not sure why there is so little interest in this endeavor. There at least seems to be some demand for kmods to support Hardware not supported by the CentOS Stream kernel (see question about ELRepo on #centos-devel few days ago). Having the recent announcement about EOL of CentOS Linux 8 in mind, I assume a possible explanations (at least for this case) is that (CentOS Linux 7/8) users:

1. haven't decided yet what to do once CentOS Linux 7/8 is EOL.
2. are planning to move to a free RHEL subscription or an other RHEL

downstream rebuild (AlmaLinux, RockyLinux, etc.). 3. tried to run CentOS Stream 8 and realized these kmod packages provided by ELRepo and others are not working anymore. These probably stick with CentOS Linux 8 for now and switch a) to an alternative RHEL downstream rebuild (AlmaLinux, RockyLinux, etc.) before EOL (supported by ELRepo and others); b) to CentOS Stream 8 once someone has fixed this issue.

Just my guess. Anyway, to allow users currently running CentOS Linux 8 (i) with kmods provided by ELRepo and others or (ii) running the CentOS Linux Plus kernel to switch to CentOS Stream 8 without changing hardware, one should either provide (i) an alternate source for these kmods or (ii) build CentOS Stream Plus kernel. Has anyone heard from Akemi / toracat about plans concerning building the plus kernel for CentOS Stream?

As I already mentioned in one of my previous emails, I do need some of these kmods locally and hence build them for CentOS Stream anyway. I do not mind the extra work (actually it might be lower once all ist set up) to provide the kmods to other CentOS Stream users as well.

I currently have SPECS ready for:

• Rebuilds of upstream drivers supported by RHEL with (almost all)

deprecated adapters [1] re-enabled: aacraid, be2iscsi, be2net, lpfc, megaraid, mlx4, mpt3sas, qla2xxx, qla4xxx

• Upstream drivers not supported by RHEL: isci, sky2
• Wireguard

The second group of packages is actually more complicated to maintain than the first. Wireguard and others highly depend on upstream support.

Is there any reason why one should not be able to build these packages in CBS? Note: The kernel modules are currently not being signed.

Just let me know how I can move this proposal forward. If there is no interest by anyone, I'll just keep on building the packages I do require in private.

Hi Peter, I have the strange feeling that you're somehow in a similar situation than I am. You may have a look at this thread on the centos list https://lists.centos.org/pipermail/centos/2021-February/353317.html

RHEL and clones are fundamentally lacking things some of us may need (skipped kernel modules, additional software packages) but the situation to get or provide them got worse over the years and today we start to feel it badly. After two decades with Red Hat based distributions it makes me rethink if I'm still on the right track. I try to see it positive but can't ignore the sad facts anymore.

Regards, Simon

On 18.02.2021 21:40, redbaronbrowser via CentOS-devel wrote:

Key questions I still can not find the answer to: What is the expected lifespan remaining on Stream 8?

The duration of the RHEL 8 Full Support Phase, 5 years. https://wiki.centos.org/FAQ/CentOSStream # Q7 https://centos.org/distro-faq/ # Q13

On 18/02/2021 23.25, redbaronbrowser via CentOS-devel wrote:

On Thursday, February 18, 2021 3:03 PM, Gena Makhomed gmm@csdoc.com wrote:

...

On 18.02.2021 21:40, redbaronbrowser via CentOS-devel wrote:

...

Key questions I still can not find the answer to: What is the expected lifespan remaining on Stream 8?

The duration of the RHEL 8 Full Support Phase, 5 years. https://wiki.centos.org/FAQ/CentOSStream # Q7 https://centos.org/distro-faq/ # Q13

Let me rephrase the question then, will Stream 8 ever recieve the resources to close the openness gap (provide all the resources expected of being the Upstream) before focus shifts to Stream 9?

Answer to Q7 leaves things vague: "Around the time the RHEL 9 Public Beta is issued, an additional set of CentOS Stream repositories and ISOs will be available. The existing CentOS Stream repositories representing RHEL 8 bits will continue to be available, and changes to these bits will continue as before for the duration of the RHEL 8 Full Support Phase. At that time, the older repositories will be discontinued, although sources will continue to be available."

It sounds like if Stream 9 is released later this year then progress in closing the openness gap for Stream 8 will come to a halt. Or as state in the FAQ: "changes to these bits will continue as before for the duration." The fact RHEL 8 has a 5 year life span does not mean Stream 8 will ever reach a meaningful Upstream status.

According to information available/provided on this mailing-list, #centos-stream, and at the CentOS Dojo my understanding is:

CentOS Stream 9 is planned to be released in Q2 this year. It will have some of the goodies people wanted to have in Stream 8 but are not yet available. Once these are available for Stream 9 one will try to "backport" these to Stream 8. However not everything can be backported as no one wants to change the building infrastructure of a currently shipping product.

To rephrase your question once more: Will Stream 8 development stop once Stream 9 is released? - No. Will Stream 9 be the "better" Stream release? - Very likely.

Best

Peter

On 18/02/2021 20.40, redbaronbrowser via CentOS-devel wrote:

On Thursday, February 18, 2021 11:13 AM, Peter Georg peter.georg@physik.uni-regensburg.de wrote:

...

Responding to my own message as there seems to be low interest.

@Brian Stinson: Any suggestions on how we could proceed in this matter?

I'm not sure why there is so little interest in this endeavor. There at least seems to be some demand for kmods to support Hardware not supported by the CentOS Stream kernel (see question about ELRepo on #centos-devel few days ago). Having the recent announcement about EOL of CentOS Linux 8 in mind, I assume a possible explanations (at least for this case) is that (CentOS Linux 7/8) users:

1. haven't decided yet what to do once CentOS Linux 7/8 is EOL.

2. are planning to move to a free RHEL subscription or an other RHEL downstream rebuild (AlmaLinux, RockyLinux, etc.).

3. tried to run CentOS Stream 8 and realized these kmod packages provided by ELRepo and others are not working anymore. These probably stick with CentOS Linux 8 for now and switch a) to an alternative RHEL downstream rebuild (AlmaLinux, RockyLinux, etc.) before EOL (supported by ELRepo and others); b) to CentOS Stream 8 once someone has fixed this issue.

   Just my guess. Anyway, to allow users currently running CentOS Linux 8 (i) with kmods provided by ELRepo and others or (ii) running the CentOS Linux Plus kernel to switch to CentOS Stream 8 without changing hardware, one should either provide (i) an alternate source for these kmods or (ii) build CentOS Stream Plus kernel. Has anyone heard from Akemi / toracat about plans concerning building the plus kernel for CentOS Stream?

   As I already mentioned in one of my previous emails, I do need some of these kmods locally and hence build them for CentOS Stream anyway. I do not mind the extra work (actually it might be lower once all ist set up) to provide the kmods to other CentOS Stream users as well.

   I currently have SPECS ready for:

• Rebuilds of upstream drivers supported by RHEL with (almost all) deprecated adapters [1] re-enabled: aacraid, be2iscsi, be2net, lpfc, megaraid, mlx4, mpt3sas, qla2xxx, qla4xxx

• Upstream drivers not supported by RHEL: isci, sky2

• Wireguard

  The second group of packages is actually more complicated to maintain than the first. Wireguard and others highly depend on upstream support.

  Is there any reason why one should not be able to build these packages in CBS? Note: The kernel modules are currently not being signed.

  Just let me know how I can move this proposal forward. If there is no interest by anyone, I'll just keep on building the packages I do require in private.

I like your drive to help get this done. I also like that Brian Stinson made the effort to be available to listen.

However, if your read forums outside of centos-devel, it seems clear there is a lot of distrust Red Hat and CentOS at this point.

Your list of possible explainations seems to be an acknowledgement that you are aware of that.

The migration of Stream having the resources to be an Upstream should have started back in 2019. It should have been able to use those resources to organically shifted CentOS 8 users into Stream 8 users. Instead of building Stream 8 up into something that provides benefits over CentOS 8, they decided to force the issue. They then announced the free RHEL licenses still without having made any additional resources available to Stream 8 users. This behavior clearly puts Stream 8 at a huge disadvantage for interest and adoption.

It would be nice to be able to help shape that Technology Previews of RHEL 8 such as getting Wireguard added. While it seem implied that in theory the new focus on Stream can make that possible, it seems from a practical sense that is still all just vapor.

Key questions I still can not find the answer to:

What is the expected lifespan remaining on Stream 8?

What is an itemized list of resources Red Hat plans to make available for us to better contribute to Stream 8?

What is the progress status on each of those items for making those resources available?

Can we please not turn this into yet an other discussion about what CentOS Stream is and what it should be? We know what it is now. We have heard about what is envisioned in the future. Community members have raised their concerns and wishes/visions.

I offer my help to improve the current version of CentOS Stream 8. This does not depend on any of the proposed future features to be available.

So far CentOS should still have a head start in attracting SIGs around Stream. But AlmaLinux has stated they are setting aside $1 million per year in funding their project. It seems to me at some point this year they will have their own community manager and resources for SIG groups. There is only so far KW's blog post issing platitudes of theoritical advantages can hold the interest of the community. > Overall, I think you are attributing lack of faith in Red Hat with lack of interest. There should be plenty of interest in the goals you are trying to accomplish but people have gotten sick of ramming their head into a brick wall. Stream 8 is in such a vague state at this moment there is nothing to really build interest around. As long as people are stuck in a wait and see mode, they might as well wait to see what other evolving clones will be offering to SIGs.

Maybe I mistake lack of interest with lack of faith, but this doesn't really matter. The consequences are the same.

For my needs Stream 8 is very similar to Linux 8: The only difference is removing an artificial border that previously allowed (non-security) updates to only happen twice a year. Whether this is a good or bad change depends on your personal needs.

There are other proposed features I'm interested in. However, these are not yet available.

Feel free to try emailing centos-questions@ about when we should be able to contribute into CBS and get signed Wireguard modules. See if you get anything close to a practical answer to that.

I will not email centos-questions@ about this matter as I prefer to discuss in the open and centos-devel@ seems to be the right place.

Best

Peter

On 18/02/2021 18.13, Peter Georg wrote:

On 15/02/2021 15.52, Peter Georg wrote:

...

On 14/02/2021 06.22, redbaronbrowser via CentOS-devel wrote:

...

On Saturday, February 13, 2021 2:45 PM, Peter Georg peter.georg@physik.uni-regensburg.de wrote:

...

On 20/01/2021 09.08, redbaronbrowser via CentOS-devel wrote:

...

There are four main goals of this SIG: First, break down changes in the Stream version 4.18.0 kernel into individual patches. When possible, document the source the patch is from and the purpose of the patch. Moving forward also document which Stream kernel package revision introduced the patch. Second, build CentOS Plus kernel for Stream. Third, package newer LTS kernels during the LTS lifespan. These would be available as alternative kernels for Stream similar to the CentOS Plus kernel. Fourth, work on tools for better delivery of live security patches to be applied via kpatch. The reasoning for this SIG is to improve the openness of kernels available through Stream. Normally a distribution upstream would have access to which patch or commit each addition to the kernel belongs to. The patch name or commit comment would also give information as to the purpose of the patch. Stream's kernel reports to be 4.18.0 but uses a tar that is 110% the size of a true 4.18.0 tar. A breakdown of the additional 10% is not provided. Hence Stream's kernel still functions similar to a downstream rather than providing what is expected for a successful upstream project. This SIG would help close the kernel openness gap by documenting the changes. Stream users would then be in a better position to track down individual patches that cause regressions or bugs. This SIG would also help smooth the adoption of Stream for users that depend on CentOS Plus kernel features. Next, this SIG should help improve feedback to the LTS kernel developers. Tracking LTS kernel changes may also help with documenting some changes in the primary Stream kernel. Lastly, this SIG should help improve kernel security by promoting methods to enable kernel security updates even when a reboot of the system is not possible.

Replying directly to the head of this topic now as I'm not sure which branch it fits best.

As announced by Brian Stinson, he was available on IRC yesterday (twice) to discuss anything related to kernels and this proposal in particular. Sadly there have only been a few people to discuss this matter.

Most of the discussion was indeed about the fifth main goal that I proposed in a follow up mail to the original proposal: Package external kernel modules

This does not mean that the other proposed goals are not worth investigating, but simply not enough people have shown up on IRC yesterday to discuss these.

Brian also mentioned that there might actually be fewer challenges blocking a SIG to build kernel modules in CBS than he initially thought (especially if secureboot is not a hard requirement, i.e. can be tackled at a later stage). So we might be able to start earlier than expected.

It seems like starting with packaging external kernel modules is a good starting point for the proposed SIG. Hence we concluded to start gathering a list of kernel modules people are interested in building for CentOS Stream here on the mailing list. Of course this is not a pure wish list, but people should also commit to be part of the process / SIG. On IRC the following kernel modules have been mentioned / discussed as potential candidates to be built by a kernel SIG. I added information where these modules can currently be retrieved for RHEL and CentOS Linux (only one source for each module). However these are probably not compatible with CentOS Stream.

-   isci (ELRepo, kmod) -   mlx4 (ELRepo, kmod) -   lustre-client (Lustre upstream, kmod and dkms) -   nvidia (nVidia, kmod and dkms) -   rocm (AMD, dkms) -   wireguard (ELRepo, kmod)

Please feel free to extend this list with modules you are interested in.      I suggest you also add the source you currently get these modules (for      RHEL/CentOS Linux) from if any is available.

I am glad you found IRC to be productive forum to move forward on the topic of kernel modules.

It does not seem like the best place to get answers to what I am looking for.

Mike McGrath wrote back on Jan 20th the following:

...

"I've confirmed with the team, the git repo is going to be all the normal git patches you would expect (IE: not arbitrarily munged together in some way).  There's one or two more things they're configuring with gitlab and they expect to have an actual repo that you can look at / poke at to validate what I'm saying in a few weeks."

It has been "a few weeks" and there has been no update.  There is no details on what the one or two things still left to configure are. Additionally, there is no status on how the one or two things are progressing.

So far, the CentOS governance board has established only *ONE* hard deadline for the transition which is the Dec 31st termination of CentOS 8 in favor of Stream.  All other deliverables to make that transition successful remain vague and unspecified.  I haven't even found a public list of all the items Red Hat considers to be deliverables as part of this transition.

Brian Stinson's offer to talk on IRC, while helpful in other aspects, gave no indication of being able to address this fundamental reality. It left me with doubts on if the disappointing follow-through on the part of Mike McGrath could be meaningfully discussed via that forum. All I was looking for was at least a status update from Mike McGrath after a few weeks and even that was not provided.

If you want me to contribute to the discussion of packaging additional kernel modules, here is all I have to say right now:

First, documenting the Stream kernel modifications needs to be the top priority.  If a kernel crash dump points to code that doesn't belong to vanilla 4.18.0, we should already have the resources to understand what the code/patch that cause the crash dump is.  That includes being able to investigate patches which cause bad interactions with other kernel modules we claim we will be supporting.

Second, if Stream is going to release new kernels more frequently than CentOS 8 then I agree this should be a goal of the Kernel SIG to address.

Third, binary only kernel module under restrictive licensing terms should never be promoted/packaged by the Kernel SIG.

Specifically, NVIDIA license mandates security problems for the victims that are lured into using their BLOB of a driver.

These drivers have a history of creating security issues including CVE-2021-1052, CVE-2021-1053 and CVE-2021-1056.

nVidia mandates it is ILLEGAL to compare the object code differences between driver versions making it illegal to meaningfully address them through kpatch.  Hence, there will NEVER be live security patching of the security issues NVIDIA inflicts upon Stream users.

There are also other issues with the license.  It might be legal to perform fuzzing against the NVIDIA driver, but if you find an issue then you can't reverse engineer the impacted code to explore if it creates a larger issue.  You might believe a problem which causes a kernel crash could also lead to a buffer overflow to run arbitrary code in kernel space but under the terms of the license you must operate with both hands tied behind your back.  Instead, you have to submit the issue to NVIDIA and hope the investigate it further to determine the full extent of the security issue.  It seems to me they have a conflict of interest in setting the timeline to investigate and adddress the problem when they have a monopoly on being able to look and modify.

The nouveau drivers are provided under superior licensing terms and do not inflict any of the draconian terms of the NVIDIA drivers.  It is my expectation to be able to someday address security issues with nouveau drivers via kpatch.

Thanks for your input. I agree that there are licensing issues, especially concerning the nvidia drivers. That's something we have to think about, hence I'd suggest to move adding these kernel modules to a later stage and start with kernel modules without this kind of issues. Actually the nvidia driver is the only one on the current list with licensing issues.

Peter

...

Back to the topic of IRC, when Stream and the Kernel SIG are in a more mature state, I will feel more comfortable discussing things via IRC. Until then using the mailing list remains the best way to contact me. Please don't misrepresent that as a lack of interest.

Responding to my own message as there seems to be low interest.

@Brian Stinson: Any suggestions on how we could proceed in this matter?

I'm not sure why there is so little interest in this endeavor. There at least seems to be some demand for kmods to support Hardware not supported by the CentOS Stream kernel (see question about ELRepo on #centos-devel few days ago). Having the recent announcement about EOL of CentOS Linux 8 in mind, I assume a possible explanations (at least for this case) is that (CentOS Linux 7/8) users:

1. haven't decided yet what to do once CentOS Linux 7/8 is EOL.
2. are planning to move to a free RHEL subscription or an other RHEL

downstream rebuild (AlmaLinux, RockyLinux, etc.). 3. tried to run CentOS Stream 8 and realized these kmod packages provided by ELRepo and others are not working anymore. These probably stick with CentOS Linux 8 for now and switch   a) to an alternative RHEL downstream rebuild (AlmaLinux, RockyLinux, etc.) before EOL (supported by ELRepo and others);   b) to CentOS Stream 8 once someone has fixed this issue.

Just my guess. Anyway, to allow users currently running CentOS Linux 8 (i) with kmods provided by ELRepo and others or (ii) running the CentOS Linux Plus kernel to switch to CentOS Stream 8 without changing hardware, one should either provide (i) an alternate source for these kmods or (ii) build CentOS Stream Plus kernel. Has anyone heard from Akemi / toracat about plans concerning building the plus kernel for CentOS Stream?

As I already mentioned in one of my previous emails, I do need some of these kmods locally and hence build them for CentOS Stream anyway. I do not mind the extra work (actually it might be lower once all ist set up) to provide the kmods to other CentOS Stream users as well.

I currently have SPECS ready for:

• Rebuilds of upstream drivers supported by RHEL with (almost all)

deprecated adapters [1] re-enabled: aacraid, be2iscsi, be2net, lpfc, megaraid, mlx4, mpt3sas, qla2xxx, qla4xxx

• Upstream drivers not supported by RHEL: isci, sky2
• Wireguard

Short update on this matter: I have updated the packages listed above (and a few more) locally for every kernel release since February: So far there have only been two cases that required manual work: i) megaraid patches had to be adapted for kernel 4.18.0-301.1 as previously deprecated and disabled adapters have been re-enabled. ii) wireguard had to be updated to the latest upstream release to work with kernel 4.18.0-301.1 (the first kernel release that changed RHEL_MINOR to 5).

In all other cases I have been able to automate rebuilding these kernel modules for every newly released kernel version. Hence I see no reason why this could not also be done within CBS.

Concerning the CBS infrastructure I assume there are no issues either, or at least no issues the Hyperscale SIG doesn't have to deal with as well to provide an alternative kernel. Note: I'm currently not signing any kmod.

So in case there is any way to move this SIG proposal forward, or some other more suitable way (under the umbrella of centos-plus or some other SIG?) to provide such kmod packages, please let me know.

Peter

The second group of packages is actually more complicated to maintain than the first. Wireguard and others highly depend on upstream support.

Is there any reason why one should not be able to build these packages in CBS? Note: The kernel modules are currently not being signed.

Just let me know how I can move this proposal forward. If there is no interest by anyone, I'll just keep on building the packages I do require in private.

Best

Peter

[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

On 5/18/21 6:43 AM, Peter Georg wrote:

So in case there is any way to move this SIG proposal forward, or some other more suitable way (under the umbrella of centos-plus or some other SIG?) to provide such kmod packages, please let me know.

I'm not certain if this is what you are asking, but the way forward with the SIG proposal is to write a SIG proposal, and bring it to the Board of Directors.

The process is here: https://wiki.centos.org/SIGGuide#SIGGuide.2FSIGProcess.Proposal

A template is here: https://wiki.centos.org/SpecialInterestGroup/ProposalTemplate

A *great* example is here: https://wiki.centos.org/SpecialInterestGroup/Hyperscale

On 5/19/21 10:09 AM, Peter Georg wrote:

On 18/05/2021 19.44, Rich Bowen wrote:

...

On 5/18/21 6:43 AM, Peter Georg wrote:

...

So in case there is any way to move this SIG proposal forward, or some other more suitable way (under the umbrella of centos-plus or some other SIG?) to provide such kmod packages, please let me know.

I'm not certain if this is what you are asking, but the way forward with the SIG proposal is to write a SIG proposal, and bring it to the Board of Directors.

Indeed it was not what I was initially asking for. However this answer of yours, and the one to redbaronbrowser later, caused me to take a step back re-think about how to approach this:

The possible SIG named "Stream Kernel SIG" discussed in this topic here has been initially brought up by redbaronbrowser and included 4 goals. I later proposed to add building kmods as a fifth goal. redbaronbrowser described his idea of a possible SIG in the inital post. However I'm not sure this qualifies as a proposal. In addition, as you mentioned, there are some open questions. I think these questions are mostly related to the initial goals #1, #3, and #4. IMHO some of these goals are not feasible within the scope of a SIG.

Hence I'm now thinking about splitting the fifth goal from this "Stream Kernel SIG" and write a proposal for a "kmods" SIG only containing this particular goal.

Whether one wants to add the second goal of redbaronbrowser's initial idea - building the CentOS Plus kernel - is a different question and imo up to the current devolpers/maintainers of kernel-plus. This can be discussed later.

Before I start writing a proposal for a "kmods SIG": Does this make sense to you?

Yes, it makes sense to me.

The board seems more disposed to consider SIGs that have a clearly defined scope. The broader the scope is, the more difficult it is to write an understandable proposal, and for people to understand what's in scope and what's not.

And, presumably, if there's a lot of overlap, SIGs can be merged or split in the future to clarify. These things aren't immutable.

On 5/19/21 6:17 AM, redbaronbrowser via CentOS-devel wrote:

On Tuesday, May 18, 2021 5:43 AM, Peter Georg peter.georg@physik.uni-regensburg.de wrote:

...

So in case there is any way to move this SIG proposal forward, or some other more suitable way (under the umbrella of centos-plus or some other SIG?) to provide such kmod packages, please let me know.

Moving the SIG proposal forward requires a CentOS board member to sponsor the SIG. Period.

Or an existing SIG chair. Or me. Related: We're in the process of rewriting/clarifying the SIG process documentation over the coming months, and that was why I sent the "SIG feedback" email 2 weeks ago.

But what's really lacking here is not a board sponsor, but an actual proposal. Someone needs to write a SIG proposal (as per my email yesterday) so that someone can step up to sponsor it.

...

I have not been able to get any board member to establish themselves as a sponsor which effectively vetos the SIG. The purposal has been sitting for 3 months with no indication any sponsor will ever be stepping forward.

Primarily because there's no proposal yet. Write a proposal, using the proposal template - https://wiki.centos.org/SpecialInterestGroup/ProposalTemplate - and I'll sponsor the proposal to the board. I'm pretty sure I said that before, but I'm saying it again here - I'm kind of the default "new SIG sponsor" if nobody else explicitly steps up to do it. Consider me your sponsor.

It confuses me why it has been indicated that with Stream the board wants more SIGs but the same level of redtape to avoid them remains in place.

We have approved three new SIGs so far this year.

Feel free to jump on the IRC channel and see if any board members become available to discuss this issue.

Sure, that would be fine, but having the discussion here ensures that it's available to more people. THere has been specific feedback and questions, right here in this thread, that remain unanswered. The way forward is to write a SIG proposal that addresses those questions.

I personally have become completely demoralized and have a better understanding now why EL Repo does not make use of any CentOS resources to function.

I like your ideas and would like to help move this forward but I have no further advise at this time.