Karsten Wade has indicated the centos-devel mailing list is the place to talk about why he thinks things are going to turn out okay. He implied on the CentOS blog there is balancing of the needs of CentOS. Mostly the article is about the “openness gap” (which is brought up seven times).
He goes through a very selective stroll through history to present his point leaving out anything that doesn’t fit his agenda.
So, let’s go back to 2002. The most polished version of pre-RHEL Red Hat Linux, code named Valhalla, was released in March 2002. It was the result of years of refining the Red Hat packages in a repository called Rawhide and then a branch was released only when it passed quality assurance.
Around the same time, a branch of Valhalla was also released as Red Hat Enterprise Linux 2.
Then in September 2002, code name Psyche replaced Valhalla. Security updates for Valhalla ended unless you became a RHEL 2 customer.
Psyche was a problem for a couple reasons. While Red Hat pointed out that it had more features than RHEL, it wasn’t just the “good” features. Psyche contained several regressions and bugs that should have never passed quality assurance. Rather, it appeared whatever was the latest packages in Rawhide just got dumped as Psyche since it was just the free community version and RH QA was focused on just RHEL.
If I recall correctly, at the time Red Hat apologized for Psyche and tried to reassure the community the problems were not intentional. Red Hat 9 (Shrike) was released in March 2003 and the first Fedora Core released in November 2003.
The continued development under FC addressed several of the bugs. But two new issues came up.
First new issue was Red Hat’s creation of the openness gap. There were now two Rawhide repositories. The public Rawhide moved to Fedora. But also a Rawhide branch that RHEL work from was kept private.
Second new issue was a COMPATIBILITY gap. Fedora wasn’t designed to be 100% API or ABI compatible with RHEL. Several times a binary built on Fedora could not be run on RHEL 2. A statically linked binary might temporarily address the issue but then security updates to the RHEL 2 libraries would not fix issues with the statically linked binary. To build a dynamically linked binary for RHEL 2 without paying for RHEL 2 required using Valhalla which was no longer getting security updates.
In May 2004, a solution to the COMPATIBILITY gap was released. The Community ENTerprise OS. As stated in it’s FAQ:
“Does CentOS change the upstream Source RPMs?” “No. CentOS' key mandate for our base and updates repositories is NOT extending or enhancing packages or features beyond those supplied by the upstream Source RPM's. CentOS strives intentionally to provide binary functionality for our users.”
Now the community had two options. Fedora for features or CentOS for compatibility. It is important to remember, CentOS strived for COMPATIBILITY by *avoiding* additional features as this fundamental concept will come up later.
In October 2004, Ubuntu would be released. This would be a challenge for CentOS. While CentOS *strive* for being 100% compatible with the commercially supported RHEL, Ubuntu LTS community edition is able to prove it is 100% identical to Ubuntu LTS with commercial support.
Over time another openness gap was created. The Linux kernel is under the General Public License v2 which requires openness including notices of what files are modified and when. The Linux kernel development team accomplishes this with a public git repository. The majority of distributions (including Fedora) do this by providing the vanilla kernel source code and then each of the patches separately. RHEL/CentOS instead violated the spirit of the GPL v2 and the spirit of openness. CentOS users are given a kernel tar-ball with all the patches already applied and a changelog. The changelog does not indicate which files are modified per item but I am willing to guess that Red Hat legal has written off that they can defend this as compliant with the legal language of the GPL v2.
This violation of openness also has technical downsides. If a newer Fedora kernel breaks something, you can rebuild the kernel RPM with select patches unapplied until you locate which specific patch in the new kernel causes the bad behavior. But performing a roll-back of patches on CentOS was made difficult because by design RH provides no clear indication exactly what each patch changes.
Instead, CentOS users can open a bugzilla issue to try to get a Red Hat employee to trace down which individual patch caused the issue. But from what I can tell that usually doesn’t work out well. Maybe someone will take notice and ask for more information. However, from what I can tell, several times a project manager just closes the ticket with a statement a new version of RHEL has been released and to start back at square one with a new ticket (to be ignored again).
So, fast forward to 2013. Karsten Wade has stated at this point he led the Red Hat team to assimilate CentOS and took a seat at the CentOS Governing Board.
In 2014 the joining was announced which included this statement of reassurance to the CentOS community:
“The Red Hat Enterprise Linux to CentOS firewall will also remain. Members and contributors to the CentOS efforts are still isolated from the RHEL Groups inside Red Hat, with the only interface being srpm / source path tracking, no sooner than is considered released. In summary: we retain an upstream.”
Then fast forward to October 2018, the Linux kernel development team releases 4.19 as a long term support kernel.
In May 2019, RHEL 8 was released with a 4.18 kernel. RHEL doesn’t get the benefit of the community effort to support the LTS kernel for 6 years and the LTS kernel community doesn’t get the benefit of RHEL supporting a kernel for 10 years. Instead, RHEL continues a policy of releasing a monolithic tar-ball that obscures what each patch changes in violation of the spirit of openness (again, by design).
Then continuing on to 2020. CentOS 6 is reaching the end of life at the end of November with no plans to continue providing security updates. RHEL Extended Life Cycle subscribers will get some select security updates but under an openness gap by RH design none will be shared through CentOS. At the same time, it appears “state level actors” are leveraging security holes more than ever before.
CentOS users are given two options. First is to upgrade to CentOS 7 but then reach the end of life again in June 2024. Or skip from CentOS 6 to CentOS 8 which is stated to be supported until 2029.
Then Red Hat announces after CentOS 6 users have performed their upgrades that the real end of life that CentOS was committed to for CentOS 8 is December 2021. It is really CentOS 7 that has a longer commitment by 2.5 years.
To reassure the community there is an FAQ with the following:
“CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release”
This is a complete reversal on a fundamental concept stated earlier by CentOS. Previously the mission was to close the COMPATIBILITY gap by *NOT* “extending or enhancing packages or features.”
But Red Hat is clever enough to rename “Rawhide” to “Stream” to make it sound better. However, at the end of the day this is the same behavior and problems that gave us Red Hat Psyche, only now it is CentOS edition of Psyche that is coming.
Karsten Wade’s main sales pitch for this is addressing the openness gap. I agree that RHEL Rawhide should never have been made private. But based on the “firewall” established between CentOS and RHEL, it should be up to the RHEL team to make their Rawhide public and not involve CentOS at all. RH self-imposed openness gaps shouldn’t be used as the basis to violate the previous assurances provided to the community.
But it turns out Red Hat was not being honest about the “firewall” from the RHEL group. In fact, one of the members of the CentOS Governance Board includes a member of a RHEL group. Also, this is not any old RHEL group member, it is Brain “Bex” Exelbierd, the RHEL member that establishes the roadmap for RHEL.
WIth CentOS going from downstream provider to upstream provider, the roadmap for CentOS stops being focused on the community and comes instead from Bex for the benefit of RHEL. Exactly what CentOS users feared would happen in 2014 is now taking place.
And this is being done in the name of openness?
Where is the improved openness for CentOS users?
Where is an open transcript, chat log or voice recording of the CentOS Governance Board meeting that selected to kill CentOS 8 in favor of CentOS Rawhide?
Where is the openness in the statement there will be a firewall between CentOS and the RHEL group? Was Bex only interfacing at governance meetings via SRPMs? Was the firewall always a lie?
Where is the openness in CentOS’ commitment to support CentOS 8 to 2029?
Where is the openness in the CentOS kernel patches?
Where is the openness in having one CentOS FAQ indicating extending features beyond upstream is *NOT* part of CentOS and at the same time have another FAQ stating CentOS features will be coming ahead of RHEL?
The nicest way I can think of to put it regarding the assertion that this closes the openness gap as a benefit to CentOS users is at this point that I believe that Karsten Wade must have an intestinal tract full of copious amounts of matter.
The most insane part is this change should be bad for Red Hat itself in the long run.
CentOS should be helping attract market share for RHEL by encouraging developers to build/test on a RHEL compatible system and giving potential RHEL customers a try before you buy method.
But CentOS hasn’t been competing well as it is. Majority of AWS instances are running Ubuntu. Several Azure instances are running Ubuntu. There are several Docker Hub images that use Ubuntu as a base image and much fewer using CentOS/RHEL.
CentOS/RHEL is no longer the preferred runtime environment to develop for and where the developers go the users will eventually follow. Bringing back the COMPATIBILITY gap is not the solution to that. It is how to make things worse.
Several of the projects Karsten Wade selected to highlight seems to suggest that Red Hat sees itself as expanding into ESXi’s market share. That is a worthy goal but should not replace promoting RHEL as a preferred runtime environment. Having a RHCE should not be reduced to knowing the next generation hypervisor and modern day BIOS. The test requires knowing much more than that and RH should continue to provide maximum possible value to the people with a RH certification.
With CentOS Rawhide (or CentOS Stream if you prefer calling it that) replacing CentOS 8, the try before you buy will go to companies like Oracle and CloudLinux. But once someone resolves the COMPATIBILITY gap by going to either of those solutions for the community edition, chances are better they will go back to the same source for commercial support later as well. This is a roadmap for RH giving up more market share. Why? For claims of openness that ring empty?
It would be nice to hear from Paul Cormier why he would allow this type of behavior and insanity on the part of Red Hat but I doubt we will ever get that level of openness.
Hi Red Baron Browser.. [you had me at the name since I helped come up with it 24 years ago]
On Wed, 23 Dec 2020 at 02:57, redbaronbrowser via CentOS-devel < centos-devel@centos.org> wrote:
Karsten Wade has indicated the centos-devel mailing list is the place to talk about why he thinks things are going to turn out okay. He implied on the CentOS blog there is balancing of the needs of CentOS. Mostly the article is about the “openness gap” (which is brought up seven times).
He goes through a very selective stroll through history to present his point leaving out anything that doesn’t fit his agenda.
So, let’s go back to 2002. The most polished version of pre-RHEL Red Hat Linux, code named Valhalla, was released in March 2002. It was the result of years of refining the Red Hat packages in a repository called Rawhide and then a branch was released only when it passed quality assurance.
If you are going back in time, you need to look at the box covers for Red Hat Linux 6.0 Originally it was just going to be RHL6 with no .0 on the end.. this was a push from management to move to a continuous delivery mechanism where we would snapshot Rawhide to a release and then keep updating 6 in small incremental parts to make it easier to sell to Enterprises. We had outstanding contracts with IBM and then SGI and then HP to make a Linux which would be kept stable for 10 years with a 20 year absolute lifetime.
The problem was that engineering and support found that too much of a change and pushed back on it hard. There were still too many changes in core software to be able to keep to that and it was felt being clear on the dot release would be clearer to customers. As can be seen with the changes needed between 6 and 6.1 and 6.2 the amount of change churning was greater than expected by our contracts so we only finally got to an enterprise account with 6.2EE (or RHEL-1.0 I like to cheekily call it). That still was not able to be maintained for 10 years as the kernel, glibc, and other tools were 'behind' the Unix API needs that the vendors wanted for their software and the changes to make that caused major disruptions requiring RHL 7 (again trying very hard to keep to just a single number). Eventually this solidified and branched off before RHL7.2 to become RHEL2.1. [It was called 2.1 because the sales logic is "never ship a 1.0 because everyone expects that to be an Alpha and never ship a 2.0 because people expect that is the Beta. 2.1 is the first number accounting people will trust to buy. I thought that was bogus until after I left Red Hat and saw that was exactly how various software procurement rules worked.]
The work on 8 was not going well so a 7.3 was released to keep a release out every 6 months. That didn't help greatly and eventually a 'shoot the programmers/qa and ship the product decision' was made which went out as RHL 8. At this point the threading code in glibc got into a working state for another enterprise release and RHL9 and RHEL-3 were released.
A side story at this time was that for years there had been a thought by technical people that boxed sets was the way that Linux could make it to the masses however the sales numbers were not happening. Up until 5.0, Red Hat had always made more on t-shirt and book sales than it did from Linux software. And while there had been a boost, it fell apart with the software retail collapse that accompanied the dot-com bust. Initial sales would be good, but returns would begin to eat into this so that after a release you had 1-2 months positive and 3-5 months negative. After the dot-com bust the numbers got bigger but there was always the thought that the next release would fix things back to where they had been. That didn't happen, and RHL-10 became Fedora-1 with no plans for boxed sets, etc. This caused all kinds of problems in the community and internal to Red Hat with some people leaving for other companies (some created rPath).
Around the same time, a branch of Valhalla was also released as Red Hat Enterprise Linux 2.
Actually I believe this was started off of before Enigma. This is why some packages are before 7.2 and some are after 7.2
Then in September 2002, code name Psyche replaced Valhalla. Security updates for Valhalla ended unless you became a RHEL 2 customer.
Psyche was a problem for a couple reasons. While Red Hat pointed out that it had more features than RHEL, it wasn’t just the “good” features. Psyche contained several regressions and bugs that should have never passed quality assurance. Rather, it appeared whatever was the latest packages in Rawhide just got dumped as Psyche since it was just the free community version and RH QA was focused on just RHEL.
If I recall correctly, at the time Red Hat apologized for Psyche and tried to reassure the community the problems were not intentional. Red Hat 9 (Shrike) was released in March 2003 and the first Fedora Core released in November 2003.
The continued development under FC addressed several of the bugs. But two new issues came up.
First new issue was Red Hat’s creation of the openness gap. There were now two Rawhide repositories. The public Rawhide moved to Fedora. But also a Rawhide branch that RHEL work from was kept private.
Second new issue was a COMPATIBILITY gap. Fedora wasn’t designed to be 100% API or ABI compatible with RHEL. Several times a binary built on Fedora could not be run on RHEL 2. A statically linked binary might temporarily address the issue but then security updates to the RHEL 2 libraries would not fix issues with the statically linked binary. To build a dynamically linked binary for RHEL 2 without paying for RHEL 2 required using Valhalla which was no longer getting security updates.
In May 2004, a solution to the COMPATIBILITY gap was released. The Community ENTerprise OS. As stated in it’s FAQ:
You accidently dropped some history here. First there was WhiteBox and then a couple other rebuilds. Then the CAOS project started building their own to meet some needs for their project. That rebuild became CENTOS. [The Community Enterprise was a backronym as it was CAOS Enterprise Operating System.. It was also jokingly referred to as Cabal Enterprise Operating System since it was a small circle of people who could do the builds and the only changes being made to the package was what you list.. remove trademarks and work out how to rebuild it to be good enough.]
“Does CentOS change the upstream Source RPMs?” “No. CentOS' key mandate for our base and updates repositories is NOT
extending or enhancing packages or features beyond those supplied by the upstream Source RPM's. CentOS strives intentionally to provide binary functionality for our users.”
Now the community had two options. Fedora for features or CentOS for compatibility. It is important to remember, CentOS strived for COMPATIBILITY by *avoiding* additional features as this fundamental concept will come up later.
In October 2004, Ubuntu would be released. This would be a challenge for CentOS. While CentOS *strive* for being 100% compatible with the commercially supported RHEL, Ubuntu LTS community edition is able to prove it is 100% identical to Ubuntu LTS with commercial support.
I am not sure that CentOS cabal ever saw Ubuntu as any sort of challenge. That was more of a Red Hat challenge.. There was enough of a challenge just getting packages to build.
I have reached maximum email parsing and am not sure I can follow where you were going from here. I am clipping here as I am EBRAINFULL.
On Wednesday, December 23, 2020 10:30 AM, Stephen John Smoogen smooge@gmail.com wrote:
Hi Red Baron Browser.. [you had me at the name since I helped come up with it 24 years ago]
Did it have anything to do with Peanuts/Snoopy?
Is there anything you can do to bring back the browser and Redneck i18n support?
The insanity of CentOS 8 having a EOL date 2.5 years before CentOS 7 would be easier to handle if it at least came with a "Reckon So" agreement button.
I have reached maximum email parsing and am not sure I can follow where you were going from here. I am clipping here as I am EBRAINFULL.
Which parts got included and left out of the history won't make sense until you get to the end.
On Wed, 23 Dec 2020 at 13:21, redbaronbrowser < redbaronbrowser@protonmail.com> wrote:
On Wednesday, December 23, 2020 10:30 AM, Stephen John Smoogen < smooge@gmail.com> wrote:
Hi Red Baron Browser.. [you had me at the name since I helped come up with it 24 years ago]
Did it have anything to do with Peanuts/Snoopy?
Only in the sense that the company I worked for at the time (Spyglass Inc) wrote custom browsers for companies like IBM, Microsoft (hello Internet Explorer 1.2/2.0), Oracle, and various ISPs. Each of those browsers were usually named by the vendor as Spyglass basically sold the basic parts as Enhanced Mosaic. We asked what Red Hat wanted to call it and I believe Donnie Barnes said "whatever you want." So we went looking for a red themed name and through a lot out. [Red Bear since someone thought Linux and GPL was communist was actually the one going ahead. At this point two fortuitous items came in.. the microwaved pizza was Red Barons and the TV was playing a Peanuts episode for some of the kids of staff. Tada.. naming slapped in on the build and sent to meet 4.0 extras.
Is there anything you can do to bring back the browser and Redneck i18n support?
Not a thing. Red Baron's source code was owned by Spyglass which was bought by WebTV and then bought by... The code for Linux was terrible because Linux was the only X11 whose primary color distribution was 16bit versus 8 or 24bit like the others. This meant that the browser didn't work correctly in that until after 4.2 was shipped. Spyglass management thought of this and the Javascript support licensed from Microsoft (via IBM) were upgrades and Red Hat had them as early bug reports.
Redneck i18n work was done by Donnie Barnes partially as a dare and partially that the other language translations were taking longer than they could wait, and anyone could do the work themselves to add another one. However no one has done so in the years since because it does take a lot of work to do it without making it 'crap'.
The insanity of CentOS 8 having a EOL date 2.5 years before CentOS 7 would be easier to handle if it at least came with a "Reckon So" agreement button.
I have reached maximum email parsing and am not sure I can follow where you were going from here. I am clipping here as I am EBRAINFULL.
Which parts got included and left out of the history won't make sense until you get to the end.
For those that lived this history from the inside enough to tell it as you both do below.... Whatever part you played in the original RHL, whether for profit or for pride, you are my heroes. Thank you for starting this project. You changed the world.
On Wed., Dec. 23, 2020, 11:30 a.m. Stephen John Smoogen, smooge@gmail.com wrote:
Hi Red Baron Browser.. [you had me at the name since I helped come up with it 24 years ago]
On Wed, 23 Dec 2020 at 02:57, redbaronbrowser via CentOS-devel < centos-devel@centos.org> wrote:
Karsten Wade has indicated the centos-devel mailing list is the place to talk about why he thinks things are going to turn out okay. He implied on the CentOS blog there is balancing of the needs of CentOS. Mostly the article is about the “openness gap” (which is brought up seven times).
He goes through a very selective stroll through history to present his point leaving out anything that doesn’t fit his agenda.
So, let’s go back to 2002. The most polished version of pre-RHEL Red Hat Linux, code named Valhalla, was released in March 2002. It was the result of years of refining the Red Hat packages in a repository called Rawhide and then a branch was released only when it passed quality assurance.
If you are going back in time, you need to look at the box covers for Red Hat Linux 6.0 Originally it was just going to be RHL6 with no .0 on the end.. this was a push from management to move to a continuous delivery mechanism where we would snapshot Rawhide to a release and then keep updating 6 in small incremental parts to make it easier to sell to Enterprises. We had outstanding contracts with IBM and then SGI and then HP to make a Linux which would be kept stable for 10 years with a 20 year absolute lifetime.
The problem was that engineering and support found that too much of a change and pushed back on it hard. There were still too many changes in core software to be able to keep to that and it was felt being clear on the dot release would be clearer to customers. As can be seen with the changes needed between 6 and 6.1 and 6.2 the amount of change churning was greater than expected by our contracts so we only finally got to an enterprise account with 6.2EE (or RHEL-1.0 I like to cheekily call it). That still was not able to be maintained for 10 years as the kernel, glibc, and other tools were 'behind' the Unix API needs that the vendors wanted for their software and the changes to make that caused major disruptions requiring RHL 7 (again trying very hard to keep to just a single number). Eventually this solidified and branched off before RHL7.2 to become RHEL2.1. [It was called 2.1 because the sales logic is "never ship a 1.0 because everyone expects that to be an Alpha and never ship a 2.0 because people expect that is the Beta. 2.1 is the first number accounting people will trust to buy. I thought that was bogus until after I left Red Hat and saw that was exactly how various software procurement rules worked.]
The work on 8 was not going well so a 7.3 was released to keep a release out every 6 months. That didn't help greatly and eventually a 'shoot the programmers/qa and ship the product decision' was made which went out as RHL 8. At this point the threading code in glibc got into a working state for another enterprise release and RHL9 and RHEL-3 were released.
A side story at this time was that for years there had been a thought by technical people that boxed sets was the way that Linux could make it to the masses however the sales numbers were not happening. Up until 5.0, Red Hat had always made more on t-shirt and book sales than it did from Linux software. And while there had been a boost, it fell apart with the software retail collapse that accompanied the dot-com bust. Initial sales would be good, but returns would begin to eat into this so that after a release you had 1-2 months positive and 3-5 months negative. After the dot-com bust the numbers got bigger but there was always the thought that the next release would fix things back to where they had been. That didn't happen, and RHL-10 became Fedora-1 with no plans for boxed sets, etc. This caused all kinds of problems in the community and internal to Red Hat with some people leaving for other companies (some created rPath).
Around the same time, a branch of Valhalla was also released as Red Hat Enterprise Linux 2.
Actually I believe this was started off of before Enigma. This is why some packages are before 7.2 and some are after 7.2
Then in September 2002, code name Psyche replaced Valhalla. Security updates for Valhalla ended unless you became a RHEL 2 customer.
Psyche was a problem for a couple reasons. While Red Hat pointed out that it had more features than RHEL, it wasn’t just the “good” features. Psyche contained several regressions and bugs that should have never passed quality assurance. Rather, it appeared whatever was the latest packages in Rawhide just got dumped as Psyche since it was just the free community version and RH QA was focused on just RHEL.
If I recall correctly, at the time Red Hat apologized for Psyche and tried to reassure the community the problems were not intentional. Red Hat 9 (Shrike) was released in March 2003 and the first Fedora Core released in November 2003.
The continued development under FC addressed several of the bugs. But two new issues came up.
First new issue was Red Hat’s creation of the openness gap. There were now two Rawhide repositories. The public Rawhide moved to Fedora. But also a Rawhide branch that RHEL work from was kept private.
Second new issue was a COMPATIBILITY gap. Fedora wasn’t designed to be 100% API or ABI compatible with RHEL. Several times a binary built on Fedora could not be run on RHEL 2. A statically linked binary might temporarily address the issue but then security updates to the RHEL 2 libraries would not fix issues with the statically linked binary. To build a dynamically linked binary for RHEL 2 without paying for RHEL 2 required using Valhalla which was no longer getting security updates.
In May 2004, a solution to the COMPATIBILITY gap was released. The Community ENTerprise OS. As stated in it’s FAQ:
You accidently dropped some history here. First there was WhiteBox and then a couple other rebuilds. Then the CAOS project started building their own to meet some needs for their project. That rebuild became CENTOS. [The Community Enterprise was a backronym as it was CAOS Enterprise Operating System.. It was also jokingly referred to as Cabal Enterprise Operating System since it was a small circle of people who could do the builds and the only changes being made to the package was what you list.. remove trademarks and work out how to rebuild it to be good enough.]
“Does CentOS change the upstream Source RPMs?” “No. CentOS' key mandate for our base and updates repositories is NOT
extending or enhancing packages or features beyond those supplied by the upstream Source RPM's. CentOS strives intentionally to provide binary functionality for our users.”
Now the community had two options. Fedora for features or CentOS for compatibility. It is important to remember, CentOS strived for COMPATIBILITY by *avoiding* additional features as this fundamental concept will come up later.
In October 2004, Ubuntu would be released. This would be a challenge for CentOS. While CentOS *strive* for being 100% compatible with the commercially supported RHEL, Ubuntu LTS community edition is able to prove it is 100% identical to Ubuntu LTS with commercial support.
I am not sure that CentOS cabal ever saw Ubuntu as any sort of challenge. That was more of a Red Hat challenge.. There was enough of a challenge just getting packages to build.
I have reached maximum email parsing and am not sure I can follow where you were going from here. I am clipping here as I am EBRAINFULL.
-- Stephen J Smoogen.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/22/20 11:57 PM, redbaronbrowser via CentOS-devel wrote:
To reassure the community there is an FAQ with the following:
“CentOS Stream will be getting fixes and features ahead of RHEL.
Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release”
This is a complete reversal on a fundamental concept stated earlier by CentOS. Previously the mission was to close the COMPATIBILITY gap by *NOT* “extending or enhancing packages or features.”
At this point, I think you should be more clear than you are being. CentOS Stream will be getting features intended for RHEL, earlier than RHEL. This creates a temporary forward compatibility gap. RHEL might not run software that was compiled on CentOS Stream, until those new features actually appear in a later RHEL release. This same compatibility gap exists between point releases of RHEL. If you build an application on RHEL 7.8, for example, it might not run on RHEL 7.7.
On the other hand, CentOS Stream will be backward compatible with RHEL. Anything that runs on RHEL should continue running on CentOS Stream.
But Red Hat is clever enough to rename “Rawhide” to “Stream” to make it sound better.
No, Stream is not Rawhide.
On 23/12/2020 18:32, Gordon Messmer wrote:
At this point, I think you should be more clear than you are being. CentOS Stream will be getting features intended for RHEL, earlier than RHEL. This creates a temporary forward compatibility gap. RHEL might not run software that was compiled on CentOS Stream, until those new features actually appear in a later RHEL release. This same compatibility gap exists between point releases of RHEL. If you build an application on RHEL 7.8, for example, it might not run on RHEL 7.7.
On the other hand, CentOS Stream will be backward compatible with RHEL. Anything that runs on RHEL should continue running on CentOS Stream.
This is simply not true. Please stop perpetuating this. I have already provided evidence that the current Stream kernel is already not compatible with RHEL and that software that runs on RHEL will not run on Stream. I have 12 examples of things that run on RHEL that do not run on Stream sat in front of me right now. I will probably have more once I test the new -259 kernel that's just been released.
Take Wireguard VPN as an example. No sooner than upstream fixed the breakage caused by -257 on Monday, -259 landed and broke it again[2].
[1] https://lists.zx2c4.com/pipermail/wireguard/2020-December/006210.html [2] https://www.wireguard.com/build-status/
On 12/23/20 12:23 PM, Phil Perry wrote:
On the other hand, CentOS Stream will be backward compatible with RHEL. Anything that runs on RHEL should continue running on CentOS Stream.
This is simply not true. Please stop perpetuating this. I have already provided evidence that the current Stream kernel is already not compatible with RHEL and that software that runs on RHEL will not run on Stream.
Right, I get that this isn't true for kernel modules. The kernel isn't subject to RHEL's ABI compatibility commitments.
If you're arguing that I should have said "any user-space software" rather than "anything", then I take your point.
On Wed, Dec 23, 2020 at 08:23:29PM +0000, Phil Perry wrote:
Take Wireguard VPN as an example. No sooner than upstream fixed the breakage caused by -257 on Monday, -259 landed and broke it again[2].
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.)
On 23/12/2020 20:50, Matthew Miller wrote:
On Wed, Dec 23, 2020 at 08:23:29PM +0000, Phil Perry wrote:
Take Wireguard VPN as an example. No sooner than upstream fixed the breakage caused by -257 on Monday, -259 landed and broke it again[2].
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.)
Why would you do that? The method we use in Enterprise Linux to deliver 3rd party out-of-tree drivers is the RHEL Driver Update Programme. It has been this way for over a decade. It works really well. It just doesn't work for Stream because the Stream kernel is not suitable for end user (Enterprise) consumption - it is a development kernel for developing the next RHEL point release.
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
On Wed, Dec 23, 2020 at 4:38 PM Phil Perry pperry@elrepo.org wrote:
On 23/12/2020 20:50, Matthew Miller wrote:
On Wed, Dec 23, 2020 at 08:23:29PM +0000, Phil Perry wrote:
Take Wireguard VPN as an example. No sooner than upstream fixed the breakage caused by -257 on Monday, -259 landed and broke it again[2].
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.)
Why would you do that? The method we use in Enterprise Linux to deliver 3rd party out-of-tree drivers is the RHEL Driver Update Programme. It has been this way for over a decade. It works really well. It just doesn't work for Stream because the Stream kernel is not suitable for end user (Enterprise) consumption - it is a development kernel for developing the next RHEL point release.
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
No. The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release.
The LF/RH/SUSE kernel module packaging system (branded as the Driver Update Program by Red Hat) relies on one of two things happening to be reasonably successful:
* Gating to ensure kABI doesn't break (RHEL-style) * Continuous automatic rebuilds as the kABI changes (SUSE-style)
At work, we've internally implemented the SUSE-style strategy with our RHEL kernel module builds, but we're able to do that because our build system is designed to handle that. Within the CentOS Project with CKI/ARK and CentOS Stream, we should be implementing the RHEL-style strategy.
More than most, I get why you're upset about the kABI always breaking as kernel updates push out, but instead of just saying "it's not suitable", we should be building solutions to *make* it suitable for the Enterprise. It's *bad* that the RHEL kernel breaks its own promises so often (which is a relatively new thing, in my experience), and we should be implementing safeguards to stop it from happening going forward.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, Dec 23, 2020 at 7:15 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Dec 23, 2020 at 4:38 PM Phil Perry pperry@elrepo.org wrote:
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
No. The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release.
I think you are correct. I also think there is a long-ish road to get here. :-) Overall, it would have the best long-term results. It requires everyone that has requirements, document their requirements as automated tests.
But, it would put a damper on "new feature that needs large kernel ABI changes to cost effectively backport", such as the OverlayFS changes done in RHEL 7 as one of many such examples. The choice to use Linux 4.18 is particularly problematic, since it wasn't an LTS kernel. :-(
5 years is a long time to wait for new breaking features in the kernel.
More than most, I get why you're upset about the kABI always breaking as kernel updates push out, but instead of just saying "it's not suitable", we should be building solutions to *make* it suitable for the Enterprise. It's *bad* that the RHEL kernel breaks its own promises so often (which is a relatively new thing, in my experience), and we should be implementing safeguards to stop it from happening going forward.
Yes. Although, in the mean-time...
On Wed, Dec 23, 2020 at 7:43 PM Mark Mielke mark.mielke@gmail.com wrote:
On Wed, Dec 23, 2020 at 7:15 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Dec 23, 2020 at 4:38 PM Phil Perry pperry@elrepo.org wrote:
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
No. The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release.
I think you are correct. I also think there is a long-ish road to get here. :-) Overall, it would have the best long-term results. It requires everyone that has requirements, document their requirements as automated tests.
I'm more optimistic. The RHEL kernel for RHEL 9 is already being developed in Fedora ELN[0] through the Always Ready Kernel project[1]. As for RHEL 8 and CentOS Stream 8, we can wire up validation testing using the Zuul instance that the project had stood up as part of Fedora CI work. That infrastructure integrates with the CentOS Pagure server and we can do all kinds of interesting things with it.
[0]: https://docs.fedoraproject.org/en-US/eln/ [1]: https://gitlab.com/cki-project/kernel-ark
But, it would put a damper on "new feature that needs large kernel ABI changes to cost effectively backport", such as the OverlayFS changes done in RHEL 7 as one of many such examples. The choice to use Linux 4.18 is particularly problematic, since it wasn't an LTS kernel. :-(
5 years is a long time to wait for new breaking features in the kernel.
I think we'd be in a better place aiming for it and merely reducing the number of times it breaks. Right now, kABI breaks pretty significantly on every single RHEL point release. And there have been *several* botched backports that have screwed up both the RHEL kernel API and kernel module builds. Even just cutting the number of times that these kinds of breakages happen in half would be a major win.
To your comment about Red Hat not using LTS kernels, LTS kernels do not maintain kABI upstream either, so it doesn't save any effort for Red Hat one way or another. If anything, being based on an LTS kernel would do Red Hat less favors because they're under pressure to conform to something similar to the upstream LTS kernel. Since the upstream LTS kernel already doesn't match the RHEL kernel lifecycle and Red Hat engineers would wind up doing a bunch of work anyway for the kABI stabilization and live kernel patching features, the non-LTS kernels are strategically better because there's less churn in them and more long-term flexibility.
More than most, I get why you're upset about the kABI always breaking as kernel updates push out, but instead of just saying "it's not suitable", we should be building solutions to *make* it suitable for the Enterprise. It's *bad* that the RHEL kernel breaks its own promises so often (which is a relatively new thing, in my experience), and we should be implementing safeguards to stop it from happening going forward.
Yes. Although, in the mean-time...
Well, since nothing can really change now, I'm looking forward a bit and trying to see how we can take advantage of the situation to better the wider community. After all, the biggest virtue of a true open source community is the ability to adapt.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, Dec 23, 2020 at 8:27 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Dec 23, 2020 at 7:43 PM Mark Mielke mark.mielke@gmail.com wrote:
On Wed, Dec 23, 2020 at 7:15 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Dec 23, 2020 at 4:38 PM Phil Perry pperry@elrepo.org wrote:
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
No. The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release.
I think you are correct. I also think there is a long-ish road to get here. :-) Overall, it would have the best long-term results. It requires everyone that has requirements, document their requirements as automated tests.
I'm more optimistic. The RHEL kernel for RHEL 9 is already being developed in Fedora ELN[0] through the Always Ready Kernel project[1]. As for RHEL 8 and CentOS Stream 8, we can wire up validation testing using the Zuul instance that the project had stood up as part of Fedora CI work. That infrastructure integrates with the CentOS Pagure server and we can do all kinds of interesting things with it. [0]: https://docs.fedoraproject.org/en-US/eln/ [1]: https://gitlab.com/cki-project/kernel-ark
All good information. Thank you. I don't know if we can all afford to invest here, but we should all be thinking about it as long as we play in the EL ecosystem.
But, it would put a damper on "new feature that needs large kernel ABI changes to cost effectively backport", such as the OverlayFS changes done in RHEL 7 as one of many such examples. The choice to use Linux 4.18 is particularly problematic, since it wasn't an LTS kernel. :-( 5 years is a long time to wait for new breaking features in the kernel.
I think we'd be in a better place aiming for it and merely reducing the number of times it breaks. Right now, kABI breaks pretty significantly on every single RHEL point release. And there have been *several* botched backports that have screwed up both the RHEL kernel API and kernel module builds. Even just cutting the number of times that these kinds of breakages happen in half would be a major win.
Yes, please. :-)
The kernel isn't the only place that breaks us - but the kernel is definitely a common place. It's possible that if this was resolved, that 50%+ of the compatibility issues between minor releases that affect us (and presumedly others) would be resolved.
We still have a few systems on RHEL 6.3, because they have a particular nVidia driver for doing GPU acceleration. The newer driver, or the newer kernel, even staying within the RHEL 6.x later minor releases, exhibits random failures. We're working through it - but, it's an example of where an improvement in this area would go a long way to enabling integration between EL and 3rd party drivers. The GPU calculations are important to be accurate, so the system stays alive in current state until we can find a way forwards.
To your comment about Red Hat not using LTS kernels, LTS kernels do not maintain kABI upstream either, so it doesn't save any effort for Red Hat one way or another. If anything, being based on an LTS kernel would do Red Hat less favors because they're under pressure to conform to something similar to the upstream LTS kernel. Since the upstream LTS kernel already doesn't match the RHEL kernel lifecycle and Red Hat engineers would wind up doing a bunch of work anyway for the kABI stabilization and live kernel patching features, the non-LTS kernels are strategically better because there's less churn in them and more long-term flexibility.
This is an interesting point. Oracle UEK R6, as used in OL 7 and OL 8, is tied to Linux 5.4.17, and I think the above might be a good summary of why this is a good thing. One part of this is to set the expectation for what kABI is supported. The other part is to gain access to newer features, and reduce the cost of maintaining a high quality back port that still adheres to this kABI. By deploying Oracle UEK R6 simultaneously to both OL 7 and OL 8, they have effectively separated the kernel from the OS distro, allowing for both elements to be achieved. This seems like something that could be useful to do for the broader EL community. However, from a RHEL/CentOS perspective, it seems like a new thing, which means vendors may now support Oracle UEK R6, or the RHEL Kernel, but they wouldn't be aware of such a new thing yet.
More than most, I get why you're upset about the kABI always breaking as kernel updates push out, but instead of just saying "it's not suitable", we should be building solutions to *make* it suitable for the Enterprise. It's *bad* that the RHEL kernel breaks its own promises so often (which is a relatively new thing, in my experience), and we should be implementing safeguards to stop it from happening going forward.
Yes. Although, in the mean-time...
Well, since nothing can really change now, I'm looking forward a bit and trying to see how we can take advantage of the situation to better the wider community. After all, the biggest virtue of a true open source community is the ability to adapt.
I will do something similar. I mostly watch this list (and many others) the last few years, but this change was something that I wanted to ensure was represented a little better. Not sure if I did a good job or not, but - once all the things that can be said are said, and the dust settles, it's exactly as you say. We will adapt. We'll take advantage of new opportunities and new capabilities, and we'll fill any gaps left over according to our needs.
On Wed, Dec 23, 2020, at 9:25 PM, Mark Mielke wrote:
On Wed, Dec 23, 2020 at 8:27 PM Neal Gompa ngompa13@gmail.com wrote:
To your comment about Red Hat not using LTS kernels, LTS kernels do not maintain kABI upstream either, so it doesn't save any effort for Red Hat one way or another. If anything, being based on an LTS kernel would do Red Hat less favors because they're under pressure to conform to something similar to the upstream LTS kernel. Since the upstream LTS kernel already doesn't match the RHEL kernel lifecycle and Red Hat engineers would wind up doing a bunch of work anyway for the kABI stabilization and live kernel patching features, the non-LTS kernels are strategically better because there's less churn in them and more long-term flexibility.
This is an interesting point. Oracle UEK R6, as used in OL 7 and OL 8, is tied to Linux 5.4.17, and I think the above might be a good summary of why this is a good thing. One part of this is to set the expectation for what kABI is supported. The other part is to gain access to newer features, and reduce the cost of maintaining a high quality back port that still adheres to this kABI. By deploying Oracle UEK R6 simultaneously to both OL 7 and OL 8, they have effectively separated the kernel from the OS distro, allowing for both elements to be achieved. This seems like something that could be useful to do for the broader EL community.
Red Hat already does this in effect, supporting RHEL 6,7,8 user spaces on RHEL 7 and 8 kernels, via containers.
V/r, James Cassell
On Thu, Dec 24, 2020 at 10:21 AM James Cassell fedoraproject@cyberpear.com wrote:
On Wed, Dec 23, 2020, at 9:25 PM, Mark Mielke wrote:
This is an interesting point. Oracle UEK R6, as used in OL 7 and OL 8, is tied to Linux 5.4.17, and I think the above might be a good summary of why this is a good thing. One part of this is to set the expectation for what kABI is supported. The other part is to gain access to newer features, and reduce the cost of maintaining a high quality back port that still adheres to this kABI. By deploying Oracle UEK R6 simultaneously to both OL 7 and OL 8, they have effectively separated the kernel from the OS distro, allowing for both elements to be achieved. This seems like something that could be useful to do for the broader EL community.
Red Hat already does this in effect, supporting RHEL 6,7,8 user spaces on RHEL 7 and 8 kernels, via containers.
I'm not sure if I follow your reasoning... If you are saying that "this permits somebody to run Linux 5.4.17 on the host, and the user spaces can be provided by containers", then sure. But, that's not really what Oracle UEK R6 is. With Oracle UEK R6, you can "yum install kernel-uek" on either OL 7.9 or OL 8.3, and in both cases, you will get Linux 5.4.17 installed that you can boot into, on the host.
Yes, the userspace is often resilient to changes in kernels - that's how containers generally work well, and that's how Oracle UEK R6 can be so easily applied against an EL 7.x or EL 8.x machines, but actually providing a choice of different kernels to use, is not something that RHEL provides today (at least not something that comes to mind for me, although perhaps I'm missing some specialty repository, such as RHE-V?).
On Thu, Dec 24, 2020 at 10:21 AM James Cassell fedoraproject@cyberpear.com wrote:
On Wed, Dec 23, 2020, at 9:25 PM, Mark Mielke wrote:
This is an interesting point. Oracle UEK R6, as used in OL 7 and OL 8, is tied to Linux 5.4.17, and I think the above might be a good summary of why this is a good thing. One part of this is to set the expectation for what kABI is supported. The other part is to gain access to newer features, and reduce the cost of maintaining a high quality back port that still adheres to this kABI. By deploying Oracle UEK R6 simultaneously to both OL 7 and OL 8, they have effectively separated the kernel from the OS distro, allowing for both elements to be achieved. This seems like something that could be useful to do for the broader EL community.
Red Hat already does this in effect, supporting RHEL 6,7,8 user spaces on RHEL 7 and 8 kernels, via containers.
I'm not sure if I follow your reasoning... If you are saying that "this permits somebody to run Linux 5.4.17 on the host, and the user spaces can be provided by containers", then sure. But, that's not really what Oracle UEK R6 is. With Oracle UEK R6, you can "yum install kernel-uek" on either OL 7.9 or OL 8.3, and in both cases, you will get Linux 5.4.17 installed that you can boot into, on the host.
Yes, the userspace is often resilient to changes in kernels - that's how containers generally work well, and that's how Oracle UEK R6 can be so easily applied against an EL 7.x or EL 8.x machines, but actually providing a choice of different kernels to use, is not something that RHEL provides today (at least not something that comes to mind for me, although perhaps I'm missing some specialty repository, such as RHE-V?).
As for Oracle UEK kernels, the UEK repo also includes some updated packages which replace the original distro packages to make things work.
If you don't need UEK you can remove the UEK kernels, disable the UEK repos and do a distro-sync to have a clean again.
Regards, Simon
On Fri, Dec 25, 2020 at 4:11 AM Simon Matter simon.matter@invoca.ch wrote:
On Thu, Dec 24, 2020 at 10:21 AM James Cassell fedoraproject@cyberpear.com wrote:
On Wed, Dec 23, 2020, at 9:25 PM, Mark Mielke wrote:
This is an interesting point. Oracle UEK R6, as used in OL 7 and OL 8, is tied to Linux 5.4.17, and I think the above might be a good summary of why this is a good thing. One part of this is to set the expectation for what kABI is supported. The other part is to gain access to newer features, and reduce the cost of maintaining a high quality back port that still adheres to this kABI. By deploying Oracle UEK R6 simultaneously to both OL 7 and OL 8, they have effectively separated the kernel from the OS distro, allowing for both elements to be achieved. This seems like something that could be useful to do for the broader EL community.
Red Hat already does this in effect, supporting RHEL 6,7,8 user spaces on RHEL 7 and 8 kernels, via containers.
I'm not sure if I follow your reasoning... If you are saying that "this permits somebody to run Linux 5.4.17 on the host, and the user spaces can be provided by containers", then sure. But, that's not really what Oracle UEK R6 is. With Oracle UEK R6, you can "yum install kernel-uek" on either OL 7.9 or OL 8.3, and in both cases, you will get Linux 5.4.17 installed that you can boot into, on the host.
Yes, the userspace is often resilient to changes in kernels - that's how containers generally work well, and that's how Oracle UEK R6 can be so easily applied against an EL 7.x or EL 8.x machines, but actually providing a choice of different kernels to use, is not something that RHEL provides today (at least not something that comes to mind for me, although perhaps I'm missing some specialty repository, such as RHE-V?).
As for Oracle UEK kernels, the UEK repo also includes some updated packages which replace the original distro packages to make things work.
If you don't need UEK you can remove the UEK kernels, disable the UEK repos and do a distro-sync to have a clean again.
Regards, Simon
A "distro-sync" isn't enough. You need to replace *every single RPM*, and pray that no RPM's have tweaks in the ir '%script' operations that are not idempotent. It's actually not a bad bet: I've done it for RHEL, CentOS, and Scientific Linux hosts to pull the host off of or onto a license.
On 24/12/2020 00:14, Neal Gompa wrote:
On Wed, Dec 23, 2020 at 4:38 PM Phil Perry pperry@elrepo.org wrote:
On 23/12/2020 20:50, Matthew Miller wrote:
On Wed, Dec 23, 2020 at 08:23:29PM +0000, Phil Perry wrote:
Take Wireguard VPN as an example. No sooner than upstream fixed the breakage caused by -257 on Monday, -259 landed and broke it again[2].
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.)
Why would you do that? The method we use in Enterprise Linux to deliver 3rd party out-of-tree drivers is the RHEL Driver Update Programme. It has been this way for over a decade. It works really well. It just doesn't work for Stream because the Stream kernel is not suitable for end user (Enterprise) consumption - it is a development kernel for developing the next RHEL point release.
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
No. The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release.
Blocking Stream kernel updates you mean?
That would certainly be an option, and I have written a yum plugin (for el7) that does the reverse and masks kmod packages from the yum transaction where the required kernel is not available yet. But for such an approach to work, it is essential that the Stream repository contains all kernel releases, not just the latest as is the case at present.
Further, we have an issue with the Stream installation images which are constantly being updates during the latest compose and feature the latest Stream kernel - these are unable to use Driver Update Disk images (DUDs) which are generally built around the point release GA kernel and are likely not compatible with newer Stream kernels.
The LF/RH/SUSE kernel module packaging system (branded as the Driver Update Program by Red Hat) relies on one of two things happening to be reasonably successful:
- Gating to ensure kABI doesn't break (RHEL-style)
- Continuous automatic rebuilds as the kABI changes (SUSE-style)
At work, we've internally implemented the SUSE-style strategy with our RHEL kernel module builds, but we're able to do that because our build system is designed to handle that. Within the CentOS Project with CKI/ARK and CentOS Stream, we should be implementing the RHEL-style strategy.
More than most, I get why you're upset about the kABI always breaking as kernel updates push out, but instead of just saying "it's not suitable", we should be building solutions to *make* it suitable for the Enterprise. It's *bad* that the RHEL kernel breaks its own promises so often (which is a relatively new thing, in my experience), and we should be implementing safeguards to stop it from happening going forward.
To be fair to Red Hat, they are not breaking their own promises (nor even the kABI by their own definition) as Red Hat only strive to retain kABI compatibility for symbols on their own defined whitelist.
What happens in reality (especially in the first 5 years during the active development phase or Stream phase) is that Red Hat branch the RHEL kernel at point release time and the 8.3 kernel, for example, stays stable for 6 months with only important bug fix and security fixes, but no new features whilst the RHEL development kernel branch for 8.4, which is now being released to Stream, gets all the big backports that will be in the 8.4 kernel, and those backports are what causes breakage of symbols that are not on the kABI whitelist but are used in the real world by many/most 3rd party drivers.
It is really important that this process happens. If it didn't, we wouldn't for example get a new WiFi stack in RHEL8.1 backported from kernel-5.2 or in RHEL8.3 backported from kernel-5.7 and none of our fancy new WiFi adapters would work.
It's also really important that this process is being opened up if Red Hat want people (the community) outside of the Red Hat kernel development team to be able to contribute to it.
So I'm absolutely not against it and nor do I want to prevent or stop it from happening. Quite the opposite - I am really looking forward to the day I can contribute simple fixes to the RHEL kernel rather than having to file a bug and wait months/years to see the incorporation of a simple upstream fix or have to open a support case and spend months dealing people that do not understand the issue. But above all I just want people to recognise that this is a *development* system and stop trying to tell people that is it a drop in replacement for CentOS Linux because it is not.
On Thu, Dec 24, 2020 at 8:22 AM Phil Perry pperry@elrepo.org wrote:
On 24/12/2020 00:14, Neal Gompa wrote:
On Wed, Dec 23, 2020 at 4:38 PM Phil Perry pperry@elrepo.org wrote:
On 23/12/2020 20:50, Matthew Miller wrote:
On Wed, Dec 23, 2020 at 08:23:29PM +0000, Phil Perry wrote:
Take Wireguard VPN as an example. No sooner than upstream fixed the breakage caused by -257 on Monday, -259 landed and broke it again[2].
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.)
Why would you do that? The method we use in Enterprise Linux to deliver 3rd party out-of-tree drivers is the RHEL Driver Update Programme. It has been this way for over a decade. It works really well. It just doesn't work for Stream because the Stream kernel is not suitable for end user (Enterprise) consumption - it is a development kernel for developing the next RHEL point release.
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
No. The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release.
Blocking Stream kernel updates you mean?
That would certainly be an option, and I have written a yum plugin (for el7) that does the reverse and masks kmod packages from the yum transaction where the required kernel is not available yet. But for such an approach to work, it is essential that the Stream repository contains all kernel releases, not just the latest as is the case at present.
Further, we have an issue with the Stream installation images which are constantly being updates during the latest compose and feature the latest Stream kernel - these are unable to use Driver Update Disk images (DUDs) which are generally built around the point release GA kernel and are likely not compatible with newer Stream kernels.
I'm a bit more ambitious here: I'd like kernel updates to not be released *at all* to users unless it's validated alongside kernel module packages.
The LF/RH/SUSE kernel module packaging system (branded as the Driver Update Program by Red Hat) relies on one of two things happening to be reasonably successful:
- Gating to ensure kABI doesn't break (RHEL-style)
- Continuous automatic rebuilds as the kABI changes (SUSE-style)
At work, we've internally implemented the SUSE-style strategy with our RHEL kernel module builds, but we're able to do that because our build system is designed to handle that. Within the CentOS Project with CKI/ARK and CentOS Stream, we should be implementing the RHEL-style strategy.
More than most, I get why you're upset about the kABI always breaking as kernel updates push out, but instead of just saying "it's not suitable", we should be building solutions to *make* it suitable for the Enterprise. It's *bad* that the RHEL kernel breaks its own promises so often (which is a relatively new thing, in my experience), and we should be implementing safeguards to stop it from happening going forward.
To be fair to Red Hat, they are not breaking their own promises (nor even the kABI by their own definition) as Red Hat only strive to retain kABI compatibility for symbols on their own defined whitelist.
What happens in reality (especially in the first 5 years during the active development phase or Stream phase) is that Red Hat branch the RHEL kernel at point release time and the 8.3 kernel, for example, stays stable for 6 months with only important bug fix and security fixes, but no new features whilst the RHEL development kernel branch for 8.4, which is now being released to Stream, gets all the big backports that will be in the 8.4 kernel, and those backports are what causes breakage of symbols that are not on the kABI whitelist but are used in the real world by many/most 3rd party drivers.
Right, but I think Brendan Conoboy has said elsewhere on this list that they want to expand the kABI whitelist as much as practically possible based on real-world data. We have a large repository of these with ELRepo (among others) and we can use that for helping the RHEL team expand the whitelist so that they don't break so often.
It is really important that this process happens. If it didn't, we wouldn't for example get a new WiFi stack in RHEL8.1 backported from kernel-5.2 or in RHEL8.3 backported from kernel-5.7 and none of our fancy new WiFi adapters would work.
Yeah, I know. The work that's done there is extremely impressive and I love that they do it.
It's also really important that this process is being opened up if Red Hat want people (the community) outside of the Red Hat kernel development team to be able to contribute to it.
I'm pretty sure that judging by Mike and Brendan's comments that this is absolutely part of the strategy here.
So I'm absolutely not against it and nor do I want to prevent or stop it from happening. Quite the opposite - I am really looking forward to the day I can contribute simple fixes to the RHEL kernel rather than having to file a bug and wait months/years to see the incorporation of a simple upstream fix or have to open a support case and spend months dealing people that do not understand the issue. But above all I just want people to recognise that this is a *development* system and stop trying to tell people that is it a drop in replacement for CentOS Linux because it is not.
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
And for the couple of third-party kernel modules we use, our build system already triggers automatic rebuilds when new kernel builds are released. Heck, I support *Fedora* just fine because of that. So the kernel thing has been a non-issue for me. However, I recognize that most people haven't put in the engineering effort *I* have to support that kind of thing, and Red Hat's kABI promises are intended to make it so people don't have to. So we need to improve processes around that to make kernel updates less painful for third party kernel module maintainers, since that's a huge part of the value of the RHEL/CentOS ecosystem.
CentOS Stream 8 may suck a bit for that *now*, but it doesn't have to remain that way. In the future, it could make both RHEL and CentOS even better in this regard if we take advantage of this opportunity.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Thu, Dec 24, 2020 at 5:38 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Dec 24, 2020 at 8:22 AM Phil Perry pperry@elrepo.org wrote:
[snip]
Further, we have an issue with the Stream installation images which are constantly being updates during the latest compose and feature the latest Stream kernel - these are unable to use Driver Update Disk images (DUDs) which are generally built around the point release GA kernel and are likely not compatible with newer Stream kernels.
I'm a bit more ambitious here: I'd like kernel updates to not be released *at all* to users unless it's validated alongside kernel module packages.
I suspect this would be problematic for some CentOS Stream use cases, but providing a mechanism (Like a second repo) to flag kernels as good-to-go with a community-defined kernel module compatibility baseline makes sense to me. The automation required seems feasible. I imagine it's a question of somebody doing the work. Seems like a good community building opportunity to me.
What happens in reality (especially in the first 5 years during the active development phase or Stream phase) is that Red Hat branch the RHEL kernel at point release time and the 8.3 kernel, for example, stays stable for 6 months with only important bug fix and security fixes, but no new features whilst the RHEL development kernel branch for 8.4, which is now being released to Stream, gets all the big backports that will be in the 8.4 kernel, and those backports are what causes breakage of symbols that are not on the kABI whitelist but are used in the real world by many/most 3rd party drivers.
Right, but I think Brendan Conoboy has said elsewhere on this list that they want to expand the kABI whitelist as much as practically possible based on real-world data. We have a large repository of these with ELRepo (among others) and we can use that for helping the RHEL team expand the whitelist so that they don't break so often.
I do think CentOS Stream could be one of the feeders for high value symbols. We have to be cognizant that sometimes a symbol can't be added to the list no matter how much we want to, because it would make backporting upstream work impossible. Because of this I think it's better to focus efforts on managing the impact of symbols changing. Work in this area can be beneficial for the providers of kernel modules and for end users at the same time.
[snip]
So I'm absolutely not against it and nor do I want to prevent or stop it from happening. Quite the opposite - I am really looking forward to the day I can contribute simple fixes to the RHEL kernel rather than having to file a bug and wait months/years to see the incorporation of a simple upstream fix or have to open a support case and spend months dealing people that do not understand the issue. But above all I just want people to recognise that this is a *development* system and stop trying to tell people that is it a drop in replacement for CentOS Linux because it is not.
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
And for the couple of third-party kernel modules we use, our build system already triggers automatic rebuilds when new kernel builds are released. Heck, I support *Fedora* just fine because of that. So the kernel thing has been a non-issue for me. However, I recognize that most people haven't put in the engineering effort *I* have to support that kind of thing, and Red Hat's kABI promises are intended to make it so people don't have to. So we need to improve processes around that to make kernel updates less painful for third party kernel module maintainers, since that's a huge part of the value of the RHEL/CentOS ecosystem.
CentOS Stream 8 may suck a bit for that *now*, but it doesn't have to remain that way. In the future, it could make both RHEL and CentOS even better in this regard if we take advantage of this opportunity.
This is really well said, Neal. With a little work on these fundamentals I suspect we can make CentOS Stream do for everybody what you've had to do for yourself.
On 12/24/20 2:37 PM, Neal Gompa wrote:
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
As I understood it, Stream is not in full swing yet, there is no active/daily contribution from RHEL team? What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/24/20 2:37 PM, Neal Gompa wrote:
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
As I understood it, Stream is not in full swing yet, there is no active/daily contribution from RHEL team?
"Active/daily" contribution is pretty much a seasonal thing unrelated to the actual processes. Right now majority of engineers contributing to RHEL 8 sources are on their holidays and end of year vacation time. You can notice these "activity dives" also in Fedora Project annual reports by Matthew Miller, like https://mattdm.org/fedora/2020nest/StateOfFedora2020.pdf. This is a well-known phenomenon in Fedora community.
Anything that goes into RHEL 8.next distribution composes is synchronized to CentOS 8 Stream. That means CentOS 8 Stream is defined by the content in RHEL 8.next distribution composes and the packages in there are contributed to only by RHEL engineers. They do not build the packages in CentOS 8 Stream themselves, at least now.
To show a similarity, Fedora ELN is also not being built explicitly: you commit to Rawhide dist-git and build a package in Rawhide, then Fedora ELN build is kicked off by a Fedora messaging bus event. Think of RHEL->CentOS 8 Stream flow in a similar way, only with a number of additional, mandatory, automated and manual, steps in between, according to various RHEL QA processes.
Since there are thousands of packages in RHEL and there are development stages where more activity happens than in other time, naturally, you may see spikes of that activity along with relative periods of a 'cooldown'. Again, in some areas of the distribution that activity might be more exposed than in the others. In past, there was a general reduction of activity close to externally visible milestones like RHEL beta releases and after that. Hopefully, we'll see it with CentOS Stream too -- after all, if all content comes there automatically from RHEL development, then slowdown in RHEL development for milestones will be reflected in CentOS Stream (almost) automatically. Sure, updates do not come immediately but that's defined by the fact that CentoS Stream tracks RHEL development composes, not RHEL dist-git commits and individual package builds.
For example, when I released FreeIPA 4.9.0 release candidate 3 upstream on December 10th, it was imported into RHEL 8.next development tree on December 11th. Later, it was built on the same day as a idm-DL1-8040020201211123410.1f8cbe47 module stream build, and spent five days in the QA process before it got into RHEL 8.next nightly composes. Once it appeared there, all SRPM packages that were included into idm-DL1-8040020201211123410.1f8cbe47 module stream build were imported into corresponding repositories on git.centos.org -- all this information can be seen in the git commits done by the scripts that import the content.
It took some time to figure out how to rebuild idm:DL1 module stream in CentOS 8 Stream. In fact, Carl had to do some manual fixes which he contributed FreeIPA upstream https://github.com/freeipa/freeipa/pull/5367 and which will appear on RHEL 8.next once FreeIPA 4.9.0 final version will be built there after holidays/vacation season ends, thus bringing it back to CentOS 8 Stream as a part of the upstream changes. But the RHEL 8.next's idm-DL1-8040020201211123410.1f8cbe47 module stream built finally landed in CentOS 8 Stream as idm-DL1-8040020201219215824.1f8cbe47 module stream build on December 19th. So, at least three days delay between RHEL 8.next compose getting the packages and CentOS 8 Stream getting their rebuilds. Less than RHEL QE team spent qualifying the build themselves.
My understanding of what is not in 'full swing' is fully automated processing of these updates. There is still a lot of domain knowledge that is not automated. Most common one is an order of builds. There is also a complexity with module stream builds -- you need to know whether a particular package belongs to a specific module stream and don't try to rebuild it without rebuilding the whole stream (and don't build the stream without collecting all changes to that stream's packages first). It is certainly possible to do so based on the datastamps in the SRPM packages in RHEL composes but my understanding is that it is not yet utilized. Hopefully, there will be a better way to do so.
What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
You certainly control when you can and want upgrade your deployment systems. It has nothing to do with the cadence of updates coming into a distribution.
I find this fixation on the kernel updates is skewing things a lot. Kernel, certainly, is important, but it is not the thing that is RHEL or CentOS distribution, alone.
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/24/20 2:37 PM, Neal Gompa wrote:
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
As I understood it, Stream is not in full swing yet, there is no active/daily contribution from RHEL team?
"Active/daily" contribution is pretty much a seasonal thing unrelated to the actual processes. Right now majority of engineers contributing to RHEL 8 sources are on their holidays and end of year vacation time. You can notice these "activity dives" also in Fedora Project annual reports by Matthew Miller, like https://mattdm.org/fedora/2020nest/StateOfFedora2020.pdf. This is a well-known phenomenon in Fedora community.
Johnny Hughes wrote that Stream is not yet ready, that certain parts are not in place. I did not bother to retain specifics since I will not be using it at any time in the future, I only need to know enough to notify or answer on Facebook where I am admin.
<snip>
What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
You certainly control when you can and want upgrade your deployment systems. It has nothing to do with the cadence of updates coming into a distribution.
I find this fixation on the kernel updates is skewing things a lot. Kernel, certainly, is important, but it is not the thing that is RHEL or CentOS distribution, alone.
It is crucial issue if you install any kernel module not provided by Red Hat (3rd party drivers). If some software that you might or might not use brakes, you can mess around your working system and fix it. But if after dnf update your system crashes or network is down, and it is bare metal system, they you are f**ked, you need to reach the system manually (I install on regular PC hardware without ILO) and reverse to prior kernel. Even if I am quick about it, it will be very embarrassing for me in front of my clients (small in number as they are) whose work will stop for that period, so I will not be caught dead using CentOS Stream, I do not need the potential headache, embarrassment.
Le 27/12/2020 à 13:08, Ljubomir Ljubojevic a écrit :
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/24/20 2:37 PM, Neal Gompa wrote:
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
As I understood it, Stream is not in full swing yet, there is no active/daily contribution from RHEL team?
"Active/daily" contribution is pretty much a seasonal thing unrelated to the actual processes. Right now majority of engineers contributing to RHEL 8 sources are on their holidays and end of year vacation time. You can notice these "activity dives" also in Fedora Project annual reports by Matthew Miller, like https://mattdm.org/fedora/2020nest/StateOfFedora2020.pdf. This is a well-known phenomenon in Fedora community.
Johnny Hughes wrote that Stream is not yet ready, that certain parts are not in place. I did not bother to retain specifics since I will not be using it at any time in the future, I only need to know enough to notify or answer on Facebook where I am admin.
<snip>
What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
You certainly control when you can and want upgrade your deployment systems. It has nothing to do with the cadence of updates coming into a distribution.
I find this fixation on the kernel updates is skewing things a lot. Kernel, certainly, is important, but it is not the thing that is RHEL or CentOS distribution, alone.
It is crucial issue if you install any kernel module not provided by Red Hat (3rd party drivers). If some software that you might or might not use brakes, you can mess around your working system and fix it. But if after dnf update your system crashes or network is down, and it is bare metal system, they you are f**ked, you need to reach the system manually (I install on regular PC hardware without ILO) and reverse to prior kernel. Even if I am quick about it, it will be very embarrassing for me in front of my clients (small in number as they are) whose work will stop for that period, so I will not be caught dead using CentOS Stream, I do not need the potential headache, embarrassment.
Move to Oracle UEK kernels should be the solution, they use the same updated version for EL7 and EL8.
Jean-Marc
*Jean-Marc LIGER* Chef du Service Informatique UFR de Médecine de Paris Centre 15 rue de l'Ecole de médecine 75270 PARIS CEDEX 06 +33 (0)1 53 10 46 82
https://www.facebook.com/univparis/ https://www.instagram.com/univ_paris/?hl=fr https://twitter.com/Univ_Paris https://www.youtube.com/channel/UC5WhlXe1I_eRCtEx7cj3ZKg?fbclid=IwAR1vwGdU3uTudr9xzGkiXoNdg0tlasAeDl2lkkTP-cyEP_snhPMfdU_Oc9w
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/24/20 2:37 PM, Neal Gompa wrote:
In the strictest sense, it obviously is not. But in a very real practical sense, it absolutely is. Aside from the kernel issues (which I firmly believe are solvable), people are generally not going to notice a difference between CentOS Linux 8 and CentOS Stream 8.
My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the spring because it was strictly better for production *and* development. I've been in the process of opportunistically switching our build targets from CentOS Linux 8 to CentOS Stream 8 most of the year. With the retirement of CentOS Linux 8, it now becomes more of a priority, but it was already going to happen.
As I understood it, Stream is not in full swing yet, there is no active/daily contribution from RHEL team?
"Active/daily" contribution is pretty much a seasonal thing unrelated to the actual processes. Right now majority of engineers contributing to RHEL 8 sources are on their holidays and end of year vacation time. You can notice these "activity dives" also in Fedora Project annual reports by Matthew Miller, like https://mattdm.org/fedora/2020nest/StateOfFedora2020.pdf. This is a well-known phenomenon in Fedora community.
Johnny Hughes wrote that Stream is not yet ready, that certain parts are not in place. I did not bother to retain specifics since I will not be using it at any time in the future, I only need to know enough to notify or answer on Facebook where I am admin.
<snip>
What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
You certainly control when you can and want upgrade your deployment systems. It has nothing to do with the cadence of updates coming into a distribution.
I find this fixation on the kernel updates is skewing things a lot. Kernel, certainly, is important, but it is not the thing that is RHEL or CentOS distribution, alone.
It is crucial issue if you install any kernel module not provided by Red Hat (3rd party drivers). If some software that you might or might not use brakes, you can mess around your working system and fix it. But if after dnf update your system crashes or network is down, and it is bare metal system, they you are f**ked, you need to reach the system manually (I install on regular PC hardware without ILO) and reverse to prior kernel. Even if I am quick about it, it will be very embarrassing for me in front of my clients (small in number as they are) whose work will stop for that period, so I will not be caught dead using CentOS Stream, I do not need the potential headache, embarrassment.
That's pretty obvious with any system, really, no need to repeat that. I think this topic was raised multiple times (by you and others already) to realize that. In an ideal collaborative world, perhaps, those 3rd-party drivers could be build and tested automatically on top of the CentOS Stream, though we are yet to reach that point of collaboration.
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
On 27.12.2020 21:48, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
Looking-glass (upside-down) marketing?
Normal approach would be "study the community needs -> develop and announce subscription plans -> announce CentOS shutdown".
Something tells me the above steps are being applied in reverse order (with 'study" step optional).
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 21:48, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Let's be clear: the above 'diff' is your own opinion and a speculation, not based on any public information. There are no facts that would support a claim that future RHEL subscription programs we are promised will all be paid ones.
I certainly hope some of those programs would support free use of RHEL. https://www.redhat.com/en/blog/faq-centos-stream-updates#Q12 unfortunately does not give a clear 'fee' or 'free' clarification yet but it certainly does not claim they will all be paid ones.
Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
Looking-glass (upside-down) marketing?
Normal approach would be "study the community needs -> develop and announce subscription plans -> announce CentOS shutdown".
Something tells me the above steps are being applied in reverse order (with 'study" step optional).
I have had no exposure to how the process was done. I am looking here at a way to produce something that would work in future, not spending time to fight with a past. Can we focus on the future? This is development list, after all.
I can see two major ways of enabling 3rd-party drivers: - using elrepo's excellent work on kmods to build a CI for CentOS Stream kernel updates and perhaps block updates based on CI results' consensus (not necessary blocked until it 100% green)
- for proprietary drivers make sure there are clear ways to enable those partners to plug into the same process as an external CI.
Both can be achieved by forming a SIG, contributing some resources, and having a shared set of common CI tools. CKI already gives a platform to base on, for kernel-specific components. Fedora Project already gives a way to organize CIs around similar topics for other components, so reusing the tooling is definitely possible.
On 27.12.2020 23:00, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 21:48, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Let's be clear: the above 'diff' is your own opinion and a speculation, not based on any public information. There are no facts that would support a claim that future RHEL subscription programs we are promised will all be paid ones.
Let's be clear: quantifier "all" is your own interpretation and was not assumed in my statement.
I certainly hope some of those programs would support free use of RHEL. https://www.redhat.com/en/blog/faq-centos-stream-updates#Q12 unfortunately does not give a clear 'fee' or 'free' clarification yet but it certainly does not claim they will all be paid ones.
We will see, although it would be much more logical to know the options *before* the CentOS Linux EOL announcement.
Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
Looking-glass (upside-down) marketing?
Normal approach would be "study the community needs -> develop and announce subscription plans -> announce CentOS shutdown".
Something tells me the above steps are being applied in reverse order (with 'study" step optional).
I have had no exposure to how the process was done. I am looking here at a way to produce something that would work in future, not spending time to fight with a past. Can we focus on the future? This is development list, after all.
Certainly we can focus on the future. I just comment what I see and can only hope that upside-down approach won't become a habit.
I can see two major ways of enabling 3rd-party drivers: - using elrepo's excellent work on kmods to build a CI for CentOS Stream kernel updates and perhaps block updates based on CI results' consensus (not necessary blocked until it 100% green)
- for proprietary drivers make sure there are clear ways to enable those partners to plug into the same process as an external CI.
Both can be achieved by forming a SIG, contributing some resources, and having a shared set of common CI tools. CKI already gives a platform to base on, for kernel-specific components. Fedora Project already gives a way to organize CIs around similar topics for other components, so reusing the tooling is definitely possible.
The primary moot point of CentOS Stream usability, as I see it, is lack of simple means to rollback (one or more) packages, to restore a predictable software behavior after another daily update breaks something.
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 23:00, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 21:48, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 12:29 PM, Alexander Bokovoy wrote:
On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Let's be clear: the above 'diff' is your own opinion and a speculation, not based on any public information. There are no facts that would support a claim that future RHEL subscription programs we are promised will all be paid ones.
Let's be clear: quantifier "all" is your own interpretation and was not assumed in my statement.
I find it strange to add 'paid' where it is not necessary needed to be if you have no facts to say so. In fact, Chris Wrights blog is very explicit: https://www.redhat.com/en/blog/centos-stream-building-innovative-future-ente... "In the first half of 2021, we plan to introduce low- or no-cost programs for a variety of use cases, including options for open source projects and communities and expansion of the Red Hat Enterprise Linux Developer subscription use cases to better serve the needs of systems administrators. We’ll share more details as these initiatives coalesce."
I certainly hope some of those programs would support free use of RHEL. https://www.redhat.com/en/blog/faq-centos-stream-updates#Q12 unfortunately does not give a clear 'fee' or 'free' clarification yet but it certainly does not claim they will all be paid ones.
We will see, although it would be much more logical to know the options *before* the CentOS Linux EOL announcement.
Yep.
Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
Looking-glass (upside-down) marketing?
Normal approach would be "study the community needs -> develop and announce subscription plans -> announce CentOS shutdown".
Something tells me the above steps are being applied in reverse order (with 'study" step optional).
I have had no exposure to how the process was done. I am looking here at a way to produce something that would work in future, not spending time to fight with a past. Can we focus on the future? This is development list, after all.
Certainly we can focus on the future. I just comment what I see and can only hope that upside-down approach won't become a habit.
I can see two major ways of enabling 3rd-party drivers: - using elrepo's excellent work on kmods to build a CI for CentOS Stream kernel updates and perhaps block updates based on CI results' consensus (not necessary blocked until it 100% green)
- for proprietary drivers make sure there are clear ways to enable those partners to plug into the same process as an external CI.
Both can be achieved by forming a SIG, contributing some resources, and having a shared set of common CI tools. CKI already gives a platform to base on, for kernel-specific components. Fedora Project already gives a way to organize CIs around similar topics for other components, so reusing the tooling is definitely possible.
The primary moot point of CentOS Stream usability, as I see it, is lack of simple means to rollback (one or more) packages, to restore a predictable software behavior after another daily update breaks something.
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
On 12/27/20 6:27 PM, Alexander Bokovoy wrote:
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
Several people explained how bug reports like that tend end up without solution and even response while Red hat continues to do what they want... So I would not hold my breath that community is going to jump through hoops any more. Majority will just migrate to another clone, leaving developers to play "find the bug" with Red Hat.
Btw., we were warned that many Red Hat people who can decide about these kind of issues are on a holiday so those who want a debate about Stream issues are generally waiting for holidays to pass.
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 6:27 PM, Alexander Bokovoy wrote:
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
Several people explained how bug reports like that tend end up without solution and even response while Red hat continues to do what they want... So I would not hold my breath that community is going to jump through hoops any more. Majority will just migrate to another clone, leaving developers to play "find the bug" with Red Hat.
I am personally not happy with 'old' CentOS because I couldn't and still cannot influence even a tiny bit of what happens with CentOS after a RHEL release. Multiple times my components were broken in CentOS because they weren't built in the right order and weren't rebuilt for months even after users opened CentOS bugs, with zero transparency into what's happens.
I am one of those people who can fix the bugs raised against CentOS Stream. I would like to see them raised. My QE teams will certainly treat CentOS Stream bugs similar to RHEL bugs as that improves our ability to deliver RHEL releases with less issues. In the end, you are the ones who get the benefit once the fixes are out there.
At least in my area of interest other distributions aren't better than 'old' CentOS or RHEL, there is no magic wand that could help fix issues without them being reported and addressed by engineers involved in upstream projects. If your's only contribution is by raising bugs, please continue doing so, it helps, really.
Btw., we were warned that many Red Hat people who can decide about these kind of issues are on a holiday so those who want a debate about Stream issues are generally waiting for holidays to pass.
I am on a holiday myself, thus can actually spend some time engaging in these discussions. Mailing lists are great for distributed communications, both in geographic disperse and in time. Debates aren't bike-scheding. If there are bits that can be evolved and elaborated on, they don't fade with time simply because a discussion has happened over that time.
On Monday, December 28, 2020 2:01 AM, Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 6:27 PM, Alexander Bokovoy wrote:
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream. This is something worth raising as a feature request if it doesn't exist yet.
Several people explained how bug reports like that tend end up without solution and even response while Red hat continues to do what they want... So I would not hold my breath that community is going to jump through hoops any more. Majority will just migrate to another clone, leaving developers to play "find the bug" with Red Hat.
I am personally not happy with 'old' CentOS because I couldn't and still cannot influence even a tiny bit of what happens with CentOS after a RHEL release. Multiple times my components were broken in CentOS because they weren't built in the right order and weren't rebuilt for months even after users opened CentOS bugs, with zero transparency into what's happens.
That indicates bugs with Red Hat's SPEC files and assumptions made by Red Hat about the behavior of their own build system. The order in which packages are built should be reflected in the BuildRequires lines.
If Red Hat was using a standard COPR environment or pulicly documented it's build system then there should be no reason it shouldn't produce the same results as CentOS. Instead it seems RHEL developers leave things out of their SPEC files based on assumptions of the behavior of RHEL's own build system. When undocumented behavior of RHEL's build system produces different results, you shouldn't be surprised when the build order is different.
This change will not fix the SPEC files and doesn't get RHEL's build system publicly documented. Instead it just shuffles the dirt around in the name of "progress." We will get a fragmented CentOS community. You will get to be frustrated with Rocky Linux and CloudLinux Clone using vanilla COPR and reproducing RHEL's poorly written SPEC files.
I am one of those people who can fix the bugs raised against CentOS Stream. I would like to see them raised. My QE teams will certainly treat CentOS Stream bugs similar to RHEL bugs as that improves our ability to deliver RHEL releases with less issues. In the end, you are the ones who get the benefit once the fixes are out there.
At least in my area of interest other distributions aren't better than 'old' CentOS or RHEL, there is no magic wand that could help fix issues without them being reported and addressed by engineers involved in upstream projects. If your's only contribution is by raising bugs, please continue doing so, it helps, really.
What fix did we get for Bugzilla #1186352? Was the lack of Stream the reason for fixing the bug being refused? Now that Stream exists, will the bug be reopen and will we get the benefit of a real fix?
There are several other bugs I can find that was treated similarly by Red Hat.
I don't expect Red Hat to be able to solve every single bug. I also acknowledge there are several cases when openning a bug has resulted in fixes for everyone.
However, the assertion that terminating CentOS 8 before 2022 will help fix bugs does not pass the smell test.
What the CentOS Governance Board has done is put a chilling effect on adoption of CentOS 8. The EOL date of CentOS 7 is now a bonus over using CentOS 8. Why would anyone do that? The lack of transcript to the board meeting makes this move even more troubling. There still is no straight answer about why slowing adoption of CentOS 8 is a good thing for the community.
There is also no clear answer right now why fragmenting the CentOS community is good for the CentOS community.
But there is one "magic wand" that can help fix issues without a bug report which is to empower the community with the resources that should come with being the upstream provider. Right now the Stream kernel has four patches which are all for debranding. The 4.18 kernel tar.xz file in the SRPM is 110% the size of a vanilla tar.xz for 4.18. What is that additional 10%? How many patches are there? How many are security patches? How many are bug patches? How many are feature patches? How many patches were added between releases? How does a CentOS Stream contributor roll back individual patches?
If killing CentOS 8 before 2022 is part of an effort to close the openness gap and to better address bugs, then let's /*REALLY*/ do that with a kernel SRPM. It is time to make the patch information fully open to assist the community in fixing bugs.
But if the kernel SRPM can be left in the horrific state it currently is, then stop hitting us with all this empty posturing about the so-called positives we get out of killing CentOS 8 early.
On ma, 28 joulu 2020, redbaronbrowser via CentOS-devel wrote:
On Monday, December 28, 2020 2:01 AM, Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 6:27 PM, Alexander Bokovoy wrote:
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream. This is something worth raising as a feature request if it doesn't exist yet.
Several people explained how bug reports like that tend end up without solution and even response while Red hat continues to do what they want... So I would not hold my breath that community is going to jump through hoops any more. Majority will just migrate to another clone, leaving developers to play "find the bug" with Red Hat.
I am personally not happy with 'old' CentOS because I couldn't and still cannot influence even a tiny bit of what happens with CentOS after a RHEL release. Multiple times my components were broken in CentOS because they weren't built in the right order and weren't rebuilt for months even after users opened CentOS bugs, with zero transparency into what's happens.
That indicates bugs with Red Hat's SPEC files and assumptions made by Red Hat about the behavior of their own build system. The order in which packages are built should be reflected in the BuildRequires lines.
The particular issue I am alluding to was an order of rebuilds of two different components, samba and ipa. idm:DL1 module stream build happened before samba and used older internal samba ABI, making IPA server components to not load into a newer compiled smbd process. The order of builds was right in released RHEL version and wrong in CentOS 8.
Could it be forced by bumping BuildRequires in IPA spec file? Yes, it should have been. This was not deemed a bug high enough to stop RHEL release at a late stage since the problem was not in RHEL.
Could CentOS module stream be rebuilt when people reported about the issue? Most likely but there was no rebuild for months, even after I and others asked about this publicly and in personal communications. We offered help. The answer from CentOS maintainers was to wait until next CentOS release where things will fix themselves due to a rebuild of RHEL content.
With CentOS Stream it is hopefully will be easier to catch and fix in RHEL itself, by enforcing the right build requirements as a feedback coming from CentOS Stream users before we reach a stage in RHEL development where such bugs would be considered 'not important' to prevent release schedule from slipping.
I am one of those people who can fix the bugs raised against CentOS Stream. I would like to see them raised. My QE teams will certainly treat CentOS Stream bugs similar to RHEL bugs as that improves our ability to deliver RHEL releases with less issues. In the end, you are the ones who get the benefit once the fixes are out there.
At least in my area of interest other distributions aren't better than 'old' CentOS or RHEL, there is no magic wand that could help fix issues without them being reported and addressed by engineers involved in upstream projects. If your's only contribution is by raising bugs, please continue doing so, it helps, really.
What fix did we get for Bugzilla #1186352? Was the lack of Stream the reason for fixing the bug being refused? Now that Stream exists, will the bug be reopen and will we get the benefit of a real fix?
The bug you refer is on me and I had no time to work on that. Sadly, still have no time for it. Any help upstream is welcome. For complex components like LDAP server or its plugins we don't get many contributions.
There are several other bugs I can find that was treated similarly by Red Hat.
I don't expect Red Hat to be able to solve every single bug. I also acknowledge there are several cases when openning a bug has resulted in fixes for everyone.
However, the assertion that terminating CentOS 8 before 2022 will help fix bugs does not pass the smell test.
I don't see where that assertion can be attributed to me. I only compare models of contribution in the 'old' CentOS and CentOS Stream. Let's get it clear: CentOS Stream gives opportunity to contribute into RHEL development with your use cases and bug fixes well before that is late, right at the time when RHEL engineers have ability to influence the changes going into a particular release. Ideally, of course, but if timing is wrong for a specific release (say, a bug is reported after RHEL beta milestone when things become much harder to argue for), then such bug report, bug fix, or a feature request can at least be highlited for a next development cycle. With 'old' CentOS model you have none of these possibilities for the next development cycle because it is already late: at the time CentOS release is out, development window might already be shut down for the next RHEL release, especially for feature requests.
What the CentOS Governance Board has done is put a chilling effect on adoption of CentOS 8. The EOL date of CentOS 7 is now a bonus over using CentOS 8. Why would anyone do that? The lack of transcript to the board meeting makes this move even more troubling. There still is no straight answer about why slowing adoption of CentOS 8 is a good thing for the community.
There is also no clear answer right now why fragmenting the CentOS community is good for the CentOS community.
I would like to get answers to those questions as well.
But there is one "magic wand" that can help fix issues without a bug report which is to empower the community with the resources that should come with being the upstream provider. Right now the Stream kernel has four patches which are all for debranding. The 4.18 kernel tar.xz file in the SRPM is 110% the size of a vanilla tar.xz for 4.18. What is that additional 10%? How many patches are there? How many are security patches? How many are bug patches? How many are feature patches? How many patches were added between releases? How does a CentOS Stream contributor roll back individual patches?
If killing CentOS 8 before 2022 is part of an effort to close the openness gap and to better address bugs, then let's /*REALLY*/ do that with a kernel SRPM. It is time to make the patch information fully open to assist the community in fixing bugs.
But if the kernel SRPM can be left in the horrific state it currently is, then stop hitting us with all this empty posturing about the so-called positives we get out of killing CentOS 8 early.
These are good questions too. I have no experience working with RHEL kernel packaging myself, but looking at the changelogs, I wonder how many individual patches would be there. One thing I noticed in past is that using tens of thousands of patches in RPM spec file will amount to tens of thousands individual invocations of /usr/bin/patch command, each with its own cost. What is a cumulative overhead of that? Sadly, I was not able to find any performance investigations for GNU Patch. May be this can be done as a synthetic test using kernel changesets (5.10 development cycle got 16K of non-merge changesets, for example).
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, December 28, 2020 6:16 AM, Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, redbaronbrowser via CentOS-devel wrote:
But there is one "magic wand" that can help fix issues without a bug report which is to empower the community with the resources that should come with being the upstream provider. Right now the Stream kernel has four patches which are all for debranding. The 4.18 kernel tar.xz file in the SRPM is 110% the size of a vanilla tar.xz for 4.18. What is that additional 10%? How many patches are there? How many are security patches? How many are bug patches? How many are feature patches? How many patches were added between releases? How does a CentOS Stream contributor roll back individual patches? If killing CentOS 8 before 2022 is part of an effort to close the openness gap and to better address bugs, then let's /REALLY/ do that with a kernel SRPM. It is time to make the patch information fully open to assist the community in fixing bugs. But if the kernel SRPM can be left in the horrific state it currently is, then stop hitting us with all this empty posturing about the so-called positives we get out of killing CentOS 8 early.
These are good questions too. I have no experience working with RHEL kernel packaging myself, but looking at the changelogs, I wonder how many individual patches would be there. One thing I noticed in past is that using tens of thousands of patches in RPM spec file will amount to tens of thousands individual invocations of /usr/bin/patch command, each with its own cost. What is a cumulative overhead of that? Sadly, I was not able to find any performance investigations for GNU Patch. May be this can be done as a synthetic test using kernel changesets (5.10 development cycle got 16K of non-merge changesets, for example).
If you run "rpmbuild -bp" against a Stream kernel SRPM and a Fedora kernel SRPM, you can then see the difference made by running patch only 4 times vs 70 times.
In my experience, the prep section of building a kernel always amounts to less than 1% of the overall build time. There is no time savings by pre-applying the patches that justifies the lack of time savings.
If building time is a concern, then rpm-build should stop erasing the entire rpmroot directory and let make reuse object files based on source files which didn't change between releases.
On Mon, Dec 28, 2020 at 7:30 AM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, December 28, 2020 6:16 AM, Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, redbaronbrowser via CentOS-devel wrote:
But there is one "magic wand" that can help fix issues without a bug report which is to empower the community with the resources that should come with being the upstream provider. Right now the Stream kernel has four patches which are all for debranding. The 4.18 kernel tar.xz file in the SRPM is 110% the size of a vanilla tar.xz for 4.18. What is that additional 10%? How many patches are there? How many are security patches? How many are bug patches? How many are feature patches? How many patches were added between releases? How does a CentOS Stream contributor roll back individual patches? If killing CentOS 8 before 2022 is part of an effort to close the openness gap and to better address bugs, then let's /REALLY/ do that with a kernel SRPM. It is time to make the patch information fully open to assist the community in fixing bugs. But if the kernel SRPM can be left in the horrific state it currently is, then stop hitting us with all this empty posturing about the so-called positives we get out of killing CentOS 8 early.
These are good questions too. I have no experience working with RHEL kernel packaging myself, but looking at the changelogs, I wonder how many individual patches would be there. One thing I noticed in past is that using tens of thousands of patches in RPM spec file will amount to tens of thousands individual invocations of /usr/bin/patch command, each with its own cost. What is a cumulative overhead of that? Sadly, I was not able to find any performance investigations for GNU Patch. May be this can be done as a synthetic test using kernel changesets (5.10 development cycle got 16K of non-merge changesets, for example).
If you run "rpmbuild -bp" against a Stream kernel SRPM and a Fedora kernel SRPM, you can then see the difference made by running patch only 4 times vs 70 times.
In my experience, the prep section of building a kernel always amounts to less than 1% of the overall build time. There is no time savings by pre-applying the patches that justifies the lack of time savings.
If building time is a concern, then rpm-build should stop erasing the entire rpmroot directory and let make reuse object files based on source files which didn't change between releases.
At thousands of patches, it's quite slow. It's slightly faster when using the "%autopatch -S git_am" mode, but it's still very slow. The kernel package is likely to never use the standard Dist-Git model as all other packages (rightfully) do, and will use the "source-git" model in which you have a full Git tree with the packaging files and you can walk through Git commits like a developer would. I am hoping that will become available with RHEL/CentOS Stream 9 with the CKI/ARK stuff.
On ma, 28 joulu 2020, redbaronbrowser wrote:
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, December 28, 2020 6:16 AM, Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, redbaronbrowser via CentOS-devel wrote:
But there is one "magic wand" that can help fix issues without a bug report which is to empower the community with the resources that should come with being the upstream provider. Right now the Stream kernel has four patches which are all for debranding. The 4.18 kernel tar.xz file in the SRPM is 110% the size of a vanilla tar.xz for 4.18. What is that additional 10%? How many patches are there? How many are security patches? How many are bug patches? How many are feature patches? How many patches were added between releases? How does a CentOS Stream contributor roll back individual patches? If killing CentOS 8 before 2022 is part of an effort to close the openness gap and to better address bugs, then let's /REALLY/ do that with a kernel SRPM. It is time to make the patch information fully open to assist the community in fixing bugs. But if the kernel SRPM can be left in the horrific state it currently is, then stop hitting us with all this empty posturing about the so-called positives we get out of killing CentOS 8 early.
These are good questions too. I have no experience working with RHEL kernel packaging myself, but looking at the changelogs, I wonder how many individual patches would be there. One thing I noticed in past is that using tens of thousands of patches in RPM spec file will amount to tens of thousands individual invocations of /usr/bin/patch command, each with its own cost. What is a cumulative overhead of that? Sadly, I was not able to find any performance investigations for GNU Patch. May be this can be done as a synthetic test using kernel changesets (5.10 development cycle got 16K of non-merge changesets, for example).
If you run "rpmbuild -bp" against a Stream kernel SRPM and a Fedora kernel SRPM, you can then see the difference made by running patch only 4 times vs 70 times.
35 patches, as applied in https://kojipkgs.fedoraproject.org//packages/kernel/5.10.2/200.fc33/data/log...
$ curl -s https://kojipkgs.fedoraproject.org//packages/kernel/5.10.2/200.fc33/data/log... | grep Applying: | wc -l 35
I don't think that is representative.
In my experience, the prep section of building a kernel always amounts to less than 1% of the overall build time. There is no time savings by pre-applying the patches that justifies the lack of time savings.
If building time is a concern, then rpm-build should stop erasing the entire rpmroot directory and let make reuse object files based on source files which didn't change between releases.
That would be a major issue with reproducible builds and with security of the build environment.
Anyway, it is just my thought on what could be affecting an RPM build with 'too many' patches.
On 12/28/20 1:30 PM, redbaronbrowser via CentOS-devel wrote:
In my experience, the prep section of building a kernel always amounts to less than 1% of the overall build time. There is no time savings by pre-applying the patches that justifies the lack of time savings.
This started with RHEL 6, and the likely motivation was to hinder Oracle Enterprise Linux, not to save build time.[1] According to Wikipedia, Oracle even provides a service breaking down Red Hat's huge diff from upstream into individual patches, not sure how effective the Red Hat approach is, if the goal is indeed to stop OEL.[2, last paragraph]
[1] http://www.h-online.com/open/news/item/Controversy-surrounds-Red-Hat-s-obfus... [2] https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Rebuilds
On Mon, Dec 28, 2020 at 2:14 PM Laurențiu Păncescu < lpancescu@centosproject.org> wrote:
On 12/28/20 1:30 PM, redbaronbrowser via CentOS-devel wrote:
In my experience, the prep section of building a kernel always amounts
to less than 1% of the overall build time. There is no time savings by pre-applying the patches that justifies the lack of time savings.
This started with RHEL 6, and the likely motivation was to hinder Oracle Enterprise Linux, not to save build time.[1] According to Wikipedia, Oracle even provides a service breaking down Red Hat's huge diff from upstream into individual patches, not sure how effective the Red Hat approach is, if the goal is indeed to stop OEL.[2, last paragraph]
[1]
http://www.h-online.com/open/news/item/Controversy-surrounds-Red-Hat-s-obfus... [2] https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Rebuilds
Going to https://oss.oracle.com/git/gitweb.cgi?p=redpatch.git;a=summary it seems the latest commit is about 3 years ago though.... donna if repository or way of providing individual patches changed in the meantime
Gianluca
On Monday, December 28, 2020 7:14 AM, Laurențiu Păncescu lpancescu@centosproject.org wrote:
On 12/28/20 1:30 PM, redbaronbrowser via CentOS-devel wrote:
In my experience, the prep section of building a kernel always amounts to less than 1% of the overall build time. There is no time savings by pre-applying the patches that justifies the lack of time savings.
This started with RHEL 6, and the likely motivation was to hinder Oracle Enterprise Linux, not to save build time.[1] According to Wikipedia, Oracle even provides a service breaking down Red Hat's huge diff from upstream into individual patches, not sure how effective the Red Hat approach is, if the goal is indeed to stop OEL.[2, last paragraph]
[1] http://www.h-online.com/open/news/item/Controversy-surrounds-Red-Hat-s-obfus... [2] https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Rebuilds
I understand the history of the of obfuscation. But keep in mind that RHEL 6 goes back to 2010 long before CentOS joining Red Hat. We accepted some limitations of being a downstream clone. That having CentOS completely separate from Red Hat is going to have specific realities that come with it. One of those realities is the upstream sets the policy. That the openness gap created by those upstream policies were out of scope for CentOS.
Things are very different to have a governance meeting which had Chris Wright, Brain Exelbierd, Rich Bowen and Carl Trieloff. Are these considered to be active members of the CentOS community or mostly employees expected to make Red Hat's interests a priority?
With those type of people at the meeting, they should have been able to make deobfuscating the patches part of balancing CentOS' needs when being switched to being the upstream provider. Clearly balancing the needs of CentOS and closing the openness gap was not priorities of that meeting.
Despite that, Karsten Wade still felt it completely appropriate to post a blog post about this closing the openness gap.
The contradictions don't stop there.
We have had Red Hat employees asking us if we would be interested in taking part in an OpenStack SIG. Red Hat being interested in there being such a CentOS SIG would be consistent with them having put Rich Bowen on the governance board.
But then we have Mike McGrath explain Red Hat is not striving for CentOS to be enterprise grade and it is a mistake for anyone to think otherwise.
I have been involved in the OpenStack community but there is no way I want to be involved in an OpenStack SIG for CentOS now. There is no such thing as a personal-edition of OpenStack. There is a no point in having an OpenStack SIG for a CentOS which should never be mistaken for being enterprise grade.
None of this balancing the needs of CentOS.
We get the downsides of being an upstream and not the upsides of having an the openness gap fixed for the kernel SRPM.
We get the downside of being asked to put in the work for an OpenStack SIG but also told not to expect things to be enterprise grade.
The biopolar nature of the direction we should expect of Stream is just making my head spin.
On 12/28/20 3:30 PM, redbaronbrowser via CentOS-devel wrote:
I understand the history of the of obfuscation. But keep in mind that RHEL 6 goes back to 2010 long before CentOS joining Red Hat. We accepted some limitations of being a downstream clone. That having CentOS completely separate from Red Hat is going to have specific realities that come with it. One of those realities is the upstream sets the policy. That the openness gap created by those upstream policies were out of scope for CentOS.
I only wanted to put things in context with regards to the kernel SRPM, that it's done most likely to hinder Oracle (not very successfully, and OEL also seems to provide significant original R&D from Oracle, they're doing more than a simple recompile of Red Hat sources like CentOS). This never was a problem for CentOS Linux, as their goal was to be a 1:1 clone, "bug for bug and feature for feature", and I think it's not going to be a problem for the other clones, either. I remember articles from that time about Red Hat discussing this with the Free Software Foundation, who agreed it's the best way forward given Oracle's existential menace for Red Hat, probably the biggest contributor to the kernel back then.
I have no connection with Red Hat besides rare emails with people in the CentOS team, but I find this kind of discussion over specific members of the board bordering personal attack (over 900 mails, yet you are the only one focusing on that - this is my first and last answer to that). Red Hat acquired the CentOS brand and hired the CentOS core team. CentOS is their property, period. They could have decided directly at management level what to do, nobody forced them to even create a governance board. From Mike McGrath's description, it sounds they wanted to create community around CentOS similar to the one around Fedora, but things turned out different than expected and the association in people's minds, "CentOS is also from Red Hat", ended up cannibalizing their RHEL sales so they decided to put a stop to that. The Red Hat employees on the CentOS board didn't do what they wanted, but did what the company asked them to, I see no reason to focus on specific people. But the EOL should have been announced before 8.0, so people don't waste time migrating from 6 to 8.
If they wanted to kill all clones, they could have stopped publishing the sources (they only have to provide them to their own customers, like they do for EUS updates, not to everybody on the Internet). My impression is they want to kill just CentOS Linux, the only clone whose association with Red Hat seems to give it a legitimacy that other clones don't have.
I think the best way forward for everyone is to accept what happened and deal pragmatically with the consequences for their own organization.
[1] https://www.phoronix.com/scan.php?page=news_item&px=Oracle-Faster-Linux-... [2] https://blogs.oracle.com/linux/announcing-the-unbreakable-enterprise-kernel-...
On Mon, Dec 28, 2020 at 1:09 PM Laurențiu Păncescu lpancescu@centosproject.org wrote:
Red Hat acquired the CentOS brand and hired the CentOS core team. CentOS is their property, period. They could have decided directly at management level what to do, nobody forced them to even create a governance board. From Mike McGrath's description, it sounds they wanted to create community around CentOS similar to the one around Fedora, but things turned out different than expected and the association in people's minds, "CentOS is also from Red Hat", ended up cannibalizing their RHEL sales so they decided to put a stop to that. The Red Hat employees on the CentOS board didn't do what they wanted, but did what the company asked them to, I see no reason to focus on specific people. But the EOL should have been announced before 8.0, so people don't waste time migrating from 6 to 8.
I've been trying to separate cause and effect here, as I think it is somewhat important.
Everybody has their interests here. Every single person including myself and every contributor to this discussion has their own interests, and the people that made the decision in the first place. We don't need to get into what the interests are - it's only important to realize that they exist, and they are varied. This also makes it difficult to pick a particular claim like "profit!" and pin it on people, since many people who made the decision and who support the decision, do not directly or may not even indirectly profit from the decision.
I think the cause of this outcome is that the CentOS project was in a fragile place in 2014 and prior. The insiders can describe exactly how this was or wasn't so. Although the CentOS project could have expanded with further community outreach, an offer was put on the table that was difficult to refuse. Red Hat will sponsor the project, and streamline it. Without qualifying good or bad, it was recognized at the time that this was a conflict of interest situation but had great optics and would resolve immediate problems quickly and effectively. It was almost too good to be true. However, the trust in Red Hat as an honourable champion of open source, put these legitimate concerns aside.
Fast forward, and one by one - most, if not all of the voting members of the board were slowly replaced or influenced by people with Red Hat agendas. The emphasis on how to make things "better", and "contribute" was probably one of many chords that needed to play out successfully to convince people that this was in fact not only the right choice, but the inevitable and only choice. Who was representing the large community of "users" at this point? Which of these board members still had the original itch to scratch that created CentOS in the first place?
I think the future of the CentOS project was not only in a fragile place in 2014 and prior, but it was also in a fragile place in 2014 and after. It was mentioned by a few people here how there were "two months without security updates" as one of several reasons why CentOS was already "less than" RHEL, but it wasn't really acknowledged as to why - why can't CentOS be great? I don't believe Red Hat was incapable of making it great. I think the only valid explanation is that Red Hat didn't have an interest in making it great. Red Hat could have put more people on it, or streamlined the build process and guaranteed reproducibility by building RHEL and CentOS side-by-side if de-branding was still a requirement. Red Hat could have shutdown CentOS, and made RHEL available under the same terms as CentOS. But, they saw no reason to do so. they were content to leave it with insufficient resources, running behind RHEL by weeks to months, and having no security updates for two months. There might be certain heroes in the community who did all the work, and these people might be highly respected - but is it valid for sponsors to set up an organization that relies on heroic work to do something that is still "less than" RHEL?
In 2014 and prior, the community could have fixed the problem. In 2014 and after, the community was controlled by Red Hat. This wasn't absolutely clear in 2014, but it is now absolutely clear in 2020. Not only is CentOS being moved from "just downstream" to "just upstream" from the official RHEL releases, but CentOS is no longer a separate project at any arm's length from Red Hat. It's now integrated into their internal processes, and it is not really an "upstream" as most people would assume, but it is still downstream of the Red Hat controlled private branches, making it more like a preview and less like a method for community contribution.
CentOS Stream is much more like an "insider preview" as another person has been saying. "Insider preview" is valuable, but it's a very different thing from what CentOS was, and why people chose CentOS over Ubuntu. For example, I've been reflecting with some other users on how they wanted to go with Ubuntu, but chose CentOS *only* because it had 10 year life span, instead of 5 years. By dropping the 10 years down to 5 years, CentOS Stream is now much more similar to Ubuntu LTS, and Ubuntu is far more popular in developer communities.
And this comes to consequence. Even if everybody involved was trying to make the best decision possible in their own interests, there is a very real chance that their own interests will be the ones that are impacted the most. Red Hat took control of CentOS, presumedly to take control of one of the most popular downstream builds of RHEL. But, now CentOS will be re-created with one or more new names, and the 2014 acquisition of CentOS will have been effectively undone in every way except the name of the project. Red Hat will have abandoned control of the most popular downstream. Users will move to Ubuntu, Oracle Linux, Rocky, or one of several others, and this will include their non-financial contributions such as time and effort opening bug reports and getting problems fixed, and it will include downstream support contracts. Red Hat will have gained a reputation for failing to live up to commitments, and putting all future commitments in question. Other than Red Hat staff, and a few others who want a CentOS Stream (and don't need a CentOS), the great community will remember this. This isn't just individuals having uncomfortable conversations with their bosses - this will be bosses questioning why they should use Red Hat anywhere else. Even if CentOS Stream is successful in the areas it is intended to be successful, the damage will be done. I know my procurement team will be wary of any future Red Hat contracts - and this extends to all the other products beyond RHEL. There is no reason to believe an acceptable RHEL subscription will be made available that would substitute for CentOS usage, and if we must use something else for many of our machines, we may as well use something else for *all* of our machines.
From this perspective, I almost think this change was inevitable. Red
Hat was (intentionally?) not a good steward for CentOS. It was not run like a community, with community elected board members. Red Hat put just enough resources into CentOS to prevent anybody new from trying to re-create it, but not enough to allow it to compete directly with RHEL. It stayed like this essentially "on hold" for the last 6 years, being "just good enough, but not excellent". Now, a change has been made. CentOS as it was originally envisioned, is now deprecated and soon to be eliminated forever. CentOS Stream, which is the "insider preview" for RHEL is now to be released, as the new "just good enough, but not excellent, only this time it no longer effectively competes with RHEL, but instead helps harden RHEL."
If they wanted to kill all clones, they could have stopped publishing the sources (they only have to provide them to their own customers, like they do for EUS updates, not to everybody on the Internet). My impression is they want to kill just CentOS Linux, the only clone whose association with Red Hat seems to give it a legitimacy that other clones don't have.
Killing clones is not only difficult - but hopefully some people are aware that it would also be a problem. We are now on the precipice of "clones" now breaking from RHEL on definition of content. I foresee that if the exact package lineup is too hard to align with RHEL, that it will diverge. Oracle Linux 8.5 might not match RHEL 8.5, and this may be more of a problem for RHEL than for anybody else. "Clones" is more like "imitation is the most sincere form of flattery" and less like "others cannot possibly provide the same level of quality backports". The former is just a matter of convenience and an attempt to consolidate the entire EL community around common baselines so that packages that work on one system will also work with a great deal of confidence on another system. The latter is just a matter of scale. When it comes to the Linux kernel, Oracle has basically already broken from RHEL, in that they promote and install by default the Oracle UEK, and only offer the "Red Hat Compatible Kernel" as an option if necessary.
I think the best way forward for everyone is to accept what happened and deal pragmatically with the consequences for their own organization.
Yes.
I think it's important for a lot of the concerns to be captured in Internet history, so that we can learn from it. But, it seems we are far past the point of changing anything on the Red Hat side of things. As you and others have mentioned - the wording used suggests this is all final. This is why I will not be sending anything to the new mailing list. I already sent plenty of explanation to Red Hat in 2016 and 2017, and they took no action then.
On Monday, December 28, 2020 11:17 PM, Mark Mielke mark.mielke@gmail.com wrote:
On Mon, Dec 28, 2020 at 1:09 PM Laurențiu Păncescu lpancescu@centosproject.org wrote:
I think the best way forward for everyone is to accept what happened and deal pragmatically with the consequences for their own organization.
Yes.
I think it's important for a lot of the concerns to be captured in Internet history, so that we can learn from it.
We tried that. That is what resulted in the apology for Red Hat Psyche and the promise that what is happenning now will never happen. It took 2 years for things to fully settle out.
But, it seems we are far past the point of changing anything on the Red Hat side of things. As you and others have mentioned - the wording used suggests this is all final.
Then they should have announced the full elephant instead of talking about pointing flash lights at it. It seems like even Red Hat is not sure what the full elephant is yet because they expected everyone to silently give them whatever they wanted.
If we really are to the point that the best Red Hat has to offer in "balancing" the needs of the community is as things stand, there is no point in there being a CentOS community. We have Debian, we have OpenSUSE, we have plenty of other communities that show greater respect than this.
We have been given a governance board that is packed with Red Hat employees that are *NOT* members of the CentOS community. That is not a balancing of needs.
We have been told to expect 5 year life cycles for Stream but for CentOS 8 we get only 2 years. That is not a balancing of needs. If we are going from 10 years to 5 years then continue with CentOS 8 til 2024. If we accept only 2 years for CentOS 8 then we don't deserve to expect 5 years from Stream either. There is no balancing of needs in this action by Red Hat.
We have been told Red Hat is serious about closing the openness gap so the community can contribute as an upstream. But then still leave the kernel SRPM of Stream in an obfuscated state. This hinders the ability of the community to take on the roles of an upstream. The community can't meaningfully contribute to kernel patch troubleshooting and application of patches via kpatch when things remain like this. Again, there is no balancing of the needs in this action by Red Hat.
Accepting this CATHEDRAL move is accepting Red Hat is unredeemable when it comes to working with the community. It is a complete rejection of the BIZZAR to have meaningful SIGs and other contributions.
According to O'Rielly, the subject that an elephant best fits with is Hadoop. It is fitting for describing what can be accomplished by having open and fair communication can have with a community.
What Red Hat is doing so far isn't about showing us an elephant and treating us like Hadoop. It is more like showing us a turtle with an trunk attached and treating us like powershell.
They simply can not get the results they want for the next two years by repeating the mistakes of Red Hat Psyche. As fans of Red Hat and CentOS, we should feel the need to let them know that.
On 12/28/20 9:30 AM, redbaronbrowser via CentOS-devel wrote:
Things are very different to have a governance meeting which had Chris Wright, Brain Exelbierd, Rich Bowen and Carl Trieloff. Are these considered to be active members of the CentOS community or mostly employees expected to make Red Hat's interests a priority?
With those type of people at the meeting, they should have been able to make deobfuscating the patches part of balancing CentOS' needs when being switched to being the upstream provider. Clearly balancing the needs of CentOS and closing the openness gap was not priorities of that meeting.
Despite that, Karsten Wade still felt it completely appropriate to post a blog post about this closing the openness gap.
The contradictions don't stop there.
We have had Red Hat employees asking us if we would be interested in taking part in an OpenStack SIG. Red Hat being interested in there being such a CentOS SIG would be consistent with them having put Rich Bowen on the governance board.
To be completely clear, I am not on the governing board. My presence at these meetings, as community manager (a position I have held for 3 years) was *specifically* to increase openness of the governing process.
My role there is to say things like "shouldn't this conversation be happening on the public mailing list" and "when can we expect the minutes to be published" and "please address the open issues in the governance ticket tracker."
https://www.centos.org/about/governance/ says:
Also in regular attendance:
Rich Bowen (Community Architect, attendee ex-officio)
Which is to say: not a member, doesn't have a vote.
But, yeah, I do consider myself an active member of the CentOS community. I've been running CentOS Dojos around the world for 3 years. I write our newsletters. I manage our social media presence (along with other awesome people like Ljubomir). No, I'm not writing code or rebuilding packages, but I do consider these to be important contributions.
That said, the focus of the last several meetings has indeed been closing the openness gap - this was even before the recent announcement.
This has led to more regular publishing of minutes
https://blog.centos.org/category/board-minutes/ - yes, there are some missing, but I believe that we now have a full set of minutes for the entirety of Thomas' term as Secretar.
It has also led to the ongoing work on governance documentation, and in the agreement, by the board, to open the board meetings to the community. (Yes, these decisions are minuted.)
It is, of course, understandable that you're unaware of these decisions, since the decision itself was to address that lack of openness.
Carl has been a member of the board since 2014. Brian Exelbierd has been actively involved in Fedora for years. His addition to the CentOS board was, indeed, recent, and he is the Red Hat Liaison, as documented on the CentOS wiki - his addition to the board was in line with the Liaison definition described there.
Chris Wright was there as a representative of Red Hat, who, as you are no doubt aware, has a controlling interest in the CentOS project, and pays the bills, and has done so since 2014. It's not weird that he was there. If anything, it was a mark of respect that the CTO was sent to the meeting, rather than someone lower down the chain. I am unclear what your objection is to his presence at a board meeting.
I, for one, look forward to broader community presence at Board meetings. However, I think most people will be dismayed at how boring Board meetings actually are. The notion that it's some kind of smoky back room where The Important Decisions get made is, sadly, not true.
On 12/28/20 5:14 AM, Laurențiu Păncescu wrote:
This started with RHEL 6, and the likely motivation was to hinder Oracle Enterprise Linux, not to save build time.
That was speculation at the time of the change, and no evidence has ever supported it. Most specifically: It does save build time, and it hasn't hindered Oracle EL. There was never any reason to think that it would. Even if the OEL kernel maintainers wanted specifically to start with the RHEL kernel and add their own features, it's no more difficult to do so with a tarball than it is with a tarball and thousands of patch files. Either way, they will need to apply their patches to the end result and rebase periodically on that.
It was a poorly supported argument in 2011, and it's a poorly supported argument today.
On 12/28/20 8:04 PM, Gordon Messmer wrote:
On 12/28/20 5:14 AM, Laurențiu Păncescu wrote:
This started with RHEL 6, and the likely motivation was to hinder Oracle Enterprise Linux, not to save build time.
That was speculation at the time of the change, and no evidence has ever supported it.
"No evidence" besides the official declaration published on the Red Hat Blog by Brian Stevens, Red Hat CTO at the time:
Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.
Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.
On Monday, December 28, 2020 1:41 PM, Laurențiu Păncescu lpancescu@centosproject.org wrote:
On 12/28/20 8:04 PM, Gordon Messmer wrote:
On 12/28/20 5:14 AM, Laurențiu Păncescu wrote:
This started with RHEL 6, and the likely motivation was to hinder Oracle Enterprise Linux, not to save build time.
That was speculation at the time of the change, and no evidence has ever supported it.
"No evidence" besides the official declaration published on the Red Hat Blog by Brian Stevens, Red Hat CTO at the time:
Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.
Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.
Attempting this might have made business sense back in 2011. But what it was supposed to accomplished has failed.
There was justifications given for leaving in place, but those justifications involved RHEL being the upstream provider.
We now have from Karsten Wade: "Essentially, Red Hat is filling the development and contribution gap that exists between Fedora and RHEL by shifting the place of CentOS from just downstream of RHEL to just upstream of RHEL."
Leaving the obfuscated kernel SRPM in place suggests the governance board has tagged us upstream without giving us the authority that should come with it. We are being put through the exercise of this "hard decision" so that Wade can apply his own twisted definition of "openness."
And where exactly is this Wade that would be available on the centos devel mailing list to discuss things further?
He has posted a total of four times to the mailing list. His last post was a week ago with this fun little metaphor:
"Let's bring the elephant out of the room into the sunlight, I'm getting tired of building better flashlights."
The CentOS governance had a stated obligation to be public, open and inclusive. The fact a board member is even describing the situation in terms of flash lights on an elephant is an indication they have not lived up to that obligation at all.
Where is his follow-up post to bring the elephant into the sunlight?
Where is his stance on closing the openness gap by leaving the Stream kernel SRPM obfuscated for patches?
I further submit that the following governance board member have not posted to any CentOS mailing list over the entire year of 2020. Under the concept of meritocracy, they do not have merit to be on the board as they aren't active members of the CentOS community.
These MIA governance board members are: Brian Exelbierd Mike McLean Carl Trieloff Rich Bowen
How do we initiate a vote of no confidence in these non-community board members?
On 2020/12/29 8:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them.
Yes, that's a problem for me. I mentioned it at https://lists.centos.org/pipermail/centos-devel/2020-December/075631.html . I have an LSM module named TOMOYO which is in-kernel since Linux 2.6.30 . Since Fedora cannot afford enabling LSM modules other than SELinux ( https://bugzilla.redhat.com/show_bug.cgi?id=542986 ), unlike other Linux distributions, TOMOYO is enabled in CentOS Plus kernels, which is difficult for RHEL users because CentOS Plus kernels are completely unsupported by RH.
ELrepo and others have provided support at the"point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
Any chance that RH moves from "RH is responsible for supporting all code RH is shipping" to "RH ships as much code as possible (basically any GPL code), but RH supports only some portion of shipped code" ?
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Who are the intended audience of RHEL/CentOS Linux/CentOS Stream ?
While some people mention absence of security fixes in CentOS Linux upon RHEL minor release, it is common that RHEL servers with uptime of over 1 year (i.e. no kernel updates). There are servers using kernels as of e.g. RHEL 7.3 or so. That is, while RHEL is providing security fixes quickly, not all users are applying security fixes so quickly.
Since I'm a Linux kernel developer, I don't know about trends of userspace. But let me try to think about characteristics of several distributions.
Gentoo is targeting for providing newest possible versions. But since Gentoo is a distribution which asks users to "compile", Gentoo is difficult for administrators who are not developers.
Ubuntu is targeting for easy to use, with reasonably newest versions. Since the design of Ubuntu is fundamentally different (e.g. setting root password is not mandatory, multiple Linux Security Modules are available compared to SELinux-only), CentOS Stream won't be able to behave like Ubuntu due to constraint between Fedora and RHEL.
Default gcc provided in CentOS 7 became too old to compile Linux kernels. Many projects which follow the trend want latest version of compilers. Wouldn't developers who want latest versions already using Fedora/Ubuntu ?
After all, isn't RHEL/CentOS a distribution for providing reasonably oldest versions, with plenty of documents and knowledge prepared for circumspect users?
The idea of moving CentOS Linux to CentOS Stream might be just a "The grass is always greener on the other side of fence." thing...
On 12/29/20 5:16 AM, Tetsuo Handa wrote:
Ubuntu is targeting for easy to use, with reasonably newest versions. Since the design of Ubuntu is fundamentally different (e.g. setting root password is not mandatory, multiple Linux Security Modules are available compared to SELinux-only), CentOS Stream won't be able to behave like Ubuntu due to constraint between Fedora and RHEL.
I think TOMOYO support in Ubuntu is inherited from Debian, which was proud to be the only distribution supporting all LSM frameworks. Ubuntu, and Debian since Buster, default to AppArmor. That probably makes it more likely to see the greatest amount of testing in individual packages. TOMOYO's learning mode was pretty cool, I played with it years ago in Debian.
Default gcc provided in CentOS 7 became too old to compile Linux kernels. Many projects which follow the trend want latest version of compilers. Wouldn't developers who want latest versions already using Fedora/Ubuntu ?
Red Hat provides newer versions of compilers for RHEL and CentOS, as well as interpreters like Ruby and Python, in Software Collections Library (at least part of that moved to AppStream in RHEL 8 - I thought compiler will stay in SCL because users might want to have several versions installed in parallel, but I don't find any sclo directory in the CentOS 8 repos).
I think they support each collection for 18 months, and they even guarantee the binary compatibility between new compiler like gcc-9.3.1 in devtoolset-9 and the system compiler (gcc-4.8 in RHEL7), so it's possible to link object files and shared libraries compiled with different versions of gcc. The softwarecollections.org site sadly seems no longer updated, but you can browse available packages in CentOS repos:
http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/d/
On 2020/12/29 8:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them.
Yes, that's a problem for me. I mentioned it at https://lists.centos.org/pipermail/centos-devel/2020-December/075631.html . I have an LSM module named TOMOYO which is in-kernel since Linux 2.6.30 . Since Fedora cannot afford enabling LSM modules other than SELinux ( https://bugzilla.redhat.com/show_bug.cgi?id=542986 ), unlike other Linux distributions, TOMOYO is enabled in CentOS Plus kernels, which is difficult for RHEL users because CentOS Plus kernels are completely unsupported by RH.
ELrepo and others have provided supportat the "point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
Any chance that RH moves from "RH is responsible for supporting all code RH is shipping" to "RH ships as much code as possible (basically any GPL code), but RH supports only some portion of shipped code" ?
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Who are the intended audience of RHEL/CentOS Linux/CentOS Stream ?
While some people mention absence of security fixes in CentOS Linux upon RHEL minor release, it is common that RHEL servers with uptime of over 1 year (i.e. no kernel updates). There are servers using kernels as of e.g. RHEL 7.3 or so. That is, while RHEL is providing security fixes quickly, not all users are applying security fixes so quickly.
Since I'm a Linux kernel developer, I don't know about trends of userspace. But let me try to think about characteristics of several distributions.
Gentoo is targeting for providing newest possible versions. But since Gentoo is a distribution which asks users to "compile", Gentoo is difficult for administrators who are not developers.
Ubuntu is targeting for easy to use, with reasonably newest versions. Since the design of Ubuntu is fundamentally different (e.g. setting root password is not mandatory, multiple Linux Security Modules are available compared to SELinux-only), CentOS Stream won't be able to behave like Ubuntu due to constraint between Fedora and RHEL.
Default gcc provided in CentOS 7 became too old to compile Linux kernels. Many projects which follow the trend want latest version of compilers. Wouldn't developers who want latest versions already using Fedora/Ubuntu ?
After all, isn't RHEL/CentOS a distribution for providing reasonably oldest versions, with plenty of documents and knowledge prepared for circumspect users?
The idea of moving CentOS Linux to CentOS Stream might be just a "The grass is always greener on the other side of fence." thing...
For me the characteristics of RedHat EL/CentOS have always been: * It's stable, and stable for 10 years minus the first ~1-2. * It's old and outdated, nothing to make developers happy. * It provides a quite limited package set with high stability and quality. A lot of interesting stuff (things like Tomcat) have to be installed from elsewhere without stability and high quality or easy management. * It has a lot of competitors but the long support is unique. * As an admin, if you have a lot developers around you, you ALWAYS have to defend the usage of RHEL/CentOS because ALMOST EVERY developer would like to use something else.
Now for CentOS, reduce the long support to 5 years and slightly reduce on the overall stability. What do you get? How do you sell it to your customers/users who wanted something else anyway? How do you defend your decision for RHEL/CentOS? Difficult times for all of us in this situation!
Simon
On 12/29/20 8:33 AM, Simon Matter wrote:
For me the characteristics of RedHat EL/CentOS have always been:
- It's stable, and stable for 10 years minus the first ~1-2.
- It's old and outdated, nothing to make developers happy.
- It provides a quite limited package set with high stability and quality.
A lot of interesting stuff (things like Tomcat) have to be installed from elsewhere without stability and high quality or easy management.
- It has a lot of competitors but the long support is unique.
- As an admin, if you have a lot developers around you, you ALWAYS have to
defend the usage of RHEL/CentOS because ALMOST EVERY developer would like to use something else.
For me it makes more sense to pick a distro based on application needs, it saves a lot of trouble if what you need isn't directly supported by RHEL base repos:
* if your organization or customers mandate RHEL, or you need third-party hardware drivers or expensive applications (CAD, medical imaging, FEA, physics simulations) only certified on RHEL, then RHEL it is * if you're developing upstream software like the kernel, or for statically-linked microservices or utilities in Go, Fedora has the latest upstream releases and is very stable considering how fast it changes * if you develop application with lots of dependencies, like Java enterprise apps or apps in Rails or Django, the huge selection of packages in Debian or Ubuntu LTS will provide you stable versions with security updates handled by the Debian or Canonical security teams, so your application keeps running without fighting Maven or having to choose between a security vulnerability and a newer version in a third-party repo breaking your application (Debian almost never breaks something with security updates or point releases, because they don't backport new features to old versions, proprietary hardware drivers are not an issue with Ubuntu or Debian if running on AWS, and HP and Dell offer physical servers with Ubuntu LTS support[0]) * if you need DTrace, ZFS or the best network throughput, FreeBSD can do that very well (but FreeBSD Ports is similar to Gentoo, no stable versions that only get security updates)
Backpack, formerly 37signals, wrote in 2006 that they run RHEL 6 on the database servers while their application servers use Ubuntu LTS, so I guess I'm not the only one looking at this pragmatically.[1]
I only used Debian and CentOS until now, but I think Ubuntu got so popular because they hit the sweet spot: 5 years of support and you get the same binaries and timely security updates even if you don't pay anything (10 years of support with Ubuntu Advantage, paid), big selection of packages, reasonably new versions of packages, third-party driver support from some OEMs, PPAs from Nginx and others, many developers you would hire are already familiar with it from home (I only used Debian and CentOS, not Ubuntu, though).
[0] https://linux.dell.com/files/supportmatrix/Ubuntu_Support_Matrix.pdf [1] https://signalvnoise.com/posts/3202-behind-the-scenes-the-hardware-that-powe...
On 12/29/20 8:33 AM, Simon Matter wrote:
For me the characteristics of RedHat EL/CentOS have always been:
- It's stable, and stable for 10 years minus the first ~1-2.
- It's old and outdated, nothing to make developers happy.
- It provides a quite limited package set with high stability and
quality. A lot of interesting stuff (things like Tomcat) have to be installed from elsewhere without stability and high quality or easy management.
- It has a lot of competitors but the long support is unique.
- As an admin, if you have a lot developers around you, you ALWAYS have
to defend the usage of RHEL/CentOS because ALMOST EVERY developer would like to use something else.
For me it makes more sense to pick a distro based on application needs, it saves a lot of trouble if what you need isn't directly supported by RHEL base repos:
- if your organization or customers mandate RHEL, or you need
third-party hardware drivers or expensive applications (CAD, medical imaging, FEA, physics simulations) only certified on RHEL, then RHEL it is
- if you're developing upstream software like the kernel, or for
statically-linked microservices or utilities in Go, Fedora has the latest upstream releases and is very stable considering how fast it changes
- if you develop application with lots of dependencies, like Java
enterprise apps or apps in Rails or Django, the huge selection of packages in Debian or Ubuntu LTS will provide you stable versions with security updates handled by the Debian or Canonical security teams, so your application keeps running without fighting Maven or having to choose between a security vulnerability and a newer version in a third-party repo breaking your application (Debian almost never breaks something with security updates or point releases, because they don't backport new features to old versions, proprietary hardware drivers are not an issue with Ubuntu or Debian if running on AWS, and HP and Dell offer physical servers with Ubuntu LTS support[0])
- if you need DTrace, ZFS or the best network throughput, FreeBSD can do
that very well (but FreeBSD Ports is similar to Gentoo, no stable versions that only get security updates)
Backpack, formerly 37signals, wrote in 2006 that they run RHEL 6 on the database servers while their application servers use Ubuntu LTS, so I guess I'm not the only one looking at this pragmatically.[1]
While I agree to what you wrote, there are also reasons to stay with a choice of distribution you made. It's to protect knowledge, to not double the work with internally used special software, prevent from doing wrong decisions with far reaching consequences and so on. It sometimes need a trigger to rethink hard. In your list only the first point leads to RHEL family, and this point doesn't apply to our current environment. While it was always clear that we _may_ change distribution in future, no concrete steps have been done and nothing has really forced us to do something.
But now, things have changed, with this _huge_ trigger which was launched by Red Hat some days ago.
Only time will tell what comes out of this story. My feeling is that support for choosing RHEL will become more difficult in future.
Simon
On 12/29/20 10:42 AM, Simon Matter wrote:
While I agree to what you wrote, there are also reasons to stay with a choice of distribution you made. It's to protect knowledge, to not double the work with internally used special software, prevent from doing wrong decisions with far reaching consequences and so on. It sometimes need a trigger to rethink hard. In your list only the first point leads to RHEL family, and this point doesn't apply to our current environment. While it was always clear that we _may_ change distribution in future, no concrete steps have been done and nothing has really forced us to do something.
Sure, it wasn't meant to be an exhaustive list. Some companies already have processes and extensive tooling around RHEL, from deployment to packaging big software suites in RPM repos for customers (CentOS isn't even allowed, every piece of equipment has to be supported by the designated vendor).
I think RHEL makes a lot of sense for traditional enterprises, and I hope Red Hat prospers - I don't even want to think what the Linux ecosystem would look like if they disappeared: every distro benefits from the huge contributions made by Red Hat upstream, from the kernel and gcc to Gnome and Ansible, not to speak of Fedora and the excellent documentation written by Red Hat employees. Even if you use libvirt on Debian, you'll end up consulting Red Hat documentation and use upstream features and fixes made by Red Hat. I hope they're here to stay.
On Tue, Dec 29, 2020 at 2:33 AM Simon Matter simon.matter@invoca.ch wrote:
For me the characteristics of RedHat EL/CentOS have always been:
- It's stable, and stable for 10 years minus the first ~1-2.
- It's old and outdated, nothing to make developers happy.
- It provides a quite limited package set with high stability and quality.
A lot of interesting stuff (things like Tomcat) have to be installed from elsewhere without stability and high quality or easy management.
- It has a lot of competitors but the long support is unique.
- As an admin, if you have a lot developers around you, you ALWAYS have to
defend the usage of RHEL/CentOS because ALMOST EVERY developer would like to use something else.
Now for CentOS, reduce the long support to 5 years and slightly reduce on the overall stability. What do you get? How do you sell it to your customers/users who wanted something else anyway? How do you defend your decision for RHEL/CentOS? Difficult times for all of us in this situation!
I expect we'll see a lot of people not even bothering with RHEL 8 or CentOs 8. I also expect Red Hat to review and discard this approach by the time RHEL 9 rolls around: that is very much what happened when they tried similar with Red Hat 9 back in 2003. The resistance to point releases is understandable, especially to people trained on "continuous integration, continuous development". But many businesses and developers find continuous release unsafe and destabilizing, generating constant uncertainty about their environment.
EPEL is an example of the problem. Many critical system components, such as python modules, nagios, and until very recently ansible, were available in EPEL as leading edge components only, without easy access to the previous releases. It's maddening. If you have other components that rely on stable bits, well, it's been awkward.
On 12/29/20 12:16 PM, Nico Kadel-Garcia wrote:
[...] The resistance to point releases is understandable, especially to people trained on "continuous integration, continuous development". But many businesses and developers find continuous release unsafe and destabilizing, generating constant uncertainty about their environment.
Sure they do, because it's over-hyped and it doesn't work in practice - CI is necessary, but not sufficient. Big car manufacturers here really follow best practices, every commit having to pass CI/CD to be integrated (thousands of tests in R&D), there are additional test departments, years of manual testing and many thousands of km of test drives made by human test drivers, and cars still have plenty of bugs on release. Errors are then corrected in production, there is also at least a facelift that corrects additional bugs, and when the car is discontinued, there are still bugs, from the rear camera occasionally getting confused (displaying the warning messages, but an unfocused or black image - turning the motor off and waiting 30s fixes that), the smart key only opening the trunk, but not the doors after 3-4 days (open all door windows completely to reset the smart key system), to the car suddenly engaging emergency braking while driving, seeing an obstacle or person where there isn't any. The latter is really life-threatening on the Autobahn, and I found out a smaller model from the same manufacturer is also affected by that, so it's unlikely to be a sensor malfunction on just one car.
I'm somewhat skeptical the 4 hours of automated CI that someone from Red Hat mentioned are going to be enough, without extensive human testing - but Chris Wright wrote that Stream is not suitable for production use like RHEL is.
EPEL is an example of the problem. Many critical system components, such as python modules, nagios, and until very recently ansible, were available in EPEL as leading edge components only, without easy access to the previous releases. It's maddening. If you have other components that rely on stable bits, well, it's been awkward.
I had a case of an updated dependency of a dependency that relied implicitly on the slightly changed semantics of asyncio in Python 3.3.5, so it didn't really work correctly on distro-provided 3.3.1, without any errors or warnings (the method existed, but behaved a bit differently), sometimes "losing" requests. When you have tens or thousands of dependencies, constantly updated from EPEL or PyPI, such subtle bugs will slip through. If there is any LTS distro maintaining stable packages, only fixing bugs, I think it's much safer to use that instead of trying to update everything yourself.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, December 29, 2020 7:32 AM, Laurențiu Păncescu lpancescu@centosproject.org wrote:
On 12/29/20 12:16 PM, Nico Kadel-Garcia wrote:
[...] The resistance to point releases is understandable, especially to people trained on "continuous integration, continuous development". But many businesses and developers find continuous release unsafe and destabilizing, generating constant uncertainty about their environment.
Sure they do, because it's over-hyped and it doesn't work in practice - CI is necessary, but not sufficient. Big car manufacturers here really follow best practices, every commit having to pass CI/CD to be integrated (thousands of tests in R&D), there are additional test departments, years of manual testing and many thousands of km of test drives made by human test drivers, and cars still have plenty of bugs on release. Errors are then corrected in production, there is also at least a facelift that corrects additional bugs, and when the car is discontinued, there are still bugs, from the rear camera occasionally getting confused (displaying the warning messages, but an unfocused or black image - turning the motor off and waiting 30s fixes that), the smart key only opening the trunk, but not the doors after 3-4 days (open all door windows completely to reset the smart key system), to the car suddenly engaging emergency braking while driving, seeing an obstacle or person where there isn't any. The latter is really life-threatening on the Autobahn, and I found out a smaller model from the same manufacturer is also affected by that, so it's unlikely to be a sensor malfunction on just one car.
I'm somewhat skeptical the 4 hours of automated CI that someone from Red Hat mentioned are going to be enough, without extensive human testing - but Chris Wright wrote that Stream is not suitable for production use like RHEL is.
I also have my doubts about how much CI will be ready enough before 2022.
EPEL is an example of the problem. Many critical system components, such as python modules, nagios, and until very recently ansible, were available in EPEL as leading edge components only, without easy access to the previous releases. It's maddening. If you have other components that rely on stable bits, well, it's been awkward.
I had a case of an updated dependency of a dependency that relied implicitly on the slightly changed semantics of asyncio in Python 3.3.5, so it didn't really work correctly on distro-provided 3.3.1, without any errors or warnings (the method existed, but behaved a bit differently), sometimes "losing" requests. When you have tens or thousands of dependencies, constantly updated from EPEL or PyPI, such subtle bugs will slip through. If there is any LTS distro maintaining stable packages, only fixing bugs, I think it's much safer to use that instead of trying to update everything yourself.
dnf will allows enabling COPR builds for updates. Using either Docker or KVM might be the best way to deal with fragle applications or environment requirements. Just take a snapshot before running updates and revert as needed.
On 12/29/20 6:50 PM, redbaronbrowser via CentOS-devel wrote:
dnf will allows enabling COPR builds for updates. Using either Docker or KVM might be the best way to deal with fragle applications or environment requirements. Just take a snapshot before running updates and revert as needed.
The problem is that such upstream bugs appear and disappear, but subtle effects might not be detected for weeks or months. There's a lot of value in stabilizing on a specific release and only testing and fixing bugs until you are reasonably sure that everything works right together, like Red Hat and other LTS distro vendors do. An HFT algorithm losing 2% of transactions for several months would be a major issue, and worse things can happen in critical applications in medical, military or heavy industry - it's difficult enough to get software right without constantly changing dependencies. A Docker container pulling dependencies as "whatever was on GitHub last Tuesday" isn't good enough for such applications, although for web applications it might be fine (plenty of people do just that).
On Wednesday, December 30, 2020 3:23 AM, Laurențiu Păncescu lpancescu@centosproject.org wrote:
An HFT algorithm losing 2% of transactions for several months would be a major issue, and worse things can happen in critical applications in medical, military or heavy industry - it's difficult enough to get software right without constantly changing dependencies
Wait. Step back for a moment.
Are we trying to claim Stream is what would cross the line for not complying with IEC 62304? Because as far as I can tell CentOS 7/8 and RHEL 7/8 never claimed to be compliant with IEC 62034.
Did the medical industry ever really learn it's lesson from the Therac-25? [1]
Or are they still horrifically cutting corners on basic concepts? [2]
[1] https://en.wikipedia.org/wiki/Therac-25 [2] https://hackaday.com/2018/01/22/seek-and-exploit-security-vulnerabilities-in...
I would like to visit how we can use Linux in mission critical settings but I think it is outside the scope of the impact killing CentOS 8 at the end of 2021 has.
On 12/30/20 1:23 AM, Laurențiu Păncescu wrote:
An HFT algorithm losing 2% of transactions for several months would be a major issue, and worse things can happen in critical applications in medical, military or heavy industry
The existence of applications that require a static ABI for extended periods of time is not an argument for the existence of CentOS, because those applications also require *support*. That's something that RHEL provides.
Why would anyone recommend that medical, military, or heavy industry use a self-supported OS?
On 28/12/2020 23:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them. ELrepo and others have provided support at the "point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Hi Lamar,
Unfortunately there's an issue with driver disk ISO images too, as they are based on point release GA kernels (as are most all OEM ISO driver disks), and Stream regularly releases new installation media based on the latest Stream kernel, so if you need a 3rd party driver disk to install, you are probably out of luck and may not even be able to install CentOS Stream, even if a driver were available. I'm guessing for the next year you could install CentOS Linux 8 and convert to Stream, but after 2021 that will no longer be an option.
On Tue, Dec 29, 2020 at 7:04 AM Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 23:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them. ELrepo and others have provided support at the "point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Hi Lamar,
Unfortunately there's an issue with driver disk ISO images too, as they are based on point release GA kernels (as are most all OEM ISO driver disks), and Stream regularly releases new installation media based on the latest Stream kernel, so if you need a 3rd party driver disk to install, you are probably out of luck and may not even be able to install CentOS Stream, even if a driver were available. I'm guessing for the next year you could install CentOS Linux 8 and convert to Stream, but after 2021 that will no longer be an option.
He should also consider finding a more reputable hardware vendor. Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
Why wouldn't installing from CentOS 8.3 and updating to CentOS Stream be an option? The yum repos are well defined, in well-defined RPMs. And an add-on internal RPM to point to a locked internal repository to provide a "base OS imae" snapshot is a commonplace practice for EPEL, one I've used effectively myself to lock the "update" channels from RHEL and CentOS for consistency in cluster deployment. The approach costs time and effort to set up, and some internal resources, but it provides much faster "yum update" or "yum check-update" operations than always reaching out to the central repos. It's also very useful when you need to rebuild or update 200 cluster members simultaneously, and the yum metadata alone will overwhelm your external bandwidth.
On 12/30/20 9:47 AM, Nico Kadel-Garcia wrote:
On Tue, Dec 29, 2020 at 7:04 AM Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 23:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them. ELrepo and others have provided support at the "point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Hi Lamar,
Unfortunately there's an issue with driver disk ISO images too, as they are based on point release GA kernels (as are most all OEM ISO driver disks), and Stream regularly releases new installation media based on the latest Stream kernel, so if you need a 3rd party driver disk to install, you are probably out of luck and may not even be able to install CentOS Stream, even if a driver were available. I'm guessing for the next year you could install CentOS Linux 8 and convert to Stream, but after 2021 that will no longer be an option.
He should also consider finding a more reputable hardware vendor. Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
Why wouldn't installing from CentOS 8.3 and updating to CentOS Stream be an option? The yum repos are well defined, in well-defined RPMs. And an add-on internal RPM to point to a locked internal repository to provide a "base OS imae" snapshot is a commonplace practice for EPEL, one I've used effectively myself to lock the "update" channels from RHEL and CentOS for consistency in cluster deployment. The approach costs time and effort to set up, and some internal resources, but it provides much faster "yum update" or "yum check-update" operations than always reaching out to the central repos. It's also very useful when you need to rebuild or update 200 cluster members simultaneously, and the yum metadata alone will overwhelm your external bandwidth. _______________________________________________
And dozens or hundreds of thousands of regular users with need for 3rd party drivers and no time and/or money for carefully maintained internal repositories?
On Wed, Dec 30, 2020 at 5:55 AM Ljubomir Ljubojevic centos@plnet.rs wrote:
On 12/30/20 9:47 AM, Nico Kadel-Garcia wrote:
On Tue, Dec 29, 2020 at 7:04 AM Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 23:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them. ELrepo and others have provided support at the "point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Hi Lamar,
Unfortunately there's an issue with driver disk ISO images too, as they are based on point release GA kernels (as are most all OEM ISO driver disks), and Stream regularly releases new installation media based on the latest Stream kernel, so if you need a 3rd party driver disk to install, you are probably out of luck and may not even be able to install CentOS Stream, even if a driver were available. I'm guessing for the next year you could install CentOS Linux 8 and convert to Stream, but after 2021 that will no longer be an option.
He should also consider finding a more reputable hardware vendor. Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
Why wouldn't installing from CentOS 8.3 and updating to CentOS Stream be an option? The yum repos are well defined, in well-defined RPMs. And an add-on internal RPM to point to a locked internal repository to provide a "base OS imae" snapshot is a commonplace practice for EPEL, one I've used effectively myself to lock the "update" channels from RHEL and CentOS for consistency in cluster deployment. The approach costs time and effort to set up, and some internal resources, but it provides much faster "yum update" or "yum check-update" operations than always reaching out to the central repos. It's also very useful when you need to rebuild or update 200 cluster members simultaneously, and the yum metadata alone will overwhelm your external bandwidth. _______________________________________________
And dozens or hundreds of thousands of regular users with need for 3rd party drivers and no time and/or money for carefully maintained internal repositories?
Anyone who has to this kind of 3rd party integration is already devoting time and resources to it. The space for a snapshotted internal mirror is quite modest, the cooperation with internal architects is the expensive resource And keeping people from investing a much larger engineering and cash investment in a "spacewalk" system which will spend many man-hours on hand-tuning individually tuned RPM lists for individual servers is.... well, it's one of those discussions that makes DevOps DevOps. that is ordinarily a *much* bigger expense. The primary difficulty is when Red Hat sells RHN as part of a "pure Red Hat solution" and it won't properly handle CentOS or EPEL or other third party channels. RHN in this setup would need to cache and enable RPM's that have been deleted from the upstream repositories, such as CentOS Stream and EPEL, and *that* argument requires very, very careful attention. It's vastly cheaper to just set up your own locked internal repo and stay away from the RHN tuning arguments.
The time spent for a maintained internal repository is really quite modest: the github repo I publish for useful tools for one dates back to 2014, with reposync-rhel8.sh tools from 2019 if you feel the need for those. It's often well worth the benefit in update performance and deployment to use an internal mirror. Snapshotting those takes some thought, but is possible with the old "rsnapshot" tool, which I also did some work with when it was published back in.... 2003.
In other words, these are not new problems. They have some available workarounds. I publish places to start if you'd care to try them out.
Nico Kadel-Garcia
On 12/30/20 1:26 PM, Nico Kadel-Garcia wrote:
On Wed, Dec 30, 2020 at 5:55 AM Ljubomir Ljubojevic centos@plnet.rs wrote:
On 12/30/20 9:47 AM, Nico Kadel-Garcia wrote:
On Tue, Dec 29, 2020 at 7:04 AM Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 23:32, Lamar Owen wrote:
Will Stream cut it for me? One issue that keeps getting glossed over is that many drivers that are already in-kernel, not 3rd party, but disabled by Red Hat, still have users who need them. ELrepo and others have provided support at the "point release" milestones for these "unsupported" drivers; it really looks like Stream will break this hard.
For instance, I need megaraid_sas for my servers; that's not a 3rd party binary driver, but is already in-kernel; it is intentionally not built by Red Hat. ELrepo rebuilds this AND most importantly provides a working driver disk for installs; I just don't see Red Hat providing these drivers, even in a SIG, for hardware they have already decided is "unsupported "; but I always reserve the right to be wrong.
Hi Lamar,
Unfortunately there's an issue with driver disk ISO images too, as they are based on point release GA kernels (as are most all OEM ISO driver disks), and Stream regularly releases new installation media based on the latest Stream kernel, so if you need a 3rd party driver disk to install, you are probably out of luck and may not even be able to install CentOS Stream, even if a driver were available. I'm guessing for the next year you could install CentOS Linux 8 and convert to Stream, but after 2021 that will no longer be an option.
He should also consider finding a more reputable hardware vendor. Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
Why wouldn't installing from CentOS 8.3 and updating to CentOS Stream be an option? The yum repos are well defined, in well-defined RPMs. And an add-on internal RPM to point to a locked internal repository to provide a "base OS imae" snapshot is a commonplace practice for EPEL, one I've used effectively myself to lock the "update" channels from RHEL and CentOS for consistency in cluster deployment. The approach costs time and effort to set up, and some internal resources, but it provides much faster "yum update" or "yum check-update" operations than always reaching out to the central repos. It's also very useful when you need to rebuild or update 200 cluster members simultaneously, and the yum metadata alone will overwhelm your external bandwidth. _______________________________________________
And dozens or hundreds of thousands of regular users with need for 3rd party drivers and no time and/or money for carefully maintained internal repositories?
Anyone who has to this kind of 3rd party integration is already devoting time and resources to it. The space for a snapshotted internal mirror is quite modest, the cooperation with internal architects is the expensive resource And keeping people from investing a much larger engineering and cash investment in a "spacewalk" system which will spend many man-hours on hand-tuning individually tuned RPM lists for individual servers is.... well, it's one of those discussions that makes DevOps DevOps. that is ordinarily a *much* bigger expense. The primary difficulty is when Red Hat sells RHN as part of a "pure Red Hat solution" and it won't properly handle CentOS or EPEL or other third party channels. RHN in this setup would need to cache and enable RPM's that have been deleted from the upstream repositories, such as CentOS Stream and EPEL, and *that* argument requires very, very careful attention. It's vastly cheaper to just set up your own locked internal repo and stay away from the RHN tuning arguments.
The time spent for a maintained internal repository is really quite modest: the github repo I publish for useful tools for one dates back to 2014, with reposync-rhel8.sh tools from 2019 if you feel the need for those. It's often well worth the benefit in update performance and deployment to use an internal mirror. Snapshotting those takes some thought, but is possible with the old "rsnapshot" tool, which I also did some work with when it was published back in.... 2003.
In other words, these are not new problems. They have some available workarounds. I publish places to start if you'd care to try them out.
It looks like you did not understand what I mean when I write "no time and/or money". For example, a single CentOS server in client premises. Assembled from retail PC components (large parts of the planet buy parts and assemble them or buy from small shop that will sell them desired parts and assemble it for free). Is there logic in creating and regularly maintaining local repo for a single CentOS Stream server? Or maybe 3? Or is it much more simpler to just use distribution that does not need a person to be paid just to watch if distro developers will drop a ball? I vote for fire-and-forget distro, preferably free RHEL clone, but in any case one that does not need organized effort to upgrade...
Nico Kadel-Garcia _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/30/20 3:47 AM, Nico Kadel-Garcia wrote:
He should also consider finding a more reputable hardware vendor.
Hey Nico, I'm right here.... :-) Anyway, these servers are Dell. Dell isn't a reputable vendor these days?
In any case, it's a moot point, since I didn't pick these servers, the servers picked me (that is, they were donated to us, and, to modify an old saying in my area of the world, "never look a gift server in the RAID controller." (the original expression: "never look a gift horse in the mouth.")).
Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
As mostly an end-user these days, I really couldn't care less what kind of hacks are required. I just know that we as a public charity were donated these servers to put to use; I intend to put them to use in the spirit of the US Internal Revenu Code Section 501(c)(3). They beat the two-generations-older Dell PowerEdge 1950s that they are replacing, which still use the same basic RAID chipset and have serious performance bottlenecks as KVM hosts with modern KVM.
--
They say hindsight is 20/20; can we make 2020 be hindsight now?
On Wed, Dec 30, 2020 at 12:42 PM Lamar Owen lowen@pari.edu wrote:
On 12/30/20 3:47 AM, Nico Kadel-Garcia wrote:
He should also consider finding a more reputable hardware vendor.
Hey Nico, I'm right here.... :-) Anyway, these servers are Dell. Dell isn't a reputable vendor these days?
I've not looked this year. They need careful attention to avoid changing parts or chipsets without notification and breaking compatibility with your current, stable kernel. They all do. It was why, when I did hardware evals, I made sure to get one of any new model we wanted to order, put it in my test rack, and keep it there for kernel and OS deployment testing.
In any case, it's a moot point, since I didn't pick these servers, the servers picked me (that is, they were donated to us, and, to modify an old saying in my area of the world, "never look a gift server in the RAID controller." (the original expression: "never look a gift horse in the mouth.")).
If they cost you that much work, consider buying modest Adaptec RAID controllers. MegaRAID had a bad habit of doing a lot of the RAID work in the kernel, not on the attached hardware, which was why the kernel modules were so critical. And the add-on drivers from the vendor were usually at least a year behind those in the current Red Hat kernel and were in fact a regression, as long as we stuck to a contemporary Red Hat release. RHEL or CentOS these days. I'm very surprised if you have hardware that actually needs that third-party driver, though it may take work to get the latest kernel into a bootable ISO or PXE image.
I have war stories about those drivers.
Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
As mostly an end-user these days, I really couldn't care less what kind of hacks are required. I just know that we as a public charity were donated these servers to put to use; I intend to put them to use in the spirit of the US Internal Revenu Code Section 501(c)(3). They beat the two-generations-older Dell PowerEdge 1950s that they are replacing, which still use the same basic RAID chipset and have serious performance bottlenecks as KVM hosts with modern KVM.
Makes you wonder why someone gave them away?
Back to CentOS Stream. One of the CentOS kernel advantages was that Red Hat apparently did test suites across a wide variety of hardware before considering a kernel for release. I'm afraid CentOS Stream kernels won't have that regression testing done.
On Thu, 31 Dec 2020 at 07:49, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Wed, Dec 30, 2020 at 12:42 PM Lamar Owen lowen@pari.edu wrote:
On 12/30/20 3:47 AM, Nico Kadel-Garcia wrote:
He should also consider finding a more reputable hardware vendor.
Hey Nico, I'm right here.... :-) Anyway, these servers are Dell. Dell isn't a reputable vendor these days?
I've not looked this year. They need careful attention to avoid changing parts or chipsets without notification and breaking compatibility with your current, stable kernel. They all do. It was why, when I did hardware evals, I made sure to get one of any new model we wanted to order, put it in my test rack, and keep it there for kernel and OS deployment testing.
In any case, it's a moot point, since I didn't pick these servers, the servers picked me (that is, they were donated to us, and, to modify an old saying in my area of the world, "never look a gift server in the RAID controller." (the original expression: "never look a gift horse in the mouth.")).
If they cost you that much work, consider buying modest Adaptec RAID controllers. MegaRAID had a bad habit of doing a lot of the RAID work
Adaptec controllers haven't existed as 'distinct' hardware since before 2010. Since then they have been rebranded controllers some of which are MegaRAID, LSI and other systems. For the years between 2010 and 2018, the Megaraid or LSI was a better bet than Adaptec.
in the kernel, not on the attached hardware, which was why the kernel modules were so critical. And the add-on drivers from the vendor were usually at least a year behind those in the current Red Hat kernel and were in fact a regression, as long as we stuck to a contemporary Red Hat release. RHEL or CentOS these days. I'm very surprised if you have hardware that actually needs that third-party driver, though it may take work to get the latest kernel into a bootable ISO or PXE image.
RHEL8 kernels no longer comes with the megaraid controller enabled thus you need the CentOS kernel. The third party module is just the one in the normal kernel compiled.
I have war stories about those drivers.
Those so-called "megaraid" chipsets were abominable monstrosities and there were a number of extremely poorly written vendor kernel hacks supporting them.
As mostly an end-user these days, I really couldn't care less what kind of hacks are required. I just know that we as a public charity were donated these servers to put to use; I intend to put them to use in the spirit of the US Internal Revenu Code Section 501(c)(3). They beat the two-generations-older Dell PowerEdge 1950s that they are replacing, which still use the same basic RAID chipset and have serious performance bottlenecks as KVM hosts with modern KVM.
Makes you wonder why someone gave them away?
I believe there was a US Tax benefit for companies if they traded out their hardware after 4 but before 8 years and donated their old hardware to charities. Since there was a lot of hardware being 'donated' most of the US charities only want to take items they can get extended warranties and parts from the manufacturer from.. so getting a donated Dell is better than a donated Supermicro.
Back to CentOS Stream. One of the CentOS kernel advantages was that Red Hat apparently did test suites across a wide variety of hardware before considering a kernel for release. I'm afraid CentOS Stream kernels won't have that regression testing done. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Thu, Dec 31, 2020 at 07:49:29AM -0500, Nico Kadel-Garcia wrote:
Back to CentOS Stream. One of the CentOS kernel advantages was that Red Hat apparently did test suites across a wide variety of hardware before considering a kernel for release. I'm afraid CentOS Stream kernels won't have that regression testing done.
As I understand it, they _will_ go through that suite before being released into Stream.
On 12/31/20 7:49 AM, Nico Kadel-Garcia wrote:
If they cost you that much work, consider buying modest Adaptec RAID controllers. MegaRAID had a bad habit of doing a lot of the RAID work in the kernel, not on the attached hardware, which was why the kernel modules were so critical. ...I'm very surprised if you have hardware that actually needs that third-party driver, though it may take work to get the latest kernel into a bootable ISO or PXE image.
Well: [root@grymonia ~]# lspci|grep RAID 03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2108 [Liberator] (rev 05) [root@grymonia ~]# lspci -n|grep 03.00.0 03:00.0 0104: 1000:0079 (rev 05)
See http://elrepo.org/tiki/DeviceIDs and https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Note especially the wording in the latter link: " The following adapters from the megaraid_sas driver have been removed: ... SAS0079GEN2, PCI ID 0x1000:0x0079 ...." The problem is NOT that megaraid_sas has been removed (it hasn't, see below); the problem is that PCI ID 0x1000:0x0079 has been removed from the in-kernel megaraid_sas driver. Yes, the released kernel HAS a megaraid_sas.ko.xz module, but the PCI ID of the controller in the R710 is removed by Red Hat's patches from the released module. (This is not the only hardware for which this situation is true; it honestly looks like Red Hat asked Dell what Dell wanted supported and Dell cut off anything that Dell considers to be "too old.") Here's what the modules tree looks like, along with a listing of the contents of the ELrepo kmod-megaraid_sas package:
[root@grymonia modules]# find /usr/lib/modules -name megaraid_sas* -print /usr/lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas /usr/lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas/megaraid_sas.ko /usr/lib/modules/4.18.0-193.14.2.el8_2.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz /usr/lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/megaraid_sas /usr/lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko /usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz /usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/weak-updates/megaraid_sas /usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko /usr/lib/modules/4.18.0-193.28.1.el8_2.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz /usr/lib/modules/4.18.0-193.28.1.el8_2.x86_64/weak-updates/megaraid_sas /usr/lib/modules/4.18.0-193.28.1.el8_2.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko [root@grymonia modules]# rpm -ql kmod-megaraid_sas /etc/depmod.d/kmod-megaraid_sas.conf /lib/modules/4.18.0-193.el8.x86_64 /lib/modules/4.18.0-193.el8.x86_64/extra /lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas /lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas/megaraid_sas.ko /usr/share/doc/kmod-megaraid_sas-07.710.50.00 /usr/share/doc/kmod-megaraid_sas-07.710.50.00/GPL-v2.0.txt /usr/share/doc/kmod-megaraid_sas-07.710.50.00/greylist.txt [root@grymonia modules]#
It's _not_ a 3rd party driver if the driver is in the released kernel.... this is the point that seems to continue to be glossed over, that some of these point-release-dependent drivers are restoring functionality to already in-kernel modules where that functionality has been intentionally removed by Red Hat because Red Hat won't support that hardware. If Red Hat is patching out those PCI IDs for RHEL, it's highly likely, in my opinion, that Red Hat isn't going to not patch them out in the Stream kernels (yes, I know, double negative, but all Red Hat has to do is not patch out the support for that PCI ID to restore support).
Makes you wonder why someone gave them away?
I actually don't wonder, as it's really simple and I know exactly why they donated them to us: they upgraded (the R710 is a few years old now), we needed them (they're newer by five years and much better than what we were already using), they got a tax credit, and we got effective public-support revenue for our 990 (and it delays these R710s from becoming e-waste). It's a win-win; they're still good servers with excellent performance for our workloads here, and I tend to ask for spares of everything that's donated so I can keep it running. It is far less expensive to deal with these issues than to buy new servers. -- They say hindsight is 20/20; Now, can 2020 just be hindsight already?
On Thursday, December 31, 2020 11:24 AM, Lamar Owen lowen@pari.edu wrote:
On 12/31/20 7:49 AM, Nico Kadel-Garcia wrote:
If they cost you that much work, consider buying modest Adaptec RAID controllers. MegaRAID had a bad habit of doing a lot of the RAID work in the kernel, not on the attached hardware, which was why the kernel modules were so critical. ...I'm very surprised if you have hardware that actually needs that third-party driver, though it may take work to get the latest kernel into a bootable ISO or PXE image.
Well: [root@grymonia ~]# lspci|grep RAID 03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2108 [Liberator] (rev 05) [root@grymonia ~]# lspci -n|grep 03.00.0 03:00.0 0104: 1000:0079 (rev 05)
See http://elrepo.org/tiki/DeviceIDs and https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Note especially the wording in the latter link: " The following adapters from the megaraid_sas driver have been removed: ... SAS0079GEN2, PCI ID 0x1000:0x0079 ...." The problem is NOT that megaraid_sas has been removed (it hasn't, see below); the problem is that PCI ID 0x1000:0x0079 has been removed from the in-kernel megaraid_sas driver. Yes, the released kernel HAS a megaraid_sas.ko.xz module, but the PCI ID of the controller in the R710 is removed by Red Hat's patches from the released module. (This is not the only hardware for which this situation is true; it honestly looks like Red Hat asked Dell what Dell wanted supported and Dell cut off anything that Dell considers to be "too old.") Here's what the modules tree looks like, along with a listing of the contents of the ELrepo kmod-megaraid_sas package:
[root@grymonia modules]# find /usr/lib/modules -name megaraid_sas* -print /usr/lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas /usr/lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas/megaraid_sas.ko /usr/lib/modules/4.18.0-193.14.2.el8_2.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz /usr/lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/megaraid_sas /usr/lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko /usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz /usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/weak-updates/megaraid_sas /usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko /usr/lib/modules/4.18.0-193.28.1.el8_2.x86_64/kernel/drivers/scsi/megaraid/megaraid_sas.ko.xz /usr/lib/modules/4.18.0-193.28.1.el8_2.x86_64/weak-updates/megaraid_sas /usr/lib/modules/4.18.0-193.28.1.el8_2.x86_64/weak-updates/megaraid_sas/megaraid_sas.ko [root@grymonia modules]# rpm -ql kmod-megaraid_sas /etc/depmod.d/kmod-megaraid_sas.conf /lib/modules/4.18.0-193.el8.x86_64 /lib/modules/4.18.0-193.el8.x86_64/extra /lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas /lib/modules/4.18.0-193.el8.x86_64/extra/megaraid_sas/megaraid_sas.ko /usr/share/doc/kmod-megaraid_sas-07.710.50.00 /usr/share/doc/kmod-megaraid_sas-07.710.50.00/GPL-v2.0.txt /usr/share/doc/kmod-megaraid_sas-07.710.50.00/greylist.txt [root@grymonia modules]#
It's not a 3rd party driver if the driver is in the released kernel.... this is the point that seems to continue to be glossed over, that some of these point-release-dependent drivers are restoring functionality to already in-kernel modules where that functionality has been intentionally removed by Red Hat because Red Hat won't support that hardware. If Red Hat is patching out those PCI IDs for RHEL, it's highly likely, in my opinion, that Red Hat isn't going to not patch them out in the Stream kernels (yes, I know, double negative, but all Red Hat has to do is not patch out the support for that PCI ID to restore support).
Given what I know about Dell, I think it is likely that Dell bought an OEM specified custom flavor of the Megaraid to be one of their Dell PERC.
At some point a RHEL customer requested assistance with a serious problem with their MegaRAID/PERC. RHEL support was not able to resolve the issue directly as it required a firmware update. So since Dell is a Red Hat partner, they reached out to Dell. Dell then probably said something about no longer having a contract to get firmware updates from the vendor.
RHEL support then responded to the customer that they are unable to provide a solution. The customer then gets upset about the PERC being listed as "supported" and asks why. The response would then be that RHEL is committed to attempting to support the same hardware through the entire life-cycle of RHEL 7.x but they are still limited by what the hardware/firmware vendor is willing to assist. However, based on feedback from their customers, they will remove support in the next major release (RHEL 8.0) to make the situation more clear to customers.
I can't prove any of that is what really took place but based on my experiences in the past, I think it is a possible explanation.
My understanding of Stream is RHEL packages should largely come as-is from packages previously released on Stream. Hence, Stream might get Technology Preview drivers before RHEL. However, the policy of deprecation/removal of hardware will also be reflected in Stream.
You might be able to use kernel boot arguments or tricks involving dracut and echo'ing the pci-id into /sys to working around the removal. But I haven't actually tried it with a Dell PERC.
On Thursday, December 31, 2020 6:49 AM, Nico Kadel-Garcia nkadel@gmail.com wrote:
Back to CentOS Stream. One of the CentOS kernel advantages was that Red Hat apparently did test suites across a wide variety of hardware before considering a kernel for release. I'm afraid CentOS Stream kernels won't have that regression testing done.
I am not sure that is true.
Stef Walter has given the closest to technical insight driving this decision here:
https://blog.centos.org/2020/12/centos-stream-is-continuous-delivery/
The blog post is broken into four parts. The fourth part comes the closest to talking about the need for better *automated* hardware testing. There are two key components that talks directly to the hardware that you never want to fail, GRUB and the kernel. The blog post talks about how an issue with GRUB cause systems not to boot, but given regressions with the kernel can have the same impact, I think they see it just as important to test that as well.
Rather than the problem being if hardware testing will take place, I believe the problem will be if regression tests will be complete. Reading between the lines (hopefully Stef Walter can correct me if I'm wrong), Red Hat seems to be "asking" for feedback on what areas they need to add regression tests.
The problem shouldn't be seen as if Stream will add value to the CentOS community. I am sure it can add value. Adding CD/CI is a worthy goal.
The problem is if CD/CI can be improved under a cathedral model that has no respect for the community.
Keep in mind, the meeting to renege on the CentOS 8 commitment of 10 years was back on November 11th. It has already been 7 weeks since then.
Things we should have seen if there is a two way street when interacting with the CentOS community:
(1) There was Red Hat employees on the governance meeting that have not been active members of the CentOS community. They have had plenty of time to introduce themselves to the mailing list and talk about their own thoughts regarding this "hard" decision. Instead they seem to be operating from an ivory tower.
(2) If Red Hat wants to be believed that Stream with have a life span of 5 years, they should have cut CentOS 8 to 5 years. Cutting it to a life span of 2 years does two things:
(2)(a) Red Hat has sent the message that CentOS 7 is the preferable version to stay on and CentOS 8 (and by extention Stream) are not worth adopting
(2)(b) Red Hat has sent the message that the community should not have confidence in the life span of Stream and it is just one more governance meeting it being terminated at anytime
(3) Red Hat is rubber stamping this change with "openness" over and over while still providing no promises about their policy of obfuscating the kernel patches for Stream
There is also the issue of being able to perform rollbacks when the Stream repository has no previous versions of the packages. It is enough incentive to keep future packages from having a regression for me to file a bug. They don't need to make it hard for me to address the regression myself by making it hard for me to roll it back. That again is just going to hurt adoption.
While the messaging has been pathetic for the last 7 weeks, I hope we can be careful about claiming Stream will go untested. Testing seems to be the whole point.
On Thu, Dec 31, 2020 at 12:39 PM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
On Thursday, December 31, 2020 6:49 AM, Nico Kadel-Garcia nkadel@gmail.com wrote:
Back to CentOS Stream. One of the CentOS kernel advantages was that Red Hat apparently did test suites across a wide variety of hardware before considering a kernel for release. I'm afraid CentOS Stream kernels won't have that regression testing done.
I am not sure that is true.
Stef Walter has given the closest to technical insight driving this decision here:
https://blog.centos.org/2020/12/centos-stream-is-continuous-delivery/
The blog post is broken into four parts. The fourth part comes the closest to talking about the need for better *automated* hardware testing. There are two key components that talks directly to the hardware that you never want to fail, GRUB and the kernel. The blog post talks about how an issue with GRUB cause systems not to boot, but given regressions with the kernel can have the same impact, I think they see it just as important to test that as well.
Rather than the problem being if hardware testing will take place, I believe the problem will be if regression tests will be complete. Reading between the lines (hopefully Stef Walter can correct me if I'm wrong), Red Hat seems to be "asking" for feedback on what areas they need to add regression tests.
It's expensive to set up. i used to do that for hardware and OS deployment testing for a network of over 10,000 systems, and keeping my racks populated with all the different hardware variants was... an adventure. Back in the days of LILO rather than grub, it used to be possible to set the boot loader to consider one kernel as the "default", but to beet the next time with a designated kernel one time only. It made testing kernels safer because a failed kernel installation failed reboot only *once* and only needed a power cycle to revert. I don't believe grub has ever been successfully patched to support this: I'd welcome it if they did, though i'm not testing hardware and kernels these days.
On Fri, Jan 01, 2021 at 04:50:44PM -0500, Nico Kadel-Garcia wrote:
possible to set the boot loader to consider one kernel as the "default", but to beet the next time with a designated kernel one time only. It made testing kernels safer because a failed kernel installation failed reboot only *once* and only needed a power cycle to revert. I don't believe grub has ever been successfully patched to support this: I'd welcome it if they did, though i'm not testing hardware and kernels these days.
Grub can do a number of fancy things, including "boot once" and (I think probably better) a "fallback" option. (You may want `kernel.panic = 10` or something in /etc/sysctl.conf to go along with this, depending on what kind of failure you're expecting.)
I don't believe grub has ever been successfully patched to
support this: I'd welcome it if they did, though i'm not testing hardware and kernels these days.
grub-reboot — Set the default boot menu entry for the next boot only. grub-set-default — Set the default boot menu entry for GRUB.
Set the default to be latest-1 and define one time boot with latest.
Best Regards, Strahil Nikolov
On Fri, Jan 1, 2021 at 6:26 PM Strahil Nikolov via CentOS-devel centos-devel@centos.org wrote:
I don't believe grub has ever been successfully patched to
support this: I'd welcome it if they did, though i'm not testing hardware and kernels these days.
grub-reboot — Set the default boot menu entry for the next boot only. grub-set-default — Set the default boot menu entry for GRUB.
Set the default to be latest-1 and define one time boot with latest.
Cool, I think this is new since the last time i tried to do this. It's also been a while.
On 12/29/20 7:04 AM, Phil Perry wrote:
Unfortunately there's an issue with driver disk ISO images too, as they are based on point release GA kernels (as are most all OEM ISO driver disks), and Stream regularly releases new installation media based on the latest Stream kernel, so if you need a 3rd party driver disk to install, you are probably out of luck and may not even be able to install CentOS Stream, even if a driver were available. I'm guessing for the next year you could install CentOS Linux 8 and convert to Stream, but after 2021 that will no longer be an option.
Yes, thanks for the reminder there. While I haven't decided what exactly I'm going to do yet (I'm going to wait a few months before making a final decsion), it's good to have options. And I thank you and the rest of the intrepid ELrepo team for all your fine work supporting these drivers for RHEL, even though I use them on CentOS.
On 28.12.2020 00:27, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 23:00, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 21:48, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/27/20 12:29 PM, Alexander Bokovoy wrote: > On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Let's be clear: the above 'diff' is your own opinion and a speculation, not based on any public information. There are no facts that would support a claim that future RHEL subscription programs we are promised will all be paid ones.
Let's be clear: quantifier "all" is your own interpretation and was not assumed in my statement.
I find it strange to add 'paid' where it is not necessary needed to be if you have no facts to say so. In fact, Chris Wrights blog is very explicit:
https://www.redhat.com/en/blog/centos-stream-building-innovative-future-ente...
"In the first half of 2021, we plan to introduce low- or no-cost programs for a variety of use cases, including options for open source projects and communities and expansion of the Red Hat Enterprise Linux Developer subscription use cases to better serve the needs of systems administrators. We’ll share more details as these initiatives coalesce."
"low" is in accordance with "paid". As for "as these initiatives coalesce", I see yet another vague promise. Looks like this decision has been taken *post factum* and was, so to say, unplanned.
[...]
Both can be achieved by forming a SIG, contributing some resources, and having a shared set of common CI tools. CKI already gives a platform to base on, for kernel-specific components. Fedora Project already gives a way to organize CIs around similar topics for other components, so reusing the tooling is definitely possible.
The primary moot point of CentOS Stream usability, as I see it, is lack of simple means to rollback (one or more) packages, to restore a predictable software behavior after another daily update breaks something.
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
I have doubts it would ever work, but it's worth trying, just to make sure.
On ma, 28 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 28.12.2020 00:27, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 23:00, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 21:48, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Ljubomir Ljubojevic wrote: > On 12/27/20 12:29 PM, Alexander Bokovoy wrote: >> On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Let's be clear: the above 'diff' is your own opinion and a speculation, not based on any public information. There are no facts that would support a claim that future RHEL subscription programs we are promised will all be paid ones.
Let's be clear: quantifier "all" is your own interpretation and was not assumed in my statement.
I find it strange to add 'paid' where it is not necessary needed to be if you have no facts to say so. In fact, Chris Wrights blog is very explicit:
https://www.redhat.com/en/blog/centos-stream-building-innovative-future-ente...
"In the first half of 2021, we plan to introduce low- or no-cost programs for a variety of use cases, including options for open source projects and communities and expansion of the Red Hat Enterprise Linux Developer subscription use cases to better serve the needs of systems administrators. We’ll share more details as these initiatives coalesce."
"low" is in accordance with "paid". As for "as these initiatives coalesce", I see yet another vague promise. Looks like this decision has been taken *post factum* and was, so to say, unplanned.
You keep ignoring 'no-cost' part there.
As for how decisions were taken, there was plenty of details published from the people involved (I wasn't).
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
I have doubts it would ever work, but it's worth trying, just to make sure.
Not that there are no examples of failing programs anywhere, but keeping yourself to a darker tone all the time isn't a great way to improve. Please file bugs/process requests and we'll see how those can be handled as a part of CentOS Stream programs.
On 28.12.2020 17:34, Alexander Bokovoy wrote:
On ma, 28 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 28.12.2020 00:27, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 23:00, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 21:48, Alexander Bokovoy wrote: > On su, 27 joulu 2020, Ljubomir Ljubojevic wrote: >> On 12/27/20 12:29 PM, Alexander Bokovoy wrote: >>> On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote: > > Following your approach to a detailed information about the Stream, > we've been told there are various RHEL subscription programs coming > next > year that would address use of RHEL for many existing CentOS users.
-various RHEL subscription programs +various *paid* RHEL subscription programs
Let's be clear: the above 'diff' is your own opinion and a speculation, not based on any public information. There are no facts that would support a claim that future RHEL subscription programs we are promised will all be paid ones.
Let's be clear: quantifier "all" is your own interpretation and was not assumed in my statement.
I find it strange to add 'paid' where it is not necessary needed to be if you have no facts to say so. In fact, Chris Wrights blog is very explicit:
https://www.redhat.com/en/blog/centos-stream-building-innovative-future-ente...
"In the first half of 2021, we plan to introduce low- or no-cost programs for a variety of use cases, including options for open source projects and communities and expansion of the Red Hat Enterprise Linux Developer subscription use cases to better serve the needs of systems administrators. We’ll share more details as these initiatives
coalesce."
"low" is in accordance with "paid". As for "as these initiatives coalesce", I see yet another vague promise. Looks like this decision has been taken *post factum* and was, so to say, unplanned.
You keep ignoring 'no-cost' part there.
It was said "or". "low- *or* no-cost..." Do you see? Not "and", but "or".
Will I sound too unrealistic, if I say that "low-cost" would be much, much more probable?
As for how decisions were taken, there was plenty of details published from the people involved (I wasn't).
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
I have doubts it would ever work, but it's worth trying, just to make sure.
Not that there are no examples of failing programs anywhere, but keeping yourself to a darker tone all the time isn't a great way to improve.
"Failed program" != "Killed program".
RH gives too few grounds to suspect it in good will, ATM.
Please file bugs/process requests and we'll see how those can be handled as a part of CentOS Stream programs.
Sure.
On 12/28/20 3:50 AM, Konstantin Boyandin via CentOS-devel wrote:
On 28.12.2020 17:34, Alexander Bokovoy wrote:
You keep ignoring 'no-cost' part there.
It was said "or". "low- *or* no-cost..." Do you see? Not "and", but "or".
Will I sound too unrealistic, if I say that "low-cost" would be much, much more probable?
That's a totally normal sentence construct for English speakers. It implies that both will exist, and each will be appropriate for different use cases. It does not imply that "no-cost" may or may not exist, only that it may or may not fit a particular use case.
Has this seriously devolved into a discussion requiring symbolic logic?
I'm dusting off my copy from Virginia Klenk.
________________________________________ From: CentOS-devel centos-devel-bounces@centos.org on behalf of Gordon Messmer gordon.messmer@gmail.com Sent: December 28, 2020 2:10 PM To: centos-devel@centos.org Subject: Re: [CentOS-devel] Balancing the needs around the RHEL platform
On 12/28/20 3:50 AM, Konstantin Boyandin via CentOS-devel wrote:
On 28.12.2020 17:34, Alexander Bokovoy wrote:
You keep ignoring 'no-cost' part there.
It was said "or". "low- *or* no-cost..." Do you see? Not "and", but "or".
Will I sound too unrealistic, if I say that "low-cost" would be much, much more probable?
That's a totally normal sentence construct for English speakers. It implies that both will exist, and each will be appropriate for different use cases. It does not imply that "no-cost" may or may not exist, only that it may or may not fit a particular use case.
_______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 29.12.2020 02:10, Gordon Messmer wrote:
On 12/28/20 3:50 AM, Konstantin Boyandin via CentOS-devel wrote:
On 28.12.2020 17:34, Alexander Bokovoy wrote:
You keep ignoring 'no-cost' part there.
It was said "or". "low- *or* no-cost..." Do you see? Not "and", but "or".
Will I sound too unrealistic, if I say that "low-cost" would be much, much more probable?
That's a totally normal sentence construct for English speakers. It implies that both will exist, and each will be appropriate for different use cases. It does not imply that "no-cost" may or may not exist, only that it may or may not fit a particular use case.
Quite probably. In my native language, difference in such constructs between "or" and 'xor" is usually defined by intonation - i.e., open for interpretation.
In any case, it's fruitless to guess. It's more constructive to accept what's done and move forward.
On 27/12/2020 17:27, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
The primary moot point of CentOS Stream usability, as I see it, is lack of simple means to rollback (one or more) packages, to restore a predictable software behavior after another daily update breaks something.
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
It exists, here:
https://bugzilla.redhat.com/show_bug.cgi?id=1908047
2 Weeks, no acknowledgement of the issue, no ETA for a fix, no discussion, little hope?
It's easy for Red Hat folks to come onto this list and make promises, but you have to back it up with action. The community identified an issue. The community identified the solution. It's an easy fix. RH's bit is to fix the issue. The longer it sits there ...
On ma, 28 joulu 2020, Phil Perry wrote:
On 27/12/2020 17:27, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
The primary moot point of CentOS Stream usability, as I see it, is lack of simple means to rollback (one or more) packages, to restore a predictable software behavior after another daily update breaks something.
The simplest solution for these use cases is to actually report a bug against RHEL and/or CentOS Stream and make sure it is reproducible. This would be the quickest way to get the issue backed out or fixed in a number of days. There are means to remove broken packages from RHEL composes and I hope we'd have a way to propagate those 'removals' to CentOS Stream.
This is something worth raising as a feature request if it doesn't exist yet.
It exists, here:
Thanks for pointing to that. I subscribed myself.
2 Weeks, no acknowledgement of the issue, no ETA for a fix, no discussion, little hope?
Let's see after the holiday's season ends.
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: - using elrepo's excellent work on kmods to build a CI for CentOS Stream kernel updates and perhaps block updates based on CI results' consensus (not necessary blocked until it 100% green)
- for proprietary drivers make sure there are clear ways to enable those partners to plug into the same process as an external CI.
Both can be achieved by forming a SIG, contributing some resources, and having a shared set of common CI tools. CKI already gives a platform to base on, for kernel-specific components. Fedora Project already gives a way to organize CIs around similar topics for other components, so reusing the tooling is definitely possible.
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable. Until you read what he wrote, core is that that process can not be really automated like that, and that compiling drivers for each kernel defeats the purpose of kmod drivers. Also, kmod drivers are built against RHEL kernels, not CentOS...
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: - using elrepo's excellent work on kmods to build a CI for CentOS Stream kernel updates and perhaps block updates based on CI results' consensus (not necessary blocked until it 100% green)
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build. And if it is, then I'm not sure why that approach would not be viable.
Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release."
That seems like an ideal solution.
On su, 27 joulu 2020, Gordon Messmer wrote:
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: Â - using elrepo's excellent work on kmods to build a CI for CentOS Â Â Stream kernel updates and perhaps block updates based on CI results' Â Â consensus (not necessary blocked until it 100% green)
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build. And if it is, then I'm not sure why that approach would not be viable.
Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release."
That seems like an ideal solution.
You are getting it right, Gordon, thank you for connecting the threads.
On 28/12/2020 08:02, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Gordon Messmer wrote:
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: Â - using elrepo's excellent work on kmods to build a CI for CentOS Â Â Stream kernel updates and perhaps block updates based on CI results' Â Â consensus (not necessary blocked until it 100% green)
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build. And if it is, then I'm not sure why that approach would not be viable.
Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release."
That seems like an ideal solution.
You are getting it right, Gordon, thank you for connecting the threads.
So if I understand correctly, the proposal is that if a 3rd party (elrepo or OEM) driver does not build against a Stream kernel, you would not release the kernel? That would mean to date there would not be any Stream kernel updates released[1] because _something_ breaks against *all* of them? Or have I misunderstood what is being proposed? That would be a real shame for those folks who would like to test new functionality and/or participate and feed back in the development process.
And when the above situation arises (an elrepo or OEM driver fails to build and/or weak-link against a Stream kernel), what would be the process to fix that? Elrepo drivers are developed against and built for RHEL kernels, so when they fail to build for Stream, that won't get fixed for upto 6 months until the next RHEL point release. Or to put it another way, elrepo is a downstream project, downstream of RHEL.
[1] Caveat that I've only personally tested Stream kernel releases since RHEL8.3 was released (kernels -257 and -259) but I would be highly surprised if the situation was any different for Stream subsequent to earlier point releases.
On Monday, December 28, 2020 5:43 AM, Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 08:02, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Gordon Messmer wrote:
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: Â - using elrepo's excellent work on kmods to build a CI for CentOS Â Â Stream kernel updates and perhaps block updates based on CI results' Â Â consensus (not necessary blocked until it 100% green) Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build. And if it is, then I'm not sure why that approach would not be viable. Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release." That seems like an ideal solution.
You are getting it right, Gordon, thank you for connecting the threads.
So if I understand correctly, the proposal is that if a 3rd party (elrepo or OEM) driver does not build against a Stream kernel, you would not release the kernel? That would mean to date there would not be any Stream kernel updates released[1] because something breaks against all of them? Or have I misunderstood what is being proposed? That would be a real shame for those folks who would like to test new functionality and/or participate and feed back in the development process.
If I understand correctly, the kernel would still be released but be blacklisted from being installed by dnf on some systems. Almost like a subscription based extension to RPM driver metadata to add a Conflict line after the fact to a driver package. Then dnf would process the Conflict accordingly despite it have never existed at the time the driver package was originally released.
I'm not sure how well such a system would be maintained. The inability to rollback is still a serious problem.
It also could create another problem forcing users to decide the best of two evil. They could continue to use their driver or they could continue to get security patches but not both.
It would be nice if driver conflicts was delt with at a more ganular level than allow or deny an entire kernel release. If release 249 worked fine and 250 doesn't, then as the upstream provider the Stream community should be able to look into the issue on a patch by patch basis.
If release 250 has a security patch and a bugfix patch, it would be nice to see if the security patch can still be used with the driver.
Instead, I believe what is purposed would blacklist the security patch as well just because it is bundled with a bugfix which is incompatible. I would consider such a purposal flawed.
On ma, 28 joulu 2020, Phil Perry wrote:
On 28/12/2020 08:02, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Gordon Messmer wrote:
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: ÃÂ - using elrepo's excellent work on kmods to build a CI for CentOS ÃÂ ÃÂ Stream kernel updates and perhaps block updates based on CI results' ÃÂ ÃÂ consensus (not necessary blocked until it 100% green)
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build.ÃÂ And if it is, then I'm not sure why that approach would not be viable.
Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release."
That seems like an ideal solution.
You are getting it right, Gordon, thank you for connecting the threads.
So if I understand correctly, the proposal is that if a 3rd party (elrepo or OEM) driver does not build against a Stream kernel, you would not release the kernel? That would mean to date there would not be any Stream kernel updates released[1] because _something_ breaks against *all* of them? Or have I misunderstood what is being proposed? That would be a real shame for those folks who would like to test new functionality and/or participate and feed back in the development process.
Neal and I suggested to be able to use the content (source) of elrepo drivers to be a CI test for CentOS Stream kernel updates. That might be non-blocking, might be blocking if X% of packages fail, might be blocking in 'rings of importance' and so on, it is not needed to be black and white.
And when the above situation arises (an elrepo or OEM driver fails to build and/or weak-link against a Stream kernel), what would be the process to fix that? Elrepo drivers are developed against and built for RHEL kernels, so when they fail to build for Stream, that won't get fixed for upto 6 months until the next RHEL point release. Or to put it another way, elrepo is a downstream project, downstream of RHEL.
If Elrepo drivers have actual developers, getting 6 months advance notification of their driver's failures against upcoming RHEL could be valuable to plan and fix the issues in due time.
On Mon, Dec 28, 2020 at 7:25 AM Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, Phil Perry wrote:
On 28/12/2020 08:02, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Gordon Messmer wrote:
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers:  - using elrepo's excellent work on kmods to build a CI for CentOS   Stream kernel updates and perhaps block updates based on CI results'   consensus (not necessary blocked until it 100% green)
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build. And if it is, then I'm not sure why that approach would not be viable.
Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release."
That seems like an ideal solution.
You are getting it right, Gordon, thank you for connecting the threads.
So if I understand correctly, the proposal is that if a 3rd party (elrepo or OEM) driver does not build against a Stream kernel, you would not release the kernel? That would mean to date there would not be any Stream kernel updates released[1] because _something_ breaks against *all* of them? Or have I misunderstood what is being proposed? That would be a real shame for those folks who would like to test new functionality and/or participate and feed back in the development process.
Neal and I suggested to be able to use the content (source) of elrepo drivers to be a CI test for CentOS Stream kernel updates. That might be non-blocking, might be blocking if X% of packages fail, might be blocking in 'rings of importance' and so on, it is not needed to be black and white.
Yep. There's a lot of opportunity here for what we can do here. The important part is that we have the *data* to figure out *what* to do.
And when the above situation arises (an elrepo or OEM driver fails to build and/or weak-link against a Stream kernel), what would be the process to fix that? Elrepo drivers are developed against and built for RHEL kernels, so when they fail to build for Stream, that won't get fixed for upto 6 months until the next RHEL point release. Or to put it another way, elrepo is a downstream project, downstream of RHEL.
If Elrepo drivers have actual developers, getting 6 months advance notification of their driver's failures against upcoming RHEL could be valuable to plan and fix the issues in due time.
In general, it would make *my* life easier if elrepo drivers were tracking the RHEL kernel development and getting fixed along the way, because it's a pain when I have to deal with customers who use our stuff and can't upgrade because of broken drivers. That is the situation *now* with CentOS Linux.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 28/12/2020 12:29, Neal Gompa wrote:
On Mon, Dec 28, 2020 at 7:25 AM Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, Phil Perry wrote:
And when the above situation arises (an elrepo or OEM driver fails to build and/or weak-link against a Stream kernel), what would be the process to fix that? Elrepo drivers are developed against and built for RHEL kernels, so when they fail to build for Stream, that won't get fixed for upto 6 months until the next RHEL point release. Or to put it another way, elrepo is a downstream project, downstream of RHEL.
If Elrepo drivers have actual developers, getting 6 months advance notification of their driver's failures against upcoming RHEL could be valuable to plan and fix the issues in due time.
Yes, absolutely, we (elrepo developers) will likely use the RHEL development (Stream) kernel to assess what is broken and what needs fixing, and to help develop those fixes ahead of the next RHEL point release kernel.
At present we aim to get our build systems updated, and to rebase, fix and release any broken packages as soon as possible after the RHEL point release is public. Looking at the RHEL8.3 point release on 3/11/2020, we had packages released daily on 4-9th November so it took us just under a week to update all our infrastructure, rebase code, fix build issues, rebuild sign and release our whole repository. Not bad for a voluntary effort in addition to holding down 40h/week $dayjob. Being able to develop fixes in advance will for sure help us to reduce that time frame.
In general, it would make *my* life easier if elrepo drivers were tracking the RHEL kernel development and getting fixed along the way, because it's a pain when I have to deal with customers who use our stuff and can't upgrade because of broken drivers. That is the situation *now* with CentOS Linux.
If elrepo drivers are broken on CentOS Linux (not Stream), you should file a bug, either with elrepo if the driver also does not work on RHEL and we will fix it, or with CentOS if the driver works on RHEL but not on CentOS Linux and CentOS can fix it.
If by "CentOS Linux", you actually mean CentOS Stream, by definition that's never going to happen. Elrepo may use the Stream kernel to see what's coming and help develop fixes for the next release of RHEL (see comments above), or to contribute our fixes back upstream, but elrepo driver packages are built on RHEL build systems against RHEL kernels - we can not build against something that has not been released yet, and even if we could, those packages would not work for all our current RHEL (and CentOS Linux 8) users who are running 8.3 *now* and will continue to run 8.3 kernels for at least the next 4-5 months.
On Mon, Dec 28, 2020 at 9:26 AM Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 12:29, Neal Gompa wrote:
On Mon, Dec 28, 2020 at 7:25 AM Alexander Bokovoy abokovoy@redhat.com wrote:
On ma, 28 joulu 2020, Phil Perry wrote:
And when the above situation arises (an elrepo or OEM driver fails to build and/or weak-link against a Stream kernel), what would be the process to fix that? Elrepo drivers are developed against and built for RHEL kernels, so when they fail to build for Stream, that won't get fixed for upto 6 months until the next RHEL point release. Or to put it another way, elrepo is a downstream project, downstream of RHEL.
If Elrepo drivers have actual developers, getting 6 months advance notification of their driver's failures against upcoming RHEL could be valuable to plan and fix the issues in due time.
Yes, absolutely, we (elrepo developers) will likely use the RHEL development (Stream) kernel to assess what is broken and what needs fixing, and to help develop those fixes ahead of the next RHEL point release kernel.
At present we aim to get our build systems updated, and to rebase, fix and release any broken packages as soon as possible after the RHEL point release is public. Looking at the RHEL8.3 point release on 3/11/2020, we had packages released daily on 4-9th November so it took us just under a week to update all our infrastructure, rebase code, fix build issues, rebuild sign and release our whole repository. Not bad for a voluntary effort in addition to holding down 40h/week $dayjob. Being able to develop fixes in advance will for sure help us to reduce that time frame.
In general, it would make *my* life easier if elrepo drivers were tracking the RHEL kernel development and getting fixed along the way, because it's a pain when I have to deal with customers who use our stuff and can't upgrade because of broken drivers. That is the situation *now* with CentOS Linux.
If elrepo drivers are broken on CentOS Linux (not Stream), you should file a bug, either with elrepo if the driver also does not work on RHEL and we will fix it, or with CentOS if the driver works on RHEL but not on CentOS Linux and CentOS can fix it.
If by "CentOS Linux", you actually mean CentOS Stream, by definition that's never going to happen. Elrepo may use the Stream kernel to see what's coming and help develop fixes for the next release of RHEL (see comments above), or to contribute our fixes back upstream, but elrepo driver packages are built on RHEL build systems against RHEL kernels - we can not build against something that has not been released yet, and even if we could, those packages would not work for all our current RHEL (and CentOS Linux 8) users who are running 8.3 *now* and will continue to run 8.3 kernels for at least the next 4-5 months.
I mean "CentOS Linux" as in the RHEL rebuild. That's still a thing that exists for a while yet.
On Mon, Dec 28, 2020 at 6:43 AM Phil Perry pperry@elrepo.org wrote:
On 28/12/2020 08:02, Alexander Bokovoy wrote:
On su, 27 joulu 2020, Gordon Messmer wrote:
On 12/27/20 4:31 PM, Ljubomir Ljubojevic wrote:
On 12/27/20 5:00 PM, Alexander Bokovoy wrote:
I can see two major ways of enabling 3rd-party drivers: Â - using elrepo's excellent work on kmods to build a CI for CentOS Â Â Stream kernel updates and perhaps block updates based on CI results' Â Â consensus (not necessary blocked until it 100% green)
Phil Perry from ElRepo project wrote extensively both on [CentOS] and [CentOS-devel] lists why that aproach is not viable.
I might be misreading one party or another, but I believe that what is being proposed is better integration of ELRepo's kmod packages and CentOS Stream, so that kernel updates aren't published until the modules build. And if it is, then I'm not sure why that approach would not be viable.
Neal Gompa wrote: "The correct fix here is to start blocking RHEL kernel updates against third-party Free Software kernel module packages to ensure compatibility isn't broken and the kernel ABI stops breaking on every kernel version series. The reason it keeps breaking is because there's no current mechanism in which these are tested together to validate them for release."
That seems like an ideal solution.
You are getting it right, Gordon, thank you for connecting the threads.
So if I understand correctly, the proposal is that if a 3rd party (elrepo or OEM) driver does not build against a Stream kernel, you would not release the kernel? That would mean to date there would not be any Stream kernel updates released[1] because _something_ breaks against *all* of them? Or have I misunderstood what is being proposed? That would be a real shame for those folks who would like to test new functionality and/or participate and feed back in the development process.
In my proposal, they would be blocked from being released to users in the release process unless there's an acknowledgement to release it anyway by all parties involved (kernel devs, kmod maintainer blocking kernel release, etc.). They would still exist and be accessible in something like an updates-testing repo, but they wouldn't be merged into stable release until a solution is agreed upon.
And when the above situation arises (an elrepo or OEM driver fails to build and/or weak-link against a Stream kernel), what would be the process to fix that? Elrepo drivers are developed against and built for RHEL kernels, so when they fail to build for Stream, that won't get fixed for upto 6 months until the next RHEL point release. Or to put it another way, elrepo is a downstream project, downstream of RHEL.
If the assumption is that the kernel module package shouldn't break, then the kernel would be fixed. If not, then the kernel module driver gets rebuilt and released as a new build for that situation. That provides a consistent upgrade path and keeps people from having to hold back kernels.
The idea isn't to keep people from getting new stuff, but to ensure that everyone is able to keep up reasonably well.
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
You certainly control when you can and want upgrade your deployment systems. It has nothing to do with the cadence of updates coming into a distribution.
I find this fixation on the kernel updates is skewing things a lot. Kernel, certainly, is important, but it is not the thing that is RHEL or CentOS distribution, alone.
It is crucial issue if you install any kernel module not provided by Red Hat (3rd party drivers). If some software that you might or might not use brakes, you can mess around your working system and fix it. But if after dnf update your system crashes or network is down, and it is bare metal system, they you are f**ked, you need to reach the system manually (I install on regular PC hardware without ILO) and reverse to prior kernel. Even if I am quick about it, it will be very embarrassing for me in front of my clients (small in number as they are) whose work will stop for that period, so I will not be caught dead using CentOS Stream, I do not need the potential headache, embarrassment.
That's pretty obvious with any system, really, no need to repeat that. I think this topic was raised multiple times (by you and others already) to realize that. In an ideal collaborative world, perhaps, those 3rd-party drivers could be build and tested automatically on top of the CentOS Stream, though we are yet to reach that point of collaboration.
Seams it IS needed since you think it is not that important and I think it is crucial. When you dismissed it so easily as irrelevant, I thought you were not informed.
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba + KVM ?
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
What will happen to your system when/if there is new kernel change every few days? How much "punishment" can your system handle safely?
You certainly control when you can and want upgrade your deployment systems. It has nothing to do with the cadence of updates coming into a distribution.
I find this fixation on the kernel updates is skewing things a lot. Kernel, certainly, is important, but it is not the thing that is RHEL or CentOS distribution, alone.
It is crucial issue if you install any kernel module not provided by Red Hat (3rd party drivers). If some software that you might or might not use brakes, you can mess around your working system and fix it. But if after dnf update your system crashes or network is down, and it is bare metal system, they you are f**ked, you need to reach the system manually (I install on regular PC hardware without ILO) and reverse to prior kernel. Even if I am quick about it, it will be very embarrassing for me in front of my clients (small in number as they are) whose work will stop for that period, so I will not be caught dead using CentOS Stream, I do not need the potential headache, embarrassment.
That's pretty obvious with any system, really, no need to repeat that. I think this topic was raised multiple times (by you and others already) to realize that. In an ideal collaborative world, perhaps, those 3rd-party drivers could be build and tested automatically on top of the CentOS Stream, though we are yet to reach that point of collaboration.
Seams it IS needed since you think it is not that important and I think it is crucial. When you dismissed it so easily as irrelevant, I thought you were not informed.
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba + KVM ?
This is a question for centos-questions@redhat.com not for this list.
-- Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe
StarOS, Mikrotik and CentOS/RHEL/Linux consultant _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/28/20 1:50 AM, Julien Pivotto wrote:
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba + KVM ?
This is a question for centos-questions@redhat.com not for this list.
It was not me but Red Hat employee who started suggesting outlandish ideas like Red Hat giving away hundreds of thousands or millions of no-cost subscriptions for small businesses :-) I was just being sarcastic.
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/28/20 1:50 AM, Julien Pivotto wrote:
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba + KVM ?
This is a question for centos-questions@redhat.com not for this list.
It was not me but Red Hat employee who started suggesting outlandish ideas like Red Hat giving away hundreds of thousands or millions of no-cost subscriptions for small businesses :-) I was just being sarcastic.
You seem to have a very peculiar way of reading what was said. Please re-read Chris Wright's blog post if you need more help with that.
On 12/28/20 8:35 AM, Alexander Bokovoy wrote:
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/28/20 1:50 AM, Julien Pivotto wrote:
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba
- KVM ?
This is a question for centos-questions@redhat.com not for this list.
It was not me but Red Hat employee who started suggesting outlandish ideas like Red Hat giving away hundreds of thousands or millions of no-cost subscriptions for small businesses :-) I was just being sarcastic.
You seem to have a very peculiar way of reading what was said. Please re-read Chris Wright's blog post if you need more help with that.
You have used phrases "we've been *told*"(buy people who reserve right to change their mind without notice), "for *many* existing" (many != all), then wrote that just getting RHEL subscription "would address the needs of consumers of 3rd-party drivers too". One could suggest you have peculiar way of writing nonsense and use semantics to muddle the discussion.
After 900+ mails I concluded RHEL will not provide free subscriptions if you earn from your server, once-free clone "CentOS Linux 8"was against Red Hat business model because it does not financially contribute anything concrete and, CentOS Stream needs serious changes to address 3rd party driver issue (and no downgrade option) and no amount of semantics will confuse me to start doubting those facts.
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/28/20 8:35 AM, Alexander Bokovoy wrote:
On ma, 28 joulu 2020, Ljubomir Ljubojevic wrote:
On 12/28/20 1:50 AM, Julien Pivotto wrote:
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba
- KVM ?
This is a question for centos-questions@redhat.com not for this list.
It was not me but Red Hat employee who started suggesting outlandish ideas like Red Hat giving away hundreds of thousands or millions of no-cost subscriptions for small businesses :-) I was just being sarcastic.
You seem to have a very peculiar way of reading what was said. Please re-read Chris Wright's blog post if you need more help with that.
You have used phrases "we've been *told*"(buy people who reserve right to change their mind without notice), "for *many* existing" (many != all), then wrote that just getting RHEL subscription "would address the needs of consumers of 3rd-party drivers too". One could suggest you have peculiar way of writing nonsense and use semantics to muddle the discussion.
Since those 3rd-party drivers were built against RHEL kernel, using RHEL bits would address the need, by definition. Neither of us knows whether there will be a specific answer in those new subscription programs but the technical answer is there, definitely.
I hope a subscription solution would be there too. So far, we (and I'll keep saying 'we' here) have the statements already made by the Red Hat representatives who can talk about the programs being developed. I don't myself have a reason to doubt those. Yes, I think a fair amount of trust was lost in the way how things were handled by everyone. However, the trust is something that goes in both directions. Putting others' words in my mouth is not something I'd find engaging.
Whether that is confusing to you is up to you. I am trying to find a way to address the technical problem within the CentOS Stream. After all, this is development mailing list.
On 28.12.2020 07:50, Julien Pivotto wrote:
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba +
KVM ?
This is a question for centos-questions@redhat.com not for this list.
Methinks, the question was a rhetoric one.
By the way, if anyone asks that question to centos-question@ *and* gets a response - please share the knowledge.
On Monday, December 28, 2020 4:04 AM, Konstantin Boyandin via CentOS-devel centos-devel@centos.org wrote:
On 28.12.2020 07:50, Julien Pivotto wrote:
On 28 Dec 01:24, Ljubomir Ljubojevic wrote:
On 12/27/20 3:48 PM, Alexander Bokovoy wrote:
Following your approach to a detailed information about the Stream, we've been told there are various RHEL subscription programs coming next year that would address use of RHEL for many existing CentOS users. Perhaps, those programs would address the needs of consumers of 3rd-party drivers too, before we'd reach the collaboration ideal I outlined above. Let's see how that goes.
So you think Red Hat will offer no-cost subscription to a small 5-10 employee company, not in any way related to education or non-profits, that needs 1 CPU / 16-32GB RAM Linux server for mdadm RAID10 + Samba +
KVM ?
This is a question for centos-questions@redhat.com not for this list.
Methinks, the question was a rhetoric one.
By the way, if anyone asks that question to centos-question@ and gets a response - please share the knowledge.
That still breaks up the chain of custody. Given Mike McGrath's call to action, it is important the answers be made publicly directly from a named employee of Red Hat.
It isn't clear if responses from centos-questions will come from a person with a real name or come from a generic "centos-questions" account.
Also, even if answers are copied to us verbatim, it isn't clear if Red Hat will later claim the person posting it paraphrased the answer or took it out of context.
The 2014 statement that there would remain a firewall between CentOS and RHEL was to show the joining would operate in good faith.
If you read Mike McGrath's statements carefully, it becomes clear Red Hat had no intention of honoring the foundational promises made to show good faith. It was simply just something to go "out of date." Therefore, Red Hat has no intention of openly demonstrating that the governance meeting was consistent with those foundational promises.
We need to be thinking like lawyers for this point forward.
An example of thinking like a lawyer, if a free clone is released calling itself White Hat then that is vague enough to cause brand confusion. Red Hat's trademark is not just on a single color or single item of clothing. It is a trademark on *ALL* colors and *ALL* items of clothing. Even "Socks Of Auqa" is not explicitly enough and will result in a strongly worded letter from a Red Hat lawyer.
We need to be demanding the same level that things be explicit. They need to come direct from a specific person. They need to clearly state what the expirtation date of their claims will be. When possible the statements need to be published on Red Hat letter head and notarized. And they need to be posted publicly direct from the source.
Based on what Mike McGrath has said, we aren't dealing with the Red Hat we knew and loved. We are dealing with a company that can silently expire any promise at any time just like any other enterprise. We are a long way now from having Bob Young and Marc Ewing to re-iterate that Red Hat is not like a normal enterprise and does not intend to be.
I'm calling Admiral Ackbar on the centos-questions email address.
On Wed, Dec 23, 2020 at 09:38:19PM +0000, Phil Perry wrote:
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.)
Why would you do that? The method we use in Enterprise Linux to deliver 3rd party out-of-tree drivers is the RHEL Driver Update
Because it's an out-of-tree but open source driver.
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
Right, Red Hat's gonna make business decisions about what goes into the RHEL kernel, and pulling in out-of-tree backports is pretty expensive. But this is exactly the kind of thing CentOS SIGs are empowered to do. See again the Virt/Xen SIG.
On Wednesday, December 23, 2020 6:31 PM, Matthew Miller mattdm@mattdm.org wrote:
On Wed, Dec 23, 2020 at 09:38:19PM +0000, Phil Perry wrote:
It seems like Wireguard might be a good example of something for an alternate kernel maintained by a SIG. (Like the Xen SIG does.) Why would you do that? The method we use in Enterprise Linux to deliver 3rd party out-of-tree drivers is the RHEL Driver Update
Because it's an out-of-tree but open source driver.
If Red Hat really wanted to fix this in (a) kernel, the solution would have been to accept the repeated upstream requests to backport the driver into the RHEL kernel, but that idea/request has been rejected.
Right, Red Hat's gonna make business decisions about what goes into the RHEL kernel, and pulling in out-of-tree backports is pretty expensive. But this is exactly the kind of thing CentOS SIGs are empowered to do. See again the Virt/Xen SIG.
Tools like CentOS SIG or DKMS can be useful in some cases to get the module built.
There is still a bigger picture.
What about when there is a bad interaction between WireGuard and a patch in the latest kernel for CentOS Stream? How do you roll back patches in a SRPM that has no explicit details on which patch the code modifications came from?
Please compare the content of a Fedora kernel SRPM with a CentOS Stream kernel SRPM. Then explain how it is justified for Karsten Wade to say this change is because Red Hat cares about the openness gap now.
Not only did Red Hat violate that RHEL team would not cross the firewall to interact with the CentOS by putting Brian "Bex" Exelbierd on the goverance board itself, we are also fed another lie that Red Hat cares about the openness gap. The content of the CentOS Stream kernel SRPMs is clear written proof that the openness gap is willfully by Red Hat's own design.
False pretenses was given during the joining in 2014 and again now. How is that not supposed to be upsetting?
On 12/22/20 11:57 PM, redbaronbrowser via CentOS-devel wrote:
To reassure the community there is an FAQ with the following:
“CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release”
This is a complete reversal on a fundamental concept stated earlier by CentOS. Previously the mission was to close the COMPATIBILITY gap by NOT “extending or enhancing packages or features.”
At this point, I think you should be more clear than you are being. CentOS Stream will be getting features intended for RHEL, earlier than RHEL. This creates a temporary forward compatibility gap. RHEL might not run software that was compiled on CentOS Stream, until those new features actually appear in a later RHEL release. This same compatibility gap exists between point releases of RHEL. If you build an application on RHEL 7.8, for example, it might not run on RHEL 7.7.
On the other hand, CentOS Stream will be backward compatible with RHEL. Anything that runs on RHEL should continue running on CentOS Stream.
But Red Hat is clever enough to rename “Rawhide” to “Stream” to make it sound better.
No, Stream is not Rawhide.
Well, Stream is not CentOS either given it doesn't strive for binary compatiblity which has been the defintion of CentOS.
Here is an example of how CentOS Stream has more in common with Rawhide than it does with CentOS:
Say CentOS 8 has a software release number 127 which has a bug. There was at some point a release number 127 with the same bug that was part of RHEL.
But say CentOS Stream has a software release number 127 with a bug. This may result in RHEL getting release number 128 and never getting release number 127.
Rawhide also has software release numbers that only appeared in Rawhide but never appear in RHEL. You want to say the name is different, that is fine. It still acts more like Rawhide.
In terms of being more clear, here is some key points:
The announcement of what would and would not change about the governance by CentOS joining Red Hat is here: https://forums.centos.org/viewtopic.php?t=44407
It says the board would remain public, open and inclusive. The new EOL date for CentOS 8 was made by a board that provided no transcript or chat log.
It says CentOS and the RHEL team would have a "firewall" between them. All communication between them would be only via SRPM. Brian "Bex" Exelbierd's inclusion on the board is a clear violation of that.
Karsten Wade justifies the actions of this invalid governance board by claiming seven times this is about the "openness gap." However, that gap was created by Red Hat policy. The RHEL team should have been able to make public RHEL builds ahead of time whenever they wanted to without needing CentOS to do it.
Even worse, despite Wade's claims of seeking to be "open," the Red Hat design of pre-applying the kernel patches into a tar-ball will continue. If Red Hat really wanted to close the openness gap, it should start with fixing this bizzar policy that makes it hard to track down exactly which patch introduces bugs.
On Wed, 23 Dec 2020 at 15:32, redbaronbrowser via CentOS-devel < centos-devel@centos.org> wrote:
It says CentOS and the RHEL team would have a "firewall" between them. All communication between them would be only via SRPM. Brian "Bex" Exelbierd's inclusion on the board is a clear violation of that.
Bex's position was to 'replace' Karsten's place on the board whose job was to do the same thing previously. Also your second problem is that all communication between them would only be via SRPM. 1. git.centos.org was the way EL7 was being used to communicate packages. 2. communication via git or srpm would have resulted in CentOS EL7 and EL8 never getting released because problems happen. Sometimes the chain of packages being pushed is in the wrong order of a package gets dropped. [This happened in EL4/EL5/EL6 also.. in those days it could be a multiple day/week trip of 'hey do you know who I can get to push foobaz-1.2.3-5? huh my last person is on vacation and won't be back til December.' After the acqui-hiring this was brought down quite a bit because the teams actually could communicate with each other directly.
So if there was a firewall.. it was full of holes by the time CentOS7 was getting made. I think the issue is that people have all had vastly different running assumptions of how CentOS was run versus how it has been since day one. [Warning flawed analogy ahead.] Everything ran pretty darn smoothly for what people cared so looking too far under the bonnet to see how the engine really worked was out of the question. Then when the car maker says it isn't going to make internal combustion any longer but move towards electric there are large hues and cries from people who think that it will make the car unworkable but couldn't actually have fixed their own cars for 20 years or more. [End flawed analogy]
On Wed, 23 Dec 2020 at 15:32, redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
It says CentOS and the RHEL team would have a "firewall" between them. All communication between them would be only via SRPM. Brian "Bex" Exelbierd's inclusion on the board is a clear violation of that.
Bex's position was to 'replace' Karsten's place on the board whose job was to do the same thing previously. Also your second problem is that all communication between them would only be via SRPM.
- git.centos.org was the way EL7 was being used to communicate packages.
- communication via git or srpm would have resulted in CentOS EL7 and EL8 never getting released because problems happen. Sometimes the chain of packages being pushed is in the wrong order of a package gets dropped. [This happened in EL4/EL5/EL6 also.. in those days it could be a multiple day/week trip of 'hey do you know who I can get to push foobaz-1.2.3-5? huh my last person is on vacation and won't be back til December.' After the acqui-hiring this was brought down quite a bit because the teams actually could communicate with each other directly.
So if there was a firewall.. it was full of holes by the time CentOS7 was getting made. I think the issue is that people have all had vastly different running assumptions of how CentOS was run versus how it has been since day one. [Warning flawed analogy ahead.] Everything ran pretty darn smoothly for what people cared so looking too far under the bonnet to see how the engine really worked was out of the question. Then when the car maker says it isn't going to make internal combustion any longer but move towards electric there are large hues and cries from people who think that it will make the car unworkable but couldn't actually have fixed their own cars for 20 years or more. [End flawed analogy]
Give an inch, take a mile?
It is understandable that problems came up resulting in soft-fails of the firewall.
It is one thing for a CentOS member to intitate a request to the RHEL team to address a problem impacting the CentOS community. In the best of all worlds this communication was documented in the open via a public bugzilla issue. You probably will be able to give examples in which the soft-fails was not documented in an open/public way. Maybe it is because Red Hat is unresponsive to issues filed in bugzilla. Having to bypass bugzilla to get things addressed in a more realistic time frame would be another thing Red Hat really should fix as well then. However, I would also be willing to bet that you could give details on why those soft-fails still honored the spirit of the firewall for governance.
Taking any member of the RHEL team and putting them in a seat on the CentOS goverance board is a *HARD* fail of the firewall. This is the point in which the letter and spirit of CentOS goverance was shredded and thrown away.
Having the RHEL member which decides the RHEL roadmap part of the goverance to remove 8 years of commitment on CentOS 8 is crossing a line which was indicated would never be crossed.
When Karsten Wade was notified a member of the RHEL team would be taking his place, he should have felt an obligation to object. Instead he re-affired it with a blog post that it is balancing the needs around CentOS. The primary goal of the head of the RHEL roadmap should be to balance the needs of RHEL. No one should be trying to claim that Bex was beig impartial and giving 100% to his job as well.
Put another way, it should be troubling if someone said Doctor Jack Kevorkian was put on a committee to decide organ transplants and some of his own patients would be the donors. And if someone was told a committee that included Kevorkian decided someone should only live 1 more year instead of an expected 8 additional years, it should be understandable why that is upsetting.
It is also extremely upsetting Karsten Wade decides to redress this as the best way to address the openness gap.
Yes, RHEL's private rawhide created an openness gap. Some of us requested it be public. We were told Fedora's public rawhide was good enough for the majority of users. Much like how Karsten Wade now says this change is good enough for 95% of users. RHEL was able to justify the openness gap. Fixing it should be up to them without CentOS and not used to justify putting Bex on the CentOS governance board.
In terms of balancing the needs of the CentOS platform and the openness gap, the CentOS governance board should be focus on if the CentOS Stream kernel SRPM should be of the same quality as Fedora's kernel SRPM. Or if pre-applying patches in a non-open way is acceptable. Bex has a clear conflict of interest in this area. RHEL's roadmap has been that patches remain in a pre-applied tar-ball. Having a board that puts the CentOS community first can not involve him.
On 12/24/20 12:01 PM, redbaronbrowser via CentOS-devel wrote:
In terms of balancing the needs of the CentOS platform and the openness gap, the CentOS governance board should be focus on if the CentOS Stream kernel SRPM should be of the same quality as Fedora's kernel SRPM. Or if pre-applying patches in a non-open way is acceptable.
At that point, is the question you're asking whether or not the CentOS kernel should be a rebuild of an RHEL kernel SRPM? These don't seem like questions that the CentOS maintainers would have ever even accepted for consideration.
On 12/24/20 12:01 PM, redbaronbrowser via CentOS-devel wrote:
In terms of balancing the needs of the CentOS platform and the openness gap, the CentOS governance board should be focus on if the CentOS Stream kernel SRPM should be of the same quality as Fedora's kernel SRPM. Or if pre-applying patches in a non-open way is acceptable.
At that point, is the question you're asking whether or not the CentOS kernel should be a rebuild of an RHEL kernel SRPM? These don't seem like questions that the CentOS maintainers would have ever even accepted for consideration.
In normal times, I wouldn't think of suggesting this.
The year of 2020 is clearly not normal times.
CentOS Stream is a new age. We are the upstream now. We should be able to choose the kernel and the quality level of the SRPM.
I believe what makes something CentOS is the governance.
Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want.
This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response.
Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness.
Let's also fix the availability gap. Karsten Wade vision for CentOS Stream is that 95% is good enough. For every 1 million users there are 50,000 that have their needs fall through the cracks. I think as a community we can provide better results than that.
Both openness gap and availability gap are worthy things to fix so let's fix them. But Karsten Wade isn't offering an effective fix for those issues.
On Thu, Dec 24, 2020 at 3:59 PM redbaronbrowser via CentOS-devel < centos-devel@centos.org> wrote:
On 12/24/20 12:01 PM, redbaronbrowser via CentOS-devel wrote:
In terms of balancing the needs of the CentOS platform and the openness gap, the CentOS governance board should be focus on if the CentOS Stream kernel SRPM should be of the same quality as Fedora's kernel SRPM. Or if pre-applying patches in a non-open way is
acceptable.
At that point, is the question you're asking whether or not the CentOS kernel should be a rebuild of an RHEL kernel SRPM? These don't seem like questions that the CentOS maintainers would have ever even accepted for consideration.
In normal times, I wouldn't think of suggesting this.
The year of 2020 is clearly not normal times.
CentOS Stream is a new age. We are the upstream now. We should be able to choose the kernel and the quality level of the SRPM.
I believe what makes something CentOS is the governance.
Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want.
This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response.
Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness.
Let's also fix the availability gap. Karsten Wade vision for CentOS Stream is that 95% is good enough. For every 1 million users there are 50,000 that have their needs fall through the cracks. I think as a community we can provide better results than that.
Both openness gap and availability gap are worthy things to fix so let's fix them. But Karsten Wade isn't offering an effective fix for those issues.
I agree with you the governance model needs work. You've called out Karsten a couple of times but it's not clear to me what his role is going to be for CentOS Stream going forward. That is to say, there are others also involved in CentOS so you may not want to single him out.
There are a few boundaries I know Red Hat would like to keep in place.
1) The mainline branch *is* RHEL and so anything that gets committed there must be merged by a Red Hatter.
2) No attempting to do another downstream rebuild (for those that desperately want to contribute to this, there are already other communities for it and we won't be competing with them)
3) We want a robust SIG community, and that includes welcoming things that Red Hat isn't particularly interested in. So when we say no to something in the mainline branch, SIGs should be a fairly safe place to do that where they can use official build infrastructure and distribution mirrors. I've heard many people talk about special kernel needs, enabling older hardware, kabi, etc. I'd love to see that done in a SIG. I also personally think the bar for starting a SIG should be low.
I'm not in the governance game here but the question for you, and others, is this - What sort of governance model can we put in place to accomplish these goals as well as whatever common goals we have going forward? What are our common goals from here? I've seen many technical issues brought up on the list over the last two weeks that seem solvable to me.
-Mike
On 25.12.2020 08:05, Mike McGrath wrote:
On Thu, Dec 24, 2020 at 3:59 PM redbaronbrowser via CentOS-devel <centos-devel@centos.org mailto:centos-devel@centos.org> wrote: > At that point, is the question you're asking whether or not the CentOS > kernel should be a rebuild of an RHEL kernel SRPM? These don't
seem
> like questions that the CentOS maintainers would have ever even accepted > for consideration. In normal times, I wouldn't think of suggesting this. The year of 2020 is clearly not normal times. CentOS Stream is a new age. We are the upstream now. We should be able to choose the kernel and the quality level of the SRPM. I believe what makes something CentOS is the governance. Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want. This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response. Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness. Let's also fix the availability gap. Karsten Wade vision for CentOS Stream is that 95% is good enough. For every 1 million users there are 50,000 that have their needs fall through the cracks. I think as a community we can provide better results than that. Both openness gap and availability gap are worthy things to fix so let's fix them. But Karsten Wade isn't offering an effective fix for those issues.I agree with you the governance model needs work. You've called out Karsten a couple of times but it's not clear to me what his role is going to be for CentOS Stream going forward. That is to say, there are others also involved in CentOS so you may not want to single him out.
There are a few boundaries I know Red Hat would like to keep in place.
- The mainline branch *is* RHEL and so anything that gets committed
there must be merged by a Red Hatter.
- No attempting to do another downstream rebuild (for those that
desperately want to contribute to this, there are already other communities for it and we won't be competing with them)
- We want a robust SIG community, and that includes welcoming things
that Red Hat isn't particularly interested in. So when we say no to something in the mainline branch, SIGs should be a fairly safe place to do that where they can use official build infrastructure and distribution mirrors. I've heard many people talk about special kernel needs, enabling older hardware, kabi, etc. I'd love to see that done in a SIG. I also personally think the bar for starting a SIG should be low.
An interesting question here. CentOS Linux was free to use. Can you explicitly tell whether these SIGs should be related to paid-only services of RH?
Or perhaps I can't see how that can be possible with CentOS Stream.
I'm not in the governance game here but the question for you, and others, is this - What sort of governance model can we put in place to accomplish these goals as well as whatever common goals we have going forward? What are our common goals from here? I've seen many technical
issues brought up on the list over the last two weeks that seem solvable to me.
I am sure there will be answers.
But, as far as I see, these questions should have been asked *before* the decision to bury CentOS Linux alive was taken and announced.
(so much for openness and good communication, eh?)
On Thu, Dec 24, 2020 at 7:49 PM Konstantin Boyandin via CentOS-devel < centos-devel@centos.org> wrote:
On 25.12.2020 08:05, Mike McGrath wrote:
On Thu, Dec 24, 2020 at 3:59 PM redbaronbrowser via CentOS-devel <centos-devel@centos.org mailto:centos-devel@centos.org> wrote: > At that point, is the question you're asking whether or not the CentOS > kernel should be a rebuild of an RHEL kernel SRPM? These don't
seem
> like questions that the CentOS maintainers would have ever even accepted > for consideration. In normal times, I wouldn't think of suggesting this. The year of 2020 is clearly not normal times. CentOS Stream is a new age. We are the upstream now. We should be able to choose the kernel and the quality level of the SRPM. I believe what makes something CentOS is the governance. Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want. This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response. Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness. Let's also fix the availability gap. Karsten Wade vision for CentOS Stream is that 95% is good enough. For every 1 million users there are 50,000 that have their needs fall through the cracks. I think as a community we can provide better results than that. Both openness gap and availability gap are worthy things to fix so let's fix them. But Karsten Wade isn't offering an effective fix for those issues.I agree with you the governance model needs work. You've called out Karsten a couple of times but it's not clear to me what his role is going to be for CentOS Stream going forward. That is to say, there are others also involved in CentOS so you may not want to single him out.
There are a few boundaries I know Red Hat would like to keep in place.
- The mainline branch *is* RHEL and so anything that gets committed
there must be merged by a Red Hatter.
- No attempting to do another downstream rebuild (for those that
desperately want to contribute to this, there are already other communities for it and we won't be competing with them)
- We want a robust SIG community, and that includes welcoming things
that Red Hat isn't particularly interested in. So when we say no to something in the mainline branch, SIGs should be a fairly safe place to do that where they can use official build infrastructure and distribution mirrors. I've heard many people talk about special kernel needs, enabling older hardware, kabi, etc. I'd love to see that done in a SIG. I also personally think the bar for starting a SIG should be low.
An interesting question here. CentOS Linux was free to use. Can you explicitly tell whether these SIGs should be related to paid-only services of RH?
Or perhaps I can't see how that can be possible with CentOS Stream.
I'm going to be cautious about not writing policy here, that's for others to do. I will say that we've discussed several straw-man scenarios and they all seemed to go ok to us at Red Hat. As people have actual requests let's work through them (though perhaps we should first figure out a process for doing that...)
Examples (not to be taken as law, again I'm not a policymaker in CentOS but can speak to some things that we've discussed over the last year as part of potential Stream scenarios):
We know Facebook is taking CentOS Stream right now, adding whatever magic they want to add, and are using it. If they wanted to do that in a CentOS SIG, and find others who want to do it with them. We're good with that. In fact, that sounds kind of cool to me.
Another one we discussed was forking of packages. I forget what the rules are in something like EPEL or even current CentOS SIGs. But if someone wants to maintain a fork or a new package that is of no value to Red Hat and is not legally encumbered in some way, I think they should be allowed to do that in CentOS Stream.
More to your point, if someone wants to build something on/for CentOS Stream, that is never intended for RHEL nor any of Red Hat's products. I don't think Red Hat would have a problem with that.
If someone wanted to do something in a CentOS SIG, that had nothing to do with even CentOS Stream, nor any of Red Hat's products, I'm not sure why they'd come to CentOS with that request, but even that would be worth at least discussing with someone. The board maybe? We should figure that out.
I'm not in the governance game here but the question for you, and others, is this - What sort of governance model can we put in place to accomplish these goals as well as whatever common goals we have going forward? What are our common goals from here? I've seen many technical
issues brought up on the list over the last two weeks that seem solvable to me.
I am sure there will be answers.
But, as far as I see, these questions should have been asked *before* the decision to bury CentOS Linux alive was taken and announced.
(so much for openness and good communication, eh?)
I hope I'm wrong, and would love someone in the community or in Red Hat to chime in here and correct me.... but I hope "what are our common goals" is a question that has been asked at least sometime in the last several years. If not, let's correct that now.
-Mike
-- Sincerely,
Konstantin Boyandin system administrator (ProWide Labs Ltd. - IPHost Network Monitor) _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, December 24, 2020 9:17 PM, Mike McGrath mmcgrath@redhat.com wrote:
On Thu, Dec 24, 2020 at 7:49 PM Konstantin Boyandin via CentOS-devel centos-devel@centos.org wrote:
On 25.12.2020 08:05, Mike McGrath wrote:
I'm not in the governance game here but the question for you, and others, is this - What sort of governance model can we put in place to accomplish these goals as well as whatever common goals we have going forward? What are our common goals from here? I've seen many technical
issues brought up on the list over the last two weeks that seem solvable to me.
I am sure there will be answers.
But, as far as I see, these questions should have been asked *before* the decision to bury CentOS Linux alive was taken and announced.
(so much for openness and good communication, eh?)
I hope I'm wrong, and would love someone in the community or in Red Hat to chime in here and correct me.... but I hope "what are our common goals" is a question that has been asked at least sometime in the last several years. If not, let's correct that now.
The common goals of the CentOS project and community for the last several years are explained in this document:
https://wiki.centos.org/FAQ/General#What_is_CentOS_Linux.3F
Look for question/answer #1 and #6.
On 12/24/20 8:49 PM, Konstantin Boyandin via CentOS-devel wrote:
I'm not in the governance game here but the question for you, and others, is this - What sort of governance model can we put in place to accomplish these goals as well as whatever common goals we have going forward? What are our common goals from here? I've seen many technical
issues brought up on the list over the last two weeks that seem solvable to me.
I am sure there will be answers.
But, as far as I see, these questions should have been asked *before* the decision to bury CentOS Linux alive was taken and announced.
(so much for openness and good communication, eh?)
I would encourage to you to read the threads around restructuring SIGs from early in 2020. (I'll search the archives for this in a moment, if you cannot find it.) There was participation from many of our existing SIGs about how the process could/should be improved, and some of those changes are already in place.
See also https://git.centos.org/centos/board/issues where there are a number of discussion items around updating/fixing/restructuring the governance documents to be more in line with both reality, and effective means of governing the community. We would appreciate your constructive contributions there.
The CentOS governance documents which were written in 2013-2014 were largely derived from the Apache governance documents, and that process needs to continue in order to craft governance that is more appropriate to the size and nature of the CentOS project. Stuff that works at a 300+ sub-project Foundation may be a bit cumbersome here. Community input on that process would be very welcome.
In particular, some things that the board has already discussed and approved are:
* Wider attendance at Board meetings, starting with SIG leadership being invited to the Jan 13th meeting. This was decided in October, and was derailed by the aforementioned Elephant. I hope to get that invitation sent out this week.
* Even wider attendance at board meetings, after we see how meetings go with the SIG leadership in attendance. While many directors wanted this to start immediately, we eventually decided on a phased approach.
* Lower bar to SIG creation and membership. This is complicated by two things: 1) No new SIGs have been proposed in a while, so we have nothing to test this on. 2) The noggin/AAA work that CPE is doing will make SIG ACL management easier, and so we're kinda waiting for that.
* Review of all existing governance docs, and the writing of a formal governance document, rather than a scattering of several web pages.
Each of these things needs community involvement, if you want your views to be represented in that process. And *my* job is to ensure that these conversations happen in public, rather than (checks notes ...) inside the greenhouse.
It's important to note that, yes, the Red Hat Liaison has (more accurately: can have in certain circumstances) a louder voice than most other participants. This is something that needs to be stated clearly and kept in mind, however much it may annoy some folks. Red Hat is paying the bills and so has a louder voice.
So, yeah, we (and, here, I would point largely to Karsten) have been actively working and discussing governance issues for the past 2 years. Should we have been more open about this? Sure, I think the answer to that question is *always* yes. Indeed, I tend to annoy people with my pushes for transparency around everything, but, of course, it's a balance in a project, like CentOS, where it's largely controlled by a single company. This, in turn, is why I see Stream as such a positive step - it gives the reins, at least a little more, to you, the community.
That said, I understand and empathize with the frustration and anger from the community around this change. But what I have been saying consistently since that change is that, as annoyed and frustrated as we all are, it is critical that we acknowledge that the change has happened, and figure out What's Next. I hope that everyone who wants to stay around to help figure that out is able to do this without blaming individual board members, calling for resignations, and other unhelpful personal attacks.
On Monday, January 4, 2021 8:17 AM, Rich Bowen rbowen@redhat.com wrote:
I think I need to revisit this response at this point.
It's important to note that, yes, the Red Hat Liaison has (more accurately: can have in certain circumstances) a louder voice than most other participants. This is something that needs to be stated clearly and kept in mind, however much it may annoy some folks. Red Hat is paying the bills and so has a louder voice.
Some of the people paying the bills are the ones paying an additional $1,200 per Red Hat OpenStack hypervisor per year to get licenses to run RHEL guests.
Do they get a louder voice for paying the bills?
Or is the "elephant" that Red Hat now endorses three other public clouds of which *NONE* are running Red Hat OpenStack as the method people should get their free RHEL licenses?
After buying a product from Red Hat stated to be for providing public and hybrid clouds, is Red Hat providing them a fair way to compete? Or is Red Hat gaming the system to undermine their own customers--the same ones that have already paid the bills?
I understand Red Hat may have additional announcements. But the damage is already done. The message is clear that Red Hat endorses three major clouds and those that run Red Hat OpenStack aren't worth endorsing.
Did the people paying the bills really get a louder voice?
So, yeah, we (and, here, I would point largely to Karsten) have been actively working and discussing governance issues for the past 2 years. Should we have been more open about this? Sure, I think the answer to that question is always yes. Indeed, I tend to annoy people with my pushes for transparency around everything, but, of course, it's a balance in a project, like CentOS, where it's largely controlled by a single company. This, in turn, is why I see Stream as such a positive step - it gives the reins, at least a little more, to you, the community.
What is the criteria for success for giving the reins that was discussed at the November 11th meeting?
How was it determine that criteria for success would be achieved by December 31, 2021?
That said, I understand and empathize with the frustration and anger from the community around this change. But what I have been saying consistently since that change is that, as annoyed and frustrated as we all are, it is critical that we acknowledge that the change has happened, and figure out What's Next. I hope that everyone who wants to stay around to help figure that out is able to do this without blaming individual board members, calling for resignations, and other unhelpful personal attacks.
I never was attempting to make personal attacks. I am trying to figure out how we got to this point. Is there people that had a conflict of interest in deciding December 31, 2021 is a deadline by which Stream will reasonably replace CentOS 8?
Let me put this another way. Mark Shuttleworth has given interviews stating his product has never missed a release date. There is several bugs in Lauchpad that indicate major issues with the installer and other primary components were never addressed (including in LTS releases). Bugs that can be confirmed to still exist when the product never missed it's release date. Is there ever a point in which it stops being a personal attack to say someone that takes pride in that situation is toxic to the brand and the results?
To be clear, I am not saying anyone on the board is identical to that person. I am saying what Stream is has been misrepresented as part of claiming the board decision mades sense. And seems to be preceeding with no criteria for success when esablishing the date of completion.
Consider the following: "Does this mean that CentOS Stream is the RHEL BETA test platform now? A: No. CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release"
Does that indicate people should expect bugs that would fail established Fedora release criteria to be part of Stream?
Or Karsten Wade's labeling of an blog post on testing with "Stream can cover 95% (or so) of current user workloads" -- does that frame Stream in a way that people should expect bugs in which a Fedora Go / No-Go meeting would push off the release?
We are 17% from November 11th to December 31, 2021. Have we made at least 17% progress toward the success criteria the governance board has for Stream?
You indicated you want this process to be open and transparent--help me make sense of what we have been told to expect from Stream and the results we are seeing today.
On Thu, Jan 21, 2021 at 12:16 PM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
On Monday, January 4, 2021 8:17 AM, Rich Bowen rbowen@redhat.com wrote:
I think I need to revisit this response at this point.
It's important to note that, yes, the Red Hat Liaison has (more accurately: can have in certain circumstances) a louder voice than most other participants. This is something that needs to be stated clearly and kept in mind, however much it may annoy some folks. Red Hat is paying the bills and so has a louder voice.Some of the people paying the bills are the ones paying an additional $1,200 per Red Hat OpenStack hypervisor per year to get licenses to run RHEL guests.
Do they get a louder voice for paying the bills?
Red Hat's louder voice is based on two different components. The primary form of this "louder voice" is that Red Hat employs a ton of engineers who contribute directly to the CentOS Project or the codebases that become elements of CentOS Project outputs. We speak loudly in the sense that we can choose what we do and what we don't do. In the same way committers in a project have a louder voice than other contributors as they decide what the project will and will not do. This is the way Open Source works.
The other aspect is that Red Hat has a veto over the project. We also have one in Fedora. It exists primarily as a liability control switch. Because the projects don't have separate legal entities we need to be able to protect the company. These vetos have never been used and have not been envisioned to be used for technical decisions.
Extending this metaphor, as you have, Red Hat customers do have a louder voice. As a customer you have the opportunity to influence Red Hat and therefore by extension influence the decision of what Red Hat employees are asked to do. This advantage is enjoyed by any enterprise which has their employees contribute to open source.
Or is the "elephant" that Red Hat now endorses three other public clouds of which *NONE* are running Red Hat OpenStack as the method people should get their free RHEL licenses?
I don't understand this statement. The announced programs are accessible in any cloud which supports cloud access. If someone has a cloud that doesn't support cloud access, have them talk to us at CentOS-questions@redhat.com. We can figure out what the challenges are and see if we can get them into the program.
After buying a product from Red Hat stated to be for providing public and hybrid clouds, is Red Hat providing them a fair way to compete? Or is Red Hat gaming the system to undermine their own customers--the same ones that have already paid the bills?
see above. Cloud access isn't a new program.
I understand Red Hat may have additional announcements. But the damage is already done. The message is clear that Red Hat endorses three major clouds and those that run Red Hat OpenStack aren't worth endorsing.
Did the people paying the bills really get a louder voice?
So, yeah, we (and, here, I would point largely to Karsten) have been actively working and discussing governance issues for the past 2 years. Should we have been more open about this? Sure, I think the answer to that question is always yes. Indeed, I tend to annoy people with my pushes for transparency around everything, but, of course, it's a balance in a project, like CentOS, where it's largely controlled by a single company. This, in turn, is why I see Stream as such a positive step - it gives the reins, at least a little more, to you, the community.What is the criteria for success for giving the reins that was discussed at the November 11th meeting?
How was it determine that criteria for success would be achieved by December 31, 2021?
That said, I understand and empathize with the frustration and anger from the community around this change. But what I have been saying consistently since that change is that, as annoyed and frustrated as we all are, it is critical that we acknowledge that the change has happened, and figure out What's Next. I hope that everyone who wants to stay around to help figure that out is able to do this without blaming individual board members, calling for resignations, and other unhelpful personal attacks.I never was attempting to make personal attacks. I am trying to figure out how we got to this point. Is there people that had a conflict of interest in deciding December 31, 2021 is a deadline by which Stream will reasonably replace CentOS 8?
The board didn't change to set this up. Additionally Red Hat has published how its employees are freed to act in open source projects. Those guidelines may help you understand the lack of conflicts of interest. Further, part of the reason I was brought in as the Red Hat Liaison was to create clarity where previously there had been some confusion. My appointment included an explicit statement that I will speak for Red Hat, and only for Red Hat, i.e. without a personal voice, to ensure that when Red Hat something to say people could know it was Red Hat. This provided freedom for Karsten, the previous Liaison to act personally, which he had spent a lot of time doing anyway, without it being perceived that his opinions were Red Hat's or dictated to him.
Let me put this another way. Mark Shuttleworth has given interviews stating his product has never missed a release date. There is several bugs in Lauchpad that indicate major issues with the installer and other primary components were never addressed (including in LTS releases). Bugs that can be confirmed to still exist when the product never missed it's release date. Is there ever a point in which it stops being a personal attack to say someone that takes pride in that situation is toxic to the brand and the results?
To be clear, I am not saying anyone on the board is identical to that person. I am saying what Stream is has been misrepresented as part of claiming the board decision mades sense. And seems to be preceeding with no criteria for success when esablishing the date of completion.
Consider the following: "Does this mean that CentOS Stream is the RHEL BETA test platform now? A: No. CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release"
Does that indicate people should expect bugs that would fail established Fedora release criteria to be part of Stream?
Or Karsten Wade's labeling of an blog post on testing with "Stream can cover 95% (or so) of current user workloads" -- does that frame Stream in a way that people should expect bugs in which a Fedora Go / No-Go meeting would push off the release?
Because CentOS Stream doesn't have releases, there is no ritual of go/no-go meetings or release criteria, per se. Instead we have acceptance testing built from the same tests used on RHEL releases to establish that the code isn't WIP or otherwise an experiment. There will always be bugs, that is the nature of software. We are saying that bugs in CentOS Stream will be the result of actual bugs, not because someone pushed something half-baked that we knew was broken.
regards,
bex
On 1/22/21 4:53 PM, Brian (bex) Exelbierd wrote:
On Thu, Jan 21, 2021 at 12:16 PM redbaronbrowser via CentOS-devel
Do they get a louder voice for paying the bills?
Red Hat's louder voice is based on two different components. The primary form of this "louder voice" is that Red Hat employs a ton of engineers who contribute directly to the CentOS Project or the codebases that become elements of CentOS Project outputs. We speak loudly in the sense that we can choose what we do and what we don't do. In the same way committers in a project have a louder voice than other contributors as they decide what the project will and will not do. This is the way Open Source works.
There is a silver lining here because at least few people offered to be active in producing CentOS, but were refused for various reasons. What I understood from those exchanges, main reason was CentOS dev team did not want anyone else to know how they are doing things, they wanted tight control. Even requests dev teams publish how they produce CentOS was refused. I never said anything because it did not affect me.
So this is a case of "secretive group" deciding and controlling entire project and naturally anyone else had no say in what is done beyond proposing something on the mailing list or forums and hoping group in control will accept their reasoning.
Again, as far as I got the no-cost distro I was fine with how it was run, but saying anyone else could have a say/voice is false.
On Fri, Jan 22, 2021 at 6:26 PM Ljubomir Ljubojevic centos@plnet.rs wrote:
On 1/22/21 4:53 PM, Brian (bex) Exelbierd wrote:
On Thu, Jan 21, 2021 at 12:16 PM redbaronbrowser via CentOS-devel
Do they get a louder voice for paying the bills?
Red Hat's louder voice is based on two different components. The primary form of this "louder voice" is that Red Hat employs a ton of engineers who contribute directly to the CentOS Project or the codebases that become elements of CentOS Project outputs. We speak loudly in the sense that we can choose what we do and what we don't do. In the same way committers in a project have a louder voice than other contributors as they decide what the project will and will not do. This is the way Open Source works.
There is a silver lining here because at least few people offered to be active in producing CentOS, but were refused for various reasons. What I understood from those exchanges, main reason was CentOS dev team did not want anyone else to know how they are doing things, they wanted tight control. Even requests dev teams publish how they produce CentOS was refused. I never said anything because it did not affect me.
Red Hat also didn't say anything here because it didn't affect it's needs, the same as you. We wanted a development platform for the SIG and other layered work. This is part of the cognitive dissonance we maintain with communities we sponsor. In general we leave a community alone to self-govern. We only act as a company when we need to protect the community and by extension the company from liabilities or when our direct contribution goals are impacted. In the first case we use positions like the Liaison to talk to the governing body and if required (and it NEVER has been so far) our veto. In the second case we show up and make our case in the various bodies that manage the code. You can see this mostly in CentOS SIGs or in Fedora because CentOS Linux was a rebuild and generally did not make or accept changes that weren't in the code RHEL had published.
Now that we are all working on CentOS Stream there are some competing interests in play when it comes to build systems, but I think that ultimately we will have more access and transparency. WARNING: I am not a distribution build engineer so I am speaking in generalities here.
Our build systems and software has, in some cases, not changed to keep up with modern development practices or needs. Does this make it bad or wrong? No. Does it offer room for improvement? Absolutely. Looking at some of the work individual contributors have done to align build processes and software across distributions and third-party repos, I think that it is exciting to be able to use CentOS Stream as a platform for thinking about our entire build process. This won't be fast or easy as we have to be very careful, but it should be possible in a way that CentOS couldn't before. This is possible explicitly because CentOS now is ahead of RHEL, not behind it. It is building tomorrow, not rebuilding yesterday.
The other interest though is around the actual act of making the distribution, the "turning of the crank." Here Red Hat has very specific security interests and we need to limit the ability to do specific build tasks to specific people. Having spoken to the CPE team, the engineering organization in Red Hat that is tasked with our infrastructure contribution to CentOS, I know they are looking at every option to open up what they can. One thing they have to do is to get new authentication and other practices in place, which CentOS has traditionally not had in this fashion, to allow the right access (again - I am speaking in generalities here and glossing over detail. Detail discussions are not going to be useful if I am participating :D). They will tell you that in internal meetings I am constantly harping on the need to get SIGs greater controls and build opportunities, for example. That internal meetings comment raises a great question around how to get more community participation. There is an infrastructure SIG spinning up to ensure that there is a forum for these conversations. I'd also love to see, if we can technically do it, a SIG focused on improving build systems, with an eye toward making the deliberate incremental change that lifts all of us (Fedora, CentOS, RHEL, EPEL. elrepo, etc.).
So this is a case of "secretive group" deciding and controlling entire project and naturally anyone else had no say in what is done beyond proposing something on the mailing list or forums and hoping group in control will accept their reasoning.
Again, as far as I got the no-cost distro I was fine with how it was run, but saying anyone else could have a say/voice is false.
AIUI, the way the CentOS Linux distribution is produced as a direct artifact, your statement is generally true. I suspect this was largely driven by the rebuild nature of CentOS Linux and the fact that the engineering the project did was almost exclusively in service rebuilding, not creating software. An opportunity we have, as a community, is to really embrace the full definition of contribution. If you look at Fedora, as an example, we have lots of contributors who have never written a line of code or packaged a piece of software. Their contribution is invaluable and important. CentOS can have that too, if it wants it. I hope that this will be another area where the community shows up and the project thrives. Innovation comes in many forms, let's go find the ones we want to tackle, whether code or non-code activities.
regards,
bex
On 1/25/21 3:30 PM, Brian (bex) Exelbierd wrote:
The other interest though is around the actual act of making the distribution, the "turning of the crank." Here Red Hat has very specific security interests and we need to limit the ability to do specific build tasks to specific people. Having spoken to the CPE team, the engineering organization in Red Hat that is tasked with our infrastructure contribution to CentOS, I know they are looking at every option to open up what they can. One thing they have to do is to get new authentication and other practices in place, which CentOS has traditionally not had in this fashion, to allow the right access (again - I am speaking in generalities here and glossing over detail. Detail discussions are not going to be useful if I am participating :D). They will tell you that in internal meetings I am constantly harping on the need to get SIGs greater controls and build opportunities, for example. That internal meetings comment raises a great question around how to get more community participation. There is an infrastructure SIG spinning up to ensure that there is a forum for these conversations. I'd also love to see, if we can technically do it, a SIG focused on improving build systems, with an eye toward making the deliberate incremental change that lifts all of us (Fedora, CentOS, RHEL, EPEL. elrepo, etc.).
I understand need for security, SolarWind source code injection debacle comes to mind....
On Friday, January 22, 2021 9:53 AM, Brian (bex) Exelbierd bexelbie@redhat.com wrote:
If someone has a cloud that doesn't support cloud access, have them talk to us at CentOS-questions@redhat.com. We can figure out what the challenges are and see if we can get them into the program.
I will look into that.
On Thu, Jan 21, 2021 at 12:16 PM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
Consider the following: "Does this mean that CentOS Stream is the RHEL BETA test platform now? A: No. CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release" Does that indicate people should expect bugs that would fail established Fedora release criteria to be part of Stream? Or Karsten Wade's labeling of an blog post on testing with "Stream can cover 95% (or so) of current user workloads" -- does that frame Stream in a way that people should expect bugs in which a Fedora Go / No-Go meeting would push off the release?
Because CentOS Stream doesn't have releases, there is no ritual of go/no-go meetings or release criteria, per se.
I wasn't expecting a ritual of go/no-go meetings. Based on the information given, I was expecting the testing for established criteria would have been automated.
Instead we have acceptance testing built from the same tests used on RHEL releases to establish that the code isn't WIP or otherwise an experiment.
I will agree with you that Stream is not alpha grade software.
There will always be bugs, that is the nature of software. We are saying that bugs in CentOS Stream will be the result of actual bugs, not because someone pushed something half-baked that we knew was broken.
When gnome-initial-setup fails, that is not an obscure bug that slipped past beta testing into production. That is the type of bug that is in a beta-grade product to be caught by the beta testers. And unlike the other bugs I have found, this one has already been confirmed by someone else on the mailing list.
If you want to claim people using CentOS as a desktop should take advantage of the 16 RHEL license offer, I will accept that claim.
What I am not willing to accept is that the FAQ is being honest about the state that Stream is in. You are only explaining distinctions between alpha-grade and beta-grade. You are not giving evidence this bug should be exhibited in something Red Hat attachs it's credibility to being non-beta production grade software. This by Red Hat standards should be considered *BETA*.
Just fix the FAQ to reflect the reality this bug highlights. Once Red Hat is willing to be honest with us, I will be willing to file bugs for the other regressions that exist in Stream.
On Thursday, December 24, 2020 7:05 PM, Mike McGrath mmcgrath@redhat.com wrote:
On Thu, Dec 24, 2020 at 3:59 PM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
On 12/24/20 12:01 PM, redbaronbrowser via CentOS-devel wrote:
In terms of balancing the needs of the CentOS platform and the openness gap, the CentOS governance board should be focus on if the CentOS Stream kernel SRPM should be of the same quality as Fedora's kernel SRPM. Or if pre-applying patches in a non-open way is acceptable.
At that point, is the question you're asking whether or not the CentOS kernel should be a rebuild of an RHEL kernel SRPM? These don't seem like questions that the CentOS maintainers would have ever even accepted for consideration.
In normal times, I wouldn't think of suggesting this.
The year of 2020 is clearly not normal times.
CentOS Stream is a new age. We are the upstream now. We should be able to choose the kernel and the quality level of the SRPM.
I believe what makes something CentOS is the governance.
Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want.
This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response.
Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness.
Let's also fix the availability gap. Karsten Wade vision for CentOS Stream is that 95% is good enough. For every 1 million users there are 50,000 that have their needs fall through the cracks. I think as a community we can provide better results than that.
Both openness gap and availability gap are worthy things to fix so let's fix them. But Karsten Wade isn't offering an effective fix for those issues.
I agree with you the governance model needs work. You've called out Karsten a couple of times but it's not clear to me what his role is going to be for CentOS Stream going forward. That is to say, there are others also involved in CentOS so you may not want to single him out.
It does not just need work. If I submit an RPM for inclusion in Fedora that violate policy, I don't get to say publish it anyways and we will work on the policy issue later. Nothing can proceed until the policies are honored.
In the case of terminating CentOS 8, governance policy requires the following be done to proceed:
(1) A transcript of the previous Governance Board meeting to be publish
(2) Brian "Bex" Exelbierd must resign from the Governing Board
(3) The issue of gutting CentOS 8 should be re-presented before a valid Governance Board meeting which is public, open and inclusive.
There are a few boundaries I know Red Hat would like to keep in place.
You can claim all the boundaries you want. If Red Hat is willing to lie and abuse it's ownership of the CentOS trademark to do whatever it wants then claims of boundaries stop having meaning. Instead we are being given transient guidelines in which Red Hat will self-impose or dismiss at anytime.
- The mainline branch *is* RHEL and so anything that gets committed there must be merged by a Red Hatter.
Yes, I already figured out for myself that Red Hat is recreating Windows Insider builds. We will continue to have Fedora as the fast ring. But RHEL now wants a slow ring as well. I agree there are advantages to having a RHEL Insider Slow Ring. However, I don't agree that it should involve CentOS at all. This definately can't justify putting Bex on the governance board in violation of stated CentOS policy.
- No attempting to do another downstream rebuild (for those that desperately want to contribute to this, there are already other communities for it and we won't be competing with them)
Then it stops being CentOS. If the people that used to work on CentOS now need to work on something else, then we probably can't stop Red Hat. But even if it involves some of the same people doesn't make it the same thing.
Some of the people that contributed to Python have also contributed to Nodejs. If I submit to Fedora an RPM called Python4 that uses the Nodejs source code, the RPM would be rejected as misleading. What Red Hat is doing is the same of releasing something under a misleading name. What makes CentOS is "[CentOS] retains an upstream."
There are plenty of other names Red Hat can use for Red Hat Insider Slow Ring. You can try recycling any of these names Picasso, Rembrandt, Colgate, Vanderblit, Biltmore or Baron. Or whatever you want.
If a valid CentOS governance board decides CentOS needs to die, then it needs to die. But please don't hand the name of the dead to an identity theft.
- We want a robust SIG community, and that includes welcoming things that Red Hat isn't particularly interested in. So when we say no to something in the mainline branch, SIGs should be a fairly safe place to do that where they can use official build infrastructure and distribution mirrors. I've heard many people talk about special kernel needs, enabling older hardware, kabi, etc. I'd love to see that done in a SIG. I also personally think the bar for starting a SIG should be low.
I'm not in the governance game here but the question for you, and others, is this - What sort of governance model can we put in place to accomplish these goals as well as whatever common goals we have going forward? What are our common goals from here? I've seen many technical issues brought up on the list over the last two weeks that seem solvable to me
We already have SIGs for RHEL Insider Fast Ring (Fedora) and a governance board for them (FESCo). They should be able to determine what SIGs should also provide builds for RHEL Insider Slow Ring.
When you say there are technical issues that are solvable, can you list what those are?
Are we ever going to close the openness gap for the kernel SRPM? Or are we expected to assist with troubleshooting RHEL Insider Slow Ring with our hands still tied behind our back because there is no explicit information what makes up each patch?
You have taken notice that I have brought up Karsten Wade multiple times. But you seem to have gloss over the reason why. I still have not gotten an answer as to if Wade's claim that Red Hat's desire to close the openness gap is genuine. Or is it a self-serving definition of openness gap that excluses fixing the kernel SRPM.
On 12/24/20 4:59 PM, redbaronbrowser via CentOS-devel wrote:
I believe what makes something CentOS is the governance.
Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want.
This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response.
Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness.
I'm hesitant to dignify these comments with a response, but after some thought, here we go.
First: I presume your comment about "fraudulent governance" refers to Bex's appointment as Red Hat Liaison? I'm not sure, since https://www.centos.org/about/governance/board-responsibilities/#red-hat-liai... defines the process by which a Liaison may be replaced/rotated, and that's the process that was used. If you're talking about something else, I would like to hear more.
I am the first to admit (actually, Karsten has been saying for a while) that the CentOS Governance documents are lacking. There is no central "constitution", but, rather, a handful of web pages and blog posts which define how CentOS Governance works.
This is largely due to two things:
One: CentOS has not ever been a contributor community, in the traditional "Four Opens" sense. And so there was little need to document How It Works.
Two: CentOS has always been a "do-ocracy" - folks just got down to the business of doing the work, rather than talking about it.
The move to Stream increases the urgency to more clearly, publicly, transparently document how CentOS governance works, and how to get involved in that process. This is a major task of mine in the coming months, and I welcome participation from anyone who has knowledge of effective (open source) governance.
Second: I take great exception to your ad hominem attacks on Karsten, and, elsewhere on this list, Bex. It's clear that you don't know either of these people, and have little or no insight into their motives or character.
The fact that some of us are the face of these decisions means that we get to take some of that flak. That's fair and expected. But your personal attacks are unwarranted, and I would ask that you stop. Yeah, I know that I'm responding to a week-old post, and that the conversation have become considerably more productive since then. But this needs to be called out.
On Monday, January 4, 2021 10:51 AM, Rich Bowen rbowen@redhat.com wrote:
On 12/24/20 4:59 PM, redbaronbrowser via CentOS-devel wrote:
I believe what makes something CentOS is the governance. Red Hat's behavior makes it clear they believe what make something CentOS is who owns the trademark. That they can lie about the governance rules to get whatever they want. This militant attitude on the part of Red Hat and the fraudulent governance board deserves an equally militant response. Time fix the openness gap for the kernel SRPM for real instead of blindly following Karsten Wade's empty posturing in the name of openness.
I'm hesitant to dignify these comments with a response, but after some thought, here we go.
First: I presume your comment about "fraudulent governance" refers to Bex's appointment as Red Hat Liaison? I'm not sure, since https://www.centos.org/about/governance/board-responsibilities/#red-hat-liai... defines the process by which a Liaison may be replaced/rotated, and that's the process that was used. If you're talking about something else, I would like to hear more.
You are jumping way back in the thread at this point.
The announcement that CentOS would join Red Hat is available here: https://forums.centos.org/viewtopic.php?t=44407
It included assurances as to what would not change.
One of the statements is: "The Red Hat Enterprise Linux to CentOS firewall will also remain. Members and contributors to the CentOS efforts are still isolated from the RHEL Groups inside Red Hat, with the only interface being srpm / source path tracking, no sooner than is considered released. In summary: we retain an upstream."
The "firewall" between CentOS and the RHEL team had gotten referenced several times since then.
So, having a member of the RHEL team sitting on the governance board to switch CentOS from a downstream to an upstream seem to fit exactly what we were told would not happen.
Another Red Hat employee got involved in the thread and pointed out that RH is just like every other enterprise and documents just silently expire. So, I am learning to understand Red Hat is an enterprise like "every other" and there is no long term commitments or core values.
I am the first to admit (actually, Karsten has been saying for a while) that the CentOS Governance documents are lacking. There is no central "constitution", but, rather, a handful of web pages and blog posts which define how CentOS Governance works.
This is largely due to two things:
One: CentOS has not ever been a contributor community, in the traditional "Four Opens" sense. And so there was little need to document How It Works.
Two: CentOS has always been a "do-ocracy" - folks just got down to the business of doing the work, rather than talking about it.
Sure, but how do I get down to the business of doing the work with kpatch with a vendor that obfuscates all the patches?
The excuse given over and over was the "firewall." CentOS and RHEL are two different things and one doesn't influence the policies of the other. Hence, we just had to accept RHEL's policy is the obfuscate and CentOS is firewalled from doing anything about it.
Yet the firewall can be dismissed at Red Hat's convenience. Stream's kernel still has all it's patches obfuscated. But a member of RHEL can interact with CentOS beyond the publicly stated interfaces. If we are no longer retaining an upstream but rather becoming the upstream, why is my request to get access to the individual kernel patches still silently rejected?
I honestly do not understand if closing the "openness gap" is just some sort of marketting jargon or something Red Hat really is serious about. But looking at the state of Stream's kernel srpm as it stands today, it feels like marketting jargon.
The move to Stream increases the urgency to more clearly, publicly, transparently document how CentOS governance works, and how to get involved in that process. This is a major task of mine in the coming months, and I welcome participation from anyone who has knowledge of effective (open source) governance.
Second: I take great exception to your ad hominem attacks on Karsten, and, elsewhere on this list, Bex. It's clear that you don't know either of these people, and have little or no insight into their motives or character.
The fact that some of us are the face of these decisions means that we get to take some of that flak. That's fair and expected. But your personal attacks are unwarranted, and I would ask that you stop. Yeah, I know that I'm responding to a week-old post, and that the conversation have become considerably more productive since then. But this needs to be called out.
Ok.
On Mon, Jan 04, 2021 at 11:51:03AM -0500, Rich Bowen wrote:
The move to Stream increases the urgency to more clearly, publicly, transparently document how CentOS governance works, and how to get involved in that process. This is a major task of mine in the coming months, and I welcome participation from anyone who has knowledge of effective (open source) governance.
As is I think apparent, I am very interested in helping where I can here, while hopefully not overstepping my place as someone with a vested interest in the sibling project. :)
-
Alexander Bokovoy -
Brendan Conoboy -
Brian (bex) Exelbierd -
Dan Seguin -
Gianluca Cecchi -
Gordon Messmer -
James Cassell -
Jean-Marc Liger -
Julien Pivotto -
Konstantin Boyandin -
Lamar Owen -
Laurențiu Păncescu -
Ljubomir Ljubojevic -
Mark Mielke -
Matthew Miller -
Mike McGrath -
Neal Gompa -
Nico Kadel-Garcia -
Phil Perry -
redbaronbrowser -
Rich Bowen -
Simon Matter -
Stephen John Smoogen -
Strahil Nikolov -
Tetsuo Handa