Karsten Wade has indicated the centos-devel mailing list is the place to talk about why he thinks things are going to turn out okay. He implied on the CentOS blog there is balancing of the needs of CentOS. Mostly the article is about the “openness gap” (which is brought up seven times).
He goes through a very selective stroll through history to present his point leaving out anything that doesn’t fit his agenda.
So, let’s go back to 2002. The most polished version of pre-RHEL Red Hat Linux, code named Valhalla, was released in March 2002. It was the result of years of refining the Red Hat packages in a repository called Rawhide and then a branch was released only when it passed quality assurance.
Around the same time, a branch of Valhalla was also released as Red Hat Enterprise Linux 2.
Then in September 2002, code name Psyche replaced Valhalla. Security updates for Valhalla ended unless you became a RHEL 2 customer.
Psyche was a problem for a couple reasons. While Red Hat pointed out that it had more features than RHEL, it wasn’t just the “good” features. Psyche contained several regressions and bugs that should have never passed quality assurance. Rather, it appeared whatever was the latest packages in Rawhide just got dumped as Psyche since it was just the free community version and RH QA was focused on just RHEL.
If I recall correctly, at the time Red Hat apologized for Psyche and tried to reassure the community the problems were not intentional. Red Hat 9 (Shrike) was released in March 2003 and the first Fedora Core released in November 2003.
The continued development under FC addressed several of the bugs. But two new issues came up.
First new issue was Red Hat’s creation of the openness gap. There were now two Rawhide repositories. The public Rawhide moved to Fedora. But also a Rawhide branch that RHEL work from was kept private.
Second new issue was a COMPATIBILITY gap. Fedora wasn’t designed to be 100% API or ABI compatible with RHEL. Several times a binary built on Fedora could not be run on RHEL 2. A statically linked binary might temporarily address the issue but then security updates to the RHEL 2 libraries would not fix issues with the statically linked binary. To build a dynamically linked binary for RHEL 2 without paying for RHEL 2 required using Valhalla which was no longer getting security updates.
In May 2004, a solution to the COMPATIBILITY gap was released. The Community ENTerprise OS. As stated in it’s FAQ:
“Does CentOS change the upstream Source RPMs?” “No. CentOS' key mandate for our base and updates repositories is NOT extending or enhancing packages or features beyond those supplied by the upstream Source RPM's. CentOS strives intentionally to provide binary functionality for our users.”
Now the community had two options. Fedora for features or CentOS for compatibility. It is important to remember, CentOS strived for COMPATIBILITY by *avoiding* additional features as this fundamental concept will come up later.
In October 2004, Ubuntu would be released. This would be a challenge for CentOS. While CentOS *strive* for being 100% compatible with the commercially supported RHEL, Ubuntu LTS community edition is able to prove it is 100% identical to Ubuntu LTS with commercial support.
Over time another openness gap was created. The Linux kernel is under the General Public License v2 which requires openness including notices of what files are modified and when. The Linux kernel development team accomplishes this with a public git repository. The majority of distributions (including Fedora) do this by providing the vanilla kernel source code and then each of the patches separately. RHEL/CentOS instead violated the spirit of the GPL v2 and the spirit of openness. CentOS users are given a kernel tar-ball with all the patches already applied and a changelog. The changelog does not indicate which files are modified per item but I am willing to guess that Red Hat legal has written off that they can defend this as compliant with the legal language of the GPL v2.
This violation of openness also has technical downsides. If a newer Fedora kernel breaks something, you can rebuild the kernel RPM with select patches unapplied until you locate which specific patch in the new kernel causes the bad behavior. But performing a roll-back of patches on CentOS was made difficult because by design RH provides no clear indication exactly what each patch changes.
Instead, CentOS users can open a bugzilla issue to try to get a Red Hat employee to trace down which individual patch caused the issue. But from what I can tell that usually doesn’t work out well. Maybe someone will take notice and ask for more information. However, from what I can tell, several times a project manager just closes the ticket with a statement a new version of RHEL has been released and to start back at square one with a new ticket (to be ignored again).
So, fast forward to 2013. Karsten Wade has stated at this point he led the Red Hat team to assimilate CentOS and took a seat at the CentOS Governing Board.
In 2014 the joining was announced which included this statement of reassurance to the CentOS community:
“The Red Hat Enterprise Linux to CentOS firewall will also remain. Members and contributors to the CentOS efforts are still isolated from the RHEL Groups inside Red Hat, with the only interface being srpm / source path tracking, no sooner than is considered released. In summary: we retain an upstream.”
Then fast forward to October 2018, the Linux kernel development team releases 4.19 as a long term support kernel.
In May 2019, RHEL 8 was released with a 4.18 kernel. RHEL doesn’t get the benefit of the community effort to support the LTS kernel for 6 years and the LTS kernel community doesn’t get the benefit of RHEL supporting a kernel for 10 years. Instead, RHEL continues a policy of releasing a monolithic tar-ball that obscures what each patch changes in violation of the spirit of openness (again, by design).
Then continuing on to 2020. CentOS 6 is reaching the end of life at the end of November with no plans to continue providing security updates. RHEL Extended Life Cycle subscribers will get some select security updates but under an openness gap by RH design none will be shared through CentOS. At the same time, it appears “state level actors” are leveraging security holes more than ever before.
CentOS users are given two options. First is to upgrade to CentOS 7 but then reach the end of life again in June 2024. Or skip from CentOS 6 to CentOS 8 which is stated to be supported until 2029.
Then Red Hat announces after CentOS 6 users have performed their upgrades that the real end of life that CentOS was committed to for CentOS 8 is December 2021. It is really CentOS 7 that has a longer commitment by 2.5 years.
To reassure the community there is an FAQ with the following:
“CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release”
This is a complete reversal on a fundamental concept stated earlier by CentOS. Previously the mission was to close the COMPATIBILITY gap by *NOT* “extending or enhancing packages or features.”
But Red Hat is clever enough to rename “Rawhide” to “Stream” to make it sound better. However, at the end of the day this is the same behavior and problems that gave us Red Hat Psyche, only now it is CentOS edition of Psyche that is coming.
Karsten Wade’s main sales pitch for this is addressing the openness gap. I agree that RHEL Rawhide should never have been made private. But based on the “firewall” established between CentOS and RHEL, it should be up to the RHEL team to make their Rawhide public and not involve CentOS at all. RH self-imposed openness gaps shouldn’t be used as the basis to violate the previous assurances provided to the community.
But it turns out Red Hat was not being honest about the “firewall” from the RHEL group. In fact, one of the members of the CentOS Governance Board includes a member of a RHEL group. Also, this is not any old RHEL group member, it is Brain “Bex” Exelbierd, the RHEL member that establishes the roadmap for RHEL.
WIth CentOS going from downstream provider to upstream provider, the roadmap for CentOS stops being focused on the community and comes instead from Bex for the benefit of RHEL. Exactly what CentOS users feared would happen in 2014 is now taking place.
And this is being done in the name of openness?
Where is the improved openness for CentOS users?
Where is an open transcript, chat log or voice recording of the CentOS Governance Board meeting that selected to kill CentOS 8 in favor of CentOS Rawhide?
Where is the openness in the statement there will be a firewall between CentOS and the RHEL group? Was Bex only interfacing at governance meetings via SRPMs? Was the firewall always a lie?
Where is the openness in CentOS’ commitment to support CentOS 8 to 2029?
Where is the openness in the CentOS kernel patches?
Where is the openness in having one CentOS FAQ indicating extending features beyond upstream is *NOT* part of CentOS and at the same time have another FAQ stating CentOS features will be coming ahead of RHEL?
The nicest way I can think of to put it regarding the assertion that this closes the openness gap as a benefit to CentOS users is at this point that I believe that Karsten Wade must have an intestinal tract full of copious amounts of matter.
The most insane part is this change should be bad for Red Hat itself in the long run.
CentOS should be helping attract market share for RHEL by encouraging developers to build/test on a RHEL compatible system and giving potential RHEL customers a try before you buy method.
But CentOS hasn’t been competing well as it is. Majority of AWS instances are running Ubuntu. Several Azure instances are running Ubuntu. There are several Docker Hub images that use Ubuntu as a base image and much fewer using CentOS/RHEL.
CentOS/RHEL is no longer the preferred runtime environment to develop for and where the developers go the users will eventually follow. Bringing back the COMPATIBILITY gap is not the solution to that. It is how to make things worse.
Several of the projects Karsten Wade selected to highlight seems to suggest that Red Hat sees itself as expanding into ESXi’s market share. That is a worthy goal but should not replace promoting RHEL as a preferred runtime environment. Having a RHCE should not be reduced to knowing the next generation hypervisor and modern day BIOS. The test requires knowing much more than that and RH should continue to provide maximum possible value to the people with a RH certification.
With CentOS Rawhide (or CentOS Stream if you prefer calling it that) replacing CentOS 8, the try before you buy will go to companies like Oracle and CloudLinux. But once someone resolves the COMPATIBILITY gap by going to either of those solutions for the community edition, chances are better they will go back to the same source for commercial support later as well. This is a roadmap for RH giving up more market share. Why? For claims of openness that ring empty?
It would be nice to hear from Paul Cormier why he would allow this type of behavior and insanity on the part of Red Hat but I doubt we will ever get that level of openness.