Hi folks,
I am not planning to pour gas over fire, a lot of traffic was generated due to the recent announcement however i thought i should circulate an article [1] i've bumped into today.
Please note, i'm not an RH employee nor i know the article's author, i'm just a consumer of CentOS and someone who did a bit of work on PaaS sig for OpenShift in my spare time with no employer sponsorship.
As the Xmas time is approaching it would be good to take a step back and analyse from a different perspective what RH did and still does (look at OKD 4.x and the mighty effort did by few passionate RH employees) before throwing rocks. (humanity is very good at judging and snapping in a blink of an eye).
Thanks
[1] http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-thi...
On 12/14/20 11:24 AM, Daniel Comnea wrote:
I am not planning to pour gas over fire, a lot of traffic was generated due to the recent announcement however i thought i should circulate an article [1] i've bumped into today.
... FWIW, I'm not mad. I'm just blindsided.
On 12/14/20 5:24 PM, Daniel Comnea wrote:
Hi folks,
I am not planning to pour gas over fire, a lot of traffic was generated due to the recent announcement however i thought i should circulate an article [1] i've bumped into today.
Please note, i'm not an RH employee nor i know the article's author, i'm just a consumer of CentOS and someone who did a bit of work on PaaS sig for OpenShift in my spare time with no employer sponsorship.
As the Xmas time is approaching it would be good to take a step back and analyse from a different perspective what RH did and still does (look at OKD 4.x and the mighty effort did by few passionate RH employees) before throwing rocks. (humanity is very good at judging and snapping in a blink of an eye).
Thanks
[1] http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-thi... http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-think-about/
This time I posted a larger comment with several issues, lets se if author green lights it (it is moderated).
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
********************
I have been active in CentOS community for 12 years, last 9 years I am main admin of official CentOS group on Facebook. When I tok over there was 300 members, now there is 26.800+ members. One part of this comment was originally written for a article on Medium where RH employee said "CentOS is NOT dead", and other is part of reply in CentOS mailing lists, so it might feel like it is not addressing you specifically. Sorry about that but this is Frankenstein's monster.
First thing I must notice, fatherlinux, is that your lack of using CentOS and communicating with CentOS community robs you of true understanding what "CentOS Linux" was, and what it meant. I used CentOS Linux to get a stable distro. I can trust without paying for it. I and small companies I installed CentOS for were NEVER gonna buy RHEL. Ever. We do not have enough money for it. So it was always gonna bee free Linux distro. In Serbia where I live Linux community is 80-90% Debian/Ubuntu. Between 2008-2015 I was almost alone in preaching benefits of RHEL/CentOS. I was almost ridiculed. It got so bad that I finally gave up trying to change their minds.
What your western mind does not realize is that USA is 330 million people, and Europe is another 800 million. There is money there. But India is 1,2+ billion people and China has 1,5 bn people. Entire Asia has 4bn people, and legal software and paying for licenses is very low due to general low income. But they are getting technologically better and to avoid Microsoft licenses many choose Linux. India even started teaching Linux in schools, but guess what, they use Ubuntu! Every child in India is learning how to use Ubuntu. As a admin on CentOS Facebook group, majority of newbies are from Asia, mostly from poorer countries as India, Pakistan, Thailand, Mianmar etc, Those that reach CentOS community via Facebook are true newbies. They barely know to use Facebook and majority does not know support forums and mailing list even exist on the internet. Bug report systems especially. Over the years I have built several pages of resource links, explanations, (safe) howto's, etc. I did everything to make newbies understand, appreciate and adopt CentOS and via that the RHEL way. I preached like a zealot. And zealots ARE NEEDED for RHEL, because it has many issues but only positive point was it was 99% clone of RHEL. When we explain them that there are no audio/video codecs due to legal restrictions, that kernel is backported and adding some module is not possible if they compile vanila kernel because it brakes the kABI, that they need to install centosplus kernel because driver they needed RHEL does not want to support, that CentOS tools for development are too old because of version freeze, that they need to install several 3rd party repositories to make their server work, they then asks us "but why would we use CentOS instead of Debian, Ubuntu, SUSE...?" And every single time the response was "Because Red Hat is great company that created great and very stable product and CentOS is almost total clone of RHEL, and when they learn how to manage CentOS they can go for RH Linux certification. And that is it, for them CentOS does not have any other competitive edge over other Linuxes beside "99% clone of RHEL". SELinux was more of the repelling point, vast majority would just turn it off when they read first Howto on the internet, and great efforts went to pleading with them to try to learn SELinux.
So the fact that there are 1.000 guys that do developing on CentOS does not mean that majority of CentOS users are "developers", on the contrary, vast majority of servers running CentOS, especially rented VM's are run/controlled by guys like me whose main job is not to manged Linux servers, that is only a side job and many only barely understand what they are doing. They installed the system, configured it via some tutorial, and left it running.
I am one of community members that was very vocal about death of "CentOS Linux". No one said that "CentOS" is dead, but "CentOS Linux". CentOS is Open Source Project, and "CentOS Linux" was product of that project, 99% binary compatible CLONE of RHEL. CentOS project would take RHEL source, debrand it and compile it so that CentOS is "bug-for-bug" clone of RHEL. There were many instancies where some package was not compiled against exact version of dependency package, so when CentOS dev tried to compile it they would not get same binary as RHEL's, so they would spend time locating which version of which dependency package RH devs used.
It was done for 14+ years so that "CentOS Linux" would be binary compatible with RHEL. This is important if you use some software that is only supported on RHEL, and on no other distro. Using CentOS, you were (resonably but not officially) assured that software running on CentOS works same on RHEL (and vice versa).
This binary compatibility is MAIN and for many ONLY reason why they used free-as-a-beer CentOS, some on test servers (save on 1 RHEL license for system that does not create money), and others in not-for-profit organizations and small companies (that can not afford $300+/year for RHEL license ) in actual production. Even Hosting companies offer "CentOS Linux" as option for installation, and they save a ton of money on RHEL licences, but transfer that save to cheeper "CentOS Linux" hosting. If no RHEL clone ever existed, all of the CentOS users above would not learn how to use RHEL/CentOS but would start and continue to use Debian or other Linux distributions for all those purposes.
So yes "CentOS Linux" IS DEAD, and instead of that product, RH, now owner of (once Open Source) CentOS trademark offered DIFFERENT PRODUCT, "CentOS Stream", something that was never meant to replace "CentOS Linux".
And here is main problem with "CentOS Steam": "CentOS Stream" is designed to offer a vast minority of CentOS users DEVELOPMENT platform on which they can prepare their software (that will run on RHEL) to support next RHEL minor release (like 8.1, 8.2, 8.3, 8.4, etc). Current RHEL minor version is 8.3, and "CentOS Stream" (I choose to call it "RHEL Stream") is set of packages that WILL BE RHEL 8.4 in next (up to) 6 months. Unlike other Open Souce projects (run by community via consensus), "CentOS Stream" devs will only be RH employees who might be persuaided to add something new, but that is not likely. Only part of "CentOS Stream" that will be trully Open Source (with imput from community members not employed by Red Hat will be "SiG"'s, Special Interest Groups that will develop their own software on top of RHEL's next minor release, I mean "CentOS Stream"). Kernels will be biggest problem because every time RH employee decides to work on new kernel, community from ElRepo that compiles driver modules (that RH does not WANT to support/spend resources on) will have to be recompiled. And Red Hat disables A LOT of drivers, specially for older hardware!
Considering that can be any day in a span of 6 months, any time CentOS Stream user that uses 3rd party drivers (does not run "CentOS Stream" on RHEL approved hardware!) updates kernel might lose network or even storage (drivers for unsupported RAID controler or network attached storage not officially supported by RHEL). Would you risk running your server on "CentOS Stream" if it means daily fear it will stop working and need physical access to it to troubleshoot problem (maybe it was not kernel this time but actual hardware issue?). Chris Wright's words: "To be exact, CentOS Stream is an upstream development platform for ecosystem developers. It will be updated several times a day. This is not a production operating system."
Who ever tries to minimize this issue needs to know following: Hundreds (all of them) of 3rd party drivers from ElRepo project that work on RHEL 8.2 STOPPED WORKING on RHEL 8.3, and ElRepo needed to recompile ALL OF THEM. Luckily, with "CentOS Linux" that follows RHEL minor releases this might happen only every 6 months and any driver reported to have issue is recompiled against new kernel and will work for another 6 months. Imagine such nightmare happening every time soem RH dev decides kernel need some changes?
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone. I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
****************************
On 12/14/20 10:06 PM, Ljubomir Ljubojevic wrote:
On 12/14/20 5:24 PM, Daniel Comnea wrote:
Hi folks,
I am not planning to pour gas over fire, a lot of traffic was generated due to the recent announcement however i thought i should circulate an article [1] i've bumped into today.
Please note, i'm not an RH employee nor i know the article's author, i'm just a consumer of CentOS and someone who did a bit of work on PaaS sig for OpenShift in my spare time with no employer sponsorship.
As the Xmas time is approaching it would be good to take a step back and analyse from a different perspective what RH did and still does (look at OKD 4.x and the mighty effort did by few passionate RH employees) before throwing rocks. (humanity is very good at judging and snapping in a blink of an eye).
Thanks
[1] http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-thi... http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-think-about/
This time I posted a larger comment with several issues, lets se if author green lights it (it is moderated).
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone. I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
On 12/15/20 5:23 PM, sankarshan wrote:
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I am guessing that you are refering to this part: "Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat."
I was very careful how I wrote that part. I know (electronically) members of CentOS team for over a decade, and they all did excellent job and in all these years I had nothing bad to say about them.
Even now I do not hate them, wish them ill or harm.
BUT, when they accepted that (I assume highly payed) job in Red Hat, in bulk with accepting financing from Red Hat, and who knows what demands from Red Hat management (no one except them is privy to those talks even though it concerned entire CentOS Community) they did so in such a way that Red Hat got effective control over entire CentOS project including trademarks and deciding vote and veto power. i was supportive back then, even argued with those who said it was betrayal, BUT I was NEVER ASKED before the decision was made if that should have been done. There was no public vote made or given time to opposition to make a case against such move. CentOS project belonged to those who control it (and poured A LOT of time and effort into keeping it afloat), and they made a business decision to cash in their expertise. And only reason that only few held that against them was a promise CentOS project will keep releasing free RHEL clone. Since it did not hurt the interest of community, good for them, they got stable jobs which they earned. It was a win-win, just lets not say they decide it from some altruism, that ever happened.
And now we are in this point in time where Red Hat used legal frame CentOS Borad from 2014 enabled, to blackmail RH employees currently serving as CentOS Board to hurt CentOS community. It was done without any warning to community, and Board members voted "for" even though they were initialy against it, so that means that other factors were at play beside their better judgment and wishes.
Only thing it could be, since RH employs them, is implied of perceived threat to their jobs inside RH if they anger RH higher ups by refusing to vote "for" and force RH to overrule them which would have been a true PR nightmare for Red Hat. I am even guessing that compromise was made for C7 to be supported until EOL because RH's weak story about not enough resources could not be applied to C7 which is in maintenance mode and does not have many packages that need rebuilding.
I am more stubborn then a mule, and I, personally, would not have accepted this ultimatum by RH, I would have rather resigned then be remembered as a person to vote "for" killing the main reason for existence of CentOS project. Damn the consequences.
From that standpoint I hope that they see the compensation for this
decision as enough, because there is no doubt they will be remembered as Board members that closed most popular RHEL clone for 15+ years. DOes not matter if history views it positively or negatively, IT/Linux history will remember their names and every few years someone will comment on their decision.
But I still do not judge them for doing what they think was the right decision, it is important that THEY view their conscience as clean, because I fear they will be a constant target for those disgruntled, in the years and decades to come. They chose to vote "fo"r, they did their math, argued between them selves, reached consensus and then voted unanimously, so they new what they are doing and calculated-in consequences.
On Tue, Dec 15, 2020 at 10:23 AM sankarshan < sankarshan.mukhopadhyay@gmail.com> wrote:
I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone. I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Note: I was not in the room when the voting happened. I was involved in the negotiations. The board had a tremendous impact on helping Red Hat better understand some things that needed to happen in CentOS Stream. For example, versioning it and supporting it through the full support cycle of RHEL instead of what stream was before (a sort of continuous stream with one year overlap for migrations, etc). The Board is expecting things out of CentOS Stream and we expect them to hold us to that.
It's easy to say "The Board is full of Red Hatters and they did this." But I think we all know that's not the case, some of the Red Hatters on the board were as fierce a defender of the existing CentOS community as one could possibly be. The board could have voted this down. Red Hat could have dissolved the board (as I understand the voting rules). But that didn't happen. Both sides of this came to an agreement that we - together - could live with and that represented a positive future for CentOS. A very very different future for sure, but a positive one.
-Mike
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 10:23 AM sankarshan < sankarshan.mukhopadhyay@gmail.com> wrote:
I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone. I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us.
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
Well, you all see the reaction this has garnered around the world, and it's all negative except for the Red Hat employees trying to convince us it's a good thing. Nice try.
We all know differently. And we are all now making influential choices that will hurt Red Hat.
Good job!
I'm not going to say that the announcement was the board's idea or even
that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Note: I was not in the room when the voting happened. I was involved in the negotiations. The board had a tremendous impact on helping Red Hat better understand some things that needed to happen in CentOS Stream. For example, versioning it and supporting it through the full support cycle of RHEL instead of what stream was before (a sort of continuous stream with one year overlap for migrations, etc). The Board is expecting things out of CentOS Stream and we expect them to hold us to that.
It's easy to say "The Board is full of Red Hatters and they did this." But I think we all know that's not the case, some of the Red Hatters on the board were as fierce a defender of the existing CentOS community as one could possibly be. The board could have voted this down. Red Hat could have dissolved the board (as I understand the voting rules). But that didn't happen. Both sides of this came to an agreement that we - together
- could live with and that represented a positive future for CentOS. A
very very different future for sure, but a positive one.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
s/RedHat/IBM/g
On 15 Dec 2020, at 18:29, Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 10:23 AM sankarshan sankarshan.mukhopadhyay@gmail.com wrote: I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone. I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us.
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
Well, you all see the reaction this has garnered around the world, and it's all negative except for the Red Hat employees trying to convince us it's a good thing. Nice try.
We all know differently. And we are all now making influential choices that will hurt Red Hat.
Good job!
I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Note: I was not in the room when the voting happened. I was involved in the negotiations. The board had a tremendous impact on helping Red Hat better understand some things that needed to happen in CentOS Stream. For example, versioning it and supporting it through the full support cycle of RHEL instead of what stream was before (a sort of continuous stream with one year overlap for migrations, etc). The Board is expecting things out of CentOS Stream and we expect them to hold us to that.
It's easy to say "The Board is full of Red Hatters and they did this." But I think we all know that's not the case, some of the Red Hatters on the board were as fierce a defender of the existing CentOS community as one could possibly be. The board could have voted this down. Red Hat could have dissolved the board (as I understand the voting rules). But that didn't happen. Both sides of this came to an agreement that we - together - could live with and that represented a positive future for CentOS. A very very different future for sure, but a positive one.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
--
Matt Phelps Information Technology Specialist, Systems Administrator (Computation Facility, Smithsonian Astrophysical Observatory) Center for Astrophysics | Harvard & Smithsonian
60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps@cfa.harvard.edu
cfa.harvard.edu | Facebook | Twitter | YouTube | Newsletter
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Dec 15, 2020 at 01:29:42PM -0500, Phelps, Matthew wrote:
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
I don't think this is in conflict with the FAQ or what I have been saying. The FAQ clearly talks about "shifting resources and investments".
However, I have seen many times people suggesting that this is intended to force CentOS users into buying RHEL. That really is not the case.
On Tue, Dec 15, 2020 at 1:38 PM Matthew Miller mattdm@mattdm.org wrote:
On Tue, Dec 15, 2020 at 01:29:42PM -0500, Phelps, Matthew wrote:
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
I don't think this is in conflict with the FAQ or what I have been saying. The FAQ clearly talks about "shifting resources and investments".
However, I have seen many times people suggesting that this is intended to force CentOS users into buying RHEL. That really is not the case.
Yeah... right.
Forgive my sarcasm here, but Red Hat has earned it. "What does this mean for CentOS? Nothing." "We will continue to support existing timelines." Etc. etc. I trust Red Hat as far as I can throw them, and I am not alone.
On Tue, Dec 15, 2020 at 12:30 PM Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 10:23 AM sankarshan < sankarshan.mukhopadhyay@gmail.com> wrote:
I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how
ever
rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try
to
make, at least 60% will stay on CentOS Linux 7 until EOL but will
switch
CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to
anyone.
I have to rebase my server from CentOS 6, and I am going with
Springdale
for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they
can
switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us.
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Ah, actually I didn't do that. RHEL is and has been doing fine. Don't confuse "value" with revenue. CentOS Linux no longer served any purpose at Red Hat and I'll flip it back around as I did in the previous email.
Why should Red Hat, or any company, continue to pay for something that isn't working out?
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
I don't think anyone's said that. This is a massive change and disruption for the existing CentOS community. 90% of the community (by our estimates) will be able to stay on CentOS 7 until 2024 just as they expected. We made sure the 10% on CentOS Linux 8 didn't continue to grow (thus trying to minimize impact). We aren't punishing anyone and the fact that two other clones have already popped up is a testament to that.
Well, you all see the reaction this has garnered around the world, and it's all negative except for the Red Hat employees trying to convince us it's a good thing. Nice try.
Actually, things took an interesting turn around Thursday. Once people understood what we actually announced much of the press has been very positive, and now that the shock has worn off, we're seeing quite a lot of support.
We all know differently. And we are all now making influential choices that will hurt Red Hat.
I don't mean to sound cold here but if you really want to talk about the business side of this.... If you don't have a budget and don't end up finding a home in our coming low-cost or free offerings (Fedora, CentOS Stream, UBI, or RHEL for developers, CI, Open Source, edu, mom/pop shops, etc). Then what choices are you talking about?
-Mike
Good job!
I'm not going to say that the announcement was the board's idea or even
that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Note: I was not in the room when the voting happened. I was involved in the negotiations. The board had a tremendous impact on helping Red Hat better understand some things that needed to happen in CentOS Stream. For example, versioning it and supporting it through the full support cycle of RHEL instead of what stream was before (a sort of continuous stream with one year overlap for migrations, etc). The Board is expecting things out of CentOS Stream and we expect them to hold us to that.
It's easy to say "The Board is full of Red Hatters and they did this." But I think we all know that's not the case, some of the Red Hatters on the board were as fierce a defender of the existing CentOS community as one could possibly be. The board could have voted this down. Red Hat could have dissolved the board (as I understand the voting rules). But that didn't happen. Both sides of this came to an agreement that we - together
- could live with and that represented a positive future for CentOS. A
very very different future for sure, but a positive one.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
--
*Matt Phelps*
*Information Technology Specialist, Systems Administrator*
(Computation Facility, Smithsonian Astrophysical Observatory)
Center for Astrophysics | Harvard & Smithsonian
60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps@cfa.harvard.edu
cfa.harvard.edu | Facebook http://cfa.harvard.edu/facebook | Twitter http://cfa.harvard.edu/twitter | YouTube http://cfa.harvard.edu/youtube | Newsletter http://cfa.harvard.edu/newsletter
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Dec 15, 2020 at 2:03 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 12:30 PM Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 10:23 AM sankarshan < sankarshan.mukhopadhyay@gmail.com> wrote:
I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how
ever
rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's
muscles,
members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try
to
make, at least 60% will stay on CentOS Linux 7 until EOL but will
switch
CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to
anyone.
I have to rebase my server from CentOS 6, and I am going with
Springdale
for now, and will start learning Debian. I will soon resign as admin
in
Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they
can
switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to
Debian/Ubuntu,
that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us.
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Ah, actually I didn't do that. RHEL is and has been doing fine. Don't confuse "value" with revenue. CentOS Linux no longer served any purpose at Red Hat and I'll flip it back around as I did in the previous email.
Why should Red Hat, or any company, continue to pay for something that isn't working out?
Because you committed to it? Because you, repeatedly, said you would? And we believed you?
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
I don't think anyone's said that. This is a massive change and disruption for the existing CentOS community. 90% of the community (by our estimates) will be able to stay on CentOS 7 until 2024 just as they expected. We made sure the 10% on CentOS Linux 8 didn't continue to grow (thus trying to minimize impact). We aren't punishing anyone and the fact that two other clones have already popped up is a testament to that.
I have been lambasted repeatedly as if I was stealing from Red Hat. And that Red Hat deserved to get something from us "freeloaders."
Well, you all see the reaction this has garnered around the world, and it's
all negative except for the Red Hat employees trying to convince us it's a good thing. Nice try.
Actually, things took an interesting turn around Thursday. Once people understood what we actually announced much of the press has been very positive, and now that the shock has worn off, we're seeing quite a lot of support.
We all know differently. And we are all now making influential choices that will hurt Red Hat.
I don't mean to sound cold here but if you really want to talk about the business side of this.... If you don't have a budget and don't end up finding a home in our coming low-cost or free offerings (Fedora, CentOS Stream, UBI, or RHEL for developers, CI, Open Source, edu, mom/pop shops, etc). Then what choices are you talking about?
I was instrumental in recommending my organization go with RH way back when it was free. Then Fedora, then CentOS, and, yes, paid Red Hat in some places. I will be recommending otherwise now.
-Mike
Good job!
I'm not going to say that the announcement was the board's idea or even
that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Note: I was not in the room when the voting happened. I was involved in the negotiations. The board had a tremendous impact on helping Red Hat better understand some things that needed to happen in CentOS Stream. For example, versioning it and supporting it through the full support cycle of RHEL instead of what stream was before (a sort of continuous stream with one year overlap for migrations, etc). The Board is expecting things out of CentOS Stream and we expect them to hold us to that.
It's easy to say "The Board is full of Red Hatters and they did this." But I think we all know that's not the case, some of the Red Hatters on the board were as fierce a defender of the existing CentOS community as one could possibly be. The board could have voted this down. Red Hat could have dissolved the board (as I understand the voting rules). But that didn't happen. Both sides of this came to an agreement that we - together
- could live with and that represented a positive future for CentOS. A
very very different future for sure, but a positive one.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
--
*Matt Phelps*
*Information Technology Specialist, Systems Administrator*
(Computation Facility, Smithsonian Astrophysical Observatory)
Center for Astrophysics | Harvard & Smithsonian
60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps@cfa.harvard.edu
cfa.harvard.edu | Facebook http://cfa.harvard.edu/facebook | Twitter http://cfa.harvard.edu/twitter | YouTube http://cfa.harvard.edu/youtube | Newsletter http://cfa.harvard.edu/newsletter
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/15/2020 11:02 AM, Mike McGrath wrote:
On Tue, Dec 15, 2020 at 12:30 PM Phelps, Matthew <mphelps@cfa.harvard.edu mailto:mphelps@cfa.harvard.edu> wrote:
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>> wrote: I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Ah, actually I didn't do that. RHEL is and has been doing fine. Don't confuse "value" with revenue. CentOS Linux no longer served any purpose at Red Hat and I'll flip it back around as I did in the previous email.
Why should Red Hat, or any company, continue to pay for something that isn't working out?
If RedHat needed to justify or clarify the investment it was making in CentOS -- as a reminder: *after* taking the independent project under its wing and letting others snuff themselves out as superfluous -- then the professional thing to do would have been to go to the larger community about it.
Present options, such as the rebuild being spun back out. Or discuss mechanisms for increasing CentOS->RHEL conversions. Or solicit direct funding options to get free-riders to contribute directly to CentOS Project expenses while still keeping a firewall in place.
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
I don't think anyone's said that. This is a massive change and disruption for the existing CentOS community. 90% of the community (by our estimates) will be able to stay on CentOS 7 until 2024 just as they expected. We made sure the 10% on CentOS Linux 8 didn't continue to grow (thus trying to minimize impact). We aren't punishing anyone and the fact that two other clones have already popped up is a testament to that.
No, the fact that two other clones have popped up is a testament to OSS communities' ability to cope with events. The unexpected churn from having the distro pulled out is absolutely a punishment because it creates a great deal of work for all of us to return to the operational status quo with no real benefit.
*Direct question: If the CentOS Project (via the Board), secures funding for expenses relating to the rebuild, does it get to continue CentOS Linux?** *
Well, you all see the reaction this has garnered around the world, and it's all negative except for the Red Hat employees trying to convince us it's a good thing. Nice try.
Actually, things took an interesting turn around Thursday. Once people understood what we actually announced much of the press has been very positive, and now that the shock has worn off, we're seeing quite a lot of support.
"What you actually announced" was that CentOS+CR was going to be used internally for testing against future minor releases, and that any ideal of a binary-compatible rebuild was going away.
I'm sure there's support from internal RH teams that for some reason didn't have access to internal RHEL minor release betas. I can't imagine who else this benefits in any way shape or form (except Oracle, Amazon, and promoters of Debian-derived distributions).
"RedHat EL Stream" is a useful thing, and whether that's a Preview (post-QA), a Beta (intra-QA), or Rawhide (pre-QA), there's a place for it as an official way to provide feedback. But it's entirely orthogonal from the "North American Enterprise Linux Vendor" rebuild project.
We all know differently. And we are all now making influential choices that will hurt Red Hat.
I don't mean to sound cold here but if you really want to talk about the business side of this.... If you don't have a budget and don't end up finding a home in our coming low-cost or free offerings (Fedora, CentOS Stream, UBI, or RHEL for developers, CI, Open Source, edu, mom/pop shops, etc). Then what choices are you talking about?
Anyone even tangentally associated with OSS is aware of the free rider problem. As written above, there were plenty of ways to approach this without forcing CentOS Linux to be canned prematurely in the middle of a major release support cycle, immediately after EL6 had gone EOL, and after RedHat's entry had seemingly removed the need for other projects to continue operations. Frankly, the lack of goodwill demonstrated here places both "free" RHEL and UBI into suspect categories.
-jc
On Tue, Dec 15, 2020 at 2:03 PM Japheth Cleaver cleaver@terabithia.org wrote:
On 12/15/2020 11:02 AM, Mike McGrath wrote:
On Tue, Dec 15, 2020 at 12:30 PM Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath mmcgrath@redhat.com wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us.
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Ah, actually I didn't do that. RHEL is and has been doing fine. Don't confuse "value" with revenue. CentOS Linux no longer served any purpose at Red Hat and I'll flip it back around as I did in the previous email.
Why should Red Hat, or any company, continue to pay for something that isn't working out?
If RedHat needed to justify or clarify the investment it was making in CentOS -- as a reminder: *after* taking the independent project under its wing and letting others snuff themselves out as superfluous -- then the professional thing to do would have been to go to the larger community about it.
Present options, such as the rebuild being spun back out. Or discuss mechanisms for increasing CentOS->RHEL conversions. Or solicit direct funding options to get free-riders to contribute directly to CentOS Project expenses while still keeping a firewall in place.
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
I don't think anyone's said that. This is a massive change and disruption for the existing CentOS community. 90% of the community (by our estimates) will be able to stay on CentOS 7 until 2024 just as they expected. We made sure the 10% on CentOS Linux 8 didn't continue to grow (thus trying to minimize impact). We aren't punishing anyone and the fact that two other clones have already popped up is a testament to that.
No, the fact that two other clones have popped up is a testament to OSS communities' ability to cope with events. The unexpected churn from having the distro pulled out is absolutely a punishment because it creates a great deal of work for all of us to return to the operational status quo with no real benefit.
*Direct question: If the CentOS Project (via the Board), secures funding for expenses relating to the rebuild, does it get to continue CentOS Linux?*
Direct answer: Red Hat is out of the downstream building business. None of our other products have a downstream build sponsored by Red Hat, RHEL won't either. The cost was only one of many considerations here.
-Mike
Well, you all see the reaction this has garnered around the world, and it's all negative except for the Red Hat employees trying to convince us it's a good thing. Nice try.
Actually, things took an interesting turn around Thursday. Once people understood what we actually announced much of the press has been very positive, and now that the shock has worn off, we're seeing quite a lot of support.
"What you actually announced" was that CentOS+CR was going to be used internally for testing against future minor releases, and that any ideal of a binary-compatible rebuild was going away.
I'm sure there's support from internal RH teams that for some reason didn't have access to internal RHEL minor release betas. I can't imagine who else this benefits in any way shape or form (except Oracle, Amazon, and promoters of Debian-derived distributions).
"RedHat EL Stream" is a useful thing, and whether that's a Preview (post-QA), a Beta (intra-QA), or Rawhide (pre-QA), there's a place for it as an official way to provide feedback. But it's entirely orthogonal from the "North American Enterprise Linux Vendor" rebuild project.
We all know differently. And we are all now making influential choices
that will hurt Red Hat.
I don't mean to sound cold here but if you really want to talk about the business side of this.... If you don't have a budget and don't end up finding a home in our coming low-cost or free offerings (Fedora, CentOS Stream, UBI, or RHEL for developers, CI, Open Source, edu, mom/pop shops, etc). Then what choices are you talking about?
Anyone even tangentally associated with OSS is aware of the free rider problem. As written above, there were plenty of ways to approach this without forcing CentOS Linux to be canned prematurely in the middle of a major release support cycle, immediately after EL6 had gone EOL, and after RedHat's entry had seemingly removed the need for other projects to continue operations. Frankly, the lack of goodwill demonstrated here places both "free" RHEL and UBI into suspect categories.
-jc
On Tue, 15 Dec 2020 14:51:41 -0600 Mike McGrath mmcgrath@redhat.com wrote:
*Direct question: If the CentOS Project (via the Board), secures funding for expenses relating to the rebuild, does it get to continue CentOS Linux?*
Direct answer: Red Hat is out of the downstream building business. None of our other products have a downstream build sponsored by Red Hat, RHEL won't either. The cost was only one of many considerations here.
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
On Mon, Dec 21, 2020 at 11:13 AM John Crisp jcrisp@safeandsoundit.co.uk wrote:
On Tue, 15 Dec 2020 14:51:41 -0600 Mike McGrath mmcgrath@redhat.com wrote:
*Direct question: If the CentOS Project (via the Board), secures funding for expenses relating to the rebuild, does it get to continue CentOS Linux?*
Direct answer: Red Hat is out of the downstream building business. None of our other products have a downstream build sponsored by Red Hat, RHEL won't either. The cost was only one of many considerations here.
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
-Mike
Le 21/12/2020 à 18:31, Mike McGrath a écrit :
On Mon, Dec 21, 2020 at 11:13 AM John Crisp <jcrisp@safeandsoundit.co.uk mailto:jcrisp@safeandsoundit.co.uk> wrote:
On Tue, 15 Dec 2020 14:51:41 -0600 Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>> wrote: > > > > *Direct question: If the CentOS Project (via the Board), secures > > funding for expenses relating to the rebuild, does it get to > > continue CentOS Linux?* > > > Direct answer: Red Hat is out of the downstream building business. > None of our other products have a downstream build sponsored by Red > Hat, RHEL won't either. The cost was only one of many considerations > here. > That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
-Mike
Again, a fair way of doing things would have been, Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS, it will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL.
There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled.
I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ?
-Jean-Marc
Le 21/12/2020 à 21:27, Jean-Marc Liger a écrit :
Le 21/12/2020 à 18:31, Mike McGrath a écrit :
On Mon, Dec 21, 2020 at 11:13 AM John Crisp <jcrisp@safeandsoundit.co.uk mailto:jcrisp@safeandsoundit.co.uk> wrote:
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
-Mike
Again, a fair way of doing things would have been, Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS, it will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL.
There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled.
I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ?
-Jean-Marc
I ask for a fourth time this proposal which still remains unanswered, but as we say in French, "who doesn't say word consents",therefore it is obviously a question of recovering money from the CentOS community with the subscriptionsRed Hat coming soon.
Jean-Marc
On Sat, Dec 26, 2020 at 10:11 AM Jean-Marc Liger < jean-marc.liger@parisdescartes.fr> wrote:
Le 21/12/2020 à 21:27, Jean-Marc Liger a écrit :
Le 21/12/2020 à 18:31, Mike McGrath a écrit :
On Mon, Dec 21, 2020 at 11:13 AM John Crisp jcrisp@safeandsoundit.co.uk wrote:
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
-Mike
Again, a fair way of doing things would have been, Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS, it will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL.
There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled.
I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ?
-Jean-Marc
I ask for a fourth time this proposal which still remains unanswered, but as we say in French, "who doesn't say word consents", therefore it is obviously a question of recovering money from the CentOS community with the subscriptions Red Hat coming soon.
Jean-Marc
How many operating systems do you think we need to be building. In your little text diagram above, its not clear to me what usefulness CentOS Linux is to RHEL. I understand why you'd want it (free RHEL). Why do you think we should produce it? What usefulness is it to us?
We don't want to "recover money" from the CentOS community just like our other communities. But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you providing that would result in continued sponsorship of a downstream rebuild?
-Mike
Le 26/12/2020 à 17:49, Mike McGrath a écrit :
On Sat, Dec 26, 2020 at 10:11 AM Jean-Marc Liger <jean-marc.liger@parisdescartes.fr mailto:jean-marc.liger@parisdescartes.fr> wrote:
Le 21/12/2020 à 21:27, Jean-Marc Liger a écrit :
Le 21/12/2020 à 18:31, Mike McGrath a écrit :
On Mon, Dec 21, 2020 at 11:13 AM John Crisp <jcrisp@safeandsoundit.co.uk <mailto:jcrisp@safeandsoundit.co.uk>> wrote: That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'. Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so. -Mike
Again, a fair way of doing things would have been, Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS, it will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL. There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled. I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ? -Jean-Marc
I ask for a fourth time this proposal which still remains unanswered, but as we say in French, "who doesn't say word consents",therefore it is obviously a question of recovering money from the CentOS community with the subscriptionsRed Hat coming soon. Jean-Marc
How many operating systems do you think we need to be building. In your little text diagram above, its not clear to me what usefulness CentOS Linux is to RHEL. I understand why you'd want it (free RHEL). Why do you think we should produce it? What usefulness is it to us?
To fullfil the CentOS initial Goal thought to produce a RHEL clone.
We don't want to "recover money" from the CentOS community just like our other communities. But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you providing that would result in continued sponsorship of a downstream rebuild?
-Mike
My little text diagram is simple to understand. It is no an downstream rebuild, but an upstream pre-build, with more stability than Stream and actually the missing piece of motivation to help Stream to become stronger.
Jean-Marc
On Sat, Dec 26, 2020 at 11:10 AM Jean-Marc Liger < jean-marc.liger@parisdescartes.fr> wrote:
Le 26/12/2020 à 17:49, Mike McGrath a écrit :
On Sat, Dec 26, 2020 at 10:11 AM Jean-Marc Liger < jean-marc.liger@parisdescartes.fr> wrote:
Le 21/12/2020 à 21:27, Jean-Marc Liger a écrit :
Le 21/12/2020 à 18:31, Mike McGrath a écrit :
On Mon, Dec 21, 2020 at 11:13 AM John Crisp jcrisp@safeandsoundit.co.uk wrote:
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
-Mike
Again, a fair way of doing things would have been, Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS, it will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL.
There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled.
I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ?
-Jean-Marc
I ask for a fourth time this proposal which still remains unanswered, but as we say in French, "who doesn't say word consents", therefore it is obviously a question of recovering money from the CentOS community with the subscriptions Red Hat coming soon.
Jean-Marc
How many operating systems do you think we need to be building. In your little text diagram above, its not clear to me what usefulness CentOS Linux is to RHEL. I understand why you'd want it (free RHEL). Why do you think we should produce it? What usefulness is it to us?
To fullfil the CentOS initial Goal thought to produce a RHEL clone.
We've been evaluating that goas for years and I (and others) are unconvinced why we should be doing that. None of our other products have a downstream build that we sponsor. What benefit does doing something we've already done (RHEL), and doing it again (CentOS Rebuild), have for Red Hat? Believe me, when I tell you we looked for years to find an answer to that, no one enjoyed this announcement. But when you look at it, CentOS Linux no longer makes any sense to sponsor.
We don't want to "recover money" from the CentOS community just like our other communities. But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you providing that would result in continued sponsorship of a downstream rebuild?
-Mike
My little text diagram is simple to understand. It is no an downstream rebuild, but an upstream pre-build, with more stability than Stream and actually the missing piece of motivation to help Stream to become stronger.
RHEL is a downstream rebuild of Stream. Perhaps the missing piece you need to help isn't another rebuild, but free actual RHEL?
-Mike
Le 26/12/2020 à 18:17, Mike McGrath a écrit :
On Sat, Dec 26, 2020 at 11:10 AM Jean-Marc Liger <jean-marc.liger@parisdescartes.fr mailto:jean-marc.liger@parisdescartes.fr> wrote:
Le 26/12/2020 à 17:49, Mike McGrath a écrit :
On Sat, Dec 26, 2020 at 10:11 AM Jean-Marc Liger <jean-marc.liger@parisdescartes.fr <mailto:jean-marc.liger@parisdescartes.fr>> wrote: Le 21/12/2020 à 21:27, Jean-Marc Liger a écrit :
Le 21/12/2020 à 18:31, Mike McGrath a écrit :
On Mon, Dec 21, 2020 at 11:13 AM John Crisp <jcrisp@safeandsoundit.co.uk <mailto:jcrisp@safeandsoundit.co.uk>> wrote: That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'. Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so. -Mike
Again, a fair way of doing things would have been, Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS, it will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL. There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled. I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ? -Jean-Marc
I ask for a fourth time this proposal which still remains unanswered, but as we say in French, "who doesn't say word consents",therefore it is obviously a question of recovering money from the CentOS community with the subscriptionsRed Hat coming soon. Jean-Marc How many operating systems do you think we need to be building. In your little text diagram above, its not clear to me what usefulness CentOS Linux is to RHEL. I understand why you'd want it (free RHEL). Why do you think we should produce it? What usefulness is it to us?
To fullfil the CentOS initial Goal thought to produce a RHEL clone.
We've been evaluating that goas for years and I (and others) are unconvinced why we should be doing that. None of our other products have a downstream build that we sponsor. What benefit does doing something we've already done (RHEL), and doing it again (CentOS Rebuild), have for Red Hat? Believe me, when I tell you we looked for years to find an answer to that, no one enjoyed this announcement. But when you look at it, CentOS Linux no longer makes any sense to sponsor.
As I said below, the idea is to produce a downtream RHEL clone, with CentOS Stream as the continuous beta release for RHEL, and Centos Linux the bitwise release candidate for RHEL you can last adjust with the feedback of the CentOS community. It's a simple and valuable answer to what you've been looking for years.
We don't want to "recover money" from the CentOS community just like our other communities. But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you providing that would result in continued sponsorship of a downstream rebuild? -Mike
My little text diagram is simple to understand. It is no an downstream rebuild, but an upstream pre-build, with more stability than Stream and actually the missing piece of motivation to help Stream to become stronger.
RHEL is a downstream rebuild of Stream. Perhaps the missing piece you need to help isn't another rebuild, but free actual RHEL?
What I need is a free RHEL clone for educational purposes, with no subscription and no restrictive contract terms to deal with.
Right now the winner of your fantastical move is Oracle Linux, that fewbody knows before. Every day there is another IT website to explain how easy it is to move from CentOS to Oracle Linux 8. And Project Lenix and Rocky Linux will also succeded in this trifecta order. And in the end you will loose both the value of the CentOS Brand and Community. Is this the big deal Red Hat want to achieve?
Jean-Marc
Le 26/12/2020 à 20:09, Jean-Marc Liger a écrit :
Le 26/12/2020 à 18:17, Mike McGrath a écrit :
We've been evaluating that goas for years and I (and others) are unconvinced why we should be doing that. None of our other products have a downstream build that we sponsor. What benefit does doing something we've already done (RHEL), and doing it again (CentOS Rebuild), have for Red Hat? Believe me, when I tell you we looked for years to find an answer to that, no one enjoyed this announcement. But when you look at it, CentOS Linux no longer makes any sense to sponsor.
As I said below, the idea is to produce a downtream RHEL clone, with CentOS Stream as the continuous beta release for RHEL, and Centos Linux the bitwise release candidate for RHEL you can last adjust with the feedback of the CentOS community. It's a simple and valuable answer to what you've been looking for years.
No answer again, as the end goal of this initiative is to kill CentOS Linux, maybe not to get Community money back, but surely because it reduces some RHEL sales. However this is not a Community responsibility, it is all Red Hat's fault when it bought CentOS and gave more credibility to this project without highlighting the value of RHEL over CentOS Linux. I proposed a solution for that dilemna which end all the downstream process and restored some trust we could put in Red Hat, and this advice is free of charge.
RHEL is a downstream rebuild of Stream. Perhaps the missing piece you need to help isn't another rebuild, but free actual RHEL?
And what should be the terms of this free actual RHEL we could be helpfull for ?
What I need is a free RHEL clone for educational purposes, with no subscription and no restrictive contract terms to deal with.
Right now the winner of your fantastical move is Oracle Linux, that fewbody knows before. Every day there is another IT website to explain how easy it is to move from CentOS to Oracle Linux 8. And Project Lenix and Rocky Linux will also succeded in this trifecta order. And in the end you will loose both the value of the CentOS Brand and Community. Is this the big deal Red Hat want to achieve?
Jean-Marc
On Sun, Dec 27, 2020 at 02:03:32PM +0100, Jean-Marc Liger wrote:
RHEL is a downstream rebuild of Stream. Perhaps the missing piece you need to help isn't another rebuild, but free actual RHEL?
And what should be the terms of this free actual RHEL we could be helpfull for ?
As I understand it, some of the programs are going to be introduced shortly. If they don't meet your needs, or if you have further questions, I encourage you to email centos-questions@redhat.com. This is monitored by the people designing the programs, not by sales.
On 27.12.2020 22:45, Matthew Miller wrote:
As I understand it, some of the programs are going to be introduced
shortly.
If they don't meet your needs, or if you have further questions, I
encourage
you to email centos-questions@redhat.com. This is monitored by the people designing the programs, not by sales.
I hope this is not only monitored, but answered as well. It could be nice to know an average response time (before first response, if any, and between consequent ones).
Personally, I sent a message 9 days ago (no response so far - although I am not surprised, there might be thousands of messages daily).
On su, 27 joulu 2020, Konstantin Boyandin via CentOS-devel wrote:
On 27.12.2020 22:45, Matthew Miller wrote:
As I understand it, some of the programs are going to be introduced
shortly.
If they don't meet your needs, or if you have further questions, I
encourage
you to email centos-questions@redhat.com. This is monitored by the people designing the programs, not by sales.
I hope this is not only monitored, but answered as well. It could be nice to know an average response time (before first response, if any, and between consequent ones).
Personally, I sent a message 9 days ago (no response so far - although I am not surprised, there might be thousands of messages daily).
I am not on that list myself but I am on vacation, as well as all those people. Red Hat is known to have a period of end of year vacations that literally cause visible 'dives' in activities of those red hatters.
This year was also exceptional in the sense that people weren't able to use their vacation time during 'normal' periods, piling up a lot of unused days that have to be used or they cannot be moved forward in some jurisdictions.
But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you providing that would result in continued sponsorship of a downstream rebuild?
Installation count share?
Without CenOS Linux you'll look at 1% of Linux servers running RHEL (I'm sure you have a more precise number, and I'm sure it will be as embarrassing...).
In a world without CentOS Linux why should I pay for RH courses? Why should I renew my RHCE? Why should educators choose such a rare distribution to teach Linux? At this point why not go with Ubuntu? Debian? Why teach yum/dnf when most servers will use apt?
You say us (the community) isn't providing you (Red Hat) with profit. Well, who do you think installed CentOS Linux and recommended RHEL to our pointy haired bosses that wanted support contracts?
Maybe you sold 1 RHEL subscription for any 100 CentOS we installed.
Well, now you're going to sell exactly 0 RHEL subscriptions for any of the 100 Ubuntu or Debian boxes I'm going to set up with my clients.
Before you say "Stream", the single most killer argument against Stream is that you have broken any trust when you moved the EOL of CentOS Linux 8 from 2029 to 2021.
Now Red Hat employees jump through hoops to tell us how great that'll be. What guerantees that in two year's time you're not going to kill Stream because "it didn't provide anything to RH"?
Somebody already mentioned "fool me once..."...
I'm still angry and I still cannot believe you appear to be oblivious to the huge blunder you made with this incredible EOL shortening. You've basically killed the CentOS brand in one single move.
Bye, Chris.
On Sat, Dec 26, 2020 at 4:12 PM Chris Mair chris@1006.org wrote:
But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you
providing
that would result in continued sponsorship of a downstream rebuild?
Installation count share?
Without CenOS Linux you'll look at 1% of Linux servers running RHEL (I'm sure you have a more precise number, and I'm sure it will be as embarrassing...).
In a world without CentOS Linux why should I pay for RH courses? Why should I renew my RHCE? Why should educators choose such a rare distribution to teach Linux? At this point why not go with Ubuntu? Debian? Why teach yum/dnf when most servers will use apt?
This one is interesting and one we discussed in length. Unfortunately, if you look at any "top" list. Ubuntu is clumped together, and Red Hat gets broken out into Fedora, RHEL, and CentOS. As for why educators would pick a "rare" distribution, because we're going to have a program that caters directly to education.
You say us (the community) isn't providing you (Red Hat) with profit. Well, who do you think installed CentOS Linux and recommended RHEL to our pointy haired bosses that wanted support contracts?
Maybe you sold 1 RHEL subscription for any 100 CentOS we installed.
Well, now you're going to sell exactly 0 RHEL subscriptions for any of the 100 Ubuntu or Debian boxes I'm going to set up with my clients.
Unfortunately with these last ones we've seen no evidence of this happening (where CentOS is actually leading to RHEL sales. We have seen a little evidence of the opposite. Where we once had RHEL sales, and now have CentOS. Why? "They're both built by Red Hat, right?"
Before you say "Stream", the single most killer argument against Stream is that you have broken any trust when you moved the EOL of CentOS Linux 8 from 2029 to 2021.
I think we lost a lot of trust due to a pretty serious mix-up about the EOL date announcement, but I don't think it has erased all the good Red Hat has done and continues to do.
Now Red Hat employees jump through hoops to tell us how great that'll be.
What guerantees that in two year's time you're not going to kill Stream because "it didn't provide anything to RH"?
Nothing, we and all companies kill products all the time. I don't recall this level of outrage over mugshot. We're taking a big risk with Stream and if it doesn't work out, we'll make changes there too as we should. It makes no sense to continue doing something that isn't working out.
Somebody already mentioned "fool me once..."...
I'm still angry and I still cannot believe you appear to be oblivious to the huge blunder you made with this incredible EOL shortening. You've basically killed the CentOS brand in one single move.
We're not oblivious to it. If this were any other organization or relationship, we'd help make amends by giving you your money back. That's just not an option here. And you can say we killed it all you want, what we've done is significantly change it. You may not recognize it anymore but there are many people on this list who we talked to before the announcement and that we've seen now who are actually interested in coming on this journey with us. Thats good.
I suspect the very trust that you all were putting in Red Hat to continue to produce CentOS Linux as though it were actually a 10-year enterprise-grade distribution for production was part of the problem here. To further demonstrate that problem, many of the replies I've seen look as though people did their risk assessments with "we're relying on Red Hat for our OS" instead of "We're relying on a community for our OS." I'd imagine some of you are having very awkward conversations with your management chain about this. No one using Fedora or WildFly thinks that way.
We wanted CentOS to flourish in development environments, in upstream Open Source CI, and to help with things like OpenStack. I'm not sure if we accomplished any of that. For those that think perhaps that was the mistake all those years ago, I personally agree with you.
-Mike
Bye, Chris.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/26/20 11:48 PM, Mike McGrath wrote:
I suspect the very trust that you all were putting in Red Hat to continue to produce CentOS Linux as though it were actually a 10-year enterprise-grade distribution for production was part of the problem here. To further demonstrate that problem, many of the replies I've seen look as though people did their risk assessments with "we're relying on Red Hat for our OS" instead of "We're relying on a community for our OS." I'd imagine some of you are having very awkward conversations with your management chain about this. No one using Fedora or WildFly thinks that way.
Considering Red Hat bought entire CentOS project in total secret, or should we say bribed CentOS dev group with jobs, you do not get to say "why should we continue to build it" without admitting that Red Hat's goal was to kill it all along but needed to control the clone building process first. How was it said? Embrace, Engulf, Extinguish? Either say both or neither.
On Sat, Dec 26, 2020 at 6:03 PM Ljubomir Ljubojevic centos@plnet.rs wrote:
On 12/26/20 11:48 PM, Mike McGrath wrote:
I suspect the very trust that you all were putting in Red Hat to continue to produce CentOS Linux as though it were actually a 10-year enterprise-grade distribution for production was part of the problem here. To further demonstrate that problem, many of the replies I've seen look as though people did their risk assessments with "we're relying on Red Hat for our OS" instead of "We're relying on a community for our OS." I'd imagine some of you are having very awkward conversations with your management chain about this. No one using Fedora or WildFly thinks that way.
Considering Red Hat bought entire CentOS project in total secret, or should we say bribed CentOS dev group with jobs, you do not get to say "why should we continue to build it" without admitting that Red Hat's goal was to kill it all along but needed to control the clone building process first. How was it said? Embrace, Engulf, Extinguish? Either say both or neither.
I wasn't involved in the beginning and am not privy to those discussions so I won't comment on them. I will say that many of the paid CentOS engineers are the most vocal community advocates on my team. Saying they were bribed is pretty bad form, especially if you actually know any of them. They gave you and others years of stability and often in challenging situations.
-Mike
On 12/27/20 1:18 AM, Mike McGrath wrote:
On Sat, Dec 26, 2020 at 6:03 PM Ljubomir Ljubojevic <centos@plnet.rs mailto:centos@plnet.rs> wrote:
On 12/26/20 11:48 PM, Mike McGrath wrote: > I suspect the very trust that you all were putting in Red Hat to > continue to produce CentOS Linux as though it were actually a 10-year > enterprise-grade distribution for production was part of the problem > here. To further demonstrate that problem, many of the replies I've > seen look as though people did their risk assessments with "we're > relying on Red Hat for our OS" instead of "We're relying on a community > for our OS." I'd imagine some of you are having very awkward > conversations with your management chain about this. No one using > Fedora or WildFly thinks that way. Considering Red Hat bought entire CentOS project in total secret, or should we say bribed CentOS dev group with jobs, you do not get to say "why should we continue to build it" without admitting that Red Hat's goal was to kill it all along but needed to control the clone building process first. How was it said? Embrace, Engulf, Extinguish? Either say both or neither.
I wasn't involved in the beginning and am not privy to those discussions so I won't comment on them. I will say that many of the paid CentOS engineers are the most vocal community advocates on my team. Saying they were bribed is pretty bad form, especially if you actually know any of them. They gave you and others years of stability and often in challenging situations.
I have wrote here before that both back then and now I do *not* have any problems with them taking those jobs. Good for them, they were recognized and advanced in life. And they *did* earn that with their work on CentOS. They worked on the project without any visible personal gain for years while we enjoyed fruits of their work. So any gain/reward/payment they got out of it is not in question, I (and I suspect vast majority of users) do not hold them against it.
But lets be honest and blunt. Red Hat wanted control over CentOS project (for whatever reason you want to choose to believe in) and they approached CentOS Board and in *secret* conducted crucial negotiations about the future of community project and in return they got *personal gains* out of the deal, out of giving their consent to relinquish the control over the supposedly community project. Here, you define the word that describes this phenomena. What will it be I wonder.
And veto over any Board decision Red Hat does not like and threat of loosing the great job they have in the process is the definition of the word "mobbing" I believe. All they can do to support the project was to plead with Red Hat executives and maybe threaten with PR disaster by leaving the Red Hat and/or leaking how to clone RHEL to the public. They did not have, and do not have any other power over the CentOS project, just to resign and help others.
In hindsight, because I think even they were persuaded by Red Hat representatives, my personal opinion is that CentOS Board made a bad decision when they agreed to join Red Hat in a sense that they lost their own freedom of expression, their hands are tide and they are not free to publicly voice any of their grievances or opposition.
Again, it they are fine with it, who am I to judge them? But I know I personally could not do it, I did walk away from good jobs when I was harassed without cause, and I would have done it again. I even sent my WISP users to competitors when their demands were unreasonable or they were verbally abusive (while I was operating small WISP). But that is me.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Saturday, December 26, 2020 4:48 PM, Mike McGrath mmcgrath@redhat.com wrote:
On Sat, Dec 26, 2020 at 4:12 PM Chris Mair chris@1006.org wrote:
Before you say "Stream", the single most killer argument against Stream
is that you have broken any trust when you moved the EOL of CentOS Linux 8 from 2029 to 2021.
I think we lost a lot of trust due to a pretty serious mix-up about the EOL date announcement, but I don't think it has erased all the good Red Hat has done and continues to do.
Can you quantify that? Can you list out all the good Red Hat has done and what score should be assigned to each of those items. Then also list the evil Red Hat has done nad what score should be assigned. We will then have a value of how much evil Red Hat still feels it can justify in the future while still being able to claim that at least the good still has greater weight than the evil.
I don't think your statement is as reassuring as what you intended it to be.
Now Red Hat employees jump through hoops to tell us how great that'll be. What guerantees that in two year's time you're not going to kill Stream because "it didn't provide anything to RH"?
Nothing, we and all companies kill products all the time. I don't recall this level of outrage over mugshot. We're taking a big risk with Stream and if it doesn't work out, we'll make changes there too as we should. It makes no sense to continue doing something that isn't working out.
What existing popular community project did mugshot replace?
Imagine a parallel universe in which Red Hat does the following:
(1) Red Hat declares it's support for a display server called Wayland
(2) Red Hat then declares all documentation claiming such support is now out of date but doesn't tell anyone
(3) Red Hat then has a meeting behind close doors decides to drop all assistance to contribute to Wayland and will replace it with Mir
(4) Red Hat then announces the replacement with several excuses why it is for the good of the community
Eventually there would be a wikipedia article titled "Mir (software)" with a controversy section.
This behavior with CentOS is Red Hat's version of Mir.
I would be willing to discuss this further with you on Wave but the company was not commited to Wave and replaced it.
So, I would be willing to discuss this further with you on Gtalk... but the company was not commited to Gtalk and replaced it.
So, I would be willing to discuss this further with you on Hangouts... or do I mean RCS?
Please remind me, who is the dominate provider of instant messager services at this point?
Somebody already mentioned "fool me once..."...
I'm still angry and I still cannot believe you appear to be oblivious to the huge blunder you made with this incredible EOL shortening. You've basically killed the CentOS brand in one single move.
We're not oblivious to it. If this were any other organization or relationship, we'd help make amends by giving you your money back. That's just not an option here. And you can say we killed it all you want, what we've done is significantly change it. You may not recognize it anymore but there are many people on this list who we talked to before the announcement and that we've seen now who are actually interested in coming on this journey with us. Thats good.
We already understand there is a nameless set of cardinals selected to be notified ahead of time before the CATHEDRAL announced it to the public.
None of us are asking for our money back or our time back. What was requested was that Red Hat treat the *Community* ENTerprise OS as a BAZAAR instead of leveraging it's ownership of the trademarks to bully the community into a cathedral model.
I suspect the very trust that you all were putting in Red Hat to continue to produce CentOS Linux as though it were actually a 10-year enterprise-grade distribution for production was part of the problem here. To further demonstrate that problem, many of the replies I've seen look as though people did their risk assessments with "we're relying on Red Hat for our OS" instead of "We're relying on a community for our OS." I'd imagine some of you are having very awkward conversations with your management chain about this. No one using Fedora or WildFly thinks that way.
I can't imagine why anyone would think the Community ENTERPRISE Operating System would be confused with a project striving for enterprise-grade.
But thank you for your warning about WildFly. I will pass the warning not to expect WildFly to be enterprise grade to users of Keycloak.
We wanted CentOS to flourish in development environments, in upstream Open Source CI, and to help with things like OpenStack. I'm not sure if we accomplished any of that. For those that think perhaps that was the mistake all those years ago, I personally agree with you.
The biggest problem with OpenStack/RDO is the upgrade path. OpenStack has 2 releases every year just like Fedora. But if I install Fedora 24 and then upgrade incrementally until I get to Fedora 33, most things still work as expected. If I install OpenStack Newton and then attempt to upgrade incrementally to Train, things go badly quickly. There is a lot of work to be done.
I would like to see the issues with OpenStack resolved. I would like a community built around RDO. But how Red Hat is going about things now is not about mobilizing the community. This sudden 1 year remaing on CentOS 8 just frustration and demotivating.
We may not be asking for our "money" or asking back our time back invested in CentOS. But we also aren't going to be quick to invest more time into a company that has commitments to us just go out of date.
But taking a step back. Let's say Wade's blog post is completely honest and Red Hat wants to close the openness gap. Does that make any openness gaps in Stream considered a bug? Do I file a bug report against the Stream kernel SRPM? Would anyone take it seriously? Is this something Red Hat really wants to resolve? Or is openness whatever Red Hat defines it to be internally and all other defintions are out of date?
On Sat, Dec 26, 2020 at 4:12 PM Chris Mair chris@1006.org wrote:
But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you
providing
that would result in continued sponsorship of a downstream rebuild?
Installation count share?
Without CenOS Linux you'll look at 1% of Linux servers running RHEL (I'm sure you have a more precise number, and I'm sure it will be as embarrassing...).
In a world without CentOS Linux why should I pay for RH courses? Why should I renew my RHCE? Why should educators choose such a rare distribution to teach Linux? At this point why not go with Ubuntu? Debian? Why teach yum/dnf when most servers will use apt?
This one is interesting and one we discussed in length. Unfortunately, if you look at any "top" list. Ubuntu is clumped together, and Red Hat gets broken out into Fedora, RHEL, and CentOS. As for why educators would pick a "rare" distribution, because we're going to have a program that caters directly to education.
That's exactly what Google, Apple and Microsoft do - and I absolutely do not like it and do not support any such company and teach my kids to stay away as far as they can and also explain them why.
You say us (the community) isn't providing you (Red Hat) with profit. Well, who do you think installed CentOS Linux and recommended RHEL to our pointy haired bosses that wanted support contracts?
Maybe you sold 1 RHEL subscription for any 100 CentOS we installed.
Well, now you're going to sell exactly 0 RHEL subscriptions for any of the 100 Ubuntu or Debian boxes I'm going to set up with my clients.
Unfortunately with these last ones we've seen no evidence of this happening (where CentOS is actually leading to RHEL sales. We have seen a little evidence of the opposite. Where we once had RHEL sales, and now have CentOS. Why? "They're both built by Red Hat, right?"
At least for my environments in the last two decades I can tell you you are completely wrong here. And you can not know it because nobody who bought RedHat because of me told you so. They just ordered servers from $VENDOR with bundled RH subscriptions and renewed them together with all other software.
Before you say "Stream", the single most killer argument against Stream is that you have broken any trust when you moved the EOL of CentOS Linux 8 from 2029 to 2021.
I think we lost a lot of trust due to a pretty serious mix-up about the EOL date announcement, but I don't think it has erased all the good Red Hat has done and continues to do.
Now Red Hat employees jump through hoops to tell us how great that'll be.
What guerantees that in two year's time you're not going to kill Stream because "it didn't provide anything to RH"?
Nothing, we and all companies kill products all the time. I don't recall this level of outrage over mugshot. We're taking a big risk with Stream and if it doesn't work out, we'll make changes there too as we should. It makes no sense to continue doing something that isn't working out.
Maybe this is the biggest point: we just couldn't believe RedHat is like "all companies" :(
Regards, Simon
Somebody already mentioned "fool me once..."...
I'm still angry and I still cannot believe you appear to be oblivious to the huge blunder you made with this incredible EOL shortening. You've basically killed the CentOS brand in one single move.
We're not oblivious to it. If this were any other organization or relationship, we'd help make amends by giving you your money back. That's just not an option here. And you can say we killed it all you want, what we've done is significantly change it. You may not recognize it anymore but there are many people on this list who we talked to before the announcement and that we've seen now who are actually interested in coming on this journey with us. Thats good.
I suspect the very trust that you all were putting in Red Hat to continue to produce CentOS Linux as though it were actually a 10-year enterprise-grade distribution for production was part of the problem here. To further demonstrate that problem, many of the replies I've seen look as though people did their risk assessments with "we're relying on Red Hat for our OS" instead of "We're relying on a community for our OS." I'd imagine some of you are having very awkward conversations with your management chain about this. No one using Fedora or WildFly thinks that way.
We wanted CentOS to flourish in development environments, in upstream Open Source CI, and to help with things like OpenStack. I'm not sure if we accomplished any of that. For those that think perhaps that was the mistake all those years ago, I personally agree with you.
-Mike
Bye, Chris.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 27.12.2020 05:48, Mike McGrath wrote:
On Sat, Dec 26, 2020 at 4:12 PM Chris Mair <chris@1006.org mailto:chris@1006.org> wrote: You say us (the community) isn't providing you (Red Hat) with profit. Well, who do you think installed CentOS Linux and recommended RHEL to our pointy haired bosses that wanted support contracts?
Maybe you sold 1 RHEL subscription for any 100 CentOS we installed. Well, now you're going to sell exactly 0 RHEL subscriptions for any of the 100 Ubuntu or Debian boxes I'm going to set up with my clients.
Unfortunately with these last ones we've seen no evidence of this happening (where CentOS is actually leading to RHEL sales. We have seen a little evidence of the opposite. Where we once had RHEL sales, and now have CentOS. Why? "They're both built by Red Hat, right?"
That only means your marketing people poorly did their homework.
That's the obvious metric - encourage CentOS users to supply you with data on how actively they were promoting paid RH services.
Before you say "Stream", the single most killer argument against
Stream
is that you have broken any trust when you moved the EOL of CentOS Linux 8 from 2029 to 2021.
I think we lost a lot of trust due to a pretty serious mix-up about the EOL date announcement, but I don't think it has erased all
the
good Red Hat has done and continues to do.
As our saying goes, "count your chickens when autumn comes". Let's see what comes in several years.
You've already repeated (indirectly) many times that RH basically doesn't care about losing community that doesn't bring immediate profits (in good correlation with a corporation priority #1).
On 26 Dec 16:48, Mike McGrath wrote:
On Sat, Dec 26, 2020 at 4:12 PM Chris Mair chris@1006.org wrote:
But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you
providing
that would result in continued sponsorship of a downstream rebuild?
Installation count share?
Without CenOS Linux you'll look at 1% of Linux servers running RHEL (I'm sure you have a more precise number, and I'm sure it will be as embarrassing...).
In a world without CentOS Linux why should I pay for RH courses? Why should I renew my RHCE? Why should educators choose such a rare distribution to teach Linux? At this point why not go with Ubuntu? Debian? Why teach yum/dnf when most servers will use apt?
This one is interesting and one we discussed in length. Unfortunately, if you look at any "top" list. Ubuntu is clumped together, and Red Hat gets broken out into Fedora, RHEL, and CentOS. As for why educators would pick a "rare" distribution, because we're going to have a program that caters directly to education.
You say us (the community) isn't providing you (Red Hat) with profit. Well, who do you think installed CentOS Linux and recommended RHEL to our pointy haired bosses that wanted support contracts?
Maybe you sold 1 RHEL subscription for any 100 CentOS we installed.
Well, now you're going to sell exactly 0 RHEL subscriptions for any of the 100 Ubuntu or Debian boxes I'm going to set up with my clients.
Unfortunately with these last ones we've seen no evidence of this happening (where CentOS is actually leading to RHEL sales. We have seen a little evidence of the opposite. Where we once had RHEL sales, and now have CentOS. Why? "They're both built by Red Hat, right?"
That is sales' fault. The message from Red Hat before 2014 was: you know, CentOS is community based, it can go away any time, you should buy RHEL...
That messaging was such as it led to the situation you describe, when indeed the main argument used by sales was no longer correct after RHEL acquisition.
It would have been different if sales would have spoken about the added value of RHEL vs CentOS instead of simply using fear.
On 12/26/2020 2:48 PM, Mike McGrath wrote:
On Sat, Dec 26, 2020 at 4:12 PM Chris Mair <chris@1006.org mailto:chris@1006.org> wrote:
Before you say "Stream", the single most killer argument against Stream is that you have broken any trust when you moved the EOL of CentOS Linux 8 from 2029 to 2021.
I think we lost a lot of trust due to a pretty serious mix-up about the EOL date announcement, but I don't think it has erased all the good Red Hat has done and continues to do.
This is not a "mix-up!" This is not an "oops, my bad!" Most large-scale users of CentOS operate in orgs that require some level of post-mortem analysis on major fiascos like this. If Red Hat wants to start trying to rebuild ANY goodwill, Step 0 would be releasing a clear and transparent post-mortem of what exactly happened here and why. The reasons behind this have changed, public statements and leaked statements and implications have contradicted one another, and there's clear evidence of internal concern.
Perhaps instead of digging the hole even further over the holidays, Red Hat can start responding in a healthier way than it is now.
Now Red Hat employees jump through hoops to tell us how great that'll be. What guerantees that in two year's time you're not going to kill Stream because "it didn't provide anything to RH"?
Nothing, we and all companies kill products all the time. I don't recall this level of outrage over mugshot. We're taking a big risk with Stream and if it doesn't work out, we'll make changes there too as we should. It makes no sense to continue doing something that isn't working out.
Who exactly is the "we" you're referring to? You can't have your cake and eat it too when it comes to RedHat coming at the CentOS Board and threatening an override if a decision does not go the way it wants, knowing full well that this would have far-ranging effects on all downstream users.
Somebody already mentioned "fool me once..."... I'm still angry and I still cannot believe you appear to be oblivious to the huge blunder you made with this incredible EOL shortening. You've basically killed the CentOS brand in one single move.
We're not oblivious to it. If this were any other organization or relationship, we'd help make amends by giving you your money back. That's just not an option here. And you can say we killed it all you want, what we've done is significantly change it. You may not recognize it anymore but there are many people on this list who we talked to before the announcement and that we've seen now who are actually interested in coming on this journey with us. Thats good.
Lots of people would be interested in CentOS Stream (myself included). You have killed CentOS Linux (the distribution). You're equivocating here between the two sets of bits, just like you're equivocating between Red Hat and the CentOS Project above.
I suspect the very trust that you all were putting in Red Hat to continue to produce CentOS Linux as though it were actually a 10-year enterprise-grade distribution for production was part of the problem here. To further demonstrate that problem, many of the replies I've seen look as though people did their risk assessments with "we're relying on Red Hat for our OS" instead of "We're relying on a community for our OS." I'd imagine some of you are having very awkward conversations with your management chain about this. No one using Fedora or WildFly thinks that way.
Yes. "I can't believe that a VP at Red Hat, of all places, is being such an a** on the centos-devel list! // Me neither!" is indeed a "very awkward conversation" to have with one's management chain. Many of us are quite curious about what "very awkward conversations" are happening over there.
Let me make this very clear: NO ONE out there who's responsible for more than two dozen installs of CentOS is unaware that it's a rebuild of the upstream product. No one is threatening to sue Red Hat Inc over this, because you haven't broken our contract; you've only broken our trust. Our risk assessment was that Red Hat would behave like responsible adults and the long-time leaders in the OSS community (we believed) they were, not poison its community and apparently try to salt the earth to boot.
Please stop stepping on rakes. It's making this bad situation worse.
-jc
On 26.12.2020 23:49, Mike McGrath wrote:
On Sat, Dec 26, 2020 at 10:11 AM Jean-Marc Liger <jean-marc.liger@parisdescartes.fr mailto:jean-marc.liger@parisdescartes.fr> wrote:
There would have been no downstream build sponsored by Red Hat, CentOS Linux would have kill other clones this way, as it already did for Scientific Linux 8, the CentOS Community would have been happy to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled. I eared you no answer about this proposal, could you tell me why if it's not all about grabbing more money from the CentOS Community ? -Jean-Marc
I ask for a fourth time this proposal which still remains unanswered, but as we say in French, "who doesn't say word consents",therefore it is obviously a question of recovering money from the CentOS community with the subscriptionsRed Hat coming soon. Jean-Marc
How many operating systems do you think we need to be building. In your little text diagram above, its not clear to me what usefulness CentOS Linux is to RHEL. I understand why you'd want it (free RHEL). Why do you think we should produce it? What usefulness is it to us?
That brings an obvious question: have RH obtained CentOS stuff with the only actual purpose to shut down "free RHEL"?
We don't want to "recover money" from the CentOS community just like our other communities. But as a business, since you're not providing Red Hat with profit (none of our communities are), what are you providing that would result in continued sponsorship of a downstream rebuild?
How nice.
Does advertising RH paid services and gaining you new customers count for one? I suspect I am not the only one who did that for years.
(of course I won't do that any more and will personally apologize to those unlucky who've subscribed to RH services)
On Mon, 21 Dec 2020 11:31:13 -0600 Mike McGrath mmcgrath@redhat.com wrote:
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
As I mentioned elsewhere, this is gaslighting. Again.
You are closing/removing the RH owned and distributed downstream build RHEL known as CentOS (as you are entitled too). That is Extinguish.
Ends.
Whether there is something else, by someone else, no one can be sure. But that is a different story entirely.
The curious bit will be what RH try to do to with any new version that appears eg Rocky, if they are in any way successful.
On Tue, Dec 22, 2020 at 5:03 PM John Crisp jcrisp@safeandsoundit.co.uk wrote:
On Mon, 21 Dec 2020 11:31:13 -0600 Mike McGrath mmcgrath@redhat.com wrote:
That's the 'Extinguish' bit. Comes after 'Embrace' and 'Extend'.
Uhhh, there is no Extinguish in Open Source. That's the entire point. We embraced, we extended. If others want to carry on the torch they are more than welcome to do so.
As I mentioned elsewhere, this is gaslighting. Again.
This is a pretty offensive thing to say as you've basically claimed that I'm attempting to emotionally abuse you and others on the list and goes well beyond professional candor.
You are closing/removing the RH owned and distributed downstream build
RHEL known as CentOS (as you are entitled too). That is Extinguish.
Ends.
Whether there is something else, by someone else, no one can be sure. But that is a different story entirely.
The curious bit will be what RH try to do to with any new version that appears eg Rocky, if they are in any way successful.
This argument, which I've seen restated and restated, continues to ignore all the other downstream rebuilds *including* one of Red Hat's competitors in Oracle. As soon as you add that to the mix, this argument completely falls apart. It's not free "open-builds". It's open-source. All of our source is available because we already pushed it upstream before we released it and then again in our downstream build. Just like Red Hat, you are welcome to embrace and extend it. And, just like Red Hat you can't extinguish it.
-Mike
On Tue, Dec 22, 2020 at 6:57 PM Mike McGrath mmcgrath@redhat.com wrote:
This argument, which I've seen restated and restated, continues to ignore all the other downstream rebuilds *including* one of Red Hat's competitors in Oracle. As soon as you add that to the mix, this argument completely falls apart. It's not free "open-builds". It's open-source. All of our source is available because we already pushed it upstream before we released it and then again in our downstream build. Just like Red Hat, you are welcome to embrace and extend it. And, just like Red Hat you can't extinguish it.
Oracle and Amazon both edit it extensively, and are not so gracious about sharing their modifications. Whitbox used to do a decent job, and Scientific Linux. But CERN bowed out of doing Scientific Linux releases as CentOS grew and Red Hat supported it directly. It's difficult to do.
I'll have to think about this. I may have to publish some reposync based "snapshotting" backup tools to produce labeled, datestamped "8-stream.2020-12-01" style repos. The contents except for repodata can be hardlinked, so it's not *outrageously* inefficient.
I'll have to think about this. I may have to publish some reposync based "snapshotting" backup tools to produce labeled, datestamped "8-stream.2020-12-01" style repos.
Erm... isn't this called repomanagement. Both Suse Manager(Uyuni is the upstream) and RH Sattelite do great job on that.
Best Regards, Strahil Nikolov
On Wed, Dec 23, 2020 at 12:47 AM Strahil Nikolov via CentOS-devel centos-devel@centos.org wrote:
I'll have to think about this. I may have to publish some reposync based "snapshotting" backup tools to produce labeled, datestamped "8-stream.2020-12-01" style repos.
Erm... isn't this called repomanagement. Both Suse Manager(Uyuni is the upstream) and RH Sattelite do great job on that.
Best Regards, Strahil Nikolov
The last I looked at it, RHN Satellite was very expensive and demanded manual tuning of the content for each host, and did not provide a set of reference repos for local browsing and repo comparison. I've found structures similar to vault.centos.org to be more predictable and safer to manage, much as the CentOS point releases have been useful and much as git tags associated with specific, defined states of the repo are useful. It's why I say they're taags" and find the model mentioned here that "it's all one branch" to not match reality until this public decision to discard point releases.
The older among us saw this play out with Red Hat 9: I think Red Hat was understandably frustrated with people demanding ongoing support for Red Hat 7.0 distinct from hat for Red Hat 7.3 and other, similar oint releases. The policy of "there shall be only one stream" Was reversed with the RHEL 2.1, the next official release from Red Hat
I suspect this is going to be short lived and be reversed with the next major release update.
On 12/22/20 10:36 PM, Nico Kadel-Garcia wrote:
It's why I say they're taags" and find the model mentioned here that "it's all one branch" to not match reality
For fun, I'll try again:
RHEL point releases are branches. 7.6 is a branch. 7.7 is a branch. You can continue running 7.6 and receive security updates after 7.7 is released. Those updates may include packages built specifically for 7.6, and not just a selection of the packages for 7.7. They're maintained in parallel, at the same time. They're branches.
CentOS point releases weren't individual branches. There was only one CentOS 7 branch. CentOS 7.6 was just a point in time along the lifetime of CentOS 7. 7.6 is not literally a tag, but it's the closest analogy. There was no continued support for CentOS 7.6 after CentOS 7.7 was released. If there's no parallel maintenance, there is only one branch.
In an VCS, you can create a branch and continue work, and later create another branch off of that and continue work, but if you never add any work to an older branch after a new branch is created, then you're only using branches in a very superficial sense. There are technically branches, but there's no difference in that workflow between several branches and just one, because you have just one linear history containing every commit. This resembles CentOS updates.
RHEL point releases get updates that aren't just updates for a later release. As an analogy, there are updates in the older branches that aren't in the new branches, unlike CentOS.
CentOS has just one branch:
* 7.5 \ * 7.6 \ * 7.7
RHEL has multiple branches that overlap in time:
* ---- 7.5 \ * ---- 7.6 \ * ---- 7.7
On Wed, Dec 23, 2020 at 2:14 AM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/22/20 10:36 PM, Nico Kadel-Garcia wrote:
It's why I say they're taags" and find the model mentioned here that "it's all one branch" to not match reality
For fun, I'll try again:
RHEL point releases are branches. 7.6 is a branch. 7.7 is a branch. You can continue running 7.6 and receive security updates after 7.7 is released. Those updates may include packages built specifically for 7.6, and not just a selection of the packages for 7.7. They're maintained in parallel, at the same time. They're branches.
It's a point release, with a defined and immutable content. In subversion, git, CVS, and other source control terms, it's a tag that someone doesn't want to call a tag.
CentOS point releases weren't individual branches. There was only one CentOS 7 branch. CentOS 7.6 was just a point in time along the lifetime of CentOS 7. 7.6 is not literally a tag, but it's the closest analogy. There was no continued support for CentOS 7.6 after CentOS 7.7 was released. If there's no parallel maintenance, there is only one branch.
Seriously, no one cares. Both RHEL support, and CentOS support for updates pointed users to the current software channels for the leading repos and sayd to "yum update" whatever the particular package was. If you did "yum update", with either, yes indeed you got the leading edge of *everything* and lost the stability of haveing a well defined point release. None of that means that the point releases were not essentially "tags", well-defined states of the repository useful as a stable reference for build environments, business class environments, or clusters.
As a stable set, it was effectively a tag, no matter what the upstream source management was.
RHEL point releases get updates that aren't just updates for a later release. As an analogy, there are updates in the older branches that aren't in the new branches, unlike CentOS.
As I understood from dealing with Red Hat, they're short term, for specific customers, and discarded as quickly as possible. Everything gets migrated to the main codeline or discarded. If it never got brought into the main CentOS codeline, frankly, I couldn't use it, I had too many mixed system to burn cycles resolving that kind of discrepancy.
CentOS has just one branch:
- 7.5 \
\
- 7.6
- 7.7
RHEL has multiple branches that overlap in time:
- ---- 7.5 \
\
- ---- 7.6
- ---- 7.7
As long as tweaks get folded back into the main release contents, and Red Hat was very good about that, no one cared. Stopping the support of the point releases was the whole point of Red Hat 9, and RHEL continues the practice on their support lines. The last time I called them about this (perhaps 3 years ago?), the main response to "I need to fix this particular release's issues" was to try to sell me on host-specific RPM inventory tuning with a very expensive and unwelcome RHN setup that would have been very expensive and unsustainable because my contract was short term. The "rsync for CentOS" or "reposync for RHEL" internal yum repos were *much* more effective, and also made it possible to scale up hundreds of hosts for a cluster simultaneously without choking our external bandwidth to death.
On Wed, Dec 23, 2020 at 2:14 AM Gordon Messmer gordon.messmer@gmail.com wrote:
RHEL point releases are branches. 7.6 is a branch. 7.7 is a branch. You can continue running 7.6 and receive security updates after 7.7 is released. Those updates may include packages built specifically for 7.6, and not just a selection of the packages for 7.7. They're maintained in parallel, at the same time. They're branches.
CentOS point releases weren't individual branches. There was only one CentOS 7 branch. CentOS 7.6 was just a point in time along the lifetime of CentOS 7. 7.6 is not literally a tag, but it's the closest analogy. There was no continued support for CentOS 7.6 after CentOS 7.7 was released. If there's no parallel maintenance, there is only one branch.
Honestly, I can't believe you are still on this page. This is some serious kool-aid you are drinking. I'm a little jealous.
https://vault.centos.org/8.3.2011/BaseOS/Source/SPackages/kernel-4.18.0-240....
Do you know what the ".1.1" is? It's called a branch.
And here, is 8.2:
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193....
What? .28.1 wasn't in 8.3. Check the changelog for 4.18.0-240.1.1. Where did it come from? Oh right. It's called a branch.
I know you are stuck on the "c7" being a branch, and using terms like "VCS" to fill in some background. But, as already mentioned - "c7" is a flattened set of imports and de-branding. It is representing a more complex structure, that you are either unaware of - or you are pretending doesn't exist. I can't tell which yet. But, this "flattened set of imports and de-branding" is what disappears with CentOS 8 Stream. CentOS 8 Stream is the -240, -241, -242, ... with no backports also known as no branches.
On 12/23/20 12:26 AM, Mark Mielke wrote:
hat? .28.1 wasn't in 8.3. Check the changelog for 4.18.0-240.1.1. Where did it come from? Oh right. It's called a branch.
That's right. Because it comes from RHEL, where point releases are branches. That's what we keep saying: RHEL point releases are branches.
I know you are stuck on the "c7" being a branch, and using terms like "VCS" to fill in some background. But, as already mentioned - "c7" is a flattened set of imports and de-branding.
That's right. CentOS major releases are flattened. That's what we keep saying: There is only one supported branch in CentOS at any given point in time.
That's all that Matthew was pointing out, really. If you have an application that needs an ultra-stable base OS, with security updates but no new features for more than (roughly) 6-8 months, Red Hat can provide you with such an OS. CentOS doesn't.
On Wed, Dec 23, 2020 at 1:41 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/23/20 12:26 AM, Mark Mielke wrote:
hat? .28.1 wasn't in 8.3. Check the changelog for 4.18.0-240.1.1. Where did it come from? Oh right. It's called a branch.
That's right. Because it comes from RHEL, where point releases are branches. That's what we keep saying: RHEL point releases are branches. That's right. CentOS major releases are flattened. That's what we keep saying: There is only one supported branch in CentOS at any given point in time. That's all that Matthew was pointing out, really. If you have an application that needs an ultra-stable base OS, with security updates but no new features for more than (roughly) 6-8 months, Red Hat can provide you with such an OS. CentOS doesn't.
I see hope at the end of this tunnel. You admit that RHEL minor releases are branches. You admit CentOS is a flattened represention of the RHEL minor release branches, which means that CentOS is composed of concatenated set of RHEL minor release branches plus de-branding. And, you agree that "If you have an application that needs an ultra-stable base OS, with security updates but no new features for MORE than (roughly) 6-8 months, ..."
So, please admit that if you want a stable base OS release, with no new features, for *less* than 6-8 months, then this is exactly what CentOS is.
Name one person that claimed that CentOS is supported beyond the active release. Nobody did. The claim was that CentOS is based of RHEL minor release branches, and this makes the effect the same as RHEL minor release branches. That is - Red Hat branches the RHEL minor release. Red Hat backports from X.N+1 and X.N+2 only as necessary to X.N. The X.N patch set makes in into CentOS via imports. The imports are serialized, just like *any* build system might become serialized, but this is an artifact of the process, it is not indicative that the content is not branched.
Your only point centers around how the serialization process doesn't generate any late branching. This is an irrelevant point. Nobody said it said. It has no impact on the result, which is that we have branched content visible both in the source (which is not the CentOS Git repositories... the CentOS Git repositories are *part* of the process, they are not the original source for the process), and the destination. Anybody can see these as branches:
https://vault.centos.org/8.0.1905/ https://vault.centos.org/8.1.1911/ https://vault.centos.org/8.2.2004/ https://vault.centos.org/8.3.2011/
It walks like a duck, and it talks like a duck. It has duck as a point of origin, and when you cook the duck in an oven, it tastes like a duck. But sure, for a section of the process known as "CentOS import, de-branding, build, and release" it becomes serialized.
Ironically - RHEL minor releases become serialized in the very same way. It's ironic, because you keep saying it's different - but you have no evidence that it is different.
Everything you say about CentOS applies to RHEL:
That's all that Matthew was pointing out, really. If you have an application that needs an ultra-stable base OS, with security updates but no new features for more than (roughly) 6-8 months, Red Hat can provide you with such an OS. CentOS doesn't.
I know you think RHEL EUS is "a separate support offering, and not actually different". But, just as CentOS does not have these "extended updates" in the CentOS repository (meaning as you believe "there is no branch"), the main BaseOS and AppStream repositories for RHEL also not have these "extended updates". With the exception of build environment problems, and specific packages and package updates, which Red Hat chooses not to import into the CentOS Git repositories, the set of packages available in the CentOS repositories and the set of packages in the RHEL repositories are 1:1, with the only delta being de-branding.
What you say above, about "more than (roughly) 6-8 months", requires a different subscription type, and a different repository. Whether it is the same branch in the backend for RHEL as RHEL EUS? It doesn't really matter. It is only the effect that counts.
The effect is that CentOS = RHEL for most real-life measurements, including how the release was initially baselined, and which backported source changes make it into the release. This is because CentOS is imported from RHEL.
CentOS Stream is a different thing, as while CentOS = RHEL, CentOS Stream does not match ANY RHEL release branch.
But, I think this has to be my last attempt to instruct you on this. You are free to believe whatever you wish. Other people know how this works, and understand that CentOS Stream is different. If you think it's the same thing - good luck.
On 12/23/20 2:00 PM, Mark Mielke wrote:
You admit that RHEL minor releases are branches. You admit CentOS is a flattened represention of the RHEL minor release branches, which means that CentOS is composed of concatenated set of RHEL minor release branches plus de-branding.
You're using the word "admit" as if that wasn't the point that Matthew was making to begin with, which you and Nico argued against. (In response to Matthew, Nico wrote "No. RHEL minor releases are more like source control "tags" than branches." I wrote that Matthew was correct, and you replied "This is false...")
So, please admit that if you want a stable base OS release, with no new features, for *less* than 6-8 months, then this is exactly what CentOS is.
It's pretty close, with one significant caveat: for (roughly) two months out of the year, CentOS doesn't get any updates at all, including security patches. For me, that's an awfully big risk. I would much rather get features on a regular basis than go without security patches for a month, twice per year.
Personally, I think it's irresponsible to make the claim that CentOS is 1:1 with RHEL. It isn't. RHEL is supported all of the time. CentOS is supported during (roughly) 10/12 months of the year. Now, you're free to decide that the 10/12 month support SLA is good enough for your business, and you can say so. That's fine. But these threads drag on at length because when we discuss the benefits that come with giving up the point release in favor of continuous delivery of updates, you start shouting "this is false! this is false! this is false!"
On Wed, Dec 23, 2020 at 5:43 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/23/20 2:00 PM, Mark Mielke wrote:
You admit that RHEL minor releases are branches. You admit CentOS is a flattened represention of the RHEL minor release branches, which means that CentOS is composed of concatenated set of RHEL minor release branches plus de-branding.
You're using the word "admit" as if that wasn't the point that Matthew was making to begin with, which you and Nico argued against. (In response to Matthew, Nico wrote "No. RHEL minor releases are more like source control "tags" than branches." I wrote that Matthew was correct, and you replied "This is false...")
In hindsight... I should have qualified. The first thing I said "false" to, was that Nico should stop talking and "listen" because he might "learn something". I recognized in Nico's words that he knew exactly what he was talking about, and I did not agree that he should stop talking and "listen" because he might "learn something". I wish he would speak more.
So, please admit that if you want a stable base OS release, with no new features, for *less* than 6-8 months, then this is exactly what CentOS is.
It's pretty close, with one significant caveat: for (roughly) two months out of the year, CentOS doesn't get any updates at all, including security patches. For me, that's an awfully big risk. I would much rather get features on a regular basis than go without security patches for a month, twice per year.
Yes! Finally some common ground! We both agree that CentOS Linux is a *stable* OS with security patches and backports for the "6-8 month(s)" period that the minor release is active. We both agree that the CentOS program had difficult with new releases, which did mean that in general it always runs behind RHEL by weeks to months. With the exception of high priority security fixes, basically the whole program is initially delayed by a longer amount, and then catches up, but remains behind by a few weeks. With the exception of build environment issues, CentOS and RHEL run in parallel, with CentOS slightly behind RHEL.
If anybody wants *support* for the active release, they should probably not use CentOS. They should use RHEL, or another vendor that provides *support*.
If anybody wants *patches* for prior releases, Red Hat runs a Extended Update Support program which costs extra (or comes for free with "Premium") that extends the life of the minor release up to about 2 years after the minor release was first published. Red Hat runs additional programs (at higher extra cost) for support beyond this two year period.
If people find CentOS acceptable to their requirements, it was an excellent choice. If it was not acceptable, they would have found something else.
Personally, I think it's irresponsible to make the claim that CentOS is 1:1 with RHEL. It isn't. RHEL is supported all of the time. CentOS is supported during (roughly) 10/12 months of the year. Now, you're free to decide that the 10/12 month support SLA is good enough for your business, and you can say so.
It might be, or it might not be. We all make choices. It sounds like you are considering going forwards with CentOS 8 Stream. Personally, I would consider this irresponsible. However, we all have different use cases, priorities, and internal capabilities, and what is irresponsible for one person, might be perfectly acceptable for another person.
I think Red Hat has talented people that could solve this problem. Simply, it was not a business priority. However you try to phrase it - fundamentally, it was not in Red Hat's interest to make sure CentOS was released simultaneously with RHEL. Technically possible, but there was insufficient business incentive. Now, if we compare to something like Oracle Linux, Oracle Linux is also delayed - but in some cases, the delay is as little as 6 days. I refuse to believe Red Hat couldn't achieve this, so I'm left to believe they didn't care to. :-)
That's fine. But these threads drag on at length because when we discuss the benefits that come with giving up the point release in favor of continuous delivery of updates, you start shouting "this is false! this is false! this is false!"
I didn't say "benefits of continuous delivery of updates is false". I said points you were making about how it compares to CentOS 8 were false. You can make the case for CentOS 8 Stream, without making misleading statements about CentOS 8. For example, above - it's perfectly valid to say that CentOS 8 was delayed from RHEL, and this may represent a serious security concern. This is a true statement.
I don't want to belabor this, as I really don't like getting so frustrated as to have to tell somebody what they are saying is False. But, I think it is important to point out that I am clearly in favour of continuous delivery of updates for *some use cases*. You need only read my other posts to see that I don't think CentOS 8 Stream is a bad thing. But, I, and many other people - do not agree that it is a drop-in replacement for CentOS 8, and I think the crux of this is an accurate understanding of what CentOS 8 is. Not the pedantic definition, nor the ignorant definition, but the practical real-life experience of a large number of the users, and the carpet being ripped out from under them. CentOS 8 Stream may be the future, but most of us have to deal with the past. We don't have the luxury of living on the edge.
On 12/24/20 2:18 AM, Mark Mielke wrote:
Not the pedantic definition, nor the ignorant definition, but the practical real-life experience of a large number of the users, and the carpet being ripped out from under them. CentOS 8 Stream may be the future, but most of us have to deal with the past. We don't have the luxury of living on the edge.
+1 on most of the mail
On 12/23/20 11:43 PM, Gordon Messmer wrote:
It's pretty close, with one significant caveat: for (roughly) two months out of the year, CentOS doesn't get any updates at all, including security patches. For me, that's an awfully big risk. I would much rather get features on a regular basis than go without security patches for a month, twice per year.
Every CentOS user accepts this as part of the "free" offering. Anyone that has problem with this gap has bought RHEL subscription, as would have I if it was important enough for me.
But I would not have said there are no security for entire 2 months because CentOS devs have been pushing important security updates into CR repooitory for instance, if I remember correctly. But again, you are either OK with the wait or you buy RHEL subscription, that was the deal everyone accept. I have 3-4 CentOS servers that are inside firewall and have no internet access other then SSH with ports to them turned on only when I need to access them. On intranet side all employees (3-10 people) access shares with different users but same password (to avoid confusing them since all have physical access to every PC and usually jump in when other is unavailable). I do not update them more frequently then once every few months, so I do not loose anything if I have to wait. I only have one public facing server that I update regularly but since there are no industrial secrets on it only few websites and a mail server, I am not in need of SWAT team protecting it.
On 12/24/2020 11:21 PM, Ljubomir Ljubojevic wrote:
On 12/23/20 11:43 PM, Gordon Messmer wrote:
It's pretty close, with one significant caveat: for (roughly) two months out of the year, CentOS doesn't get any updates at all, including security patches. For me, that's an awfully big risk. I would much rather get features on a regular basis than go without security patches for a month, twice per year.
Every CentOS user accepts this as part of the "free" offering. Anyone that has problem with this gap has bought RHEL subscription, as would have I if it was important enough for me.
But I would not have said there are no security for entire 2 months because CentOS devs have been pushing important security updates into CR repooitory for instance, if I remember correctly. But again, you are either OK with the wait or you buy RHEL subscription, that was the deal everyone accept.
It's also worth pointing out that in cases where we're known to be in a delay period (such as just after a point release) or where there's a critical CVE and neither RHEL nor the CentOS updates have dropped, it's not uncommon for a critical update to just be rolled internally.
Take the existing SRPM, apply patch, call it N-V-R.1+ test the fix, sign and insert into your private yum repo that you inevitably have. Done. When the upstream and/or vendor fix is released, it will silently upgrade in place over yours with the official version. Large CentOS installs have teams of Linux systems engineers capable of doing this if a relevant security fix needs to go out.
But this depends on having a predictable upstream, and a stable foundation on which to build on top of. I.e., coherent OS release management with /updates/ layered over it. CentOS CR/Stream does not have this, which is why it is not generally suitable for production use on actual, persistent boxes.
-jc
On Mon, Dec 28, 2020 at 4:06 AM Japheth Cleaver cleaver@terabithia.org wrote:
On 12/24/2020 11:21 PM, Ljubomir Ljubojevic wrote:
On 12/23/20 11:43 PM, Gordon Messmer wrote:
It's pretty close, with one significant caveat: for (roughly) two months out of the year, CentOS doesn't get any updates at all, including security patches. For me, that's an awfully big risk. I would much rather get features on a regular basis than go without security patches for a month, twice per year.
Every CentOS user accepts this as part of the "free" offering. Anyone that has problem with this gap has bought RHEL subscription, as would have I if it was important enough for me.
But I would not have said there are no security for entire 2 months because CentOS devs have been pushing important security updates into CR repooitory for instance, if I remember correctly. But again, you are either OK with the wait or you buy RHEL subscription, that was the deal everyone accept.
It's also worth pointing out that in cases where we're known to be in a delay period (such as just after a point release) or where there's a critical CVE and neither RHEL nor the CentOS updates have dropped, it's not uncommon for a critical update to just be rolled internally.
Take the existing SRPM, apply patch, call it N-V-R.1+ test the fix, sign and insert into your private yum repo that you inevitably have. Done. When the upstream and/or vendor fix is released, it will silently upgrade in place over yours with the official version. Large CentOS installs have teams of Linux systems engineers capable of doing this if a relevant security fix needs to go out.
Unless, of course, internal '%{name}-%{version}-%{release}" number exceeds that of the published release. Been there, done that. It requires careful control of versions and release numbering, with no use of "Epoch" settings, to avoid your internal kernel being considered more "recent" than whatever version and release number the next upstream kernel provides. Large CentOS installs typically have one or two engineers who might handle both kernel integration and release integration. especially because it crosses departments with hardware testing, QA, and resource management for the reboots. In a bunch of places I've worked, it's been *me*.
But this depends on having a predictable upstream, and a stable foundation on which to build on top of. I.e., coherent OS release management with /updates/ layered over it. CentOS CR/Stream does not have this, which is why it is not generally suitable for production use on actual, persistent boxes.
Yeah. We're going to wind up with people declining the update to CentOS 8, or maintaining internal "snapshot" repos, probably timestamped. You could actually do it with "rsnapshot" and more sane snapshot numbering scheme.
On 12/23/2020 10:41 AM, Gordon Messmer wrote:
On 12/23/20 12:26 AM, Mark Mielke wrote:
I know you are stuck on the "c7" being a branch, and using terms like "VCS" to fill in some background. But, as already mentioned - "c7" is a flattened set of imports and de-branding.
That's right. CentOS major releases are flattened. That's what we keep saying: There is only one supported branch in CentOS at any given point in time.
That's all that Matthew was pointing out, really. If you have an application that needs an ultra-stable base OS, with security updates but no new features for more than (roughly) 6-8 months, Red Hat can provide you with such an OS. CentOS doesn't.
The point is that whatever CentOS Linux was, and whatever its installed base of users "needed," was known stability. Everyone running CentOS (at scale, not neophytes) is aware that they don't get frozen release tracking, they don't get direct access to EUS, that they have to deal with their own lifecycle management outside of RHN, and to pay attention to minor release drops and be prepared to manually forward over important updates from CR/Stream/Updates in the (hopefully brief) time period between the RHEL release and CentOS's rebuild and update forklift.
CentOS Linux provided a *paradigm* for that... whatever you want to call it. Stream does not. As you've written, if we had an issue with CentOS Linux's operation in this regard, we wouldn't be using it in the first place.
-jc
On 12/23/20 1:14 AM, Gordon Messmer wrote:
On 12/22/20 10:36 PM, Nico Kadel-Garcia wrote:
It's why I say they're taags" and find the model mentioned here that "it's all one branch" to not match reality
For fun, I'll try again:
RHEL point releases are branches. 7.6 is a branch. 7.7 is a branch. You can continue running 7.6 and receive security updates after 7.7 is released. Those updates may include packages built specifically for 7.6, and not just a selection of the packages for 7.7. They're maintained in parallel, at the same time. They're branches.
CentOS point releases weren't individual branches. There was only one CentOS 7 branch. CentOS 7.6 was just a point in time along the lifetime of CentOS 7. 7.6 is not literally a tag, but it's the closest analogy. There was no continued support for CentOS 7.6 after CentOS 7.7 was released. If there's no parallel maintenance, there is only one branch.
In an VCS, you can create a branch and continue work, and later create another branch off of that and continue work, but if you never add any work to an older branch after a new branch is created, then you're only using branches in a very superficial sense. There are technically branches, but there's no difference in that workflow between several branches and just one, because you have just one linear history containing every commit. This resembles CentOS updates.
RHEL point releases get updates that aren't just updates for a later release. As an analogy, there are updates in the older branches that aren't in the new branches, unlike CentOS.
CentOS has just one branch:
- 7.5
\ * 7.6 \ * 7.7
RHEL has multiple branches that overlap in time:
- ---- 7.5
\ * ---- 7.6 \ * ---- 7.7
This is 100% exactly accurate. AND, this is how it has been since the beginning of CentOS Linux.
Stream is really no different than this. Each major version is one tree (IE CentOS Stream 8, CentOS Stream 9)
There will be 5 years for each stream tree (after the release of the RHEL 8.0 or RHEL 9.0 official release). This results in about 2 years of overlap (or maybe slightly more) of 2 versions of Stream being active at the same time so you can plan a migration from one to the other.
On Wednesday, December 23, 2020 10:00 AM, Johnny Hughes johnny@centos.org wrote:
On 12/23/20 1:14 AM, Gordon Messmer wrote:
On 12/22/20 10:36 PM, Nico Kadel-Garcia wrote:
It's why I say they're taags" and find the model mentioned here that "it's all one branch" to not match reality
For fun, I'll try again: RHEL point releases are branches. 7.6 is a branch. 7.7 is a branch. You can continue running 7.6 and receive security updates after 7.7 is released. Those updates may include packages built specifically for 7.6, and not just a selection of the packages for 7.7. They're maintained in parallel, at the same time. They're branches. CentOS point releases weren't individual branches. There was only one CentOS 7 branch. CentOS 7.6 was just a point in time along the lifetime of CentOS 7. 7.6 is not literally a tag, but it's the closest analogy. There was no continued support for CentOS 7.6 after CentOS 7.7 was released. If there's no parallel maintenance, there is only one branch. In an VCS, you can create a branch and continue work, and later create another branch off of that and continue work, but if you never add any work to an older branch after a new branch is created, then you're only using branches in a very superficial sense. There are technically branches, but there's no difference in that workflow between several branches and just one, because you have just one linear history containing every commit. This resembles CentOS updates. RHEL point releases get updates that aren't just updates for a later release. As an analogy, there are updates in the older branches that aren't in the new branches, unlike CentOS. CentOS has just one branch:
- 7.5 \ * 7.6 \ * 7.7
RHEL has multiple branches that overlap in time:
- ---- 7.5 \ * ---- 7.6 \ * ---- 7.7
This is 100% exactly accurate. AND, this is how it has been since the beginning of CentOS Linux.
Stream is really no different than this. Each major version is one tree (IE CentOS Stream 8, CentOS Stream 9)
There will be 5 years for each stream tree (after the release of the RHEL 8.0 or RHEL 9.0 official release). This results in about 2 years of overlap (or maybe slightly more) of 2 versions of Stream being active at the same time so you can plan a migration from one to the other.
Thank you for taking the time to share your perspective.
However, if Stream is "really no different" then give is 5 years CentOS 8. Make the End of Life of CentOS 8 the same as CentOS 7.
More importantly, before continuing to justify this change please work to give us the governance board we were promised.
I don't understand why you of all people would allow your name to be used to give credability to an invalid governance board!
Here is the announcement explained how the CentOS governance would work with CentOS joining forces with Red Hat: https://forums.centos.org/viewtopic.php?t=44407
Key items:
(1) "The key operating points of the Board are going to be: Public, Open, and Inclusive."
(2) "The Red Hat Enterprise Linux to CentOS firewall will also remain. Members and contributors to the CentOS efforts are still isolated from the RHEL Groups inside Red Hat, with the only interface being srpm / source path tracking, no sooner than is considered released. In summary: we retain an upstream."
The governance board meeting that gutted CentOS' End of Life commitment date wasn't public, open and inclusive. There was no announcement for community particpation. There is no public transcript or chat log of the board meeting.
There also was no firewall between CentOS and RHEL for the governance meeting. Brian "Bex" Exelbierd, a member of the RHEL team, was allowed to be part of the board in violation of stated CentOS governance policy.
Kadsten Wade's blog post also explained that CentOS will *NOT* retain an upstream. It will *replace* being the upstream.
Daniel Comnea (the person that started this thread) has asked we think before getting mad. I have read through his blog post. I have consider it all carefully including the his points about poor communication. But it is not just about poor communication, the community of CentOS was lied to about how governance would work after CentOS joined Red Hat.
It isn't just that it was "possible" to communicate better. It was *required* that it be public, open and inclusive governance board. Not behind closed door with a member of the RHEL team.
Comnea has pointed out that the hate toward Rich Bowen and Chris Wright is not fair. I agree with Comnea on this point. But Rich Bown and Chris Wright's documents talk as if this decision was made by a valid governance board. No one within Red Hat seems to be stopping to ask if Red Hat honored it's stated commitments on how governance would work. Was lying about how the governance process would work really expected to produce no backlash?
I completely believe Comnea that Chris Wright is committed to Open Source. I'm not questioning that. I am also not claiming there is a conspiracy.
I am saying Red Hat employees should feel an obligation to speak up about how this was an invalid governance board. Brian "Bex" Exelbierd and the rest of the RHEL team has no right to breach the firewall when making this decision. This is exactly what the community feared would happen.
That is why I am still mad even after carefully reading Comnea's blog post. I do not like being lied to. It makes me mad.
On 12/15/20 10:29 AM, Phelps, Matthew wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us.
Thanks for admitting that the reason Red Hat did this was financial. This BS about it being "a better way for Community input into RHEL" is just that, BS.
Can we stop with the charade that this is supposed to be a good thing for the CentOS community? It's not. It was never intended to be. It's a punishment for us getting "free Red Hat" all these years.
That is not a fair characterization of that decision.
Red Hat built up CentOS Stream as a stable LTS distribution. At some point, a rational person looking at the use cases for CentOS Stream and CentOS would naturally ask if it makes sense to continue putting human work hours into producing CentOS when it's considerably more difficult due to trying to reproduce the exact state of RHEL's build roots through reverse engineering, and for that effort, it's worse for 99.(some number of nines)% of users. There are no security updates for at least two months a year, even when they're needed. New features (rare as they are) roll out slower. Support for new hardware rolls out slower.
The vast majority of self-supported deployments would be better off choosing CentOS Stream, and having both makes that a lot less clear. Producing CentOS is very expensive, and provides no value for the vast majority of users. It provides no value to Red Hat, either. And saying so doesn't mean that Red Hat is making a cash grab.
Time that engineers spend rebuilding CentOS is time that they aren't improving CentOS Stream. (This is, very much, a zero-sum system.) In other words, continuing to make CentOS is a missed opportunity to make CentOS Stream better, which would have made RHEL better, which would have made CentOS better. That's what it means for CentOS to not work *for* Red Hat. Red Hat can provide a better self-supported distribution by discontinuing effort on CentOS.
On Tue, Dec 15, 2020 at 2:31 PM Gordon Messmer gordon.messmer@gmail.com wrote:
Red Hat built up CentOS Stream as a stable LTS distribution. At some point, a rational person looking at the use cases for CentOS Stream and CentOS would naturally ask if it makes sense to continue putting human work hours into producing CentOS when it's considerably more difficult due to trying to reproduce the exact state of RHEL's build roots through reverse engineering, and for that effort, it's worse for 99.(some number of nines)% of users. There are no security updates for at least two months a year, even when they're needed. New features (rare as they are) roll out slower. Support for new hardware rolls out slower.
The "considerable more effort to reverse-engineer" is a problem that was created by Red Hat. The 2014 "acquisition" of CentOS was supposed to resolve that. That RHEL is not reproducible is a Red Hat design problem which CentOS inherits. CentOS is not the cause of this problem.
If Red Hat was really concerned about making things easier, and better - they would make the binaries free, and sell support subscriptions. There doesn't need to be a separate CentOS distribution in this case. Instead, the subscription model is tied to the binaries, and the "secret sauce" and "do not use our brand" legal requirements which forces CentOS to be a re-engineered attempt to reproduce RHEL from source. The Red Hat brand could be everywhere, instead of creating the rather muddy "CentOS is upstream of RHEL", making RHEL be the derivative product? I think we all know why Red Hat won't do this.
The vast majority of self-supported deployments would be better off choosing CentOS Stream, and having both makes that a lot less clear. Producing CentOS is very expensive, and provides no value for the vast majority of users. It provides no value to Red Hat, either. And saying so doesn't mean that Red Hat is making a cash grab.
If this was true - then RHEL could also abandon minor releases. This is not how vendors certify that their products work with RHEL, and it will be a major problem for CentOS 8 Stream, just as it would be a major problem for an RHEL 8 Stream (which is what CentOS 8 Stream should be!). I don't believe that the CentOS board or the Red Hat management team are innocent and unaware of this. Messaging such as "if you require a stable release, you must buy RHEL" makes it clear what is really going on here. CentOS 8 is being eliminated as part of a determined business strategy. It is predatory. Allowing CentOS 8 Stream to exist only if CentOS 8 is destroyed, under legal threat, is essentially defeat. It is saying "you can only exist if you do not provide the same product we do".
Time that engineers spend rebuilding CentOS is time that they aren't improving CentOS Stream. (This is, very much, a zero-sum system.) In other words, continuing to make CentOS is a missed opportunity to make CentOS Stream better, which would have made RHEL better, which would have made CentOS better. That's what it means for CentOS to not work *for* Red Hat. Red Hat can provide a better self-supported distribution by discontinuing effort on CentOS.
Which users of CentOS are voting for CentOS Stream to replace CentOS Stable? Do you have a list? Is anybody other than Red Hat on this list? Do you think a poll of the users would show that they agree with this conclusion, or would it show the exact opposite? Are these users irrelevant to the discussion?
This whole thread is fairly ingenuine. It is a lot of justification after the fact, for CentOS 7 to be eliminated by 2024, and CentOS 8 to be eliminated by 2021, leaving nothing to replace it with except RHEL 8.
Enterprising individuals will fill this hole. Enterprising individuals created CentOS to fill this hole originally, and Enterprising individuals can do it again. Whether Rocky or something else. Most likely, Ubuntu will steal market-share that Red Hat will never get back. I also expect a number of Red Hat engineers to leave as a result of this news, as it will be evidence that the company they work for is not the company they agreed to join.
I have the ability to choose what gets deployed after EL 7 in our company. I was heading down the EL 8 path. I'm not qualifying which EL 7 or which EL 8, because we use many variants, including RHEL 7. After this event, I am forced to consider whether EL 8 has any place in our organization at all. The module system is already a bit of a mess, but I was willing to see it improve. Missing devel packages were annoying, but they could be rebuilt from source. This event that essentially destroys CentOS 8 is a sobering reminder that nothing is certain. When we decide what OS to use behind EL 7, we need to pick a distribution that will have a large ecosystem of vendors behind it. CentOS 7 helped build that ecosystem. The deprecation and elimination of CentOS 8 by 2021, is a huge "Dead End" sign that should not be ignored.
On Wed, Dec 16, 2020 at 2:18 AM Mark Mielke mark.mielke@gmail.com wrote:
On Tue, Dec 15, 2020 at 2:31 PM Gordon Messmer gordon.messmer@gmail.com wrote:
Red Hat built up CentOS Stream as a stable LTS distribution. At some point, a rational person looking at the use cases for CentOS Stream and CentOS would naturally ask if it makes sense to continue putting human work hours into producing CentOS when it's considerably more difficult due to trying to reproduce the exact state of RHEL's build roots through reverse engineering, and for that effort, it's worse for 99.(some number of nines)% of users. There are no security updates for at least two months a year, even when they're needed. New features (rare as they are) roll out slower. Support for new hardware rolls out slower.
The "considerable more effort to reverse-engineer" is a problem that was created by Red Hat. The 2014 "acquisition" of CentOS was supposed to resolve that. That RHEL is not reproducible is a Red Hat design problem which CentOS inherits. CentOS is not the cause of this problem.
If Red Hat was really concerned about making things easier, and better
- they would make the binaries free, and sell support subscriptions.
There doesn't need to be a separate CentOS distribution in this case. Instead, the subscription model is tied to the binaries, and the "secret sauce" and "do not use our brand" legal requirements which forces CentOS to be a re-engineered attempt to reproduce RHEL from source. The Red Hat brand could be everywhere, instead of creating the rather muddy "CentOS is upstream of RHEL", making RHEL be the derivative product? I think we all know why Red Hat won't do this.
The vast majority of self-supported deployments would be better off choosing CentOS Stream, and having both makes that a lot less clear. Producing CentOS is very expensive, and provides no value for the vast majority of users. It provides no value to Red Hat, either. And saying so doesn't mean that Red Hat is making a cash grab.
If this was true - then RHEL could also abandon minor releases. This is not how vendors certify that their products work with RHEL, and it will be a major problem for CentOS 8 Stream, just as it would be a major problem for an RHEL 8 Stream (which is what CentOS 8 Stream should be!). I don't believe that the CentOS board or the Red Hat management team are innocent and unaware of this. Messaging such as "if you require a stable release, you must buy RHEL" makes it clear what is really going on here. CentOS 8 is being eliminated as part of a determined business strategy. It is predatory. Allowing CentOS 8 Stream to exist only if CentOS 8 is destroyed, under legal threat, is essentially defeat. It is saying "you can only exist if you do not provide the same product we do".
Well, I would certainly be happy if Red Hat dropped minor releases from RHEL. Officially, Red Hat advises ISVs to not target specific minor releases already, and that compatibility is assured by following reasonable practices and the documentation around ABI guarantees in RHEL.
Minor releases are a *major* headache, and I'd be happier to see them apply Stream methodology to RHEL with RHEL 9 and just respin install trees and ISOs regularly for easing deployments like CentOS Stream will be doing.
Time that engineers spend rebuilding CentOS is time that they aren't improving CentOS Stream. (This is, very much, a zero-sum system.) In other words, continuing to make CentOS is a missed opportunity to make CentOS Stream better, which would have made RHEL better, which would have made CentOS better. That's what it means for CentOS to not work *for* Red Hat. Red Hat can provide a better self-supported distribution by discontinuing effort on CentOS.
Which users of CentOS are voting for CentOS Stream to replace CentOS Stable? Do you have a list? Is anybody other than Red Hat on this list? Do you think a poll of the users would show that they agree with this conclusion, or would it show the exact opposite? Are these users irrelevant to the discussion?
This whole thread is fairly ingenuine. It is a lot of justification after the fact, for CentOS 7 to be eliminated by 2024, and CentOS 8 to be eliminated by 2021, leaving nothing to replace it with except RHEL 8.
Enterprising individuals will fill this hole. Enterprising individuals created CentOS to fill this hole originally, and Enterprising individuals can do it again. Whether Rocky or something else. Most likely, Ubuntu will steal market-share that Red Hat will never get back. I also expect a number of Red Hat engineers to leave as a result of this news, as it will be evidence that the company they work for is not the company they agreed to join.
I have the ability to choose what gets deployed after EL 7 in our company. I was heading down the EL 8 path. I'm not qualifying which EL 7 or which EL 8, because we use many variants, including RHEL 7. After this event, I am forced to consider whether EL 8 has any place in our organization at all. The module system is already a bit of a mess, but I was willing to see it improve. Missing devel packages were annoying, but they could be rebuilt from source. This event that essentially destroys CentOS 8 is a sobering reminder that nothing is certain. When we decide what OS to use behind EL 7, we need to pick a distribution that will have a large ecosystem of vendors behind it. CentOS 7 helped build that ecosystem. The deprecation and elimination of CentOS 8 by 2021, is a huge "Dead End" sign that should not be ignored.
I can say as someone operating as an ISV that I'm very confident in CentOS Stream 8.
On Wed, Dec 16, 2020 at 2:29 AM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Dec 16, 2020 at 2:18 AM Mark Mielke mark.mielke@gmail.com wrote:
The vast majority of self-supported deployments would be better off choosing CentOS Stream, and having both makes that a lot less clear. Producing CentOS is very expensive, and provides no value for the vast majority of users. It provides no value to Red Hat, either. And saying so doesn't mean that Red Hat is making a cash grab.
If this was true - then RHEL could also abandon minor releases. This is not how vendors certify that their products work with RHEL, and it will be a major problem for CentOS 8 Stream, just as it would be a major problem for an RHEL 8 Stream (which is what CentOS 8 Stream should be!). I don't believe that the CentOS board or the Red Hat management team are innocent and unaware of this. Messaging such as "if you require a stable release, you must buy RHEL" makes it clear what is really going on here. CentOS 8 is being eliminated as part of a determined business strategy. It is predatory. Allowing CentOS 8 Stream to exist only if CentOS 8 is destroyed, under legal threat, is essentially defeat. It is saying "you can only exist if you do not provide the same product we do".
Well, I would certainly be happy if Red Hat dropped minor releases from RHEL. Officially, Red Hat advises ISVs to not target specific minor releases already, and that compatibility is assured by following reasonable practices and the documentation around ABI guarantees in RHEL.
The problem is that 10 years is a very long time to require both source and binary, forwards and backwards compatibility, and without point releases - you essentially have no ability to introduce changes to interfaces. And this includes, no ability to upgrade important packages. It might be possible with some system whereby packages are built statically, or multiple versions of libraries (both source and binaries) are installed in parallel, or containers are used to run programs in the environments they were designed for. But, that's not what we have with RHEL or CentOS today. What we have today, is a system where feature changes are introduced in new minor releases, and important patches are introduced in channels. Point releases exist for an important reason. They might introduce effort for contributors, but they also reduce risk for users, and they provide a baseline for certification.
I think *if* RHEL abandoned minor releases, then the CentOS problem would disappear as a problem on its own. However, this would then re-introduce the problem that:
1. CentOS 8 would be a direct competitor for RHEL 8, which is very likely the reason why this change is being introduced in the first place. 2. Users might not agree. RHEL 8 without point releases is not a valid option, for the same reason that CentOS 8 without point releases is not an option. Feature changes could arrive any day, and break Enterprises every day.
I think abandoning point releases, is basically abandoning RHEL's bread and butter. I can't see it happening. It doesn't matter what ISV think or which path is easier for ISV.
Also, I don't mean that judgmentally on you... it also doesn't matter what I think. My contribution here is only to ensure a few additional points are captured. I have no confidence that my contribution will change the outcome. :-)
On Tue, Dec 15, 2020 at 1:00 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 10:23 AM sankarshan sankarshan.mukhopadhyay@gmail.com wrote:
I am going to snip a lot of this note and respond to a specific part.
On Tue, 15 Dec 2020 at 21:33, Ljubomir Ljubojevic centos@plnet.rs wrote:
It looks like "fatherlinux" chose to not allow my comment. I see he allowed some other comments and replied to some other, but mine is missing, so I will post it here:
[snipped]
To conclude: When RH employed CentOS Core team in 2014 they promised that nothing will change for "CentOS Linux". According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of "CentOS Linux" happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided "to vote for changes unanimously". Red Had flexed it's muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat. Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with "free RHEL licensees" for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone. I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
The RHT - CentOS bits happened in 2014. I am certain that the statements from the CentOS team were made with the best intentions and were not meant to masquerade anything. Holding the entire phenomenal CentOS crew (all of whom have spent long years building this community with love) to a statement made way back in 2014 seems and is a bit unfair. Realities change and it would be reasonably obvious that strategic plans determined CentOS-as-upstream-of-RHEL to be the need of the hour rather than continue with the focus of CentOS as it has been.
Please pause for a moment and think about the individuals being denigrated on the lists. These are not the evil, malicious and villainous characters they are being demonized as. For what it is worth we've likely met them in person, shared a joke or a beverage. I doubt they like the outcome any more than we in the community do.
Being kind, being respectful and being an ally does not take a lot. Let's be that while we find how best to preserve our interests, businesses and energies.
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Note: I was not in the room when the voting happened. I was involved in the negotiations. The board had a tremendous impact on helping Red Hat better understand some things that needed to happen in CentOS Stream. For example, versioning it and supporting it through the full support cycle of RHEL instead of what stream was before (a sort of continuous stream with one year overlap for migrations, etc). The Board is expecting things out of CentOS Stream and we expect them to hold us to that.
It's easy to say "The Board is full of Red Hatters and they did this." But I think we all know that's not the case, some of the Red Hatters on the board were as fierce a defender of the existing CentOS community as one could possibly be. The board could have voted this down. Red Hat could have dissolved the board (as I understand the voting rules). But that didn't happen. Both sides of this came to an agreement that we - together - could live with and that represented a positive future for CentOS. A very very different future for sure, but a positive one.
I am not angry at all the folks who kept centos alive. I am not even angry at redhat. What I am disappointed is that this decision felt it was like one of those laws passed late at night so there was no way for people to comment on it. I feel this, to quote Steve Jobs, changes everything, so if "CentOS" stands for "Community Enterprise OS" instead of "that strange growth I found on my left butt cheek in a midsummer evening," the Community should be informed what were the options (and consequences) and let them have a say in the decision process.
That is it. I am just disappointed in the process. However, this is a done deal. We will move on, adapt, and prevail.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/15/20 6:59 PM, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
I can only analyze in hindsight, no way around it.
I thought of that too, but there is a little snag. If Red Hat was unable to pay for "CentOS Linux" rebuild, what effectively PREVENTED Red Hat to announce that they can not continue to finance rebuild project, and to ASK/ALLOW FOR DONATIONS that would be used for rebuild project?
If Red Hat REALLY wanted (or still wants!) for CentOS project to continue rebuilding RHEL clone, all it had to due is announce that donations in hardware or manpower are needed for it to continue.
Here is a way out!:
Even NOW it is still NOT TOO LATE to decide that Red hat employees that work in CentOS project are free to spend their free time rebuilding clone, and can use hardware donated by 3rd parties to organize rebuild.
Hardware is going to be donated for rebuild anyway for Rocky and Lenix projects, why not enable people and organizations invested in continued life of RHEL clone to pull RHEL out of the "dark chazam" where Gray Wizard fell :-) Effort can be separated from "CentOS Stream" and "CentOS Linux" can continue it's life...
On 15/12/2020 17:59, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Thank you for this clarification although it was fairly apparent to everyone what the driver was behind this change.
I'd like to thank Red Hat for supporting the CentOS Project from 2014 to 2020. You did a good thing by stepping in to save the project from disintegration back in 2014. Thanks for that, CentOS would probably have survived without you but you did the right thing and stepped up when you were needed.
However...
While Red Hat may *legally* own the CentOS Project, I do not believe you are *morally* entitled to do what you have done. CentOS is not just about the project and the contributors to it. It's more than that. It has millions of users, so many that no-one really knows how many there are. Lots of those users may be large corporations "freeloading" as Red Hat probably see it but others, those are small users running single machines or just a few. Those users are *your* future.
You (Red Hat) made a lot of promises both in 2014 and as late as last year when Chris Wright said something along the lines of classic CentOS Linux is not going anywhere. It's all very well to say that things change, well of course they do, but when they do, you have an obligation to live up to your promises and the recent actions were in no way doing that.
I believe the correct action for Red Hat to have taken would have been to say "we have decided that we no longer wish to fund the CentOS Project as it no longer aligns with our business purposes. So, in order not to let down the millions of users of CentOS Linux, we have decided to set up a foundation and donate the trade marks and domain names (that we acquired for almost nothing)".
With a decent legal founding, you could have made it takeover proof so that none of your competitors could acquire it. You could have done this and asked a number of the larger companies that have CentOS as part of their portfolio to sponsor the foundation - the Googles/AWS/OVH/cpanel's of this world could easily have stepped up and funded a FTE or 2 by donating to the foundation and you could have transferred some or all of the existing people who work on CentOS to that foundation and let *them* run it. Those hosting companies spin up new CentOS instances all the time and a cent or two donation on each instance would most likely fund most of what's required. And the people who are now scrambling around attempting to set up new hardware and build environments, they could be supporting the CentOS Linux Foundation instead.
The fact that you decided to take CentOS Linux out the back and shoot it in the head is a betrayal of your company's promises over the last 6 or 7 years. It's exactly what everyone was afraid of when Red Hat took over CentOS in 2014 and despite numerous questions, you all said "no no, it's safe with us". Some of us remember those days and arguing with people about whether it was a good thing or not and a lot of us said "Trust Red Hat, see what they do, look at their actions not their words". Well we did.
You should rename CentOS Stream to Red Hat Stream Linux (RHSL) and remove CentOS from the Red Hat family altogether. Donate the trade marks and logos and domain names and the tooling needed to produce CentOS Linux. Set up a foundation. Get the big players who offer CentOS to users to help fund the foundation. Ask the employees who work on CentOS on a daily basis if they'd like to stay with Red Hat or transfer to the new foundation. Find some way in which users can contribute to the foundation and ensure its future.
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free.
You can say that you think people are coming round to this. I do not agree. I have read all of the feedback on IRC, all of the feedback on the CentOS forums, all the feedback on the mailing lists. This is *not* a popular change. It's tarnishing and poisoning Red Hat's reputation and until it's addressed it will continue to do so. You can help to fix this before Red Hat becomes tarred with the same brush as that other big company with the big red logo and the not so great reputation. This is NOT just a $$$ decision, it has other ramifications and right now, Red Hat are the bad guys and will remain so until this is addressed.
You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
Trevor Hemsley
On 15/12/2020 19:39, Trevor Hemsley via CentOS-devel wrote:
On 15/12/2020 17:59, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Thank you for this clarification although it was fairly apparent to everyone what the driver was behind this change.
I'd like to thank Red Hat for supporting the CentOS Project from 2014 to 2020. You did a good thing by stepping in to save the project from disintegration back in 2014. Thanks for that, CentOS would probably have survived without you but you did the right thing and stepped up when you were needed.
However...
While Red Hat may *legally* own the CentOS Project, I do not believe you are *morally* entitled to do what you have done. CentOS is not just about the project and the contributors to it. It's more than that. It has millions of users, so many that no-one really knows how many there are. Lots of those users may be large corporations "freeloading" as Red Hat probably see it but others, those are small users running single machines or just a few. Those users are *your* future.
You (Red Hat) made a lot of promises both in 2014 and as late as last year when Chris Wright said something along the lines of classic CentOS Linux is not going anywhere. It's all very well to say that things change, well of course they do, but when they do, you have an obligation to live up to your promises and the recent actions were in no way doing that.
I believe the correct action for Red Hat to have taken would have been to say "we have decided that we no longer wish to fund the CentOS Project as it no longer aligns with our business purposes. So, in order not to let down the millions of users of CentOS Linux, we have decided to set up a foundation and donate the trade marks and domain names (that we acquired for almost nothing)".
With a decent legal founding, you could have made it takeover proof so that none of your competitors could acquire it. You could have done this and asked a number of the larger companies that have CentOS as part of their portfolio to sponsor the foundation - the Googles/AWS/OVH/cpanel's of this world could easily have stepped up and funded a FTE or 2 by donating to the foundation and you could have transferred some or all of the existing people who work on CentOS to that foundation and let *them* run it. Those hosting companies spin up new CentOS instances all the time and a cent or two donation on each instance would most likely fund most of what's required. And the people who are now scrambling around attempting to set up new hardware and build environments, they could be supporting the CentOS Linux Foundation instead.
The fact that you decided to take CentOS Linux out the back and shoot it in the head is a betrayal of your company's promises over the last 6 or 7 years. It's exactly what everyone was afraid of when Red Hat took over CentOS in 2014 and despite numerous questions, you all said "no no, it's safe with us". Some of us remember those days and arguing with people about whether it was a good thing or not and a lot of us said "Trust Red Hat, see what they do, look at their actions not their words". Well we did.
You should rename CentOS Stream to Red Hat Stream Linux (RHSL) and remove CentOS from the Red Hat family altogether. Donate the trade marks and logos and domain names and the tooling needed to produce CentOS Linux. Set up a foundation. Get the big players who offer CentOS to users to help fund the foundation. Ask the employees who work on CentOS on a daily basis if they'd like to stay with Red Hat or transfer to the new foundation. Find some way in which users can contribute to the foundation and ensure its future.
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free.
You can say that you think people are coming round to this. I do not agree. I have read all of the feedback on IRC, all of the feedback on the CentOS forums, all the feedback on the mailing lists. This is *not* a popular change. It's tarnishing and poisoning Red Hat's reputation and until it's addressed it will continue to do so. You can help to fix this before Red Hat becomes tarred with the same brush as that other big company with the big red logo and the not so great reputation. This is NOT just a $$$ decision, it has other ramifications and right now, Red Hat are the bad guys and will remain so until this is addressed.
You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
Trevor Hemsley
Well said Trevor.
Copied to centos-questions@redhat.com
On Tue, Dec 15, 2020 at 11:40 AM Trevor Hemsley via CentOS-devel < centos-devel@centos.org> wrote:
On 15/12/2020 17:59, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Thank you for this clarification although it was fairly apparent to everyone what the driver was behind this change.
I'd like to thank Red Hat for supporting the CentOS Project from 2014 to 2020. You did a good thing by stepping in to save the project from disintegration back in 2014. Thanks for that, CentOS would probably have survived without you but you did the right thing and stepped up when you were needed.
However...
While Red Hat may *legally* own the CentOS Project, I do not believe you are *morally* entitled to do what you have done. CentOS is not just about the project and the contributors to it. It's more than that. It has millions of users, so many that no-one really knows how many there are. Lots of those users may be large corporations "freeloading" as Red Hat probably see it but others, those are small users running single machines or just a few. Those users are *your* future.
You (Red Hat) made a lot of promises both in 2014 and as late as last year when Chris Wright said something along the lines of classic CentOS Linux is not going anywhere. It's all very well to say that things change, well of course they do, but when they do, you have an obligation to live up to your promises and the recent actions were in no way doing that.
I believe the correct action for Red Hat to have taken would have been to say "we have decided that we no longer wish to fund the CentOS Project as it no longer aligns with our business purposes. So, in order not to let down the millions of users of CentOS Linux, we have decided to set up a foundation and donate the trade marks and domain names (that we acquired for almost nothing)".
With a decent legal founding, you could have made it takeover proof so that none of your competitors could acquire it. You could have done this and asked a number of the larger companies that have CentOS as part of their portfolio to sponsor the foundation - the Googles/AWS/OVH/cpanel's of this world could easily have stepped up and funded a FTE or 2 by donating to the foundation and you could have transferred some or all of the existing people who work on CentOS to that foundation and let *them* run it. Those hosting companies spin up new CentOS instances all the time and a cent or two donation on each instance would most likely fund most of what's required. And the people who are now scrambling around attempting to set up new hardware and build environments, they could be supporting the CentOS Linux Foundation instead.
The fact that you decided to take CentOS Linux out the back and shoot it in the head is a betrayal of your company's promises over the last 6 or 7 years. It's exactly what everyone was afraid of when Red Hat took over CentOS in 2014 and despite numerous questions, you all said "no no, it's safe with us". Some of us remember those days and arguing with people about whether it was a good thing or not and a lot of us said "Trust Red Hat, see what they do, look at their actions not their words". Well we did.
You should rename CentOS Stream to Red Hat Stream Linux (RHSL) and remove CentOS from the Red Hat family altogether. Donate the trade marks and logos and domain names and the tooling needed to produce CentOS Linux. Set up a foundation. Get the big players who offer CentOS to users to help fund the foundation. Ask the employees who work on CentOS on a daily basis if they'd like to stay with Red Hat or transfer to the new foundation. Find some way in which users can contribute to the foundation and ensure its future.
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free.
You can say that you think people are coming round to this. I do not agree. I have read all of the feedback on IRC, all of the feedback on the CentOS forums, all the feedback on the mailing lists. This is *not* a popular change. It's tarnishing and poisoning Red Hat's reputation and until it's addressed it will continue to do so. You can help to fix this before Red Hat becomes tarred with the same brush as that other big company with the big red logo and the not so great reputation. This is NOT just a $$$ decision, it has other ramifications and right now, Red Hat are the bad guys and will remain so until this is addressed.
You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
Trevor Hemsley
I agree 100% with what Trevor so marvelously said here.
On Tue, Dec 15, 2020 at 1:41 PM Trevor Hemsley trevor.hemsley@ntlworld.com wrote:
On 15/12/2020 17:59, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Thank you for this clarification although it was fairly apparent to everyone what the driver was behind this change.
I'd like to thank Red Hat for supporting the CentOS Project from 2014 to 2020. You did a good thing by stepping in to save the project from disintegration back in 2014. Thanks for that, CentOS would probably have survived without you but you did the right thing and stepped up when you were needed.
However...
While Red Hat may *legally* own the CentOS Project, I do not believe you are *morally* entitled to do what you have done. CentOS is not just about the project and the contributors to it. It's more than that. It has millions of users, so many that no-one really knows how many there are. Lots of those users may be large corporations "freeloading" as Red Hat probably see it but others, those are small users running single machines or just a few. Those users are *your* future.
You (Red Hat) made a lot of promises both in 2014 and as late as last year when Chris Wright said something along the lines of classic CentOS Linux is not going anywhere. It's all very well to say that things change, well of course they do, but when they do, you have an obligation to live up to your promises and the recent actions were in no way doing that.
I believe the correct action for Red Hat to have taken would have been to say "we have decided that we no longer wish to fund the CentOS Project as it no longer aligns with our business purposes. So, in order not to let down the millions of users of CentOS Linux, we have decided to set up a foundation and donate the trade marks and domain names (that we acquired for almost nothing)".
With a decent legal founding, you could have made it takeover proof so that none of your competitors could acquire it. You could have done this and asked a number of the larger companies that have CentOS as part of their portfolio to sponsor the foundation - the Googles/AWS/OVH/cpanel's of this world could easily have stepped up and funded a FTE or 2 by donating to the foundation and you could have transferred some or all of the existing people who work on CentOS to that foundation and let *them* run it. Those hosting companies spin up new CentOS instances all the time and a cent or two donation on each instance would most likely fund most of what's required. And the people who are now scrambling around attempting to set up new hardware and build environments, they could be supporting the CentOS Linux Foundation instead.
The fact that you decided to take CentOS Linux out the back and shoot it in the head is a betrayal of your company's promises over the last 6 or 7 years. It's exactly what everyone was afraid of when Red Hat took over CentOS in 2014 and despite numerous questions, you all said "no no, it's safe with us". Some of us remember those days and arguing with people about whether it was a good thing or not and a lot of us said "Trust Red Hat, see what they do, look at their actions not their words". Well we did.
You should rename CentOS Stream to Red Hat Stream Linux (RHSL) and remove CentOS from the Red Hat family altogether. Donate the trade marks and logos and domain names and the tooling needed to produce CentOS Linux. Set up a foundation. Get the big players who offer CentOS to users to help fund the foundation. Ask the employees who work on CentOS on a daily basis if they'd like to stay with Red Hat or transfer to the new foundation. Find some way in which users can contribute to the foundation and ensure its future.
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free.
You can say that you think people are coming round to this. I do not agree. I have read all of the feedback on IRC, all of the feedback on the CentOS forums, all the feedback on the mailing lists. This is *not* a popular change. It's tarnishing and poisoning Red Hat's reputation and until it's addressed it will continue to do so. You can help to fix this before Red Hat becomes tarred with the same brush as that other big company with the big red logo and the not so great reputation. This is NOT just a $$$ decision, it has other ramifications and right now, Red Hat are the bad guys and will remain so until this is addressed.
You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
I'm in this weird position where I'm regularly hearing from people that thought that Red Hat made some sort of "We'll never change and CentOS Linux will be around forever" announcement. I'd suggest everyone go back and re-read the original press release (I was not involved with the original agreement) - https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-force...
You can nitpick at words, or take a quote out of context. But don't be naive and pretend we had some grand plan for all of this from the beginning. Just like anyone, Red Hat changes and makes decisions based on the best information we have at the time. CentOS Linux made sense in 2014, it doesn't make sense in 2020.
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users. And I know there are community members out there who are actively contributing time on QE as you did Trevor. You are a very small minority on this project and I hope we can win you over in CentOS Stream. As for the rest of you, where were you?
And sure, we could have turned CentOS back over to some non-Red Hat foundation. But the fact is contrary to popular belief, we actually like the engineers that work on CentOS, we like many of the users who have cultivated relationships with Red Hat over the years. We intend on going forward with a healthy, mutually beneficial relationship there. If that's not you, and you're ready to leave - I think that's unfortunate but I understand.
-Mike
Trevor Hemsley
On Dec 15, 2020, at 3:50 PM, Mike McGrath mmcgrath@redhat.com wrote:
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff).
The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
And sure, we could have turned CentOS back over to some non-Red Hat foundation. But the fact is contrary to popular belief, we actually like the engineers that work on CentOS, we like many of the users who have cultivated relationships with Red Hat over the years.
That would still be possible spinning CentOS out to its own foundation.
We intend on going forward with a healthy, mutually beneficial relationship there.
At this point, it is likely in RedHat's best interest to be crystal clear by their expectations on what constitutes a "mutually beneficial relationship" in their eyes. Cause back in 2014, having CentOS under RedHat but continuing business as usual was seen as a "mutually beneficial relationship". So what happens if today's mutually beneficial relationship is not tomorrows?
On Tue, Dec 15, 2020 at 1:03 PM Jim Jagielski jim@jimjag.com wrote:
On Dec 15, 2020, at 3:50 PM, Mike McGrath mmcgrath@redhat.com wrote:
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff)
The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
I know that "same package set that RH ships to customers" has been an effective shorthand for "good enough for me," and that our assurances of trusting the automated ci and the power of community collaboration may be inspiring less trust right now, but I am confident that CentOS Stream will be a good, stable distro for current users. My team that hosts infra for various open source projects has already begun converting some of our production services from CentOS 8 to CentOS Stream.
How will Stream work for the numerous projects and tools (some open source some other-wise) that have targeted RHEL/CentOS X.Y versions for compatibility? Things such as Lustre, IBM Spectrum Scale, OpenZFS, Mellanox OFED, Intel Omni-Path, etc. (these are ones I’m very familiar with being in HPC, but I’m sure there are others). Will there be mechanisms within Stream to set things at a release version of RHEL (maybe via a set of repos?) so that Stream can be used without having to carefully manage package versions via yum/dnf version lock, etc?
A lot of the above tools/projects don’t bring support for an X.Y version of RHEL until 1-4 weeks after the RHEL release drops, which has historically worked out well anyway for CentOS because of its inherent delay also. For instance I’m not sure I could get Mellanox drivers for the 8.4 kernel (that I understand is currently in stream). I’ve been using most of the above mentioned tools for the better part of the past 7 years and they don’t survive across an update from CentOS X.Y —> X.(Y+1) very often, if at all.
JD
On Dec 15, 2020, at 17:30, Jason Brooks jbrooks@redhat.com wrote:
On Tue, Dec 15, 2020 at 1:03 PM Jim Jagielski jim@jimjag.com wrote:
On Dec 15, 2020, at 3:50 PM, Mike McGrath mmcgrath@redhat.com wrote:
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff)
The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
I know that "same package set that RH ships to customers" has been an effective shorthand for "good enough for me," and that our assurances of trusting the automated ci and the power of community collaboration may be inspiring less trust right now, but I am confident that CentOS Stream will be a good, stable distro for current users. My team that hosts infra for various open source projects has already begun converting some of our production services from CentOS 8 to CentOS Stream.
-- Jason Brooks He/Him Manager, Community Architects & Infrastructure Red Hat Open Source Program Office (OSPO) https://community.redhat.com | https://osci.io
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Dec 15, 2020 at 4:21 PM JD Maloney jdphotography7@gmail.com wrote:
How will Stream work for the numerous projects and tools (some open source some other-wise) that have targeted RHEL/CentOS X.Y versions for compatibility? Things such as Lustre, IBM Spectrum Scale, OpenZFS, Mellanox OFED, Intel Omni-Path, etc. (these are ones I’m very familiar with being in HPC, but I’m sure there are others). Will there be mechanisms within Stream to set things at a release version of RHEL (maybe via a set of repos?) so that Stream can be used without having to carefully manage package versions via yum/dnf version lock, etc?
A lot of the above tools/projects don’t bring support for an X.Y version of RHEL until 1-4 weeks after the RHEL release drops, which has historically worked out well anyway for CentOS because of its inherent delay also. For instance I’m not sure I could get Mellanox drivers for the 8.4 kernel (that I understand is currently in stream). I’ve been using most of the above mentioned tools for the better part of the past 7 years and they don’t survive across an update from CentOS X.Y —> X.(Y+1) very often, if at all.
There is a connection between point release orientation and chronic delays adapting software to updates to the underlying OS. Anybody who has been in ops for more than a few months takes it as a given and it's easy to rattle off a litany of reasons why it can't be any other way. If we stick with point releases, that might be true. Yet, if those same vendors are participants in a continuous development model, those delays could be reduced, perhaps even eliminated. If this is a thing members of the community want to develop, it's probably the kind of thing CentOS Stream should enable.
On Tue, Dec 15, 2020 at 11:06 PM Brendan Conoboy blc@redhat.com wrote:
On Tue, Dec 15, 2020 at 4:21 PM JD Maloney jdphotography7@gmail.com wrote:
How will Stream work for the numerous projects and tools (some open source some other-wise) that have targeted RHEL/CentOS X.Y versions for compatibility? Things such as Lustre, IBM Spectrum Scale, OpenZFS, Mellanox OFED, Intel Omni-Path, etc. (these are ones I’m very familiar with being in HPC, but I’m sure there are others). Will there be mechanisms within Stream to set things at a release version of RHEL (maybe via a set of repos?) so that Stream can be used without having to carefully manage package versions via yum/dnf version lock, etc?
A lot of the above tools/projects don’t bring support for an X.Y version of RHEL until 1-4 weeks after the RHEL release drops, which has historically worked out well anyway for CentOS because of its inherent delay also. For instance I’m not sure I could get Mellanox drivers for the 8.4 kernel (that I understand is currently in stream). I’ve been using most of the above mentioned tools for the better part of the past 7 years and they don’t survive across an update from CentOS X.Y —> X.(Y+1) very often, if at all.
There is a connection between point release orientation and chronic delays adapting software to updates to the underlying OS. Anybody who has been in ops for more than a few months takes it as a given and it's easy to rattle off a litany of reasons why it can't be any other way. If we stick with point releases, that might be true. Yet, if those same vendors are participants in a continuous development model, those delays could be reduced, perhaps even eliminated. If this is a thing members of the community want to develop, it's probably the kind of thing CentOS Stream should enable.
At least from the OpenZFS perspective, I expect there to be no significant issues adapting to CentOS Stream 8 once AMIs are available. The CI infrastructure is able to continuously test Fedora kernel updates, which move much more quickly than CentOS/RHEL ones. The only reason it hasn't been tracking CentOS Stream 8 so far is because of the lack of images to use.
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks jbrooks@redhat.com wrote:
On Tue, Dec 15, 2020 at 1:03 PM Jim Jagielski jim@jimjag.com wrote:
On Dec 15, 2020, at 3:50 PM, Mike McGrath mmcgrath@redhat.com wrote:
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff)
The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
I know that "same package set that RH ships to customers" has been an effective shorthand for "good enough for me," and that our assurances of trusting the automated ci and the power of community collaboration may be inspiring less trust right now, but I am confident that CentOS Stream will be a good, stable distro for current users. My team that hosts infra for various open source projects has already begun converting some of our production services from CentOS 8 to CentOS Stream.
Honestly, I think CentOS Stream will be amazing. There's a lot of amazing potential here, and the opportunity to influence RHEL/CentOS itself through CentOS Stream is something that shouldn't be underestimated. From my perspective, CentOS Stream is a more exciting platform because it's making CentOS itself into more of a true community project.
However, the real kick in the teeth was the lifespan being cut in half with CentOS Stream 8. That is where I feel most of the trust was damaged, because it violated the community contract that has been established as a norm for the past decade and a half. From my perspective as a contributing distribution engineer working in CentOS, that change is disruptive on a level in which it's hard to recover from. Strictly speaking, from a promotionary point of view for using and relying on it, this is a downgrade. And I *want* to be able to promote this as the truly *community* enterprise Linux platform where the *community* can work in partnership with Red Hat to develop the greatest enterprise Linux platform to build anything and everything.
I think most folks know by now that I'm not talking out of my rear when I say that I'm willing to put work in to help with this, and I have even already contributed to CentOS Stream because I think it's awesome. But the lifespan of CentOS Stream 8 not matching Red Hat Enterprise Linux 8 makes CentOS *itself* lose one of the differentiators that made it attractive over openSUSE Leap and Ubuntu LTS. CentOS Stream with the 10 year lifespan is an absolute no-brainer compared to openSUSE Leap or Ubuntu LTS that would enhance the goal of making Red Hat platforms the default Linux platform everywhere (dev, CI, and production).
I'm also willing to help with developing strategies for supporting on-ramps from CentOS Stream to Red Hat Enterprise Linux. I have been testing convert2rhel, given feedback in code reviews for the tool, and so on.
I want to make the story better between CentOS and RHEL. I want to help Red Hat make RHEL more appealing to more audiences. There is a lot of value that you get with a subscription for Red Hat Enterprise Linux that you can't get with CentOS (such as certifications and assurances related to that, Red Hat Insights, kernel live patching, engineering support and product assistance, etc.), and highlighting those things and reinforcing the value of those things to the community at large would probably help with driving growth here.
I strongly believe a combination of an excellent platform and a good story for upgrading to supported tiers would make the ecosystem more attractive for all. Because to me, the trade for open source software is that you give either your time+effort or money. If you can't give time+effort, then paying money to someone who will is a fair trade. Making that easy and obvious would help things considerably, in my view.
-- 真実はいつも一つ!/ Always, there's only one truth!
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks jbrooks@redhat.com wrote:
[...]
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
Isn't that what fedora is used for?
On Tue, Dec 15, 2020 at 5:42 PM Mauricio Tavares raubvogel@gmail.com wrote:
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks jbrooks@redhat.com wrote:
[...]
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
Isn't that what fedora is used for?
Fedora is used as a starting point for major release alphas and betas, i.e., 7.0 Beta, 8.0 Beta, etc. After the major release beta comes out all automatic connection between Fedora and RHEL ceases. RHEL 8.2 was based on 8.1 + upstream changes, 8.1 was based on 8.0 plus upstream changes. There simply hasn't been a place where people outside the Red Hat firewall can see, use, and influence the direction of the next minor release, as it is being created. That's what Stream is meant to do.
On 12/15/2020 6:18 PM, Brendan Conoboy wrote:
On Tue, Dec 15, 2020 at 5:42 PM Mauricio Tavares <raubvogel@gmail.com mailto:raubvogel@gmail.com> wrote:
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks <jbrooks@redhat.com <mailto:jbrooks@redhat.com>> wrote: > [...] > No. I was on that team too, and growing CentOS beyond just consumption > and into contribution was something we emphasized throughout. Our > primary intent, the reason the whole thing got started, was that we > needed to provide our layered projects with a slow-moving community > distro to layer atop. That's why we put so much effort into the SIGs, > and into opening up the build processes and tools. Even with that work > done, until we opened up RHEL development itself, contributions to the > core of CentOS were basically blocked. Now, in addition to the layered > project need, which hasn't gone away, we need a distro to open up RHEL > development, and CentOS Stream is that distro. > Isn't that what fedora is used for?
Fedora is used as a starting point for major release alphas and betas, i.e., 7.0 Beta, 8.0 Beta, etc. After the major release beta comes out all automatic connection between Fedora and RHEL ceases. RHEL 8.2 was based on 8.1 + upstream changes, 8.1 was based on 8.0 plus upstream changes. There simply hasn't been a place where people outside the Red Hat firewall can see, use, and influence the direction of the next minor release, as it is being created. That's what Stream is meant to do.
Minor release updates very rarely have a need for significant influence, but I'm unsure how this is supposed to relate to actual RHEL minor version Beta releases. Will there still even be RHEL 8.x Beta Releases?
But beyond that, the above statement does not seem to be compatible with the following:
On 12/15/2020 3:35 PM, Johnny Hughes wrote:
You guys keep calling it beta .. it is not.
The RHEL team is not grabbing brand new software (like the do in Rawhide, for example) and trying to roll that into RHEL. They are going to do one of three type of updates.
A security update
A bugfix update.
An Enhancement update.
For #1 and #2 .. you want those rolled in and you want them rolled in ASAP. RHEAs do not make up that many of the updates. You are getting these after QA testing a couple months early at most.
There's no meaningful "influence" at this point beyond filing BZs about things that break. While that's certainly better than nothing, if RedHat wanted to accept bugs from non-RHEL binaries, especially in-house rebuilds explicitly targeting 100% binary compatibility, it could easily have done so at any point in the past.
There's still a conflation of upstream and downstream here. The "direction" of any aspect of RHEL is already going to be quite set by the time it gets into CentOS Stream... as is appropriate for a downstream.
-jc
On Tue, Dec 15, 2020 at 6:42 PM Japheth Cleaver cleaver@terabithia.org wrote:
On 12/15/2020 6:18 PM, Brendan Conoboy wrote:
On Tue, Dec 15, 2020 at 5:42 PM Mauricio Tavares raubvogel@gmail.com wrote:
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks jbrooks@redhat.com wrote:
[...]
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
Isn't that what fedora is used for?
Fedora is used as a starting point for major release alphas and betas, i.e., 7.0 Beta, 8.0 Beta, etc. After the major release beta comes out all automatic connection between Fedora and RHEL ceases. RHEL 8.2 was based on 8.1 + upstream changes, 8.1 was based on 8.0 plus upstream changes. There simply hasn't been a place where people outside the Red Hat firewall can see, use, and influence the direction of the next minor release, as it is being created. That's what Stream is meant to do.
Minor release updates very rarely have a need for significant influence, but I'm unsure how this is supposed to relate to actual RHEL minor version Beta releases. Will there still even be RHEL 8.x Beta Releases?
There'll be more opportunity to shape things once the 9 stream starts, during that stream 8 / stream 9 overlap period. Fedora is the place for big changes, but it can be a long wait for that stuff to get into RHEL, so Stream helps address this.
On Tue, Dec 15, 2020 at 10:06 PM Jason Brooks jbrooks@redhat.com wrote:
On Tue, Dec 15, 2020 at 6:42 PM Japheth Cleaver cleaver@terabithia.org wrote:
On 12/15/2020 6:18 PM, Brendan Conoboy wrote:
On Tue, Dec 15, 2020 at 5:42 PM Mauricio Tavares raubvogel@gmail.com wrote:
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks jbrooks@redhat.com wrote:
[...]
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
Isn't that what fedora is used for?
Fedora is used as a starting point for major release alphas and betas, i.e., 7.0 Beta, 8.0 Beta, etc. After the major release beta comes out all automatic connection between Fedora and RHEL ceases. RHEL 8.2 was based on 8.1 + upstream changes, 8.1 was based on 8.0 plus upstream changes. There simply hasn't been a place where people outside the Red Hat firewall can see, use, and influence the direction of the next minor release, as it is being created. That's what Stream is meant to do.
Minor release updates very rarely have a need for significant influence, but I'm unsure how this is supposed to relate to actual RHEL minor version Beta releases. Will there still even be RHEL 8.x Beta Releases?
There'll be more opportunity to shape things once the 9 stream starts, during that stream 8 / stream 9 overlap period. Fedora is the place for big changes, but it can be a long wait for that stuff to get into RHEL, so Stream helps address this.
Your CTO stated that
"To be exact, CentOS Stream is an upstream development platform for ecosystem developers. It will be updated several times a day. This is not a production operating system."
That sounds like it is for alpha releases, not minor changes.
-- Jason Brooks He/Him Manager, Community Architects & Infrastructure Red Hat Open Source Program Office (OSPO) https://community.redhat.com | https://osci.io
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Dec 15, 2020 at 6:49 PM Japheth Cleaver cleaver@terabithia.org wrote:
On 12/15/2020 6:18 PM, Brendan Conoboy wrote:
On Tue, Dec 15, 2020 at 5:42 PM Mauricio Tavares raubvogel@gmail.com wrote:
On Tue, Dec 15, 2020 at 6:30 PM Jason Brooks jbrooks@redhat.com wrote:
[...]
No. I was on that team too, and growing CentOS beyond just consumption and into contribution was something we emphasized throughout. Our primary intent, the reason the whole thing got started, was that we needed to provide our layered projects with a slow-moving community distro to layer atop. That's why we put so much effort into the SIGs, and into opening up the build processes and tools. Even with that work done, until we opened up RHEL development itself, contributions to the core of CentOS were basically blocked. Now, in addition to the layered project need, which hasn't gone away, we need a distro to open up RHEL development, and CentOS Stream is that distro.
Isn't that what fedora is used for?
Fedora is used as a starting point for major release alphas and betas, i.e., 7.0 Beta, 8.0 Beta, etc. After the major release beta comes out all automatic connection between Fedora and RHEL ceases. RHEL 8.2 was based on 8.1 + upstream changes, 8.1 was based on 8.0 plus upstream changes. There simply hasn't been a place where people outside the Red Hat firewall can see, use, and influence the direction of the next minor release, as it is being created. That's what Stream is meant to do.
Minor release updates very rarely have a need for significant influence, but I'm unsure how this is supposed to relate to actual RHEL minor version Beta releases.
That's a very interesting and unexpected standpoint. Do you only use the features present in the .0 release?
Will there still even be RHEL 8.x Beta Releases?
We have one planned for 8.4. In the future, who knows?
But beyond that, the above statement does not seem to be compatible with the following: On 12/15/2020 3:35 PM, Johnny Hughes wrote:
You guys keep calling it beta .. it is not.
The RHEL team is not grabbing brand new software (like the do in Rawhide, for example) and trying to roll that into RHEL. They are going to do one of three type of updates.
A security update
A bugfix update.
An Enhancement update.
For #1 and #2 .. you want those rolled in and you want them rolled in ASAP. RHEAs do not make up that many of the updates. You are getting these after QA testing a couple months early at most.
There's no meaningful "influence" at this point beyond filing BZs about things that break. While that's certainly better than nothing, if RedHat wanted to accept bugs from non-RHEL binaries, especially in-house rebuilds explicitly targeting 100% binary compatibility, it could easily have done so at any point in the past.
There's still a conflation of upstream and downstream here. The "direction" of any aspect of RHEL is already going to be quite set by the time it gets into CentOS Stream... as is appropriate for a downstream.
I think you've succinctly expressed that we have not done a good enough job of painting and sustaining a picture of the potential breadth of CentOS Stream and what it could mean to add additional facets to the community's self identity.
On 12/15/20 1:03 PM, Jim Jagielski wrote:
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff).
The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
It is actually just as Jason Brooks has spelled out—we needed a slower-moving platform for our layered projects success. CentOS Linux was our best bet in 2013, when projects like OpenStack (RDO) and oVirt were growing and running into pains.
If anything, the community of CentOS Linux users was one of our biggest concerns in terms of not wanting to scare any of them away, nor wanting them to think we were suddenly making CentOS Linux the same thing as RHEL.
- Karsten
On Dec 15, 2020, at 10:07 PM, Karsten Wade kwade@redhat.com wrote:
On 12/15/20 1:03 PM, Jim Jagielski wrote:
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff). The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
It is actually just as Jason Brooks has spelled out—we needed a slower-moving platform for our layered projects success. CentOS Linux was our best bet in 2013, when projects like OpenStack (RDO) and oVirt were growing and running into pains.
Again, the main concern was that with such layered products, it was deemed better if instead of people using Canonical, they stayed in the RedHat family, and officially having CentOS supported as a RedHat "effort" was the solution.
Yes, people were not going to run OpenStack (or OpenShift) on Fedora, nor did it make sense to try to fold those directly into RHEL. CentOS was the "perfect" solution. The goal of SIGs was to determine what layered products, and in what format, people wanted. But the idea that CentOS was intended to be a 50/50 bidirectional codebase is simply rewriting history. The claim that the CentOS community never changed from what it was, and what RedHat *knew* it was, and what RedHat over the years (at least publicly) constantly indicated they were 100% happy about (That CentOS was a community of *users*) just seems like after the fact justification, with the sole intent of placing the blame ON CENTOS.
Le 16/12/2020 à 12:35, Jim Jagielski a écrit :
On Dec 15, 2020, at 10:07 PM, Karsten Wade kwade@redhat.com wrote:
On 12/15/20 1:03 PM, Jim Jagielski wrote:
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff). The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
It is actually just as Jason Brooks has spelled out—we needed a slower-moving platform for our layered projects success. CentOS Linux was our best bet in 2013, when projects like OpenStack (RDO) and oVirt were growing and running into pains.
Again, the main concern was that with such layered products, it was deemed better if instead of people using Canonical, they stayed in the RedHat family, and officially having CentOS supported as a RedHat "effort" was the solution.
Yes, people were not going to run OpenStack (or OpenShift) on Fedora, nor did it make sense to try to fold those directly into RHEL. CentOS was the "perfect" solution. The goal of SIGs was to determine what layered products, and in what format, people wanted. But the idea that CentOS was intended to be a 50/50 bidirectional codebase is simply rewriting history. The claim that the CentOS community never changed from what it was, and what RedHat *knew* it was, and what RedHat over the years (at least publicly) constantly indicated they were 100% happy about (That CentOS was a community of *users*) just seems like after the fact justification, with the sole intent of placing the blame ON CENTOS.
It now seems crystal clear Red Hat purchased CentOS 6 years ago as it was the best OpenStack infrastructure for their purpose. And the best value of this CentOS - Red Hat joint effort was not the binary rebuild of RHEL, but all the additionnal SIGs provided with CentOS 7.
With Red Hat now focused on OpenShift, this golden age as ended and CentOS Linux wasn't necessary in the suitable form it had always been for years. This turned in a way that betrayed all the Red Hat promises, the Community Entreprise OS was first and only interested for.
As in 2003 where Red Hat was the leading distro in the Linux World, the trust has been broken again and many will flee to Debian or Ubuntu LTS, but I personnaly dislike .deb and their system organisation.
I also have no time to wait for Rocky or Lenix, it will be a difficult goal to achieve for the moment, or even jump to Springdale as this team is not strong enough for the long run.
IBM/Red Hat has became the new evil for my point of vue. So between all the evils I know, let's choose the lesser to deal with at the moment. And Oracle actually renewed the promise IBM/Red Hat has just renied, a binary rebuild of RHEL stucked on the 2029 EOL, with some awesome UEK kernels I already use in production for years now.
https://blogs.oracle.com/linux/need-a-stable,-rhel-compatible-alternative-to...
I will still get an eye on s/CentOS/RHEL/g Stream, but the distro I beloved since fourteen years has just fall in the grave, 2020 December the 8th, RIP CentOS.
Jean-Marc LIGER
Le 16/12/2020 à 12:35, Jim Jagielski a écrit :
On Dec 15, 2020, at 10:07 PM, Karsten Wade kwade@redhat.com wrote:
On 12/15/20 1:03 PM, Jim Jagielski wrote:
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff). The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
It is actually just as Jason Brooks has spelled out—we needed a slower-moving platform for our layered projects success. CentOS Linux was our best bet in 2013, when projects like OpenStack (RDO) and oVirt were growing and running into pains.
Again, the main concern was that with such layered products, it was deemed better if instead of people using Canonical, they stayed in the RedHat family, and officially having CentOS supported as a RedHat "effort" was the solution.
Yes, people were not going to run OpenStack (or OpenShift) on Fedora, nor did it make sense to try to fold those directly into RHEL. CentOS was the "perfect" solution. The goal of SIGs was to determine what layered products, and in what format, people wanted. But the idea that CentOS was intended to be a 50/50 bidirectional codebase is simply rewriting history. The claim that the CentOS community never changed from what it was, and what RedHat *knew* it was, and what RedHat over the years (at least publicly) constantly indicated they were 100% happy about (That CentOS was a community of *users*) just seems like after the fact justification, with the sole intent of placing the blame ON CENTOS.
It now seems crystal clear Red Hat purchased CentOS 6 years ago as it was the best OpenStack infrastructure for their purpose. And the best value of this CentOS - Red Hat joint effort was not the binary rebuild of RHEL, but all the additionnal SIGs provided with CentOS 7.
With Red Hat now focused on OpenShift, this golden age as ended and CentOS Linux wasn't necessary in the suitable form it had always been for years. This turned in a way that betrayed all the Red Hat promises, the Community Entreprise OS was first and only interested for.
As in 2003 where Red Hat was the leading distro in the Linux World, the trust has been broken again and many will flee to Debian or Ubuntu LTS,
Now that you mention 2003, I remember that time and one thing became clear to me. This new direction with CentOS is most likely inspired from the top of RedHat. It can only happen with the support of the top management and it is, if you ask me, inspired by the same people who decided the things back in 2003. Unfortunately they seem to have missed to realize the reasons why things have worked back then.
I know that Red Hat was and is free to decide what they want. But I can assure you that the only reason why quite a number RHEL subscriptions have been sold to the companies where I have worked in the past is that there was a project called CentOS! That was the reason why I didn't move to Debian or something else and I was the person who brought Linux into the companies, where they had everything from Windows Server to Novell Netware to different UNIX systems but no Linux.
Regards, Simon
Hi,
On 17. Dec 2020, at 09:03, Simon Matter simon.matter@invoca.ch wrote:
I know that Red Hat was and is free to decide what they want. But I can assure you that the only reason why quite a number RHEL subscriptions have been sold to the companies where I have worked in the past is that there was a project called CentOS!
if that is really the case (I think it could well be), the business decision of RedHat maybe makes perfect sense.
Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
Sounds like a win/win-Situation, doesn't it?
Peter.
On Thu., Dec. 17, 2020, 3:13 a.m. Peter Eckel, lists@eckel-edv.de wrote:
Hi,
On 17. Dec 2020, at 09:03, Simon Matter simon.matter@invoca.ch wrote:
I know that Red Hat was and is free to decide what they want. But I can assure you that the only reason why quite a number RHEL subscriptions
have
been sold to the companies where I have worked in the past is that there was a project called CentOS!
if that is really the case (I think it could well be), the business decision of RedHat maybe makes perfect sense.
Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
Sounds like a win/win-Situation, doesn't it?
Not at all. Vendors only just recognize CentOS as a valid vendor string and provide support for it. They do not recognize Rocky or other (including CentOS Stream) , and you are talking about repeating history and reengineering something all over again that the community already built.
It is difficult to see this as anything except sabotage. It is in RHEL interest, but not in user interest.
How can Red Hat eliminating what CentOS already was, to permit somebody else to start over from scratch without being allowed to use the branding and restarting the legal concerns, be considered a win /win?
You can put lipstick on a pig, but you still have a pig.
Peter. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Hi,
On 17. Dec 2020, at 09:03, Simon Matter simon.matter@invoca.ch wrote:
I know that Red Hat was and is free to decide what they want. But I can assure you that the only reason why quite a number RHEL subscriptions have been sold to the companies where I have worked in the past is that there was a project called CentOS!
if that is really the case (I think it could well be), the business decision of RedHat maybe makes perfect sense.
Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
Sounds like a win/win-Situation, doesn't it?
No.
A lot of those people who helped Red Hat to grow are now extremely disappointed and disillusioned. Trust was the driving force here and it is no more.
There is one more thing that may make a lot of former Red Hat supporters leave the party. Everyone can see how big companies make big $$$ with some of Red Hats work. Oracle sells support subscriptions for their Red Hat rebuild and make a business of it. Amazon uses Red Hats work to sell their cloud stuff and the list goes on endless. Then Red Hat writes here:
https://www.redhat.com/en/blog/centos-stream-building-innovative-future-ente...
"Facebook runs millions of servers supporting its vast global social network, all of which have been migrated (or are migrating) to an operating system they derive from CentOS Stream."
Can you read it, _millions_ of servers, and they don't pay for RHN subscriptions. Of course they don't need to because they can rebuild whatever they want for their own needs.
But, SMB companies are expected to pay big $$$ for their 5 servers. Do you see how David and Goliath are treated very differently? How are you going to explain this to your company/customer?
Regards, Simon
On 12/17/20 9:13 AM, Peter Eckel wrote:
Hi,
On 17. Dec 2020, at 09:03, Simon Matter simon.matter@invoca.ch wrote:
I know that Red Hat was and is free to decide what they want. But I can assure you that the only reason why quite a number RHEL subscriptions have been sold to the companies where I have worked in the past is that there was a project called CentOS!
if that is really the case (I think it could well be), the business decision of RedHat maybe makes perfect sense.
Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
Sounds like a win/win-Situation, doesn't it?
No, it does not. Because so far Red hat was viewed as champion of Open Source and we "freeloaders" felt morally obligated to help Red Hat in any way we could. It was the right and honest thing to do.
Since Red Hat displayed greedy and stab-in-the-back attitude (buy hiding what wanted to do before they were ready), there is absolutely no moral obligation to help them in any way, and many now even have negative feelings towards another "greedy company".
Before this my message was "If you are going to spend the money on Linux, it is best to spend it on RHEL, they give so much to community it is only fair."
Since few days ago my message is "I do not like them anymore, and I do not have trust in them, so better stay clear from them."
CentOS project leaders had the same philosophy in mind when they refused to add extra packages to CentOS repositories like non-free codecs, 3rd party drivers (ElRepo had to be created separately) or even some desktop apps or KDE, MATE, etc. All of that was redirected to Red Hat controlled EPEL or 3rd party repositories.
But Rocky Linux and Lenix (CloudLinux) do not have to be constrained with these compliance, why should they when most likely Red Hat will do their best to complicate creation of other clones any way they can. You can say what ever you want, but I and others do not trust them/you to be better then their worst deed.
And there is no legal obligation to use RHEL and not clones in production, especially if CloudLinux develops a business model that will enhance FOSS clone and eventually spin off from RHEL into competitor just like Oracle did. Even Rocky Linux could be backed by some new company that will offer paid-for support in production.
Up until this backstabbing act any company that would try to steal support income from Red Hat would have been declared greedy by CentOS and even Linux community at large. Even today I do not like Oracle because they became direct competitor to Red Hat who was spending money on development, bugfixes, etc.
But since Red Hat is now in same category as Oracle, greedy corporation, EL/Linux community will WELCOME another player in paid-support for RHEL clones, and stand by them as long as their actions support needs of "us freeloaders". Do you really think CloudLinux decided to spend $1 million because they are altruists? I do not. They have seen Red Hat hang them selves (nobody provoked them) and saw unique one-in-a-lifetime opportunity to expand their portfolio from only light hosting clone based on RHEL source to all-purpose distro that will help them expand their paid-for support offer to baremetal servers and workstations, maybe even laptops. All they have to do is to publish binary clone and then expand on that ecosystem by adding repos like ElRepo, EPEL, CentOSPlus, and maybe non-free repo and they will be huge success and make bundle of money, well worth the investment they are making.
And you know what? I am going to support them, and bee happy for them. And direct any money spending THEIR WAY.
Hi, I don't want to put more fire in this discussion but I read enough and now I feel like I need to reply. This is my personal opinion and completely not related to what Red Hat thinks: How would be the linux world if, after reading all these complains about Red Hat being greedy and not committed to open source as community think it should be and about people turning to RHEL rebuilds or other distro, would announce: "Red Hat is hereby announcing Red Hat Enterprise Linux on December 31st 2021. Red Hat will be committed in releasing only Red Hat CoreOS based on rebuilds from Rocky Linux opensource community as pure downstream to it" Some times we should look at things with different perspectives.
Thanks.
I see what you mean, and you saw what I meant.
Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
When I said in a previous message about developing real world products, it takes 20-30 VMs to test, validate, audit, etc, etc. CentOS was *crucial* for this. (Yes, of course patches for issues were tracked/applied accordingly).
I mentioned assurance (in a security context) earlier, too. It's important. This is outside of the container discussion, too.
You can't always statically link all your apps (*grin*), it's not about runtime memory anymore, but more about sideband updates for important libs that you want to see updated for fixes (api compliant), and still run.
I am, of course, coming from a different angle to most on this list (maybe).
________________________________ From: CentOS-devel centos-devel-bounces@centos.org on behalf of Sandro Bonazzola sbonazzo@redhat.com Sent: December 17, 2020 6:03 AM To: The CentOS developers mailing list. Subject: Re: [CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…
Hi, I don't want to put more fire in this discussion but I read enough and now I feel like I need to reply. This is my personal opinion and completely not related to what Red Hat thinks: How would be the linux world if, after reading all these complains about Red Hat being greedy and not committed to open source as community think it should be and about people turning to RHEL rebuilds or other distro, would announce: "Red Hat is hereby announcing Red Hat Enterprise Linux on December 31st 2021. Red Hat will be committed in releasing only Red Hat CoreOS based on rebuilds from Rocky Linux opensource community as pure downstream to it" Some times we should look at things with different perspectives.
Thanks. --
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEAhttps://www.redhat.com/
sbonazzo@redhat.commailto:sbonazzo@redhat.com
[https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png]https://www.redhat.com/
Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.
Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin dan.seguin@cord3inc.com ha scritto:
I see what you mean, and you saw what I meant.
Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
Like https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri... ?
Not really, no.
Read the rest: nobody (i.e. clients) can run anything developed on this platform without paying for the platform it was targeted for, correct?
I think you missed my point.
________________________________ From: CentOS-devel centos-devel-bounces@centos.org on behalf of Sandro Bonazzola sbonazzo@redhat.com Sent: December 17, 2020 7:44 AM To: The CentOS developers mailing list. Subject: Re: [CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…
Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin <dan.seguin@cord3inc.commailto:dan.seguin@cord3inc.com> ha scritto:
I see what you mean, and you saw what I meant.
Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
Like https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri... ?
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEAhttps://www.redhat.com/
sbonazzo@redhat.commailto:sbonazzo@redhat.com
[https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png]https://www.redhat.com/
Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. https://mojo.redhat.com/docs/DOC-1199578
On 12/17/20 7:04 AM, Dan Seguin wrote:
Not really, no.
Read the rest: nobody (i.e. clients) can run anything developed on this platform without paying for the platform it was targeted for, correct?
I think you missed my point.
*From:* CentOS-devel centos-devel-bounces@centos.org on behalf of Sandro Bonazzola sbonazzo@redhat.com *Sent:* December 17, 2020 7:44 AM *To:* The CentOS developers mailing list. *Subject:* Re: [CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…
Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin <dan.seguin@cord3inc.com mailto:dan.seguin@cord3inc.com> ha scritto:
I see what you mean, and you saw what I meant. Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
Like https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri... https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/ ?
All CentOS users who can not use stream (many will be able to, if they want .. some will not) should provide their use case to centos-questions@redhat.com
Red Hat is absolutely looking to provide more options for previous CentOS users.
There is also no reason 3rd party devel will not support Stream. It is absolutely in their best interest to do so.
On Thu, Dec 17, 2020 at 10:00 AM Johnny Hughes johnny@centos.org wrote:
There is also no reason 3rd party devel will not support Stream. It is absolutely in their best interest to do so.
It depends upon the type of vendor. Most vendors have the exact opposite premise. They do the absolute minimum testing, and they delay updates.
In the case of IBM Rational ClearCase, for example, they provide support at least 90 days after the availability of the RHEL minor release.
In the case of several hardware vendors - we are still stuck with RHEL 4, 5, and 6 in production use cases, because they either don't support 7, or only their newest version supports 7.
I don't agree with them. But, especially when it comes to specialized use cases - and the choice of products are 1) bad or 2) worse, we are limited in our deployment choices. Most of these vendors only barely acknowledge the existence of CentOS. I don't see them acknowledging CentOS 8 Stream for a very long time.
On Sun, Dec 20, 2020 at 09:34:09PM +0200, Strahil Nikolov via CentOS-devel wrote:
centos-questions@redhat.com
Yeah, and only an automatic response is received.
I assure you it goes to real people. It's just kind of overwhelming for them right now.
On Sun, Dec 20, 2020 at 2:42 PM Matthew Miller mattdm@mattdm.org wrote:
On Sun, Dec 20, 2020 at 09:34:09PM +0200, Strahil Nikolov via CentOS-devel wrote:
centos-questions@redhat.com
Yeah, and only an automatic response is received.
I assure you it goes to real people. It's just kind of overwhelming for them right now.
Be nice to them, too. Senior engineers who answer such lines give up *very quickly* if they get yelled at. Being compelled to support a foolish idea is soul draining. Being forced to explain something very sensible to someone who refuses to listen can cause strokes. Working with someone who sounds weird but actually knows what they're talking about and leads you to the solution you never heard of.... is educational and why some of us like answering help lines after decades in the business.
Am 17.12.20 um 13:44 schrieb Sandro Bonazzola:
Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin <dan.seguin@cord3inc.com mailto:dan.seguin@cord3inc.com> ha scritto:
I see what you mean, and you saw what I meant. Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
Like https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri... https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/ ?
Sandro, as you make the pointer - can you explain what production system means? While being the usage border of such license? Most people here are in production businesses. So, what should we think when making such suggestion? Thanks.
-- Leon
Il giorno gio 17 dic 2020 alle ore 16:16 Leon Fauster via CentOS-devel < centos-devel@centos.org> ha scritto:
Am 17.12.20 um 13:44 schrieb Sandro Bonazzola:
Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin <dan.seguin@cord3inc.com mailto:dan.seguin@cord3inc.com> ha scritto:
I see what you mean, and you saw what I meant. Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
Like
https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri...
<
https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri...
?
Sandro, as you make the pointer - can you explain what production system means? While being the usage border of such license? Most people here are in production businesses. So, what should we think when making such suggestion? Thanks.
Please refer to https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux... and https://developers.redhat.com/terms-and-conditions
-- Leon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
I've looked in detail at these "no cost RHEL". They are severely encumbered if you try to do it for real. Several of the suggestions in this thread from Red Hat staff demonstrate how little they know about what it is like to be a Red Hat customer.
On Thu., Dec. 17, 2020, 10:45 a.m. Sandro Bonazzola, sbonazzo@redhat.com wrote:
Il giorno gio 17 dic 2020 alle ore 16:16 Leon Fauster via CentOS-devel < centos-devel@centos.org> ha scritto:
Am 17.12.20 um 13:44 schrieb Sandro Bonazzola:
Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin <dan.seguin@cord3inc.com mailto:dan.seguin@cord3inc.com> ha scritto:
I see what you mean, and you saw what I meant. Will there be a "free" RHEL distro that people can develop on that won't need a paid license?
Like
https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri...
<
https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscri...
?
Sandro, as you make the pointer - can you explain what production system means? While being the usage border of such license? Most people here are in production businesses. So, what should we think when making such suggestion? Thanks.
Please refer to https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux... and https://developers.redhat.com/terms-and-conditions
-- Leon
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA https://www.redhat.com/
sbonazzo@redhat.com https://www.redhat.com/
*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. https://mojo.redhat.com/docs/DOC-1199578*
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/17/20 10:09 AM, Mark Mielke wrote:
I've looked in detail at these "no cost RHEL". They are severely encumbered if you try to do it for real. Several of the suggestions in this thread from Red Hat staff demonstrate how little they know about what it is like to be a Red Hat customer.
On Thu., Dec. 17, 2020, 10:45 a.m. Sandro Bonazzola, <sbonazzo@redhat.com mailto:sbonazzo@redhat.com> wrote:
Il giorno gio 17 dic 2020 alle ore 16:16 Leon Fauster via CentOS-devel <centos-devel@centos.org <mailto:centos-devel@centos.org>> ha scritto: Am 17.12.20 um 13:44 schrieb Sandro Bonazzola: > > > Il giorno gio 17 dic 2020 alle ore 13:25 Dan Seguin > <dan.seguin@cord3inc.com <mailto:dan.seguin@cord3inc.com> <mailto:dan.seguin@cord3inc.com <mailto:dan.seguin@cord3inc.com>>> ha scritto: > > I see what you mean, and you saw what I meant. > > > Will there be a "free" RHEL distro that people can develop on that > won't need a paid license? > > Like > https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/ <https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/> > <https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/ <https://developers.redhat.com/blog/2016/03/31/no-cost-rhel-developer-subscription-now-available/>> > ? Sandro, as you make the pointer - can you explain what production system means? While being the usage border of such license? Most people here are in production businesses. So, what should we think when making such suggestion? Thanks. Please refer to https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux# <https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux#> and https://developers.redhat.com/terms-and-conditions <https://developers.redhat.com/terms-and-conditions>
As i said .. Red Hat is currently creating NEW terms for using RHEL. The way to interact with them is centos-questions@redhat.com.
There will be new use cases and new terms for those. It should likely happen by the end of QTR1 2021.
The current version for development rhel does indeed have many restrictions. Hence, the reason for the new list and the itesm in QTR1 2021.
Hello Ljubomir (and all!).
I never heard your name until a few days ago.
Based on what you wrote in this thread, I first want to thank you personally for what you did all these years.
I'd like to provide my own POV regarding some of the issues you raise, but before that, let me briefly introduce myself.
I use Linux since 1993. Started with MCC, then SLS, then played for some months the game of "I can do this myself" (what's later been called Linux
From Scratch), then Slackware, and then Debian. For my own machines, for
most of this time, I used Debian. At home I only moved briefly to Ubuntu for some years and now recently to Fedora, but can definitely see myself going back to Debian.
At work, I started in a place that used RHL, and also added quite a lot of Debian there. Then moved to another place that had RHL, and worked on replacing this with RHEL (3, at 2005). I worked there for quite some time, and also there introduced Debian in some places, but in the important places, there wasn't really a question - for production - RHEL, and for testing/debug - CentOS.
Then I moved to Red Hat, and here I work on oVirt/RHV.
I am not representing Red Hat in any capacity. I am just a developer, and my job is somewhat far from the discussion about RHEL/CentOS/etc. - in my day job, I am mostly a user of these.
On Thu, Dec 17, 2020 at 11:01 AM Ljubomir Ljubojevic centos@plnet.rs wrote:
On 12/17/20 9:13 AM, Peter Eckel wrote:
Hi,
On 17. Dec 2020, at 09:03, Simon Matter simon.matter@invoca.ch wrote:
I know that Red Hat was and is free to decide what they want. But I can assure you that the only reason why quite a number RHEL subscriptions have been sold to the companies where I have worked in the past is that there was a project called CentOS!
if that is really the case (I think it could well be), the business decision of RedHat maybe makes perfect sense.
Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
Sounds like a win/win-Situation, doesn't it?
No, it does not. Because so far Red hat was viewed as champion of Open Source and we "freeloaders" felt morally obligated to help Red Hat in any way we could. It was the right and honest thing to do.
Since Red Hat displayed greedy and stab-in-the-back attitude (buy hiding what wanted to do before they were ready), there is absolutely no moral
I wasn't part of the discussion around CentOS - neither in 2014 nor now - and the news from last week was a shock to me as well. But if you now go back and read the announcement from 2014, you can very clearly see that from the very beginning, Red Hat didn't consider, or implied, or suggested, or anything like that, that it sees CentOS as a cheap/free RHEL replacement for the poor. It wasn't presented as _charity_. It was presented, and AFAICT _was_, for the benefit of Red Hat. Go read it. There is nothing new here.
obligation to help them in any way, and many now even have negative feelings towards another "greedy company".
Before this my message was "If you are going to spend the money on Linux, it is best to spend it on RHEL, they give so much to community it is only fair."
I also think/thought/talked like that in past jobs.
But I also want to add another reason: Spend it on Red Hat/RHEL, simply because they are the best, and worth your money. If you do not think so, don't.
Since few days ago my message is "I do not like them anymore, and I do not have trust in them, so better stay clear from them."
I definitely feel your pain. I felt the same way for several days now, and slowly got used to the new situation, and am now mostly ok with it.
I am not saying it was nice. I am just saying, that right now, I do agree with upper management here, if they say we simply had no choice but break this "promise" (of support till 2029), as bad as the community would accept this.
CentOS project leaders had the same philosophy in mind when they refused to add extra packages to CentOS repositories like non-free codecs, 3rd party drivers (ElRepo had to be created separately) or even some desktop apps or KDE, MATE, etc.
There were at least two other reasons, AFAICT:
1. People do/did not want that. They wanted exactly what CentOS said it is trying to do - bug-for-bug compatibility with RHEL - so that's what CentOS did.
2. Doing CentOS as-is was already hard enough. I do not think I have to remind people the situation it was in, before the Red Hat "acquisition".
All of that was redirected to Red Hat controlled EPEL or 3rd party repositories.
But Rocky Linux and Lenix (CloudLinux) do not have to be constrained with these compliance, why should they when most likely Red Hat will do their best to complicate creation of other clones any way they can.
If you ask me, CentOS Stream is a giant step forward, for anyone that wants to rebuild RHEL. I am not sure why consider it "the best Red Hat can do to complicate" this.
You can say what ever you want, but I and others do not trust them/you to be better then their worst deed.
Fair enough.
And there is no legal obligation to use RHEL and not clones in production, especially if CloudLinux develops a business model that will enhance FOSS clone and eventually spin off from RHEL into competitor just like Oracle did. Even Rocky Linux could be backed by some new company that will offer paid-for support in production.
Of course!
As other, more senior Red Hatters already said on this thread: Game on.
We do not look for mercy. We think we are good. If you prefer doing business with Oracle, or CloudLinux, or a (new) company behind Rocky Linux - go ahead.
If you ask me, a _business_, making money, that decides to base their supply chain on the promise of a community project, instead of a contract with a company, is taking a significant risk. Nobody prevents this, but I'd personally not do that.
Up until this backstabbing act any company that would try to steal support income from Red Hat would have been declared greedy by CentOS and even Linux community at large. Even today I do not like Oracle because they became direct competitor to Red Hat who was spending money on development, bugfixes, etc.
But since Red Hat is now in same category as Oracle, greedy corporation, EL/Linux community will WELCOME another player in paid-support for RHEL clones, and stand by them as long as their actions support needs of "us freeloaders". Do you really think CloudLinux decided to spend $1 million because they are altruists? I do not. They have seen Red Hat hang them selves (nobody provoked them) and saw unique one-in-a-lifetime opportunity to expand their portfolio from only light hosting clone based on RHEL source to all-purpose distro that will help them expand their paid-for support offer to baremetal servers and workstations, maybe even laptops. All they have to do is to publish binary clone and then expand on that ecosystem by adding repos like ElRepo, EPEL, CentOSPlus, and maybe non-free repo and they will be huge success and make bundle of money, well worth the investment they are making.
And you know what? I am going to support them, and bee happy for them. And direct any money spending THEIR WAY.
Very well. I am not saying you should not.
I do not remember where I read it, but I read somewhere an estimation that continuing full support of CentOS 8 until 2029 would have cost Red Hat something like $30-$40 million. I have no reason to think this is way off. So $1 million suddenly does not look that much.
I'd like to use this opportunity to address some other issues raised recently (in this thread, perhaps also elsewhere). I'll not quote the text I am replying to, I hope that's ok.
What is Free Software, and what is Open Source? A lot was written about this, and I am not going to repeat everything. For current discussion, the main point I'd like to make is: Open Source is a business model. It's not (mainly) about giving the famous 4 freedoms to users of your software, or even about Linus's law "With enough eyeballs, all bugs are shallow". Red Hat got/gets, for free, the code of many FOSS projects. In return, Red Hat gives back _all_ the code it added/adds to these, and other, projects. If you measure the amount of "Being open source" for a particular company, by dividing to value of what it gave, with the value of what it got, or, for that matter, with its revenue/income/ whatever, I am fully certain you'll not find any other company in the world, in a size similar to Red Hat's, with even close-by ratio. Think about this. Go ahead and think about the companies you know. Sure, many companies "give back", but how much? Even those that give a lot, in money, or money-worth, have, AFAIK, way lower ratios. And, BTW, why does Red Hat do that? Because the people here, like me, make it do so. Both management, and owners (now IBM, other investors before that), realize that if a significant change is done to the "wrong" side, too many people here will quit, and put the company in such a bad position, that it will simply not be worth it. memo-list was on fire, for the last week. Not fire - more like a nuclear explosion. Believe me. But people slowly understand, and things slowly calm down. I currently simply do not believe that Red Hat can become evil, in this sense. It will simply be dissolved.
To add to that: IMO, any person/company/whatever that says "Red Hat pissed me off, I do not want ever to have anything to do with them", and want to be honest to themselves, must now go and start helping Debian. Doing, or using, a RHEL rebuild, is not that, and does not make sense. To me. You want competition? The only real competition for Red Hat, IMO, is Debian. Go ahead, help them. Or, if you prefer, SuSE. But a RHEL rebuild still keeps you tightly-related to Red Hat, strongly dependent on Red Hat, as much as you claim you hate it.
I am now going to take a big risk and talk about Ubuntu. It might be the stupidest thing I did, ever. Ubuntu/Canonical are not doing well, financially. The only reason they keep being so well-known, common, etc., is because they have funding, which is not based on their income. If rich people, that made their money (also) thanks to FOSS, want to contribute back, by financing it, such as by doing Ubuntu, that's adorable, really. But this isn't Red Hat. Red Hat is a group of people that is trying to make a leaving from Open Source itself. You might claim this is simply not interesting, or irrelevant, or stupid, or whatever. You might claim that Red Hat, as-is, simply has no place in the FOSS world. But many people do not agree, including both the people inside Red Hat, and also many of its customers, and so it exists. I'd personally find it very sad if it turns out that this is wrong - that there simply is no way to make a business, make a leaving, from doing FOSS - that I, personally, must either give up on my principles and work for a non-FOSS company, or give up on having FOSS as my day job (and do it only as a spare-time hobby). I still have hopes.
Re the help that people like you provided to Red Hat, for free, by spending effort on CentOS, then push for using/buying RHEL: Can you please think for a minute, and explain why CentOS Stream is not almost the same? For people like you (and me!), that want to play with Linux, do interesting stuff with/on it, learn it, etc., but not (yet) make money from it, build a business around it, does the difference between CentOS Linux and CentOS Stream really matter? Why? You do not have to answer right now. You can try it, then decide. I completely ignored it until the announcement last week, and now, since the oVirt project realized that migration to Stream is the most reasonable choice, started moving, and already yesterday ran into a bug and fixed it this morning, https://bugzilla.redhat.com/1908602. Is that really so bad? You might say: But I do not have money, and want/need to run real stuff in production, and need the stability, security fixes, etc. Fine. Was CentOS so good for you so far? CentOS was also weeks or months behind RHEL for all of its lifetime, sometimes (before the Red Hat acquisition) many months. Why do you think Stream will be that much worse?
Re timing etc.: I was really shocked last week, like everyone else. But now I think everything basically makes sense, to me. When I heard about providing only 1 year, I was shocked. 1 year? I was a sysadmin, and I know that doing the upgrades, at the time, e.g. from RHL 7 to RHEL 3, and later from RHEL 3 to 6 (yes, we did skip there 4-5), took me and my team more than a year, closer to two years. But: If any of the existing or emerging RHEL rebuilds is already good for you, or will be good enough in a year, migration to it from CentOS should not take a year. Not more than a few weeks, IMO (including testing, automation, whatever, etc.). If it's not production, you are most welcome to migrate to Stream, and depending on your needs, it might make much _more_ sense to you, than to consider a RHEL rebuild. And this also should not take that long. And if you want RHEL, and want to use it for development, see the existing options, and if you think they are not enough, or do not suit well your particular situation, others on this thread also discussed this - contact Red Hat. And if you are simply running production stuff, and your business relies on RHEL, if you ask me - RHEL is simply what you want. And if you think otherwise - just do your own calculations and buy (or not) what you think is best for you. This indeed might require a longer time, but would still make sense to do in a year.
Re the CentOS _brand_: People keep saying: OK, you realized that taking over CentOS was a mistake. Fine. Just give it back to the community. Now, let's have an exercise. Red Hat claims that CentOS costs it money. A lot of money. And that if CentOS Linux would continue as-is, even without Red Hat's financing, it would still cost it quite a lot of money. See the linked article at start of this thread, and other stuff here, for details. But it's not important if you accept this fact as presented or not. Just accept that this is Red Hat's stand - that CentOS Linux will continue costing it money, as-is. Now, suppose that Red Hat, expecting the huge noise/backlash such an announcement would cause, would simply quitely try to find contributions somewhere and keep the project as-is. Would this help covering these costs? Would a thread like current exist? Would people seriously start thinking about what it means to have something like CentOS Linux? How much it costs? etc.? I do not think so. So Red Hat decided it's best for everyone to simply publicly kill it. To make it extremely clear that Red Hat is not involved anymore in any RHEL rebuilds. That from Red Hat's POV, if you want CentOS Stream, as presented, you are most welcome to try it, that if you want RHEL, you have several options for getting it, but that if you want a RHEL rebuild, then CentOS isn't one, and Red Hat is not going to be involved in one. That if anyone wants to make one, fine - but Red Hat is not part of that. Now, let's be honest: People on this thread keep saying two opposite, contradicting things: One, that CentOS, as a name, is worth almost nothing - say, not much more than the cost of keeping the domain name centos.org, and the other, that CentOS is a huge thing, and that killing it as-is is a huge crime. Please be honest. If you think the former, what's the problem? Just create a new cool name, and start collecting people to join you. If you think the latter, please realize that this costs money - a lot - and that Red Hat decided that it does not want to spend this money anymore. I personally think that's fair.
Last but not least: About the implied pressure of Red Hat on the CentOS board, that led to current state. I wasn't part of this, or related in any way, and basically all I know about it is from this list. So what I write here is purely a guess. Isn't it possible, that the board, even including non-Red-Hatters, was simply convinced that Red Hat's POV is legitimate? That just keeping CentOS as-is for 9 years would make such a damage to Red Hat, that CentOS itself would simply not exist anymore? That for the interests of CentOS, its users, it's simply unavoidable to do something like what was eventually done? People might disagree about the details (e.g. 1 year, or less, or more), but does it sound so crazy that they simply agreed?
If you ask me, there simply is no more place for a project like CentOS in this world. It does not make sense. I do not believe any of the "community competitors" will survive. I think there is enough space only for a single large-scale community distro, and this is Debian. All the others will simply remain small, or disappear, or become companies. And these companies will realize the hard work of having a business. But that's just a guess. Time will tell.
If you read so far, thanks for your time. Also, thanks again, really, for your hard work along all these years. Do not think that Red Hatters do not appreciate that.
Best regards,
On 12/17/2020 5:06 AM, Yedidyah Bar David wrote:
Re the CentOS_brand_: People keep saying: OK, you realized that taking over CentOS was a mistake. Fine. Just give it back to the community. Now, let's have an exercise. Red Hat claims that CentOS costs it money. A lot of money. And that if CentOS Linux would continue as-is, even without Red Hat's financing, it would still cost it quite a lot of money. See the linked article at start of this thread, and other stuff here, for details. But it's not important if you accept this fact as presented or not. Just accept that this is Red Hat's stand - that CentOS Linux will continue costing it money, as-is. Now, suppose that Red Hat, expecting the huge noise/backlash such an announcement would cause, would simply quitely try to find contributions somewhere and keep the project as-is. Would this help covering these costs? Would a thread like current exist? Would people seriously start thinking about what it means to have something like CentOS Linux? How much it costs? etc.? I do not think so.
So Red Hat decided it's best for everyone to simply publicly kill it. To make it extremely clear that Red Hat is not involved anymore in any RHEL rebuilds. That from Red Hat's POV, if you want CentOS Stream, as presented, you are most welcome to try it, that if you want RHEL, you have several options for getting it, but that if you want a RHEL rebuild, then CentOS isn't one, and Red Hat is not going to be involved in one. That if anyone wants to make one, fine - but Red Hat is not part of that.
I think this supposition undervalues the awareness of the folks on this list, or who have been a part of this industry for a long time.
RedHat is a for-profit company, borne from a foundation of open-source. Most of the users are also for-profit companies of one type or another. One for-profit company is not going to simply "donate" to another one without some sort of return, even if it isn't a great one. But everyone in the OSS community who's been here for a while is well aware of the communal challenges around free-ridership. The relationship between RHL/RHEL and rebuilds is _not_ the same relationship as between, say, software companies and Amazon selling it as a service. As others have said on this thread, they've worked hard to FIND WAYS to throw money at RedHat to help ensure stability, development, and resources continue, and RedHat itself has rebuffed them or been non-responsive.
Had RedHat said "Real Talk: Running CentOS is expensive and we need a better way to justify this or get support for non-revenue operations more broadly", there are people on this list who would have jumped. There are multi-billion dollar companies that prize stability but could not internally justify a switch to all-RHEL pricing, given that rebuilds operated with a level of risk that was acceptable to them (or, more likely, acceptable to their sysadmin and linux engineering teams). If you're paying 20-25 senior Linux engineers, moving from OSS to a commercial framework is a tough sell. A common-good/private-profit synergy where support contracts were paired with contributions earmarked for CentOS operations and associated PR boosts would have been a no-brainer, and that synergy was what was implied in 2014. The expectation was that if your quasi-official rebuild project was in danger of closing up, there would have been outreach. There wasn't. This came out of the blue for anyone outside the bubble.
People on this thread keep saying two opposite, contradicting things: One, that CentOS, as a name, is worth almost nothing - say, not much more than the cost of keeping the domain name centos.org, and the other, that CentOS is a huge thing, and that killing it as-is is a huge crime. Please be honest. If you think the former, what's the problem? Just create a new cool name, and start collecting people to join you. If you think the latter, please realize that this costs money - a lot - and that Red Hat decided that it does not want to spend this money anymore. I personally think that's fair.
The fact that CentOS Linux was *not* spun off -- not allowed a chance to solicit funding directly, years after the funding problems from before -- but was, in fact "killed" is precisely the sign that this was about RedHat punishing the rebuild userbase, and not out of any sort of desire to maintain symbiosis with the (much larger) EL-derivative community or develop further ways to improve the balance sheets of both RHEL and the CentOS sides of the house.
Of course!
As other, more senior Red Hatters already said on this thread: Game on.
We do not look for mercy. We think we are good. If you prefer doing business with Oracle, or CloudLinux, or a (new) company behind Rocky Linux - go ahead.
If CentOS needed a row of hardware to continue functioning (or it would die), there would have been donations. But when a new Vice-President comes on the list and throws it back at the post-2014 not meeting expectations that clearly were not indicated (and, in fact, were the polar opposite of what a downstream support distribution is intended to be), you're seeing more and more goodwill drying up. The other EL-derivatives *also* didn't operate for-profit, and closed down operations because they trusted that RH was doing the right thing. Clearly that trust, that RH would be the grownups here, was severely misplaced. EL-users have invested significant time and energy in the RH ecosystem, and we don't really *need* competing rebuild projects, which all, by definition, are trying to do nearly identical things.
The direct revenue to RHEL is not changing overnight. No one on this list *wants* to do business with Oracle, and no one wants to be enemies with RedHat. But leadership has needlessly created a hostile environment (not just a cool one, or a post-amicable-divorce one, but a /hostile/ one) from a merciful one.
It's baffling.
-jc
On 12/17/20 2:06 PM, Yedidyah Bar David wrote:
Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
Sounds like a win/win-Situation, doesn't it?
No, it does not. Because so far Red hat was viewed as champion of Open Source and we "freeloaders" felt morally obligated to help Red Hat in any way we could. It was the right and honest thing to do.
Since Red Hat displayed greedy and stab-in-the-back attitude (buy hiding what wanted to do before they were ready), there is absolutely no moral
I wasn't part of the discussion around CentOS - neither in 2014 nor now - and the news from last week was a shock to me as well. But if you now go back and read the announcement from 2014, you can very clearly see that from the very beginning, Red Hat didn't consider, or implied, or suggested, or anything like that, that it sees CentOS as a cheap/free RHEL replacement for the poor. It wasn't presented as _charity_. It was presented, and AFAICT _was_, for the benefit of Red Hat. Go read it. There is nothing new here.
It is very possible that back then it was genuine sentiment. I am talking about last few months when Red Had decided to kill the clone. Main point which many miss is that Red Hat *acquired* CentOS trademark in 2014, and after that (and few years) it decided to not fund it any more. If Red Hat is so community oriented and altruistic, why not only announce that Red hat wil not FUND clone any more, and allow CentOS community at large to try to provide funding? But Rad Hat did not do that, right? Higher ups decided to kill the clone, forbid the clone under CentOS trademark (RH controls) and morph CentOS users away from clone supported for 10-years in hope to switch enough users making money to RHEL subscription.
When RHEL 6.0 (2011) was released there was ~6 months gap before first clones were released. I remember how much I was pissed (at Oracle) that because of Oracle's attempt to steal market share from RH we CentOS users had to suffer. In order to slow down Oracle, RH employed several tactics like releasing kernel source without visible patches to mask which exact changes were made and "using secret sauce" in how parts of RHEL were built. I understood the logic and though it was ok, RH had to protect it self, even if CentOS was lagging behind. Then I asked CentOS devs if they can secretly get some help from RH, but they said no way. CentOS started to make some progress, and suddenly how CentOS was made became even more secretive, numerous people offering help were turned down, and I thought it little strange, but I was using something CentOS devs made so they had my full support. My private guess was that some deal was made that CentOS project will not share how they rebuild RHEL 6, but I could not guess what could have got in return. Help to finish rebuild so allied clone is still in the race to compete with Oracle clone? Who knows, but I did have such thought back then.
So just like RH in 2011 delayed cometition (free clones) for just 6 months, RH is now delaying most of the competition for several months. But no one expected such strong backlash. One of the CentOS Board members was commenting on Twitter that he expected backlash but not the petition on change.org.
In 2011 RH just delayed us using CentOS, and most of us forgiven it and accepted it. But killing it after securing trademark and making community impotent by forcing CebtOS Board to choose between jobs and clone? That was some nasty s**t, like one of the parents going on vacation with a child to his/hers homeland and then refusing to return the child.
obligation to help them in any way, and many now even have negative feelings towards another "greedy company".
Before this my message was "If you are going to spend the money on Linux, it is best to spend it on RHEL, they give so much to community it is only fair."
I also think/thought/talked like that in past jobs.
But I also want to add another reason: Spend it on Red Hat/RHEL, simply because they are the best, and worth your money. If you do not think so, don't.
Your "problem" is that you run in different circles then majority of CentOS users. I have been preaching virtues of CentOS/RHEL and that "best" does not fly with majority of users of other distro's, you could argue with some of them for months and never change their mind about Debian/Ubuntu being better. Also, how would you argue that RHEL is better the Oracle Linux when 99% of it is same code? Where does "better" can be measured and that it is not subjective?
That is why "giving back to community" ALWAYS had much better resonance with large majority of Linux users, it triggers their morality, sence of karmic justice.
Since few days ago my message is "I do not like them anymore, and I do not have trust in them, so better stay clear from them."
I definitely feel your pain. I felt the same way for several days now, and slowly got used to the new situation, and am now mostly ok with it.
I am not saying it was nice. I am just saying, that right now, I do agree with upper management here, if they say we simply had no choice but break this "promise" (of support till 2029), as bad as the community would accept this.
Again, altruistic choice of allowing community to try to fund it was always the option, even now, but that would not serve ulterior motives of RH (move users to RHEL licenses).
CentOS project leaders had the same philosophy in mind when they refused to add extra packages to CentOS repositories like non-free codecs, 3rd party drivers (ElRepo had to be created separately) or even some desktop apps or KDE, MATE, etc.
There were at least two other reasons, AFAICT:
- People do/did not want that. They wanted exactly what CentOS said it
is trying to do - bug-for-bug compatibility with RHEL - so that's what CentOS did.
There were several attempts in different periods in time where expansion of activities was attempted. It was always shut down by the Board. I was one that for C5 and C6 wanted to organize something like "Desktop SIG", but could not get approval. I created DentOS repository (70 rebuilded package for C5 including Skype) by my self but since it was unknown 3rd party repository it never gained any traction. I was one of the largest advocates of making easier the Desktop/Workstaion use of CentOS becuase I thought it would be much appealing for Linux users if you could more easily install desktop/laptop PC. Ubuntu's meteoric rise can be directly attributed to Fedora not shipping audio/video codecs for mp3, divix, etc while Ubuntu shiped it. And Ubuntu users tells newbie "all I have to do is turn on ppa and install codec, no 3rd party repositories needed, no clasing of repositores." And newbie would choose Ubuntu, not Fedora of CentOS...
- Doing CentOS as-is was already hard enough. I do not think I have to
remind people the situation it was in, before the Red Hat "acquisition".
I do not remember and donation campaigns before acquisition....
All of that was redirected to Red Hat controlled EPEL or 3rd party repositories.
But Rocky Linux and Lenix (CloudLinux) do not have to be constrained with these compliance, why should they when most likely Red Hat will do their best to complicate creation of other clones any way they can.
If you ask me, CentOS Stream is a giant step forward, for anyone that wants to rebuild RHEL. I am not sure why consider it "the best Red Hat can do to complicate" this.
You can say what ever you want, but I and others do not trust them/you to be better then their worst deed.
Fair enough.
And there is no legal obligation to use RHEL and not clones in production, especially if CloudLinux develops a business model that will enhance FOSS clone and eventually spin off from RHEL into competitor just like Oracle did. Even Rocky Linux could be backed by some new company that will offer paid-for support in production.
Of course!
As other, more senior Red Hatters already said on this thread: Game on.
We do not look for mercy. We think we are good. If you prefer doing business with Oracle, or CloudLinux, or a (new) company behind Rocky Linux - go ahead.
If you ask me, a _business_, making money, that decides to base their supply chain on the promise of a community project, instead of a contract with a company, is taking a significant risk. Nobody prevents this, but I'd personally not do that.
Outside "Western world" where RHEL support has significant presence, and "English speaking" parts of the world, how many countries have support in native language? RH has offices in only 17 European countries , out of 50. Around the world, in how many of 195 countries in the world does Red Hat provide support? Do you provide support in Hungarian, Serbo-Croatian, Slovak, etc, etc.?
Most of small businesses with several employees do not have their own IT admin, they hire contractors to deploy and manage their Linux servers. Now, what do you think owner of the small business is going to rather pay, RHEL license + contractor cost or only contractor cost? Or which contractor will win the bid, first charging ONLY contractor fee or the second one that adds $300/year for RHEL license on top? First one of course. So when I managed to convince owners they need dedicated storage server with RAID (you would be shocked how many small businesses STILL do not have file servers or even monthly backups of date form their PC's!), there was no way that I could sell them RHEL license. If they pay for something they would pay for Windows, they know how to use those. So all I could offer them was FREE Linux, on regular PC with 2 HDD's in mdraid without monitor was all I needed. No ECC memory, server-class MB's, U rack cases, etc. If there was no free CentOS, I would have gone with Ubuntu (or maybe Debian) and just as now I am the one who is responsible to manage them, and I know enough to not need emergency support from any vendor. Since they sit inside private networks, no internet facing ports, and barely have user security (Samba with all folders writable by all), I update them only from time to time, every few months, otherwise I do not touch them in not really necessary, to not jinx them while they work without issues. I only deployed 4-5 such servers, several other companies I manage have small NAS-es or keep data on Windows PC's :-( But even if I expand my business and start offering support to much more small business, none of them will want to pay yearly licenses. So I will either install free clone, TrueNAS/FreeNAS (FreeBSD based with ZFS!) or Debian-based. What I settle on will be for all such deployments, and my only contribution in direction of paid-for licenses would be my online recommendations, based on who I feel deserve the praise.
Up until this backstabbing act any company that would try to steal support income from Red Hat would have been declared greedy by CentOS and even Linux community at large. Even today I do not like Oracle because they became direct competitor to Red Hat who was spending money on development, bugfixes, etc.
But since Red Hat is now in same category as Oracle, greedy corporation, EL/Linux community will WELCOME another player in paid-support for RHEL clones, and stand by them as long as their actions support needs of "us freeloaders". Do you really think CloudLinux decided to spend $1 million because they are altruists? I do not. They have seen Red Hat hang them selves (nobody provoked them) and saw unique one-in-a-lifetime opportunity to expand their portfolio from only light hosting clone based on RHEL source to all-purpose distro that will help them expand their paid-for support offer to baremetal servers and workstations, maybe even laptops. All they have to do is to publish binary clone and then expand on that ecosystem by adding repos like ElRepo, EPEL, CentOSPlus, and maybe non-free repo and they will be huge success and make bundle of money, well worth the investment they are making.
And you know what? I am going to support them, and bee happy for them. And direct any money spending THEIR WAY.
Very well. I am not saying you should not.
I do not remember where I read it, but I read somewhere an estimation that continuing full support of CentOS 8 until 2029 would have cost Red Hat something like $30-$40 million. I have no reason to think this is way off. So $1 million suddenly does not look that much.
Again, with green light from Red Hat (never gonna happen) community could try to finance cloning effort and new blood could be introduced to actually work on it (also NEVER gonna happen).
I'd like to use this opportunity to address some other issues raised recently (in this thread, perhaps also elsewhere). I'll not quote the text I am replying to, I hope that's ok.
What is Free Software, and what is Open Source? A lot was written about this, and I am not going to repeat everything. For current discussion, the main point I'd like to make is: Open Source is a business model. It's not (mainly) about giving the famous 4 freedoms to users of your software, or even about Linus's law "With enough eyeballs, all bugs are shallow". Red Hat got/gets, for free, the code of many FOSS projects. In return, Red Hat gives back _all_ the code it added/adds to these, and other, projects. If you measure the amount of "Being open source" for a particular company, by dividing to value of what it gave, with the value of what it got, or, for that matter, with its revenue/income/ whatever, I am fully certain you'll not find any other company in the world, in a size similar to Red Hat's, with even close-by ratio. Think about this. Go ahead and think about the companies you know. Sure, many companies "give back", but how much? Even those that give a lot, in money, or money-worth, have, AFAIK, way lower ratios. And, BTW, why does Red Hat do that? Because the people here, like me, make it do so. Both management, and owners (now IBM, other investors before that), realize that if a significant change is done to the "wrong" side, too many people here will quit, and put the company in such a bad position, that it will simply not be worth it. memo-list was on fire, for the last week. Not fire - more like a nuclear explosion. Believe me. But people slowly understand, and things slowly calm down. I currently simply do not believe that Red Hat can become evil, in this sense. It will simply be dissolved.
Up to only 9 days ago I would agree. But announcing CentOS project is FORBIDDEN to maintain RHEL clone (RH owns trademark and controls the CentOS Board and does not want anyone else to finance cloning) it all invalidates any and all past deeds.
There is excellent speech in TV Show "Newsroom". Watch it to the very end, it expresses my exact feeling for Red Hat at this moment. "America is not the greatest country in the world ANYMORE". (There is "it used to be..." continuation that is important to watch) https://www.youtube.com/watch?v=bIpKfw17-yY Same sentiment is in the Linux community since Dec 8th, RHEL is not the greatest supporter of FOSS ANYMORE!
To add to that: IMO, any person/company/whatever that says "Red Hat pissed me off, I do not want ever to have anything to do with them", and want to be honest to themselves, must now go and start helping Debian. Doing, or using, a RHEL rebuild, is not that, and does not make sense. To me. You want competition? The only real competition for Red Hat, IMO, is Debian. Go ahead, help them. Or, if you prefer, SuSE. But a RHEL rebuild still keeps you tightly-related to Red Hat, strongly dependent on Red Hat, as much as you claim you hate it.
Do you really think I am fighting here to get back CentOS clone because I hate RHEL, how things are done? I chose rpm as a package of choice in 2005-2006, and chose CentOS as distro ini 2008. It makes most sence to me, I like how stable/boring it is, I like 10 year support, I like point releases which gives you option to update only up to certain date if you have issues with next minor release, I like stable kABI n between, I like weak-updates for drivers (kmod). The rest I got used to and accepted, and if I can choose, I would choose that CentOS community is allowed to fund clone. There is still little time for RH to say community at large does not understand them, and they did not think of it at first, and because of that they will hand over clone rebuild to community, to keep peace, show good will, or what ever.
That would also prove to you Red Hat employees who do not like what happened that Red Hat is still company you are portraying it to be.
But that is not going to happen, and Red Hat employees will have to swallow this bitter pill, and accept the new reality.
And actually, the sentiment is ONLY about CentOS 8 systems already deployed. Only one (C8) I have is my 4-year old laptop, but by moving it to Springdale I will not have to reinstall anything else on it while it works. I doubt it will live next 9 years until C8 EOL.
I have 1 C7 that is safe next 4 years, and 2 C6. One is private Samba+KVM and can function for next several years without reinstalling it since it is not accessible except to 5-6 employees. Other is my main web/mail/storage server that I do not know what to do. My small WISP is practically dead and I should remove server from its uplink (10/10Mbps free of charge), so moving mails and web sites (one is Joomla 1.5, never cared enough to update it since word of mouth is enough for my side job) to hosting service or better hosted VM is best scenario for it, and then I can convert it to TrueNAS of some Linux with ZFS support. That decision will wait, C6 Samba/NFS not on the internet can wait for me to choose.
I am now going to take a big risk and talk about Ubuntu. It might be the stupidest thing I did, ever. Ubuntu/Canonical are not doing well, financially. The only reason they keep being so well-known, common, etc., is because they have funding, which is not based on their income. If rich people, that made their money (also) thanks to FOSS, want to contribute back, by financing it, such as by doing Ubuntu, that's adorable, really. But this isn't Red Hat. Red Hat is a group of people that is trying to make a leaving from Open Source itself. You might claim this is simply not interesting, or irrelevant, or stupid, or whatever. You might claim that Red Hat, as-is, simply has no place in the FOSS world. But many people do not agree, including both the people inside Red Hat, and also many of its customers, and so it exists. I'd personally find it very sad if it turns out that this is wrong - that there simply is no way to make a business, make a leaving, from doing FOSS - that I, personally, must either give up on my principles and work for a non-FOSS company, or give up on having FOSS as my day job (and do it only as a spare-time hobby). I still have hopes.
Re the help that people like you provided to Red Hat, for free, by spending effort on CentOS, then push for using/buying RHEL: Can you please think for a minute, and explain why CentOS Stream is not almost the same? For people like you (and me!), that want to play with Linux, do interesting stuff with/on it, learn it, etc., but not (yet) make money from it, build a business around it, does the difference between CentOS Linux and CentOS Stream really matter? Why? You do not have to answer right now. You can try it, then decide. I completely ignored it until the announcement last week, and now, since the oVirt project realized that migration to Stream is the most reasonable choice, started moving, and already yesterday ran into a bug and fixed it this morning, https://bugzilla.redhat.com/1908602. Is that really so bad? You might say: But I do not have money, and want/need to run real stuff in production, and need the stability, security fixes, etc. Fine. Was CentOS so good for you so far? CentOS was also weeks or months behind RHEL for all of its lifetime, sometimes (before the Red Hat acquisition) many months. Why do you think Stream will be that much worse?
Re timing etc.: I was really shocked last week, like everyone else. But now I think everything basically makes sense, to me. When I heard about providing only 1 year, I was shocked. 1 year? I was a sysadmin, and I know that doing the upgrades, at the time, e.g. from RHL 7 to RHEL 3, and later from RHEL 3 to 6 (yes, we did skip there 4-5), took me and my team more than a year, closer to two years. But: If any of the existing or emerging RHEL rebuilds is already good for you, or will be good enough in a year, migration to it from CentOS should not take a year. Not more than a few weeks, IMO (including testing, automation, whatever, etc.). If it's not production, you are most welcome to migrate to Stream, and depending on your needs, it might make much _more_ sense to you, than to consider a RHEL rebuild. And this also should not take that long. And if you want RHEL, and want to use it for development, see the existing options, and if you think they are not enough, or do not suit well your particular situation, others on this thread also discussed this - contact Red Hat. And if you are simply running production stuff, and your business relies on RHEL, if you ask me - RHEL is simply what you want. And if you think otherwise - just do your own calculations and buy (or not) what you think is best for you. This indeed might require a longer time, but would still make sense to do in a year.
Re the CentOS _brand_: People keep saying: OK, you realized that taking over CentOS was a mistake. Fine. Just give it back to the community. Now, let's have an exercise. Red Hat claims that CentOS costs it money. A lot of money. And that if CentOS Linux would continue as-is, even without Red Hat's financing, it would still cost it quite a lot of money. See the linked article at start of this thread, and other stuff here, for details. But it's not important if you accept this fact as presented or not. Just accept that this is Red Hat's stand - that CentOS Linux will continue costing it money, as-is. Now, suppose that Red Hat, expecting the huge noise/backlash such an announcement would cause, would simply quitely try to find contributions somewhere and keep the project as-is. Would this help covering these costs? Would a thread like current exist? Would people seriously start thinking about what it means to have something like CentOS Linux? How much it costs? etc.? I do not think so. So Red Hat decided it's best for everyone to simply publicly kill it. To make it extremely clear that Red Hat is not involved anymore in any RHEL rebuilds. That from Red Hat's POV, if you want CentOS Stream, as presented, you are most welcome to try it, that if you want RHEL, you have several options for getting it, but that if you want a RHEL rebuild, then CentOS isn't one, and Red Hat is not going to be involved in one. That if anyone wants to make one, fine - but Red Hat is not part of that. Now, let's be honest: People on this thread keep saying two opposite, contradicting things: One, that CentOS, as a name, is worth almost nothing - say, not much more than the cost of keeping the domain name centos.org, and the other, that CentOS is a huge thing, and that killing it as-is is a huge crime. Please be honest. If you think the former, what's the problem? Just create a new cool name, and start collecting people to join you. If you think the latter, please realize that this costs money - a lot - and that Red Hat decided that it does not want to spend this money anymore. I personally think that's fair.
Last but not least: About the implied pressure of Red Hat on the CentOS board, that led to current state. I wasn't part of this, or related in any way, and basically all I know about it is from this list. So what I write here is purely a guess. Isn't it possible, that the board, even including non-Red-Hatters, was simply convinced that Red Hat's POV is legitimate? That just keeping CentOS as-is for 9 years would make such a damage to Red Hat, that CentOS itself would simply not exist anymore? That for the interests of CentOS, its users, it's simply unavoidable to do something like what was eventually done? People might disagree about the details (e.g. 1 year, or less, or more), but does it sound so crazy that they simply agreed?
Johnny Huges wrote in one of the lists main or devel that Board was against, but new Red hat can veto any Board decission. And Johnny was personally first against it and negotiations were held. It is pretty detailed.
Here are excerpts from 2 mails he wrote:
"The CentOS Project board has a Red Hat Liaison. That position is documented here:
https://www.centos.org/about/governance/board-responsibilities/#red-hat-liai...
Also see role of Liaison here (and look at B:):
https://www.centos.org/about/governance/voting/
The bottom line is .. a decision of the CentOS Board has been made and we don't have to like it. We do have to do it, regardless of if we like it."
and
"That is correct .. so, the Red Hat Liaison can use Section B. of the Governance to dictate a vote. If the board FORCES the use of this clause, then whatever was wanted (in this case by Red Hat) would get inacted in its entirety with no real input from the board.
https://www.centos.org/about/governance/voting/
The CentOS Board knows this, so we had a dialoge with Red Hat instead. Red Hat presented their case and listened to our response. There was a significant back and forth.
So, no one 'FORCED' the board to do anything. Red Hat told us what they were going to do (what you quoted). The board then made many recommendations in a back and forth negotiation. The board then made a decision. The decision was reluctant .. but it was unanimous.
And now this is the way forward."
If you ask me, there simply is no more place for a project like CentOS in this world. It does not make sense. I do not believe any of the "community competitors" will survive. I think there is enough space only for a single large-scale community distro, and this is Debian. All the others will simply remain small, or disappear, or become companies. And these companies will realize the hard work of having a business. But that's just a guess. Time will tell.
If you read so far, thanks for your time. Also, thanks again, really, for your hard work along all these years. Do not think that Red Hatters do not appreciate that.
Best regards,
I do not believe that any distro will be "last mans standing". There is enough people that have different desires which are not Debian, but that also depends on other companies in Linux world. If Red hat doubles down on RHEL clone basing and making cloning more difficult, then yeah, majority of us will move to Debian and Red hat will not have system admins who would even think of RHEL way of doing stuff.
On Thu, Dec 17, 2020 at 8:06 AM Yedidyah Bar David didi@redhat.com wrote:
But I also want to add another reason: Spend it on Red Hat/RHEL, simply because they are the best, and worth your money. If you do not think so, don't.
I believed in this point strongly in 2015. Let me share some details about what forced this to change.
We were an RHEL-only shop, with a ~$0.5M USD / year annual renewal. We were preparing to deploy a private cloud, with expectations of quadrupling our Linux instances in the company. According to the contract, this would have been around $2M USD / year annual renewal. I did some review of what Red Hat provided us, and I came to a few problematic conclusions. One conclusion was that for the past few years, we were averaging a support cost of $20K USD per ticket, and these were for the most part very basic tickets that should never have been opened in the first place. For more complex tickets, including one that I opened to test and evaluate the system - the ticket actually went nowhere. The support person was unwiling to push the developers to fix a problem. I had to escalate and push hard, and then participate in the bugzilla.redhat.com discussions before finally getting the attention of the backend Red Hat developer who accidentally broke it, who quickly agreed to fix it.
I don't really want to get into all the details of what right, and what went wrong. I want you to think about this number for a second. $20K USD per support ticket, including basic support problems. Which of you can easily justify this to your directors? Would you agree to pay this out of your own pocket, if you were an owner of the company?
I did a lot of analysis, and had many discussions with Red Hat staff, who by the way - were all great. I believe emails I sent were circulated around Red Hat in the 2016 and 2017 timeframe, although I don't know if this was just the sales and technical people that worked with us, or if they weren't further. There was a great deal of interest in the problem I was describing, but no ability for anybody to do anything about it. The subscription model was fixed.
The Red Hat subscription model is particularly problematic in that *all* systems must be subscribed, whether they require support or not. The question at renewal time, and the wording of the contract is not "which systems require support?" The question is "how many Red Hat systems do you have?" This means that test systems, development system, customer test systems, are all counted as individual subscriptions of some type.
This also means that when we want to horizontally scale a service, such as by having it be split across 3 virtual machines instead of 1, we need 3 licenses. There are some allowances in there such as how the Standard Subscription permits 2 VM per subscription, but you eventually conclude that you must subscribe the entire hypervisor. Except - we use many different operating systems on the hypervisors, and we are now talking about licensing every hypervisor for Red Hat, even if the system might not be running a Red Hat host or guest. Effectively, we end up being charged for running Ubuntu or Windows unless we design our system to limit the migration of virtual machines so that we have a "RHEL-only cluster" and a "non-RHEL cluster".
The Red Hat sales person had no levers to get around this. Red Hat is fundamentally sold per subscription. Even at bulk discount, the per-subscription discount was no more than 20% per system.
I wanted to pay Red Hat for the services we received. I could justify this. However, we were looking at a $2M USD RHEL expense for the next year, and $4M USD for the year after that. I could not justify this, when so many alternatives existed that provide what was substantially the same content and service.
Red Hat priced themselves out of the market. I didn't do this. Red Hat did this. This is why Oracle or Facebook cannot use Red Hat. Oracle Linux might not exist if Red Hat had offered an arrangement with Oracle. It is fine to live by a code that says "my product is worth $X", but then it better be worth $X, or people won't buy it.
Do I think Red Hat has done great things for the community? Yes, with the caveat that a great number of companies are doing great things for the community, and Red Hat is not alone.
Can I justify $20K USD per ticket? Absolutely not. Red Hat is not worth $20K USD per ticket. At scale, we can fund an entire team to build something like CentOS for this cost, and that's exactly what happens. This isn't being cheap. This is business. If Red Hat is in the business of Open Source, including reselling the works of many others including myself, Red Hat should know this.
On Thu, Dec 17, 2020 at 8:48 PM Mark Mielke mark.mielke@gmail.com wrote:
On Thu, Dec 17, 2020 at 8:06 AM Yedidyah Bar David didi@redhat.com wrote:
But I also want to add another reason: Spend it on Red Hat/RHEL, simply because they are the best, and worth your money. If you do not think so, don't.
I believed in this point strongly in 2015. Let me share some details about what forced this to change.
We were an RHEL-only shop, with a ~$0.5M USD / year annual renewal. We were preparing to deploy a private cloud, with expectations of quadrupling our Linux instances in the company. According to the contract, this would have been around $2M USD / year annual renewal. I did some review of what Red Hat provided us, and I came to a few problematic conclusions. One conclusion was that for the past few years, we were averaging a support cost of $20K USD per ticket, and these were for the most part very basic tickets that should never have been opened in the first place. For more complex tickets, including one that I opened to test and evaluate the system - the ticket actually went nowhere. The support person was unwiling to push the developers to fix a problem. I had to escalate and push hard, and then participate in the bugzilla.redhat.com discussions before finally getting the attention of the backend Red Hat developer who accidentally broke it, who quickly agreed to fix it.
I don't really want to get into all the details of what right, and what went wrong. I want you to think about this number for a second. $20K USD per support ticket, including basic support problems. Which of you can easily justify this to your directors? Would you agree to pay this out of your own pocket, if you were an owner of the company?
I did a lot of analysis, and had many discussions with Red Hat staff,
who by the way - were all great. I believe emails I sent were circulated around Red Hat in the 2016 and 2017 timeframe, although I don't know if this was just the sales and technical people that worked with us, or if they weren't further. There was a great deal of interest in the problem I was describing, but no ability for anybody to do anything about it. The subscription model was fixed.
The Red Hat subscription model is particularly problematic in that *all* systems must be subscribed, whether they require support or not. The question at renewal time, and the wording of the contract is not "which systems require support?" The question is "how many Red Hat systems do you have?" This means that test systems, development system, customer test systems, are all counted as individual subscriptions of some type.
This also means that when we want to horizontally scale a service, such as by having it be split across 3 virtual machines instead of 1, we need 3 licenses. There are some allowances in there such as how the Standard Subscription permits 2 VM per subscription, but you eventually conclude that you must subscribe the entire hypervisor. Except - we use many different operating systems on the hypervisors, and we are now talking about licensing every hypervisor for Red Hat, even if the system might not be running a Red Hat host or guest. Effectively, we end up being charged for running Ubuntu or Windows unless we design our system to limit the migration of virtual machines so that we have a "RHEL-only cluster" and a "non-RHEL cluster".
The Red Hat sales person had no levers to get around this. Red Hat is fundamentally sold per subscription. Even at bulk discount, the per-subscription discount was no more than 20% per system.
I wanted to pay Red Hat for the services we received. I could justify this. However, we were looking at a $2M USD RHEL expense for the next year, and $4M USD for the year after that. I could not justify this, when so many alternatives existed that provide what was substantially the same content and service.
Red Hat priced themselves out of the market. I didn't do this. Red Hat did this. This is why Oracle or Facebook cannot use Red Hat. Oracle Linux might not exist if Red Hat had offered an arrangement with Oracle. It is fine to live by a code that says "my product is worth $X", but then it better be worth $X, or people won't buy it.
Do I think Red Hat has done great things for the community? Yes, with the caveat that a great number of companies are doing great things for the community, and Red Hat is not alone.
Can I justify $20K USD per ticket? Absolutely not. Red Hat is not worth $20K USD per ticket. At scale, we can fund an entire team to build something like CentOS for this cost, and that's exactly what happens. This isn't being cheap. This is business. If Red Hat is in the business of Open Source, including reselling the works of many others including myself, Red Hat should know this.
I have to assume you actually downloaded and installed RHEL. To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain.
Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts. We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.
We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.
We have subject matter experts and sometimes project leads those critical upstream projects. If you've got a strange problem, or need a feature implemented, we've got the people who can solve it. The best in the industry (at least for those who need the best, not everyone does).
And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.
-Mike
On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath mmcgrath@redhat.com wrote:
On Thu, Dec 17, 2020 at 8:48 PM Mark Mielke mark.mielke@gmail.com wrote:
On Thu, Dec 17, 2020 at 8:06 AM Yedidyah Bar David didi@redhat.com wrote:
But I also want to add another reason: Spend it on Red Hat/RHEL, simply because they are the best, and worth your money. If you do not think so, don't.
I believed in this point strongly in 2015. Let me share some details about what forced this to change.
We were an RHEL-only shop, with a ~$0.5M USD / year annual renewal. We were preparing to deploy a private cloud, with expectations of quadrupling our Linux instances in the company. According to the contract, this would have been around $2M USD / year annual renewal. I did some review of what Red Hat provided us, and I came to a few problematic conclusions. One conclusion was that for the past few years, we were averaging a support cost of $20K USD per ticket, and these were for the most part very basic tickets that should never have been opened in the first place. For more complex tickets, including one that I opened to test and evaluate the system - the ticket actually went nowhere. The support person was unwiling to push the developers to fix a problem. I had to escalate and push hard, and then participate in the bugzilla.redhat.com discussions before finally getting the attention of the backend Red Hat developer who accidentally broke it, who quickly agreed to fix it.
I don't really want to get into all the details of what right, and what went wrong. I want you to think about this number for a second. $20K USD per support ticket, including basic support problems. Which of you can easily justify this to your directors? Would you agree to pay this out of your own pocket, if you were an owner of the company?
I did a lot of analysis, and had many discussions with Red Hat staff, who by the way - were all great. I believe emails I sent were circulated around Red Hat in the 2016 and 2017 timeframe, although I don't know if this was just the sales and technical people that worked with us, or if they weren't further. There was a great deal of interest in the problem I was describing, but no ability for anybody to do anything about it. The subscription model was fixed.
The Red Hat subscription model is particularly problematic in that *all* systems must be subscribed, whether they require support or not. The question at renewal time, and the wording of the contract is not "which systems require support?" The question is "how many Red Hat systems do you have?" This means that test systems, development system, customer test systems, are all counted as individual subscriptions of some type.
This also means that when we want to horizontally scale a service, such as by having it be split across 3 virtual machines instead of 1, we need 3 licenses. There are some allowances in there such as how the Standard Subscription permits 2 VM per subscription, but you eventually conclude that you must subscribe the entire hypervisor. Except - we use many different operating systems on the hypervisors, and we are now talking about licensing every hypervisor for Red Hat, even if the system might not be running a Red Hat host or guest. Effectively, we end up being charged for running Ubuntu or Windows unless we design our system to limit the migration of virtual machines so that we have a "RHEL-only cluster" and a "non-RHEL cluster".
The Red Hat sales person had no levers to get around this. Red Hat is fundamentally sold per subscription. Even at bulk discount, the per-subscription discount was no more than 20% per system.
I wanted to pay Red Hat for the services we received. I could justify this. However, we were looking at a $2M USD RHEL expense for the next year, and $4M USD for the year after that. I could not justify this, when so many alternatives existed that provide what was substantially the same content and service.
Red Hat priced themselves out of the market. I didn't do this. Red Hat did this. This is why Oracle or Facebook cannot use Red Hat. Oracle Linux might not exist if Red Hat had offered an arrangement with Oracle. It is fine to live by a code that says "my product is worth $X", but then it better be worth $X, or people won't buy it.
Do I think Red Hat has done great things for the community? Yes, with the caveat that a great number of companies are doing great things for the community, and Red Hat is not alone.
Can I justify $20K USD per ticket? Absolutely not. Red Hat is not worth $20K USD per ticket. At scale, we can fund an entire team to build something like CentOS for this cost, and that's exactly what happens. This isn't being cheap. This is business. If Red Hat is in the business of Open Source, including reselling the works of many others including myself, Red Hat should know this.
I have to assume you actually downloaded and installed RHEL. To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain.
Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts. We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.
We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.
We have subject matter experts and sometimes project leads those critical upstream projects. If you've got a strange problem, or need a feature implemented, we've got the people who can solve it. The best in the industry (at least for those who need the best, not everyone does).
And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.
I think what would be nice to have would be more affordable "self-support" options, even for production uses. Believe it or not, I often tell folks that the best way to support Fedora and CentOS is to buy a RHEL subscription. I used to tell folks that purchasing the self-support options at $50~$150 per box per year is not a bad way to support the excellent work that you folks do. Unfortunately, those options no longer exist, and now self-support options for RHEL are just simply too expensive.
I don't presume to know much about how the balance of things work out for full support vs self-support buys of RHEL subscriptions, but I think a lot of us would be willing to buy RHEL for our workloads if it was more affordable. I'm *not* even saying "entirely free", I'm saying making it "impulse-buy" levels of cheap that people would buy it and use it to have RHEL and support the work Red Hat does. And companies that are in the business of running large fleets of systems but have to go without support due to the low margin/budget could take advantage of that and put it into their cost calculus. Sure, $50 isn't $0, but it does become one of the lowest parts of the cost of building out a system if you're already doing self-support anyway. That also makes the conversion story to higher-value support tiers *so* much easier, too.
That's not to say that having no-cost RHEL options for specific production cases (HPC, research, etc.) shouldn't happen (I know those places often have a budget at zero or in the red most of the time), but I think it'd be great to make RHEL more of an option for everyone else, too.
-- 真実はいつも一つ!/ Always, there's only one truth!
On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath mmcgrath@redhat.com wrote:
I wanted to pay Red Hat for the services we received. I could justify this. However, we were looking at a $2M USD RHEL expense for the next year, and $4M USD for the year after that. I could not justify this, when so many alternatives existed that provide what was substantially the same content and service.
I just explained how Red Hat's subscription model and pricing forced me to review if we could justify a Red Hat cost increase from $0.5M USD annually to $4M USD annually for essentially no increase in service levels, and I found it was not. We could fund a small army with $4M USD annually, and build our own distribution. This is at least 20 people worth of salaries.
Mike McGrath's answer below was not "I see your point, I will look into this and fix this" as it should have been. Instead, his point was "I think you are ignorant and not aware of our value."
This level of arrogance ("having or revealing an exaggerated sense of one's own importance or abilities") meant I was forced to reduce our contribution to Red Hat from a value we were willing to pay of $0.5M, down to <$0.1M, and with this new CentOS choice, it will likely drop to $0, by investing in alternatives.
I want it to be perfectly clear that Red Hat management is making Red Hat a difficult proposition for us to sell, even for proponents and admirers of Red Hat, and these choices have direct consequences on the Red Hat bottom line. Most other vendors we work with provide large discounts to support cases such as ours, so that it is a win/win. Red Hat provided no such option, and effectively priced itself out of the market. 2 years later when they saw that we were not bluffing, they offered a discount that only kicks in above $1M, and would still be about 20% discount. These terms are unacceptable.
Do you think I wanted to use my technical skills to rip out Red Hat binaries from everywhere in our company in a short time period? I didn't. But, I had no choice. After a year of discussions, Red Hat gave me no choice.
Let's dissect Mike McGrath's answer in a little more detail:
I have to assume you actually downloaded and installed RHEL. To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain.
Any large deployment cannot use Red Hat servers to deploy RHEL, but requires in-house CDN. Also, any large deployment requires customization, which then involves overlaying RHEL packages with non-RHEL packages including packages from external sources such as EPEL, and in-house packages. The CDN cost is therefore *ours*, not *yours*.
Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts. We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.
We have a team of in-house people providing front-end support for our users. Red Hat did not eliminate the need for this team. If we quadrupled our installs as was projected, the brunt of this cost would have been borne by our in-house team, not by Red Hat. The Red Hat subscription model is broken. The Red Hat proposition becomes problematic with scale.
Red Hat may have 1,000 people working on hardening RHEL, but the global community has millions of people, and the global community includes people such as myself who contribute back. Red Hat's contribution here is significant, but it is not infinite. I have contributed to several fixes to Red Hat Enterprise Linux, and Red Hat deploys components that I have worked on. It's great that you contribute back - just as it is great that all companies who work on F/OSS portions of Linux contribute back. Red Hat is one of many companies who contribute back, and this means that the Red Hat value proposition is not infinite. If Red Hat prices itself out of the market, how will Red Hat support these 1,000 people? How is choosing to force me to reduce our RHEL deployment from $0.5M USD annually to $0.1M USD annually, because of unacceptable subscription terms, in the greater interest of Red Hat?
We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.
This paywall service is sometimes useful, but most of the time problematic. Our users don't have access to it. I usually avoid using it wherever possible, and recommend it be avoided. This information should be publicly accessible.
We have subject matter experts and sometimes project leads those critical upstream projects. If you've got a strange problem, or need a feature implemented, we've got the people who can solve it. The best in the industry (at least for those who need the best, not everyone does).
So does Oracle, or Google, or AWS, or hundreds of others. I participate on the devel mailing lists, and I pay attention to who contributes and who provides answers. Red Hat is definitely on this list, but Red Hat is not alone - and for the most part, this service is a "cost of doing business", and not directly tied to a subscription. When I had an issue with Qemu and live migrations of nested virtualization, it was Oracle that contributed the fixes for live migration of nested virtualization. I don't want to reduce the value of Red Hat here - I want to make it clear that Red Hat is one of several important players.
And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.
Especially the "you're welcome" at the end is arrogance. Actually, Red Hat has plenty of bugs after "hardening", many of which I have had to deal with, including getting fixes made upstream. But, I don't find Red Hat support that useful for this. Too often, I have to figure out for myself what is wrong, and I may as well describe the problem on bugzilla.redhat.com myself. I don't need to open a support ticket and go through somebody else to do this.
I know you don't want to hear my story. You would prefer the story that Red Hat is awesome, that Red Hat can set whatever price they wish and it should be considered a bargain, and anybody who disagrees is ignorant. But, you are talking about your customers here - including customers showing good will and willingness to negotiate. You forced me to substantially remove Red Hat from our systems, by refusing to negotiate acceptable terms. You did this. I didn't do this.
On Fri, Dec 18, 2020 at 11:12 AM Mark Mielke mark.mielke@gmail.com wrote:
On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath mmcgrath@redhat.com wrote:
I wanted to pay Red Hat for the services we received. I could justify this. However, we were looking at a $2M USD RHEL expense for the next year, and $4M USD for the year after that. I could not justify this, when so many alternatives existed that provide what was substantially the same content and service.
I just explained how Red Hat's subscription model and pricing forced me to review if we could justify a Red Hat cost increase from $0.5M USD annually to $4M USD annually for essentially no increase in service levels, and I found it was not. We could fund a small army with $4M USD annually, and build our own distribution. This is at least 20 people worth of salaries.
Mike McGrath's answer below was not "I see your point, I will look into this and fix this" as it should have been. Instead, his point was "I think you are ignorant and not aware of our value."
This level of arrogance ("having or revealing an exaggerated sense of one's own importance or abilities") meant I was forced to reduce our contribution to Red Hat from a value we were willing to pay of $0.5M, down to <$0.1M, and with this new CentOS choice, it will likely drop to $0, by investing in alternatives.
I want it to be perfectly clear that Red Hat management is making Red Hat a difficult proposition for us to sell, even for proponents and admirers of Red Hat, and these choices have direct consequences on the Red Hat bottom line. Most other vendors we work with provide large discounts to support cases such as ours, so that it is a win/win. Red Hat provided no such option, and effectively priced itself out of the market. 2 years later when they saw that we were not bluffing, they offered a discount that only kicks in above $1M, and would still be about 20% discount. These terms are unacceptable.
Do you think I wanted to use my technical skills to rip out Red Hat binaries from everywhere in our company in a short time period? I didn't. But, I had no choice. After a year of discussions, Red Hat gave me no choice.
Let's dissect Mike McGrath's answer in a little more detail:
I have to assume you actually downloaded and installed RHEL. To do so
you would have used our CDN and a fairly extensive (and audited and secured) supply chain.
Any large deployment cannot use Red Hat servers to deploy RHEL, but requires in-house CDN. Also, any large deployment requires customization, which then involves overlaying RHEL packages with non-RHEL packages including packages from external sources such as EPEL, and in-house packages. The CDN cost is therefore *ours*, not *yours*.
Over 1,000 people work on the actual RHEL bits *after* the community has
already worked on it and many more support those efforts. We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.
We have a team of in-house people providing front-end support for our users. Red Hat did not eliminate the need for this team. If we quadrupled our installs as was projected, the brunt of this cost would have been borne by our in-house team, not by Red Hat. The Red Hat subscription model is broken. The Red Hat proposition becomes problematic with scale.
Red Hat may have 1,000 people working on hardening RHEL, but the global community has millions of people, and the global community includes people such as myself who contribute back. Red Hat's contribution here is significant, but it is not infinite. I have contributed to several fixes to Red Hat Enterprise Linux, and Red Hat deploys components that I have worked on. It's great that you contribute back - just as it is great that all companies who work on F/OSS portions of Linux contribute back. Red Hat is one of many companies who contribute back, and this means that the Red Hat value proposition is not infinite. If Red Hat prices itself out of the market, how will Red Hat support these 1,000 people? How is choosing to force me to reduce our RHEL deployment from $0.5M USD annually to $0.1M USD annually, because of unacceptable subscription terms, in the greater interest of Red Hat?
We also have an extensive KBase and are working for more "in your face"
ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.
This paywall service is sometimes useful, but most of the time problematic. Our users don't have access to it. I usually avoid using it wherever possible, and recommend it be avoided. This information should be publicly accessible.
We have subject matter experts and sometimes project leads those
critical upstream projects. If you've got a strange problem, or need a feature implemented, we've got the people who can solve it. The best in the industry (at least for those who need the best, not everyone does).
So does Oracle, or Google, or AWS, or hundreds of others. I participate on the devel mailing lists, and I pay attention to who contributes and who provides answers. Red Hat is definitely on this list, but Red Hat is not alone - and for the most part, this service is a "cost of doing business", and not directly tied to a subscription. When I had an issue with Qemu and live migrations of nested virtualization, it was Oracle that contributed the fixes for live migration of nested virtualization. I don't want to reduce the value of Red Hat here - I want to make it clear that Red Hat is one of several important players.
And finally, while you are only calculating Red Hat value via support
tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.
Especially the "you're welcome" at the end is arrogance. Actually, Red Hat has plenty of bugs after "hardening", many of which I have had to deal with, including getting fixes made upstream. But, I don't find Red Hat support that useful for this. Too often, I have to figure out for myself what is wrong, and I may as well describe the problem on bugzilla.redhat.com myself. I don't need to open a support ticket and go through somebody else to do this.
I know you don't want to hear my story. You would prefer the story that Red Hat is awesome, that Red Hat can set whatever price they wish and it should be considered a bargain, and anybody who disagrees is ignorant. But, you are talking about your customers here - including customers showing good will and willingness to negotiate. You forced me to substantially remove Red Hat from our systems, by refusing to negotiate acceptable terms. You did this. I didn't do this.
-- Mark Mielke mark.mielke@gmail.com
Mark, I've heard responses like this for two weeks. You describe me as arrogant, and not listening, and I can only assume that you think we at Red Hat are fools who don't understand the enterprise. You've used a lot of fairly charged language in your response and you and many others have the "I'm going to show Red Hat" attitude. The problem is, and I think too few people realize this:
Red Hat isn't aiming for total global domination
So when you say I forced you to go somewhere else. You have to understand that in the open-source world, and in Red Hat's business plan, we know alternatives exist. That's the whole point of it. If you don't like the level of service you're getting. Go somewhere else. But don't pretend that the RHEL bits don't matter and that minimizing RHEL's contributions on a CentOS-devel mailing list will teach us something. You posted that your relationship with Red Hat boils down to cost per ticket - you did that, not me. I think if we've learned anything in the last two weeks it's that the bits seem to matter very much to people. They matter so much that people are feverishly trying to recreate RHEL instead of going to one of those many alternatives that already exist.
Just know that I mean it when I say, for those of you that are moving on. We wish you luck, we understand, and we'll see you around.
-Mike
On Fri, Dec 18, 2020 at 2:04 PM Mike McGrath mmcgrath@redhat.com wrote:
On Fri, Dec 18, 2020 at 11:12 AM Mark Mielke mark.mielke@gmail.com wrote:
I just explained how Red Hat's subscription model and pricing forced me to review if we could justify a Red Hat cost increase from $0.5M USD annually to $4M USD annually for essentially no increase in service levels, and I found it was not. We could fund a small army with $4M USD annually, and build our own distribution. This is at least 20 people worth of salaries.
Even in your response below, this shows a high level of tone deafness. You are hearing what you want to hear. For example:
Mark, I've heard responses like this for two weeks. You describe me as arrogant, and not listening, and I can only assume that you think we at Red Hat are fools who don't understand the enterprise. You've used a lot of fairly charged language in your response and you and many others have the "I'm going to show Red Hat" attitude. The problem is, and I think too few people realize this: Red Hat isn't aiming for total global domination
I never claimed you were trying for total global domination. I claimed your subscription model is fundamentally broke, and does not scale. I claimed that you made it difficult for me to justify RHEL for our company going forwards. You are not listening.
So when you say I forced you to go somewhere else. You have to understand that in the open-source world, and in Red Hat's business plan, we know alternatives exist. That's the whole point of it. If you don't like the level of service you're getting. Go somewhere else. But don't pretend that the RHEL bits don't matter and that minimizing RHEL's contributions on a CentOS-devel mailing list will teach us something. You posted that your relationship with Red Hat boils down to cost per ticket - you did that, not me. I think if we've learned anything in the last two weeks it's that the bits seem to matter very much to people. They matter so much that people are feverishly trying to recreate RHEL instead of going to one of those many alternatives that already exist.
You didn't hear my argument at all. You focused on one point about ticket cost, and ignored everything else. It's no wonder you don't get it. Read my above statement I quoted, and tell me how my *one* argument is about ticket costs. My point is that I had a choice - choose RHEL for our projected future use, and see our subscription costs go from $0.5M to $4M for essentially no increase in service levels, or choose something else.
Just know that I mean it when I say, for those of you that are moving on. We wish you luck, we understand, and we'll see you around.
I don't think you do understand. And that is the true tragedy here.
On 12/18/2020 11:18 AM, Mark Mielke wrote:
On Fri, Dec 18, 2020 at 2:04 PM Mike McGrath mmcgrath@redhat.com wrote:
Just know that I mean it when I say, for those of you that are moving on. We wish you luck, we understand, and we'll see you around.
I don't think you do understand. And that is the true tragedy here.
It's clear from other emails that there is a massive amount of internal RedHat debate about this decision still going on. I sincerely hope a decision-maker can try to fix this situation before a point-of-no-return is reached. This scorched earth approach from a RedHat rep is absurd considering the relative size of the EL-derivatives vs paid support contracts in install footprint. Nobody wants to have to explain to a RedHat Vice President the realities of a F/OSS ecosystem when RedHat is the company that virtually codified the industry's best-practices on how to do it.
More explicitly: Nobody wants an EL war; nobody wants to fork. Nobody wants more of a headache than we've already all had this year for reasons unrelated to Linux. But those things can happen, and it certainly wouldn't take $34B to do it.
This could have been "just" a question about a business decision -- one that would've been entirely (if painfully) understandable had it been a direct result of the IBM purchase. It could have been "just" a question of how much stability enabling yum-cron and CR/Stream in prod. actually removes. Instead, this is turning into a four-alarm fire about whether RedHat the company has the self-awareness that we all assumed it did about the trust, deference, and good will it has received from others.
-jc
On Fri, Dec 18, 2020 at 3:41 PM Japheth Cleaver cleaver@terabithia.org wrote:
On 12/18/2020 11:18 AM, Mark Mielke wrote:
On Fri, Dec 18, 2020 at 2:04 PM Mike McGrath mmcgrath@redhat.com wrote:
Just know that I mean it when I say, for those of you that are moving on. We wish you luck, we understand, and we'll see you around.
I don't think you do understand. And that is the true tragedy here.
It's clear from other emails that there is a massive amount of internal RedHat debate about this decision still going on. I sincerely hope a decision-maker can try to fix this situation before a point-of-no-return is reached. This scorched earth approach from a RedHat rep is absurd considering the relative size of the EL-derivatives vs paid support contracts in install footprint. Nobody wants to have to explain to a RedHat Vice President the realities of a F/OSS ecosystem when RedHat is the company that virtually codified the industry's best-practices on how to do it.
More explicitly: Nobody wants an EL war; nobody wants to fork. Nobody wants more of a headache than we've already all had this year for reasons unrelated to Linux. But those things can happen, and it certainly wouldn't take $34B to do it.
Exactly.
The community is better equipped than ever to produce an un-encumbered EL distribution. It would be a nuisance, it would be initially approached with sadness and reluctance, and it would have consequences, but the result might be a necessary next step in EL history. Personally, I don't see why Red Hat would want this. I wish Red Hat would listen and fix their subscription model to be compatible with customer requirements, eliminating the need for another EL distribution. I would be wary of challenging the EL community in this way. There are thousands of extremely intelligent, well resourced, and now motivated people to face such a challenge. Words like "feverish" are misleading as it implies some sort of illness or weariness. If necessary, this is an exciting challenge for this type of person. They have basically been practicing how to do this for years now, and they are being called upon to flex their considerable talents and resources to solve a real problem that directly impacts their day-to-day life? This is an invitation for greatness. This call is more like "hackers of the world, unite!"
This could have been "just" a question about a business decision -- one that would've been entirely (if painfully) understandable had it been a direct result of the IBM purchase. It could have been "just" a question of how much stability enabling yum-cron and CR/Stream in prod. actually removes. Instead, this is turning into a four-alarm fire about whether RedHat the company has the self-awareness that we all assumed it did about the trust, deference, and good will it has received from others.
I think it would result in a different sentiment and choice of language, but perhaps not a different result. I still like many people at Red Hat, I still appreciate all they have done for the community, and I still want to reward them for their efforts wherever possible. If Red Hat offered compatible terms tomorrow, I would review them, and consider reverting all of our decisions. However, this seems like an unlikely outcome. I suspect Red Hat will announce a new "free" RHEL, which continues to be encumbered in some problematic way, and we will be back to where we started. Also, given the recent choice to shut down CentOS 8, there would always be the threat that Red Hat would shut down this new "free" RHEL subscription type. A level of trust has been betrayed, that may never be restored. Ultimately, unless the software is free in both source and binary form, we eventually have this problem all over again.
The reality is that once Red Hat chose to stand firm behind their subscription model with us, and once we invested in the effort to essentially remove our dependencies on RHEL, we realized that we were better off. Red Hat forced us to develop our own expertise, and now we can choose deployment architectures based upon design requirements, instead of subscription requirements. Given the same budget, we can buy twice as much hardware without RHEL subscriptions, as with RHEL subscriptions, which for a business with ballooning compute requirements is a major win. And since many of our companies build our own Linux distributions anyways (for example, we use Yocto to produce many of our products), this is all familiar territory.
Le 18/12/2020 à 20:03, Mike McGrath a écrit :
On Fri, Dec 18, 2020 at 11:12 AM Mark Mielke <mark.mielke@gmail.com mailto:mark.mielke@gmail.com> wrote:
On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>> wrote: >> I wanted to pay Red Hat for the services we received. I could justify >> this. However, we were looking at a $2M USD RHEL expense for the next >> year, and $4M USD for the year after that. I could not justify this, >> when so many alternatives existed that provide what was substantially >> the same content and service. I just explained how Red Hat's subscription model and pricing forced me to review if we could justify a Red Hat cost increase from $0.5M USD annually to $4M USD annually for essentially no increase in service levels, and I found it was not. We could fund a small army with $4M USD annually, and build our own distribution. This is at least 20 people worth of salaries. Mike McGrath's answer below was not "I see your point, I will look into this and fix this" as it should have been. Instead, his point was "I think you are ignorant and not aware of our value." This level of arrogance ("having or revealing an exaggerated sense of one's own importance or abilities") meant I was forced to reduce our contribution to Red Hat from a value we were willing to pay of $0.5M, down to <$0.1M, and with this new CentOS choice, it will likely drop to $0, by investing in alternatives. I want it to be perfectly clear that Red Hat management is making Red Hat a difficult proposition for us to sell, even for proponents and admirers of Red Hat, and these choices have direct consequences on the Red Hat bottom line. Most other vendors we work with provide large discounts to support cases such as ours, so that it is a win/win. Red Hat provided no such option, and effectively priced itself out of the market. 2 years later when they saw that we were not bluffing, they offered a discount that only kicks in above $1M, and would still be about 20% discount. These terms are unacceptable. Do you think I wanted to use my technical skills to rip out Red Hat binaries from everywhere in our company in a short time period? I didn't. But, I had no choice. After a year of discussions, Red Hat gave me no choice. Let's dissect Mike McGrath's answer in a little more detail: > I have to assume you actually downloaded and installed RHEL. To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain. Any large deployment cannot use Red Hat servers to deploy RHEL, but requires in-house CDN. Also, any large deployment requires customization, which then involves overlaying RHEL packages with non-RHEL packages including packages from external sources such as EPEL, and in-house packages. The CDN cost is therefore *ours*, not *yours*. > Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts. We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream. We have a team of in-house people providing front-end support for our users. Red Hat did not eliminate the need for this team. If we quadrupled our installs as was projected, the brunt of this cost would have been borne by our in-house team, not by Red Hat. The Red Hat subscription model is broken. The Red Hat proposition becomes problematic with scale. Red Hat may have 1,000 people working on hardening RHEL, but the global community has millions of people, and the global community includes people such as myself who contribute back. Red Hat's contribution here is significant, but it is not infinite. I have contributed to several fixes to Red Hat Enterprise Linux, and Red Hat deploys components that I have worked on. It's great that you contribute back - just as it is great that all companies who work on F/OSS portions of Linux contribute back. Red Hat is one of many companies who contribute back, and this means that the Red Hat value proposition is not infinite. If Red Hat prices itself out of the market, how will Red Hat support these 1,000 people? How is choosing to force me to reduce our RHEL deployment from $0.5M USD annually to $0.1M USD annually, because of unacceptable subscription terms, in the greater interest of Red Hat? > We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run. This paywall service is sometimes useful, but most of the time problematic. Our users don't have access to it. I usually avoid using it wherever possible, and recommend it be avoided. This information should be publicly accessible. > We have subject matter experts and sometimes project leads those critical upstream projects. If you've got a strange problem, or need a feature implemented, we've got the people who can solve it. The best in the industry (at least for those who need the best, not everyone does). So does Oracle, or Google, or AWS, or hundreds of others. I participate on the devel mailing lists, and I pay attention to who contributes and who provides answers. Red Hat is definitely on this list, but Red Hat is not alone - and for the most part, this service is a "cost of doing business", and not directly tied to a subscription. When I had an issue with Qemu and live migrations of nested virtualization, it was Oracle that contributed the fixes for live migration of nested virtualization. I don't want to reduce the value of Red Hat here - I want to make it clear that Red Hat is one of several important players. > And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome. Especially the "you're welcome" at the end is arrogance. Actually, Red Hat has plenty of bugs after "hardening", many of which I have had to deal with, including getting fixes made upstream. But, I don't find Red Hat support that useful for this. Too often, I have to figure out for myself what is wrong, and I may as well describe the problem on bugzilla.redhat.com <http://bugzilla.redhat.com> myself. I don't need to open a support ticket and go through somebody else to do this. I know you don't want to hear my story. You would prefer the story that Red Hat is awesome, that Red Hat can set whatever price they wish and it should be considered a bargain, and anybody who disagrees is ignorant. But, you are talking about your customers here - including customers showing good will and willingness to negotiate. You forced me to substantially remove Red Hat from our systems, by refusing to negotiate acceptable terms. You did this. I didn't do this. -- Mark Mielke <mark.mielke@gmail.com <mailto:mark.mielke@gmail.com>>
Mark, I've heard responses like this for two weeks. You describe me as arrogant, and not listening, and I can only assume that you think we at Red Hat are fools who don't understand the enterprise. You've used a lot of fairly charged language in your response and you and many others have the "I'm going to show Red Hat" attitude. The problem is, and I think too few people realize this:
Red Hat isn't aiming for total global domination
So when you say I forced you to go somewhere else. You have to understand that in the open-source world, and in Red Hat's business plan, we know alternatives exist. That's the whole point of it. If you don't like the level of service you're getting. Go somewhere else. But don't pretend that the RHEL bits don't matter and that minimizing RHEL's contributions on a CentOS-devel mailing list will teach us something. You posted that your relationship with Red Hat boils down to cost per ticket - you did that, not me. I think if we've learned anything in the last two weeks it's that the bits seem to matter very much to people. They matter so much that people are feverishly trying to recreate RHEL instead of going to one of those many alternatives that already exist.
Just know that I mean it when I say, for those of you that are moving on. We wish you luck, we understand, and we'll see you around.
-Mike
It's sad to say, but CentOS-devel mailing list was more intersting without some Red Hat proud boys.
Jean-Marc
On Fri, Dec 18, 2020 at 09:03:42PM +0100, Jean-Marc Liger wrote:
Jesus... enough already.
Referring to people as Proud Boys isn't welcome on this list. I don't know if this is a regional thing or not but in this country at this moment in time Proud Boys, well, just stop it would ya?
Mike,
If you think telling people "we'll see you around" isn't being dismissive and condescending you're wrong. You're not helping the situation at all.
Also, could everyone learn to trim their replies?
On Fri, Dec 18, 2020 at 6:53 PM John R. Dennison jrd@gerdesas.com wrote:
On Fri, Dec 18, 2020 at 09:03:42PM +0100, Jean-Marc Liger wrote:
Mike,
If you think telling people "we'll see you around" isn't being dismissive and condescending you're wrong. You're not helping the situation at all.
Email does a pretty poor job of tone. Please note that up until now, I haven't been personally attacked on the list. The two replies before yours took it to that level.
By "We'll see you around" I really did mean that in a literal sense. I do actually attend conferences (and even their crappier counterpart the "virtual" conference until this COVID stuff is done). I personally know many of the people on this list. So when I say "I'll see you around". I mean "no hard feelings if you're not running a Red Hat OS next time I see you." I re-read it several times and I don't understand how that sounded condescending, but I'll take your word for it.
-Mike
Also, could everyone learn to trim their replies?
-- Humans hate to admit error even as they stand there, black and smoldering, with the stub of a cigarette in one hand, in the middle of a wide crater containing them and the remains of a sign that once read "DANGER: VOLATILE EXPLOSIVES".
-- James Nicoll (1961-), Canadian freelance game and speculative fiction reviewer, Usenet article (2005) _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/18/20 9:03 PM, Jean-Marc Liger wrote:
It's sad to say, but CentOS-devel mailing list was more intersting without some Red Hat proud boys.
I was just a small-time contributor to CentOS (maintaining the Vagrant images for 6 and 7), and I'm sad to see it go - sad, not angry. Seeing the discussion degenerate in personal attacks makes me even sadder. Disclosure: I do not, and never did work for Red Hat.
Your email address is from a French domain - I would like to believe you were not aware the "proud boys" is the name of a neo-fascist organization based in the USA, which was quite a few times in the US news recently.[1] That kind of comment crosses a line for me, no matter what Mr. McGrath might have decided or written above.
For people that never heard of Mike McGrath, he was the founder and architect of OpenShift, and, according to his LinkedIn page, corporate vice-president of Linux Engineering at Red Hat and management lead for RHEL8. I don't know him personally, but, management position aside, I think he probably is an intelligent and technologically apt person, who deserves more respect than invoking Godwin's law.[2] I'm also not sure how many upper managers at other companies would have spent many hours since the announcement on IRC and this mailing list, only to be (sometimes) shouted at and insulted. Mr. McGrath, in case you're reading this, sad as I am to see CentOS Linux go, many thanks for the years of financial and engineering support that Red Hat donated to the CentOS Project. I would also like to express my heartfelt thanks to Karanbir, Fabian and Brian, who have helped me all these years.
CentOS Stream might be great even for some productions roles, time will tell. It's definitely not a substitute for people needing 10 years of support or binary compatibility with RHEL (especially for expensive, proprietary hardware, whose vendors only support RHEL). I would have found it better if Red Hat did this before CentOS Linux 8 was released, or at least to have communicated openly that they are considering sunsetting it, but that doesn't change the reality: CentOS Linux, as we knew it and loved it, is dead, and we'll have to migrate to one of several RHEL rebuilds (OEL and Springdale were available for years, and Red Hat will continue to publish sources), or to some other LTS distro.
Best wishes, Laurențiu
[1] https://en.wikipedia.org/wiki/Proud_Boys [2] https://en.wikipedia.org/wiki/Godwin%27s_law
Le 19/12/2020 à 11:24, Laurențiu Păncescu a écrit :
On 12/18/20 9:03 PM, Jean-Marc Liger wrote:
It's sad to say, but CentOS-devel mailing list was more intersting without some Red Hat proud boys.
I was just a small-time contributor to CentOS (maintaining the Vagrant images for 6 and 7), and I'm sad to see it go - sad, not angry. Seeing the discussion degenerate in personal attacks makes me even sadder. Disclosure: I do not, and never did work for Red Hat.
Your email address is from a French domain - I would like to believe you were not aware the "proud boys" is the name of a neo-fascist organization based in the USA, which was quite a few times in the US news recently.[1] That kind of comment crosses a line for me, no matter what Mr. McGrath might have decided or written above.
For people that never heard of Mike McGrath, he was the founder and architect of OpenShift, and, according to his LinkedIn page, corporate vice-president of Linux Engineering at Red Hat and management lead for RHEL8. I don't know him personally, but, management position aside, I think he probably is an intelligent and technologically apt person, who deserves more respect than invoking Godwin's law.[2] I'm also not sure how many upper managers at other companies would have spent many hours since the announcement on IRC and this mailing list, only to be (sometimes) shouted at and insulted. Mr. McGrath, in case you're reading this, sad as I am to see CentOS Linux go, many thanks for the years of financial and engineering support that Red Hat donated to the CentOS Project. I would also like to express my heartfelt thanks to Karanbir, Fabian and Brian, who have helped me all these years.
CentOS Stream might be great even for some productions roles, time will tell. It's definitely not a substitute for people needing 10 years of support or binary compatibility with RHEL (especially for expensive, proprietary hardware, whose vendors only support RHEL). I would have found it better if Red Hat did this before CentOS Linux 8 was released, or at least to have communicated openly that they are considering sunsetting it, but that doesn't change the reality: CentOS Linux, as we knew it and loved it, is dead, and we'll have to migrate to one of several RHEL rebuilds (OEL and Springdale were available for years, and Red Hat will continue to publish sources), or to some other LTS distro.
Best wishes, Laurențiu
[1] https://en.wikipedia.org/wiki/Proud_Boys [2] https://en.wikipedia.org/wiki/Godwin%27s_law
Ok, let's consider it was a mistake to write it that way, I took you remarque your remark in consideration and apologize, as reading some answers puts more fuel on fire.
Red Hat must stop saying us ithis decision was a good one for the CentOS Communiity, it'not, it might have been a good one for IBM/Red Hat business and nothing all.
Many of us couldn't afford RHEL registration, so we were regularly contributed, at a low or high level depending on our motivation and our time, to the Red Hat ecosystem, not the Red Hat business model, knowing the latter cannot sustain the long term without the former.
Twenty years ago there was a company with best Unix Solaris and ZFS and Java which became arrogant and sold us expensive SUN Stations with poor IDE disks instead of strong SCSI one's. We bought DELL Servers with Red Hat Linux... And we can easily change some piece again.
A fair way of doing things would have been : "Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS Linux. It will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL."
Red Hat would have kill all the clones by releasing CentOS Linux first, the Community would have been happy and not anger to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled.
Jean-Marc
On Fri, 18 Dec 2020 13:03:47 -0600 Mike McGrath mmcgrath@redhat.com wrote:
problem is, and I think too few people realize this:
Red Hat isn't aiming for total global domination
Absolute complete and utter nonsense.
That is what any company aims for, even if they don't achieve it. It is the simple objective, and outcome, of capitalism. Beat the rest.
So this is about growing/expanding market share, which then equates to profit. Why wouldn't you?
(I'm not arguing the right or wrongs of this, but it is the natural conclusion, if you don't get broken up for monopolisation.)
If you argue anything else you really are quite simply gaslighting.
I really wish all the Hatters who have been wheeled out in defence of this decision stop their corporate mealy mouthed blather, which anyone can see straight through, and be honest.
Oh, and stop treating every CentOS user as a 'free loader'.
It really is quite offensive and you are aren't winning any hearts and minds right now.
Could we not use this list to share whatever nefarious purpose we assign to Red Hat? It's off topic and not civil.
On Mon, Dec 21, 2020 at 11:06 AM John Crisp jcrisp@safeandsoundit.co.uk wrote:
On Fri, 18 Dec 2020 13:03:47 -0600 Mike McGrath mmcgrath@redhat.com wrote:
problem is, and I think too few people realize this:
Red Hat isn't aiming for total global domination
Absolute complete and utter nonsense.
That is what any company aims for, even if they don't achieve it. It is the simple objective, and outcome, of capitalism. Beat the rest.
So this is about growing/expanding market share, which then equates to profit. Why wouldn't you?
(I'm not arguing the right or wrongs of this, but it is the natural conclusion, if you don't get broken up for monopolisation.)
If you argue anything else you really are quite simply gaslighting.
You have grossly oversimplified a complicated situation and completely ignored corporate responsibility which is something Red Hat takes very seriously.
I really wish all the Hatters who have been wheeled out in defence of this decision stop their corporate mealy mouthed blather, which anyone can see straight through, and be honest.
Oh, and stop treating every CentOS user as a 'free loader'.
It really is quite offensive and you are aren't winning any hearts and minds right now.
I'll admit something about Mark's reply didn't bring out the best in me. However, I'll take every opportunity to dote on the team and on Red Hat, they've done some amazing things that we have all benefited from. There is a lot more to Red Hat than the dollars and cents you're trying to distill us down into.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Mon, 21 Dec 2020 11:29:52 -0600 Mike McGrath mmcgrath@redhat.com wrote:
On Mon, Dec 21, 2020 at 11:06 AM John Crisp jcrisp@safeandsoundit.co.uk wrote:
On Fri, 18 Dec 2020 13:03:47 -0600 Mike McGrath mmcgrath@redhat.com wrote:
Red Hat isn't aiming for total global domination
That is what any company aims for, even if they don't achieve it. It is the simple objective, and outcome, of capitalism. Beat the rest.
So this is about growing/expanding market share, which then equates to profit. Why wouldn't you?
If you argue anything else you really are quite simply gaslighting.
You have grossly oversimplified a complicated situation and completely ignored corporate responsibility which is something Red Hat takes very seriously.
^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is called gaslighting.
As a good friend of mine said many moons ago. "Life is simple. It is people that make it complicated."
And for good reason. It is easy to hide amongst corporate complexity and a wall of long arguments over minutiae. Take a simple idea and complicate it.
Business really IS simple. Buy or create something, add value, and sell for more money than it cost you = profit.
How to increase that may be complex. But the base formula is not.
Of course, you can add morals & all that jazz. But none of it gets away from the fact that companies need profit, and shareholders need dividends, and the simplest way to increase profit is to either cut your highest costs - usually jobs - and send them somewhere cheaper, or increase market share, aka 'domination'.
Corporate responsibility? Yeah, I understand that. We like to be huggy too. But with no profit it doesn't mean a thing. You are out of business.
But of course, you know all this.
My point was/is please stop the platitudes and excuses and just say it like it is. Stop trying to pretend it is really complicated, and we couldn't possibly understand your feelings, when in fact it is fundamentally really simple.
It really is quite offensive and you are aren't winning any hearts and minds right now.
I'll admit something about Mark's reply didn't bring out the best in me. However, I'll take every opportunity to dote on the team and on Red Hat, they've done some amazing things that we have all benefited
Indeed. Seeing more and more frustrated RH people desperately try to defend the indefensible. I almost feel sorry for many there who probably feel quite betrayed, and those like you having to do the rounds on lists like this getting their ears bent trying to placate people with businesses that have just been destroyed over night (luckily I'm not one) by telling them what a great company RedHat is. Gaslighting.
There is a lot more to Red Hat than the dollars and cents you're trying to distill us down into.
There may well be, but I'm afraid when you do distil it down, no matter which way you try and cut it, the bottom line is always money.
If it wasn't for money, CentOS as we know it would not be being cut.
No, I don't doubt for a moment that you are all lovely people to meet down the bar (and I have met one or two). But this is a decision based on $$$ presumably trying to convert a cost into revenue and extinguishing the 'free loaders'.
Probably rammed home when the new RedHat owners asked about how you were going to increase revenue. They aren't known for beating about the bush with 'loss making' business. I'd guess a marketing and beancounter assessment of how many CentOS users converted to RH didn't fair well.
So this was a quick way to shave a few million in costs overnight, potentially gain a load of new subscribers, and get more testing by the rest. And your morals went to hell in a handcart.
It is really that simple. Please don't try and pretend it is otherwise.
Have a safe Xmas & New Year.
On Tue, Dec 22, 2020 at 6:32 PM John Crisp jcrisp@safeandsoundit.co.uk wrote:
On Mon, 21 Dec 2020 11:29:52 -0600 Mike McGrath mmcgrath@redhat.com wrote:
On Mon, Dec 21, 2020 at 11:06 AM John Crisp jcrisp@safeandsoundit.co.uk wrote:
On Fri, 18 Dec 2020 13:03:47 -0600 Mike McGrath mmcgrath@redhat.com wrote:
Red Hat isn't aiming for total global domination
That is what any company aims for, even if they don't achieve it. It is the simple objective, and outcome, of capitalism. Beat the rest.
So this is about growing/expanding market share, which then equates to profit. Why wouldn't you?
If you argue anything else you really are quite simply gaslighting.
You have grossly oversimplified a complicated situation and completely ignored corporate responsibility which is something Red Hat takes very seriously.
^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is called gaslighting.
As a good friend of mine said many moons ago. "Life is simple. It is people that make it complicated."
And for good reason. It is easy to hide amongst corporate complexity and a wall of long arguments over minutiae. Take a simple idea and complicate it.
Business really IS simple. Buy or create something, add value, and sell for more money than it cost you = profit.
How to increase that may be complex. But the base formula is not.
Of course, you can add morals & all that jazz. But none of it gets away from the fact that companies need profit, and shareholders need dividends, and the simplest way to increase profit is to either cut your highest costs - usually jobs - and send them somewhere cheaper, or increase market share, aka 'domination'.
Corporate responsibility? Yeah, I understand that. We like to be huggy too. But with no profit it doesn't mean a thing. You are out of business.
But of course, you know all this.
My point was/is please stop the platitudes and excuses and just say it like it is. Stop trying to pretend it is really complicated, and we couldn't possibly understand your feelings, when in fact it is fundamentally really simple.
It really is quite offensive and you are aren't winning any hearts and minds right now.
I'll admit something about Mark's reply didn't bring out the best in me. However, I'll take every opportunity to dote on the team and on Red Hat, they've done some amazing things that we have all benefited
Indeed. Seeing more and more frustrated RH people desperately try to defend the indefensible. I almost feel sorry for many there who probably feel quite betrayed, and those like you having to do the rounds on lists like this getting their ears bent trying to placate people with businesses that have just been destroyed over night (luckily I'm not one) by telling them what a great company RedHat is. Gaslighting.
There is a lot more to Red Hat than the dollars and cents you're trying to distill us down into.
There may well be, but I'm afraid when you do distil it down, no matter which way you try and cut it, the bottom line is always money.
If it wasn't for money, CentOS as we know it would not be being cut.
No, I don't doubt for a moment that you are all lovely people to meet down the bar (and I have met one or two). But this is a decision based on $$$ presumably trying to convert a cost into revenue and extinguishing the 'free loaders'.
Probably rammed home when the new RedHat owners asked about how you were going to increase revenue. They aren't known for beating about the bush with 'loss making' business. I'd guess a marketing and beancounter assessment of how many CentOS users converted to RH didn't fair well.
So this was a quick way to shave a few million in costs overnight, potentially gain a load of new subscribers, and get more testing by the rest. And your morals went to hell in a handcart.
It is really that simple. Please don't try and pretend it is otherwise.
Have a safe Xmas & New Year.
And herein lies the problem. I'm doing my best to answer questions. You are angry and have directed your anger at me, all of which is completely understandable. I could do without the namecalling but if you can't help yourself, I get it. I'm here doing my best to make sure people understand what is going on. And you're right, one of the many factors that went into this decision was business health and yes, as you point out, revenue is tied to business health. The problem is you've made several uninformed claims about Red Hat's business. For one, infrastructure revenues, as were publicly reported up through last year, have remained basically the same for years at Red Hat. That's public knowledge and is available in our form 10k, feel free to go read it from last year and previous years. I don't know who the anonymous "ex Red Hat exec" was quoted earlier but I don't think the entire paid Linux OS market is worth $10 billion. That just doesn't pass the sniff test though I guess that's for some analyst somewhere to figure out.
If Red Hat's new "owners" as you put it want to increase revenue, they wouldn't be doing that with RHEL, they'd be doing it with one of the emerging technologies that were growing at somewhere around triple the rate of our infrastructure revenue (Like an OpenShift or OpenStack). Again, there's no secret here, this was all available information up until about a year ago and not much has actually changed in the last year in the Linux OS market. Why would you invest a dollar into a bank account that could give 10% return, when another bank account would give you 30 or 40% return?
So, of course, this is where it gets complicated. Many, many factors went into this decision across many people in Red Hat and on the board. I think you honestly believe we had some meeting inside Red Hat and the agenda was "How to make more money by killing CentOS." Instead, we had several discussions both inside Red Hat, outside Red Hat, etc on what to do about CentOS over at least a couple of years that I was involved. It's been clear to us for a while that this model was pretty unhealthy. We all had our own wants out of this and I suspect no one got everything they wanted.
Many people on this list are looking to find a way forward here - to understand what is going on and provide suggestions where they see it might help. It is forgivable that you and others would jump to conclusions that equate to IBM, or greed, or revenue because you don't have all the information we do. But just know that when you again try to boil this all down to just a revenue discussion, you're missing a lot of detail that actually went into this.
-Mike
On Tue, Dec 22, 2020 at 7:14 PM Mike McGrath mmcgrath@redhat.com wrote:
And herein lies the problem. I'm doing my best to answer questions. You are angry and have directed your anger at me, all of which is completely understandable. I could do without the namecalling but if you can't help yourself, I get it. I'm here doing my best to make sure people understand what is going on. And you're right, one of the many factors that went into this decision was business health and yes, as you point out, revenue is tied to business health. The problem is you've made several uninformed claims about Red Hat's business. For one, infrastructure revenues, as were publicly reported up through last year, have remained basically the same for years at Red Hat.
Sorry for the self reply, I realized this last sentence was sufficiently ambiguous that I'd clarify it. Infrastructure revenues had constant growth, not significantly up or down. Anyway, we are now WAY off-topic for a -devel list so for those that want to know more about how to read form 10k's and public SEC documents, find me on IRC after the new year :)
-Mike
В 20:13 -0600 на 22.12.2020 (вт), Mike McGrath написа:
On Tue, Dec 22, 2020 at 7:14 PM Mike McGrath mmcgrath@redhat.com wrote:
And herein lies the problem. I'm doing my best to answer questions. You are angry and have directed your anger at me, all of which is completely understandable. I could do without the namecalling but if you can't help yourself, I get it. I'm here doing my best to make sure people understand what is going on. And you're right, one of the many factors that went into this decision was business health and yes, as you point out, revenue is tied to business health. The problem is you've made several uninformed claims about Red Hat's business. For one, infrastructure revenues, as were publicly reported up through last year, have remained basically the same for years at Red Hat.
Sorry for the self reply, I realized this last sentence was sufficiently ambiguous that I'd clarify it. Infrastructure revenues had constant growth, not significantly up or down. Anyway, we are now WAY off-topic for a -devel list so for those that want to know more about how to read form 10k's and public SEC documents, find me on IRC after the new year :)
Hey Mike,
Is there a chance to prolong the Stream's life any further ? Maybe with a SIG ? I think that if stream's life was a little bit longer , Stream could have a better adoption .
Best Regards, Strahil Nikolov
On Tue, Dec 22, 2020, 11:56 PM Strahil Nikolov via CentOS-devel < centos-devel@centos.org> wrote:
В 20:13 -0600 на 22.12.2020 (вт), Mike McGrath написа:
On Tue, Dec 22, 2020 at 7:14 PM Mike McGrath mmcgrath@redhat.com wrote:
And herein lies the problem. I'm doing my best to answer questions. You are angry and have directed your anger at me, all of which is completely understandable. I could do without the namecalling but if you can't help yourself, I get it. I'm here doing my best to make sure people understand what is going on. And you're right, one of the many factors that went into this decision was business health and yes, as you point out, revenue is tied to business health. The problem is you've made several uninformed claims about Red Hat's business. For one, infrastructure revenues, as were publicly reported up through last year, have remained basically the same for years at Red Hat.
Sorry for the self reply, I realized this last sentence was sufficiently ambiguous that I'd clarify it. Infrastructure revenues had constant growth, not significantly up or down. Anyway, we are now WAY off-topic for a -devel list so for those that want to know more about how to read form 10k's and public SEC documents, find me on IRC after the new year :)
Hey Mike,
Is there a chance to prolong the Stream's life any further ? Maybe with a SIG ? I think that if stream's life was a little bit longer , Stream could have a better adoption .
Best Regards, Strahil Nikolov
Low chance but not impossible. We just shook things up and the results of that shakeup are a little unclear. The question for a longer lifecycle on stream will depend entirely on why one of the soon-to-be announced RHEL programs won't work for you. So that's a discussion for another time.
-Mike
_______________________________________________
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 23.12.2020 08:14, Mike McGrath wrote:
On Tue, Dec 22, 2020 at 6:32 PM John Crisp <jcrisp@safeandsoundit.co.uk mailto:jcrisp@safeandsoundit.co.uk> wrote:
On Mon, 21 Dec 2020 11:29:52 -0600 Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>> wrote: > On Mon, Dec 21, 2020 at 11:06 AM John Crisp > <jcrisp@safeandsoundit.co.uk <mailto:jcrisp@safeandsoundit.co.uk>> wrote: > > > On Fri, 18 Dec 2020 13:03:47 -0600 > > Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>> wrote:
[...snip...]
So, of course, this is where it gets complicated. Many, many factors went into this decision across many people in Red Hat and on the board. I think you honestly believe we had some meeting inside Red Hat and the agenda was "How to make more money by killing CentOS." Instead, we had several discussions both inside Red Hat, outside Red Hat, etc on what to do about CentOS over at least a couple of years that I was involved. It's been clear to us for a while that this model was pretty unhealthy. We all had our own wants out of this and I suspect no one got everything they wanted.
Many people on this list are looking to find a way forward here - to understand what is going on and provide suggestions where they see it might help. It is forgivable that you and others would jump to conclusions that equate to IBM, or greed, or revenue because you don't have all the information we do. But just know that when you again try to boil this all down to just a revenue discussion, you're missing a lot of detail that actually went into this.
Oh, thank you so much, I didn't hope it could be forgivable. Now that we can actually be absolved, it's so much simpler to breathe.
Translating your last statements into English: you (those outside of RH) have not that information, you won't have that information, thus you may not come to any conclusions.
Again, this is extremely simple. The fundamental flaw of CentOS Linux was it didn't bring profits directly. On the contrary, the majority of sysadmins were using CentOS instead of RHEL, since CentOS was stable enough, was quite simple to maintain - and unless any specific requirements were rpesent, didn't require paying expensive bills from RH. That simple.
The missed lot of details doesn't change the above simple assumption. CentOS Linux was too good an alternative for majority of systems, thus that "Carthago delenda est". That simple. No need to permeate an aura of mystery.
What's really funny is the constant flow of BMWs (bovine metabolic wastes) from RH, trying to "prove" that CentOS Stream is a drop-on replacement for CentOS Linux (which it isn't) and that it fits 95% of CentOS Linux use cases (which it doesn't).
Happy New Year!
On Wed, Dec 23, 2020 at 7:50 PM Konstantin Boyandin via CentOS-devel < centos-devel@centos.org> wrote:
On 23.12.2020 08:14, Mike McGrath wrote:
On Tue, Dec 22, 2020 at 6:32 PM John Crisp <jcrisp@safeandsoundit.co.uk mailto:jcrisp@safeandsoundit.co.uk> wrote:
On Mon, 21 Dec 2020 11:29:52 -0600 Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>>
wrote:
> On Mon, Dec 21, 2020 at 11:06 AM John Crisp > <jcrisp@safeandsoundit.co.uk <mailto:jcrisp@safeandsoundit.co.uk>> wrote: > > > On Fri, 18 Dec 2020 13:03:47 -0600 > > Mike McGrath <mmcgrath@redhat.com <mailto:mmcgrath@redhat.com>> wrote:
[...snip...]
So, of course, this is where it gets complicated. Many, many factors went into this decision across many people in Red Hat and on the board. I think you honestly believe we had some meeting inside Red Hat and the agenda was "How to make more money by killing CentOS." Instead, we had several discussions both inside Red Hat, outside Red Hat, etc on what to do about CentOS over at least a couple of years that I was involved. It's been clear to us for a while that this model was pretty unhealthy. We all had our own wants out of this and I suspect no one got everything they wanted.
Many people on this list are looking to find a way forward here - to understand what is going on and provide suggestions where they see it might help. It is forgivable that you and others would jump to conclusions that equate to IBM, or greed, or revenue because you don't have all the information we do. But just know that when you again try to boil this all down to just a revenue discussion, you're missing a lot of detail that actually went into this.
Oh, thank you so much, I didn't hope it could be forgivable. Now that we can actually be absolved, it's so much simpler to breathe.
Translating your last statements into English: you (those outside of RH) have not that information, you won't have that information, thus you may not come to any conclusions.
Again, this is extremely simple. The fundamental flaw of CentOS Linux was it didn't bring profits directly. On the contrary, the majority of sysadmins were using CentOS instead of RHEL, since CentOS was stable enough, was quite simple to maintain - and unless any specific requirements were rpesent, didn't require paying expensive bills from RH. That simple.
It didn't have to bring in profits, but it did have to be mutually beneficial. In the beginning it was, but back then OpenStack was the future of computing. Things change. Take a look at Fedora - it doesn't bring in profits. But it is extremely useful to us and to the community. As Matthew can tell you, it takes a lot of hard work to run a community like that and actively balance the needs of Red Hat and the community. We run several communities that don't bring in profits. Red Hat also belongs to several foundations and other open-source groups, some of which we don't have any productization plans but we do them because we think they're good for the industry.
And really all I'm getting at here is that there is more to Red Hat than profits, we've proven that over and over. If this decision has caused you to lose that trust, to make you think that Red Hat cares only about profits, that's your right.
The missed lot of details doesn't change the above simple assumption. CentOS Linux was too good an alternative for majority of systems, thus that "Carthago delenda est". That simple. No need to permeate an aura of mystery.
What's really funny is the constant flow of BMWs (bovine metabolic wastes) from RH, trying to "prove" that CentOS Stream is a drop-on replacement for CentOS Linux (which it isn't) and that it fits 95% of CentOS Linux use cases (which it doesn't).
CentOS Stream isn't a drop-in replacement for CentOS Linux. There's not some crazy messaging to make people think they're the same. But they're also not entirely different and are far more compatible than what reporters and many were writing about on the day of the announcement (which is why we've worked so hard to correct that).
Let me give you one benefit of Stream that didn't exist in CentOS Linux that I *CANNOT* believe hasn't come up yet. Are you aware that there were at two months in 2020 where CentOS Linux 8 received no security updates of any kind? When the next y stream comes out, the team goes to building that y stream and stop working on the current version. People touting security and stability either weren't even aware of this, or have just decided not to discuss it. I can't imagine Stream would have that problem.
But no, make no mistake. Chris's blog was correct. CentOS Stream and CentOS Linux are different things in terms of deliverables. While at the same time RHEL and CentOS Stream are extremely closely related and the nuance between them will take a while for people to understand.
-Mike
Happy New Year!
-- Sincerely,
Konstantin Boyandin system administrator (ProWide Labs Ltd. - IPHost Network Monitor) _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 24.12.2020 09:16, Mike McGrath wrote:
On Wed, Dec 23, 2020 at 7:50 PM Konstantin Boyandin via CentOS-devel <centos-devel@centos.org mailto:centos-devel@centos.org> wrote: > Many people on this list are looking to find a way forward here -
to
> understand what is going on and provide suggestions where they see it > might help. It is forgivable that you and others would jump to > conclusions that equate to IBM, or greed, or revenue because you don't > have all the information we do. But just know that when you again try > to boil this all down to just a revenue discussion, you're missing a lot > of detail that actually went into this. Oh, thank you so much, I didn't hope it could be forgivable. Now that we can actually be absolved, it's so much simpler to breathe. Translating your last statements into English: you (those outside of RH) have not that information, you won't have that information, thus
you
may not come to any conclusions. Again, this is extremely simple. The fundamental flaw of CentOS Linux was it didn't bring profits directly. On the contrary, the majority of sysadmins were using CentOS instead of RHEL, since CentOS was stable enough, was quite simple to maintain - and unless any specific requirements were rpesent, didn't require paying expensive bills from RH. That simple.
It didn't have to bring in profits, but it did have to be mutually beneficial. In the beginning it was, but back then OpenStack was the future of computing. Things change. Take a look at Fedora - it doesn't
bring in profits. But it is extremely useful to us and to the community. As Matthew can tell you, it takes a lot of hard work to run a community like that and actively balance the needs of Red Hat and the community. We run several communities that don't bring in profits. Red
Hat also belongs to several foundations and other open-source groups, some of which we don't have any productization plans but we do them because we think they're good for the industry.
"Mutually beneficial" - for RH and whom, exactly?
CentOS Board, with all due respect, definitely doesn't speak for the entire community.
And really all I'm getting at here is that there is more to Red Hat than profits, we've proven that over and over. If this decision has caused you to lose that trust, to make you think that Red Hat cares only about profits, that's your right.
I am not your customer (although I worked closely with many of your customers, so I have impression about your tech. support and the rest), so obviously that proof evades me.
Since the exact reasoning of CentOS dismantling won't be a matter of public knowledge, I have no reasons to change the viewpoint. It was/is all about profit.
Of course I appreciate RH caring about open source (no irony here), but again, for any corporation profit is the first priority. All the rest is secondary.
The missed lot of details doesn't change the above simple assumption. CentOS Linux was too good an alternative for majority of systems, thus that "Carthago delenda est". That simple. No need to permeate an aura of mystery. What's really funny is the constant flow of BMWs (bovine metabolic wastes) from RH, trying to "prove" that CentOS Stream is a drop-on replacement for CentOS Linux (which it isn't) and that it fits 95% of CentOS Linux use cases (which it doesn't).
CentOS Stream isn't a drop-in replacement for CentOS Linux. There's not some crazy messaging to make people think they're the same. But they're also not entirely different and are far more compatible than what reporters and many were writing about on the day of the announcement (which is why we've worked so hard to correct that).
Of course they are not entirely different - they both have "CentOS" string in them.
I am glad you avoided mentioning that statement that "CentOS Stream would cover 95% of CentOS Linux use cases" (in my words). For clarification, can *you*, explicitly, either agree to that statement or disagree with it?
Let me give you one benefit of Stream that didn't exist in CentOS Linux that I *CANNOT* believe hasn't come up yet. Are you aware that there were at two months in 2020 where CentOS Linux 8 received no security updates of any kind? When the next y stream comes out, the team goes to building that y stream and stop working on the current version. People touting security and stability either weren't even aware of this, or have just decided not to discuss it. I can't imagine Stream would have that problem.
That kind of sh*t happened now and then all the time. As a sysadmin, I got used to be constantly wary and bring emergency measures in similar cases; sometimes, in a matter of hours. So, to me, it's a regular event.
That difference makes sense for those who will actually be using CentOS Stream. People on this list provided detailed explanation why CentOS Stream can't fit many typical CentOS Linux use cases.
For generic use - for generic systems, such as VPSes - CentOS Stream can be very useful, when nothing really mission critical depends on it (when a system can be safely rebuilt, when another Stream update breaks it). I suppose it can be very useful in cases similar to Fedora - for development purposes.
But no, make no mistake. Chris's blog was correct. CentOS Stream and CentOS Linux are different things in terms of deliverables. While at the same time RHEL and CentOS Stream are extremely closely related and the nuance between them will take a while for people to understand.
Yes. And CentOS Linux users, meanwhile, just migrate to other distributions (Rocky, Lenix, Springdale, whatever), lifting a burden of unnecessary expenses off RH.
Apart from RH's reputation losses (I suppose RH just doesn't care about that), there remains not much to grieve upon.
On Tue, Dec 22, 2020 at 7:32 PM John Crisp jcrisp@safeandsoundit.co.uk wrote:
This is called gaslighting.
As a good friend of mine said many moons ago. "Life is simple. It is people that make it complicated."
John? Get off the nice peoples' case, please? They're unlikely to be allies if you're rude to them.
Your friend seems confused. Making up rules or goals is trivial. Actually doing the work, well, computers are all 1's and 0's, they couldn't be *complicated*.
On 12/17/2020 12:03 AM, Simon Matter wrote:
Le 16/12/2020 à 12:35, Jim Jagielski a écrit :
On Dec 15, 2020, at 10:07 PM, Karsten Wade kwade@redhat.com wrote:
On 12/15/20 1:03 PM, Jim Jagielski wrote:
Whose fault is that? And, to be honest, I never recall such an expectation ever being vocalized during my tenure @ RedHat (FTR: I was one of the people inside OSAS who drove the CentOS "acquisition" along w/ Carl Trieloff). The whole intent back then was "as long as there is going to be this huge community of 'free-loading' users out there, they might as well be under the RHEL/Fedora umbrella, rather than Canonical or elsewhere." I guess somewhere along the line that changed. The issue isn't that the situation changed but rather that up until very recently, promises were still being made and then RedHat backed out of those promises.
It is actually just as Jason Brooks has spelled out—we needed a slower-moving platform for our layered projects success. CentOS Linux was our best bet in 2013, when projects like OpenStack (RDO) and oVirt were growing and running into pains.
Again, the main concern was that with such layered products, it was deemed better if instead of people using Canonical, they stayed in the RedHat family, and officially having CentOS supported as a RedHat "effort" was the solution.
Yes, people were not going to run OpenStack (or OpenShift) on Fedora, nor did it make sense to try to fold those directly into RHEL. CentOS was the "perfect" solution. The goal of SIGs was to determine what layered products, and in what format, people wanted. But the idea that CentOS was intended to be a 50/50 bidirectional codebase is simply rewriting history. The claim that the CentOS community never changed from what it was, and what RedHat *knew* it was, and what RedHat over the years (at least publicly) constantly indicated they were 100% happy about (That CentOS was a community of *users*) just seems like after the fact justification, with the sole intent of placing the blame ON CENTOS.
It now seems crystal clear Red Hat purchased CentOS 6 years ago as it was the best OpenStack infrastructure for their purpose. And the best value of this CentOS - Red Hat joint effort was not the binary rebuild of RHEL, but all the additionnal SIGs provided with CentOS 7.
With Red Hat now focused on OpenShift, this golden age as ended and CentOS Linux wasn't necessary in the suitable form it had always been for years. This turned in a way that betrayed all the Red Hat promises, the Community Entreprise OS was first and only interested for.
As in 2003 where Red Hat was the leading distro in the Linux World, the trust has been broken again and many will flee to Debian or Ubuntu LTS,
Now that you mention 2003, I remember that time and one thing became clear to me. This new direction with CentOS is most likely inspired from the top of RedHat. It can only happen with the support of the top management and it is, if you ask me, inspired by the same people who decided the things back in 2003. Unfortunately they seem to have missed to realize the reasons why things have worked back then.
I do wonder if there was some sort of generational change at root here that has not been made explicit. Or perhaps some underlying tribal knowledge about how OSS works that failed to get transmitted? It's easy to make assumptions that stability and ideology gets passed down automatically, but just like in real life it takes eternal vigilance to re-instill foundational concepts.
I refuse to believe RedHat itself has /actually/ been this myopic _this entire time_ about its relationship with the wider Enterprise Linux -derived community, and how this feeds back into revenue, stability, paid support, certifications, training, and the all-important "mindshare." Maybe that's just wishful thinking, though?
On 12/16/2020 4:17 PM, Mark Mielke wrote:
On Wed, Dec 16, 2020 at 6:52 PM Mike McGrath mmcgrath@redhat.com wrote:
I think many of you think that some implicit guarantee was made, or are applying some standard to CentOS similar to what you would a contractual agreement and those will never be the same thing.
This isn't how Free / Open Source projects work. It is not normal for a community that exists precisely to provide a particular feature, is "acquired" by a company that claims to have the community interest at heart, and then leverages this power to replace the product with something that provides value to the company, and does not directly compete with company.
I have no doubt of your sincerity. However, I also believe that you may have been surrounded by other people with similar conflicts of interest and created a sort of "echo chamber" that after several months made it seem entirely reasonable to do.
This reaction was entirely predictable... and I mean *entirely*. How in the world was this missed? Frankly, this would have hurt far less if this /had/ been blamed on IBM, and I'm rather shocked no one above the Board felt the need to push the hard call UP rather than scapegoat down. That could be the most tone-deaf thing going on here.
-jc
On Tue, Dec 15, 2020 at 3:50 PM Mike McGrath mmcgrath@redhat.com wrote:
On Tue, Dec 15, 2020 at 1:41 PM Trevor Hemsley < trevor.hemsley@ntlworld.com> wrote:
On 15/12/2020 17:59, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Thank you for this clarification although it was fairly apparent to everyone what the driver was behind this change.
I'd like to thank Red Hat for supporting the CentOS Project from 2014 to 2020. You did a good thing by stepping in to save the project from disintegration back in 2014. Thanks for that, CentOS would probably have survived without you but you did the right thing and stepped up when you were needed.
However...
While Red Hat may *legally* own the CentOS Project, I do not believe you are *morally* entitled to do what you have done. CentOS is not just about the project and the contributors to it. It's more than that. It has millions of users, so many that no-one really knows how many there are. Lots of those users may be large corporations "freeloading" as Red Hat probably see it but others, those are small users running single machines or just a few. Those users are *your* future.
You (Red Hat) made a lot of promises both in 2014 and as late as last year when Chris Wright said something along the lines of classic CentOS Linux is not going anywhere. It's all very well to say that things change, well of course they do, but when they do, you have an obligation to live up to your promises and the recent actions were in no way doing that.
I believe the correct action for Red Hat to have taken would have been to say "we have decided that we no longer wish to fund the CentOS Project as it no longer aligns with our business purposes. So, in order not to let down the millions of users of CentOS Linux, we have decided to set up a foundation and donate the trade marks and domain names (that we acquired for almost nothing)".
With a decent legal founding, you could have made it takeover proof so that none of your competitors could acquire it. You could have done this and asked a number of the larger companies that have CentOS as part of their portfolio to sponsor the foundation - the Googles/AWS/OVH/cpanel's of this world could easily have stepped up and funded a FTE or 2 by donating to the foundation and you could have transferred some or all of the existing people who work on CentOS to that foundation and let *them* run it. Those hosting companies spin up new CentOS instances all the time and a cent or two donation on each instance would most likely fund most of what's required. And the people who are now scrambling around attempting to set up new hardware and build environments, they could be supporting the CentOS Linux Foundation instead.
The fact that you decided to take CentOS Linux out the back and shoot it in the head is a betrayal of your company's promises over the last 6 or 7 years. It's exactly what everyone was afraid of when Red Hat took over CentOS in 2014 and despite numerous questions, you all said "no no, it's safe with us". Some of us remember those days and arguing with people about whether it was a good thing or not and a lot of us said "Trust Red Hat, see what they do, look at their actions not their words". Well we did.
You should rename CentOS Stream to Red Hat Stream Linux (RHSL) and remove CentOS from the Red Hat family altogether. Donate the trade marks and logos and domain names and the tooling needed to produce CentOS Linux. Set up a foundation. Get the big players who offer CentOS to users to help fund the foundation. Ask the employees who work on CentOS on a daily basis if they'd like to stay with Red Hat or transfer to the new foundation. Find some way in which users can contribute to the foundation and ensure its future.
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free.
You can say that you think people are coming round to this. I do not agree. I have read all of the feedback on IRC, all of the feedback on the CentOS forums, all the feedback on the mailing lists. This is *not* a popular change. It's tarnishing and poisoning Red Hat's reputation and until it's addressed it will continue to do so. You can help to fix this before Red Hat becomes tarred with the same brush as that other big company with the big red logo and the not so great reputation. This is NOT just a $$$ decision, it has other ramifications and right now, Red Hat are the bad guys and will remain so until this is addressed.
You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
I'm in this weird position where I'm regularly hearing from people that thought that Red Hat made some sort of "We'll never change and CentOS Linux will be around forever" announcement. I'd suggest everyone go back and re-read the original press release (I was not involved with the original agreement) - https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-force...
I'd suggest everyone go to this post where Rich Bowen assured us *last year*, via the CentOS blog, that nothing would change:
https://blog.centos.org/2019/07/ibm-red-hat-and-centos/
I quote, "Red Hat always has and will continue to be a champion for open source and projects like CentOS." and, in particular, "Our mission, governance, and objectives remain the same. We will continue to execute the existing project roadmap."
At that point the RHEL 8 roadmap was published, and CentOS 8.0 was being worked on. It was released two months later. So, the "existing roadmap" included the full course of CentOS 8.
Sure, things change. But all Red Hat had to do was wait until RHEL/CentOS 9 to enact this, and give us plenty of warning. You're paying for CentOS 7 engineering through 2024 (you BETTER be!), so what additional cost would there be to continue CentOS 8 until then? By then it would be in maintenance mode and resources could be drawn down, or moved, or whatever and minimal effort would be needed. Meanwhile, Stream would still be spinning up, and you'd still get whatever benefit you want from it.
I just don't understand the timing here.
And I'm mad as hell Red Hat reneged on their commitment to us.
You can nitpick at words, or take a quote out of context. But don't be
naive and pretend we had some grand plan for all of this from the beginning. Just like anyone, Red Hat changes and makes decisions based on the best information we have at the time. CentOS Linux made sense in 2014, it doesn't make sense in 2020.
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users. And I know there are community members out there who are actively contributing time on QE as you did Trevor. You are a very small minority on this project and I hope we can win you over in CentOS Stream. As for the rest of you, where were you?
And sure, we could have turned CentOS back over to some non-Red Hat foundation. But the fact is contrary to popular belief, we actually like the engineers that work on CentOS, we like many of the users who have cultivated relationships with Red Hat over the years. We intend on going forward with a healthy, mutually beneficial relationship there. If that's not you, and you're ready to leave - I think that's unfortunate but I understand.
-Mike
Trevor Hemsley
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Dear Mike,
You are right and at the very same time insanely wrong:
``` CentOS Linux made sense in 2014, it doesn't make sense in 2020. ```
The whole discussion, podcasts, articles and how popular CentOS still prove otherwise.
``` You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users. ```
**CentOS had one job - provide a stable platform based on RHEL sources.** CentOS do not have to evolve. This does not make any sense... (That why I said that you are insanely wrong). You might ask why CentOS didn't start "next revolutionary" OS like CoreOS (that I personally really liked) - because there was no need.
" When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong."
CentOS is/was used in the biggest clusters around the world. This clusters made calculations that made a breakthrough in fields of physic, medicine, computer science and much much more. CentOS is/was in any serious industry like health care, finances or less demanding but still quite crucial for our lives like media, marketing, or social media and much more. I believe that there was no better place to put your talent.
When you are using a system like CentOS, your stack is likely evolving - but on the very stable platform. Your company could change database version a few times on **one** CentOS version. Your company can run the hottest application backend on CentOS. Did you know that PostgreSQL 12 was released for EL (Enterprise Linux) version 6? One system that could be used for like 8 (you can check number) major releases of the database. And it doesn't really matter if you are using a public cloud, private cloud, containers workloads. There is always OS. So for admins "evolutions" in many cases is same s*** different day.
I'm also deeply sorry that admins/devops/adminops/whateverops are not evolving as fast as javascript frameworks. A lot of people like it this way. They won't support solutions that are unstable - because in case of a f***up, they are the one to take a hit.
When a company that I work for decided to use Scientific Linux (v6 and v7) as a base instead of CentOS many people were sceptical. When I decided that we won't make version 8 based on CentOS I already knew that CentOS soul was sold:
- CentOS blocked koji (x86_64, I'm really grateful that arm koji was [maybe still is] open) - CentOS never wrote how to do modules and how it's done. It's always like "there is repo and we are using koji". - CentOS removed the most of *-devel packages because RHEL removed them. It doesn't make any sense. It brought multiple times on epel-devel mailing list. CentOS was never supported, so why remove unsupported packages instead of moving them to "all" or "whatever" repo? - The packages that are required to build a base system are not present in both CentOS and RHEL -> it means that you cannot rebuild the system yourself.
To sum this up - CentOS was great part of our stack. The part that was solid and now it's gone. I didn't have to evolve, we don't need fancier screwdriver just the one that works.
Best, Alex
PS. I'm just sorry for RH. It used to great company. I'm sorry that the next quarter revenue is more important than clients, users, community, market adaptation and keeping your promises.
On 12/15/20 9:50 PM, Mike McGrath wrote:
On Tue, Dec 15, 2020 at 1:41 PM Trevor Hemsley <trevor.hemsley@ntlworld.com mailto:trevor.hemsley@ntlworld.com> wrote:
On 15/12/2020 17:59, Mike McGrath wrote:
I'd also just add that while I find Johnny's characterization of what happened accurate, Ljubomir took a couple of leaps that I don't think existed. Red Hat decided not to continue paying actual money for what was actively harming us and no longer providing the value that it once did. No one, not even the board, could force Red Hat to continue paying for this project which was just not working for us. I'm not going to say that the announcement was the board's idea or even that they were happy about it. I think the previous course and speed of CentOS was well understood. But that no longer worked for Red Hat who is paying for people, servers, swag, etc. The list goes on.
Thank you for this clarification although it was fairly apparent to everyone what the driver was behind this change. I'd like to thank Red Hat for supporting the CentOS Project from 2014 to 2020. You did a good thing by stepping in to save the project from disintegration back in 2014. Thanks for that, CentOS would probably have survived without you but you did the right thing and stepped up when you were needed. However... While Red Hat may *legally* own the CentOS Project, I do not believe you are *morally* entitled to do what you have done. CentOS is not just about the project and the contributors to it. It's more than that. It has millions of users, so many that no-one really knows how many there are. Lots of those users may be large corporations "freeloading" as Red Hat probably see it but others, those are small users running single machines or just a few. Those users are *your* future. You (Red Hat) made a lot of promises both in 2014 and as late as last year when Chris Wright said something along the lines of classic CentOS Linux is not going anywhere. It's all very well to say that things change, well of course they do, but when they do, you have an obligation to live up to your promises and the recent actions were in no way doing that. I believe the correct action for Red Hat to have taken would have been to say "we have decided that we no longer wish to fund the CentOS Project as it no longer aligns with our business purposes. So, in order not to let down the millions of users of CentOS Linux, we have decided to set up a foundation and donate the trade marks and domain names (that we acquired for almost nothing)". With a decent legal founding, you could have made it takeover proof so that none of your competitors could acquire it. You could have done this and asked a number of the larger companies that have CentOS as part of their portfolio to sponsor the foundation - the Googles/AWS/OVH/cpanel's of this world could easily have stepped up and funded a FTE or 2 by donating to the foundation and you could have transferred some or all of the existing people who work on CentOS to that foundation and let *them* run it. Those hosting companies spin up new CentOS instances all the time and a cent or two donation on each instance would most likely fund most of what's required. And the people who are now scrambling around attempting to set up new hardware and build environments, they could be supporting the CentOS Linux Foundation instead. The fact that you decided to take CentOS Linux out the back and shoot it in the head is a betrayal of your company's promises over the last 6 or 7 years. It's exactly what everyone was afraid of when Red Hat took over CentOS in 2014 and despite numerous questions, you all said "no no, it's safe with us". Some of us remember those days and arguing with people about whether it was a good thing or not and a lot of us said "Trust Red Hat, see what they do, look at their actions not their words". Well we did. You should rename CentOS Stream to Red Hat Stream Linux (RHSL) and remove CentOS from the Red Hat family altogether. Donate the trade marks and logos and domain names and the tooling needed to produce CentOS Linux. Set up a foundation. Get the big players who offer CentOS to users to help fund the foundation. Ask the employees who work on CentOS on a daily basis if they'd like to stay with Red Hat or transfer to the new foundation. Find some way in which users can contribute to the foundation and ensure its future. It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free. You can say that you think people are coming round to this. I do not agree. I have read all of the feedback on IRC, all of the feedback on the CentOS forums, all the feedback on the mailing lists. This is *not* a popular change. It's tarnishing and poisoning Red Hat's reputation and until it's addressed it will continue to do so. You can help to fix this before Red Hat becomes tarred with the same brush as that other big company with the big red logo and the not so great reputation. This is NOT just a $$$ decision, it has other ramifications and right now, Red Hat are the bad guys and will remain so until this is addressed. You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
I'm in this weird position where I'm regularly hearing from people that thought that Red Hat made some sort of "We'll never change and CentOS Linux will be around forever" announcement. I'd suggest everyone go back and re-read the original press release (I was not involved with the original agreement) - https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-force... https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-forces
You can nitpick at words, or take a quote out of context. But don't be naive and pretend we had some grand plan for all of this from the beginning. Just like anyone, Red Hat changes and makes decisions based on the best information we have at the time. CentOS Linux made sense in 2014, it doesn't make sense in 2020.
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong. For whatever reason, CentOS never grew beyond a community of users. And I know there are community members out there who are actively contributing time on QE as you did Trevor. You are a very small minority on this project and I hope we can win you over in CentOS Stream. As for the rest of you, where were you?
And sure, we could have turned CentOS back over to some non-Red Hat foundation. But the fact is contrary to popular belief, we actually like the engineers that work on CentOS, we like many of the users who have cultivated relationships with Red Hat over the years. We intend on going forward with a healthy, mutually beneficial relationship there. If that's not you, and you're ready to leave - I think that's unfortunate but I understand.
-Mike
Trevor Hemsley
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/15/20 3:50 PM, Mike McGrath wrote:
On Tue, Dec 15, 2020 at 1:41 PM Trevor Hemsley <trevor.hemsley@ntlworld.com mailto:trevor.hemsley@ntlworld.com> wrote:
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free. ... You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
I'm in this weird position where I'm regularly hearing from people that thought that Red Hat made some sort of "We'll never change and CentOS Linux will be around forever" announcement. I'd suggest everyone go back and re-read the original press release (I was not involved with the original agreement) - https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-force... https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-forces
First, Mike, thanks for taking the time to reply here; I for one greatly appreciate it. I like Red Hat, and I like CentOS, and I count several current and former Red Hat employees as long-time close acquaintances, and even friends. Finally got to meet Bill Nottingham a few weeks ago, but there are many from the 'old days' I've not yet had opportunity to meet in person.
You can nitpick at words, or take a quote out of context. But don't be naive and pretend we had some grand plan for all of this from the beginning. Just like anyone, Red Hat changes and makes decisions based on the best information we have at the time. CentOS Linux made sense in 2014, it doesn't make sense in 2020.
The quote that has me somewhat riled up is from https://www.redhat.com/en/blog/transforming-development-experience-within-ce... : "CentOS Stream is parallel to existing CentOS builds; this means that nothing changes for current users of CentOS Linux and services, even those that begin to explore the newly-released CentOS 8. " In the context of the posted statement, and being the first sentence under the heading of "What does this mean for CentOS," this is to me a clear statement that nothing changes with the then-current (2019) CentOS model especially for users of CentOS 8; had CentOS 8 not been so specifically called out I personally wouldn't feel quite so blindsided. Yes, business happens; yes, real life happens; yes promises get broken. But broken promises produce broken relationships and have consequences, such as broken trust.
I started deployment into production based on that September 24, 2019 statement, and the published 2029 EOL date; I specifically waited on CentOS 8's ecosystem to mature for upgrades from C6 rather than go ahead an upgrade to C7 based on this statement, delaying, in one instance, a workstation upgrade six months, with the user of that workstation complaining about the delay nearly daily, and my reply being "In order to get your new system to be stable until 2029 instead of 2024 I still think we should go with CentOS 8 for you;" if I had known then what I know now she would have gotten CentOS 7 and I would have had many less headaches for most of a year.
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong.
There are a number of things that could have been done early on to make things different, in my opinion. And you're right: the CentOS community has seen very little change in many years. Some call it stagnation; some call it stability. Some might even call it denial.
Using the Debian model from the get-go (unstable -> testing -> stable) instead of (Fedora+secretsauce -> RHEL x.0 beta) might have helped back then; that IS what CentOS Stream is doing, adding that 'testing' portion, although you guys at Red Hat might prefer to not call it that. The sequence becomes, if I'm thinking this through correctly, (Fedora+secretsauce -> RHEL x.0 Beta -> RHEL x.0 -> for $point in 1 2 3 4 5 6 do (CentOS x Stream -> RHEL x.${point}); done) or somesuch.
But then there's the 'secretsauce' part of the issue. RHEL development has been opaque since RHLEE 6.2E; RHL development was opaque prior to that, even. I was on the Beta Team back in the day; I remember the city code names, the NDA, etc; I still have my executed copy of that NDA, for that matter, so there are certain cities I never speak of (:-) in case you missed the humor); still have some of the boxed sets from those days, too. Lack of transparency is not a new issue. CentOS Stream, in this regard, is a very refreshing development and will likely be a big win for transparency. But will CentOS Stream eliminate the 'secretsauce' bridge currently existing between Fedora and RHEL? (It may be covered in one of the numerous posts about it, so forgive me if I missed it). Will CentOS 9 Stream begin with a Fedora snapshot? (How that develops may not be even known by you guys at Red Hat as far as I know). If a transparent path from Fedora through the entire Full Support phase of RHEL can be developed in the form of CentOS Stream, I think that's a very good thing. I wouldn't mind using that myself, as long as hardware drivers don't frequently break that I need to use. And currently they break every stinking point release!!!
As for the rest of you, where were you?
The mailing list archives are full of messages over the years from users offering to help in various ways. Me personally? Well, my name and email were still in the changelog for PostgreSQL through the end of CentOS 4, at least; I did my contributions upstream when I was able to spend the time to do so. Life happens; those contributions had to yield to RL issues.
And sure, we could have turned CentOS back over to some non-Red Hat foundation. But the fact is contrary to popular belief, we actually like the engineers that work on CentOS, we like many of the users who have cultivated relationships with Red Hat over the years. We intend on going forward with a healthy, mutually beneficial relationship there. If that's not you, and you're ready to leave - I think that's unfortunate but I understand.
I seem to be one of the very few in my local area still advocating Red Hat-based systems. Most of the Linux talent in my area is rabidly anti-Red Hat. It got exponentially worse last week. Several years back, I got to meet Michael Tiemann and Mark Webbink in Asheville; watched TruthHappens the first time; everything was so positive. Michael accepted my invitation to tour $dayjob, and he seemed to have a great time. For a while the local LUGs were less anti-Red Hat.... How times have changed in the LUG-scape around here.
If it could be done over? The most detrimental statements made, in my opinion, were those statements that could be read to imply a commitment by Red Hat for CentOS 8 until 2029; this was after all the published roadmap by the CentOS Prject prior to December. In the press release you linked to, there is a now-dead link to a FAQ page; thanks to the Wayback Machine, I can re-read this FAQ page ( https://web.archive.org/web/20140906203025/http://community.redhat.com/cento... ) and see that maybe it was wishful thinking the way I read the very positively-worded spin that FAQ put on virtually all questions.
And then there's the statement I quoted at the very beginning of this reply; there was wording that strongly implied a commitment that "nothing changes" specifically for CentOS 8. The September 24, 2019 statement, in my opinion, is what primarily set the stage for this backlash you see today.
Best regards.
On Wed, Dec 16, 2020 at 11:59 AM Lamar Owen lowen@pari.edu wrote:
On 12/15/20 3:50 PM, Mike McGrath wrote:
On Tue, Dec 15, 2020 at 1:41 PM Trevor Hemsley <trevor.hemsley@ntlworld.com mailto:trevor.hemsley@ntlworld.com>
wrote:
It's not too late to do the right thing. Red Hat can still back off this betrayal of the community that use CentOS Linux and set CentOS Linux free. ... You can hope it'll go away but it won't. Red Hat will always be the company that broke its promises and killed CentOS Linux.
I'm in this weird position where I'm regularly hearing from people that thought that Red Hat made some sort of "We'll never change and CentOS Linux will be around forever" announcement. I'd suggest everyone go back and re-read the original press release (I was not involved with the original agreement) -
https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-force...
<
https://www.redhat.com/en/about/press-releases/red-hat-and-centos-join-force...
First, Mike, thanks for taking the time to reply here; I for one greatly appreciate it. I like Red Hat, and I like CentOS, and I count several current and former Red Hat employees as long-time close acquaintances, and even friends. Finally got to meet Bill Nottingham a few weeks ago, but there are many from the 'old days' I've not yet had opportunity to meet in person.
You can nitpick at words, or take a quote out of context. But don't be naive and pretend we had some grand plan for all of this from the beginning. Just like anyone, Red Hat changes and makes decisions based on the best information we have at the time. CentOS Linux made sense in 2014, it doesn't make sense in 2020.
The quote that has me somewhat riled up is from
https://www.redhat.com/en/blog/transforming-development-experience-within-ce... : "CentOS Stream is parallel to existing CentOS builds; this means that nothing changes for current users of CentOS Linux and services, even those that begin to explore the newly-released CentOS 8. " In the context of the posted statement, and being the first sentence under the heading of "What does this mean for CentOS," this is to me a clear statement that nothing changes with the then-current (2019) CentOS model especially for users of CentOS 8; had CentOS 8 not been so specifically called out I personally wouldn't feel quite so blindsided. Yes, business happens; yes, real life happens; yes promises get broken. But broken promises produce broken relationships and have consequences, such as broken trust.
That was accurate at the time of that writing as no decisions had been made.
I started deployment into production based on that September 24, 2019 statement, and the published 2029 EOL date; I specifically waited on CentOS 8's ecosystem to mature for upgrades from C6 rather than go ahead an upgrade to C7 based on this statement, delaying, in one instance, a workstation upgrade six months, with the user of that workstation complaining about the delay nearly daily, and my reply being "In order to get your new system to be stable until 2029 instead of 2024 I still think we should go with CentOS 8 for you;" if I had known then what I know now she would have gotten CentOS 7 and I would have had many less headaches for most of a year.
It makes a lot of sense that people would be upset about this. We very much should have set better expectations at the launch of CentOS 8 but at that point no specific dates around CentOS8 had been decided other than to release it.
You may not like it, but the CentOS community didn't evolve in any way with the industry. When I think about the talent on this list, and in IRC, I can't help but wonder what went wrong.
There are a number of things that could have been done early on to make things different, in my opinion. And you're right: the CentOS community has seen very little change in many years. Some call it stagnation; some call it stability. Some might even call it denial.
Using the Debian model from the get-go (unstable -> testing -> stable) instead of (Fedora+secretsauce -> RHEL x.0 beta) might have helped back then; that IS what CentOS Stream is doing, adding that 'testing' portion, although you guys at Red Hat might prefer to not call it that. The sequence becomes, if I'm thinking this through correctly, (Fedora+secretsauce -> RHEL x.0 Beta -> RHEL x.0 -> for $point in 1 2 3 4 5 6 do (CentOS x Stream -> RHEL x.${point}); done) or somesuch.
But then there's the 'secretsauce' part of the issue. RHEL development has been opaque since RHLEE 6.2E; RHL development was opaque prior to that, even. I was on the Beta Team back in the day; I remember the city code names, the NDA, etc; I still have my executed copy of that NDA, for that matter, so there are certain cities I never speak of (:-) in case you missed the humor); still have some of the boxed sets from those days, too. Lack of transparency is not a new issue. CentOS Stream, in this regard, is a very refreshing development and will likely be a big win for transparency. But will CentOS Stream eliminate the 'secretsauce' bridge currently existing between Fedora and RHEL? (It may be covered in one of the numerous posts about it, so forgive me if I missed it). Will CentOS 9 Stream begin with a Fedora snapshot? (How that develops may not be even known by you guys at Red Hat as far as I know). If a transparent path from Fedora through the entire Full Support phase of RHEL can be developed in the form of CentOS Stream, I think that's a very good thing. I wouldn't mind using that myself, as long as hardware drivers don't frequently break that I need to use. And currently they break every stinking point release!!!
The RHEL9 bootstrap process has already started with Fedora ELN and I'm expecting the CentOS Stream 9 code/builds to be showing up in the next few months, sometime before May. (note, the 5 year lifecycle of CentOS stream doesn't actually starts then, we don't start the clock until RHEL9 ships so these are all "pre-releases).
 As for the rest of you, where were you?
The mailing list archives are full of messages over the years from users offering to help in various ways. Me personally? Well, my name and email were still in the changelog for PostgreSQL through the end of CentOS 4, at least; I did my contributions upstream when I was able to spend the time to do so. Life happens; those contributions had to yield to RL issues.
And sure, we could have turned CentOS back over to some non-Red Hat foundation. But the fact is contrary to popular belief, we actually like the engineers that work on CentOS, we like many of the users who have cultivated relationships with Red Hat over the years. We intend on going forward with a healthy, mutually beneficial relationship there. If that's not you, and you're ready to leave - I think that's unfortunate but I understand.
I seem to be one of the very few in my local area still advocating Red Hat-based systems. Most of the Linux talent in my area is rabidly anti-Red Hat. It got exponentially worse last week. Several years back, I got to meet Michael Tiemann and Mark Webbink in Asheville; watched TruthHappens the first time; everything was so positive.  Michael accepted my invitation to tour $dayjob, and he seemed to have a great time. For a while the local LUGs were less anti-Red Hat.... How times have changed in the LUG-scape around here.
If it could be done over? The most detrimental statements made, in my opinion, were those statements that could be read to imply a commitment by Red Hat for CentOS 8 until 2029; this was after all the published roadmap by the CentOS Prject prior to December. In the press release you linked to, there is a now-dead link to a FAQ page; thanks to the Wayback Machine, I can re-read this FAQ page (
https://web.archive.org/web/20140906203025/http://community.redhat.com/cento... ) and see that maybe it was wishful thinking the way I read the very positively-worded spin that FAQ put on virtually all questions.
People being upset about the CentOS Stream 8 dates makes a ton of sense to me. But people thinking that we'd be shipping CentOS Linux 1,500 in the year 2984 doesn't. Taking someone's words about our intentions at some time or plans at some time, and then expanding them across an infinite timeline is just not how anything works.
And then there's the statement I quoted at the very beginning of this reply; there was wording that strongly implied a commitment that "nothing changes" specifically for CentOS 8. The September 24, 2019 statement, in my opinion, is what primarily set the stage for this backlash you see today.
Again, that was accurate at that time. If I could go back in time not do a CentOS Linux 8 release, I would have. But it wasn't in the cards.
-Mike
Best regards.
On 16 Dec 17:08, Mike McGrath wrote:
It makes a lot of sense that people would be upset about this. We very much should have set better expectations at the launch of CentOS 8 but at that point no specific dates around CentOS8 had been decided other than to release it.
Mike,
Somehow it ended up on https://wiki.centos.org/About/Product , so someone took the decision to write it here.
On Wed, Dec 16, 2020 at 5:12 PM Julien Pivotto roidelapluie@inuits.eu wrote:
On 16 Dec 17:08, Mike McGrath wrote:
It makes a lot of sense that people would be upset about this. We very much should have set better expectations at the launch of CentOS 8 but at that point no specific dates around CentOS8 had been decided other than
to
release it.
Mike,
Somehow it ended up on https://wiki.centos.org/About/Product , so someone took the decision to write it here.
Yes, I'm going to explain what happened as I've covered on IRC.
First, we didn't say anything about the EOL date because we didn't have an agreement at that time and that's the screwup. That absolutely could have been handled better.
What happened after that was a community member, not knowing any differently edited the wiki with a date that everyone thought was the right date. This was not an unreasonable thing for them to do. Any other date written after that probably got it from the wiki.
I think many of you think that some implicit guarantee was made, or are applying some standard to CentOS similar to what you would a contractual agreement and those will never be the same thing.
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort. Anyone mixing "free" and "enterprise" at work need to accept any risks that come along with that, I always did when I ran CentOS in production. Even with all of that, we gave a year's notice for 8 and let 7 continue in its natural life. We provided a viable (but not identical) alternative, and are working to find ways too get free RHEL to people. We're going to stand by that all of that, why? Because at the end of the day, any comparisons to us and Oracle, or "Embrace, Extend, Extinguish" are unfounded. We made a very unpopular decision here, I get that. But the difference is while we stand by that decision we actually do care about the impact it has had and are trying to make it right with many of you.
This is a good time to remind people of part of Chris Wright's announcement https://www.redhat.com/en/blog/centos-stream-building-innovative-future-enterprise-linux and centos-questions@redhat.com. This is a mailing list (not a sales lead generator). If you're using CentOS Linux today, and feel you cannot use stream. Email us and tell us why. The people who are creating new free and low-cost RHEL programs want to hear from you. We don't know who you are. And even if you are a Red Hat customer, previously you likely hid your CentOS deployments from us and so we don't know about them. And I repeat: this isn't going to our sales team, they don't have access to this list, this is about making sure we structure our future RHEL programs correctly.
-Mike
-- Julien Pivotto _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 16 Dec 17:51, Mike McGrath wrote:
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort.
That's the point. It does not seem a community decision. It happened in the board and every occasion I can think about, we were told CentOS Linux would stay as it.
What we are seeing in the unfortunate new rebuilds is that there is still a community willing to support this.
I am not saying that we won't try stream, but I would have like to have it at full steam speed before this announcement, and when it is fully working, then, having one year to switch.
On Wed, Dec 16, 2020 at 6:52 PM Mike McGrath mmcgrath@redhat.com wrote:
I think many of you think that some implicit guarantee was made, or are applying some standard to CentOS similar to what you would a contractual agreement and those will never be the same thing.
This isn't how Free / Open Source projects work. It is not normal for a community that exists precisely to provide a particular feature, is "acquired" by a company that claims to have the community interest at heart, and then leverages this power to replace the product with something that provides value to the company, and does not directly compete with company.
I have no doubt of your sincerity. However, I also believe that you may have been surrounded by other people with similar conflicts of interest and created a sort of "echo chamber" that after several months made it seem entirely reasonable to do.
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort. Anyone mixing "free" and "enterprise" at work need to accept any risks that come along with that, I always did when I ran CentOS in production. Even with all of that, we gave a year's notice for 8 and let 7 continue in its natural life. We provided a viable (but not identical) alternative, and are working to find ways too get free RHEL to people. We're going to stand by that all of that, why? Because at the end of the day, any comparisons to us and Oracle, or "Embrace, Extend, Extinguish" are unfounded. We made a very unpopular decision here, I get that. But the difference is while we stand by that decision we actually do care about the impact it has had and are trying to make it right with many of you.
Why is this a Red Hat decision to make? You say "community supported and best effort", but then you speak in terms of Red Hat's interests alone. Why can't it be a community decision, supported by the community? Is this because Red Hat took ownership of the branding, and stated that the branding could not be used for the original purpose any longer?
This is a good time to remind people of part of Chris Wright's announcement and centos-questions@redhat.com. This is a mailing list (not a sales lead generator). If you're using CentOS Linux today, and feel you cannot use stream. Email us and tell us why. The people who are creating new free and low-cost RHEL programs want to hear from you. We don't know who you are. And even if you are a Red Hat customer, previously you likely hid your CentOS deployments from us and so we don't know about them. And I repeat: this isn't going to our sales team, they don't have access to this list, this is about making sure we structure our future RHEL programs correctly.
I spent a great deal of effort trying to explain to Red Hat the problems with their subscription model in 2015 through 2018. It went nowhere. Red Hat made no meaningful effort to adjust their subscription model to be compatible with our requirements. The proposals made to compromise were non-proposals. I wanted to spend money on Red Hat, but against reasonable terms - and no reasonable terms were offered.
CentOS 8 Stream is not a replacement for CentOS 8, and everybody in this thread knows it. This means that a choice to use CentOS 8 Stream, is a choice to use something entirely new - something somewhere in between RHEL 8 and Fedora 33, and this is not a reasonable suggestion for many "Enterprise" use cases.
The idea that an "Enterprise" is "hiding CentOS" from Red Hat is part of the dangerous ethos that is on display here. Red Hat may substantially contribute to the community - but Red Hat is for the most part an assembly of free / open source projects that were given to Red Hat to use freely. Red Hat is reselling the works of others. Like thousands (millions?) of others, I contribute back to upstream projects. I also contribute fixes which Red Hat packages and builds into products that they sell to Enterprise customers. Red Hat isn't paying my salary. Where is my cut? I don't actually want a cut. I want Red Hat to be a hero of Free / Open Source, and show leadership in how to do things right. Killing CentOS 8 is an example of how to do things wrong.
The question is what happens next. There are thousands of people with budget, skill, and now motivation, to do something in response to this choice. Will these people embrace CentOS 8 Stream for development, and pay for RHEL 8 for stability? Or, will these people reject this conclusion, and either rebuild CentOS as it was originally intended, or simply move to another vendor that is not as predatory? Oracle Linux and Ubuntu both look like saints today.
On Wed, Dec 16, 2020 at 6:52 PM Mike McGrath mmcgrath@redhat.com wrote:
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort.
Name two people who are genuinely happy about it. Different motivations are understandable, but do you see anyone in the user community who advocated for it? I see a number of CentOS users and RHEL licensees who are alarmed and seriously considering dumping both in favor of Ubuntu for next year's platforms.
Nico Kadel-Garcia
One of the main reasons CentOS was so great is that anybody who wanted to try a startup company's software who used CentOS (for dev) could pull it down and run it *WITH GUARANTEES* that it would run with ASSURANCE . Because small/medium companies could *RELY* on a stable, vetted OS.
The up-sell for the client was RHEL (which was trackable from CentOS), if they wanted to run an OS with support *only* to run a certain specified product as an appliance, say.
For small/medium companies CentOS was a smart way to run dev/test/release products, tested on *validated* UPSTREAM OS'.
________________________________ From: CentOS-devel centos-devel-bounces@centos.org on behalf of Mike McGrath mmcgrath@redhat.com Sent: December 16, 2020 6:51 PM To: The CentOS developers mailing list.; Lamar Owen Subject: Re: [CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort. Anyone mixing "free" and "enterprise" at work need to accept any risks that come along with that, I always did when I ran CentOS in production.
...
This is a good time to remind people of part of Chris Wright's announcementhttps://www.redhat.com/en/blog/centos-stream-building-innovative-future-enterprise-linux and centos-questions@redhat.commailto:centos-questions@redhat.com. This is a mailing list (not a sales lead generator). If you're using CentOS Linux today, and feel you cannot use stream. Email us and tell us why. The people who are creating new free and low-cost RHEL programs want to hear from you. We don't know who you are. And even if you are a Red Hat customer, previously you likely hid your CentOS deployments from us and so we don't know about them. And I repeat: this isn't going to our sales team, they don't have access to this list, this is about making sure we structure our future RHEL programs correctly.
-Mike
On 12/17/20 5:02 AM, Dan Seguin wrote:
One of the main reasons CentOS was so great is that anybody who wanted to try a startup company's software who used CentOS (for dev) could pull it down and run it *WITH GUARANTEES* that it would run with ASSURANCE . Because small/medium companies could *RELY* on a stable, vetted OS.
Guarantees? Assurance? With CentOS? There was a stated goal of as close as possible to 100% binary compatibility, but there were never any guarantees with CentOS.
The CentOS developers have done and are doing their jobs very well indeed, and as time went on their project became more and more trusted due to the high quality. But there has never been a guarantee or assurance.
On 17.12.2020 06:51, Mike McGrath wrote:
On Wed, Dec 16, 2020 at 5:12 PM Julien Pivotto <roidelapluie@inuits.eu mailto:roidelapluie@inuits.eu> wrote:
On 16 Dec 17:08, Mike McGrath wrote: > > It makes a lot of sense that people would be upset about this. We very > much should have set better expectations at the launch of CentOS 8 but at > that point no specific dates around CentOS8 had been decided other than to > release it. Mike, Somehow it ended up on https://wiki.centos.org/About/Product <https://wiki.centos.org/About/Product> , so someone took the decision to write it here.
Yes, I'm going to explain what happened as I've covered on IRC.
First, we didn't say anything about the EOL date because we didn't have an agreement at that time and that's the screwup. That absolutely could have been handled better.
What happened after that was a community member, not knowing any differently edited the wiki with a date that everyone thought was the right date. This was not an unreasonable thing for them to do. Any other date written after that probably got it from the wiki.
I think many of you think that some implicit guarantee was made, or are applying some standard to CentOS similar to what you would a contractual agreement and those will never be the same thing.
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort. Anyone mixing "free" and "enterprise" at work need to accept any risks that come along with that, I always did when I ran CentOS in production. Even with all of that, we gave a year's notice for 8 and let 7 continue in its natural life. We provided a viable (but not identical) alternative, and are working to find ways too get free RHEL to people. We're going to stand by that all of that, why? Because at the end of the day, any comparisons to us and Oracle, or "Embrace, Extend, Extinguish" are unfounded. We made a very unpopular decision here, I get that. But the
difference is while we stand by that decision we actually do care about the impact it has had and are trying to make it right with many of you.
I suppose you (RH) have included, as the impact's component, the fact your (RH's) words and whatever promises you (RH) do are now of little value.
From my viewpoint, it's dead simple. CentOS was purchased for the sake of reaching a business goal. Now that goal is reached, CentOS Linux is buried (nothing personal, a business decision), and the uproar can be safely ignored - after all, it's mostly taken from people who aren't RH customers.
Some will be converted to RH customers (see also below), some will be free-to-use testers at CentOS Stream, all the rest are free to go wherever they wish.
This is a good time to remind people of part of Chris Wright's announcement
https://www.redhat.com/en/blog/centos-stream-building-innovative-future-enterprise-linux
and centos-questions@redhat.com mailto:centos-questions@redhat.com. This is a mailing list (not a sales lead generator). If you're using CentOS Linux today, and feel you cannot use stream. Email us and tell us why. The people who are creating new free and low-cost RHEL programs want to hear from you. We don't know who you are. And even if you are a Red Hat customer, previously you likely hid your CentOS deployments from us and so we don't know about them. And I repeat: this isn't going to our sales team, they don't have access to this list, this is about making sure we structure our future RHEL programs correctly.
"Low-cost" is the keyword, isn't it?
Corporation is all about the profit. Period. If you can't use the nice CentOS Stream (and thus help RH to serve as free tester and help RH to get more profits), RH will provide you with low-cost replacement.
It may even be free. Say, for a year. Just business decision, nothing personal.
(BTW, I mailed at that address, I am quite curious to see what can be proposed).
I think Red Hat did everything we could to stress that this rebuild was community supported and best effort. Anyone mixing "free" and "enterprise" at work need to accept any risks that come along with that, I always did when I ran CentOS in production. Even with all of that, we gave a year's notice for 8 and let 7 continue in its natural life. We provided a viable (but not identical) alternative, and are working to find ways too get free RHEL to people. We're going to stand by that all of that, why? Because at the end of the day, any comparisons to us and Oracle, or "Embrace, Extend, Extinguish" are unfounded. We made a very unpopular decision here, I get that. But the difference is while we stand by that decision we actually do care about the impact it has had and are trying to make it right with many of you.
Not unpopular, but impulsive.Timing is wrong - should have been done earlier or later , but not now.The RHEL programs should have been established before announcing such stuff. Stream could be great, could be worse - we will see. With those mistakes , many people will be disgusted by anything RH has touched which includes Stream,and will reduce the addoption rate by a magninute.Even I'm considering if I should invest more time learning RHEL products and switching to other interesting stuff. Best Regards,Strahil Nikolov
Dear Mike,
Am 17.12.20 um 00:08 schrieb Mike McGrath:
People being upset about the CentOS Stream 8 dates makes a ton of sense to me. But people thinking that we'd be shipping CentOS Linux 1,500 in the year 2984 doesn't. Taking someone's words about our intentions at some time or plans at some time, and then expanding them across an infinite timeline is just not how anything works.
I appreciate that RH kept supporting the CentOS distribution since the acquirement in late 2013 but
why you/RH doesn't recognize that the problem that the _community_ have is about the changed EOL of CentOS Linux EIGHT and not NINE or TEN etc. Why are you making a mockery of it? People call such behavior arrogant.
(even when you say that this date was never communicated -> defacto means that there is a consensus)
You stated elsewhere that RH do not want to put more resources into it. Okay, but RH are months to late with this decision (well C8 GA is over a year ago). If RH mess this up then it should take the consequences. In this case keeping providing the resources for CentOS Linux EIGHT until 2029. And to be honest; the major part of the effort (bootstrapping, build chain etc) is already paid. Not that much effort remains, right?
Also stated was that CentOS Linux is for RH not useful anymore because of UBI, Devel Programm etc.. Well, this is very biased - a point of view of an "President of Engineering"? Many people said that such decision (shortening EOL) can only be done by an marketing department. I would take this and put it upside down. A marketing department would never kill a community (an ecosystem for the NG of DevOps, $[NEXTBIGTHING}) for the cash cow of the organization, right?
Saying that - this is complete decoupled from the second announcement regarding CentoSO Stream. Please do not mix this up.
-- Leon
On 12/16/20 6:08 PM, Mike McGrath wrote:
... It makes a lot of sense that people would be upset about this. We very much should have set better expectations at the launch of CentOS 8 but at that point no specific dates around CentOS8 had been decided other than to release it.
The 2019 statement appears partly to have been intended to alleviate concerns about CentOS Linux 8 going away. It seems to have done that job too well.
The RHEL9 bootstrap process has already started with Fedora ELN ..
Ok, good to know, thanks for the pointer. To those who might not have found it, read https://docs.fedoraproject.org/en-US/eln/
... People being upset about the CentOS Stream 8 dates makes a ton of sense to me. But people thinking that we'd be shipping CentOS Linux 1,500 in the year 2984 doesn't. Taking someone's words about our intentions at some time or plans at some time, and then expanding them across an infinite timeline is just not how anything works.
In my particular case I had no expectations for CentOS > 8.
...The September 24, 2019 statement, in my opinion, is what primarily set the stage for this backlash you see today.
Again, that was accurate at that time. If I could go back in time not do a CentOS Linux 8 release, I would have. But it wasn't in the cards.
A real pity.
I just saw the official joint statement come down from Fermilab about this; they basically said that they would say something more concrete during Q1 2021.
And thanks for the links and reminders for the centos-questions@redhat.com email address. Time to write an email, I guess.