[CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…

On 12/17/20 2:06 PM, Yedidyah Bar David wrote:
>>>
>>> Some other distribution will step in for CentOS Linux. Rocky, Lenix, Springsdale, whatever. That distribution/s will take the role of CentOS in paving the path for RHEL without RedHat having to paying for it.
>>>
>>> Sounds like a win/win-Situation, doesn't it?
>>>
>>
>> No, it does not. Because so far Red hat was viewed as champion of Open
>> Source and we "freeloaders" felt morally obligated to help Red Hat in
>> any way we could. It was the right and honest thing to do.
>>
>> Since Red Hat displayed greedy and stab-in-the-back attitude (buy hiding
>> what wanted to do before they were ready), there is absolutely no moral
> 
> I wasn't part of the discussion around CentOS - neither in 2014 nor now -
> and the news from last week was a shock to me as well. But if you now go
> back and read the announcement from 2014, you can very clearly see that
> from the very beginning, Red Hat didn't consider, or implied, or suggested,
> or anything like that, that it sees CentOS as a cheap/free RHEL replacement
> for the poor. It wasn't presented as _charity_. It was presented, and AFAICT
> _was_, for the benefit of Red Hat. Go read it. There is nothing new here.

It is very possible that back then it was genuine sentiment. I am
talking about last few months when Red Had decided to kill the clone.
Main point which many miss is that Red Hat *acquired* CentOS trademark
in 2014, and after that (and few years) it decided to not fund it any
more. If Red Hat is so community oriented and altruistic, why not only
announce that Red hat wil not FUND clone any more, and allow CentOS
community at large to try to provide funding? But Rad Hat did not do
that, right? Higher ups decided to kill the clone, forbid the clone
under CentOS trademark (RH controls) and morph CentOS users away from
clone supported for 10-years in hope to switch enough users making money
to RHEL subscription.

When RHEL 6.0 (2011) was released there was ~6 months gap before first
clones were released. I remember how much I was pissed (at Oracle) that
because of Oracle's attempt to steal market share from RH we CentOS
users had to suffer. In order to slow down Oracle, RH employed several
tactics like releasing kernel source without visible patches to mask
which exact changes were made and "using secret sauce" in how parts of
RHEL were built. I understood the logic and though it was ok, RH had to
protect it self, even if CentOS was lagging behind. Then I asked CentOS
devs if they can secretly get some help from RH, but they said no way.
CentOS started to make some progress, and suddenly how CentOS was made
became even more secretive, numerous people offering help were turned
down, and I thought it little strange, but I was using something CentOS
devs made so they had my full support. My private guess was that some
deal was made that CentOS project will not share how they rebuild RHEL
6, but I could not guess what could have got in return. Help to finish
rebuild so allied clone is still in the race to compete with Oracle
clone? Who knows, but I did have such thought back then.

So just like RH in 2011 delayed cometition (free clones) for just 6
months, RH is now delaying most of the competition for several months.
But no one expected such strong backlash. One of the CentOS Board
members was commenting on Twitter that he expected backlash but not the
petition on change.org.

In 2011 RH just delayed us using CentOS, and most of us forgiven it and
accepted it. But killing it after securing trademark and making
community impotent by forcing CebtOS Board to choose between jobs and
clone? That was some nasty s**t, like one of the parents going on
vacation with a child to his/hers homeland and then refusing to return
the child.

> 
>> obligation to help them in any way, and many now even have negative
>> feelings towards another "greedy company".
>>
>> Before this my message was "If you are going to spend the money on
>> Linux, it is best to spend it on RHEL, they give so much to community it
>> is only fair."
> 
> I also think/thought/talked like that in past jobs.
> 
> But I also want to add another reason: Spend it on Red Hat/RHEL, simply
> because they are the best, and worth your money. If you do not think so,
> don't.
> 
Your "problem" is that you run in different circles then majority of
CentOS users. I have been preaching virtues of CentOS/RHEL and that
"best" does not fly with majority of users of other distro's, you could
argue with some of them for months and never change their mind about
Debian/Ubuntu being better.
Also, how would you argue that RHEL is better the Oracle Linux when 99%
of it is same code? Where does "better" can be measured and that it is
not subjective?

That is why "giving back to community" ALWAYS had much better resonance
with large majority of Linux users, it triggers their morality, sence of
karmic justice.


>>
>> Since few days ago my message is "I do not like them anymore, and I do
>> not have trust in them, so better stay clear from them."
> 
> I definitely feel your pain. I felt the same way for several days now,
> and slowly got used to the new situation, and am now mostly ok with it.
> 
> I am not saying it was nice. I am just saying, that right now, I do
> agree with upper management here, if they say we simply had no choice
> but break this "promise" (of support till 2029), as bad as the community
> would accept this.

Again, altruistic choice of allowing community to try to fund it was
always the option, even now, but that would not serve ulterior motives
of RH (move users to RHEL licenses).

> 
>>
>> CentOS project leaders had the same philosophy in mind when they refused
>> to add extra packages to CentOS repositories like non-free codecs, 3rd
>> party drivers (ElRepo had to be created separately) or even some desktop
>> apps or KDE, MATE, etc.
> 
> There were at least two other reasons, AFAICT:
> 
> 1. People do/did not want that. They wanted exactly what CentOS said it
> is trying to do - bug-for-bug compatibility with RHEL - so that's what
> CentOS did.

There were several attempts in different periods in time where expansion
of activities was attempted. It was always shut down by the Board. I was
one that for C5 and C6 wanted to organize something like "Desktop SIG",
but could not get approval. I created DentOS repository (70 rebuilded
package for C5 including Skype) by my self but since it was unknown 3rd
party repository it never gained any traction. I was one of the largest
advocates of making easier the Desktop/Workstaion use of CentOS becuase
I thought it would be much appealing for Linux users if you could more
easily install desktop/laptop PC. Ubuntu's meteoric rise can be directly
attributed to Fedora not shipping audio/video codecs for mp3, divix, etc
while Ubuntu shiped it. And Ubuntu users tells newbie "all I have to do
is turn on ppa and install codec, no 3rd party repositories needed, no
clasing of repositores." And newbie would choose Ubuntu, not Fedora of
CentOS...

> 
> 2. Doing CentOS as-is was already hard enough. I do not think I have to
> remind people the situation it was in, before the Red Hat "acquisition".
> 
I do not remember and donation campaigns before acquisition....


>> All of that was redirected to Red Hat controlled
>> EPEL or 3rd party repositories.
>>
>> But Rocky Linux and Lenix (CloudLinux) do not have to be constrained
>> with these compliance, why should they when most likely Red Hat will do
>> their best to complicate creation of other clones any way they can.
> 
> If you ask me, CentOS Stream is a giant step forward, for anyone that
> wants to rebuild RHEL. I am not sure why consider it "the best Red Hat
> can do to complicate" this.
> 
>> You
>> can say what ever you want, but I and others do not trust them/you to be
>> better then their worst deed.
> 
> Fair enough.
> 
>>
>> And there is no legal obligation to use RHEL and not clones in
>> production, especially if CloudLinux develops a business model that will
>> enhance FOSS clone and eventually spin off from RHEL into competitor
>> just like Oracle did. Even Rocky Linux could be backed by some new
>> company that will offer paid-for support in production.
> 
> Of course!
> 
> As other, more senior Red Hatters already said on this thread: Game on.
> 
> We do not look for mercy. We think we are good. If you prefer doing
> business with Oracle, or CloudLinux, or a (new) company behind Rocky
> Linux - go ahead.
> 
> If you ask me, a _business_, making money, that decides to base their
> supply chain on the promise of a community project, instead of a
> contract with a company, is taking a significant risk. Nobody prevents
> this, but I'd personally not do that.

Outside "Western world" where RHEL support has significant presence, and
"English speaking" parts of the world, how many countries have support
in native language? RH has offices in only 17 European countries  , out
of 50. Around the world, in how many of 195 countries in the world does
Red Hat provide support?
Do you provide support in Hungarian, Serbo-Croatian, Slovak, etc, etc.?

Most of small businesses with several employees do not have their own IT
admin, they hire contractors to deploy and manage their Linux servers.
Now, what do you think owner of the small business is going to rather
pay, RHEL license + contractor cost or only contractor cost? Or which
contractor will win the bid, first charging ONLY contractor fee or the
second one that adds $300/year for RHEL license on top? First one of
course. So when I managed to convince owners they need dedicated storage
server with RAID (you would be shocked how many small businesses STILL
do not have file servers or even monthly backups of date form their
PC's!), there was no way that I could sell them RHEL license. If they
pay for something they would pay for Windows, they know how to use
those. So all I could offer them was FREE Linux, on regular PC with 2
HDD's in mdraid without monitor was all I needed. No ECC memory,
server-class MB's, U rack cases, etc.
If there was no free CentOS, I would have gone with Ubuntu (or maybe
Debian) and just as now I am the one who is responsible to manage them,
and I know enough to not need emergency support from any vendor. Since
they sit inside private networks, no internet facing ports, and barely
have user security (Samba with all folders writable by all), I update
them only from time to time, every few months, otherwise I do not touch
them in not really necessary, to not jinx them while they work without
issues.
I only deployed 4-5 such servers, several other companies I manage have
small NAS-es or keep data on Windows PC's :-(
But even if I expand my business and start offering support to much more
small business, none of them will want to pay yearly licenses. So I will
either install free clone, TrueNAS/FreeNAS (FreeBSD based with ZFS!) or
Debian-based. What I settle on will be for all such deployments, and my
only contribution in direction of paid-for licenses would be my online
recommendations, based on who I feel deserve the praise.


> 
>>
>> Up until this backstabbing act any company that would try to steal
>> support income from Red Hat would have been declared greedy by CentOS
>> and even Linux community at large. Even today I do not like Oracle
>> because they became direct competitor to Red Hat who was spending money
>> on development, bugfixes, etc.
>>
>> But since Red Hat is now in same category as Oracle, greedy corporation,
>> EL/Linux community will WELCOME another player in paid-support for RHEL
>> clones, and stand by them as long as their actions support needs of "us
>> freeloaders". Do you really think CloudLinux decided to spend $1 million
>> because they are altruists? I do not. They have seen Red Hat hang them
>> selves (nobody provoked them) and saw unique one-in-a-lifetime
>> opportunity to expand their portfolio from only light hosting clone
>> based on RHEL source to all-purpose distro that will help them expand
>> their paid-for support offer to baremetal servers and workstations,
>> maybe even laptops. All they have to do is to publish binary clone and
>> then expand on that ecosystem by adding repos like ElRepo, EPEL,
>> CentOSPlus, and maybe non-free repo and they will be huge success and
>> make bundle of money, well worth the investment they are making.
>>
>> And you know what? I am going to support them, and bee happy for them.
>> And direct any money spending THEIR WAY.
> 
> Very well. I am not saying you should not.
> 
> I do not remember where I read it, but I read somewhere an estimation
> that continuing full support of CentOS 8 until 2029 would have cost
> Red Hat something like $30-$40 million. I have no reason to think this
> is way off. So $1 million suddenly does not look that much.

Again, with green light from Red Hat (never gonna happen) community
could try to finance cloning effort and new blood could be introduced to
actually work on it (also NEVER gonna happen).

> 
> I'd like to use this opportunity to address some other issues raised
> recently (in this thread, perhaps also elsewhere). I'll not quote
> the text I am replying to, I hope that's ok.
> 
> What is Free Software, and what is Open Source? A lot was written about
> this, and I am not going to repeat everything. For current discussion,
> the main point I'd like to make is: Open Source is a business model.
> It's not (mainly) about giving the famous 4 freedoms to users of your
> software, or even about Linus's law "With enough eyeballs, all bugs are
> shallow". Red Hat got/gets, for free, the code of many FOSS projects.
> In return, Red Hat gives back _all_ the code it added/adds to these,
> and other, projects. If you measure the amount of "Being open source"
> for a particular company, by dividing to value of what it gave, with
> the value of what it got, or, for that matter, with its revenue/income/
> whatever, I am fully certain you'll not find any other company in the
> world, in a size similar to Red Hat's, with even close-by ratio. Think
> about this. Go ahead and think about the companies you know. Sure, many
> companies "give back", but how much? Even those that give a lot, in
> money, or money-worth, have, AFAIK, way lower ratios. And, BTW, why
> does Red Hat do that? Because the people here, like me, make it do so.
> Both management, and owners (now IBM, other investors before that),
> realize that if a significant change is done to the "wrong" side, too
> many people here will quit, and put the company in such a bad position,
> that it will simply not be worth it. memo-list was on fire, for the
> last week. Not fire - more like a nuclear explosion. Believe me. But
> people slowly understand, and things slowly calm down. I currently
> simply do not believe that Red Hat can become evil, in this sense.
> It will simply be dissolved.

Up to only 9 days ago I would agree. But announcing CentOS project is
FORBIDDEN to maintain RHEL clone (RH owns trademark and controls the
CentOS Board and does not want anyone else to finance cloning) it all
invalidates any and all past deeds.

There is excellent speech in TV Show "Newsroom". Watch it to the very
end, it expresses my exact feeling for Red Hat at this moment. "America
is not the greatest country in the world ANYMORE". (There is "it used to
be..." continuation that is important to watch)
https://www.youtube.com/watch?v=bIpKfw17-yY
Same sentiment is in the Linux community since Dec 8th, RHEL is not the
greatest supporter of FOSS ANYMORE!

> 
> To add to that: IMO, any person/company/whatever that says "Red Hat
> pissed me off, I do not want ever to have anything to do with them",
> and want to be honest to themselves, must now go and start helping
> Debian. Doing, or using, a RHEL rebuild, is not that, and does not
> make sense. To me. You want competition? The only real competition
> for Red Hat, IMO, is Debian. Go ahead, help them. Or, if you prefer,
> SuSE. But a RHEL rebuild still keeps you tightly-related to Red Hat,
> strongly dependent on Red Hat, as much as you claim you hate it.

Do you really think I am fighting here to get back CentOS clone because
I hate RHEL, how things are done? I chose rpm as a package of choice in
2005-2006, and chose CentOS as distro ini 2008. It makes most sence to
me, I like how stable/boring it is, I like 10 year support, I like point
releases which gives you option to update only up to certain date if you
have issues with next minor release, I like stable kABI n between, I
like weak-updates for drivers (kmod). The rest I got used to and
accepted, and if I can choose, I would choose that CentOS community is
allowed to fund clone. There is still little time for RH to say
community at large does not understand them, and they did not think of
it at first, and because of that they will hand over clone rebuild to
community, to keep peace, show good will, or what ever.

That would also prove to you Red Hat employees who do not like what
happened that Red Hat is still company you are portraying it to be.

But that is not going to happen, and Red Hat employees will have to
swallow this bitter pill, and accept the new reality.



And actually, the sentiment is ONLY about CentOS 8 systems already
deployed. Only one (C8) I have is my 4-year old laptop, but by moving it
to Springdale I will not have to reinstall anything else on it while it
works. I doubt it will live next 9 years until C8 EOL.

I have 1 C7 that is safe next 4 years, and 2 C6. One is private
Samba+KVM and can function for next several years without reinstalling
it since it is not accessible except to 5-6 employees. Other is my main
web/mail/storage server that I do not know what to do. My small WISP is
practically dead and I should remove server from its uplink (10/10Mbps
free of charge), so moving mails and web sites (one is Joomla 1.5, never
cared enough to update it since word of mouth is enough for my side job)
to hosting service or better hosted VM is best scenario for it, and then
I can convert it to TrueNAS of some Linux with ZFS support. That
decision will wait, C6 Samba/NFS not on the internet can wait for me to
choose.



> 
> I am now going to take a big risk and talk about Ubuntu. It might
> be the stupidest thing I did, ever. Ubuntu/Canonical are not doing
> well, financially. The only reason they keep being so well-known,
> common, etc., is because they have funding, which is not based on
> their income. If rich people, that made their money (also) thanks
> to FOSS, want to contribute back, by financing it, such as by doing
> Ubuntu, that's adorable, really. But this isn't Red Hat. Red Hat is
> a group of people that is trying to make a leaving from Open Source
> itself. You might claim this is simply not interesting, or irrelevant,
> or stupid, or whatever. You might claim that Red Hat, as-is, simply
> has no place in the FOSS world. But many people do not agree, including
> both the people inside Red Hat, and also many of its customers, and
> so it exists. I'd personally find it very sad if it turns out that
> this is wrong - that  there simply is no way to make a business,
> make a leaving, from doing FOSS - that I, personally, must either
> give up on my principles and work for a non-FOSS company, or give
> up on having FOSS as my day job (and do it only as a spare-time
> hobby). I still have hopes.
> 
> Re the help that people like you provided to Red Hat, for free, by
> spending effort on CentOS, then push for using/buying RHEL: Can you
> please think for a minute, and explain why CentOS Stream is not almost
> the same? For people like you (and me!), that want to play with Linux,
> do interesting stuff with/on it, learn it, etc., but not (yet) make
> money from it, build a business around it, does the difference between
> CentOS Linux and CentOS Stream really matter? Why? You do not have to
> answer right now. You can try it, then decide. I completely ignored
> it until the announcement last week, and now, since the oVirt project
> realized that migration to Stream is the most reasonable choice,
> started moving, and already yesterday ran into a bug and fixed it
> this morning, https://bugzilla.redhat.com/1908602. Is that really
> so bad? You might say: But I do not have money, and want/need to run
> real stuff in production, and need the stability, security fixes,
> etc. Fine. Was CentOS so good for you so far? CentOS was also weeks
> or months behind RHEL for all of its lifetime, sometimes (before
> the Red Hat acquisition) many months. Why do you think Stream will
> be that much worse?
> 
> Re timing etc.: I was really shocked last week, like everyone else.
> But now I think everything basically makes sense, to me. When I heard
> about providing only 1 year, I was shocked. 1 year? I was a sysadmin,
> and I know that doing the upgrades, at the time, e.g. from RHL 7 to
> RHEL 3, and later from RHEL 3 to 6 (yes, we did skip there 4-5), took
> me and my team more than a year, closer to two years. But: If any of
> the existing or emerging RHEL rebuilds is already good for you, or
> will be good enough in a year, migration to it from CentOS should not
> take a year. Not more than a few weeks, IMO (including testing,
> automation, whatever, etc.). If it's not production, you are most
> welcome to migrate to Stream, and depending on your needs, it might
> make much _more_ sense to you, than to consider a RHEL rebuild.
> And this also should not take that long. And if you want RHEL, and
> want to use it for development, see the existing options, and if
> you think they are not enough, or do not suit well your particular
> situation, others on this thread also discussed this - contact
> Red Hat. And if you are simply running production stuff, and your
> business relies on RHEL, if you ask me - RHEL is simply what you
> want. And if you think otherwise - just do your own calculations
> and buy (or not) what you think is best for you. This indeed might
> require a longer time, but would still make sense to do in a year.
> 
> Re the CentOS _brand_: People keep saying: OK, you realized that
> taking over CentOS was a mistake. Fine. Just give it back to the
> community. Now, let's have an exercise. Red Hat claims that CentOS
> costs it money. A lot of money. And that if CentOS Linux would
> continue as-is, even without Red Hat's financing, it would still
> cost it quite a lot of money. See the linked article at start of
> this thread, and other stuff here, for details. But it's not
> important if you accept this fact as presented or not. Just accept
> that this is Red Hat's stand - that CentOS Linux will continue
> costing it money, as-is. Now, suppose that Red Hat, expecting the
> huge noise/backlash such an announcement would cause, would simply
> quitely try to find contributions somewhere and keep the project
> as-is. Would this help covering these costs? Would a thread like
> current exist? Would people seriously start thinking about what
> it means to have something like CentOS Linux? How much it costs?
> etc.? I do not think so. So Red Hat decided it's best for
> everyone to simply publicly kill it. To make it extremely clear
> that Red Hat is not involved anymore in any RHEL rebuilds. That
> from Red Hat's POV, if you want CentOS Stream, as presented, you
> are most welcome to try it, that if you want RHEL, you have
> several options for getting it, but that if you want a RHEL
> rebuild, then CentOS isn't one, and Red Hat is not going to be
> involved in one. That if anyone wants to make one, fine - but
> Red Hat is not part of that. Now, let's be honest: People on
> this thread keep saying two opposite, contradicting things: One,
> that CentOS, as a name, is worth almost nothing - say, not much
> more than the cost of keeping the domain name centos.org, and
> the other, that CentOS is a huge thing, and that killing it
> as-is is a huge crime. Please be honest. If you think the former,
> what's the problem? Just create a new cool name, and start
> collecting people to join you. If you think the latter, please
> realize that this costs money - a lot - and that Red Hat decided
> that it does not want to spend this money anymore. I personally
> think that's fair.
> 
> Last but not least: About the implied pressure of Red Hat on
> the CentOS board, that led to current state. I wasn't part of
> this, or related in any way, and basically all I know about it
> is from this list. So what I write here is purely a guess. Isn't
> it possible, that the board, even including non-Red-Hatters, was
> simply convinced that Red Hat's POV is legitimate? That just
> keeping CentOS as-is for 9 years would make such a damage to
> Red Hat, that CentOS itself would simply not exist anymore? That
> for the interests of CentOS, its users, it's simply unavoidable
> to do something like what was eventually done? People might
> disagree about the details (e.g. 1 year, or less, or more), but
> does it sound so crazy that they simply agreed?

Johnny Huges wrote in one of the lists main or devel that Board was
against, but new Red hat can veto any Board decission. And Johnny was
personally first against it and negotiations were held. It is pretty
detailed.

Here are excerpts from 2 mails he wrote:

"The CentOS Project board has a Red Hat Liaison.  That position is
documented here:

https://www.centos.org/about/governance/board-responsibilities/#red-hat-liaison-responsibilities

Also see role of Liaison here (and look at B:):

https://www.centos.org/about/governance/voting/

The bottom line is .. a decision of the CentOS Board has been made and
we don't have to like it.  We do have to do it, regardless of if we like
it."

and

"That is correct .. so, the Red Hat Liaison can use Section B. of the
Governance to dictate a vote. If the board FORCES the use of this
clause, then whatever was wanted (in this case by Red Hat) would get
inacted in its entirety with no real input from the board.

https://www.centos.org/about/governance/voting/

The CentOS Board knows this, so we had a dialoge with Red Hat instead.
Red Hat presented their case and listened to our response.  There was a
significant back and forth.

So, no one 'FORCED' the board to do anything.  Red Hat told us what they
were going to do (what you quoted).  The board then made many
recommendations in a back and forth negotiation.  The board then made a
decision.  The decision was reluctant .. but it was unanimous.

And now this is the way forward."


> 
> If you ask me, there simply is no more place for a project like
> CentOS in this world. It does not make sense. I do not believe
> any of the "community competitors" will survive. I think there
> is enough space only for a single large-scale community distro,
> and this is Debian. All the others will simply remain small, or
> disappear, or become companies. And these companies will realize
> the hard work of having a business. But that's just a guess.
> Time will tell.
> 
> If you read so far, thanks for your time. Also, thanks again,
> really, for your hard work along all these years. Do not think
> that Red Hatters do not appreciate that.
> 
> Best regards,
> 
I do not believe that any distro will be "last mans standing". There is
enough people that have different desires which are not Debian, but that
also depends on other companies in Linux world. If Red hat doubles down
on RHEL clone basing and making cloning more difficult, then yeah,
majority of us will move to Debian and Red hat will not have system
admins who would even think of RHEL way of doing stuff.



-- 
Ljubomir Ljubojevic
(Love is in the Air)
PL Computers
Serbia, Europe

StarOS, Mikrotik and CentOS/RHEL/Linux consultant