[CentOS] CentOS 8 future ("Long goodbye"?)

Wed Dec 16 16:45:39 UTC 2020
Valeri Galtsev <galtsev at kicp.uchicago.edu>

My apologies about top posting.

I join Matthew on all counts.

The following might sound as a rant, but it is not, given the circumstances we have been put into.

First, and most important: thank you CentOS team for all great work you have done during all these years. As user who used results of your work without giving much back (not counting maintaining public mirror, or helping others on the list whenever I felt my expertise adequate), I can not express how high I value what you gave to all of us.

Now, that CentOS as we knew it (as a “binary replica” of RedHat Enterprise) ceases to exist many of us are trying to figure out new long term solution for their “enterprise” sort of systems. Luckily I only partly have to do that, as for servers I already did migration quite long ago. My mentioning it on this list was causing more annoyance than I would like to, so I stopped mentioning it. But now it is time to mention it again, just to help everyone arrive at best decision. But first some thoughts on migration to different Linux Distro:

One of obvious possibilities is to migrate to some other “binary clone” of RHEL. One can find several, Oracle Linux (even though many are cautious of Oracle, they - Oracle - didn’t drown out of existence mysql so far, maybe thanks to mariadb fork existence, …), Scientific Linux (which is effort of really small team, and I evaluated it well below CentOS when I had to make decision, and it confirmed true over time), and others... However, once RedHat (or rather its owner IBM) made fundamental decision, it is not as much about the one who clones (binary rebuilds) of RHEL, as it is about RHEL itself. At least fo me it is. As, by undermining trust, even if they roll everything back to what it was, the trust is already lost by the knowledge of everyone that any moment they can do that in a future. This alternative is just out of question for me. Will I maintain RHEL for my current or potential future employer? Yes, definitely. Will I recommend fair (and way cheaper, better, longer lasting) alternative? By all means, yes, and with my experience of migration, and documented migration steps, etc...

Another possibility for pure Linux folks is switch to different distro. Not with 10 years life cycle (here RedHat was unique), but shorter one, yet with much easier upgrade from one release to another. [Even knowing about Ubuntu LTS] Debian would be my choice, which I am going to pursue for CentOS number crunchers and workstations I maintain. Laptops are Debian clone Ubuntu since long ago. This will be “rolling release", i.e. mostly you will have to upgrade packages to latest release, and constantly will take chance something will break with change of internals of given software from one release to another. It will be more work (for 24/7/365 servers most gravely notable). But it may outweigh the single event when your “enterprise” life is cancelled one day, and you have to redo the whole infrastructure all at once. Think about it and about peace of mind avoiding that eventuality.

This leads me at last to telling that my sever infrastructure was migrated long ago to FreeBSD. One can chose different BSD successor based on one’s own assessment of suitability. First of all, pure Linux folk, it is not that challenging as one may think. I would say here the same thing I was telling to my users who we just starting to use UNIX (or Linux). How many command do you need to know to start using UNIX? Just 5-6 is enough. Start doing things, and in a couple of Months you will feel you know everything. In 6 Months you will be top expert: the one who knows what he knows and knows what he doesn’t know. My choice was based on the following facts: FreeBSD is most widely used (even Microsoft was once noticed to run some of their servers on FreeBSD). FreeBSD has excellent documentation. FreeBSD community is as eager to help the one who got stuck with something as our CentOS community is. They have as excellent experts as Johnny, Matthew, ... sorry I can not mention everyone, that will take separate huge post...

And now, with my servers gone to FreeBSD long ago, I can share this nice experience. On FreeBSD (base system is separate, and Linux, BTW, decided to go same excellent way), and extra stuff can be added from huge port collection, most part of which is available as binary packages. Ports/packages are up to their maintainers, and pretty much all of the ones I use are available as different versions, still maintained and patched, so you not necessarily have to upgrade to latest version when it is released. In this respect, individual ports or packages can live as “enterprise” portions of your ecosystem themselves (each with its own life cycle, still…) This actually is not as challenging as it may sound, as long before end of life of some package version (like PHP-5), at every update you will get warning that it will be end of life soon (starts multiple Months before the day it is going to happen…).

Anyway, what I am leading to is: life with FreeBSD is not as challenging as it may seem before you try. Just try it. And it is more “enterprisish” than with Linux “rolling release” in my opinion. Though I must confess, I have much less “rolling” Linux release server experience, than I have FreeBSD experience.

I did not even mention FreeBSD jails which essentially are containers with the same base system mounted read-only (or you can several base system versions, e.g. 12.x and 11.x for some or another jails). Anyway, jails can be long separate post…


The length of my post suggests that it is a “log goodbye” which it probably will be.


Thanks again to CentOS, the hard working excellent team. Whatever falls on you from us has nothing to do with you, but rather with RHEL/IBM, and the trust they decided to throw out of window.


Valeri

> On Dec 15, 2020, at 7:04 PM, Phelps, Matthew <mphelps at cfa.harvard.edu> wrote:
> 
> On Tue, Dec 15, 2020 at 7:41 PM Johnny Hughes <johnny at centos.org> wrote:
> 
>> On 12/15/20 6:12 PM, Joshua Kramer wrote:
>>>> I don't think there will be a course change either, but for different
>>>> reasons. The motivation isn't "cashing/selling out". It's... actually
>> the
>>>> stated motivation
>>>> https://www.redhat.com/en/blog/faq-centos-stream-updates#Q2
>>> 
>>> First, I will note that I think the idea of creating *a version of*
>>> CentOS that is called "Stream", with the intent that it leads RHEL by
>>> a bit, is a GREAT idea, for exactly the stated reasons!
>>> 
>>> There's one problem I have with this asserted motivation.  Stream is
>>> not being done as *a version of* CentOS.  It is being done as *THE*
>>> CentOS, which means you're discontinuing point releases.  As far as
>>> "maintaining CentOS point releases to follow RHEL"- this is what is
>>> being discontinued.  How much money, in developer time and other
>>> incidentals, does this cost RedHat per year?  Of course this is a
>>> proprietary number.  But let's imagine that this number is $250k per
>>> year.  Out of what was it, about $433M of profit (2019)?  So it would
>>> cost RedHat 0.06% of profit to hire more developers to keep issuing
>>> CentOS point releases.
>>> 
>>> What does RedHat "buy" in return for spending 0.06% of its profit on
>>> maintaining point releases?
>>> -Community trust and goodwill.  Those members of the community that
>>> cannot afford RedHat licenses for whatever reason still know that the
>>> #1 player in the Linux marketplace still has their back.  Then when
>>> those folks move on to enterprises that can afford RH licensing (and
>>> in some cases demand it), will select RedHat because of this trust and
>>> goodwill.  They will be highly likely to recommend other RedHat
>>> products- since it all "works together" and they'll know RHEL (i.e.
>>> CentOS) well.  Also note that this trust and goodwill means
>>> "convenience", even within enterprises that have a large budget with
>>> RedHat.  If I have a project and I want to spin up 100 OS instances
>>> just for the heck of it, I can.  I don't need to ask anyone, I don't
>>> need to reserve or download any entitlement key files.  I don't need
>>> to debug weird problems when entitlement key files don't work.
>>> -Control of part of the ecosystem.  Those companies that build their
>>> products to run on RHEL (or in RHEL containers) are able to (and
>>> encouraged to) certify those products on RHEL because they are able to
>>> use CentOS.
>>> 
>>> But more to the point, what does RedHat LOSE by saving 0.06% of its
>>> profit?  The damage to community trust and goodwill far exceeds the
>>> gains that would be realized if the status quo were kept in place.
>>> Yes, it's true that many of the folks who used CentOS would never turn
>>> into paying customers.  But due to this situation, you have thousands
>>> of system administrators who are actively looking to completely
>>> abandon the RedHat ecosystem altogether.  When it comes time to
>>> recommend products... they aren't going to recommend RHEL.  They
>>> aren't going to recommend JBoss, or Fuse, or 3Scale API management.
>>> It's clear that RedHat can't be trusted with some parts of its
>>> portfolio, so why should we trust ANY of its products?
>> 
>> So don't trust them.  Move to something else if you think something is
>> better.
>> 
>>> 
>>> If it is 100% factually correct that the ONLY motivation for killing
>>> point releases is the stated motivation, then it's just a simple
>>> matter of finding a spare $250k (or whatever that cost is) from the
>>> almost-half-a-billion dollar corporate coin purse.  The return on
>>> investment has been, and will continue to be, immeasurable...
>> $250K is not even close.  That is one employee, when you also take into
>> account unemployment insurance, HR, medical insurance etc.  now multiply
>> that by 8.  Now, outfit those 8 employees to work from home .. all over
>> the world, different countries, different laws.
>> 
>> .. THEN buy 30 machines minimum (servers, not workstations) for
>> building and testing, buy a service contract for those 30 machines, host
>> the bandwidth required to sync out to 600 worldwide servers.
>> 
>> We need all the CI machines .. that is a bunch of blade servers for
>> that.  They need service contacts too.
>> 
>> In any event it doesn't matter.  The decision is made. If people don't
>> want to use CentOS Stream, then don't.  The decision is not changing.
>> _______________________________________________
>> 
> 
> We won't.
> 
> Thanks for all your work in the past. Good luck to you.
> 
> And to Red Hat I have one more thing to say:
> 
> Buh bye!
> 
> 
> ###
> 
> 
> -- 
> 
> *Matt Phelps*
> 
> *Information Technology Specialist, Systems Administrator*
> 
> (Computation Facility, Smithsonian Astrophysical Observatory)
> 
> Center for Astrophysics | Harvard & Smithsonian
> 
> 
> 60 Garden Street | MS 39 | Cambridge, MA 02138
> email: mphelps at cfa.harvard.edu
> 
> 
> cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
> <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
> | Newsletter <http://cfa.harvard.edu/newsletter>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos