[CentOS-devel] Balancing the needs around the RHEL platform

On Mon, Dec 28, 2020 at 1:09 PM Laurențiu Păncescu
<lpancescu at centosproject.org> wrote:
> Red Hat acquired the CentOS brand and hired the CentOS core team.
> CentOS is their property, period.  They could have decided directly at
> management level what to do, nobody forced them to even create a
> governance board.  From Mike McGrath's description, it sounds they
> wanted to create community around CentOS similar to the one around
> Fedora, but things turned out different than expected and the
> association in people's minds, "CentOS is also from Red Hat", ended up
> cannibalizing their RHEL sales so they decided to put a stop to that.
> The Red Hat employees on the CentOS board didn't do what they wanted,
> but did what the company asked them to, I see no reason to focus on
> specific people.  But the EOL should have been announced before 8.0, so
> people don't waste time migrating from 6 to 8.

I've been trying to separate cause and effect here, as I think it is
somewhat important.

Everybody has their interests here. Every single person including
myself and every contributor to this discussion has their own
interests, and the people that made the decision in the first place.
We don't need to get into what the interests are - it's only important
to realize that they exist, and they are varied. This also makes it
difficult to pick a particular claim like "profit!" and pin it on
people, since many people who made the decision and who support the
decision, do not directly or may not even indirectly profit from the
decision.

I think the cause of this outcome is that the CentOS project was in a
fragile place in 2014 and prior. The insiders can describe exactly how
this was or wasn't so. Although the CentOS project could have expanded
with further community outreach, an offer was put on the table that
was difficult to refuse. Red Hat will sponsor the project, and
streamline it. Without qualifying good or bad, it was recognized at
the time that this was a conflict of interest situation but had great
optics and would resolve immediate problems quickly and effectively.
It was almost too good to be true. However, the trust in Red Hat as an
honourable champion of open source, put these legitimate concerns
aside.

Fast forward, and one by one - most, if not all of the voting members
of the board were slowly replaced or influenced by people with Red Hat
agendas. The emphasis on how to make things "better", and "contribute"
was probably one of many chords that needed to play out successfully
to convince people that this was in fact not only the right choice,
but the inevitable and only choice. Who was representing the large
community of "users" at this point? Which of these board members still
had the original itch to scratch that created CentOS in the first
place?

I think the future of the CentOS project was not only in a fragile
place in 2014 and prior, but it was also in a fragile place in 2014
and after. It was mentioned by a few people here how there were "two
months without security updates" as one of several reasons why CentOS
was already "less than" RHEL, but it wasn't really acknowledged as to
why - why can't CentOS be great? I don't believe Red Hat was incapable
of making it great. I think the only valid explanation is that Red Hat
didn't have an interest in making it great. Red Hat could have put
more people on it, or streamlined the build process and guaranteed
reproducibility by building RHEL and CentOS side-by-side if
de-branding was still a requirement. Red Hat could have shutdown
CentOS, and made RHEL available under the same terms as CentOS. But,
they saw no reason to do so. they were content to leave it with
insufficient resources, running behind RHEL by weeks to months, and
having no security updates for two months. There might be certain
heroes in the community who did all the work, and these people might
be highly respected - but is it valid for sponsors to set up an
organization that relies on heroic work to do something that is still
"less than" RHEL?

In 2014 and prior, the community could have fixed the problem. In 2014
and after, the community was controlled by Red Hat. This wasn't
absolutely clear in 2014, but it is now absolutely clear in 2020. Not
only is CentOS being moved from "just downstream" to "just upstream"
from the official RHEL releases, but CentOS is no longer a separate
project at any arm's length from Red Hat. It's now integrated into
their internal processes, and it is not really an "upstream" as most
people would assume, but it is still downstream of the Red Hat
controlled private branches, making it more like a preview and less
like a method for community contribution.

CentOS Stream is much more like an "insider preview" as another person
has been saying. "Insider preview" is valuable, but it's a very
different thing from what CentOS was, and why people chose CentOS over
Ubuntu. For example, I've been reflecting with some other users on how
they wanted to go with Ubuntu, but chose CentOS *only* because it had
10 year life span, instead of 5 years. By dropping the 10 years down
to 5 years, CentOS Stream is now much more similar to Ubuntu LTS, and
Ubuntu is far more popular in developer communities.

And this comes to consequence. Even if everybody involved was trying
to make the best decision possible in their own interests, there is a
very real chance that their own interests will be the ones that are
impacted the most. Red Hat took control of CentOS, presumedly to take
control of one of the most popular downstream builds of RHEL. But, now
CentOS will be re-created with one or more new names, and the 2014
acquisition of CentOS will have been effectively undone in every way
except the name of the project. Red Hat will have abandoned control of
the most popular downstream. Users will move to Ubuntu, Oracle Linux,
Rocky, or one of several others, and this will include their
non-financial contributions such as time and effort opening bug
reports and getting problems fixed, and it will include downstream
support contracts. Red Hat will have gained a reputation for failing
to live up to commitments, and putting all future commitments in
question. Other than Red Hat staff, and a few others who want a CentOS
Stream (and don't need a CentOS), the great community will remember
this. This isn't just individuals having uncomfortable conversations
with their bosses - this will be bosses questioning why they should
use Red Hat anywhere else. Even if CentOS Stream is successful in the
areas it is intended to be successful, the damage will be done. I know
my procurement team will be wary of any future Red Hat contracts - and
this extends to all the other products beyond RHEL. There is no reason
to believe an acceptable RHEL subscription will be made available that
would substitute for CentOS usage, and if we must use something else
for many of our machines, we may as well use something else for *all*
of our machines.

>From this perspective, I almost think this change was inevitable. Red
Hat was (intentionally?) not a good steward for CentOS. It was not run
like a community, with community elected board members. Red Hat put
just enough resources into CentOS to prevent anybody new from trying
to re-create it, but not enough to allow it to compete directly with
RHEL. It stayed like this essentially "on hold" for the last 6 years,
being "just good enough, but not excellent". Now, a change has been
made. CentOS as it was originally envisioned, is now deprecated and
soon to be eliminated forever. CentOS Stream, which is the "insider
preview" for RHEL is now to be released, as the new "just good enough,
but not excellent, only this time it no longer effectively competes
with RHEL, but instead helps harden RHEL."

> If they wanted to kill all clones, they could have stopped publishing
> the sources (they only have to provide them to their own customers, like
> they do for EUS updates, not to everybody on the Internet).  My
> impression is they want to kill just CentOS Linux, the only clone whose
> association with Red Hat seems to give it a legitimacy that other clones
> don't have.

Killing clones is not only difficult - but hopefully some people are
aware that it would also be a problem. We are now on the precipice of
"clones" now breaking from RHEL on definition of content. I foresee
that if the exact package lineup is too hard to align with RHEL, that
it will diverge. Oracle Linux 8.5 might not match RHEL 8.5, and this
may be more of a problem for RHEL than for anybody else. "Clones" is
more like "imitation is the most sincere form of  flattery" and less
like "others cannot possibly provide the same level of quality
backports". The former is just a matter of convenience and an attempt
to consolidate the entire EL community around common baselines so that
packages that work on one system will also work with a great deal of
confidence on another system. The latter is just a matter of scale.
When it comes to the Linux kernel, Oracle has basically already broken
from RHEL, in that they promote and install by default the Oracle UEK,
and only offer the "Red Hat Compatible Kernel" as an option if
necessary.

> I think the best way forward for everyone is to accept what happened and
> deal pragmatically with the consequences for their own organization.

Yes.

I think it's important for a lot of the concerns to be captured in
Internet history, so that we can learn from it. But, it seems we are
far past the point of changing anything on the Red Hat side of things.
As you and others have mentioned - the wording used suggests this is
all final. This is why I will not be sending anything to the new
mailing list. I already sent plenty of explanation to Red Hat in 2016
and 2017, and they took no action then.

-- 
Mark Mielke <mark.mielke at gmail.com>