[CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…

Fri Dec 18 02:37:56 UTC 2020
Mark Mielke <mark.mielke at gmail.com>

On Thu, Dec 17, 2020 at 8:06 AM Yedidyah Bar David <didi at redhat.com> wrote:
> But I also want to add another reason: Spend it on Red Hat/RHEL, simply
> because they are the best, and worth your money. If you do not think so,
> don't.

I believed in this point strongly in 2015. Let me share some details
about what forced this to change.

We were an RHEL-only shop, with a ~$0.5M USD / year annual renewal. We
were preparing to deploy a private cloud, with expectations of
quadrupling our Linux instances in the company. According to the
contract, this would have been around $2M USD / year annual renewal. I
did some review of what Red Hat provided us, and I came to a few
problematic conclusions. One conclusion was that for the past few
years, we were averaging a support cost of $20K USD per ticket, and
these were for the most part very basic tickets that should never have
been opened in the first place. For more complex tickets, including
one that I opened to test and evaluate the system - the ticket
actually went nowhere. The support person was unwiling to push the
developers to fix a problem. I had to escalate and push hard, and then
participate in the bugzilla.redhat.com discussions before finally
getting the attention of the backend Red Hat developer who
accidentally broke it, who quickly agreed to fix it.

I don't really want to get into all the details of what right, and
what went wrong. I want you to think about this number for a second.
$20K USD per support ticket, including basic support problems. Which
of you can easily justify this to your directors? Would you agree to
pay this out of your own pocket, if you were an owner of the company?

I did a lot of analysis, and had many discussions with Red Hat staff,
who by the way - were all great. I believe emails I sent were
circulated around Red Hat in the 2016 and 2017 timeframe, although I
don't know if this was just the sales and technical people that worked
with us, or if they weren't further. There was a great deal of
interest in the problem I was describing, but no ability for anybody
to do anything about it. The subscription model was fixed.

The Red Hat subscription model is particularly problematic in that
*all* systems must be subscribed, whether they require support or not.
The question at renewal time, and the wording of the contract is not
"which systems require support?" The question is "how many Red Hat
systems do you have?" This means that test systems, development
system, customer test systems, are all counted as individual
subscriptions of some type.

This also means that when we want to horizontally scale a service,
such as by having it be split across 3 virtual machines instead of 1,
we need 3 licenses. There are some allowances in there such as how the
Standard Subscription permits 2 VM per subscription, but you
eventually conclude that you must subscribe the entire hypervisor.
Except - we use many different operating systems on the hypervisors,
and we are now talking about licensing every hypervisor for Red Hat,
even if the system might not be running a Red Hat host or guest.
Effectively, we end up being charged for running Ubuntu or Windows
unless we design our system to limit the migration of virtual machines
so that we have a "RHEL-only cluster" and a "non-RHEL cluster".

The Red Hat sales person had no levers to get around this. Red Hat is
fundamentally sold per subscription. Even at bulk discount, the
per-subscription discount was no more than 20% per system.

I wanted to pay Red Hat for the services we received. I could justify
this. However, we were looking at a $2M USD RHEL expense for the next
year, and $4M USD for the year after that. I could not justify this,
when so many alternatives existed that provide what was substantially
the same content and service.

Red Hat priced themselves out of the market. I didn't do this. Red Hat
did this. This is why Oracle or Facebook cannot use Red Hat. Oracle
Linux might not exist if Red Hat had offered an arrangement with
Oracle. It is fine to live by a code that says "my product is worth
$X", but then it better be worth $X, or people won't buy it.

Do I think Red Hat has done great things for the community? Yes, with
the caveat that a great number of companies are doing great things for
the community, and Red Hat is not alone.

Can I justify $20K USD per ticket? Absolutely not. Red Hat is not
worth $20K USD per ticket. At scale, we can fund an entire team to
build something like CentOS for this cost, and that's exactly what
happens. This isn't being cheap. This is business. If Red Hat is in
the business of Open Source, including reselling the works of many
others including myself, Red Hat should know this.

Mark Mielke <mark.mielke at gmail.com>