On Thu, Dec 17, 2020 at 8:48 PM Mark Mielke <mark.mielke@gmail.com> wrote:
On Thu, Dec 17, 2020 at 8:06 AM Yedidyah Bar David <didi@redhat.com> wrote:
> But I also want to add another reason: Spend it on Red Hat/RHEL, simply
> because they are the best, and worth your money. If you do not think so,
> don't.

I believed in this point strongly in 2015. Let me share some details
about what forced this to change.

We were an RHEL-only shop, with a ~$0.5M USD / year annual renewal. We
were preparing to deploy a private cloud, with expectations of
quadrupling our Linux instances in the company. According to the
contract, this would have been around $2M USD / year annual renewal. I
did some review of what Red Hat provided us, and I came to a few
problematic conclusions. One conclusion was that for the past few
years, we were averaging a support cost of $20K USD per ticket, and
these were for the most part very basic tickets that should never have
been opened in the first place. For more complex tickets, including
one that I opened to test and evaluate the system - the ticket
actually went nowhere. The support person was unwiling to push the
developers to fix a problem. I had to escalate and push hard, and then
participate in the bugzilla.redhat.com discussions before finally
getting the attention of the backend Red Hat developer who
accidentally broke it, who quickly agreed to fix it.

I don't really want to get into all the details of what right, and
what went wrong. I want you to think about this number for a second.
$20K USD per support ticket, including basic support problems. Which
of you can easily justify this to your directors? Would you agree to
pay this out of your own pocket, if you were an owner of the company? 
I did a lot of analysis, and had many discussions with Red Hat staff,
who by the way - were all great. I believe emails I sent were
circulated around Red Hat in the 2016 and 2017 timeframe, although I
don't know if this was just the sales and technical people that worked
with us, or if they weren't further. There was a great deal of
interest in the problem I was describing, but no ability for anybody
to do anything about it. The subscription model was fixed.

The Red Hat subscription model is particularly problematic in that
*all* systems must be subscribed, whether they require support or not.
The question at renewal time, and the wording of the contract is not
"which systems require support?" The question is "how many Red Hat
systems do you have?" This means that test systems, development
system, customer test systems, are all counted as individual
subscriptions of some type.

This also means that when we want to horizontally scale a service,
such as by having it be split across 3 virtual machines instead of 1,
we need 3 licenses. There are some allowances in there such as how the
Standard Subscription permits 2 VM per subscription, but you
eventually conclude that you must subscribe the entire hypervisor.
Except - we use many different operating systems on the hypervisors,
and we are now talking about licensing every hypervisor for Red Hat,
even if the system might not be running a Red Hat host or guest.
Effectively, we end up being charged for running Ubuntu or Windows
unless we design our system to limit the migration of virtual machines
so that we have a "RHEL-only cluster" and a "non-RHEL cluster".

The Red Hat sales person had no levers to get around this. Red Hat is
fundamentally sold per subscription. Even at bulk discount, the
per-subscription discount was no more than 20% per system.

I wanted to pay Red Hat for the services we received. I could justify
this. However, we were looking at a $2M USD RHEL expense for the next
year, and $4M USD for the year after that. I could not justify this,
when so many alternatives existed that provide what was substantially
the same content and service.

Red Hat priced themselves out of the market. I didn't do this. Red Hat
did this. This is why Oracle or Facebook cannot use Red Hat. Oracle
Linux might not exist if Red Hat had offered an arrangement with
Oracle. It is fine to live by a code that says "my product is worth
$X", but then it better be worth $X, or people won't buy it.

Do I think Red Hat has done great things for the community? Yes, with
the caveat that a great number of companies are doing great things for
the community, and Red Hat is not alone.

Can I justify $20K USD per ticket? Absolutely not. Red Hat is not
worth $20K USD per ticket. At scale, we can fund an entire team to
build something like CentOS for this cost, and that's exactly what
happens. This isn't being cheap. This is business. If Red Hat is in
the business of Open Source, including reselling the works of many
others including myself, Red Hat should know this.


I have to assume you actually downloaded and installed RHEL.  To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain.  

Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts.  We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.

We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.

We have subject matter experts and sometimes project leads those critical upstream projects.  If you've got a strange problem, or need a feature implemented, we've got the people who can solve it.  The best in the industry (at least for those who need the best, not everyone does).

And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.

             -Mike