[CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…

On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath <mmcgrath at redhat.com> wrote:
>> I wanted to pay Red Hat for the services we received. I could justify
>> this. However, we were looking at a $2M USD RHEL expense for the next
>> year, and $4M USD for the year after that. I could not justify this,
>> when so many alternatives existed that provide what was substantially
>> the same content and service.

I just explained how Red Hat's subscription model and pricing forced
me to review if we could justify a Red Hat cost increase from $0.5M
USD annually to $4M USD annually for essentially no increase in
service levels, and I found it was not. We could fund a small army
with $4M USD annually, and build our own distribution. This is at
least 20 people worth of salaries.

Mike McGrath's answer below was not "I see your point, I will look
into this and fix this" as it should have been. Instead, his point was
"I think you are ignorant and not aware of our value."

This level of arrogance ("having or revealing an exaggerated sense of
one's own importance or abilities") meant I was forced to reduce our
contribution to Red Hat from a value we were willing to pay of $0.5M,
down to <$0.1M, and with this new CentOS choice, it will likely drop
to $0, by investing in alternatives.

I want it to be perfectly clear that Red Hat management is making Red
Hat a difficult proposition for us to sell, even for proponents and
admirers of Red Hat, and these choices have direct consequences on the
Red Hat bottom line. Most other vendors we work with provide large
discounts to support cases such as ours, so that it is a win/win. Red
Hat provided no such option, and effectively priced itself out of the
market. 2 years later when they saw that we were not bluffing, they
offered a discount that only kicks in above $1M, and would still be
about 20% discount. These terms are unacceptable.

Do you think I wanted to use my technical skills to rip out Red Hat
binaries from everywhere in our company in a short time period? I
didn't. But, I had no choice. After a year of discussions, Red Hat
gave me no choice.

Let's dissect Mike McGrath's answer in a little more detail:

> I have to assume you actually downloaded and installed RHEL.  To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain.

Any large deployment cannot use Red Hat servers to deploy RHEL, but
requires in-house CDN. Also, any large deployment requires
customization, which then involves overlaying RHEL packages with
non-RHEL packages including packages from external sources such as
EPEL, and in-house packages. The CDN cost is therefore *ours*, not
*yours*.

> Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts.  We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.

We have a team of in-house people providing front-end support for our
users. Red Hat did not eliminate the need for this team. If we
quadrupled our installs as was projected, the brunt of this cost would
have been borne by our in-house team, not by Red Hat. The Red Hat
subscription model is broken. The Red Hat proposition becomes
problematic with scale.

Red Hat may have 1,000 people working on hardening RHEL, but the
global community has millions of people, and the global community
includes people such as myself who contribute back. Red Hat's
contribution here is significant, but it is not infinite. I have
contributed to several fixes to Red Hat Enterprise Linux, and Red Hat
deploys components that I have worked on. It's great that you
contribute back - just as it is great that all companies who work on
F/OSS portions of Linux contribute back. Red Hat is one of many
companies who contribute back, and this means that the Red Hat value
proposition is not infinite. If Red Hat prices itself out of the
market, how will Red Hat support these 1,000 people? How is choosing
to force me to reduce our RHEL deployment from $0.5M USD annually to
$0.1M USD annually, because of unacceptable subscription terms, in the
greater interest of Red Hat?

> We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.

This paywall service is sometimes useful, but most of the time
problematic. Our users don't have access to it. I usually avoid using
it wherever possible, and recommend it be avoided. This information
should be publicly accessible.

> We have subject matter experts and sometimes project leads those critical upstream projects.  If you've got a strange problem, or need a feature implemented, we've got the people who can solve it.  The best in the industry (at least for those who need the best, not everyone does).

So does Oracle, or Google, or AWS, or hundreds of others. I
participate on the devel mailing lists, and I pay attention to who
contributes and who provides answers. Red Hat is definitely on this
list, but Red Hat is not alone - and for the most part, this service
is a "cost of doing business", and not directly tied to a
subscription. When I had an issue with Qemu and live migrations of
nested virtualization, it was Oracle that contributed the fixes for
live migration of nested virtualization. I don't want to reduce the
value of Red Hat here - I want to make it clear that Red Hat is one of
several important players.

> And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.

Especially the "you're welcome" at the end is arrogance. Actually, Red
Hat has plenty of bugs after "hardening", many of which I have had to
deal with, including getting fixes made upstream. But, I don't find
Red Hat support that useful for this. Too often, I have to figure out
for myself what is wrong, and I may as well describe the problem on
bugzilla.redhat.com myself. I don't need to open a support ticket and
go through somebody else to do this.

I know you don't want to hear my story. You would prefer the story
that Red Hat is awesome, that Red Hat can set whatever price they wish
and it should be considered a bargain, and anybody who disagrees is
ignorant. But, you are talking about your customers here - including
customers showing good will and willingness to negotiate. You forced
me to substantially remove Red Hat from our systems, by refusing to
negotiate acceptable terms. You did this. I didn't do this.

-- 
Mark Mielke <mark.mielke at gmail.com>