[CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…

On Fri, Dec 18, 2020 at 11:12 AM Mark Mielke <mark.mielke at gmail.com> wrote:

> On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath <mmcgrath at redhat.com> wrote:
> >> I wanted to pay Red Hat for the services we received. I could justify
> >> this. However, we were looking at a $2M USD RHEL expense for the next
> >> year, and $4M USD for the year after that. I could not justify this,
> >> when so many alternatives existed that provide what was substantially
> >> the same content and service.
>
> I just explained how Red Hat's subscription model and pricing forced
> me to review if we could justify a Red Hat cost increase from $0.5M
> USD annually to $4M USD annually for essentially no increase in
> service levels, and I found it was not. We could fund a small army
> with $4M USD annually, and build our own distribution. This is at
> least 20 people worth of salaries.
>
> Mike McGrath's answer below was not "I see your point, I will look
> into this and fix this" as it should have been. Instead, his point was
> "I think you are ignorant and not aware of our value."
>
> This level of arrogance ("having or revealing an exaggerated sense of
> one's own importance or abilities") meant I was forced to reduce our
> contribution to Red Hat from a value we were willing to pay of $0.5M,
> down to <$0.1M, and with this new CentOS choice, it will likely drop
> to $0, by investing in alternatives.
>
> I want it to be perfectly clear that Red Hat management is making Red
> Hat a difficult proposition for us to sell, even for proponents and
> admirers of Red Hat, and these choices have direct consequences on the
> Red Hat bottom line. Most other vendors we work with provide large
> discounts to support cases such as ours, so that it is a win/win. Red
> Hat provided no such option, and effectively priced itself out of the
> market. 2 years later when they saw that we were not bluffing, they
> offered a discount that only kicks in above $1M, and would still be
> about 20% discount. These terms are unacceptable.
>
> Do you think I wanted to use my technical skills to rip out Red Hat
> binaries from everywhere in our company in a short time period? I
> didn't. But, I had no choice. After a year of discussions, Red Hat
> gave me no choice.
>
> Let's dissect Mike McGrath's answer in a little more detail:
>
> > I have to assume you actually downloaded and installed RHEL.  To do so
> you would have used our CDN and a fairly extensive (and audited and
> secured) supply chain.
>
> Any large deployment cannot use Red Hat servers to deploy RHEL, but
> requires in-house CDN. Also, any large deployment requires
> customization, which then involves overlaying RHEL packages with
> non-RHEL packages including packages from external sources such as
> EPEL, and in-house packages. The CDN cost is therefore *ours*, not
> *yours*.
>
> > Over 1,000 people work on the actual RHEL bits *after* the community has
> already worked on it and many more support those efforts.  We push all our
> code upstream before we release it so presumably, you got some value out of
> RHEL or you would have just used upstream.
>
> We have a team of in-house people providing front-end support for our
> users. Red Hat did not eliminate the need for this team. If we
> quadrupled our installs as was projected, the brunt of this cost would
> have been borne by our in-house team, not by Red Hat. The Red Hat
> subscription model is broken. The Red Hat proposition becomes
> problematic with scale.
>
> Red Hat may have 1,000 people working on hardening RHEL, but the
> global community has millions of people, and the global community
> includes people such as myself who contribute back. Red Hat's
> contribution here is significant, but it is not infinite. I have
> contributed to several fixes to Red Hat Enterprise Linux, and Red Hat
> deploys components that I have worked on. It's great that you
> contribute back - just as it is great that all companies who work on
> F/OSS portions of Linux contribute back. Red Hat is one of many
> companies who contribute back, and this means that the Red Hat value
> proposition is not infinite. If Red Hat prices itself out of the
> market, how will Red Hat support these 1,000 people? How is choosing
> to force me to reduce our RHEL deployment from $0.5M USD annually to
> $0.1M USD annually, because of unacceptable subscription terms, in the
> greater interest of Red Hat?
>
> > We also have an extensive KBase and are working for more "in your face"
> ways to let you know something is up with your servers via services like
> insights so you can fix them before there's an impact to the services you
> run.
>
> This paywall service is sometimes useful, but most of the time
> problematic. Our users don't have access to it. I usually avoid using
> it wherever possible, and recommend it be avoided. This information
> should be publicly accessible.
>
> > We have subject matter experts and sometimes project leads those
> critical upstream projects.  If you've got a strange problem, or need a
> feature implemented, we've got the people who can solve it.  The best in
> the industry (at least for those who need the best, not everyone does).
>
> So does Oracle, or Google, or AWS, or hundreds of others. I
> participate on the devel mailing lists, and I pay attention to who
> contributes and who provides answers. Red Hat is definitely on this
> list, but Red Hat is not alone - and for the most part, this service
> is a "cost of doing business", and not directly tied to a
> subscription. When I had an issue with Qemu and live migrations of
> nested virtualization, it was Oracle that contributed the fixes for
> live migration of nested virtualization. I don't want to reduce the
> value of Red Hat here - I want to make it clear that Red Hat is one of
> several important players.
>
> > And finally, while you are only calculating Red Hat value via support
> tickets..... It sounds like you rarely needed it so on behalf of the
> engineering and QE team I'd say... you're welcome.
>
> Especially the "you're welcome" at the end is arrogance. Actually, Red
> Hat has plenty of bugs after "hardening", many of which I have had to
> deal with, including getting fixes made upstream. But, I don't find
> Red Hat support that useful for this. Too often, I have to figure out
> for myself what is wrong, and I may as well describe the problem on
> bugzilla.redhat.com myself. I don't need to open a support ticket and
> go through somebody else to do this.
>
> I know you don't want to hear my story. You would prefer the story
> that Red Hat is awesome, that Red Hat can set whatever price they wish
> and it should be considered a bargain, and anybody who disagrees is
> ignorant. But, you are talking about your customers here - including
> customers showing good will and willingness to negotiate. You forced
> me to substantially remove Red Hat from our systems, by refusing to
> negotiate acceptable terms. You did this. I didn't do this.
>
> --
> Mark Mielke <mark.mielke at gmail.com>
>

Mark, I've heard responses like this for two weeks.  You describe me as
arrogant, and not listening, and I can only assume that you think we at Red
Hat are fools who don't understand the enterprise.  You've used a lot of
fairly charged language in your response and you and many others have the
"I'm going to show Red Hat" attitude.  The problem is, and I think too few
people realize this:

Red Hat isn't aiming for total global domination

So when you say I forced you to go somewhere else.  You have to understand
that in the open-source world, and in Red Hat's business plan, we know
alternatives exist.  That's the whole point of it.  If you don't like the
level of service you're getting.  Go somewhere else.  But don't pretend
that the RHEL bits don't matter and that minimizing RHEL's contributions on
a CentOS-devel mailing list will teach us something.  You posted that your
relationship with Red Hat boils down to cost per ticket - you did that, not
me.  I think if we've learned anything in the last two weeks it's that the
bits seem to matter very much to people.  They matter so much that people
are feverishly trying to recreate RHEL instead of going to one of those
many alternatives that already exist.

Just know that I mean it when I say, for those of you that are moving on.
We wish you luck, we understand, and we'll see you around.

              -Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20201218/b2a8e02a/attachment.html>