[CentOS-devel] Before You Get Mad About The CentOS Stream Change, Think About…

On Fri, Dec 18, 2020 at 9:58 AM Mike McGrath <mmcgrath at redhat.com> wrote:
>
>
>
> On Thu, Dec 17, 2020 at 8:48 PM Mark Mielke <mark.mielke at gmail.com> wrote:
>>
>> On Thu, Dec 17, 2020 at 8:06 AM Yedidyah Bar David <didi at redhat.com> wrote:
>> > But I also want to add another reason: Spend it on Red Hat/RHEL, simply
>> > because they are the best, and worth your money. If you do not think so,
>> > don't.
>>
>> I believed in this point strongly in 2015. Let me share some details
>> about what forced this to change.
>>
>> We were an RHEL-only shop, with a ~$0.5M USD / year annual renewal. We
>> were preparing to deploy a private cloud, with expectations of
>> quadrupling our Linux instances in the company. According to the
>> contract, this would have been around $2M USD / year annual renewal. I
>> did some review of what Red Hat provided us, and I came to a few
>> problematic conclusions. One conclusion was that for the past few
>> years, we were averaging a support cost of $20K USD per ticket, and
>> these were for the most part very basic tickets that should never have
>> been opened in the first place. For more complex tickets, including
>> one that I opened to test and evaluate the system - the ticket
>> actually went nowhere. The support person was unwiling to push the
>> developers to fix a problem. I had to escalate and push hard, and then
>> participate in the bugzilla.redhat.com discussions before finally
>> getting the attention of the backend Red Hat developer who
>> accidentally broke it, who quickly agreed to fix it.
>>
>> I don't really want to get into all the details of what right, and
>> what went wrong. I want you to think about this number for a second.
>> $20K USD per support ticket, including basic support problems. Which
>> of you can easily justify this to your directors? Would you agree to
>> pay this out of your own pocket, if you were an owner of the company?
>>
>> I did a lot of analysis, and had many discussions with Red Hat staff,
>> who by the way - were all great. I believe emails I sent were
>> circulated around Red Hat in the 2016 and 2017 timeframe, although I
>> don't know if this was just the sales and technical people that worked
>> with us, or if they weren't further. There was a great deal of
>> interest in the problem I was describing, but no ability for anybody
>> to do anything about it. The subscription model was fixed.
>>
>> The Red Hat subscription model is particularly problematic in that
>> *all* systems must be subscribed, whether they require support or not.
>> The question at renewal time, and the wording of the contract is not
>> "which systems require support?" The question is "how many Red Hat
>> systems do you have?" This means that test systems, development
>> system, customer test systems, are all counted as individual
>> subscriptions of some type.
>>
>> This also means that when we want to horizontally scale a service,
>> such as by having it be split across 3 virtual machines instead of 1,
>> we need 3 licenses. There are some allowances in there such as how the
>> Standard Subscription permits 2 VM per subscription, but you
>> eventually conclude that you must subscribe the entire hypervisor.
>> Except - we use many different operating systems on the hypervisors,
>> and we are now talking about licensing every hypervisor for Red Hat,
>> even if the system might not be running a Red Hat host or guest.
>> Effectively, we end up being charged for running Ubuntu or Windows
>> unless we design our system to limit the migration of virtual machines
>> so that we have a "RHEL-only cluster" and a "non-RHEL cluster".
>>
>> The Red Hat sales person had no levers to get around this. Red Hat is
>> fundamentally sold per subscription. Even at bulk discount, the
>> per-subscription discount was no more than 20% per system.
>>
>> I wanted to pay Red Hat for the services we received. I could justify
>> this. However, we were looking at a $2M USD RHEL expense for the next
>> year, and $4M USD for the year after that. I could not justify this,
>> when so many alternatives existed that provide what was substantially
>> the same content and service.
>>
>> Red Hat priced themselves out of the market. I didn't do this. Red Hat
>> did this. This is why Oracle or Facebook cannot use Red Hat. Oracle
>> Linux might not exist if Red Hat had offered an arrangement with
>> Oracle. It is fine to live by a code that says "my product is worth
>> $X", but then it better be worth $X, or people won't buy it.
>>
>> Do I think Red Hat has done great things for the community? Yes, with
>> the caveat that a great number of companies are doing great things for
>> the community, and Red Hat is not alone.
>>
>> Can I justify $20K USD per ticket? Absolutely not. Red Hat is not
>> worth $20K USD per ticket. At scale, we can fund an entire team to
>> build something like CentOS for this cost, and that's exactly what
>> happens. This isn't being cheap. This is business. If Red Hat is in
>> the business of Open Source, including reselling the works of many
>> others including myself, Red Hat should know this.
>>
>
> I have to assume you actually downloaded and installed RHEL.  To do so you would have used our CDN and a fairly extensive (and audited and secured) supply chain.
>
> Over 1,000 people work on the actual RHEL bits *after* the community has already worked on it and many more support those efforts.  We push all our code upstream before we release it so presumably, you got some value out of RHEL or you would have just used upstream.
>
> We also have an extensive KBase and are working for more "in your face" ways to let you know something is up with your servers via services like insights so you can fix them before there's an impact to the services you run.
>
> We have subject matter experts and sometimes project leads those critical upstream projects.  If you've got a strange problem, or need a feature implemented, we've got the people who can solve it.  The best in the industry (at least for those who need the best, not everyone does).
>
> And finally, while you are only calculating Red Hat value via support tickets..... It sounds like you rarely needed it so on behalf of the engineering and QE team I'd say... you're welcome.
>

I think what would be nice to have would be more affordable
"self-support" options, even for production uses. Believe it or not, I
often tell folks that the best way to support Fedora and CentOS is to
buy a RHEL subscription. I used to tell folks that purchasing the
self-support options at $50~$150 per box per year is not a bad way to
support the excellent work that you folks do. Unfortunately, those
options no longer exist, and now self-support options for RHEL are
just simply too expensive.

I don't presume to know much about how the balance of things work out
for full support vs self-support buys of RHEL subscriptions, but I
think a lot of us would be willing to buy RHEL for our workloads if it
was more affordable. I'm *not* even saying "entirely free", I'm saying
making it "impulse-buy" levels of cheap that people would buy it and
use it to have RHEL and support the work Red Hat does. And companies
that are in the business of running large fleets of systems but have
to go without support due to the low margin/budget could take
advantage of that and put it into their cost calculus. Sure, $50 isn't
$0, but it does become one of the lowest parts of the cost of building
out a system if you're already doing self-support anyway. That also
makes the conversion story to higher-value support tiers *so* much easier, too.

That's not to say that having no-cost RHEL options for specific
production cases (HPC, research, etc.) shouldn't happen (I know those
places often have a budget at zero or in the red most of the time),
but I think it'd be great to make RHEL more of an option for everyone
else, too.




--
真実はいつも一つ！/ Always, there's only one truth!