[CentOS-devel] [EXT] Re: https://blog.centos.org/2020/12/future-is-centos-stream/

>On 10/12/2020 16.50, Rich Bowen wrote:
>> >> >>* On 12/9/20 12:32 PM, David Hrbáč wrote:
*>>>*     I don't use CentOS Stream, I use RHEL. I use RHEL to develop software
*>>>*     for RHEL and compatible OS clones, including CentOS. If Stream
*>>>* retains
*>>>*     binary compatibility, and specifically kernel ABI compatibility, then
*>>>*     the users of the software packages we develop can continue to use
*>>>* them.
*>>>*     If not, they can't. Simple as that. So please don't push rolling
*>>>* kernel
*>>>*     updates to Stream that break the kernel ABI.
*>>>>>>>>>* Rolling kernel updates are going to kill all the traditional HPC
*>>>* clusters. Almost 25% of the TOP 500 HPC clusters run CentOS. See
*>>>* https://www.top500.org/statistics/list/
<https://www.top500.org/statistics/list/>
*>>>* <https://www.top500.org/statistics/list/
<https://www.top500.org/statistics/list/>>
*>> >>* I was under the impression that practically all of those run custom
*>>* kernels anyway, right?
*
> Obviously in no way generally valid: In the last ~5 years I have been
> involved in / responsible for the administration of two systems that
> have been listed in the TOP500 (both lower half of TOP500, but top field
> of Green 500 at their respective time). Both systems are running CentOS
> 7 with vanilla kernel. We only add external 3rd party kernel modules as
> required.

> There have been plans/conversations/ideas about updating the newer
> system to CentOS Linux 8 soon. However, with the changed perspective of
> CentOS, this plan is now dead as well. Depending on how CentOS Stream
> works out (i.e., kernel ABI compatibility to current RHEL minor
> release), switching to CentOS Stream might be an option.

Same here, almost a decade in HPC and have admin'd multiple systems in
the Top 100, attended SC conferences regularly, as well as other
gatherings in the HPC community, all the systems I know of personally,
and I know of a few in the Top 10; dozens in the Top 100, currently
run vanilla kernels.  While I can't speak for all of the vast amount
of Supercomputing centers, from all my experience I don't know of many
places that run custom kernels.


>> Can you elaborate on what you mean by rolling kernel updates?  If you
>> mean wholesale kernel rebases (e.g. kernel-5.8 -> kernel-5.9), that's
>> not what Stream will be doing.

>> Stream will include more frequent updates of the *RHEL* kernel, where
>> we take the RHEL kABI whitelist very seriously.  It is literally the
>> RHEL kernel.  In the context of HPC deployments, they could use Stream
>> and choose when to update and reboot those machines based on the
>> kernel updates that come out, just as the would with CentOS Linux.

This isn't possible, I'll use an example from IBM as maybe that'll
help illustrate it better.  When RHEL 8.3 was released (for example
but same applies to 7.8, 7.9, 8.1, 8.2, etc.), we couldn't just
upgrade to it as the kernel changes break the ability to build the
kernel module for IBM Spectrum Scale -- the cluster's main shared file
system (eg. running mmbuildgpl would fail, you would have to downgrade
the kernel to the prior .x release's version to get it to run); and
the Lustre project is in the same boat; btw these two file systems
service the VAST majority of HPC systems around the world.  We also
have to build Mellanox (Nvidia Networking) OFED against the kernel and
use their drivers that are released with RHEL updates.  Upgrading the
kernel to something ahead of the latest official RHEL release (eg. the
current, what will be the 8.4 kernel, in Stream) would break both
cluster networking and storage pretty much every time, it has every .x
--> .(x+1) release I've ever seen over the past 4 years, which would
render the entire cluster unusable.  Thus I just do not see a way as
you guys describe it where Stream could ever be a viable option as we
need the kernel to freeze, have vendors get their drivers and software
to work with it, THEN deploy it.  What is in Stream will always be too
far ahead of what the vendors have available and built/tested against.
We could use yum/dnf to lock the kernel versions (and some other
supporting packages) but that could have other implications and adds a
lot of fragility as one is essentially trying to un-stream a stream
release.

This move has alienated the HPC community from CentOS, and the naivety
that RH folks are showing about the HPC ecosystem is even more
saddening.  HPC users that are on CentOS 7 will likely get stuck there
for a bit, while they re-evaluate offerings and look for where to jump
to next, which is sad as 8 has/had a lot of nice things.  Those who
have already jumped to CentOS 8 are now in a very tricky spot, which
can not be understated.  Red Hat's lack of plan to go along with this
announcement is mind boggling, something I'd expect from some Silicon
Valley startup, not a seasoned industry player.  A "hang tight we'll
get back to you on a new offering in 1H 2021" (and if you don't
like/can't afford our offer congrats you now have 6 months to lift and
shift your entire environment and the dozens to hundreds of end-user
applications you support) is an insult to folks who run these systems.

I completely get the benefits of CentOS Stream for RHEL development,
it makes a lot of sense and for that I think Stream is a great thing.
Also likely a good thing for the hyperscalers who likely run most, if
not all, their stuff in user space and have large development budgets
to support their operations. Though I've yet to see an answer from RH,
about why both CentOS proper and Stream couldn't continue as they were
since they aren't mutually exclusive.  We're left to assume it's about
extracting more money from users and (an attempt at) forcing more
community/vendor involvement further upstream.

JD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20201212/a68a6132/attachment.html>