On 10/12/2020 16.50, Rich Bowen wrote:
- On 12/9/20 12:32 PM, David Hrbáč wrote:
*>>>* I don't use CentOS Stream, I use RHEL. I use RHEL to develop software *>>>* for RHEL and compatible OS clones, including CentOS. If Stream *>>>* retains *>>>* binary compatibility, and specifically kernel ABI compatibility, then *>>>* the users of the software packages we develop can continue to use *>>>* them. *>>>* If not, they can't. Simple as that. So please don't push rolling *>>>* kernel *>>>* updates to Stream that break the kernel ABI. *>>>>>>>>>* Rolling kernel updates are going to kill all the traditional HPC *>>>* clusters. Almost 25% of the TOP 500 HPC clusters run CentOS. See *>>>* https://www.top500.org/statistics/list/ https://www.top500.org/statistics/list/ *>>>* <https://www.top500.org/statistics/list/ https://www.top500.org/statistics/list/> *>> >>* I was under the impression that practically all of those run custom *>>* kernels anyway, right? *
Obviously in no way generally valid: In the last ~5 years I have been involved in / responsible for the administration of two systems that have been listed in the TOP500 (both lower half of TOP500, but top field of Green 500 at their respective time). Both systems are running CentOS 7 with vanilla kernel. We only add external 3rd party kernel modules as required.
There have been plans/conversations/ideas about updating the newer system to CentOS Linux 8 soon. However, with the changed perspective of CentOS, this plan is now dead as well. Depending on how CentOS Stream works out (i.e., kernel ABI compatibility to current RHEL minor release), switching to CentOS Stream might be an option.
Same here, almost a decade in HPC and have admin'd multiple systems in the Top 100, attended SC conferences regularly, as well as other gatherings in the HPC community, all the systems I know of personally, and I know of a few in the Top 10; dozens in the Top 100, currently run vanilla kernels. While I can't speak for all of the vast amount of Supercomputing centers, from all my experience I don't know of many places that run custom kernels.
Can you elaborate on what you mean by rolling kernel updates? If you mean wholesale kernel rebases (e.g. kernel-5.8 -> kernel-5.9), that's not what Stream will be doing.
Stream will include more frequent updates of the *RHEL* kernel, where we take the RHEL kABI whitelist very seriously. It is literally the RHEL kernel. In the context of HPC deployments, they could use Stream and choose when to update and reboot those machines based on the kernel updates that come out, just as the would with CentOS Linux.
This isn't possible, I'll use an example from IBM as maybe that'll help illustrate it better. When RHEL 8.3 was released (for example but same applies to 7.8, 7.9, 8.1, 8.2, etc.), we couldn't just upgrade to it as the kernel changes break the ability to build the kernel module for IBM Spectrum Scale -- the cluster's main shared file system (eg. running mmbuildgpl would fail, you would have to downgrade the kernel to the prior .x release's version to get it to run); and the Lustre project is in the same boat; btw these two file systems service the VAST majority of HPC systems around the world. We also have to build Mellanox (Nvidia Networking) OFED against the kernel and use their drivers that are released with RHEL updates. Upgrading the kernel to something ahead of the latest official RHEL release (eg. the current, what will be the 8.4 kernel, in Stream) would break both cluster networking and storage pretty much every time, it has every .x --> .(x+1) release I've ever seen over the past 4 years, which would render the entire cluster unusable. Thus I just do not see a way as you guys describe it where Stream could ever be a viable option as we need the kernel to freeze, have vendors get their drivers and software to work with it, THEN deploy it. What is in Stream will always be too far ahead of what the vendors have available and built/tested against. We could use yum/dnf to lock the kernel versions (and some other supporting packages) but that could have other implications and adds a lot of fragility as one is essentially trying to un-stream a stream release.
This move has alienated the HPC community from CentOS, and the naivety that RH folks are showing about the HPC ecosystem is even more saddening. HPC users that are on CentOS 7 will likely get stuck there for a bit, while they re-evaluate offerings and look for where to jump to next, which is sad as 8 has/had a lot of nice things. Those who have already jumped to CentOS 8 are now in a very tricky spot, which can not be understated. Red Hat's lack of plan to go along with this announcement is mind boggling, something I'd expect from some Silicon Valley startup, not a seasoned industry player. A "hang tight we'll get back to you on a new offering in 1H 2021" (and if you don't like/can't afford our offer congrats you now have 6 months to lift and shift your entire environment and the dozens to hundreds of end-user applications you support) is an insult to folks who run these systems.
I completely get the benefits of CentOS Stream for RHEL development, it makes a lot of sense and for that I think Stream is a great thing. Also likely a good thing for the hyperscalers who likely run most, if not all, their stuff in user space and have large development budgets to support their operations. Though I've yet to see an answer from RH, about why both CentOS proper and Stream couldn't continue as they were since they aren't mutually exclusive. We're left to assume it's about extracting more money from users and (an attempt at) forcing more community/vendor involvement further upstream.
JD
-
JD Maloney