[CentOS-devel] Documenting the CentOS Linux 8 EOL process

Fri Jul 16 12:13:25 UTC 2021
Nico Kadel-Garcia <nkadel at gmail.com>

On Fri, Jul 16, 2021 at 3:44 AM Neal Gompa <ngompa13 at gmail.com> wrote:
>
> On Thu, Jul 15, 2021 at 7:25 PM Nico Kadel-Garcia <nkadel at gmail.com> wrote:
> >
> > On Thu, Jul 15, 2021 at 4:36 PM Johnny Hughes <johnny at centos.org> wrote:
> >
> > > Other than the kernels, I'm sorry but you would be hard pressed to find
> > > something in Stream that doesn't also happen in CentOS and even RHEL.
> > > It just happens a month or two earlier in Stream.  BUT, so do bugfixes :)
> >
> > I hope you understand that we expect both gross and subtle failures of
> > the beta testing in CentOS 8 Stream. I expect it with python modules,
> > when the author of one module fails to restrict versions of other
> > modules, especially modules brought in by dependencies, to versions
> > that are compatible because they cannot *predict* that the next
> > version will be compatible. It can be very difficult to anticipate
> > those: ansible is a great example right now, since the latest ansible
> > release requires python 3.8 and the panoply of necessary modules are
> > not yet available anywhere.
> >
>
> I hope you understand that we've actually taken care to deal with
> issues like that. The CentOS/RHEL platform inherits all the work the
> Fedora Python team and myself have done over the years to simplify,
> automate, and rationalize Python packaging. The Python dependency

I see the efforts. I've run into and tried to resolve them myself
myself, especially with python sphinx and the dependency chains. The
python generator has difficulty with dependencies, and dependencies,
are updated and the original module authors did not know and did not
publish requirements of modules *less* than an incompatible version.
It's an underlying python architecture issue, one that I and others
rely on RPM's to keep consistent. It's tricky.

> generator *already* deals with most of those problems for us. The
> modularity tooling makes it relatively trivial for us to rebootstrap
> the world of Python modules for a new Python version if we so choose.
> Now, Red Hat has not currently elected to do this, but that doesn't mean it
> isn't possible.

Many things are possible. Enacting modularity for python module
resolution would destabilize them further, modularity is *not* woking
well.

> > It's one of the dangers of the "streaming" model, when unanticipated
> > dependencies are discovered in the field.  It's why I expect people to
> > use rsync or reposync tools to generate internal mirrors with locked
> > snapshots, which they used to do with CentOS point releases.
>
> You mean like how people *already* did it because they thought regular
> CentOS updates were "too dangerous"? Frankly, I don't buy what you're

It's familiar turf. Many used the point releases for creating new
hosts and disabled the update channels at system build time, precisely
to avoid randomized out-of-band updates.  They did not run "yum
update" at build time or after deployment, they did only "yum update
--security", and switched their configs to pint to the new point
releases only after testing and review.

> selling here. To make matters worse, the previous model gave you
> *zero* opportunity to resolve issues with updates if they were buggy.

It's a trade-off. Bugs created by unplanned vendor updates are very
bad indeed and preventable, and most preventable by locking down the
update process. Red Hat tried to sell spacewalk setups or RHN to
provide that kind of update management, but it turned out to be far
too micro-managed and expensive for most environments. The update
channels were available: they simply are not, and should not be,
turned on by default for high stability, production grade, systems.
The idea that updates will not introduce bugs is not a historically
well-founded one, especially for updates that have not had yet gone
through beta testing, and that instead are part of the beta testing
cycle as CentOS 8 Stream is now designed for.

> They just stayed broken for months or years. At least now there's a
> chance of them getting fixed in a reasonable time window.

> Nico, you spend far too much time looking at CentOS Stream the wrong
> way. You should be looking at this as an opportunity to bring value to
> the folks you work with leveraging the CentOS platform and to the
> CentOS ecosystem as a whole. You can validate things early and often,

I do so, because I'm aggressive about catching issues before the
people who sign my paychecks. I'm not looking forward to expanding my
mock setups to include CentOs 8 Stream as well as CentOS 8. The fiasco
is very familiar from Red Hat 9, which tried the same "we do not have
point releases" stunt, and was abandoned with RHEL. I anticipate that
CentOS 9 or CentOS 10 will abandon most of this replayed debate, or
people will abandon CentOS in favor of one of the alternatives. I
remember switching from White Box Linux to CenTOS, and a dalliance
with Scientific Linux when CentOS took a really, really long time to
release CentOS 6.

> identify specific issues, contribute fixes, and be part of the process
> to raise the quality bar for Enterprise Linux as a whole. And that's
> the bare minimum! You could also go so much further if you wanted by
> developing features that are useful to you and your organization in
> Fedora and bringing them to RHEL/CentOS through the RHEL Feature
> Proposal process handled by the CentOS Stream Feature SIG[1]. This

I publish. I've had repeated problems with the Fedora registration process.

> allows folks in the CentOS community to lobby for improvements in the
> platform in an avenue that was never available to them before as
> non-customers.
>
> The whole point of CentOS Stream is continuous improvement. In a very
> real way, CentOS Stream allows CentOS to truly earn its name as the
> "Community Enterprise Operating System". We, the community, now have
> the power to contribute, develop, and deliver *the* community
> enterprise Linux platform. We've never had that before. We are the
> folks with the opportunity to literally *set the standard* for
> Enterprise Linux. Don't waste it!

IWe had nearly all of that flexibility with EPEL (for components not
in CentOS), with RPMforge (which admittedly seems to have dried up as
Dag Weiers found other work to do, I took him drinking about the time
he stepped back from that), with RPMfusion, and with IUS (which I find
handy for nignx, haproxy, and git updates). It's not been as well
integrated as I'd hope that CentOS 8 Steam might be, but they've
certainly existed as development and testing environments.

> And I'm putting my money where my mouth is. I've been actively
> contributing to the platform since it launched. Even before this whole
> self-inflicted disaster happened, I had already switched my CentOS
> machines to CentOS Stream. I've been working in the Hyperscale SIG[2]
> to develop features that I hope will be part of RHEL one day. And I'm
> already looking forward to CentOS Stream 9.
>
> [1]: https://wiki.centos.org/SpecialInterestGroup/StreamFeatureRequest
> [2]: https://wiki.centos.org/SpecialInterestGroup/Hyperscale

Unfortunately, we weren't asked. It's not what was on the menu, and
we're not being allowed to send it back to finish baking. We're having
to set up our own ovens at our own tables. I *publish* some of those
oven tools, thank you for reminding me that I need to update them to
include CentOS 8 Stream.