[CentOS-devel] freetype package missed in repo

On Fri, Aug 6, 2021 at 9:30 AM Troy Dawson <tdawson at redhat.com> wrote:
>
> On Thu, Aug 5, 2021 at 3:54 PM Nico Kadel-Garcia <nkadel at gmail.com> wrote:
>>
>> Because this kind of unwelcome stunt is brand new as best I can tell,
>> never pulled in 25 years as far as I can find.
>
> I understand the feeling, but let's take a deep breath.
>
> RHEL hasn't been around for 25 years, so you know you are exaggerating.

I did not say "RHEL". I said Red Hat. I date back to at least Red Hat
4.0, in 1996, so that's not exaggeration. That's precision. OK, 4.0
was published in October, so 24 years and 10 months. I *think* I
worked with a Red Hat 3.0.3, which was published in May, 1996, but I
can't swear when that box got to my hands.

> And yes, it has happened before, at least for Scientific Linux.

I did not list Scientific Linux. They are not the reference
distribution for Red Hat based distributions. Red Hat Linux was, and
now RHEL is. I've worked extensively with Scientific Linux and am
still active on the primary mailing list, which has quieted down a
great deal since CERN decided not to try to publish a Scientific Linux
8.

I'm not aware of any such instances for Scientific Linux since I first
looked at it with Scientific Linux 5.

> I definitely remember RHEL pushing some srpm's to their ftp area (back when they released srpm's), and then removing those srpm's.

Name them, please, because I don't remember that *ever* happening in
any Red Hat published operating system, whether Red Hat Linux or RHEL.
It happens quite casually in EPEL, which has turned out to be a big
problem for people who expect to be able to build a system from
scratch on Thursday that matches what they built on Tuesday.

> We (Scientific Linux) automatically pulled down those srpm's, built them, and pushed them out.
> We (Scientific Linux) were contacted once by Red Hat, another time by our own users, letting us know that we'd released a package not in RHEL, and we removed it from our repos.
> What happened?
> Real people, and a real world happened.
> Something happened in RHEL's workflow that was allowing srpms to be pushed.  This workflow was eventually fixed.
> We (Scientific Linux) were not properly testing our packages before pushing them out.  This workflow was eventually fixed.

I'm considerably less alarmed by Scientific Linux withdrawing an
accidentally published package. Moreover that doesn't sound like a
matching case, that sounds like a licensing issue where the package
had to be altogether withdrawn. What package was that? An Oracle JDK
package?

I'm alarmed about a number of factors of CentOS 8 Stream. I previously
expressed concern that CentOS 8 Stream, as a beta platform for RHEL,
will have packages published which never make it to RHEL, which is
confusing enough. But having them withdrawn from CentOS, because "it's
the same as the lower numbered build published for RHEL"? That means
updates are happening *without* making it into CentOS 8 Stream first,
and that there is no plan to match updates that were, in fact,
published and field tested in CentOS 8 Stream.

If Red Hat is not going to integrate the successful results of beta
testing, why are they bothering with CentOS 8 Stream? And why bother
to *delete* the package, and break existing depolyments.

The simplest working solution for this case would have been to accept
the higher numbered "beta-tested" build as canonical, and to update
RHEL to match the CentOS tested update, not to discard the higher
numbered build. This would have acknowledged the role of CentOS 8
Stream as a beta test environment for RHEL 8, and in this case as a
*successful* beta test environment, rather than an orphaned one as
occurred with this particular software package.

We also need to be careful about language. I've been very careful
about mine, referring to the particular build of freetype-devel as
"retired", and had someone try to say "retired" meant the software was
withdrawn. No, I spoke pretty specifically of the build, I don't see
how that seemed confusing. And it just happened above where you
accused me of exxageration, not realizing I had said and referred to
"Red Hat", not merely "RHEL". That one.... I won't take the blame for
that confusion: Red Hat's decision years ago to switch the names of
their published operating systems as "RHEL" and to revise their
numbering scheme was roughly as much fun as the SunOS and Solaris
naming confusion. When I say I learned a lesson with "Red Hat 4.2 when
it came out", folks underestimate my experience by a decade. We just
saw that happen.

> Again, deep breath.
> Please remember this (CentOS Stream) is a new workflow.
> There are real people, and a real world, behind it.
> There will be hickups and problems in that workflow.
> We (Red Hat) need feedback, so we know about those problems.  We appreciate that.
> Once we (Red Hat) know, and understand the scope of the problem, it's time to figure out how to solve that problem.
> We (Red Hat) also appreciate input on how to solve the problem.

Your note is very thoughtful. In this case, I urge Red Hat *not* to
casually delete RPMs from CentOS 8 Streams. Where possible, they
should be either accepted as a successfully tested package for RHEL,
or if the package failed, to revise and publish the fixed package in
the CentOS 8 Stream. I urge you to avoid what I think is a confusing
and difficult to validate belief, that a package with a different RPM
release number can be considered identical to another RPM, even if the
source code at compilation and the contents of all compiled libraries
are identical. We, as users of RPMs, can't verify that without a great
deal of extra work, especially when the relevant RPM and SRPM have
been expunged from the public repositories from which they were
previously deleted.

Expunging historical artifacts such as successfully built and used
RPMs should be undertaken only for *extraordinary* reasons, such as a
licensing violation or an extremely dangerous security issue.

> We (Red Hat) appreciate feedback, both on the problems, and the solutions.
> It's much easier for that feedback to be digested (understood and empathised) when things aren't exaggerated.
> It is also very useful when we find out "why" people are doing things.
> Using this example, it's helpful to know "why" they are trying to reinstall freetype.
> Is this a security audit that requires every package to be reinstalled?

That.... is a good question. One thing *I* do which is broken by this
kind of package withdrawal is to do a reference build, with a "yum
update" command, then don an "rpm -qa --qf
'%{name}-%{version}-%{release}.%{arch}.rpm]n' | LANG=C sort -n",
precisely to get a full list of every element of my build environment
and to be able to replicate it precisely, particularly when building
CentOS versus RHEL, or a building a docker or hardware mock or other
working environment.

EPEL makes this much more difficult with their frequent withdrawals of
previously available builds of packages, and they compel me to
maintain an internal mirror which aggregates only, it never prunes.
I'm going to *hate* to have to do this for CentOS 8 Stream repos.

> Is this part of someone's QA that requires every package to be reinstalled?
> Did you accidentally remove a file and need to re-install the package.

It's especially likely to be needed if someone screws up a config
file. I myself went through conniptions recently because someone was
editing "/etc/init.d/jenkins" rather than "/etc/sysconfig/jenkins",
and Jenkins updates weren't getting applied ot the init script. It
took some adventures with deleting the modified init script and
re-installing the package to get a clean copy with the right
timestamps, to allow future updates to work. Not that is a Red Hat
published config file, but it's the kind of issue that could compel a
re-install.

In this case, the withdrawal of the most recent "freetype" builds
would also break "yum install freetype-devel" or anything that depends
on freetype-devel.

> Knowing the "why" helps us (Red Hat) understand the priority and scope.
>
> Troy
>
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel