On 04.08.21 03:33, Nico Kadel-Garcia wrote:

On Tue, Aug 3, 2021 at 7:49 PM Leon Fauster via CentOS-devel centos-devel@centos.org wrote:

...

I wonder why this package not available anymore?

# LANG=C yum reinstall freetype Last metadata expiration check: 1:04:59 ago on Wed Aug 4 00:38:21 2021. Installed package freetype-2.9.1-5.el8.x86_64 (from baseos) not available. Error: No packages marked for reinstall.

Any clues?

Because the CentOS 8.4 package is freetype-2.9.1-4, according to the local mirrors. Did you perhaps install that from a CentOS 8 Stream repo?

Here my context: I am comparing two nodes based on CS8 (Centos 8 Stream ). One have

freetype-2.9.1-5.el8.x86_64 and the other have freetype-2.9.1-4.el8_3.1.x86_64

The mirror http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ shows freetype-2.9.1-4.el8_3.1.x86_64 has the latest.

I wonder where this version 2.9.1-5 is coming from? The node was regularly installed with C8 and then swapped to CS8 ...

# LANG=C rpm -q freetype -i |head -17 Name : freetype Version : 2.9.1 Release : 5.el8 Architecture: x86_64 Install Date: Thu Mar 25 23:54:00 2021 Group : System Environment/Libraries Size : 811895 License : (FTL or GPLv2+) and BSD and MIT and Public Domain and zlib with acknowledgement Signature : RSA/SHA256, Wed Dec 2 16:11:08 2020, Key ID 05b555b38483c65d Source RPM : freetype-2.9.1-5.el8.src.rpm Build Date : Mon Nov 16 22:58:59 2020 Build Host : x86-02.mbox.centos.org Relocations : (not relocatable) Packager : CentOS Buildsys bugs@centos.org Vendor : CentOS URL : http://www.freetype.org Summary : A free and portable font rendering engine

I see it here

https://koji.mbox.centos.org/koji/packageinfo?packageID=408

but not on the mirrors ...

A retired package?

-- Leon

On 04.08.21 14:00, Josh Boyer wrote:

On Wed, Aug 4, 2021 at 6:14 AM Leon Fauster via CentOS-devel centos-devel@centos.org wrote:

...

<snip>

...

Here my context: I am comparing two nodes based on CS8 (Centos 8 Stream ). One have

freetype-2.9.1-5.el8.x86_64 and the other have freetype-2.9.1-4.el8_3.1.x86_64

At one point in time during RHEL 8.4 development, freetype-2.9.1-5.el8 was set to be shipped. However, it only fixed a CVE and that CVE was already fixed by the freetype-2.9.1-4.el8_3.1 that as shipped as part of a batch update. There was no reason to ship a build that didn't do anything, so it was dropped on the RHEL side.

My educated guess is that Stream 8 picked up the -5.el8 build during the course of RHEL 8.4 development as expected, and then when it was dropped on the RHEL side it used the -4.el8_3.1 update because that is indeed the latest available even today.

This is one of the unintended consequences of how Stream 8 is produced.

Thanks for the explanation. I did not though that such activity would come so much to the front and produce a installable artifact. But it looks like that such dropped rpms do not have a serious impact (at least this one).

...

The mirror http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ shows freetype-2.9.1-4.el8_3.1.x86_64 has the latest.

I wonder where this version 2.9.1-5 is coming from? The node was regularly installed with C8 and then swapped to CS8 ...

<snip>

...

I see it here

https://koji.mbox.centos.org/koji/packageinfo?packageID=408

but not on the mirrors ...

A retired package?

Not retired, just a build that will never be shipped at this point.

I will incorporate this insight into our plausibility checks ... Thanks.

-- Leon

On 8/4/21 5:21 PM, Nico Kadel-Garcia wrote:

On Wed, Aug 4, 2021 at 8:01 AM Josh Boyer jwboyer@redhat.com wrote:

...

On Wed, Aug 4, 2021 at 6:14 AM Leon Fauster via CentOS-devel centos-devel@centos.org wrote:

...

On 04.08.21 03:33, Nico Kadel-Garcia wrote:

...

On Tue, Aug 3, 2021 at 7:49 PM Leon Fauster via CentOS-devel centos-devel@centos.org wrote:

...

I wonder why this package not available anymore?

# LANG=C yum reinstall freetype Last metadata expiration check: 1:04:59 ago on Wed Aug 4 00:38:21 2021. Installed package freetype-2.9.1-5.el8.x86_64 (from baseos) not available. Error: No packages marked for reinstall.

Any clues?

Because the CentOS 8.4 package is freetype-2.9.1-4, according to the local mirrors. Did you perhaps install that from a CentOS 8 Stream repo?

Here my context: I am comparing two nodes based on CS8 (Centos 8 Stream ). One have

freetype-2.9.1-5.el8.x86_64 and the other have freetype-2.9.1-4.el8_3.1.x86_64

At one point in time during RHEL 8.4 development, freetype-2.9.1-5.el8 was set to be shipped. However, it only fixed a CVE and that CVE was already fixed by the freetype-2.9.1-4.el8_3.1 that as shipped as part of a batch update. There was no reason to ship a build that didn't do anything, so it was dropped on the RHEL side.

This kind of behavior is a powerful reminder of one of the problems of CentOS 8 Stream. Has there *ever* been a CentOS package published and then yanked back from the public repos before? I can't think of any, and I'm afraid it's likely to recur as other beta packages are tested and dropped without notice.

The "simple" solution is the same as that for the frequently ephemeral packages in EPEL: maintain, and deploy only from, designated internal snapshots for continuing access to discarded packages used in your clusters or consistently deployed working environments. It's a pain in the keister, and not even addressable by spacewalk unless spacewalk is set to maintain its own, purely aggregated and never pruned internal mirrors.

You don't think doing CentOS Stream 8 is a pain in the keister?

I just want to point out that there a lot people doing a lot of hard work and a bunch of fighting to make the pieces of stream that are still available publicly available.

We have put ourselves and our careers on the line to get every possible concession we can for the community in CentOS Stream 8. We fight for this at every opportunity.

An example was making the koji system available publicly so you can get ANYTHING we have ever built. You can get this package (or anything else ever built) there.

I get it .. you don't like Stream. OK, then don't use it. But the people who are maintaining Stream 8 are doing the best that we can. It might not be enough for you to want to continue to use Stream. It is what we can do, however.

On Wed, Aug 4, 2021 at 7:01 AM Josh Boyer jwboyer@redhat.com wrote:

On Wed, Aug 4, 2021 at 6:14 AM Leon Fauster via CentOS-devel centos-devel@centos.org wrote:

...

On 04.08.21 03:33, Nico Kadel-Garcia wrote:

...

On Tue, Aug 3, 2021 at 7:49 PM Leon Fauster via CentOS-devel centos-devel@centos.org wrote:

...

I wonder why this package not available anymore?

# LANG=C yum reinstall freetype Last metadata expiration check: 1:04:59 ago on Wed Aug 4 00:38:21 2021. Installed package freetype-2.9.1-5.el8.x86_64 (from baseos) not available. Error: No packages marked for reinstall.

Any clues?

Because the CentOS 8.4 package is freetype-2.9.1-4, according to the local mirrors. Did you perhaps install that from a CentOS 8 Stream repo?

Here my context: I am comparing two nodes based on CS8 (Centos 8 Stream ). One have

freetype-2.9.1-5.el8.x86_64 and the other have freetype-2.9.1-4.el8_3.1.x86_64

At one point in time during RHEL 8.4 development, freetype-2.9.1-5.el8 was set to be shipped. However, it only fixed a CVE and that CVE was already fixed by the freetype-2.9.1-4.el8_3.1 that as shipped as part of a batch update. There was no reason to ship a build that didn't do anything, so it was dropped on the RHEL side.

My educated guess is that Stream 8 picked up the -5.el8 build during the course of RHEL 8.4 development as expected, and then when it was dropped on the RHEL side it used the -4.el8_3.1 update because that is indeed the latest available even today.

This is one of the unintended consequences of how Stream 8 is produced.

This guess is exactly correct. 2.9.1-5 is identical to 2.9.1-4.el8_3.1, which is the only reason we felt it was appropriate to drop the former build entirely. If/when this package is updated again in the 8 lifecycle, I expect the release will be at least -6, providing the correct upgrade path regardless of which release a user currently has installed.

...

The mirror http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/ shows freetype-2.9.1-4.el8_3.1.x86_64 has the latest.

I wonder where this version 2.9.1-5 is coming from? The node was regularly installed with C8 and then swapped to CS8 ...

# LANG=C rpm -q freetype -i |head -17 Name : freetype Version : 2.9.1 Release : 5.el8 Architecture: x86_64 Install Date: Thu Mar 25 23:54:00 2021 Group : System Environment/Libraries Size : 811895 License : (FTL or GPLv2+) and BSD and MIT and Public Domain and zlib with acknowledgement Signature : RSA/SHA256, Wed Dec 2 16:11:08 2020, Key ID 05b555b38483c65d Source RPM : freetype-2.9.1-5.el8.src.rpm Build Date : Mon Nov 16 22:58:59 2020 Build Host : x86-02.mbox.centos.org Relocations : (not relocatable) Packager : CentOS Buildsys bugs@centos.org Vendor : CentOS URL : http://www.freetype.org Summary : A free and portable font rendering engine

I see it here

https://koji.mbox.centos.org/koji/packageinfo?packageID=408

but not on the mirrors ...

A retired package?

Not retired, just a build that will never be shipped at this point.

josh

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

On Thu, Aug 5, 2021 at 10:51 AM Nico Kadel-Garcia nkadel@gmail.com wrote:

On Wed, Aug 4, 2021 at 9:03 PM Carl George carl@redhat.com wrote:

...

On Wed, Aug 4, 2021 at 7:01 AM Josh Boyer jwboyer@redhat.com wrote:

...

...

...

I see it here

https://koji.mbox.centos.org/koji/packageinfo?packageID=408

but not on the mirrors ...

A retired package?

Not retired, just a build that will never be shipped at this point.

josh

Since it was published at one time, and has now been deleted from the public repos with extreme prejudice, in what way is this not "retired"?

Perhaps we have a simple lexicon issue. Retired to me means the package (SRPM) has been removed from the distribution entirely. In this specific instance, that is not the case. We have a build of a package that was dropped, but the package and all of the functionality previously provided by it is still available with the current build.

I admit this is a first. I've never seen Red Hat or CentOS pull this stunt. Has this *ever* been done before?

I'm not sure I would call it a stunt. It's certainly not ideal and we've explained how it happened already.

To answer your question, in exceedingly rare cases we have removed builds in the past, even from RHEL CDN. I have no idea if that was reflected in CentOS Linux or not.

josh

On Thu, Aug 5, 2021 at 11:08 AM Carl George carl@redhat.com wrote:

On Thu, Aug 5, 2021 at 9:51 AM Nico Kadel-Garcia nkadel@gmail.com wrote:

...

On Wed, Aug 4, 2021 at 9:03 PM Carl George carl@redhat.com wrote:

...

On Wed, Aug 4, 2021 at 7:01 AM Josh Boyer jwboyer@redhat.com wrote:

...

...

...

I see it here

https://koji.mbox.centos.org/koji/packageinfo?packageID=408

but not on the mirrors ...

A retired package?

Not retired, just a build that will never be shipped at this point.

josh

Since it was published at one time, and has now been deleted from the public repos with extreme prejudice, in what way is this not "retired"?

I admit this is a first. I've never seen Red Hat or CentOS pull this stunt. Has this *ever* been done before? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

Retired would be if the package was removed entirely. `dnf install freetype` still works, therefore the package is not retired.

"dnf install freetype-2.9.1-5" does not. Neither does "dnf reinstall freetype" for this previously deployed host. He'd have to do "dnf downgrade freetype" or "rpm -U --oldpackage" for freetype, freetype-devel, and any other RPM's built from the same SRPM.

The 2.9.1-5 build is identical to the 2.9.1-4.el8_3.1 build. We removed the duplicate (which follows what happened in the RHEL nightly compose). Why is this so upsetting to you?

Because this kind of unwelcome stunt is brand new as best I can tell, never pulled in 25 years as far as I can find. And whether this particular dropped package is identical to an inconsistently numbered release is no guarantee that next week, it won't be a testing gcc or glibc version, tested in what is now RHEL 8 beta and discarded for no predictable or published reason.

It's also unlikely in the extreme that they're identical. Even if the checksums of compiled software match, datestamps on the compiled libraries would not. That's not likely to make a big deal, but it's not "identical".

On Thu, Aug 5, 2021 at 7:32 PM Brian Stinson brian@bstinson.com wrote:

On Thu, Aug 5, 2021, at 17:54, Nico Kadel-Garcia wrote:

...

On Thu, Aug 5, 2021 at 11:08 AM Carl George carl@redhat.com wrote:

...

On Thu, Aug 5, 2021 at 9:51 AM Nico Kadel-Garcia nkadel@gmail.com wrote:

...

On Wed, Aug 4, 2021 at 9:03 PM Carl George carl@redhat.com wrote:

...

On Wed, Aug 4, 2021 at 7:01 AM Josh Boyer jwboyer@redhat.com wrote:

...

...

> I see it here > > https://koji.mbox.centos.org/koji/packageinfo?packageID=408 > > but not on the mirrors ... > > A retired package?

Not retired, just a build that will never be shipped at this point.

josh

Since it was published at one time, and has now been deleted from the public repos with extreme prejudice, in what way is this not "retired"?

I admit this is a first. I've never seen Red Hat or CentOS pull this stunt. Has this *ever* been done before? _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

Retired would be if the package was removed entirely. `dnf install freetype` still works, therefore the package is not retired.

"dnf install freetype-2.9.1-5" does not. Neither does "dnf reinstall freetype" for this previously deployed host. He'd have to do "dnf downgrade freetype" or "rpm -U --oldpackage" for freetype, freetype-devel, and any other RPM's built from the same SRPM.

dnf distro-sync may be helpful here too

...

...

The 2.9.1-5 build is identical to the 2.9.1-4.el8_3.1 build. We removed the duplicate (which follows what happened in the RHEL nightly compose). Why is this so upsetting to you?

Because this kind of unwelcome stunt is brand new as best I can tell, never pulled in 25 years as far as I can find. And whether this particular dropped package is identical to an inconsistently numbered release is no guarantee that next week, it won't be a testing gcc or glibc version, tested in what is now RHEL 8 beta and discarded for no predictable or published reason.

CentOS Stream as presented on the mirrors is meant to reflect what’s going into the next minor version of RHEL, so if RHEL decides not to ship a build we need to reflect that. The builds in the buildsystem (which you already found) will stick around if you’re interested in specific versions. I think the Stream 9 workflows may help with some of the visibility here as we roll them out.

...

It's also unlikely in the extreme that they're identical. Even if the checksums of compiled software match, datestamps on the compiled libraries would not. That's not likely to make a big deal, but it's not "identical".

I’m curious about this, what is important about matching library datestamps?

Ordinarily? Nothing. But libraries are not the only files in RPMs. Also even when the source code is identical, libraries can differ due to their compilation. That's not likely to be significantly different in a mock or koji build environment, but it *can*, especially due to small changes in gcc or glibc. And there's that word, "likely". And worse, some source control systems are prone to programmers activating datestamp, author or commit references in text fields, and some programmers use those sorts of source control generated data for "version" or comment fields. This used to be very popular indeed with Subversion source control, it wasn't well supported at first with git and remains much less popular. Some programmers even generated their version.h file with every compilation: From memory, that was part of the bootstrapping procedure of compiling gcc.

I'm not saying this is likely with freetype, or likely to break things on a specific project, people are usually sensible about its use. But if the git repos have such features enabled for specific configuration files, well, checksum consistency of compiled software is unlikely. with that software, especially if it gets embedded in a "version.h" file.

On 06.08.21 15:29, Troy Dawson wrote:

It is also very useful when we find out "why" people are doing things. Using this example, it's helpful to know "why" they are trying to reinstall freetype. Is this a security audit that requires every package to be reinstalled? Is this part of someone's QA that requires every package to be reinstalled? Did you accidentally remove a file and need to re-install the package. Knowing the "why" helps us (Red Hat) understand the priority and scope.

Its a QA process that identified that a package would be downgraded, if distro-sync would be done. Downgrades are classified as security issue, therefore a manually interaction was done. The reinstall activity was done by an operator to verify that the current installed pkg is really not in the repos anymore ...

So IMHO, for the sake of "high level" processes out there. Leaving the already published rpm in the repos would hurt less then removing it.

$ rpm -q --qf '%{SIZE}\n' freetype 811871

-- Leon

On Fri, Aug 6, 2021 at 9:30 AM Troy Dawson tdawson@redhat.com wrote:

On Thu, Aug 5, 2021 at 3:54 PM Nico Kadel-Garcia nkadel@gmail.com wrote:

...

Because this kind of unwelcome stunt is brand new as best I can tell, never pulled in 25 years as far as I can find.

I understand the feeling, but let's take a deep breath.

RHEL hasn't been around for 25 years, so you know you are exaggerating.

I did not say "RHEL". I said Red Hat. I date back to at least Red Hat 4.0, in 1996, so that's not exaggeration. That's precision. OK, 4.0 was published in October, so 24 years and 10 months. I *think* I worked with a Red Hat 3.0.3, which was published in May, 1996, but I can't swear when that box got to my hands.

And yes, it has happened before, at least for Scientific Linux.

I did not list Scientific Linux. They are not the reference distribution for Red Hat based distributions. Red Hat Linux was, and now RHEL is. I've worked extensively with Scientific Linux and am still active on the primary mailing list, which has quieted down a great deal since CERN decided not to try to publish a Scientific Linux 8.

I'm not aware of any such instances for Scientific Linux since I first looked at it with Scientific Linux 5.

I definitely remember RHEL pushing some srpm's to their ftp area (back when they released srpm's), and then removing those srpm's.

Name them, please, because I don't remember that *ever* happening in any Red Hat published operating system, whether Red Hat Linux or RHEL. It happens quite casually in EPEL, which has turned out to be a big problem for people who expect to be able to build a system from scratch on Thursday that matches what they built on Tuesday.

We (Scientific Linux) automatically pulled down those srpm's, built them, and pushed them out. We (Scientific Linux) were contacted once by Red Hat, another time by our own users, letting us know that we'd released a package not in RHEL, and we removed it from our repos. What happened? Real people, and a real world happened. Something happened in RHEL's workflow that was allowing srpms to be pushed. This workflow was eventually fixed. We (Scientific Linux) were not properly testing our packages before pushing them out. This workflow was eventually fixed.

I'm considerably less alarmed by Scientific Linux withdrawing an accidentally published package. Moreover that doesn't sound like a matching case, that sounds like a licensing issue where the package had to be altogether withdrawn. What package was that? An Oracle JDK package?

I'm alarmed about a number of factors of CentOS 8 Stream. I previously expressed concern that CentOS 8 Stream, as a beta platform for RHEL, will have packages published which never make it to RHEL, which is confusing enough. But having them withdrawn from CentOS, because "it's the same as the lower numbered build published for RHEL"? That means updates are happening *without* making it into CentOS 8 Stream first, and that there is no plan to match updates that were, in fact, published and field tested in CentOS 8 Stream.

If Red Hat is not going to integrate the successful results of beta testing, why are they bothering with CentOS 8 Stream? And why bother to *delete* the package, and break existing depolyments.

The simplest working solution for this case would have been to accept the higher numbered "beta-tested" build as canonical, and to update RHEL to match the CentOS tested update, not to discard the higher numbered build. This would have acknowledged the role of CentOS 8 Stream as a beta test environment for RHEL 8, and in this case as a *successful* beta test environment, rather than an orphaned one as occurred with this particular software package.

We also need to be careful about language. I've been very careful about mine, referring to the particular build of freetype-devel as "retired", and had someone try to say "retired" meant the software was withdrawn. No, I spoke pretty specifically of the build, I don't see how that seemed confusing. And it just happened above where you accused me of exxageration, not realizing I had said and referred to "Red Hat", not merely "RHEL". That one.... I won't take the blame for that confusion: Red Hat's decision years ago to switch the names of their published operating systems as "RHEL" and to revise their numbering scheme was roughly as much fun as the SunOS and Solaris naming confusion. When I say I learned a lesson with "Red Hat 4.2 when it came out", folks underestimate my experience by a decade. We just saw that happen.

Again, deep breath. Please remember this (CentOS Stream) is a new workflow. There are real people, and a real world, behind it. There will be hickups and problems in that workflow. We (Red Hat) need feedback, so we know about those problems. We appreciate that. Once we (Red Hat) know, and understand the scope of the problem, it's time to figure out how to solve that problem. We (Red Hat) also appreciate input on how to solve the problem.

Your note is very thoughtful. In this case, I urge Red Hat *not* to casually delete RPMs from CentOS 8 Streams. Where possible, they should be either accepted as a successfully tested package for RHEL, or if the package failed, to revise and publish the fixed package in the CentOS 8 Stream. I urge you to avoid what I think is a confusing and difficult to validate belief, that a package with a different RPM release number can be considered identical to another RPM, even if the source code at compilation and the contents of all compiled libraries are identical. We, as users of RPMs, can't verify that without a great deal of extra work, especially when the relevant RPM and SRPM have been expunged from the public repositories from which they were previously deleted.

Expunging historical artifacts such as successfully built and used RPMs should be undertaken only for *extraordinary* reasons, such as a licensing violation or an extremely dangerous security issue.

We (Red Hat) appreciate feedback, both on the problems, and the solutions. It's much easier for that feedback to be digested (understood and empathised) when things aren't exaggerated. It is also very useful when we find out "why" people are doing things. Using this example, it's helpful to know "why" they are trying to reinstall freetype. Is this a security audit that requires every package to be reinstalled?

That.... is a good question. One thing *I* do which is broken by this kind of package withdrawal is to do a reference build, with a "yum update" command, then don an "rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}.rpm]n' | LANG=C sort -n", precisely to get a full list of every element of my build environment and to be able to replicate it precisely, particularly when building CentOS versus RHEL, or a building a docker or hardware mock or other working environment.

EPEL makes this much more difficult with their frequent withdrawals of previously available builds of packages, and they compel me to maintain an internal mirror which aggregates only, it never prunes. I'm going to *hate* to have to do this for CentOS 8 Stream repos.

Is this part of someone's QA that requires every package to be reinstalled? Did you accidentally remove a file and need to re-install the package.

It's especially likely to be needed if someone screws up a config file. I myself went through conniptions recently because someone was editing "/etc/init.d/jenkins" rather than "/etc/sysconfig/jenkins", and Jenkins updates weren't getting applied ot the init script. It took some adventures with deleting the modified init script and re-installing the package to get a clean copy with the right timestamps, to allow future updates to work. Not that is a Red Hat published config file, but it's the kind of issue that could compel a re-install.

In this case, the withdrawal of the most recent "freetype" builds would also break "yum install freetype-devel" or anything that depends on freetype-devel.

Knowing the "why" helps us (Red Hat) understand the priority and scope.

Troy

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

On Sat, Aug 7, 2021 at 7:52 AM Neal Gompa ngompa13@gmail.com wrote:

While I wish this was how it is handled with Red Hat Enterprise Linux (as it is in Fedora for technical reasons), Red Hat has yanked packages from RHEL over the years, even in post-release updates. Sometimes because they were accidentally pushed, sometimes because there was no change (that is what freetype falls into), sometimes because they broke things badly (first round of BootHole fixes were like that, I think?), and so on. I'm aware of at least one case where content was pushed out because it was too late and the documentation was updated to indicate to not rely on it because it was going to disappear in the next point release. Though it is rare, content does disappear from RHEL across minor releases too.

First, *thank you* for using the phrase "point release". That came up in a previous thread, with a very strange claim that RHEL and CentOS don't do point releases.

Second, what does "pushed out" mean here? Pushed out to the repos, or pushed out to a future publication? I'm afraid the word is ambiguous. And "discontinued" is not the same process as "yanked out of the published repos".

Can you name any specific RPM that got deleted from the public repos from Red Hat Linux, RHEL, or CentOS? I'd really like to see or hear of a specific example, not a "we've done that!". EPEL, yeah, they do that and it's crazy making when trying to synchronize test environments. I'd really prefer not to have to tune CentOS mirror tools to aggregate RPMs only, rather than mirroring all content including "repodata". That will mean generating repodata on the fly in my local repos and slowing the publication of an updated repo. I'd anticipated a need to do some of that to have stabilized internal repos, but don't want to have to generate repodata as well to catch references to such "discarded" RPMs.

On Sun, Aug 8, 2021 at 7:45 PM Carl George carl@redhat.com wrote:

On Sat, Aug 7, 2021 at 6:32 AM Nico Kadel-Garcia nkadel@gmail.com wrote:

...

On Fri, Aug 6, 2021 at 9:30 AM Troy Dawson tdawson@redhat.com wrote:

...

On Thu, Aug 5, 2021 at 3:54 PM Nico Kadel-Garcia nkadel@gmail.com wrote:

...

Because this kind of unwelcome stunt is brand new as best I can tell, never pulled in 25 years as far as I can find.

I understand the feeling, but let's take a deep breath.

RHEL hasn't been around for 25 years, so you know you are exaggerating.

I did not say "RHEL". I said Red Hat. I date back to at least Red Hat 4.0, in 1996, so that's not exaggeration. That's precision. OK, 4.0 was published in October, so 24 years and 10 months. I *think* I worked with a Red Hat 3.0.3, which was published in May, 1996, but I can't swear when that box got to my hands.

...

And yes, it has happened before, at least for Scientific Linux.

I did not list Scientific Linux. They are not the reference distribution for Red Hat based distributions. Red Hat Linux was, and now RHEL is. I've worked extensively with Scientific Linux and am still active on the primary mailing list, which has quieted down a great deal since CERN decided not to try to publish a Scientific Linux 8.

I'm not aware of any such instances for Scientific Linux since I first looked at it with Scientific Linux 5.

...

I definitely remember RHEL pushing some srpm's to their ftp area (back when they released srpm's), and then removing those srpm's.

Name them, please, because I don't remember that *ever* happening in any Red Hat published operating system, whether Red Hat Linux or RHEL. It happens quite casually in EPEL, which has turned out to be a big problem for people who expect to be able to build a system from scratch on Thursday that matches what they built on Tuesday.

...

We (Scientific Linux) automatically pulled down those srpm's, built them, and pushed them out. We (Scientific Linux) were contacted once by Red Hat, another time by our own users, letting us know that we'd released a package not in RHEL, and we removed it from our repos. What happened? Real people, and a real world happened. Something happened in RHEL's workflow that was allowing srpms to be pushed. This workflow was eventually fixed. We (Scientific Linux) were not properly testing our packages before pushing them out. This workflow was eventually fixed.

I'm considerably less alarmed by Scientific Linux withdrawing an accidentally published package. Moreover that doesn't sound like a matching case, that sounds like a licensing issue where the package had to be altogether withdrawn. What package was that? An Oracle JDK package?

I'm alarmed about a number of factors of CentOS 8 Stream. I previously expressed concern that CentOS 8 Stream, as a beta platform for RHEL, will have packages published which never make it to RHEL, which is confusing enough. But having them withdrawn from CentOS, because "it's the same as the lower numbered build published for RHEL"? That means updates are happening *without* making it into CentOS 8 Stream first, and that there is no plan to match updates that were, in fact, published and field tested in CentOS 8 Stream.

If Red Hat is not going to integrate the successful results of beta testing, why are they bothering with CentOS 8 Stream? And why bother to *delete* the package, and break existing depolyments.

The simplest working solution for this case would have been to accept the higher numbered "beta-tested" build as canonical, and to update RHEL to match the CentOS tested update, not to discard the higher numbered build. This would have acknowledged the role of CentOS 8 Stream as a beta test environment for RHEL 8, and in this case as a *successful* beta test environment, rather than an orphaned one as occurred with this particular software package.

We also need to be careful about language. I've been very careful about mine, referring to the particular build of freetype-devel as "retired", and had someone try to say "retired" meant the software was withdrawn. No, I spoke pretty specifically of the build, I don't see how that seemed confusing. And it just happened above where you accused me of exxageration, not realizing I had said and referred to "Red Hat", not merely "RHEL". That one.... I won't take the blame for that confusion: Red Hat's decision years ago to switch the names of their published operating systems as "RHEL" and to revise their numbering scheme was roughly as much fun as the SunOS and Solaris naming confusion. When I say I learned a lesson with "Red Hat 4.2 when it came out", folks underestimate my experience by a decade. We just saw that happen.

...

Again, deep breath. Please remember this (CentOS Stream) is a new workflow. There are real people, and a real world, behind it. There will be hickups and problems in that workflow. We (Red Hat) need feedback, so we know about those problems. We appreciate that. Once we (Red Hat) know, and understand the scope of the problem, it's time to figure out how to solve that problem. We (Red Hat) also appreciate input on how to solve the problem.

Your note is very thoughtful. In this case, I urge Red Hat *not* to casually delete RPMs from CentOS 8 Streams. Where possible, they should be either accepted as a successfully tested package for RHEL, or if the package failed, to revise and publish the fixed package in the CentOS 8 Stream. I urge you to avoid what I think is a confusing and difficult to validate belief, that a package with a different RPM release number can be considered identical to another RPM, even if the source code at compilation and the contents of all compiled libraries are identical. We, as users of RPMs, can't verify that without a great deal of extra work, especially when the relevant RPM and SRPM have been expunged from the public repositories from which they were previously deleted.

You can verify it in much less time than you've spent writing antagonistic emails on this thread.

https://git.centos.org/rpms/freetype/c/19e0ad98e6e7570372b3601ecf949b023c603...

It's also intellectually dishonest to use the word expunged to describe this. Expunged means to remove completely. The duplicate build and it's artifacts are still available in our build system.

https://koji.mbox.centos.org/koji/buildinfo?buildID=14760

In this case, that's not the public repo by which the package was originally distributed nor is it directly enabled on any CentOS system. And if you'll check, I've been really consistent about saying "expunged from the public repos". Is that admittedly public resource a repo? Without associated repodata or yum access to it, I would not call it one, especially because the word "repo" has a pretty specific meaning for RPM based operating systems.

I've been accused of things a few times in this thread, several times now by someone confused by what I thought was clear language. Can we agree that this is such an occasion? I thought "expunged from the public repos" was intelligible, and mild by comparison to, "yoinked for no apparent reason from every public yum repo", which is more precise but a bit ruder. Hopefully after these threads it's clear that such deletions cause confusion and break working configuration management tools.