Hi,
We're working on validating CentOS 8 for some desktop use cases at work, and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Turns out that we need libzstd, which on the previous machine was sourced from the EPEL repo, but the epel8 package got retired 3 months ago because the package is now in RHEL's BaseOS:
https://src.fedoraproject.org/rpms/zstd/history/dead.package?identifier=epel...
The package is only in BaseOS in 8.2 though, and CentOS 8.2 is not out -- the only repo that has it now is 8-stream:
https://mirrors.edge.kernel.org/centos/8-stream/BaseOS/x86_64/os/Packages/
8.1.1911 does not have the package: https://mirrors.edge.kernel.org/centos/8.1.1911/BaseOS/x86_64/os/Packages/
Also, the version in BaseOS (if 8-stream is up to date) is 1.4.2, which is older than the last version in EPEL (1.4.4).
Is anyone else in the same situation, and how do you work around it? Since EPEL is a sort of "rolling release" does it make sense to just track 8-stream if you're using it, or are people resorting to hosting these key packages in internal repos?
Thanks,
Sent from Michel Alexandre Salim on Thursday, 14 May 2020 21:32 To: centos-devel@centos.org; epel-devel@lists.fedoraproject.org Hi,
We're working on validating CentOS 8 for some desktop use cases at work, and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Turns out that we need libzstd, which on the previous machine was sourced from the EPEL repo, but the epel8 package got retired 3 months ago because the package is now in RHEL's BaseOS:
https://src.fedoraproject.org/rpms/zstd/history/dead.package?identifier=e pel8
The package is only in BaseOS in 8.2 though, and CentOS 8.2 is not out -- the only repo that has it now is 8-stream:
https://mirrors.edge.kernel.org/centos/8- stream/BaseOS/x86_64/os/Packages/
8.1.1911 does not have the package: https://mirrors.edge.kernel.org/centos/8.1.1911/BaseOS/x86_64/os/Packag es/
Also, the version in BaseOS (if 8-stream is up to date) is 1.4.2, which is older than the last version in EPEL (1.4.4).
Is anyone else in the same situation, and how do you work around it? Since EPEL is a sort of "rolling release" does it make sense to just track 8- stream if you're using it, or are people resorting to hosting these key packages in internal repos?
Same situation, needing package versions that are no longer in the official repos.
I resorted to internally running a limited archiving mirror for needed repos, including EPEL8. This mirror does not delete any packages, to ensure reproducibility of processes that require specific package versions. It is not public, but if you need x86_64 binary or Source packages for libzstd 1.4.4 from EPEL8, I can provide them.
Thanks,
-- Michel Alexandre Salim profile: https://keybase.io/michel_slm chat via email: https://delta.chat/ GPG key: 96A7 A6ED FB4D 2113 4056 3257 CAF9 AD10 ACB1 BEF2
Regards, Antal
On Thu, May 14, 2020 at 12:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
We're working on validating CentOS 8 for some desktop use cases at work, and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Turns out that we need libzstd, which on the previous machine was sourced from the EPEL repo, but the epel8 package got retired 3 months ago because the package is now in RHEL's BaseOS:
https://src.fedoraproject.org/rpms/zstd/history/dead.package?identifier=epel...
The package is only in BaseOS in 8.2 though, and CentOS 8.2 is not out -- the only repo that has it now is 8-stream:
https://mirrors.edge.kernel.org/centos/8-stream/BaseOS/x86_64/os/Packages/
8.1.1911 does not have the package: https://mirrors.edge.kernel.org/centos/8.1.1911/BaseOS/x86_64/os/Packages/
Also, the version in BaseOS (if 8-stream is up to date) is 1.4.2, which is older than the last version in EPEL (1.4.4).
Is anyone else in the same situation, and how do you work around it? Since EPEL is a sort of "rolling release" does it make sense to just track 8-stream if you're using it, or are people resorting to hosting these key packages in internal repos?
Thanks,
There is the EPEL archives. This is fairly recent, but we did a snapshot of 8.1 a bit before 8.2 came out. We plan on doing this for each minor release (7 and 8). Fedora Archives are not on all the mirrors, so it might be a little slower, depending on where you get it from, but at least it's there. https://archives.fedoraproject.org/pub/archive/epel/8.1/
Troy
On 5/14/20 1:03 PM, Troy Dawson wrote:
On Thu, May 14, 2020 at 12:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
https://src.fedoraproject.org/rpms/zstd/history/dead.package?identifier=epel...
The package is only in BaseOS in 8.2 though, and CentOS 8.2 is not out -- the only repo that has it now is 8-stream:
https://mirrors.edge.kernel.org/centos/8-stream/BaseOS/x86_64/os/Packages/
8.1.1911 does not have the package: https://mirrors.edge.kernel.org/centos/8.1.1911/BaseOS/x86_64/os/Packages/
There is the EPEL archives. This is fairly recent, but we did a snapshot of 8.1 a bit before 8.2 came out. We plan on doing this for each minor release (7 and 8). Fedora Archives are not on all the mirrors, so it might be a little slower, depending on where you get it from, but at least it's there. https://archives.fedoraproject.org/pub/archive/epel/8.1/
Oh, that's an option, thanks. the archives site tends not to be mirrored as widely though, so if we want to use it we might end up mirroring it internally.
Per what Paul said (using Stream) I suppose we could also enable the Stream repo but set it at a lower priority so we only use packages that are not found elsewhere.
Thanks,
This should normally be an edge case that only occurs during the time between a RHEL point release and the corresponding CentOS Linux rebuild. Historically that usually takes about a month. As more and more content is built in CentOS Stream we hope to be able to reduce that. In the case of zstd, the EPEL maintainer jumped the gun and retired the package two months before the RHEL 8.2 release (despite me asking to retire it at 8.2 GA), so there was a longer gap than necessary.
https://bugzilla.redhat.com/show_bug.cgi?id=1806759
Regarding using CentOS Stream for your use case, be aware that EPEL is built against RHEL, so it's possible to run into issues where an EPEL package doesn't work on CentOS Stream. This is a known issue that we haven't come to a conclusion yet on how to handle. On a related note, EPEL faces similar challenges when RHEL bumps a library soname. The maintainer has to choose between updating the package to be compatible with RHEL and breaking installation on CentOS, or leaving RHEL installations broken until CentOS catches up. Anecdotally I think most maintainers choose the latter. It's not an exact science.
On Thu, May 14, 2020 at 6:23 PM Michel Alexandre Salim michel@michel-slm.name wrote:
On 5/14/20 1:03 PM, Troy Dawson wrote:
On Thu, May 14, 2020 at 12:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
https://src.fedoraproject.org/rpms/zstd/history/dead.package?identifier=epel...
The package is only in BaseOS in 8.2 though, and CentOS 8.2 is not out -- the only repo that has it now is 8-stream:
https://mirrors.edge.kernel.org/centos/8-stream/BaseOS/x86_64/os/Packages/
8.1.1911 does not have the package: https://mirrors.edge.kernel.org/centos/8.1.1911/BaseOS/x86_64/os/Packages/
There is the EPEL archives. This is fairly recent, but we did a snapshot of 8.1 a bit before 8.2 came out. We plan on doing this for each minor release (7 and 8). Fedora Archives are not on all the mirrors, so it might be a little slower, depending on where you get it from, but at least it's there. https://archives.fedoraproject.org/pub/archive/epel/8.1/
Oh, that's an option, thanks. the archives site tends not to be mirrored as widely though, so if we want to use it we might end up mirroring it internally.
Per what Paul said (using Stream) I suppose we could also enable the Stream repo but set it at a lower priority so we only use packages that are not found elsewhere.
Thanks,
-- Michel Alexandre Salim profile: https://keybase.io/michel_slm chat via email: https://delta.chat/ GPG key: 96A7 A6ED FB4D 2113 4056 3257 CAF9 AD10 ACB1 BEF2 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Thu, May 14, 2020 at 7:44 PM Carl George carl@redhat.com wrote:
This should normally be an edge case that only occurs during the time between a RHEL point release and the corresponding CentOS Linux rebuild. Historically that usually takes about a month. As more and
A few have been that fast. Some have taken up to six months: I'm thinking of CentOS 7.2.
Regarding using CentOS Stream for your use case, be aware that EPEL is built against RHEL, so it's possible to run into issues where an EPEL package doesn't work on CentOS Stream. This is a known issue that we haven't come to a conclusion yet on how to handle. On a related note, EPEL faces similar challenges when RHEL bumps a library soname. The
It's one reason I spend some time with "mock", myself, to be able to backport dependency chains and resolve such discrepancies at least with an internal repository.
On Thu, May 14, 2020 at 08:14:31PM -0400, Nico Kadel-Garcia wrote:
A few have been that fast. Some have taken up to six months: I'm thinking of CentOS 7.2.
RHEL Release Date - 7.2: 2015-11-19 CentOS Release Date - 7.2-1511: 2015-12-14
Perhaps you're thinking of the 6.0 fiasco?
John
On Thu, May 14, 2020 at 3:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
We're working on validating CentOS 8 for some desktop use cases at work, and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Do not get me *started* on the ansible version fun and games, or the confusing state of the python3 for various EPEL, RHEL and CentOS migrations. The situation is exacerbated when RHEL elects to use a kind-of-sort-of distinct naming scheme for software previously published via EPEL.
It's an ongoing problem. EPEL's decision to show only the most recent versions of RPMs, and to trim old RPMs out, is a destabilizing problem and why I make hrdlinked snapshots of EPEL using "rsnapshot" for internal access to old packages.
On Thu, 14 May 2020 at 20:00, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, May 14, 2020 at 3:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
We're working on validating CentOS 8 for some desktop use cases at work, and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Do not get me *started* on the ansible version fun and games, or the confusing state of the python3 for various EPEL, RHEL and CentOS migrations. The situation is exacerbated when RHEL elects to use a kind-of-sort-of distinct naming scheme for software previously published via EPEL.
It's an ongoing problem. EPEL's decision to show only the most recent versions of RPMs, and to trim old RPMs out, is a destabilizing problem and why I make hrdlinked snapshots of EPEL using "rsnapshot" for internal access to old packages.
To be clear here.. the 'decision' is that EPEL is built using the same build system that Fedora uses. The Fedora build system does not keep older versions of packages in its composes for a space and so EPEL can not keep older versions of packages either. We tried several 'hacks' to do this and they broke the Fedora side or didn't do what we wanted on the EPEL side either.
At this point it is either someone finding the time to deep dive into pungi and other tools to make it work for this 2 different case mode or moving EPEL to a different build system. Both are giant projects which no one has had the time to do.
On Fri, May 15, 2020 at 8:02 AM Stephen John Smoogen smooge@gmail.com wrote:
On Thu, 14 May 2020 at 20:00, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, May 14, 2020 at 3:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
We're working on validating CentOS 8 for some desktop use cases at work, and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Do not get me *started* on the ansible version fun and games, or the confusing state of the python3 for various EPEL, RHEL and CentOS migrations. The situation is exacerbated when RHEL elects to use a kind-of-sort-of distinct naming scheme for software previously published via EPEL.
It's an ongoing problem. EPEL's decision to show only the most recent versions of RPMs, and to trim old RPMs out, is a destabilizing problem and why I make hrdlinked snapshots of EPEL using "rsnapshot" for internal access to old packages.
To be clear here.. the 'decision' is that EPEL is built using the same build system that Fedora uses. The Fedora build system does not keep older versions of packages in its composes for a space and so EPEL can not keep older versions of packages either. We tried several 'hacks' to do this and they broke the Fedora side or didn't do what we wanted on the EPEL side either.
That is not clear at all. Build systems normally build new versions of software and deploy them to some target space. The decision to delete older packages is a very distinct one. Review of that workspace and expiration of old packages pretty much *must* be a distinct one. Deletion of obsolete packages in anticipation of their activation in an entirely distinct repository is a very distinct decision.
EPEL does duel publication of upstream RHEL published packages with ansible. I really don't see any reason why the older EPEL packages can't be left in place until well after they are published in CentOS, not merely RHEL, as a courtesy to the far larger CentOS user community. And yes, I've had to build my own internal package of an older EPEL release until CentOS published a release myself a few times. It's going on right now with python 3 packages as the migration to "python3" based package names continues.
These can be automated. But publication to a common workspace is not the same as preservation of an internal workspace. I could believe that synchronization among them are automated.
At this point it is either someone finding the time to deep dive into pungi and other tools to make it work for this 2 different case mode or moving EPEL to a different build system. Both are giant projects which no one has had the time to do.
Or, simply stop deleting the older RPMs from the mirrors.There is a cost, mirroring would take more scanning and more bandwidth to keep the mirrors. Metadata would also accumulate and cause some additional burden. But let's not confuse a model with one repositories building workspace with the same project's publication workspace. I've actually been through this one a few times in public and private software repositories: it can take a bit of thought.
On Fri, 15 May 2020 at 09:06, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Fri, May 15, 2020 at 8:02 AM Stephen John Smoogen smooge@gmail.com wrote:
On Thu, 14 May 2020 at 20:00, Nico Kadel-Garcia nkadel@gmail.com
wrote:
On Thu, May 14, 2020 at 3:32 PM Michel Alexandre Salim michel@michel-slm.name wrote:
Hi,
We're working on validating CentOS 8 for some desktop use cases at
work,
and noticed that after working fine on a machine that's installed several months ago, it's now failing on a freshly-installed machine.
Do not get me *started* on the ansible version fun and games, or the confusing state of the python3 for various EPEL, RHEL and CentOS migrations. The situation is exacerbated when RHEL elects to use a kind-of-sort-of distinct naming scheme for software previously published via EPEL.
It's an ongoing problem. EPEL's decision to show only the most recent versions of RPMs, and to trim old RPMs out, is a destabilizing problem and why I make hrdlinked snapshots of EPEL using "rsnapshot" for internal access to old packages.
To be clear here.. the 'decision' is that EPEL is built using the same
build system that Fedora uses. The Fedora build system does not keep older versions of packages in its composes for a space and so EPEL can not keep older versions of packages either. We tried several 'hacks' to do this and they broke the Fedora side or didn't do what we wanted on the EPEL side either.
That is not clear at all. Build systems normally build new versions of software and deploy them to some target space. The decision to delete older packages is a very distinct one. Review of that workspace and expiration of old packages pretty much *must* be a distinct one. Deletion of obsolete packages in anticipation of their activation in an entirely distinct repository is a very distinct decision.
I believe Dennis Gilmore and others have tried to explain this multiple times in the past, but it seems that it just doesn't get through. Here is my take on it.. which is not as nice as theirs and probably flawed because I am tired.
The Fedora build system means everything from pkgs, pdc, bodhi, koji, pungi, and several other tools. The way that this system is built is to build a complete operating system with the 'compose' being just like what is in rawhide or a release. Everything is tagged to be in the compose, a fresh tree is generated, createrepo and other items are built on that tree, and that tree then replaces the old one on the download servers. Just like F32 directories and rawhide do not have older copies of packages.. neither does EPEL. So to the system there are no old rpms to keep around.. [and trying to keep them seems to end up with a good many mirrors carrying new packages but old repodata (or vice versa.. new repodata but no new rpms) so can't be used by yum/dnf]
Of course there could be other solutions that would fix this problem.. but each one takes time and energy that no one seems to have the time to volunteer to get done. If you have the time to do so, I am sure people will welcome it.
On 5/14/20 4:59 PM, Nico Kadel-Garcia wrote:
It's an ongoing problem. EPEL's decision to show only the most recent versions of RPMs, and to trim old RPMs out, is a destabilizing problem and why I make hrdlinked snapshots of EPEL using "rsnapshot" for internal access to old packages.
So that's a problem we have with Fedora (e.g. hardware regressions preventing some hardware from being in the latest kernel, *but* if they were not updating often enough, the last known-good version might not be available anymore unless you pull from Koji).
But it does not really apply to the EPEL retirement. If a decision is made to retire packages for a given branch, it does not matter how many versions you keep, they will all be retired, right?
Perhaps having additional metadata in dist-git (e.g. "retire from EL 8.2 onwards") would allow maintainers to keep building as normal, we can publish epel/8-all, epel/8.1, epel/8.2 etc. repos where the first repo has all packages, and the others have symlinks to 8-all but filtering out the retired packages?
But that won't work if there are ABI incompatibilities... so yeah, overall I think the best solution is to just try and get CentOS releases out of the door sooner rather than later.
Or have a "purgatory" repo where packages retired in EL 8.2 get to live until, say, a month after CentOS 8.2 is GA? Again, seems like too much work.
In our case, hopefully this is a one-off (it's because we deploy RPM-with-zstd internally).
On Fri, May 15, 2020 at 10:37 AM Michel Alexandre Salim michel@michel-slm.name wrote:
Or have a "purgatory" repo where packages retired in EL 8.2 get to live until, say, a month after CentOS 8.2 is GA? Again, seems like too much work.
Or, perhaps we could archive things before a release. I guess my reply got trimmed off and/or ignored.
There is the EPEL archives.
This is fairly recent, but we did a snapshot of 8.1 a bit before 8.2 came out. We plan on doing this for each minor release (7 and 8). Fedora Archives are not on all the mirrors, so it might be a little slower, depending on where you get it from, but at least it's there.
https://archives.fedoraproject.org/pub/archive/epel/8.1/
Troy
-
Antal Nemeš -
Carl George -
John R. Dennison -
Michel Alexandre Salim -
Nico Kadel-Garcia -
Stephen John Smoogen -
Troy Dawson