Thank you for your response filling in the gaps! Very useful :)

On Wed, Jun 15, 2022 at 08:37:22AM -0400, Josh Boyer wrote:

...

If I look in CentOS-STream-9-20220613.0 (latest-CentOS-stream, currently), it indeed has libselinux-3.4-2 that we want. However, if I sort the mirror by date [7] nothing seems to have updated since 2022-06-06? So, if I'm on the right track here -- what promotes the "production" compose to the public mirrors?

A compose is pushed to the mirrors manually, approximately weekly. I believe the rationale here is that we don't want to overload the mirror network with daily pushes as that will likely result in many mirrors being stale because they never have time to quiesce.

I'm not sure when the next mirror push is, but hopefully soon.

So we in OpenStack/OpenDev provide CI resources from 7/8-stream/9-stream and with the overall switch to the -stream model this is becoming ... more of an issue.

This has bitten us several times now; the last time I can remember was somehow "ping" stopped working as a regular user which broke all sorts of jobs and had to be worked around for > a week too.

We can probably agree that ~2 weeks for bug fixes to the distro is really making things difficult? A lot is communication I guess; the fix in question was committed on 31st May, it's 16th June here and I see the package in the mirror now; but even in the upstream bug nobody seemed to have an idea of when the fix would make it out there (your mail helped clear up some of the process, thank you).

As a CI provider, we do actually want to be testing very close to commits. OpenStack installs and stresses lots of things, and we love finding bugs and fixing them upstream :) A job broken for a day or three while upstream fixes are processed is workable; but as things drag out we need to start committing work-arounds, that then flows on to crappy hacks making it into releases, etc. and it all just gets ugly.

We *could* do something like add repos to the latest production/ compose at the start of each job and dnf update; but I am assuming that centos infra don't want our ~hundreds-to-low-thousands/day CI nodes hitting composes.stream.centos.org -- and even so we don't want to do that because transient network errors make CI jobs unreliable, so we avoid that as much as possible.

Another idea is that we could mirror in the latest composes to our CI mirror infrastructure and then update CI nodes from there, so the packages are cloud-local -- if we had a way to rsync from composes.stream.centos.org to our mirrors? I'm not sure if centos infra want to do that either. It also relies on the centos side publishing composes atomically, so we don't half-mirror something (locally, we use AFS -- so we mirror to the R/W partition and only once we're in sync do we atomically release that volume to the mirrors).

As an outsider, it seems to me the best solution is probably to greatly increase the cadence of pushing the packages out to the official mirrors. The "weeks" type cadence is probably OK for well tested updates -- the type of thing non-stream used to be I guess. But if we're in a model now of basically being a testing distro, I think we need to get the testing out there faster.

rawhide is mirrored and I imagine that has much higher delta than 9-stream would? Assuming the compose builds and passing internal testing, could it be pushed as much as daily?

Something practical I can look at doing is update [1] with some of this info.

Please let me know if there's a better forum to discuss this. I'm not exactly sure how, but if any of the resources OpenDev/OpenStack has can help, I'll be happy to get that going.

Thanks

-i

[1] https://docs.centos.org/en-US/stream-contrib/quickstart/