[CentOS-devel] How does a 9-stream package move from Koji to released-to-mirrors?

Thu Jun 16 01:15:47 UTC 2022
Ian Wienand <iwienand at redhat.com>

Thank you for your response filling in the gaps!  Very useful :)

On Wed, Jun 15, 2022 at 08:37:22AM -0400, Josh Boyer wrote:
> > If I look in CentOS-STream-9-20220613.0 (latest-CentOS-stream,
> > currently), it indeed has libselinux-3.4-2 that we want.  However, if
> > I sort the mirror by date [7] nothing seems to have updated since
> > 2022-06-06?  So, if I'm on the right track here -- what promotes the
> > "production" compose to the public mirrors?
> A compose is pushed to the mirrors manually, approximately weekly.  I
> believe the rationale here is that we don't want to overload the
> mirror network with daily pushes as that will likely result in many
> mirrors being stale because they never have time to quiesce.
> I'm not sure when the next mirror push is, but hopefully soon.

So we in OpenStack/OpenDev provide CI resources from
7/8-stream/9-stream and with the overall switch to the -stream model
this is becoming ... more of an issue.

This has bitten us several times now; the last time I can remember was
somehow "ping" stopped working as a regular user which broke all sorts
of jobs and had to be worked around for > a week too.

We can probably agree that ~2 weeks for bug fixes to the distro is
really making things difficult?  A lot is communication I guess; the
fix in question was committed on 31st May, it's 16th June here and I
see the package in the mirror now; but even in the upstream bug nobody
seemed to have an idea of when the fix would make it out there (your
mail helped clear up some of the process, thank you).

As a CI provider, we do actually want to be testing very close to
commits.  OpenStack installs and stresses lots of things, and we love
finding bugs and fixing them upstream :) A job broken for a day or
three while upstream fixes are processed is workable; but as things
drag out we need to start committing work-arounds, that then flows on
to crappy hacks making it into releases, etc. and it all just gets

We *could* do something like add repos to the latest production/
compose at the start of each job and dnf update; but I am assuming
that centos infra don't want our ~hundreds-to-low-thousands/day CI
nodes hitting composes.stream.centos.org -- and even so we don't want
to do that because transient network errors make CI jobs unreliable,
so we avoid that as much as possible.

Another idea is that we could mirror in the latest composes to our CI
mirror infrastructure and then update CI nodes from there, so the
packages are cloud-local -- if we had a way to rsync from
composes.stream.centos.org to our mirrors?  I'm not sure if centos
infra want to do that either.  It also relies on the centos side
publishing composes atomically, so we don't half-mirror something
(locally, we use AFS -- so we mirror to the R/W partition and only
once we're in sync do we atomically release that volume to the

As an outsider, it seems to me the best solution is probably to
greatly increase the cadence of pushing the packages out to the
official mirrors.  The "weeks" type cadence is probably OK for well
tested updates -- the type of thing non-stream used to be I guess.
But if we're in a model now of basically being a testing distro, I
think we need to get the testing out there faster.

rawhide is mirrored and I imagine that has much higher delta than
9-stream would?  Assuming the compose builds and passing internal
testing, could it be pushed as much as daily?

Something practical I can look at doing is update [1] with some of
this info.

Please let me know if there's a better forum to discuss this.  I'm not
exactly sure how, but if any of the resources OpenDev/OpenStack has
can help, I'll be happy to get that going.



[1] https://docs.centos.org/en-US/stream-contrib/quickstart/