[CentOS-devel] First round of RHEL programs announced

Thu Feb 4 00:26:41 UTC 2021
Davide Cavalca <dcavalca at fb.com>

On Fri, 2021-01-29 at 23:09 +0000, redbaronbrowser via CentOS-devel
> On Friday, January 29, 2021 4:58 PM, Mark Mielke <
> mark.mielke at gmail.com> wrote:
> > This is why "Facebook" (as one example of scale in context for this
> > discussion) is using CentOS Stream, and not using RHEL.
> Is there any public interview that Facebook is using the latest
> Stream packages in production?

Rich already linked to the latest talk below. I had a more recent one
at SCALE 18x that went in a bit more detail but the recording was
botched. I will be talking about CentOS Stream at FB specifically at
DevConf.cz later this month:

> It seems to me that Facebook is doing it's own CI/CD tests and avoid
> regressions that would impact it.
> I'm willing to accept that Facebook is using select versions of
> packages from Stream.  I have a harder time believing they are using
> Stream the same way one of us would of running yum and expecting
> things to continue to work.

We do "rolling OS updates". We keep dated snapshots of the repo and
whenever we do an update we shard a change to dnf.conf to include the
repo with the new snapshot and run 'dnf upgrade' from Chef. We test
this on a few machines and then (slowly) roll it across the fleet over
a couple of weeks. Sometimes we might have to add a quirk or two to the
Chef recipe that manages the upgrade, but it generally works pretty
well. None of this is specific to Stream: we'd been doing this with
CentOS 6 and 7 as well (and you can find older talks I gave where we
cover it). With Stream it's just easier as there's only one repo in
play, instead of having to track "updates" and then resync the main
repo whenever a point release is cut.

> Facebook is also probably retaining an internal vault of previous
> versions to allow them to revert.  Again, that is not the same as
> what is exposed to most of us.

We do keep the dated snapshots around, but we never rollback updates.
If something goes wrong we stop the rollout, fix it (often with some
logic in Chef) and resume the rollout.

> It would be nice to see whatever tests Facebook considers important
> contributed back into Stream's own CD/CI but so far I haven't gotten
> an answer as to when/if public access to updating the tests will be
> available.

Sure, happy to participate in this when something becomes available.