On Fri, 2021-01-29 at 23:09 +0000, redbaronbrowser via CentOS-devel wrote:
On Friday, January 29, 2021 4:58 PM, Mark Mielke < mark.mielke@gmail.com> wrote:
This is why "Facebook" (as one example of scale in context for this discussion) is using CentOS Stream, and not using RHEL.
Is there any public interview that Facebook is using the latest Stream packages in production?
Rich already linked to the latest talk below. I had a more recent one at SCALE 18x that went in a bit more detail but the recording was botched. I will be talking about CentOS Stream at FB specifically at DevConf.cz later this month: https://devconfcz2021.sched.com/event/gmOa/centos-stream-at-facebook
It seems to me that Facebook is doing it's own CI/CD tests and avoid regressions that would impact it.
I'm willing to accept that Facebook is using select versions of packages from Stream. I have a harder time believing they are using Stream the same way one of us would of running yum and expecting things to continue to work.
We do "rolling OS updates". We keep dated snapshots of the repo and whenever we do an update we shard a change to dnf.conf to include the repo with the new snapshot and run 'dnf upgrade' from Chef. We test this on a few machines and then (slowly) roll it across the fleet over a couple of weeks. Sometimes we might have to add a quirk or two to the Chef recipe that manages the upgrade, but it generally works pretty well. None of this is specific to Stream: we'd been doing this with CentOS 6 and 7 as well (and you can find older talks I gave where we cover it). With Stream it's just easier as there's only one repo in play, instead of having to track "updates" and then resync the main repo whenever a point release is cut.
Facebook is also probably retaining an internal vault of previous versions to allow them to revert. Again, that is not the same as what is exposed to most of us.
We do keep the dated snapshots around, but we never rollback updates. If something goes wrong we stop the rollout, fix it (often with some logic in Chef) and resume the rollout.
It would be nice to see whatever tests Facebook considers important contributed back into Stream's own CD/CI but so far I haven't gotten an answer as to when/if public access to updating the tests will be available.
Sure, happy to participate in this when something becomes available.
Cheers Davide