[CentOS-devel] Balancing the needs around the RHEL platform

On pe, 25 joulu 2020, Ljubomir Ljubojevic wrote:
>On 12/24/20 2:37 PM, Neal Gompa wrote:
>> In the strictest sense, it obviously is not. But in a very real
>> practical sense, it absolutely is. Aside from the kernel issues (which
>> I firmly believe are solvable), people are generally not going to
>> notice a difference between CentOS Linux 8 and CentOS Stream 8.
>>
>> My CentOS Linux 8 boxes were replaced with CentOS Stream 8 back in the
>> spring because it was strictly better for production *and*
>> development. I've been in the process of opportunistically switching
>> our build targets from CentOS Linux 8 to CentOS Stream 8 most of the
>> year. With the retirement of CentOS Linux 8, it now becomes more of a
>> priority, but it was already going to happen.
>>
>
>As I understood it, Stream is not in full swing yet, there is no
>active/daily contribution from RHEL team?

"Active/daily" contribution is pretty much a seasonal thing unrelated to
the actual processes. Right now majority of engineers contributing to
RHEL 8 sources are on their holidays and end of year vacation time. You
can notice these "activity dives" also in Fedora Project annual reports by
Matthew Miller, like https://mattdm.org/fedora/2020nest/StateOfFedora2020.pdf.
This is a well-known phenomenon in Fedora community.

Anything that goes into RHEL 8.next distribution composes is synchronized to
CentOS 8 Stream. That means CentOS 8 Stream is defined by the content in
RHEL 8.next distribution composes and the packages in there are
contributed to only by RHEL engineers. They do not build the packages in
CentOS 8 Stream themselves, at least now.

To show a similarity, Fedora ELN is also not being built explicitly: you
commit to Rawhide dist-git and build a package in Rawhide, then Fedora
ELN build is kicked off by a Fedora messaging bus event. Think of
RHEL->CentOS 8 Stream flow in a similar way, only with a number of
additional, mandatory, automated and manual, steps in between, according
to various RHEL QA processes.

Since there are thousands of packages in RHEL and there are development
stages where more activity happens than in other time, naturally, you
may see spikes of that activity along with relative periods of a
'cooldown'. Again, in some areas of the distribution that activity might
be more exposed than in the others. In past, there was a general
reduction of activity close to externally visible milestones like RHEL
beta releases and after that. Hopefully, we'll see it with CentOS Stream
too -- after all, if all content comes there automatically from RHEL
development, then slowdown in RHEL development for milestones will be
reflected in CentOS Stream (almost) automatically. Sure, updates do not
come immediately but that's defined by the fact that CentoS Stream
tracks RHEL development composes, not RHEL dist-git commits and
individual package builds.

For example, when I released FreeIPA 4.9.0 release candidate 3 upstream
on December 10th, it was imported into RHEL 8.next development tree on
December 11th. Later, it was built on the same day as a
idm-DL1-8040020201211123410.1f8cbe47 module stream build, and spent five
days in the QA process before it got into RHEL 8.next nightly composes.
Once it appeared there, all SRPM packages that were included into
idm-DL1-8040020201211123410.1f8cbe47 module stream build were imported
into corresponding repositories on git.centos.org -- all this
information can be seen in the git commits done by the scripts that
import the content.

It took some time to figure out how to rebuild idm:DL1 module stream in
CentOS 8 Stream. In fact, Carl had to do some manual fixes which he
contributed FreeIPA upstream https://github.com/freeipa/freeipa/pull/5367 
and which will appear on RHEL 8.next once FreeIPA 4.9.0 final version
will be built there after holidays/vacation season ends, thus bringing
it back to CentOS 8 Stream as a part of the upstream changes. But the
RHEL 8.next's idm-DL1-8040020201211123410.1f8cbe47 module stream built
finally landed in CentOS 8 Stream as
idm-DL1-8040020201219215824.1f8cbe47 module stream build on December
19th. So, at least three days delay between RHEL 8.next compose getting
the packages and CentOS 8 Stream getting their rebuilds. Less than RHEL
QE team spent qualifying the build themselves.

My understanding of what is not in 'full swing' is fully automated
processing of these updates.  There is still a lot of domain knowledge
that is not automated. Most common one is an order of builds. There is
also a complexity with module stream builds -- you need to know whether
a particular package belongs to a specific module stream and don't try
to rebuild it without rebuilding the whole stream (and don't build the
stream without collecting all changes to that stream's packages first).
It is certainly possible to do so based on the datastamps in the SRPM
packages in RHEL composes but my understanding is that it is not yet
utilized. Hopefully, there will be a better way to do so.

>What will happen to your system when/if there is new kernel change every
>few days? How much "punishment" can your system handle safely?

You certainly control when you can and want upgrade your deployment
systems. It has nothing to do with the cadence of updates coming into a
distribution.

I find this fixation on the kernel updates is skewing things a lot.
Kernel, certainly, is important, but it is not the thing that is RHEL or
CentOS distribution, alone.

-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland