[CentOS] Blog article about the state of CentOS

Wed Jun 24 16:27:33 UTC 2020
Lamar Owen <lowen at pari.edu>

On 6/20/20 6:50 AM, Peter wrote:
> On 20/06/20 3:50 am, Johnny Hughes wrote:
>
>> And EL8 is exponentially harder with an entirely new build system and
>> the requirement to build modules.
>
> But it seems like every major release has had reasons to be 
> exponentially harder than the last.  With 7 it was the shift to using 
> the git sources instead of the SRPMS that were previously provided by 
> Red Hat, thereby necessitating an entirely new build system, plus the 
> change to systemd and the introduction of a new point release 
> numbering scheme, not to mention the move to entirely new 
> infrastructure because of the change to Red Hat sponsorship. 

Using git to build sources from, while a change, doesn't significantly 
increase the number of packages that need building. The actual package 
building - compilation - stage can take days on even the fastest 
hardware.  Yes, the build infrastructure is different, and yes there is 
a learning curve involved, but I have found that the mechanics of 
building the source is not really that different from doing it with 
source RPMs.  Instead of 'wget src.rpm; rpmbuild --rebuild src.rpm' you 
do a 'git clone $rpm-package; git checkout $tag; get_sources.sh; 
rpmbuild -ba $tree' for a full manual build; I've now done this a few 
times, and it's not really difficult, just a few more keys to press.  
What can't now be used is the %{BUILDTIME} embedded in the source RPM 
that can give you clues as to build order; a git commit of a thousand 
packages can be atomic and all at the same time.

The use of systemd has nothing at all to do with the build difficulty; 
it's just another package for that purpose; likewise, the new release 
numbering has nothing to do with build and QA difficulty.

> ...
> It would appear that the package build was completed on the 4th of 
> May, why didn't we get a CR repo dump this time around so that CentOS 
> users wouldn't have to wait another month before getting critical 
> updates?

CentOS users who can't wait for CentOS to release 'critical' updates 
really should reconsider using something else; that might be RHEL, that 
might be something completely different.  This is part of the calculated 
risk of using CentOS, and this hasn't changed since 2.1. That's not 
going to change, nor can it thanks to all the reasons Johnny mentioned.  
Building a full distribution from sources can be a hard problem.

As far as the first build iteration completion date is concerned, that 
should be considered a rough draft build; if the QA process finds binary 
incompatibility (library linkage versions for the most part) then 
several if not all packages need rebuilding as part of the QA stage.  Do 
you REALLY want to use first-draft stage 1 packages in production?  You 
can look in the %{BUILDTIME} query tag for build order; use the 
following command to get the order:
rpm -qa --queryformat "%{BUILDTIME} %{NAME} %{EPOCH} %{VERSION} 
%{RELEASE}\n" | sort

Or you can go through the released packages on centos.org and look at 
the build dates to see how many packages were built pretty late; for 
instance, bpftool-4.18.0-193.el8.x86_64.rpm apparently wasn't built in 
the released version until 2020-05-29.  I'm reasonably sure the first 
pass at this was built probably a month earlier based on the dates of 
other packages, but something was probably found in QA that required a 
rebuild, or the rebuild depended on something else that was late to build.

AND since this rebuilt package HAS TO KEEP THE SAME 
Name/Epoch/Version/Release (NEVR) tuple to be RHEL-compatible, the first 
N=bpftool V=4.18.0 R= 193.el8 could not be released; who knows, there 
may have been a dozen of the same NVR (bpftool's %{EPOCH} is (none) and 
so I don't count it)...   If the first N=bpftool V=4.18.0 R= 193.el8 
were to be released, then how, within the RPM update decision mechanism 
that only uses the EVR to update, is dnf/yum/rpm going to know the 
version built on 2020-05-29 is updating the one built on say 
2020-04-26?  No, ${BUILDTIME} is NOT used for update decisions, sorry, 
and bumping %{EPOCH} is the nuclear sledgehammer of RPM versioning and 
thus is strongly discouraged.  For complete compatibility you want every 
package to have the same NEVR in CentOS as in RHEL except where 
CentOS-specific changes had to be made (branding, mostly).

And then there's modularity in CentOS 8, which means packages might have 
to build against a whole 'nother set of packages (I'm thinking 
specifically of the three different PostgreSQL versions that are 
modularized, or the mariadb versus MySQL modules with their libraries, 
and so on) that does indeed dramatically increase the number of package 
builds that not only need building but then need QA testing.  Or do you 
want to run untested packages in production to get the packages sooner?