[CentOS-devel] Balancing the needs around the CentOS platform

On 19/12/2020 10:09, Mark Mielke wrote:
> On Sat, Dec 19, 2020 at 4:34 AM Mark Mielke <mark.mielke at gmail.com> wrote:
>> 2. Minor release milestones to stabilize branches. We have breakage
>> with most minor release upgrades, and the stabilization process is an
>> important method of isolating users from being affected by this. This
>> is why CentOS 8 Stream is being said "for developers", while RHEL 8
>> would be "for production". It is being said, because it is a real
>> thing. If you truly believed minor release milestones were unnecessary
>> for CentOS 8 Stream, then you would also believe that minor release
>> milestones were unnecessary for RHEL 8.
> 
> I should provide real-life examples for you to consider. I will just a
> pick few that come to mind without thinking too hard.
> 
> The first example, is EL 7.8 and the GlusterFS update. This broke our
> loadbuild process across multiple product teams. In this particular
> case, several teams build a custom version of Qemu as part of their
> Yocto build process. The error showed up like this:
> 
> build   01-Dec-2020 08:05:20   |
> .../tmp/work/x86_64-linux/qemu-native/2.7.0-r1/qemu-2.7.0/block/gluster.c:
> In function ‘qemu_gluster_truncate’:
> build   01-Dec-2020 08:05:20   |
> .../tmp/work/x86_64-linux/qemu-native/2.7.0-r1/qemu-2.7.0/block/gluster.c:1000:5:
> error: too few arguments to function ‘glfs_ftruncate’
> build   01-Dec-2020 08:05:20   |      ret = glfs_ftruncate(s->fd, offset);
> build   01-Dec-2020 08:05:20   |      ^
> build   01-Dec-2020 08:05:20   | In file incl
> 
> GlusterFS changed the function glfs_ftruncate to require a second
> argument. Red Hat chose to upgrade GlusterFS as part of EL 7.8. Qemu
> 2.7.0 doesn't know about this change. Builds failed for three of our
> product teams.
> 
> By design, we did early adopter testing of 7.8 before mass upgrading
> everyone, and we discovered this problem, and were able to get code
> changes into the various product releases. In this case, we chose to
> disable GlusterFS from the Qemu build process as GlusterFS was an
> accidental dependency. Once it was disabled, the above code stopped
> being compiled, and the emergency was averted.
> 
> Now, imagine the same scenario with "CentOS 8 Stream". Are you
> expecting that users will perform each set up their own milestone
> release processes? Are we going to test every single package that gets
> released to "CentOS 8 Stream" incrementally to ensure that our systems
> don't experience massive breakage?
> 
> Another example that hit us with EL 7.7, is an update to elfutils that
> was incompatible with binutils and gcc:
> 
> https://wiki.gentoo.org/wiki/Binutils_2.32_upgrade_notes/elfutils_0.175:_unable_to_initialize_decompress_status_for_section_.debug_info
> 
> In this case, we still don't have a solution - but are downgrading
> elfutils for 7.7+ until we do decide what to do about it. Still, we
> were able to detect it in our 7.7 early adopter testing, and decide on
> a temporary solution *before* rolling it out to the users.
> 
> I mentioned this problem earlier this week:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1489542
> 
> This was a change to autofs in EL 7.4 that caused it to drop automount
> maps every 10 minutes in our environment, causing actual breakage when
> paths were accessing during these intervals. Again, we were able to
> detect this in our EL 7.4 early adopter testing, and defer roll out of
> EL 7.4 until after we came up with a solution.
> 
> These are just a few of the regular things that we encounter all of
> the time. I don't see how CentOS 8 Stream can address these, unless
> you promise to treat CentOS 8 Stream as a minor release, and never
> introduce changes in API, new features, or changes in behaviour.
> 
> For all of these, when upstream or downstream - we contribute back
> analysis, bug reports, and fixes. However, I cannot be deploying
> CentOS 8 Stream in our environment. It wouldn't even be an option. We
> would choose something else.
> 
> I think when you say "95% of users would be ok with CentOS 8 Stream",
> you are talking about basic usage. You are not talking about
> Enterprise use cases.
> 

Your post illustrates nicely where CentOS Stream fits into the 
ecosystem. Running CentOS Stream would allow your developers early 
opportunity to identify, feed back, fix and test any issues _before_ 
they affect your RHEL production systems. This is great as long as you 
are running RHEL on your production systems and using CentOS Stream as 
your CI development system. Clearly CentOS Stream is not suitable as a 
replacement from your RHEL production systems, and as you have 
previously stated, if you can not afford the RHEL licensing costs then 
you must look elsewhere.