I wrote a blog post to share with you:
https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platfo...
Below is a fair summary of the blog post, but I encourage you to read the whole thing for the context around the "availability gap" and the "openness gap":
Summary ======= Letting an outsider take care of the availability gap after the RHL to Fedora/RHEL split meant there was no vehicle for closing the openness gap. Red Hat needed to create or bring-in a solution that was highly available (easy to get) in order to close the gap around openness (easy to contribute to). That was the joining forces with CentOS. And now we think we have something that can cover both the availability and openness gaps, please try it out.
All along Red Has been making a Linux OS that we want to be open, to be available, to be stable enough, and to be sustainable. And now we think we have the next evolution in that decades of work, in the form of CentOS Stream. =======
Kind regards,
- Karsten
On 19/12/2020 07.44, Karsten Wade wrote:
I wrote a blog post to share with you:
https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platfo...
Below is a fair summary of the blog post, but I encourage you to read the whole thing for the context around the "availability gap" and the "openness gap":
Summary
Letting an outsider take care of the availability gap after the RHL to Fedora/RHEL split meant there was no vehicle for closing the openness gap. Red Hat needed to create or bring-in a solution that was highly available (easy to get) in order to close the gap around openness (easy to contribute to). That was the joining forces with CentOS. And now we think we have something that can cover both the availability and openness gaps, please try it out.
All along Red Has been making a Linux OS that we want to be open, to be available, to be stable enough, and to be sustainable. And now we think we have the next evolution in that decades of work, in the form of CentOS Stream. =======
Kind regards,
- Karsten
Hi Karsten,
I agree with you that CentOS Stream might actually be a good way forward and consider (even looking forward to) switching from CentOS Linux to CentOS Stream for the reasons you mentioned. However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower. This topic has already been mentioned several times on this mailing list by various users. So far feedback has been positive here on the list.
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility. The only way I can think of implementing this is by offering an optional "RHEL kernel repository" containing the (updated) kernel of the current RHEL minor release. Feedback to this proposal has been positive so far as well, but no concrete feedback yet. I even suggested this to centos-questions@redhat.com. So far I only got a response that I shall wait till next year for no- and low-cost RHEL offerings. Let's wait and see if I get a useful answer to my proposal.
Sorry for mentioning this issue once again here on the mailing list, but I just hope that by mentioning it often enough, the possibility of this issue/proposal being acknowledged goes up.
This issue (and the fact that the Devel repository is non existing for CentOS Stream yet) is (are) the only blocker(s) for me that keep me from switching to CentOS Stream. Otherwise I very likely would have switched by now.
As you can see, I, and I assume others as well, want to get involved in CentOS Stream. However there are still some issues to solve and the infrastructure how to get involved is not yet in place. I'm looking forward to and am curious how all of this works out in the next months.
Best,
Peter
On 12/19/20 1:13 AM, Peter Georg wrote:
However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower.
Do you have a reason to think that > 5% of users need 3rd party drivers?
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility.
Stream is going to continue to get the same kernel that RHEL gets, it's just going to get updates earlier.
In the past, users with binary 3rd party drivers would need to wait a while after a point release before they could update their kernel. In CentOS Stream, they'll have to wait a while after a kernel package update before they can update their kernel.
There's really very little changing, here. The point releases are going away, so the date of kernel updates is less predictable, but the solution hasn't changed at all: If you use a binary 3rd party kernel module, you need a process in place to block kernel updates until your 3rd party kernel modules are available.
On Sat, Dec 19, 2020 at 3:09 PM Gordon Messmer gordon.messmer@gmail.com wrote:
Stream is going to continue to get the same kernel that RHEL gets, it's just going to get updates earlier.
This is false.
In the past, users with binary 3rd party drivers would need to wait a while after a point release before they could update their kernel. In CentOS Stream, they'll have to wait a while after a kernel package update before they can update their kernel.
This is hugely problematic.
There's really very little changing, here. The point releases are going away, so the date of kernel updates is less predictable, but the solution hasn't changed at all: If you use a binary 3rd party kernel module, you need a process in place to block kernel updates until your 3rd party kernel modules are available.
This is hugely problematic.
But, I want to respond with details to a different post.
On 19/12/2020 20:09, Gordon Messmer wrote:
On 12/19/20 1:13 AM, Peter Georg wrote:
However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower.
Do you have a reason to think that > 5% of users need 3rd party drivers?
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility.
Stream is going to continue to get the same kernel that RHEL gets, it's just going to get updates earlier.
No, it's not, That's the point. It's going to have the unstable kernel from the NEXT point release so it will not be the same kernel as the current RHEL point release. Case in point, RHEL 8.3 has 4.18.0-240 series kernels, Stream had changed to a 4.18.0-257 kernel within 2 or 3 days of CentOS 8.3 being released.
In the past, users with binary 3rd party drivers would need to wait a while after a point release before they could update their kernel. In CentOS Stream, they'll have to wait a while after a kernel package update before they can update their kernel.
And that change happened on a predictable, approximately 6 month interval. And required the third party drivers to be rebuild once per point release and usually, by the time the CentOS build was done, those drivers were already rebuilt and already available since they had been done for the preceding RHEL point release.
So we're going from a once every 6 months rebuild to a once every time the RHEL kernel developers feel like changing. Those changes were abstracted from the CentOS and RHEL kernels by the fact that KABI changes only happened (intentionally!) at point releases. Now that will change whenever someone in the kernel development team commits a breaking change. So we're going from a rebuild the drivers every 6 months to rebuild the drivers whenever RH feel like breaking it. That could be once a month or once a week or every time the kernel is updated or even once every 6 months like stable RHEL (though the chances of that are slim IMO). And since RHEL will still be using the old kernel series, that means that the third party yum repos have to invent a new way to cater for both RHEL and Stream since they will be different.
Trevor
On 19/12/2020 20:37, Trevor Hemsley via CentOS-devel wrote:
On 19/12/2020 20:09, Gordon Messmer wrote:
On 12/19/20 1:13 AM, Peter Georg wrote:
However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower.
Do you have a reason to think that > 5% of users need 3rd party drivers?
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility.
Stream is going to continue to get the same kernel that RHEL gets, it's just going to get updates earlier.
No, it's not, That's the point. It's going to have the unstable kernel from the NEXT point release so it will not be the same kernel as the current RHEL point release. Case in point, RHEL 8.3 has 4.18.0-240 series kernels, Stream had changed to a 4.18.0-257 kernel within 2 or 3 days of CentOS 8.3 being released.
In the past, users with binary 3rd party drivers would need to wait a while after a point release before they could update their kernel. In CentOS Stream, they'll have to wait a while after a kernel package update before they can update their kernel.
And that change happened on a predictable, approximately 6 month interval. And required the third party drivers to be rebuild once per point release and usually, by the time the CentOS build was done, those drivers were already rebuilt and already available since they had been done for the preceding RHEL point release.
So we're going from a once every 6 months rebuild to a once every time the RHEL kernel developers feel like changing. Those changes were abstracted from the CentOS and RHEL kernels by the fact that KABI changes only happened (intentionally!) at point releases. Now that will change whenever someone in the kernel development team commits a breaking change. So we're going from a rebuild the drivers every 6 months to rebuild the drivers whenever RH feel like breaking it. That could be once a month or once a week or every time the kernel is updated or even once every 6 months like stable RHEL (though the chances of that are slim IMO). And since RHEL will still be using the old kernel series, that means that the third party yum repos have to invent a new way to cater for both RHEL and Stream since they will be different.
Trevor
This whole argument is all a matter of perspective, from which side of the fence you are looking.
As a *developer* of 3rd party driver packages *for RHEL*, CentOS Stream will work well for me as a CI develop system to help me develop driver packages for the *next* RHEL release (8.4). I get to see what changes are *coming* and can develop against them. That said, I kind of already had that ability from the old RHEL point release betas but I gather moving forward we will get far more kernel updates to Stream than we have to date. To be clear, we develop for RHEL. We will not be developing *for* Stream, but we may well use Stream to help develop for the next RHEL point release.
So for me as a developer, Stream is fine. But for my users, the 2 million or so people who use and depend upon the software package we distribute, Stream is a non-starter. These packages are developed for RHEL (and CentOS Linux as a binary compatible clone). Stream is not a replacement for CentOS Linux. CentOS Linux is dead, gone (or rather going). Those Enterprise Linux users viewing Stream as a replacement to CentOS Linux, they are trying to bash a square peg into a round hole if they have a requirement to use 3rd party drivers.
On 19/12/2020 21.09, Gordon Messmer wrote:
On 12/19/20 1:13 AM, Peter Georg wrote:
However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower.
Do you have a reason to think that > 5% of users need 3rd party drivers?
I never said that. I just assume that the amount x of use cases CentOS can not cover plus the amount y of users that need 3rd party drivers is greater than 5%. So yes, you are right, it is an assumption by myself, but so has been the assumption of 95% by Karsten. There are many 3rd party driver / kernel modules you probably do not need. That is good for you, but other people depend on these. I wish all software/hardware I use would be supported 100% by upstream kernel, sadly this is not the reality. Even if they are supported upstream, hardware vendors often supply backports for EL (e.g. AMD ROCm).
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility.
Stream is going to continue to get the same kernel that RHEL gets, it's just going to get updates earlier.
In the past, users with binary 3rd party drivers would need to wait a while after a point release before they could update their kernel. In CentOS Stream, they'll have to wait a while after a kernel package update before they can update their kernel.
There's really very little changing, here. The point releases are going away, so the date of kernel updates is less predictable, but the solution hasn't changed at all: If you use a binary 3rd party kernel module, you need a process in place to block kernel updates until your 3rd party kernel modules are available
It's not that simple. Currently we have a kernel build 4.18.0-x.y in CentOS 8. As Trevor already told you, kABI changes only (there might be exceptions) happen when x is changed. x is fixed for a particular minor release.
In practice this currently (i.e. with CentOS Linux) means that at some point all 3rd party drivers will support a particular kernel version x and all its updated versions x.y. I.e. we are good to upgrade to this particular minor release. We can even update the kernel within this minor release to get security fixes.
With CentOS Stream x is changing frequently, y does not exist at all. In the best case, the same build x used in a minor release is also available in CentOS Stream (actually it should be available in any case). Up to this point everything seems fine as you said, I "just" lose the opportunity to update the kernel as long as any 3rd party driver I have installed is not yet available for version x+? [1]. Nothing new, just like it has always been? Here's the tricky part: In CentOS Linux I can update the kernel only changing "y" and retaining kABI compatibility to fix security issues. In CentOS Stream these security fixes will be part of a new kernel version x+? not having any guarantee about kABI. Any kernel version x.y released in RHEL will NOT be available in CentOS Stream.
Maybe this somewhat longer explanation helps you to understand this issue.
[1] One should also note here that I need to have installed this particular kernel version x as it will not be available from official repositories once x+1 is released. Which in turn means that I somewhere have to store ALL ever release kernel (and directly dependent) rpms as I never know which version "x" will be used for the next minor release and hence be the "chosen" version all 3rd party drivers/kernel modules will be built against.
Peter
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
s
On Sat, Dec 19, 2020 at 1:13 AM Peter Georg peter.georg@physik.uni-regensburg.de wrote:
On 19/12/2020 07.44, Karsten Wade wrote: I agree with you that CentOS Stream might actually be a good way forward and consider (even looking forward to) switching from CentOS Linux to CentOS Stream for the reasons you mentioned. However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower. This topic has already been mentioned several times on this mailing list by various users. So far feedback has been positive here on the list.
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility. The only way I can think of implementing this is by offering an optional "RHEL kernel repository" containing the (updated) kernel of the current RHEL minor release. Feedback to this proposal has been positive so far as well, but no concrete feedback yet. I even suggested this to centos-questions@redhat.com. So far I only got a response that I shall wait till next year for no- and low-cost RHEL offerings. Let's wait and see if I get a useful answer to my proposal.
Sorry for mentioning this issue once again here on the mailing list, but I just hope that by mentioning it often enough, the possibility of this issue/proposal being acknowledged goes up.
Hi Peter,
Thanks for mentioning it again- I would encourage anybody else who is using kernel modules that aren't included in the stock kernel to mail centos-questions@redhat.com and let us know about it.
Just on list I've read that the ongoing ability to use CentOSPlus, ELRepo, and a few miscellaneous third party drivers is the difference between being able to adopt CentOS Stream or not. My impression is that if CentOS Stream can provide some mechanism by which people will be unable to 'dnf update' their way into an unusable system for want of a compatible kernel module, that somewhat resolves the issue. I picture a SIG being formed of community members to create this and oversee the multiple implementation details that would be necessary per use case (EG, don't stop ELRepo, provide something that works smoothly with it).
Coincidentally, for anybody who uses the word "stable" or "unstable", it's always helpful when you outline what you mean by that. I'm personally aware of 4 different meanings, and suspect at least 2 more are being used here! This is a good practice for both on-list discussions and reports to centos-questions@redhat.com.
Finally, I want to mention that a number of Red Hatters are now or about to be on an end of year holiday. Consequently, some people who would normally be joining in constructive conversations are away from their email. Please nobody take this as a lack of interest or concern, they'll be back bright and early next year.
On 19/12/2020 22.42, Brendan Conoboy wrote:
On Sat, Dec 19, 2020 at 1:13 AM Peter Georg peter.georg@physik.uni-regensburg.de wrote:
On 19/12/2020 07.44, Karsten Wade wrote: I agree with you that CentOS Stream might actually be a good way forward and consider (even looking forward to) switching from CentOS Linux to CentOS Stream for the reasons you mentioned. However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower. This topic has already been mentioned several times on this mailing list by various users. So far feedback has been positive here on the list.
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility. The only way I can think of implementing this is by offering an optional "RHEL kernel repository" containing the (updated) kernel of the current RHEL minor release. Feedback to this proposal has been positive so far as well, but no concrete feedback yet. I even suggested this to centos-questions@redhat.com. So far I only got a response that I shall wait till next year for no- and low-cost RHEL offerings. Let's wait and see if I get a useful answer to my proposal.
Sorry for mentioning this issue once again here on the mailing list, but I just hope that by mentioning it often enough, the possibility of this issue/proposal being acknowledged goes up.
Hi Peter,
Thanks for mentioning it again- I would encourage anybody else who is using kernel modules that aren't included in the stock kernel to mail centos-questions@redhat.com and let us know about it.
Thanks for your support!
Just on list I've read that the ongoing ability to use CentOSPlus, ELRepo, and a few miscellaneous third party drivers is the difference between being able to adopt CentOS Stream or not. My impression is that if CentOS Stream can provide some mechanism by which people will be unable to 'dnf update' their way into an unusable system for want of a compatible kernel module, that somewhat resolves the issue. I picture a SIG being formed of community members to create this and oversee the multiple implementation details that would be necessary per use case (EG, don't stop ELRepo, provide something that works smoothly with it).
I think it is not enough to provide "some mechanism by which people will be unable to 'dnf update' their way into an unusable system" due to incompatible kernel module is enough. One also needs to be able to install a new system with a particular kernel version x that works with a 3rd party driver. This could be solved by keeping all ever released kernel version in the repository. To reduce the amount of kernel versions to store, I suggested to limit it to the version provided by the most recent minor release.
In addition the kernel version provided in a minor release might also be updated to fix security issues etc. These retain kABI compatibility (at least in most cases) to the "base" minor release version. These kernel versions are not available in CentOS Stream at any time.
Hence my suggestion has been to introduce a "RHEL kernel" repository that provides the current RHEL minor releases' kernel (and directly dependent RPMs). Whenever there is an update to the minor release's kernel (without changing the RHEL minor release), this version will be added to the "RHEL kernel" repository. I.e., versions 4.18.0-x.y, which are never being released for CentOS Stream.
Once a new RHEL minor release is available, the "RHEL kernel" repository is moved to vault (or deleted?) and we re-start with an empty "RHEL kernel" repository. We will then add the new RHEL minor release's kernel.
This explanation is somewhat complicated and a simpler, but maybe more controversal, explanation might just describe the suggested "RHEL kernel" repository as "CentOS Linux limited to the kernel (and directly dependent) packages".
Coincidentally, for anybody who uses the word "stable" or "unstable", it's always helpful when you outline what you mean by that. I'm personally aware of 4 different meanings, and suspect at least 2 more are being used here! This is a good practice for both on-list discussions and reports to centos-questions@redhat.com.
Finally, I want to mention that a number of Red Hatters are now or about to be on an end of year holiday. Consequently, some people who would normally be joining in constructive conversations are away from their email. Please nobody take this as a lack of interest or concern, they'll be back bright and early next year.
On 2020/12/20 9:03, Peter Georg wrote:
Hence my suggestion has been to introduce a "RHEL kernel" repository that provides the current RHEL minor releases' kernel (and directly dependent RPMs). Whenever there is an update to the minor release's kernel (without changing the RHEL minor release), this version will be added to the "RHEL kernel" repository. I.e., versions 4.18.0-x.y, which are never being released for CentOS Stream.
Yes, my idea resembles your suggestion. My feeling ( https://lists.centos.org/pipermail/centos-devel/2020-December/075631.html ) is "Why are we using different binary rpm files between RHEL and CentOS Linux? Why RH does not allow CentOS Linux users to use the same binary rpm files shipped for RHEL?" They are basically the same source code (the only difference should be de-branding).
If RH allows, as an exception, CentOS Linux users to use RHEL kernels, the time lag between RHEL and CentOS Linux will be significantly reduced. Also, community can convert the overhead of de-branding into the resource for improving quality of RHEL kernels, which would be a happy thing for both RH and community.
Also, "third party drivers" are not limited to hardware device drivers needed for booting a system. For example, a system could be booted without a kernel module for e.g. antivirus software's on-access scanning, but some enterprise systems are not allowed to operate without e.g. on-access scanning. Therefore, some enterprise systems cannot update kernels as soon as a new RHEL minor release becomes available. And buying RHEL's EUS/ELS license only for obtaining security fixes for kernels during that period (presumably a few months) is not an option. If updated kernels for previous RHEL minor release were available to everybody, everybody will be able to operate with all "third party drivers". Since the security requirement is getting tighter and tighter, the time lag between "ready to build" and "ready to install" should not be ignored.
Once a new RHEL minor release is available, the "RHEL kernel" repository is moved to vault (or deleted?) and we re-start with an empty "RHEL kernel" repository. We will then add the new RHEL minor release's kernel.
I hope that "RHEL kernel" for older RHEL minor releases are not deleted when a new RHEL minor release becomes available.
On Sat, Dec 19, 2020 at 8:51 PM Tetsuo Handa < from-centos@i-love.sakura.ne.jp> wrote:
On 2020/12/20 9:03, Peter Georg wrote:
Hence my suggestion has been to introduce a "RHEL kernel" repository
that provides the current
RHEL minor releases' kernel (and directly dependent RPMs). Whenever
there is an update to the
minor release's kernel (without changing the RHEL minor release), this
version will be added to
the "RHEL kernel" repository. I.e., versions 4.18.0-x.y, which are never
being released for
CentOS Stream.
Yes, my idea resembles your suggestion. My feeling ( https://lists.centos.org/pipermail/centos-devel/2020-December/075631.html ) is "Why are we using different binary rpm files between RHEL and CentOS Linux? Why RH does not allow CentOS Linux users to use the same binary rpm files shipped for RHEL?" They are basically the same source code (the only difference should be de-branding).
FWIW, we did actually discuss doing this. It boiled down to business reasons, and in particular the security of our supply chain (recent news only emphasizes this). The idea of using a single build is popular among some on our engineering team though. I'm sure it will be proposed again as things settle down, we can re-evaluate then.
If RH allows, as an exception, CentOS Linux users to use RHEL kernels, the
time lag between RHEL and CentOS Linux will be significantly reduced. Also, community can convert the overhead of de-branding into the resource for improving quality of RHEL kernels, which would be a happy thing for both RH and community.
I'm pretty sure mixing RHEL content like that would be a violation of our terms. I don't see that changing.
Also, "third party drivers" are not limited to hardware device drivers needed for booting a system. For example, a system could be booted without a kernel module for e.g. antivirus software's on-access scanning, but some enterprise systems are not allowed to operate without e.g. on-access scanning. Therefore, some enterprise systems cannot update kernels as soon as a new RHEL minor release becomes available. And buying RHEL's EUS/ELS license only for obtaining security fixes for kernels during that period (presumably a few months) is not an option. If updated kernels for previous RHEL minor release were available to everybody, everybody will be able to operate with all "third party drivers". Since the security requirement is getting tighter and tighter, the time lag between "ready to build" and "ready to install" should not be ignored.
Once a new RHEL minor release is available, the "RHEL kernel" repository
is moved to vault (or deleted?)
and we re-start with an empty "RHEL kernel" repository. We will then add
the new RHEL minor release's kernel.
I hope that "RHEL kernel" for older RHEL minor releases are not deleted when a new RHEL minor release becomes available.
I don't think we have a particular retention policy for stream though even if we did, a simple local mirror would solve your problem, right?
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
I analyzed vmcore (kernel crash dumps) when I was working at a support center as a technical staff for troubleshooting RHEL/CentOS servers. When using CentOS in enterprise servers (despite RH does not provide support for CentOS, some companies/organizations might provide fault-analysis service for CentOS), all binary/src/debuginfo rpm files are expected to be available without delay and remain available even after end of life. And
On 2020/12/20 12:11, Mike McGrath wrote:
Once a new RHEL minor release is available, the "RHEL kernel" repository is moved to vault (or deleted?) and we re-start with an empty "RHEL kernel" repository. We will then add the new RHEL minor release's kernel.
I hope that "RHEL kernel" for older RHEL minor releases are not deleted when a new RHEL minor release becomes available.
I don't think we have a particular retention policy for stream though even if we did, a simple local mirror would solve your problem, right?
since the support center I was working at and administrators who install RHEL or CentOS or Ubuntu are different companies/organizations, maintaining local mirror servers does not solve my problem.
What is sometimes troublesome and risky is that we need debuginfo rpm files. I worry that availability of src rpm and debuginfo rpm is rather unreliable. If debuginfo rpm files were unavailable, it becomes impossible to contribute like https://lkml.org/lkml/2014/9/19/230 .
RH has infrastructure for serving all these files, but CentOS's infrastructure looks poor. For example, although http://debuginfo.centos.org/8/x86_64/Packages/ contains kernel-debuginfo-4.18.0-*.rpm , http://debuginfo.centos.org/8-stream/ is currently empty.
If binary/src/debuginfo rpm files for CentOS Stream will be removed like Fedora, the support center I was working at would have to give up providing fault-analysis service for CentOS Stream. If RH is expecting CentOS Stream users to contribute with testing/debugging, I think that improving such distribution infrastructure is important.
If CentOS Linux is moved to CentOS Stream, will this availability problem be solved?
On 20.12.2020 04:42, Brendan Conoboy wrote:
Thanks for mentioning it again- I would encourage anybody else who is using kernel modules that aren't included in the stock kernel to mail centos-questions@redhat.com and let us know about it.
Out of curiosity: does a human being answer to such emails (OK to wait if you are getting tons of emails), and how much time it could take before getting a *live* human response?
So far all I see is a "bounce" message with ad blurb and promise to send newsletter on great RH plans for ex-CentOS Linux users. I do not need that, actually - I'd prefer to read real-life human being's answers.
Thanks.
On Sat, Dec 19, 2020 at 6:09 PM Konstantin Boyandin via CentOS-devel < centos-devel@centos.org> wrote:
On 20.12.2020 04:42, Brendan Conoboy wrote:
Thanks for mentioning it again- I would encourage anybody else who is using kernel modules that aren't included in the stock kernel to mail centos-questions@redhat.com and let us know about it.
Out of curiosity: does a human being answer to such emails (OK to wait if you are getting tons of emails), and how much time it could take before getting a *live* human response?
So far all I see is a "bounce" message with ad blurb and promise to send newsletter on great RH plans for ex-CentOS Linux users. I do not need that, actually - I'd prefer to read real-life human being's answers.
Right now they're aggregating data to find any blind spots we don't know about. A surprising number of requests have just been to request info when the programs are done. You might get an answer next week but I wouldn't expect anything until the new year. Lots of PTO this time of year.
-Mike
Thanks.
-- Sincerely,
Konstantin Boyandin system administrator (ProWide Labs Ltd. - IPHost Network Monitor) _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Sat, Dec 19, 2020 at 4:09 PM Konstantin Boyandin via CentOS-devel centos-devel@centos.org wrote:
On 20.12.2020 04:42, Brendan Conoboy wrote:
Thanks for mentioning it again- I would encourage anybody else who is using kernel modules that aren't included in the stock kernel to mail centos-questions@redhat.com and let us know about it.
Out of curiosity: does a human being answer to such emails (OK to wait if you are getting tons of emails), and how much time it could take before getting a *live* human response?
So far all I see is a "bounce" message with ad blurb and promise to send newsletter on great RH plans for ex-CentOS Linux users. I do not need that, actually - I'd prefer to read real-life human being's answers.
I'm not familiar with the exact plans, but I suspect the answer to this depends on the nature and volume of the feedback provided.
On 19/12/2020 21:42, Brendan Conoboy wrote:
On Sat, Dec 19, 2020 at 1:13 AM Peter Georg peter.georg@physik.uni-regensburg.de wrote:
On 19/12/2020 07.44, Karsten Wade wrote: I agree with you that CentOS Stream might actually be a good way forward and consider (even looking forward to) switching from CentOS Linux to CentOS Stream for the reasons you mentioned. However I disagree on one point: You mention CentOS Stream can cover "95% (or so)" of the current users. As long as 3rd party drivers are not working with CentOS Stream the number will be lower. This topic has already been mentioned several times on this mailing list by various users. So far feedback has been positive here on the list.
In my opinion, the only way to guarantee compatibility is by 100% kernel ABI compatibility. The only way I can think of implementing this is by offering an optional "RHEL kernel repository" containing the (updated) kernel of the current RHEL minor release. Feedback to this proposal has been positive so far as well, but no concrete feedback yet. I even suggested this to centos-questions@redhat.com. So far I only got a response that I shall wait till next year for no- and low-cost RHEL offerings. Let's wait and see if I get a useful answer to my proposal.
Sorry for mentioning this issue once again here on the mailing list, but I just hope that by mentioning it often enough, the possibility of this issue/proposal being acknowledged goes up.
Hi Peter,
Thanks for mentioning it again- I would encourage anybody else who is using kernel modules that aren't included in the stock kernel to mail centos-questions@redhat.com and let us know about it.
Just on list I've read that the ongoing ability to use CentOSPlus, ELRepo, and a few miscellaneous third party drivers is the difference between being able to adopt CentOS Stream or not. My impression is that if CentOS Stream can provide some mechanism by which people will be unable to 'dnf update' their way into an unusable system for want of a compatible kernel module, that somewhat resolves the issue. I picture a SIG being formed of community members to create this and oversee the multiple implementation details that would be necessary per use case (EG, don't stop ELRepo, provide something that works smoothly with it).
Hi Brendan,
I fear this is not really practicable. 3rd Party kernel driver updates are largely delivered for RHEL using Red Hat's Driver Update Programme framework that has existed since the early days of RHEL5 [1-4]. It is totally dependent upon the relatively stable kABI that exists in RHEL between point releases. CentOS Stream does not provide this. You are proposing we try banging a square peg into a round hole. As it stands, CentOS Stream is great as a CI development system for us to use to develop packages for the next point release of RHEL (e.g, 8.4), not for us to develop packages intended to be consumed today by users running Stream.
The solution is to make regular RHEL kernel updates available in the Stream repository, on either an opt-in or opt-out basis, depending upon one's intended use case. To suggest that a SIG or other mechanism could be implemented to regularly rebuild 3rd party content is simply not feasible for a number of reasons. Firstly, it's not simply a matter of rebuilding 3rd party drivers against each newer kernel. Many of our own drivers will require a code rebase for each rebuild or patches updating that no longer apply cleanly. This is not something that can be readily automated. Secondly, as I understand it, as we move forward and Red Hat kernel development moves towards the CentOS Stream model, we will be seeing nightly builds so this work will need to be undertaken on a daily basis which again is simply not feasible, nor sustainable. Even if we could rebuild our content on a daily basis, by the time we've updated our build systems with the latest CentOS Stream kernel, rebased , fixed and rebuilt 50+ packages, signed them, put them through QA and released them, and our global mirror network has synced, CentOS Stream will have likely released the next daily kernel update and we are back to square one. Square peg, round hole.
I am encouraged that people within Red Hat are starting to recognise and acknowledge some of the challenges we face, but the conversation seems to be stuck at the same place and we are not moving forward to discuss and implement the very workable solution which has been proposed. Can regular RHEL kernels be populated into a Stream repository, either by default or as an optional add on? If so, when can we expect this to happen?
Or maybe Red Hat's solution is for any affected CentOS Stream customers to email centos-questions@redhat.com and be given free RHEL licenses? It would be nice to have some clarity around this.
[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/htm...
[2] http://people.redhat.com/jcm/kmods/RHEL5.2/docs/RHEL5_DUP_InstallQuickstart....
[3] http://people.redhat.com/jcm/el6/dup/docs/old/pre-release/whitepaper.pdf
On Sat, Dec 19, 2020 at 4:40 PM Phil Perry pperry@elrepo.org wrote:
I fear this is not really practicable. 3rd Party kernel driver updates are largely delivered for RHEL using Red Hat's Driver Update Programme framework that has existed since the early days of RHEL5 [1-4]. It is totally dependent upon the relatively stable kABI that exists in RHEL between point releases. CentOS Stream does not provide this. You are proposing we try banging a square peg into a round hole. As it stands, CentOS Stream is great as a CI development system for us to use to develop packages for the next point release of RHEL (e.g, 8.4), not for us to develop packages intended to be consumed today by users running Stream.
Hi Phil,
Yes, it's pretty clear to all that we're going to need to develop new mechanisms to make this work.
The solution is to make regular RHEL kernel updates available in the Stream repository, on either an opt-in or opt-out basis, depending upon one's intended use case. To suggest that a SIG or other mechanism could be implemented to regularly rebuild 3rd party content is simply not feasible for a number of reasons. Firstly, it's not simply a matter of rebuilding 3rd party drivers against each newer kernel. Many of our own drivers will require a code rebase for each rebuild or patches updating that no longer apply cleanly. This is not something that can be readily automated. Secondly, as I understand it, as we move forward and Red Hat kernel development moves towards the CentOS Stream model, we will be seeing nightly builds so this work will need to be undertaken on a daily basis which again is simply not feasible, nor sustainable. Even if we could rebuild our content on a daily basis, by the time we've updated our build systems with the latest CentOS Stream kernel, rebased , fixed and rebuilt 50+ packages, signed them, put them through QA and released them, and our global mirror network has synced, CentOS Stream will have likely released the next daily kernel update and we are back to square one. Square peg, round hole.
Stream kernels + third party driver adaption logistics are complicated, but I don't think they're impossible. We probably have a different understanding of the constants and the variables in this equation, so let's keep talking about it and see what we can do. Again, a lot of Red Hatters are going to be absent the next couple weeks, but that's just bad timing, not disinterest.
I am encouraged that people within Red Hat are starting to recognise and acknowledge some of the challenges we face, but the conversation seems to be stuck at the same place and we are not moving forward to discuss and implement the very workable solution which has been proposed. Can regular RHEL kernels be populated into a Stream repository, either by default or as an optional add on? If so, when can we expect this to happen?
My reading of Q14 on https://centos.org/distro-faq/ suggests this would not be allowed.
Or maybe Red Hat's solution is for any affected CentOS Stream customers to email centos-questions@redhat.com and be given free RHEL licenses? It would be nice to have some clarity around this.
Whether you're blocked from using CentOS Stream or RHEL, we definitely want to hear the details, so we can make one or both viable options.
On 12/19/20 4:39 PM, Phil Perry wrote:
Many of our own drivers will require a code rebase for each rebuild or patches updating that no longer apply cleanly. This is not something that can be readily automated. Secondly, as I understand it, as we move forward and Red Hat kernel development moves towards the CentOS Stream model, we will be seeing nightly builds so this work will need to be undertaken on a daily basis which again is simply not feasible, nor sustainable. Even if we could rebuild our content on a daily basis, by the time we've updated our build systems with the latest CentOS Stream kernel, rebased , fixed and rebuilt 50+ packages, signed them, put them through QA and released them, and our global mirror network has synced, CentOS Stream will have likely released the next daily kernel update and we are back to square one.
What are your thoughts on publishing a "kernel" repo that includes not only the 3rd party modules, but the kernel as well? CentOS Stream users who need 3rd party modules could blacklist kernel packages from the primary repos, and get both kernels and modules from the elrepo kernel repository. If the RHEL kernel release cadence is the target that you think elrepo can support, then you could rebuild the RHEL kernel srpm rather than a CentOS Stream one (or use the Stream package that eventually makes it into an RHEL release).
I think you'd be able to continue supporting both RHEL and CentOS Stream with a single repo by rebuilding one more package.
On 20/12/2020 22:40, Gordon Messmer wrote:
On 12/19/20 4:39 PM, Phil Perry wrote:
Many of our own drivers will require a code rebase for each rebuild or patches updating that no longer apply cleanly. This is not something that can be readily automated. Secondly, as I understand it, as we move forward and Red Hat kernel development moves towards the CentOS Stream model, we will be seeing nightly builds so this work will need to be undertaken on a daily basis which again is simply not feasible, nor sustainable. Even if we could rebuild our content on a daily basis, by the time we've updated our build systems with the latest CentOS Stream kernel, rebased , fixed and rebuilt 50+ packages, signed them, put them through QA and released them, and our global mirror network has synced, CentOS Stream will have likely released the next daily kernel update and we are back to square one.
What are your thoughts on publishing a "kernel" repo that includes not only the 3rd party modules, but the kernel as well? CentOS Stream users who need 3rd party modules could blacklist kernel packages from the primary repos, and get both kernels and modules from the elrepo kernel repository. If the RHEL kernel release cadence is the target that you think elrepo can support, then you could rebuild the RHEL kernel srpm rather than a CentOS Stream one (or use the Stream package that eventually makes it into an RHEL release).
I think you'd be able to continue supporting both RHEL and CentOS Stream with a single repo by rebuilding one more package.
Yes, it is the RHEL kernel(s) that are required and a separate kernel repository is exactly what I've been requesting, on either an opt-in or opt-out basis.
Technically, rebuilding the RHEL kernel(s) is not something that is difficult to do, but this is something I would really like to see Red Hat / CentOS make available as part of Stream since a wider need for it has been identified. This is something that has been repeatedly requested. It's a really small concession that is a really big deal to any potential CentOS Stream users reliant on 3rd party drivers. It is something I would have liked to have seen the CentOS board negotiate as part of the discussions, but as these all occurred behind closed doors, without any prior knowledge of or consultation with the community, it was not possible to highlight such issues in advance or have any input into those discussions. This is not a difficult thing for Red Hat to fix if they have the will. Lets give the folks at Red Hat / CentOS time to offer up some solutions.
If Red Hat are unable to provide a solution, then it becomes a simple logistical issue for users to pull in the kernels that will exist as part of other community rebuild projects (e.g, to run a Rocky or whatever kernel on Stream).
On Sun, Dec 20, 2020 at 8:01 PM Phil Perry pperry@elrepo.org wrote:
Technically, rebuilding the RHEL kernel(s) is not something that is difficult to do, but this is something I would really like to see Red Hat / CentOS make available as part of Stream since a wider need for it has been identified. This is something that has been repeatedly
Given that subtle gcc changes sometimes interact unfortunately with kernel compilation, I *would not* do kernel compilation from CentOS 8 Stream based systems. Locking down the build environment for kernels is especially vital because it can be *very* difficult to sort out the subtleties of the "include" file changes of critical tools like gcc.
If Red Hat are unable to provide a solution, then it becomes a simple logistical issue for users to pull in the kernels that will exist as part of other community rebuild projects (e.g, to run a Rocky or whatever kernel on Stream).
Our friends at Red Hat already do something like this for "real-time" kernels referred to as "kernel-rt", which is where I last encountered the need for hand-tuning kernels. It's in the "centos/8/rt/" in the yum merrors.
Hi Phil,
Thanks for your thoughtful and thorough responses. From my perspective, Brendan is a great person to be speaking with here, I'm glad you're both engaging in this thread.
I have a side point to address that comes up, about decision making in open source projects. It's a topic I didn't address directly in my blog post.
On 12/20/20 5:01 PM, Phil Perry wrote:
It is something I would have liked to have seen the CentOS board negotiate as part of the discussions, but as these all occurred behind closed doors, without any prior knowledge of or consultation with the community, it was not possible to highlight such issues in advance or have any input into those discussions. This is not a difficult thing for Red Hat to fix if they have the will. Lets give the folks at Red Hat / CentOS time to offer up some solutions.
I appreciate you see more time is needed to come up with solutions.
Frankly, bringing incomplete solutions to the community and wider user base is the intended and correct approach.
For products and customers, there is an expectation of announcements and offerings being of a certain finish and polish.
For open source projects, there has to be community input and wider feedback for solutions to end up actually working for people. It's an important part of how the innovation happens.
Right here in this thread, we are doing the work to help create solutions, so thanks again to all.
You reasonably ask why this couldn't have been done from the beginning as an open discussion, rather than springing a seemingly-disappointing surprise on people.
My response is that a large portion of the discussion has been happening in the open regarding our plans for the Project and the distro once we all joined forces in 2014.
For example:
* There have been different "CentOS Distros" for almost seven years. * Plans for evolution were discussed openly and often; the conversations around the numbering scheme for CentOS Linux 7 were all about that. * Open build and CI/CD systems were put in place. * Layered projects were invited to own and maintain any change to the core distro and still call it "CentOS". * Visibility into the RHEL 8 build/rebuild showed where RHEL Engineering wanted to go with building and shipping the distro. * This is in addition to many other smaller communications, presentations, and so forth.
During those seven years, if you spent time close to the Project, there was sometimes a cloud over a discussion, which was, "when is Red Hat going to drop the other shoe?"—the first shoe being the joining forces in 2014.
I think there have never been any shoes being dropped. I think the core intention around solving the issues of open, available, sustainable, and stable-enough-for-different-use-cases is what this has been about since 31 October 1994.
In January 2014 Red Hat acknowledged the world's association of the CentOS brand with Red Hat. Red Hat became the stewards of the brand, empowering the CentOS Board to take good care of that brand out here in the day-to-day.
In all of that, we have conducted as much of the discussions and decision making in the open, communal spaces as possible at the time. But there are also discussions and decisions to have in private business space. That is the nature of business.
Red Hat (as a business) invited the non-Red Hat and non-managers who are Board Directors to have discussions about RHEL Engineering development, business, and resource plans for the coming years. The conversations contained non-public topics. The result of those discussions is where we are today. We announced to begin having open conversations as soon as we could, thanks to the help of a lot of people. If it could have been one minute sooner or have included one other deserving contributor in those discussions, we would have done so.
We have all trusted the code over the years and the people in the CentOS Project and Red Hat to give us a Linux that is quite good enough for our needs. In making decisions, the Board have listened to and choosen to trust the Red Hatters who brought their requests to us. Similarly, we all trust the Linux kernel developers with our lives and livelihood.
I wrote near the top "seemingly-disappointing surprise", and I want to acknowledge while there is a very real disappointment people feel, the question is still open if this news turns out to be disappointing by the time Dec 2021 arrives.
The major difference for me in terms of "can we make a CentOS distro that is open/available/sustainable/stable-enough" is that for the first time since before 2003, Red Hat Engineering have the community distro (CentOS Stream now, Red Hat Linux back then) as a top-line priority. That is why I am so hopeful this can succeed for all of us.
Best regards,
- Karsten
On Sat, Dec 19, 2020 at 1:44 AM Karsten Wade kwade@redhat.com wrote:
I wrote a blog post to share with you:
https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platfo...
Below is a fair summary of the blog post, but I encourage you to read the whole thing for the context around the "availability gap" and the "openness gap":
Hi Karsten:
It's good to hear your perspective. I understand you are trying to do something noble and with value.
However, these are significant reasons why CentOS Linux is superior to CentOS Stream:
1. Bug-for-bug compatibility with RHEL. This is important or a variety of reasons, particularly including reproducibility. If something works in CentOS 8 Stream, but fails in RHEL 8, this, or if it works in RHEL 8, but fails in CentOS 8 Stream, this means any testing efforts are invalid. In strange cases which happen in real life - code that relies on bugs will break if the bug is fixed.
2. Minor release milestones to stabilize branches. We have breakage with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
3. CentOS brand. CentOS was just getting recognized by vendors as existing by vendors who have install scripts and runtime scripts that literally say things like "if /etc/system-release doesn't contain a recognized string, then fail". I get questions like "can we use Ubuntu or CentOS?" There is no guarantee that CentOS 8 Stream will be recognized by these vendors ever. The term wishful thinking comes to mind for me.
I don't agree with you that CentOS cannot be two things. It's quite normal for most projects to have an "upstream" and a "LTS" branch. This seems like an after the fact justification for some compromises that were made behind closed doors.
I don't think this decision is in tune with what the CentOS users want. CentOS 8 Stream addresses a set of requirements that CentOS did not address previously, but it does so by abandoning the very reason that CentOS existed in the first place. If you wanted to know for sure - you would take a referendum. I think there is a reason why no referendum was taken.
Personally, I think:
1. CentOS 8 Stream should have been called RHEL 8 Stream. 2. CentOS 8 should have continued to exist until a suitable replacement was provided, with input from the community.
The choice to make the decision without consultation with the community, is a pretty major violation of trust. No matter the intent - no matter the impossible situation you may have felt was thrust upon you, to even act like CentOS belongs to the community would require some sort of public discussion on the matter. By choosing to proceed without this discussion, the people involved made it clear that the opinion of the community does not matter to your decision making process.
On Sat, Dec 19, 2020 at 4:34 AM Mark Mielke mark.mielke@gmail.com wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
I should provide real-life examples for you to consider. I will just a pick few that come to mind without thinking too hard.
The first example, is EL 7.8 and the GlusterFS update. This broke our loadbuild process across multiple product teams. In this particular case, several teams build a custom version of Qemu as part of their Yocto build process. The error showed up like this:
build 01-Dec-2020 08:05:20 | .../tmp/work/x86_64-linux/qemu-native/2.7.0-r1/qemu-2.7.0/block/gluster.c: In function ‘qemu_gluster_truncate’: build 01-Dec-2020 08:05:20 | .../tmp/work/x86_64-linux/qemu-native/2.7.0-r1/qemu-2.7.0/block/gluster.c:1000:5: error: too few arguments to function ‘glfs_ftruncate’ build 01-Dec-2020 08:05:20 | ret = glfs_ftruncate(s->fd, offset); build 01-Dec-2020 08:05:20 | ^ build 01-Dec-2020 08:05:20 | In file incl
GlusterFS changed the function glfs_ftruncate to require a second argument. Red Hat chose to upgrade GlusterFS as part of EL 7.8. Qemu 2.7.0 doesn't know about this change. Builds failed for three of our product teams.
By design, we did early adopter testing of 7.8 before mass upgrading everyone, and we discovered this problem, and were able to get code changes into the various product releases. In this case, we chose to disable GlusterFS from the Qemu build process as GlusterFS was an accidental dependency. Once it was disabled, the above code stopped being compiled, and the emergency was averted.
Now, imagine the same scenario with "CentOS 8 Stream". Are you expecting that users will perform each set up their own milestone release processes? Are we going to test every single package that gets released to "CentOS 8 Stream" incrementally to ensure that our systems don't experience massive breakage?
Another example that hit us with EL 7.7, is an update to elfutils that was incompatible with binutils and gcc:
https://wiki.gentoo.org/wiki/Binutils_2.32_upgrade_notes/elfutils_0.175:_una...
In this case, we still don't have a solution - but are downgrading elfutils for 7.7+ until we do decide what to do about it. Still, we were able to detect it in our 7.7 early adopter testing, and decide on a temporary solution *before* rolling it out to the users.
I mentioned this problem earlier this week:
https://bugzilla.redhat.com/show_bug.cgi?id=1489542
This was a change to autofs in EL 7.4 that caused it to drop automount maps every 10 minutes in our environment, causing actual breakage when paths were accessing during these intervals. Again, we were able to detect this in our EL 7.4 early adopter testing, and defer roll out of EL 7.4 until after we came up with a solution.
These are just a few of the regular things that we encounter all of the time. I don't see how CentOS 8 Stream can address these, unless you promise to treat CentOS 8 Stream as a minor release, and never introduce changes in API, new features, or changes in behaviour.
For all of these, when upstream or downstream - we contribute back analysis, bug reports, and fixes. However, I cannot be deploying CentOS 8 Stream in our environment. It wouldn't even be an option. We would choose something else.
I think when you say "95% of users would be ok with CentOS 8 Stream", you are talking about basic usage. You are not talking about Enterprise use cases.
-- Mark Mielke mark.mielke@gmail.com
On 19/12/2020 10:09, Mark Mielke wrote:
On Sat, Dec 19, 2020 at 4:34 AM Mark Mielke mark.mielke@gmail.com wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
I should provide real-life examples for you to consider. I will just a pick few that come to mind without thinking too hard.
The first example, is EL 7.8 and the GlusterFS update. This broke our loadbuild process across multiple product teams. In this particular case, several teams build a custom version of Qemu as part of their Yocto build process. The error showed up like this:
build 01-Dec-2020 08:05:20 | .../tmp/work/x86_64-linux/qemu-native/2.7.0-r1/qemu-2.7.0/block/gluster.c: In function ‘qemu_gluster_truncate’: build 01-Dec-2020 08:05:20 | .../tmp/work/x86_64-linux/qemu-native/2.7.0-r1/qemu-2.7.0/block/gluster.c:1000:5: error: too few arguments to function ‘glfs_ftruncate’ build 01-Dec-2020 08:05:20 | ret = glfs_ftruncate(s->fd, offset); build 01-Dec-2020 08:05:20 | ^ build 01-Dec-2020 08:05:20 | In file incl
GlusterFS changed the function glfs_ftruncate to require a second argument. Red Hat chose to upgrade GlusterFS as part of EL 7.8. Qemu 2.7.0 doesn't know about this change. Builds failed for three of our product teams.
By design, we did early adopter testing of 7.8 before mass upgrading everyone, and we discovered this problem, and were able to get code changes into the various product releases. In this case, we chose to disable GlusterFS from the Qemu build process as GlusterFS was an accidental dependency. Once it was disabled, the above code stopped being compiled, and the emergency was averted.
Now, imagine the same scenario with "CentOS 8 Stream". Are you expecting that users will perform each set up their own milestone release processes? Are we going to test every single package that gets released to "CentOS 8 Stream" incrementally to ensure that our systems don't experience massive breakage?
Another example that hit us with EL 7.7, is an update to elfutils that was incompatible with binutils and gcc:
https://wiki.gentoo.org/wiki/Binutils_2.32_upgrade_notes/elfutils_0.175:_una...
In this case, we still don't have a solution - but are downgrading elfutils for 7.7+ until we do decide what to do about it. Still, we were able to detect it in our 7.7 early adopter testing, and decide on a temporary solution *before* rolling it out to the users.
I mentioned this problem earlier this week:
https://bugzilla.redhat.com/show_bug.cgi?id=1489542
This was a change to autofs in EL 7.4 that caused it to drop automount maps every 10 minutes in our environment, causing actual breakage when paths were accessing during these intervals. Again, we were able to detect this in our EL 7.4 early adopter testing, and defer roll out of EL 7.4 until after we came up with a solution.
These are just a few of the regular things that we encounter all of the time. I don't see how CentOS 8 Stream can address these, unless you promise to treat CentOS 8 Stream as a minor release, and never introduce changes in API, new features, or changes in behaviour.
For all of these, when upstream or downstream - we contribute back analysis, bug reports, and fixes. However, I cannot be deploying CentOS 8 Stream in our environment. It wouldn't even be an option. We would choose something else.
I think when you say "95% of users would be ok with CentOS 8 Stream", you are talking about basic usage. You are not talking about Enterprise use cases.
Your post illustrates nicely where CentOS Stream fits into the ecosystem. Running CentOS Stream would allow your developers early opportunity to identify, feed back, fix and test any issues _before_ they affect your RHEL production systems. This is great as long as you are running RHEL on your production systems and using CentOS Stream as your CI development system. Clearly CentOS Stream is not suitable as a replacement from your RHEL production systems, and as you have previously stated, if you can not afford the RHEL licensing costs then you must look elsewhere.
Le 19/12/2020 à 10:34, Mark Mielke a écrit :
On Sat, Dec 19, 2020 at 1:44 AM Karsten Wade kwade@redhat.com wrote:
I wrote a blog post to share with you:
https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platfo...
Below is a fair summary of the blog post, but I encourage you to read the whole thing for the context around the "availability gap" and the "openness gap":
Hi Karsten:
It's good to hear your perspective. I understand you are trying to do something noble and with value.
However, these are significant reasons why CentOS Linux is superior to CentOS Stream:
- Bug-for-bug compatibility with RHEL. This is important or a variety
of reasons, particularly including reproducibility. If something works in CentOS 8 Stream, but fails in RHEL 8, this, or if it works in RHEL 8, but fails in CentOS 8 Stream, this means any testing efforts are invalid. In strange cases which happen in real life - code that relies on bugs will break if the bug is fixed.
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
- CentOS brand. CentOS was just getting recognized by vendors as
existing by vendors who have install scripts and runtime scripts that literally say things like "if /etc/system-release doesn't contain a recognized string, then fail". I get questions like "can we use Ubuntu or CentOS?" There is no guarantee that CentOS 8 Stream will be recognized by these vendors ever. The term wishful thinking comes to mind for me.
I don't agree with you that CentOS cannot be two things. It's quite normal for most projects to have an "upstream" and a "LTS" branch. This seems like an after the fact justification for some compromises that were made behind closed doors.
I don't think this decision is in tune with what the CentOS users want. CentOS 8 Stream addresses a set of requirements that CentOS did not address previously, but it does so by abandoning the very reason that CentOS existed in the first place. If you wanted to know for sure
- you would take a referendum. I think there is a reason why no
referendum was taken.
Personally, I think:
- CentOS 8 Stream should have been called RHEL 8 Stream.
- CentOS 8 should have continued to exist until a suitable
replacement was provided, with input from the community.
The choice to make the decision without consultation with the community, is a pretty major violation of trust. No matter the intent
- no matter the impossible situation you may have felt was thrust upon
you, to even act like CentOS belongs to the community would require some sort of public discussion on the matter. By choosing to proceed without this discussion, the people involved made it clear that the opinion of the community does not matter to your decision making process.
As I previously said, a fair way of doing things would have been : "Hey guys since we Red Hat have bought CentOS, making a downstream release of RHEL is just a nonsense, It costs us time and money we can save. So let's reverse the process and make RHEL a downstream of CentOS Linux. It will now be Fedora ELN - > CentOS Stream - > CentOS Linux - > RHEL."
Red Hat would have kill all the clones by releasing CentOS Linux first, the Community would have been happy and not anger to help to get a better RHEL in the Stream process, and Red Hat folks could have put all the value of their brand and specificities in their final products, backed with a strong ecosystem they could have controlled.
Jean-Marc
On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not mimic.
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is
No. RHEL minor releases are more like source control "tags" than branches. They have stable installation images one can refer and start new deployments from, and ongoing work is branched from that tag. This has been the case since the earliest Red Hat, not even RHEL, releases since I encountered them back in 1995.
a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not mimic.
Nothing personal, but "nonsense". See above. The PXE images and kernels in the installation media, in particular, mattered. I used to publish quite large Red Hat based deployments, and you *never* simply ran "yum update" from the live update streams at the time of building your new images or testing environments. It's why I published tools like my reposync scripts, to generate a lockable internal mirror and avoid the streaming update dangers without investing in the very large cost of an RHN setup.
Having a stable repository for a full reference for updating an environment or especially build and testing environments synchronously is vital. It's also been very useful for "mock", since one can turn off the update channels and use the current of specific CentOS point releases for builidng software. If you're doing that with RHEL, well, frankly I'd use a locked down internal mirror. My reposync scripts and rsync tools are published at https://github.com/nkadel/nkadel-rsync-scripts . They're pretty easily linked to a filesystem that does hardlinks and allows snapshotting to produce a "current" and a datestamped, locked down copy of the repos, much like the old "rsnapshot" tool did for rsync backups. And I was a maintainer on that tool for a while, too.
Am 19.12.20 um 18:49 schrieb Nico Kadel-Garcia:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is
No. RHEL minor releases are more like source control "tags" than branches. They have stable installation images one can refer and start new deployments from, and ongoing work is branched from that tag. This has been the case since the earliest Red Hat, not even RHEL, releases since I encountered them back in 1995.
a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not mimic.
Nothing personal, but "nonsense". See above. The PXE images and kernels in the installation media, in particular, mattered. I used to publish quite large Red Hat based deployments, and you *never* simply ran "yum update" from the live update streams at the time of building your new images or testing environments. It's why I published tools like my reposync scripts, to generate a lockable internal mirror and avoid the streaming update dangers without investing in the very large cost of an RHN setup.
IIRC thats the reason why CentOS Linux has a kickstart directory in the repo tree.
Having a stable repository for a full reference for updating an environment or especially build and testing environments synchronously is vital. It's also been very useful for "mock", since one can turn off the update channels and use the current of specific CentOS point releases for builidng software. If you're doing that with RHEL, well, frankly I'd use a locked down internal mirror. My reposync scripts and rsync tools are published at https://github.com/nkadel/nkadel-rsync-scripts . They're pretty easily linked to a filesystem that does hardlinks and allows snapshotting to produce a "current" and a datestamped, locked down copy of the repos, much like the old "rsnapshot" tool did for rsync backups. And I was a maintainer on that tool for a while, too. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Leon
On Sat, Dec 19, 2020 at 12:50 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is
No. RHEL minor releases are more like source control "tags" than branches. They have stable installation images one can refer and start new deployments from, and ongoing work is branched from that tag. This has been the case since the earliest Red Hat, not even RHEL, releases since I encountered them back in 1995.
Exactly. Matthew: You are pretending as if RHEL beta releases, and RHEL stabilization doesn't exist for some important reason,
These gates +/- exist between RHEL Stream and RHEL:
1. Changes first go through designer testing and user testing of some sort. "Try this fix". This is typically limited to only a few users who reported the problem.
2. Changes were typically made in the "next" branch, which I believe was often +2 minor releases. This would see the first QA testing, and the start of some integration testing. Historically this has been an internal process to Red Hat, which is a problem as it also limits exposure of the QA testing to Red Hat, and if any behaviours are changing, it limits the ability for users to be ready for upstream changes coming down the pipeline. Changes may sit in this stabilization branch for weeks to months.
3. Changes would backported into the +1 minor release, and get periodically rolled into a beta program, that would allow the outside world the first glimpse of what they might be getting, including changes documented in release notes. Presumedly Red Hat is doing some form of QA and certification during this period, although this is also a bit of a black box. These release could be analyzed and if users were interested in testing, they could go through the special effort of testing. As the beta program requires some setup to participate in, many vendors do not. However, of the people that do try out the beta - presumedly at least some bugs are found and fixed, before they make it into a release. Changes may sit in this stabilization branch for weeks to months.
4. Changes are released into an official minor release, Users receive release notes that they can analyze for impact. Users can and should run early adopter programs, to ensure that the large drop of changes will not impact their workflows. Anybody who arbitrarily does "yum update" at this point is playing with fire. Red Hat QA processes are insufficient to detect and prevent every problem that might affect customers. In many cases, the problems are simply missed, and pass through the above gates without detection. In many other cases, the problems are impossible to have been predicted by Red Hat, because Red Hat does not know how people use RHEL in real-life. For example, Red Hat cannot run all our product builds before release RHEL, so how are they supposed to be able to detect if our product builds will break? This is why there are release notes and minor releases, to provide this level of insulation.
Where exactly, does CentOS 8 Stream fit into this? I'm thinking you are cutting at 3 + 4 out of the picture, and then some people on this list are essentially claiming 3 + 4 provides no value for 95% of the "users" or 95% of the "installed base" (not sure which at this point, since the numbers are quite different).
Downstream of RHEL, there are additional gates. Including what Nico is talking about, and what anybody with a significant deployment will include, which is internal repositories and internal "stable" branches of RHEL. We want all of the prior branches and gates, but we know that mistakes still make it through these gates. So, we analyze every change that comes out of RHEL, and if it might impact our workflows, we do testing of our workflows before releasing the packages to our internal systems. In my case we had a "latest" branch which included content covered by the above gates, a "testing" branch which includes content that was being tentatively approved, and a "stable" branch which includes content that was approved.
If the people in favour of abandoning CentOS 8 for CentOS 8 Stream effectively remove gates 3 + 4 from this process, you introduce unacceptable risk to our programs, and this will force us to choose some other solution. "Us" is a really big "us". I think by numbers, "us" is almost your entire community. If you abandon "us" as is described, the consequence of this is that "us" will go somewhere else. "Us" is the reason why CentOS 8 existed in the first place. Red Hat and the CentOS board can ignore "us", but they can't dictate to "us". You can't force us to use this CentOS 8 Stream merely by claiming it is not really different at all.
I see Facebook mentioned several times as evidence that CentOS 8 Stream is no big deal. I imagine that Facebook has their own stabilization processes, and they are effectively abandoning *RHEL*. I need to be clear of what I mean by this. In this scenario where CentOS 8 Stream is upstream of RHEL 8, and CentOS 8 is cancelled, the situation we are left with is that RHEL is deciding when to cut new CentOS 8 Stream milestones and stabilize them. There is a lot of emphasis on the Red Hat QA processes, and how they are very important, but I just explained that many bugs escape the Red Hat QA process, and we catch them. In effect, the Red Hat milestones are not necessarily good ones, and this is where Amazon Linux and others have gone, and it sounds like Facebook as well. The community can pick a different process for snapshot of CentOS 8 Stream and stabilization. The community can run its own beta programs separately from RHEL, effectively making RHEL obsolete. This may be hard to imagine for individual users who are accustomed to Red Hat controlling everything they are allowed to see and do, but it's actually a problem for companies that are being asked to pay $4M USD annual for RHEL subscription fees for the right to use a collection of free software. We can do our own testing and stabilization for $4M USD annually. We don't need Red Hat. And this is the problem with the Red Hat subscription model, and the problem with CentOS 8 Stream. Red Hat has priced itself out of essential markets. This isn't about "if you can't afford RHEL, you can use CentOS 8 Stream". This is about the value proposition of RHEL. If the value proposition of RHEL is lower than the value proposition of doing it ourselves, we would be financially irresponsible to choose RHEL.
If you don't think a few large companies can do as good a job as Red Hat at branching, and stabilizing - you are mistaken. This is what we do for *our* products. We know exactly how to do this. We have build systems and test systems. As described above, we catch problems that Red Hat does not catch. I didn't really comment on Mike McGraph's comment that "we are the best", because I don't really want to go there - but I would be cautious about claiming words like "best". There is a pretty significant difference between "we are highly valuable contributors" and "we are the best".
The reason we align on Red Hat, is because it is beneficial for us to do so. The QA testing and such is useful, for sure - but more useful is to have a unified "EL" environment where packages can be deployed in multiple different "EL" distributions with confidence, and vendor solutions will generally work across all "EL" distributions. If Red Hat makes this too difficult too achieve, there is an outcome that has not been described here: Red Hat will become the vendor lock-in speciality solution that nobody chooses to use.
Based upon several posts in centos-devel, It has become clear to me that many CentOS board members, CentOS contributors, and Red Hat staff, do not fully comprehend the value that is brought to the table by allowing large companies to consolidate on "EL" as a standard. This is not "EL Next", this is "EL". It is fine for this value to not be recognized - but when failure to recognize this value has the consequence of CentOS/RHEL actually *removing* CentOS as the common standard around which all EL distributions align, we will simply pick a new standard. The rest of us can work together just fine, and we don't need Red Hat.
It will just be a terrible shame. None of us wants to see this. We want Red Hat to be the unifying force. We want to help Red Hat. But, if Red Hat doesn't want us around any more - it will be more like a break up with a first girlfriend. Life goes on.
It's up to you, CentOS/RHEL. You basically have these choices:
1. CentOS 8 stays, and "EL" distributions use CentOS 8 as a common baseline.
2. RHEL 8, including minor releases, stays accessible in source form, so that the community can build a clone that aligns with RHEL 8 for the purposes of *standards*, not for the purpose of *cheap*.
3. RHEL 8 hides the source behind a legal framework that makes it difficult or impossible for "bug-for-bug" compatibility to be maintained. Either the community reverse engineers this (for example, OpenJDK 8 is kind of like this), or the community creates its own baselines and ignores RHEL. RHEL becomes irrelevant.
It will just be a terrible shame. None of us wants to see this. We want Red Hat to be the unifying force. We want to help Red Hat. But, if Red Hat doesn't want us around any more - it will be more like a break up with a first girlfriend. Life goes on.
It's up to you, CentOS/RHEL. You basically have these choices:
- CentOS 8 stays, and "EL" distributions use CentOS 8 as a common
baseline.
- RHEL 8, including minor releases, stays accessible in source form,
so that the community can build a clone that aligns with RHEL 8 for the purposes of *standards*, not for the purpose of *cheap*.
- RHEL 8 hides the source behind a legal framework that makes it
difficult or impossible for "bug-for-bug" compatibility to be maintained. Either the community reverse engineers this (for example, OpenJDK 8 is kind of like this), or the community creates its own baselines and ignores RHEL. RHEL becomes irrelevant.
For some days now I was thinking exactly about this:
What is Red Hat going to do to stop rebuilders like Oracle in future?
I mean, it doesn't make sense at all to stop CentOS Linux, whose users are for sure the most loyal to Red Hat, and let the big profit makers like Oracle go on and even help them to improve their profits.
So, what are you, Red Hat, going to do to prevent this?
I'm sure you have plans and I'm interested to hear how these plans look like.
I'm really sure Red Hat has to stop rebuilders somehow, otherwise a large number of CentOS Linux users will just move on to another clone and nobody will use CentOS Stream in the end. That would be a real lose-lose situation.
Regards, Simon
On Sun, Dec 20, 2020 at 8:08 AM Simon Matter simon.matter@invoca.ch wrote:
It will just be a terrible shame. None of us wants to see this. We want Red Hat to be the unifying force. We want to help Red Hat. But, if Red Hat doesn't want us around any more - it will be more like a break up with a first girlfriend. Life goes on.
It's up to you, CentOS/RHEL. You basically have these choices:
- CentOS 8 stays, and "EL" distributions use CentOS 8 as a common
baseline.
- RHEL 8, including minor releases, stays accessible in source form,
so that the community can build a clone that aligns with RHEL 8 for the purposes of *standards*, not for the purpose of *cheap*.
- RHEL 8 hides the source behind a legal framework that makes it
difficult or impossible for "bug-for-bug" compatibility to be maintained. Either the community reverse engineers this (for example, OpenJDK 8 is kind of like this), or the community creates its own baselines and ignores RHEL. RHEL becomes irrelevant.
For some days now I was thinking exactly about this:
What is Red Hat going to do to stop rebuilders like Oracle in future?
I mean, it doesn't make sense at all to stop CentOS Linux, whose users are for sure the most loyal to Red Hat, and let the big profit makers like Oracle go on and even help them to improve their profits.
So, what are you, Red Hat, going to do to prevent this?
I'm sure you have plans and I'm interested to hear how these plans look like.
I'm really sure Red Hat has to stop rebuilders somehow, otherwise a large number of CentOS Linux users will just move on to another clone and nobody will use CentOS Stream in the end. That would be a real lose-lose situation.
Regards, Simon
We're not stopping rebuilders. We're just no longer sponsoring them. This is in line with the rest of our products.
-Mike
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 12/20/20 5:41 PM, Mike McGrath wrote:
We're not stopping rebuilders. We're just no longer sponsoring them. This is in line with the rest of our products.
Then allowing people and companies from CentOS community to sponsor/donate should not be a problem, and it would be smart PR move. At least for CentOS 8 until 2029...
On 12/19/20 9:49 AM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is
No. RHEL minor releases are more like source control "tags" than branches.
I promise: You do not understand RHEL better than Matthew Miller does. Just stop for a moment and listen to him. You might learn something new.
CentOS point releases are more like source control tags than branches. If you have only used CentOS and not RHEL, then I can see why you might be confused about this point. You are accurately describing CentOS.
RHEL, however, branches at point releases. You can pin an RHEL host to a specific point release and continue to get only security and bug fixes, and no new features (which are introduced only at point releases). You can't do that with CentOS, because there's only one branch.
The difference is the same as source control. When you check out a branch, you're going to get content not at a specific point in time, but all of the latest changes to that branch since it was made. That's what you'll get from RHEL if you install a specific point release, pin it, and apply updates. It's meaningful and supported to have an RHEL 7.5 host today. That host can be fully up to date and secure, and still be running the 7.5 branch. When you check out a tag, however, you're getting a specific point in time and no change since. That's what you get when you install an old CentOS point release. CentOS point releases don't really have any other meaning. If you have a host running 7.5.1804, that just means that you've stopped applying updates, and that's not supported by anyone.
a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not mimic.
Nothing personal, but "nonsense". See above. The PXE images and kernels in the installation media, in particular, mattered. I used to publish quite large Red Hat based deployments, and you *never* simply ran "yum update" from the live update streams at the time of building your new images or testing environments. It's why I published tools like my reposync scripts, to generate a lockable internal mirror and avoid the streaming update dangers without investing in the very large cost of an RHN setup.
You're not making the argument that you intend to make.
RHEL supports pinning hosts to specific point releases in order to ensure that they get only bug and security fixes for their lifetime. Satellite supports creating views for fully reproducible environments. RHEL supports these things out of the box, while CentOS users need to roll their own solution.
You just ridiculed Matthew, and then repeated the point that he was making, using your own words.
On Sat, Dec 19, 2020 at 4:15 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 9:49 AM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is
No. RHEL minor releases are more like source control "tags" than branches.
I promise: You do not understand RHEL better than Matthew Miller does. Just stop for a moment and listen to him. You might learn something new.
This is false. Nico is describing a similar system to what we do.
CentOS point releases are more like source control tags than branches. If you have only used CentOS and not RHEL, then I can see why you might be confused about this point. You are accurately describing CentOS.
This is false. I don't think you understand the RHEL release process. Check my other post. Red Hat has internal branching that you were unaware of (apparently), and CentOS is using the later branch, not the earlier branch. You can clearly see when this happens when RHEL names the package ".el7_6" for a package that is newly introduced in RHEL 7.6. Normally, they just use ".el7" if it is in the latest branch, but if they backport to the branch that CentOS picks up, it shows up as ".el7_6" in the RHEL repositories.
RHEL, however, branches at point releases. You can pin an RHEL host to a specific point release and continue to get only security and bug fixes, and no new features (which are introduced only at point releases). You can't do that with CentOS, because there's only one branch.
This is partially false. Red Hat branched *before* the point releases, and CentOS picks these up. However, Red Hat also continues with EUS branches *after* the next minor release is cut.
The difference is the same as source control. When you check out a branch, you're going to get content not at a specific point in time, but all of the latest changes to that branch since it was made. That's what you'll get from RHEL if you install a specific point release, pin it, and apply updates. It's meaningful and supported to have an RHEL 7.5 host today. That host can be fully up to date and secure, and still be running the 7.5 branch. When you check out a tag, however, you're getting a specific point in time and no change since. That's what you get when you install an old CentOS point release. CentOS point releases don't really have any other meaning. If you have a host running 7.5.1804, that just means that you've stopped applying updates, and that's not supported by anyone.
This conclusion is false. It is ignoring that Red Hat branched prior to the cut of the release, before CentOS saw it, and it is ignoring that Red Hat maintains the branch accessible to CentOS until after the next release is cut.
On 12/19/20 1:25 PM, Mark Mielke wrote:
On Sat, Dec 19, 2020 at 4:15 PM Gordon Messmer gordon.messmer@gmail.com wrote:
CentOS point releases are more like source control tags than branches. If you have only used CentOS and not RHEL, then I can see why you might be confused about this point. You are accurately describing CentOS.
This is false. I don't think you understand the RHEL release process. Check my other post. Red Hat has internal branching that you were unaware of (apparently), and CentOS is using the later branch, not the earlier branch.
Sure, but this conversation isn't about the specific mechanics of RHEL's branching. Matthew pointed out that minor releases of RHEL are branches, and that CentOS releases have no branches. Nico made an inaccurate analogy to VCS tags and branches. Matthew is correct, though: RHEL major releases have branches and CentOS releases do not. (Or, they have just one branch. Whatever phrasing you prefer.)
On Sat, Dec 19, 2020 at 6:05 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 1:25 PM, Mark Mielke wrote:
On Sat, Dec 19, 2020 at 4:15 PM Gordon Messmer gordon.messmer@gmail.com wrote:
CentOS point releases are more like source control tags than branches. If you have only used CentOS and not RHEL, then I can see why you might be confused about this point. You are accurately describing CentOS.
This is false. I don't think you understand the RHEL release process. Check my other post. Red Hat has internal branching that you were unaware of (apparently), and CentOS is using the later branch, not the earlier branch.
Sure, but this conversation isn't about the specific mechanics of RHEL's branching. Matthew pointed out that minor releases of RHEL are branches, and that CentOS releases have no branches. Nico made an inaccurate analogy to VCS tags and branches. Matthew is correct, though: RHEL major releases have branches and CentOS releases do not. (Or, they have just one branch. Whatever phrasing you prefer.)
The problem with this conclusion, is that you are ignoring the reality of approximately (in the example of something reaching RHEL X.Y):
1) RHEL Master (days?) -> RHEL X.N+2 Staging (weeks to months)-> RHEL X.N+1 Staging or Beta (weeks to months) -> RHEL X.N Stable (weeks to months) -> CentOS X.N Stable (weeks to months)
And the claim of several people on this is that the following is "no big deal":
1) RHEL Master (days?) -> CentOS X Stream (days?)
So you can say that CentOS only has one branch - but that is only if you ignore what processes and gates existed prior to that branch. I think the conclusion that CentOS 7 was just one branch is misleading at the very least, and false in the only picture that matters.
On Sat, Dec 19, 2020 at 6:13 PM Mark Mielke mark.mielke@gmail.com wrote:
The problem with this conclusion, is that you are ignoring the reality of approximately (in the example of something reaching RHEL X.Y):
- RHEL Master (days?) -> RHEL X.N+2 Staging (weeks to months)-> RHEL
X.N+1 Staging or Beta (weeks to months) -> RHEL X.N Stable (weeks to months) -> CentOS X.N Stable (weeks to months)
And the claim of several people on this is that the following is "no big deal":
- RHEL Master (days?) -> CentOS X Stream (days?)
So you can say that CentOS only has one branch - but that is only if you ignore what processes and gates existed prior to that branch. I think the conclusion that CentOS 7 was just one branch is misleading at the very least, and false in the only picture that matters.
I presumed everybody understood this, but perhaps I was in an error. In order to ensure we are all on the same page here, let's consider a real-life scenario.
https://access.redhat.com/articles/3078
This shows:
RHEL 8.3 - 4.18.0-240 RHEL 8.2 - 4.18.0-193 RHEL 8.1 - 4.18.0-147
Let's pick RHEL 8.2 as an example. Once the release was branched, it enters a stabilization and testing period ("staging" period where it remains private, and then a "beta" period where it might be formally announced and made available to the public). During this time - upstream has already moved on with -194, -195, -196, ... eventually reached -240 for RHEL 8.3.
If any problems are encountered during this stabilization period or during the life of the minor release, or - if any high priority bug fix, such as a security fix becamse available upstream and are important enough to backport, then *only* the changes necessary will be backported. In the RHEL 8.2 case, this would become 4.18.0-193.x.
In the case of CentOS 8.2, this included:
https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.... https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.... https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.... https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.... https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.... https://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193....
Upgrades within the minor release, such as .14.2 -> .19.1, would be considered "safer", because rather than picking up *all* the upstream changes - only the changes that were considered important enough, and safe enough, to backport to the RHEL 8.2 stream, were made available in RHEL 8.2, and then made available in CentOS 8.2.
With CentOS 8 Stream, none of the above happens any more. Instead, you will get -241, -242, -243, -244, ... there is no stable version. There is no baseline to align against or certify against. There is only "LATEST". Your choice is to pick up "LATEST", or stay behind. No security patches. No important bug fixes. You must stay exactly where you are until *all* of your vendors provide support for the new kernel.
In many cases, such as user space API, the public interfaces make an attempt to be forwards and backwards compatible. However, in many cases - this fails. If I build something for EL 8.2, it will *probably* work in EL 8.3. However, if I build something in EL 8.3, there is no guarantee it will work in EL 8.2. The kernel is particularly problematic, because the kernel API is not covered by the same compatibility criteria, and things like EL 7.5 can introduced updates to the VFS layer to support OverlayFS, and then break any systems that hook into the VFS layer, such as in our case for RHEL 7.5 - IBM Rational ClearCase. Consequently, we need to wait for IBM to produce a version of ClearCase that supports RHEL 7.5, which is only normally expected 90 days after RHEL 7.5 is released.
This same thing happens all over EL. It's actually difficult for me to comprehend that people think this is no big deal, but I understand that for me this is a regular problem, and for many of the people posting, it seems you may not be fully aware. But, I think I'm describing a problem that is known to a majority of the users of EL, but perhaps not to the developers of CentOS, and this may be the fundamental problem of how everything went wrong.
On Sat, Dec 19, 2020 at 6:05 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 1:25 PM, Mark Mielke wrote:
On Sat, Dec 19, 2020 at 4:15 PM Gordon Messmer gordon.messmer@gmail.com wrote:
CentOS point releases are more like source control tags than branches. If you have only used CentOS and not RHEL, then I can see why you might be confused about this point. You are accurately describing CentOS.
This is false. I don't think you understand the RHEL release process. Check my other post. Red Hat has internal branching that you were unaware of (apparently), and CentOS is using the later branch, not the earlier branch.
Sure, but this conversation isn't about the specific mechanics of RHEL's branching. Matthew pointed out that minor releases of RHEL are branches, and that CentOS releases have no branches. Nico made an inaccurate analogy to VCS tags and branches. Matthew is correct, though: RHEL major releases have branches and CentOS releases do not. (Or, they have just one branch. Whatever phrasing you prefer.)
I want to make sure we are clear about this, and why the mechanics matter, especially in the context of "CentOS 8 Stream":
CentOS has a branch, which is provided by Red Hat. This is not a periodic dump of "RHEL Latest", but it is a periodic import of an RHEL branch. The importance of this is that what you are seeing on the "c7" branch (in the case of CentOS 7), is a flattened representation from a more complex branching strategy. Technically you can call this "one branch", but this is a very superficial view of what is going on, that glosses over super important details.
RHEL 7.8 is a separate branch from RHEL 7.9. While RHEL 7.8 was the latest minor release published, Red Hat was periodically importing content and de-branding from RHEL 7.8 - not from RHEL Latest. Once RHEL 7.9 was released, the RHEL 7.9 content was imported onto the CentOS 7 branch, and de-branded.
Let's look at Firefox ESR for an example of this in action:
https://git.centos.org/rpms/firefox/commits/c7
Scroll down to where el7_8 becomes el7_9, and see these two commits:
812ff3 import firefox-68.12.0-1.el7_8 - https://git.centos.org/rpms/firefox/c/812ff34cfe1f0f157d1af7ba2375167451be82... e98f2c import firefox-78.3.0-1.el7_9 - https://git.centos.org/rpms/firefox/c/e98f2c4cf130b5dd05e5cb7bb86f0a1cccea78...
In the first commit, see how the changelog has this entry at the top:
* Thu Aug 20 2020 Jan Horak jhorak@redhat.com - Update to 68.12.0 build1
Now check the second commit and see how the above lines are deleted (and one date is even updated, strangely!) and replaced with:
* Fri Sep 18 2020 Jan Horak jhorak@redhat.com - Update to 78.3.0 build1
* Tue Aug 18 2020 Jan Horak jhorak@redhat.com - 78.2.0-3 - Update to 78.2.0 build1
* Fri Jul 24 2020 Jan Horak jhorak@redhat.com - Update to 68.11.0 build1
We don't know what CentOS 8 Stream will look like for real - but my presumption without any evidence to the contrary, would be that:
According to the CentOS 8 model, Firefox ESR 68.x would have been maintained until Aug 20, 2020 when the final version of 68.12.0 was released. Then, the first change after this would have been Firefox ESR 78.3 on Sep 18, 2020.
According to the CentOS 8 Stream model, Firefox ESR 68.x would have been maintained until Jul 24, 2020 when the 68.11.0 was released. Then, the first change after this would have been Firefox ESR 78.2 on Aug 18, 2020.
I picked a relatively tame example just to demonstrate how the approach changes what happens, and could change the outcome and experience for the user. But, in this case, of some important note, is that Firefox ESR has its own "test" / "stabilization" process. When it was first released:
https://mail.mozilla.org/pipermail/enterprise/2020-July/002326.html
"Firefox ESR 78.0 is the first release from the ESR78 branch, which is slated to replace ESR68 in September. We recommend that organizations use this opportunity to test this new version in their environment, ahead of the ESR68 End Of Life. Firefox ESR 78.0.1 fixed a last-minute issue found prior to the 78.0 release."
Firefox ESR 78.1 and Firefox ESR 78.2 had a similar caveat, and also specified that wide deployment would begin with "78.3.0esr in September":
https://mail.mozilla.org/pipermail/enterprise/2020-August/002462.html
"This is the final scheduled release of the ESR 68.x branch. With the release of 78.3.0esr in September, we will begin offering automatic updates from earlier ESR versions to the 78.x branch. We recommend that organizations use this opportunity to test this new version in their environment ahead of the ESR68 End Of Life."
As you can see from the above timeline - if CentOS 7 was run according to the CentOS 8 Stream model, CentOS would have received the *beta* version of Firefox ESR 78.2, while if it had followed the CentOS 7 and CentOS 8 model, CentOS would have received the *production* version of Firefox ESR 78.3.
This is just one of hundreds of packages with problems like this. CentOS 8 Stream is not "Enterprise Linux". CentOS 8 Stream is "the first stage of stabilization for Enterprise Linux". It does not seem appropriate to use for environments that are risk-averse. This had been clearly stated by many people in Red Hat and outside Red Hat who do know what they are talking about. It is just that some people are of the belief that this makes it ok for "95% of users". But, who are these 95%? Who are these users who are not risk-averse who choose CentOS as their distribution? I can't stop you from using CentOS 8 Stream in a production environment without building your own stabilization process to substitute for what you just lost - but I can warn you, that it sounds like a terrible idea.
On 12/19/20 9:05 PM, Mark Mielke wrote:
As you can see from the above timeline - if CentOS 7 was run according to the CentOS 8 Stream model, CentOS would have received the*beta* version of Firefox ESR 78.2, while if it had followed the CentOS 7 and CentOS 8 model, CentOS would have received the*production* version of Firefox ESR 78.3.
Why do you think Firefox 78.2 was a beta release?
On Sun, Dec 20, 2020 at 6:59 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 9:05 PM, Mark Mielke wrote:
As you can see from the above timeline - if CentOS 7 was run according to the CentOS 8 Stream model, CentOS would have received the*beta* version of Firefox ESR 78.2, while if it had followed the CentOS 7 and CentOS 8 model, CentOS would have received the*production* version of Firefox ESR 78.3.
Why do you think Firefox 78.2 was a beta release?
"Beta" from https://en.wikipedia.org/wiki/Software_release_life_cycle#Beta :
"Beta, named after the second letter of the Greek alphabet, is the software development phase following alpha. Software in the beta stage is also known as betaware.[6] A Beta phase generally begins when the software is feature complete but likely to contain a number of known or unknown bugs.[7] Software in the beta phase will generally have many more bugs in it than completed software, speed or performance issues, and may still cause crashes or data loss. The focus of beta testing is reducing impacts to users, often incorporating usability testing. The process of delivering a beta version to the users is called beta release and this is typically the first time that the software is available outside of the organization that developed it. Software beta releases can either be public or private, depending on whether they are openly available or only available to a limited audience. Beta version software is often useful for demonstrations and previews within an organization and to prospective customers. Some developers refer to this stage as a preview, preview release, prototype, technical preview / technology preview (TP),[8] or early access. Since the introduction of Windows 8, Microsoft has called pre-release software a preview rather than beta. All pre-release builds released through the Windows Insider Program launched in 2014 are termed "Insider Preview builds". "beta" may also indicate something more like a release candidate, or as a form of time-limited demo, or marketing technique.[9]
Beta testers are people who actively report issues of beta software. They are usually customers or representatives of prospective customers of the organization that develops the software. Beta testers tend to volunteer their services free of charge but often receive versions of the product they test, discounts on the release version, or other incentives."
Firefox ESR 78.2 release notes, as I already quoted from https://mail.mozilla.org/pipermail/enterprise/2020-August/002462.html :
"This is the final scheduled release of the ESR 68.x branch. With the release of 78.3.0esr in September, we will begin offering automatic updates from earlier ESR versions to the 78.x branch. We recommend that organizations use this opportunity to test this new version in their environment ahead of the ESR68 End Of Life."
Firefox ESR has had this process for several ESR releases at this point. Everybody who is part of the process knows what to expect. Internally to our company, we also began testing with the release of Firefox ESR 78.0, 78.1, and 78.2, so we could provide feedback into 78.3, which was the version that was pushed out broadly.
I don't know if we are playing word games here - or if you truly believe it is a responsible choice to broadly deploy an early access version to a set of "Enterprise" customers. I'll use whatever word you want, as long as we agree that CentOS 8 Stream is for people who are *developing* CentOS. It is not for "Enterprise production deployments". I will comment further on another one of your posts.
-- Mark Mielke mark.mielke@gmail.com
On 12/20/20 7:09 PM, Mark Mielke wrote:
On Sun, Dec 20, 2020 at 6:59 PM Gordon Messmer gordon.messmer@gmail.com wrote:
Why do you think Firefox 78.2 was a beta release?
"This is the final scheduled release of the ESR 68.x branch. With the release of 78.3.0esr in September, we will begin offering automatic updates from earlier ESR versions to the 78.x branch. We recommend that organizations use this opportunity to test this new version in their environment ahead of the ESR68 End Of Life."
I don't think that Mozilla would agree with your characterization. They do not describe ESR releases as betas anywhere I've ever seen. Having two mature versions in production and in support does not make the newest one a beta. I mean... you don't think RHEL 8 is a beta because RHEL 7 is still supported, right?
Is it the word "test" that you associate with beta? Because *of course* you should test a product before you deploy it. You should also test your software on RHEL before you deploy it. That doesn't make RHEL a beta, it's just a part of you doing due diligence with your own configuration.
I don't know if we are playing word games here - or if you truly believe it is a responsible choice to broadly deploy an early access version to a set of "Enterprise" customers. I'll use whatever word you want, as long as we agree that CentOS 8 Stream is for people who are *developing* CentOS. It is not for "Enterprise production deployments". I will comment further on another one of your posts.
Sure. I'm not saying that people who want an enterprise production environment should use CentOS Stream for that purpose. You should probably look at RHEL for that.
I *would* say that I expect CentOS Stream to be better than CentOS is today, for most purposes.
On Sun, Dec 20, 2020 at 11:40 PM Gordon Messmer gordon.messmer@gmail.com wrote:
Is it the word "test" that you associate with beta? Because *of course* you should test a product before you deploy it. You should also test your software on RHEL before you deploy it. That doesn't make RHEL a beta, it's just a part of you doing due diligence with your own configuration.
Firefox ESR team: We are making 78.0, 78.1, and 78.2 available for organizations to test with. We will make 78.3 widely available, and an automatic update.
RHEL 7: We will start from 78.3, because we are a responsible Enterprise Linux company, and we work with the Firefox ESR team, and we understand that 78.3 is the version that should be used to replace 68.
CentOS 7: We will start from 78.3, because RHEL started with 78.3.
CentOS 7 Stream: We will start from 78.2, to test 78 before dropping it on our RHEL customers. (CentOS 7 Stream does not exist - but if it had...)
Gordon: I see no problem with CentOS 7 Stream strategy. Seems fine to me. What's wrong?
I don't know if we are playing word games here - or if you truly believe it is a responsible choice to broadly deploy an early access version to a set of "Enterprise" customers. I'll use whatever word you want, as long as we agree that CentOS 8 Stream is for people who are *developing* CentOS. It is not for "Enterprise production deployments". I will comment further on another one of your posts.
Sure. I'm not saying that people who want an enterprise production environment should use CentOS Stream for that purpose. You should probably look at RHEL for that.
Maybe. Maybe not. Going back to my original problem - the Red Hat subscription model is broken and does not scale. Are you using RHEL at scale? Why or why not?
I *would* say that I expect CentOS Stream to be better than CentOS is today, for most purposes.
How are you defining "most purposes"? Do most purposes include "Enterprise production deployments" or not? Because, CentOS 8 exists for "Enterprise production deployments", which I think is "most purposes". You are just ignoring this use case, and *then* claiming that CentOS 8 Stream is acceptable for "most purposes".
On 12/20/20 8:55 PM, Mark Mielke wrote:
On Sun, Dec 20, 2020 at 11:40 PM Gordon Messmer gordon.messmer@gmail.com wrote:
Is it the word "test" that you associate with beta? Because *of course* you should test a product before you deploy it. You should also test your software on RHEL before you deploy it. That doesn't make RHEL a beta, it's just a part of you doing due diligence with your own configuration.
RHEL 7: We will start from 78.3, because we are a responsible Enterprise Linux company, and we work with the Firefox ESR team, and we understand that 78.3 is the version that should be used to replace 68.
As far as I can tell, when Red Hat updated Firefox 45 to 52, they started with 52.0. When they updated from 52 to 60, they started with 60.1 (ESR channel users weren't updated until 60.2). When they updated from 60 to 68, they started with 68.1 (ESR channel users weren't updated until 68.2). When they updated from 68 to 78, they started with 78.4:
https://vault.centos.org/7.3.1611/updates/Source/SPackages/ https://vault.centos.org/7.5.1804/updates/Source/SPackages/ https://vault.centos.org/7.7.1908/updates/Source/SPackages/ https://vault.centos.org/7.9.2009/updates/Source/SPackages/
https://mail.mozilla.org/pipermail/enterprise/2019-October/001869.html https://mail.mozilla.org/pipermail/enterprise/2018-September/000246.html
I'm not sure where to find the Firefox enterprise list archives from March 2017, but I don't see anything that supports the idea that Mozilla has ever described the initial ESR releases as betas, and I don't see anything to support the idea that Red Hat keeps the same release schedule as Mozilla. I think you're cherry picking data to build your argument and attributing *your* motivations to other people without cause.
Gordon: I see no problem with CentOS 7 Stream strategy. Seems fine to me. What's wrong?
You're right, here. I don't see a problem with it.
On Mon, Dec 21, 2020 at 12:48 AM Gordon Messmer gordon.messmer@gmail.com wrote:
I'm not sure where to find the Firefox enterprise list archives from March 2017, but I don't see anything that supports the idea that Mozilla has ever described the initial ESR releases as betas, and I don't see anything to support the idea that Red Hat keeps the same release schedule as Mozilla. I think you're cherry picking data to build your argument and attributing *your* motivations to other people without cause.
https://support.mozilla.org/en-US/kb/firefox-esr-release-cycle
"The ESR version will also have a three cycle (at least 12 weeks) overlap between the time of a new release and the end-of-life of the previous release to permit testing and certification before you deploy the new version to your organization."
I can't argue with you any more, Gordon. It's really draining, and I don't think we are ever going to agree. I'm sharing information, and you can choose what you want to do with it.
All I can say is that I've been on the Firefox ESR mailing lists from the start, and been part of the planning for distribution, and when I've noticed Red Hat doing smart things - I've nodded in my head and said "good job, you did the right thing."
So, if you are trying to convince me that sometimes Red Hat and CentOS didn't do the right thing ... fine, I'm sure that happens. But, comparing my experiences to yours with Firefox ESR, I have known about the three cycle overlap for a long time now, and this is not new information to me. Is it new information to you?
On 12/20/20 10:06 PM, Mark Mielke wrote:
So, if you are trying to convince me that sometimes Red Hat and CentOS didn't do the right thing ... fine, I'm sure that happens.
I'm not saying that, at all. I just don't think that using the current ESR before the last one is EOL is the wrong thing to do. To the best of my knowledge, Mozilla supports users making that switch whenever they have completed testing. The automatic upgrade doesn't happen when the last release is EOL because that's the "right thing", it's just because that's the deadline. That's when users no longer have a choice (other than to continue running unsupported software).
But, comparing my experiences to yours with Firefox ESR, I have known about the three cycle overlap for a long time now, and this is not new information to me. Is it new information to you?
No.
On Mon, Dec 21, 2020 at 1:19 AM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/20/20 10:06 PM, Mark Mielke wrote:
So, if you are trying to convince me that sometimes Red Hat and CentOS didn't do the right thing ... fine, I'm sure that happens.
I'm not saying that, at all. I just don't think that using the current ESR before the last one is EOL is the wrong thing to do. To the best of my knowledge, Mozilla supports users making that switch whenever they have completed testing. The automatic upgrade doesn't happen when the last release is EOL because that's the "right thing", it's just because that's the deadline. That's when users no longer have a choice (other than to continue running unsupported software).
We are not on the same wavelength here. I don't know who your users are, but they are not the same as mine.
I think any "Enterprise Linux" distribution should not automatically make a Firefox ESR version the default version, before Mozilla has made it the default version. If Mozilla has not made it an automatic update, then RHEL should not, and CentOS should not.
But, do whatever you want. As long as your employer is happy with your assessment of risk, it's not my problem.
On 12/21/20 5:40 AM, Gordon Messmer wrote:
I *would* say that I expect CentOS Stream to be better than CentOS is today, for most purposes.
AHAHAHAHAHAHAHA, You crack me up :-D
On Sat, Dec 19, 2020 at 4:15 PM Gordon Messmer gordon.messmer@gmail.com wrote:
I promise: You do not understand RHEL better than Matthew Miller does. Just stop for a moment and listen to him. You might learn something new.
I might indeed learn something from him. Let him say it himself, eh? I might be a bit crotchety about this, but I'm pointing out details and facts. Take it as the MIT vs. Harvard approach: at presentations, the Harvard people identify themselves and their department before they ask a question. The MIT people just say "you violated the second law of thermodynamics on page two....." If you're curious, look me up: I've deployed many thousands of Red Hat based systems for more than 20 years.
You just ridiculed Matthew, and then repeated the point that he was making, using your own words.
Mark Mielke covered most of this, no need to repeat his analysis or mine
Matthew, did I insult you or offend you in some way? Feel free to let me know privately if I did, I don't want to be a *jerk* to people trying to explain what they see.
On Sat, Dec 19, 2020 at 10:54:31PM -0500, Nico Kadel-Garcia wrote:
Matthew, did I insult you or offend you in some way? Feel free to let me know privately if I did, I don't want to be a *jerk* to people trying to explain what they see.
I am not insulted or offended. Thanks for checking, though (seriously).
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not
Am I missing something? The last time I stared at RHEL release media the packages and their versions were a one-for-one match to the CentOs point release, except for the CentOS-only packages such as "epel-release" and packages with distinct trademarks or licenses. It's been a couple of years since I grabbed RHEL release media, I usually use a licensed RHEL OS image as needed these days. Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
lsoOn Sat, Dec 19, 2020 at 11:28 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
On Sat, Dec 19, 2020 at 04:34:53AM -0500, Mark Mielke wrote:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not
Am I missing something? The last time I stared at RHEL release media the packages and their versions were a one-for-one match to the CentOs point release, except for the CentOS-only packages such as "epel-release" and packages with distinct trademarks or licenses. It's been a couple of years since I grabbed RHEL release media, I usually use a licensed RHEL OS image as needed these days. Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
I'm pretty sure Matthew Miller is talking about EUS releases, and he is missing that all RHEL minor releases are actually branches whether EUS or not. Red Hat is developing N+1, and N+2, while publishing N, and mirroring N to CentOS 7 and 8. I covered this in more detail in my other posts.
There does seem to be a pattern that the people who think CentOS 8 Stream is not a big change, seem to be the same people who are not really aware of these additional branches. I also see how Red Hat presenting a "c8" branch might re-enforce the belief that CentOS 8 is already basically the same as CentOS 8 Stream. Unfortunately, this means that there will be a surprise and shock factor when the light bulb goes on and they realize what just happened.
-- Mark Mielke mark.mielke@gmail.com
On Sun, Dec 20, 2020 at 12:21:48AM -0500, Mark Mielke wrote:
I'm pretty sure Matthew Miller is talking about EUS releases, and he is missing that all RHEL minor releases are actually branches whether EUS or not. Red Hat is developing N+1, and N+2, while publishing N, and mirroring N to CentOS 7 and 8. I covered this in more detail in my other posts.
I am a little confused by your messages on this topic. You are vehemently telling me that I'm missing something and don't understand, while you explain in greater detail the same thing I said: RHEL minor releases are actually branches, while CentOS Linux rebuild minor releases never were.
On Sun, Dec 20, 2020 at 2:21 PM Matthew Miller mattdm@mattdm.org wrote:
On Sun, Dec 20, 2020 at 12:21:48AM -0500, Mark Mielke wrote:
I'm pretty sure Matthew Miller is talking about EUS releases, and he is missing that all RHEL minor releases are actually branches whether EUS or not. Red Hat is developing N+1, and N+2, while publishing N, and mirroring N to CentOS 7 and 8. I covered this in more detail in my other posts.
I am a little confused by your messages on this topic. You are vehemently telling me that I'm missing something and don't understand, while you explain in greater detail the same thing I said: RHEL minor releases are actually branches, while CentOS Linux rebuild minor releases never were.
The obvious answer is that I don't think we are saying the same thing.
Let's go back to your original response to me:
- Minor release milestones to stabilize branches. We have breakage
with most minor release upgrades, and the stabilization process is an important method of isolating users from being affected by this. This is why CentOS 8 Stream is being said "for developers", while RHEL 8 would be "for production". It is being said, because it is a real thing. If you truly believed minor release milestones were unnecessary for CentOS 8 Stream, then you would also believe that minor release milestones were unnecessary for RHEL 8.
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_. So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not mimic.
I claimed CentOS Linux was superior because it provided "Minor release milestones to stabilize branches". You claimed "It's important to note that the CentOS Linux rebuild never actually had this."
Your conclusion is not correct, for the reasons that I outlined in other responses. As I respect you too much to believe that you are playing word games, I am presuming this is a genuine misunderstanding, which is why I spent considerable effort detailing examples, including Firefox ESR breakdown of impact, to explain why what you are saying does not match what I am saying, and why what I am saying matters.
Let's go into closer detail on your claim, snipping specific sections to help dissect:
RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates. For CentOS Linux, a minor release is a point where updates pause for a while while the team scrambles to rebuild a large update of many packages and then those packages all updated in a big chunk _on the single CentOS branch_.
The CentOS Linux branches are imports from the RHEL minor release branches. You can see this by looking at the history of them, and comparing the versions that are published to CentOS Linux, and the versions that are published to RHEL for any given minor release. The "c7" branch looks more like this:
|-- ... --|-- RHEL 7.0 branch imports + de-branding --|-- RHEL 7.1 branch imports + de-branding --|-- ... --|
Technically this is exposed as one branch called "c7", but this is a flattened set of outputs. Essentially, "c7" is *not* a source branch. It is the serialized result of performing a process against a set of more complicated branches. This means that to say "RHEL minor releases are branched, CentOS minor releases are not" is to ignore that CentOS = RHEL + De-Branding, and that the CentOS branch is an export for the RHEL branches.
Therefore, for an *active* release of RHEL/CentOS, this statement you made is false:
So this is always been extra value that Red Hat Enterprise Linux provides that CentOS Linux did not mimic.
This is *not* an extra value of RHEL over CentOS. The extra value of RHEL over CentOS, is *support*.
What you are, however, referring to, perhaps by accident, or perhaps by misunderstanding, is RHEL EUS. For RHEL EUS, your statement becomes correct:
So this is always been extra value that Red Hat Enterprise Linux *Extended Update Support* provides that CentOS Linux did not mimic.
I think there is a large section of people that do not understand the difference, and therefore they do not understand exactly what the different between CentOS Linux and CentOS Stream represents. These people don't understand what is being lost with CentOS Stream, and they will be a little shocked when they find out.
Fedora doesn't do this. Fedora breaks this with smaller incremental major versions. Fedora is very similar to CentOS Stream, and as such, I can see why you would be comfortable with it.
On Sun, Dec 20, 2020 at 2:21 PM Matthew Miller mattdm@mattdm.org wrote:
On Sun, Dec 20, 2020 at 12:21:48AM -0500, Mark Mielke wrote:
I'm pretty sure Matthew Miller is talking about EUS releases, and he is missing that all RHEL minor releases are actually branches whether EUS or not. Red Hat is developing N+1, and N+2, while publishing N, and mirroring N to CentOS 7 and 8. I covered this in more detail in my other posts.
I am a little confused by your messages on this topic. You are vehemently telling me that I'm missing something and don't understand, while you explain in greater detail the same thing I said: RHEL minor releases are actually branches, while CentOS Linux rebuild minor releases never were.
Ahh. "No one cares". It's not that we don't appreciate someone doing the back end work safely and consistently. The back end git branching and tag versus release used for tracking development. We see the RPMs and SRPMs, not the git history. And what we see, or I see, is that the point releases are used just like git tags. They're a reference point for the software building system. When I use "mock", I'm pointng at the "CentOS 7" or "CentOS 8" mirrors, and those are pointed to the point release and expected to have consistent, reference images as a mass. We have a similar same thing with docker images and any internal corporate "golden images". The stable reference content is very like that of a git or other source control tag, you expect the same dang thing if you run the same building tools again. And as I understand the CentOS releases, it is *designed* to be as close as possible to the RHEL labeled point release.
Discarding the point releases discards that reference, OK, I don't know what else to call it in this context but a tag. If I have to do this from scratch.... well, I'm going to have to do something like get a complete set of installation media and take my reference repositories from that, or otherwise burn time and disk space I'm not happy to spend.
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
In RHEL, a minor release is a branch. You can install RHEL 7.8, and keep a host on RHEL 7.8 until the end of its life cycle. If you want long term support for an OS with minimal changes, but continued support, that's a thing that RHEL provides.
CentOS doesn't offer that. If you're using CentOS, you're always on the "latest" branch. You either apply feature updates that come with each point release to get security updates, or you stop applying updates entirely and run the risks that come with not applying security or bugfix updates.
Mark is confusing the issue somewhat. I *think* he is trying to say that when we say that CentOS point releases have no branches, we're saying that there's no QA, which is absolutely not what we're saying. We're not talking about the back end development process, we're talking about the products that are delivered to customers. Customers can choose what branch of the RHEL product to deploy on their systems, and how long to use a given point release. CentOS users don't get that level of support.
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
If I may say, I didn't see him say that. If you call Red Hat about current CVE's, the updates are in the main update channels.
In RHEL, a minor release is a branch. You can install RHEL 7.8, and keep a host on RHEL 7.8 until the end of its life cycle. If you want long term support for an OS with minimal changes, but continued support, that's a thing that RHEL provides.
And for CentOS, you point them to the vault archives of the old OS for installation media, and apply the updates as needed from the main channel. Does Red Hat publish new sub distributions and installation media for RHEL 7.0, 7.1, etc? The last time I needed that, they walked me though building my own PXE image as needed.
CentOS doesn't offer that. If you're using CentOS, you're always on the "latest" branch. You either apply feature updates that come with each point release to get security updates, or you stop applying updates entirely and run the risks that come with not applying security or bugfix updates.
Unless you point to vault.centos.org or, if you're thoughtful, set up an internal mirror of vault. It's just like what I did with RHEL installation media to have internal point releases accessible in yum repos without paying for RHN and doing host-by-host individual tuning.
Getting the point releases published as a testable, designated set of packages is critical for stability in large environments. In that way, they very much resemble a "git tag".
Mark is confusing the issue somewhat. I *think* he is trying to say that when we say that CentOS point releases have no branches, we're saying that there's no QA, which is absolutely not what we're saying. We're not talking about the back end development process, we're talking about the products that are delivered to customers. Customers can choose what branch of the RHEL product to deploy on their systems, and how long to use a given point release. CentOS users don't get that level of support.
Can you think of or name *any* who get updates published only for their older version of RHEL, rather than published as part of the main channels? Back in a previous role, I sometimes got beta kernel updates via a back channel, but the updates showed up quite soon in the main codeline's kernel. Way back when, sometimes my employers *sent* kernel tuning patches.
On 12/20/20 4:54 PM, Nico Kadel-Garcia wrote:
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
If I may say, I didn't see him say that.
I had quoted it above.
If you call Red Hat about current CVE's, the updates are in the main update channels.
Yes, they're in the main update channels, but they'll *also* be in update channels for RHEL minor releases that are still supported. There are no such channels for CentOS minor releases that aren't the most recent release.
In RHEL, a minor release is a branch. You can install RHEL 7.8, and keep a host on RHEL 7.8 until the end of its life cycle. If you want long term support for an OS with minimal changes, but continued support, that's a thing that RHEL provides.
And for CentOS, you point them to the vault archives of the old OS for installation media, and apply the updates as needed from the main channel.
That's often true, but not necessarily so, because of the problem that Mark described in his email yesterday: "If I build something for EL 8.2, it will *probably* work in EL 8.3. However, if I build something in EL 8.3,there is no guarantee it will work in EL 8.2."
Linux ABIs aren't forward compatible. Updates prepared for the current release may or may not actually work when applied to an older release.
You cannot, therefore, reliably keep a CentOS system on a non-current minor release and still get security or bug fixes.
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
If you disagree with me that CentOS is bug-for-bug compatible with RHEL for *active* releases, then please provide a real life example. Otherwise, stop saying this rubbish.
In RHEL, a minor release is a branch. You can install RHEL 7.8, and keep a host on RHEL 7.8 until the end of its life cycle. If you want long term support for an OS with minimal changes, but continued support, that's a thing that RHEL provides.
You can also install CentOS 7.9, and keep a host on CentOS 7.8 until the end of its life cycle. That is what RHEL is. If you can prove otherwise, then please prove it. Show us an example of where this is not true.
You and Matthew are confusing RHEL with RHEL EUS. They are not even the same subscription type! Somebody who buys RHEL, is not permitted to access EUS, without either having a Premium subscription, or a separate EUS subscription.
As I said - provide an example. Otherwise, please stop misinforming people.
CentOS doesn't offer that. If you're using CentOS, you're always on the "latest" branch. You either apply feature updates that come with each point release to get security updates, or you stop applying updates entirely and run the risks that come with not applying security or bugfix updates.
This is false. As I showed real-life examples for. Please provide your own counter-examples for an actual discussion, rather than your own uninformed opinion.
Mark is confusing the issue somewhat. I *think* he is trying to say that when we say that CentOS point releases have no branches, we're saying that there's no QA, which is absolutely not what we're saying. We're not talking about the back end development process, we're talking about the products that are delivered to customers. Customers can choose what branch of the RHEL product to deploy on their systems, and how long to use a given point release. CentOS users don't get that level of support.
This is also false. Moving CentOS from "downstream" to "upstream" absolutely affects where QA fits into the process. This is a fundamental thing that is being ripped out from under you - and you don't even realize.
Let's say there is a process to properly "bake" an Enterprise Linux release. It includes a number of steps, which I have outlined in prior posts. CentOS as it was originally created, and as existed until CentOS 8, was to take the output of this finished process and reproduce it bug-for-bug. Essentially RHEL baked a cookie, and CentOS baked a cookie, and the outcome is the exact same except one says "baked RHEL" and one says "baked by CentOS".
CentOS 8 Stream, is the output at an earlier stage in this baking process. In this analogy, I would equate CentOS 8 Stream to something like the batter. It's a necessary component to baking the cookie, but it is an unfinished product.
In this analogy, you - and a few others like you - are eating the batter and saying "tastes like cookie to me! good job, Red Hat!" But, you don't even realize the value of RHEL. You don't realize that you previously had a finished cookie, but now you just have the cookie batter because Red Hat is not baking it for you. They are just giving you the batter.
One day - the light bulb will go on, and you will be "omg... I just realized what Mark was trying to tell me."
Or, maybe not.
On Sun, Dec 20, 2020 at 10:19 PM Mark Mielke mark.mielke@gmail.com wrote:
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
If you disagree with me that CentOS is bug-for-bug compatible with RHEL for *active* releases, then please provide a real life example. Otherwise, stop saying this rubbish.
In RHEL, a minor release is a branch. You can install RHEL 7.8, and keep a host on RHEL 7.8 until the end of its life cycle. If you want long term support for an OS with minimal changes, but continued support, that's a thing that RHEL provides.
You can also install CentOS 7.9, and keep a host on CentOS 7.8 until the end of its life cycle. That is what RHEL is. If you can prove otherwise, then please prove it. Show us an example of where this is not true.
You and Matthew are confusing RHEL with RHEL EUS. They are not even the same subscription type! Somebody who buys RHEL, is not permitted to access EUS, without either having a Premium subscription, or a separate EUS subscription.
As I said - provide an example. Otherwise, please stop misinforming people.
CentOS doesn't offer that. If you're using CentOS, you're always on the "latest" branch. You either apply feature updates that come with each point release to get security updates, or you stop applying updates entirely and run the risks that come with not applying security or bugfix updates.
This is false. As I showed real-life examples for. Please provide your own counter-examples for an actual discussion, rather than your own uninformed opinion.
Mark is confusing the issue somewhat. I *think* he is trying to say that when we say that CentOS point releases have no branches, we're saying that there's no QA, which is absolutely not what we're saying. We're not talking about the back end development process, we're talking about the products that are delivered to customers. Customers can choose what branch of the RHEL product to deploy on their systems, and how long to use a given point release. CentOS users don't get that level of support.
This is also false. Moving CentOS from "downstream" to "upstream" absolutely affects where QA fits into the process. This is a fundamental thing that is being ripped out from under you - and you don't even realize.
Let's say there is a process to properly "bake" an Enterprise Linux release. It includes a number of steps, which I have outlined in prior posts. CentOS as it was originally created, and as existed until CentOS 8, was to take the output of this finished process and reproduce it bug-for-bug. Essentially RHEL baked a cookie, and CentOS baked a cookie, and the outcome is the exact same except one says "baked RHEL" and one says "baked by CentOS".
CentOS 8 Stream, is the output at an earlier stage in this baking process. In this analogy, I would equate CentOS 8 Stream to something like the batter. It's a necessary component to baking the cookie, but it is an unfinished product.
In this analogy, you - and a few others like you - are eating the batter and saying "tastes like cookie to me! good job, Red Hat!" But, you don't even realize the value of RHEL. You don't realize that you previously had a finished cookie, but now you just have the cookie batter because Red Hat is not baking it for you. They are just giving you the batter.
One day - the light bulb will go on, and you will be "omg... I just realized what Mark was trying to tell me."
Or, maybe not.
-- Mark Mielke mark.mielke@gmail.com
On Sun, Dec 20, 2020 at 10:19 PM Mark Mielke mark.mielke@gmail.com wrote:
In RHEL, a minor release is a branch. You can install RHEL 7.8, and keep a host on RHEL 7.8 until the end of its life cycle. If you want long term support for an OS with minimal changes, but continued support, that's a thing that RHEL provides.
You can also install CentOS 7.9, and keep a host on CentOS 7.8 until the end of its life cycle. That is what RHEL is. If you can prove otherwise, then please prove it. Show us an example of where this is not true.
Grrr... an unfortunate typo for me. But, I'll fix it and expand:
You can install CentOS 7.8, and keep a host on CentOS 7.8, receiving security patches that are on both RHEL 7.8 and CentOS 7.8, until the end of the RHEL 7.8 and CentOS 7.8 life cycle. This is because RHEL 7.8 is a branch, and CentOS is regularly importing changes from this branch, which means that CentOS 7.8 is also a branch.
Once RHEL 7.9 is out, RHEL 7.8 is "end of life", and similarly, CentOS 7.8 is also "end of life". Once RHEL 7.9 is out, the "c7" branches are updated to reflect RHEL 7.9, and follow the RHEL 7.9 branch, as CentOS 7.9. If there was a RHEL 7.10, then it would be on a private 7.10 branch in RHEL, and run in parallel to the 7.9 that is being imported into CentOS 7.9.
Everything you are saying about RHEL being some branch that lives longer than CentOS applies to RHEL EUS, not RHEL:
You and Matthew are confusing RHEL with RHEL EUS. They are not even the same subscription type! Somebody who buys RHEL, is not permitted to access EUS, without either having a Premium subscription, or a separate EUS subscription.
On Sun, Dec 20, 2020 at 10:23 PM Mark Mielke mark.mielke@gmail.com wrote:
Once RHEL 7.9 is out, RHEL 7.8 is "end of life", and similarly, CentOS 7.8 is also "end of life". Once RHEL 7.9 is out, the "c7" branches are updated to reflect RHEL 7.9, and follow the RHEL 7.9 branch, as CentOS 7.9. If there was a RHEL 7.10, then it would be on a private 7.10 branch in RHEL, and run in parallel to the 7.9 that is being imported into CentOS 7.9.
Here is a picture for you:
https://access.redhat.com/support/policy/updates/errata#RHEL8_Life_Cycle
See how the "RHEL" in dark blue "Minor Release" are not overlapping. This RHEL is what is used to produce CentOS. All RHEL changes are published to the CentOS branches, and built into CentOS a short time later. This includes backported security fixes and other critical fixes that are received throughout the active life of every minor release. The RHEL/CentOS branch is a stabilized branch that has already gone through all design processes including a public beta and a milestone release with release notes documenting the changes that may cause breakage for users.
Gordon and Matthew are referring to the "RHEL EUS", which is the lighter blue. This is a separate subscription type from RHEL, and it is NOT published to the CentOS branches. You must pay extra to gain access to RHEL EUS. It requires either a Premium subscription, or an add-on "Extended Update Support" subscription. This separate subscription type does allow you to stay specific previously active minor releases and continue to receive critical updates without having to upgrade to the next RHEL minor release.
I think if we were in a room with a whiteboard - many of these confusions could be much more easily dispelled. A lot of what is going on is simply misinformation and wishful thinking. It's not intentional - it's just that you never knew the things I'm trying to share with you.
On 12/20/20 7:19 PM, Mark Mielke wrote:
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
CentOS is compatible with the current release. I don't think anyone is saying that it isn't. I honestly can't figure out why you're telling me that I'm wrong and then arguing what seems to be a completely different point. Unless...
You and Matthew are confusing RHEL with RHEL EUS.
Are you objecting because you think that RHEL and RHEL EUS are different products? I would have described EUS as a different support contract for the same product, but at that point you're *really* parsing words carefully.
Mark is confusing the issue somewhat. I *think* he is trying to say that when we say that CentOS point releases have no branches, we're saying that there's no QA, which is absolutely not what we're saying. We're not talking about the back end development process, we're talking about the products that are delivered to customers. Customers can choose what branch of the RHEL product to deploy on their systems, and how long to use a given point release. CentOS users don't get that level of support.
This is also false. Moving CentOS from "downstream" to "upstream" absolutely affects where QA fits into the process. This is a fundamental thing that is being ripped out from under you - and you don't even realize.
I think that your position here implies that end-users who run beta releases are responsible for the quality of Red Hat's releases, and I don't know any reason to believe that. In particular, Red Hat employees tell us that "literally almost nobody uses them":
http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-thi...
I don't have any reason to disbelieve that, so when I think about how Red Hat is able to manage a high-quality release, I tend to think that it's because they have a rigorous testing process for the software they distribute, and they have repeatedly told us in this list that CentOS Stream updates will have gone through that process.
On Mon, Dec 21, 2020 at 12:22 AM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/20/20 7:19 PM, Mark Mielke wrote:
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org wrote:
It's important to note that the CentOS Linux rebuild never actually had this. RHEL minor releases are actually branches, and you can stay at a minor release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
CentOS is compatible with the current release. I don't think anyone is saying that it isn't. I honestly can't figure out why you're telling me that I'm wrong and then arguing what seems to be a completely different point. Unless...
The point that is getting lost - is that CentOS 8 Stream is not the hardened release you think it is. CentOS 8 and RHEL 8 is. CentOS 8 is not.
I've tried to explain the branching strategy to you, that should have made it clear to you that the "current release" is a branch, and it is not latest. But, we're not stuck in word games about what a branch is, or what a tag is, and you are totally missing the point.
I think others do get the point, so we can just leave it at that.
You and Matthew are confusing RHEL with RHEL EUS.
Are you objecting because you think that RHEL and RHEL EUS are different products? I would have described EUS as a different support contract for the same product, but at that point you're *really* parsing words carefully.
I'm objecting to you saying things that are not correct. "RHEL" and "CentOS" are equivalent today. "CentOS Stream" is not equivalent. "RHEL EUS" is what you are describing.
If you want to claim that I am playing a word game about this, then this is another place where I think others will get the point, so we can just leave it at that.
Mark is confusing the issue somewhat. I *think* he is trying to say that when we say that CentOS point releases have no branches, we're saying that there's no QA, which is absolutely not what we're saying. We're not talking about the back end development process, we're talking about the products that are delivered to customers. Customers can choose what branch of the RHEL product to deploy on their systems, and how long to use a given point release. CentOS users don't get that level of support.
This is also false. Moving CentOS from "downstream" to "upstream" absolutely affects where QA fits into the process. This is a fundamental thing that is being ripped out from under you - and you don't even realize.
I think that your position here implies that end-users who run beta releases are responsible for the quality of Red Hat's releases, and I don't know any reason to believe that. In particular, Red Hat employees tell us that "literally almost nobody uses them": http://crunchtools.com/before-you-get-mad-about-the-centos-stream-change-thi... I don't have any reason to disbelieve that, so when I think about how Red Hat is able to manage a high-quality release, I tend to think that it's because they have a rigorous testing process for the software they distribute, and they have repeatedly told us in this list that CentOS Stream updates will have gone through that process.
Red Hat never claimed that CentOS Stream will be just as hardened as CentOS. They never claimed it will get the same process. Doing so would make RHEL obsolete, so it's not even an intelligent thing for them to claim. This is more like wishful thinking.
Good luck with your CentOS 8 Stream deployments.
On 12/20/20 9:32 PM, Mark Mielke wrote:
Red Hat never claimed that CentOS Stream will be just as hardened as CentOS. They never claimed it will get the same process. Doing so would make RHEL obsolete, so it's not even an intelligent thing for them to claim. This is more like wishful thinking.
I think they have, repeatedly. That's now I interpret their statements:
"no one is rolling in packages here straight from a new Fedora version or from Rawhide. These changes are point release type changes. That is the type of changes you see from 8.2 to 8.3 or 7.8 to 7.9 ... not major changes."
https://lists.centos.org/pipermail/centos/2020-December/352374.html
"no, CentOS Stream isn't beta. Packages landing in Stream have already passed QA and gating."
https://lists.centos.org/pipermail/centos/2020-December/352383.html
Making a reliable distribution doesn't make RHEL obsolete. There's still a lot of value in an OS that is itself semantically versioned, and has paid support contracts.
There's no evidence to support the idea that CentOS Stream will be unreliable because a reliable system would compete with RHEL. And lots of evidence to the contrary, including the extent to which Red Hat publishes their work in a form usable to rebuild the distribution.
On Mon, Dec 21, 2020 at 2:28 AM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/20/20 9:32 PM, Mark Mielke wrote:
Red Hat never claimed that CentOS Stream will be just as hardened as CentOS. They never claimed it will get the same process. Doing so would make RHEL obsolete, so it's not even an intelligent thing for them to claim. This is more like wishful thinking.
I think they have, repeatedly. That's now I interpret their statements: "no one is rolling in packages here straight from a new Fedora version or from Rawhide. These changes are point release type changes. That is the type of changes you see from 8.2 to 8.3 or 7.8 to 7.9 ... not major changes." https://lists.centos.org/pipermail/centos/2020-December/352374.html
Rawhide is a totally different beast. I lived on rawhide for a year just to learn from it back around 2005. I did learn a lot. It did hurt a lot. :-)
"no, CentOS Stream isn't beta. Packages landing in Stream have already passed QA and gating." https://lists.centos.org/pipermail/centos/2020-December/352383.html
If "Red Hat Enterprise Linux" worth hardening had occurred, we wouldn't need CentOS 8 Stream. But, let's just see what it looks like for real... I'm genuinely curious as it may help us decode the secret sauce, and extrapolate the impacts...
Here is a recently updated package from AppStream:
http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/postgr...
A new version of PostgreSQL. Great! I'm a fan of PostgreSQL. :-)
According to the date on the mirror directory, it says: 2020-12-15 17:57
That's pretty recent. But, the date it was published shouldn't matter - surely there will be quite a lot "QA and gating" that occurred, right?
Build Date : Tue 15 Dec 2020 12:01:55 PM
Hmm... Built on Dec 15, 2020 @ 12:01:55 PM, published on Dec 15, 2020 @ 17:57 PM. Well, let's check the change log:
* Wed Nov 18 2020 Patrik Novotný panovotn@redhat.com - 10.15-1 - Rebase to upstream release 10.15 Resolves: rhbz#1898213 Resolves: rhbz#1898341 Resolves: rhbz#1901567
This one is interesting. In particular, it makes me wonder what is happening behind the scenes. I've seen two seemingly conflicting descriptions of what CentOS 8 Stream is:
1. CentOS 8 Stream is where development happens first, allowing the CentOS community to influence what goes into CentOS 8. This would imply "LATEST". This suggests that CentOS community is in control of the direction of RHEL.
2. CentOS 8 Stream is the "next" (N+1?) version of RHEL 8. This might NOT be "LATEST", but only "LATEST" as shared by Red Hat. Perhaps they are still maintaining a private N+2? This says that the CentOS community has only limited influence. They are seeing N+1, but not seeing N+2.
If the QA was being performed *after* it was built, then this means the change was queued from some time after November 18, until December 15, before being published within 6 hours (assumes the same time zones). Less than 6 hours of post-build testing was performed before releasing it to the public.
If the QA was being performed *before* it was built, then this implies that the build being published is not the build that Red Hat did QA on. The build procedure might be the same, but the actual build in CentOS 8 Stream is *not* the original build that was tested. This would suggest that a private Red Hat branch exists, possibly the N+2 branch? And this means that the end-to-end testing occurred some time between Nov 18 and Dec 15, on a private Red Hat branch, hidden from the CentOS community.
If somebody who actually knows how this works for real could explain, it would be enlightening? Which of the above conclusions is the closest to being right?
Now, I picked PostgreSQL above as it is a favourite of mine, and it happened to be recently updated in AppStream. But, more important to everybody else, is the kernel:
http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4....
By the date on the mirror, it seems to say it was published on 2020-12-04 00:39. Let's look inside at build date, and the change log:
Build Date : Thu 03 Dec 2020 05:32:49 PM
This is similar to the PostgreSQL update, as it was built on Dec 3 @ 5:32:49 PM, and published to the mirror by Dec 4 @ 00:39, around 7 hours later?
Inside, however, the changelog
* Wed Dec 02 2020 Jan Stancek jstancek@redhat.com [4.18.0-257.el8]
Hmmm... The changelog is for Dec 2nd. The build date was Dec 3rd. The deploy date was Dec 4th.
Maybe this was an urgent change of some sort, and very tiny - so it was easy to QA?
Oh uh....
* Wed Dec 02 2020 Jan Stancek jstancek@redhat.com [4.18.0-257.el8] - [hv] hv: vmbus: Allow cleanup of VMBUS_CONNECT_CPU if disconnected (Mohammed Gamal) [1886096] - [hv] hv: vmbus: Add parsing of VMbus interrupt in ACPI DSDT (Mohammed Gamal) [1886096] .. 167 more lines ...
Wow. That's a pretty big change to QA in less than a day. And what about the prior change?
* Mon Nov 30 2020 Jan Stancek jstancek@redhat.com [4.18.0-256.el8] - [kernel] PM: hibernate: Batch hibernate and resume IO requests (Lenny Szubowicz) [1868096] - [net] tunnels: Fix off-by-one in lower MTU bounds for ICMP/ICMPv6 replies (Antoine Tenart) [1895765] ... 124 more lines ...
Another big change, and so is the one after that, on Nov 27!
In fact, if you look at the changes since November 2, the changelog from 4.18.0-257 to 4.18.0-240 is 6149 lines long.
You just changed my mind!
*YOU* should definitely use CentOS 8 Stream! All of *YOU*. That way, by the time you test all of these changes, and report all the bugs, it won't affect *ME*. :-) :-) :-)
Making a reliable distribution doesn't make RHEL obsolete. There's still a lot of value in an OS that is itself semantically versioned, and has paid support contracts.
Yes there is. There is value, because it is important. That's what I'm trying to say. :-)
There's no evidence to support the idea that CentOS Stream will be unreliable because a reliable system would compete with RHEL. And lots of evidence to the contrary, including the extent to which Red Hat publishes their work in a form usable to rebuild the distribution.
"Unreliable" and "reliable" are extremes. RHEL 8 isn't 100% reliable, and CentOS 8 Stream isn't 100% unreliable.
However, RHEL 8 will be a lot more reliable if *YOU* test CentOS 8 Stream on *all* of your systems. The more, the better.
Thank you!
On Mon, Dec 21, 2020 at 4:28 AM Mark Mielke mark.mielke@gmail.com wrote:
On Mon, Dec 21, 2020 at 2:28 AM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/20/20 9:32 PM, Mark Mielke wrote:
Red Hat never claimed that CentOS Stream will be just as hardened as CentOS. They never claimed it will get the same process. Doing so would make RHEL obsolete, so it's not even an intelligent thing for them to claim. This is more like wishful thinking.
I think they have, repeatedly. That's now I interpret their statements: "no one is rolling in packages here straight from a new Fedora version or from Rawhide. These changes are point release type changes. That is the type of changes you see from 8.2 to 8.3 or 7.8 to 7.9 ... not major changes." https://lists.centos.org/pipermail/centos/2020-December/352374.html
Rawhide is a totally different beast. I lived on rawhide for a year just to learn from it back around 2005. I did learn a lot. It did hurt a lot. :-)
These days Rawhide isn't *quite* as painful as it was back in 2005. :)
"no, CentOS Stream isn't beta. Packages landing in Stream have already passed QA and gating." https://lists.centos.org/pipermail/centos/2020-December/352383.html
If "Red Hat Enterprise Linux" worth hardening had occurred, we wouldn't need CentOS 8 Stream. But, let's just see what it looks like for real... I'm genuinely curious as it may help us decode the secret sauce, and extrapolate the impacts...
Here is a recently updated package from AppStream:
http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/postgr...
A new version of PostgreSQL. Great! I'm a fan of PostgreSQL. :-)
According to the date on the mirror directory, it says: 2020-12-15 17:57
That's pretty recent. But, the date it was published shouldn't matter
- surely there will be quite a lot "QA and gating" that occurred,
right?
Build Date : Tue 15 Dec 2020 12:01:55 PM
Hmm... Built on Dec 15, 2020 @ 12:01:55 PM, published on Dec 15, 2020 @ 17:57 PM. Well, let's check the change log:
- Wed Nov 18 2020 Patrik Novotný panovotn@redhat.com - 10.15-1
- Rebase to upstream release 10.15 Resolves: rhbz#1898213 Resolves: rhbz#1898341 Resolves: rhbz#1901567
This one is interesting. In particular, it makes me wonder what is happening behind the scenes. I've seen two seemingly conflicting descriptions of what CentOS 8 Stream is:
- CentOS 8 Stream is where development happens first, allowing the
CentOS community to influence what goes into CentOS 8. This would imply "LATEST". This suggests that CentOS community is in control of the direction of RHEL.
- CentOS 8 Stream is the "next" (N+1?) version of RHEL 8. This might
NOT be "LATEST", but only "LATEST" as shared by Red Hat. Perhaps they are still maintaining a private N+2? This says that the CentOS community has only limited influence. They are seeing N+1, but not seeing N+2.
If the QA was being performed *after* it was built, then this means the change was queued from some time after November 18, until December 15, before being published within 6 hours (assumes the same time zones). Less than 6 hours of post-build testing was performed before releasing it to the public.
If the QA was being performed *before* it was built, then this implies that the build being published is not the build that Red Hat did QA on. The build procedure might be the same, but the actual build in CentOS 8 Stream is *not* the original build that was tested. This would suggest that a private Red Hat branch exists, possibly the N+2 branch? And this means that the end-to-end testing occurred some time between Nov 18 and Dec 15, on a private Red Hat branch, hidden from the CentOS community.
If somebody who actually knows how this works for real could explain, it would be enlightening? Which of the above conclusions is the closest to being right?
Now, I picked PostgreSQL above as it is a favourite of mine, and it happened to be recently updated in AppStream. But, more important to everybody else, is the kernel:
http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-4....
By the date on the mirror, it seems to say it was published on 2020-12-04 00:39. Let's look inside at build date, and the change log:
Build Date : Thu 03 Dec 2020 05:32:49 PM
This is similar to the PostgreSQL update, as it was built on Dec 3 @ 5:32:49 PM, and published to the mirror by Dec 4 @ 00:39, around 7 hours later?
Inside, however, the changelog
- Wed Dec 02 2020 Jan Stancek jstancek@redhat.com [4.18.0-257.el8]
Hmmm... The changelog is for Dec 2nd. The build date was Dec 3rd. The deploy date was Dec 4th.
Maybe this was an urgent change of some sort, and very tiny - so it was easy to QA?
Oh uh....
- Wed Dec 02 2020 Jan Stancek jstancek@redhat.com [4.18.0-257.el8]
- [hv] hv: vmbus: Allow cleanup of VMBUS_CONNECT_CPU if disconnected
(Mohammed Gamal) [1886096]
- [hv] hv: vmbus: Add parsing of VMbus interrupt in ACPI DSDT
(Mohammed Gamal) [1886096] .. 167 more lines ...
Wow. That's a pretty big change to QA in less than a day. And what about the prior change?
- Mon Nov 30 2020 Jan Stancek jstancek@redhat.com [4.18.0-256.el8]
- [kernel] PM: hibernate: Batch hibernate and resume IO requests
(Lenny Szubowicz) [1868096]
- [net] tunnels: Fix off-by-one in lower MTU bounds for ICMP/ICMPv6
replies (Antoine Tenart) [1895765] ... 124 more lines ...
Another big change, and so is the one after that, on Nov 27!
In fact, if you look at the changes since November 2, the changelog from 4.18.0-257 to 4.18.0-240 is 6149 lines long.
You just changed my mind!
*YOU* should definitely use CentOS 8 Stream! All of *YOU*. That way, by the time you test all of these changes, and report all the bugs, it won't affect *ME*. :-) :-) :-)
Making a reliable distribution doesn't make RHEL obsolete. There's still a lot of value in an OS that is itself semantically versioned, and has paid support contracts.
Yes there is. There is value, because it is important. That's what I'm trying to say. :-)
There's no evidence to support the idea that CentOS Stream will be unreliable because a reliable system would compete with RHEL. And lots of evidence to the contrary, including the extent to which Red Hat publishes their work in a form usable to rebuild the distribution.
"Unreliable" and "reliable" are extremes. RHEL 8 isn't 100% reliable, and CentOS 8 Stream isn't 100% unreliable.
However, RHEL 8 will be a lot more reliable if *YOU* test CentOS 8 Stream on *all* of your systems. The more, the better.
Judging by this response, I assume you're not familiar with how distribution testing typically works. At least for the past ten years, distribution testing of packages as they're updated has been almost entirely automated. Rather than spending ridiculous amounts of time having *people* test it, the gates are enforced by scripts, Ansible playbooks, etc. that execute workflows to validate behavior of a package. In the last couple of years, Red Hat has been rewriting their test suite for public consumption and pushing them into Fedora, so that packages can be validated at the earliest stage with Fedora ELN.
But let's focus on RHEL/CentOS for the moment. At the RHEL/CentOS Stream level, stuff is pushed to the RHEL internal Git server *first*. Their internal Git server is wired up with checks to run on commits being pushed so that tests run on the commit *before* it lands. Once it passes *those* tests, the commit lands and the package is built in "Brew" (the RHEL Koji instance). Once the package is built there, a series of integration and system validation tests are run before the build is accepted for release. Once *that* happens, it is pushed to the CentOS Git server (git.centos.org), and then the CentOS Stream build is kicked off. Once that build is made, it is then run through the CentOS test suite and validated based on that before pushing it out for release to CentOS Stream.
With the kernel in particular, the test suite probably validated it for release in less than half a day on the RHEL side. It was then pushed on December 3[1], Carl debranded it shortly after[2] and built on December 4[3], and released later that day after it passed the CentOS functional tests.
As someone who has also designed such systems in the past, I'd say that's actually pretty good. There's further work to be done, for sure, but there always is. It's all about continuous improvement. That's why I made suggestions about bringing ELRepo and other open source kernel modules into the CentOS Project as SIGs so that we can *add* to the kernel automated testing and prevent broken expectations with things like the kernel ABI that Red Hat maintains for the RHEL kernel[4][5]. Because if you look at CentOS Stream as the gift and opportunity that it truly is, then there are amazing things we can accomplish together in the CentOS Project.
[1]: https://git.centos.org/rpms/kernel/c/cf5e7d39f59af389825122cf70ca8ad664364ec... [2]: https://git.centos.org/rpms/kernel/c/1af03967e6580e55d84053c181555449e58da90... [3]: https://koji.mbox.centos.org/koji/buildinfo?buildID=14937 [4]: https://lists.centos.org/pipermail/centos-devel/2020-December/075593.html [5]: https://lists.centos.org/pipermail/centos-devel/2020-December/075615.html
-- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, Dec 21, 2020 at 8:44 AM Neal Gompa ngompa13@gmail.com wrote:
On Mon, Dec 21, 2020 at 4:28 AM Mark Mielke mark.mielke@gmail.com wrote:
If somebody who actually knows how this works for real could explain, it would be enlightening? Which of the above conclusions is the closest to being right?
Judging by this response, I assume you're not familiar with how distribution testing typically works. At least for the past ten years, distribution testing of packages as they're updated has been almost entirely automated. Rather than spending ridiculous amounts of time having *people* test it, the gates are enforced by scripts, Ansible playbooks, etc. that execute workflows to validate behavior of a package. In the last couple of years, Red Hat has been rewriting their test suite for public consumption and pushing them into Fedora, so that packages can be validated at the earliest stage with Fedora ELN.
Correct. I was not familiar with how Red Hat does distribution testing. I suspected, but I did not know. Your information helps fill in the missing gaps.
But let's focus on RHEL/CentOS for the moment. At the RHEL/CentOS Stream level, stuff is pushed to the RHEL internal Git server *first*. Their internal Git server is wired up with checks to run on commits being pushed so that tests run on the commit *before* it lands. Once it passes *those* tests, the commit lands and the package is built in "Brew" (the RHEL Koji instance). Once the package is built there, a series of integration and system validation tests are run before the build is accepted for release. Once *that* happens, it is pushed to the CentOS Git server (git.centos.org), and then the CentOS Stream build is kicked off. Once that build is made, it is then run through the CentOS test suite and validated based on that before pushing it out for release to CentOS Stream.
Good. CI/CD in action.
With the kernel in particular, the test suite probably validated it for release in less than half a day on the RHEL side. It was then pushed on December 3[1], Carl debranded it shortly after[2] and built on December 4[3], and released later that day after it passed the CentOS functional tests.
Without adding qualitative judgements here, this confirms that fundamentally:
1. There is an upstream RHEL-private branch. To reach the upstream RHEL-private branch, it must pass a set of automated tests. 2. Once a set of automated integration and system validation tests pass, which can complete in as little as 12 hours, the change is published to CentOS 8 Stream.
As someone who has also designed such systems in the past, I'd say that's actually pretty good. There's further work to be done, for sure, but there always is. It's all about continuous improvement. That's why I made suggestions about bringing ELRepo and other open source kernel modules into the CentOS Project as SIGs so that we can *add* to the kernel automated testing and prevent broken expectations with things like the kernel ABI that Red Hat maintains for the RHEL kernel[4][5]. Because if you look at CentOS Stream as the gift and opportunity that it truly is, then there are amazing things we can accomplish together in the CentOS Project.
Yes, it is good. I can easily see why this is of great value to people who produce content that needs to align with EL, especially to people such as yourself who understand the value of moving the automated testing upstream so that Red Hat would not automatically approve a change that would break your downstream use case, or even better - that Red Hat would include your downstream code into their upstream and do the testing for you (automated, to minimize expenses).
CentOS 8 Stream is a great option for people such as yourself - and should be a great option for other vendors. Unfortunately, most vendors are much more lazier than yourself (by historical evidence, and by their response to questions like "Do you support 7.9?"), so it isn't clear that you represent most vendors. Please take that as a compliment, although - if I'm being honest, I think you are doing what you should be doing, and they should take this as an insult. :-)
The value of CentOS 8 Stream as a new option, was never a question for me. I am sorry if my responses might have lead to the conclusion that I don't like CentOS 8 Stream. I *do* like it.
The problem is that CentOS 8 Stream is being sold as a replacement for CentOS 8, and phrases like "good enough for 95% of use cases" are being thrown around. I believe this to be a naive or ingenuous statement. However, not everybody is doing this. A significant number of people, including senior leadership from Red Hat, are clearly saying "CentOS is for development, RHEL is for production". These people are telling the truth, at least on 2020-12-21.
In some future, where the automated testing performed by a change lands in CentOS 8 Stream is good enough for production use, then it may well be "good enough for 95% of use cases". I don't believe it is, and I think anybody who believes it is today - is not making a correct assessment, and if they deploy CentOS 8 Stream in production without providing their own stabilization process, including what amounts to creating a baseline, and running their own patch program separate from RHEL/CentOS, they are very likely to have a rude awakening. They will be stuck in a hell where a change escapes Red Hat's automated testing, and they will have trouble upgrading or downgrading. Eventually, they will create their own repository that is locked to a specific baseline, and they will only permit packages in once it passes there own internal criteria, and they will then realize that certain breaking changes are not acceptable in their environment but that the breaking change is a dependency of some other change they do want. They will then need to learn to backport their own patches. And so on, until eventually they are doing a crappy job of re-implementing what CentOS already was. Probably, they will give up and move to some other distribution. Probably, it will not be RHEL.
Or, maybe all will be fine, and I have no idea what I am talking about. :-)
But, none of my concerns about dropping CentOS 8, are intended to imply that CentOS 8 Stream is bad. CentOS 8 Stream is good, and I totally get why you want it - and why I should want it.
On Mon, Dec 21, 2020 at 2:51 PM Mark Mielke mark.mielke@gmail.com wrote:
On Mon, Dec 21, 2020 at 8:44 AM Neal Gompa ngompa13@gmail.com wrote:
On Mon, Dec 21, 2020 at 4:28 AM Mark Mielke mark.mielke@gmail.com wrote:
If somebody who actually knows how this works for real could explain, it would be enlightening? Which of the above conclusions is the closest to being right?
Judging by this response, I assume you're not familiar with how distribution testing typically works. At least for the past ten years, distribution testing of packages as they're updated has been almost entirely automated. Rather than spending ridiculous amounts of time having *people* test it, the gates are enforced by scripts, Ansible playbooks, etc. that execute workflows to validate behavior of a package. In the last couple of years, Red Hat has been rewriting their test suite for public consumption and pushing them into Fedora, so that packages can be validated at the earliest stage with Fedora ELN.
Correct. I was not familiar with how Red Hat does distribution testing. I suspected, but I did not know. Your information helps fill in the missing gaps.
But let's focus on RHEL/CentOS for the moment. At the RHEL/CentOS Stream level, stuff is pushed to the RHEL internal Git server *first*. Their internal Git server is wired up with checks to run on commits being pushed so that tests run on the commit *before* it lands. Once it passes *those* tests, the commit lands and the package is built in "Brew" (the RHEL Koji instance). Once the package is built there, a series of integration and system validation tests are run before the build is accepted for release. Once *that* happens, it is pushed to the CentOS Git server (git.centos.org), and then the CentOS Stream build is kicked off. Once that build is made, it is then run through the CentOS test suite and validated based on that before pushing it out for release to CentOS Stream.
Good. CI/CD in action.
With the kernel in particular, the test suite probably validated it for release in less than half a day on the RHEL side. It was then pushed on December 3[1], Carl debranded it shortly after[2] and built on December 4[3], and released later that day after it passed the CentOS functional tests.
Without adding qualitative judgements here, this confirms that fundamentally:
- There is an upstream RHEL-private branch. To reach the upstream
RHEL-private branch, it must pass a set of automated tests. 2. Once a set of automated integration and system validation tests pass, which can complete in as little as 12 hours, the change is published to CentOS 8 Stream.
As someone who has also designed such systems in the past, I'd say that's actually pretty good. There's further work to be done, for sure, but there always is. It's all about continuous improvement. That's why I made suggestions about bringing ELRepo and other open source kernel modules into the CentOS Project as SIGs so that we can *add* to the kernel automated testing and prevent broken expectations with things like the kernel ABI that Red Hat maintains for the RHEL kernel[4][5]. Because if you look at CentOS Stream as the gift and opportunity that it truly is, then there are amazing things we can accomplish together in the CentOS Project.
Yes, it is good. I can easily see why this is of great value to people who produce content that needs to align with EL, especially to people such as yourself who understand the value of moving the automated testing upstream so that Red Hat would not automatically approve a change that would break your downstream use case, or even better - that Red Hat would include your downstream code into their upstream and do the testing for you (automated, to minimize expenses).
CentOS 8 Stream is a great option for people such as yourself - and should be a great option for other vendors. Unfortunately, most vendors are much more lazier than yourself (by historical evidence, and by their response to questions like "Do you support 7.9?"), so it isn't clear that you represent most vendors. Please take that as a compliment, although - if I'm being honest, I think you are doing what you should be doing, and they should take this as an insult. :-)
Believe me, I know *exactly* what you're talking about. My experience with those (nameless) horrible vendors is why I deliberately didn't do that at $DAYJOB. But I also think those vendors are going to be in for a rude awakening as RHEL continues to move faster and faster to support everything that paying customers are asking of Red Hat. I'm just ahead of the curve because I got burned by a vendor before, and I wouldn't inflict that on my customers if I can help it. :)
The value of CentOS 8 Stream as a new option, was never a question for me. I am sorry if my responses might have lead to the conclusion that I don't like CentOS 8 Stream. I *do* like it.
The problem is that CentOS 8 Stream is being sold as a replacement for CentOS 8, and phrases like "good enough for 95% of use cases" are being thrown around. I believe this to be a naive or ingenuous statement. However, not everybody is doing this. A significant number of people, including senior leadership from Red Hat, are clearly saying "CentOS is for development, RHEL is for production". These people are telling the truth, at least on 2020-12-21.
I'll say this flat-out as an independent user and contributor: CentOS Stream is *more* ready for production than CentOS Linux is. I personally run CentOS Stream 8 for my CentOS boxes because I've seen nothing but continuous improvement. Things *work* better on Stream for me than they did with the last checkpoint. That's why I switched from CentOS Linux 8 to CentOS Stream 8 back in the spring.
For some Red Hatters, there's some vested interest from some folks to maintain the confusion, but here's the cold, hard truth: CentOS Stream 8 isn't being tested any less than Red Hat Enterprise Linux. RHEL has no extra gates on top of CentOS Stream, since the RHEL tests happen *before* pushing to CentOS Stream.
The main reason for RHEL to have point releases is for easy delivery for certification validation. If you're using CentOS, those certifications do not apply to you anyway. :)
And a final note: CentOS has *never* supported staying on a particular point release. It's *always* been a "stream" on the major version. The difference is that we're not doing semi-annual large dumps and just pushing them out as they come.
In some future, where the automated testing performed by a change lands in CentOS 8 Stream is good enough for production use, then it may well be "good enough for 95% of use cases". I don't believe it is, and I think anybody who believes it is today - is not making a correct assessment, and if they deploy CentOS 8 Stream in production without providing their own stabilization process, including what amounts to creating a baseline, and running their own patch program separate from RHEL/CentOS, they are very likely to have a rude awakening. They will be stuck in a hell where a change escapes Red Hat's automated testing, and they will have trouble upgrading or downgrading. Eventually, they will create their own repository that is locked to a specific baseline, and they will only permit packages in once it passes there own internal criteria, and they will then realize that certain breaking changes are not acceptable in their environment but that the breaking change is a dependency of some other change they do want. They will then need to learn to backport their own patches. And so on, until eventually they are doing a crappy job of re-implementing what CentOS already was. Probably, they will give up and move to some other distribution. Probably, it will not be RHEL.
The other way to look at this is that you can add your own circumstances to the test suites and make sure they don't break you after they do it once. :)
The way the CentOS Stream project is set up, Red Hat is trying to work *with* the community to make a better project and a better product. I think that's awesome and should be applauded.
Or, maybe all will be fine, and I have no idea what I am talking about. :-)
But, none of my concerns about dropping CentOS 8, are intended to imply that CentOS 8 Stream is bad. CentOS 8 Stream is good, and I totally get why you want it - and why I should want it.
Honestly, I think everything will be fine. This whole thing was *very* poorly handled by Red Hat, but having lived through the RHL->RHEL/Fedora change, I can say it could have gone worse. :)
On 12/21/20 5:43 AM, Neal Gompa wrote:
distribution testing of packages as they're updated has been almost entirely automated. Rather than spending ridiculous amounts of time having*people* test it
Thank you, Neal.
Most of the technical conferences that I attend, year after year, include groups that stress the difficulty of improving anything due to the "That's the way we've always done it" barrier.
I think that behind a lot of the fear of using CentOS Stream there's an implicit belief that the more time passes between a piece of software being built and an end-user deploying that software, the more reliable or trustworthy that software will be. I can't think of a mechanism by which that could possibly be true for pre-release packages. Letting packages sit, idle, until the next point release doesn't make them more reliable. (And Red Hatters have told us that there are virtually no beta testers doing any testing).
On the other hand, it is theoretically possible that bugs will be found through real-world testing after release. At this point, I think it's all but explicit that some of the most vocal objectors view all software as a "beta" until someone else has used it for some period of time. For those people, the value of the point releases is that they believe they can reasonably expect to wait for an arbitrary period after a point release to let other users work out the bugs. Rather than testing software themselves, they're relying on the broader community to deploy software and report bugs. To be clear, though, there's no evidence that doing so is necessary or effective. If it were, we'd expect to see a flurry of fixes for bugs that were introduced BY the point release after each point release.
There's a non-trivial amount of "that's the way we've always done it" in many of the arguments against moving to CentOS Stream. In my opinion, the more we do to push back against the idea that humans should be involved in the QA process itself, and the more we do to expand our trust in automated tests (which we also continue to expand and improve), the better we become as a community and an industry.
On Mon, 21 Dec 2020 at 16:18, Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/21/20 5:43 AM, Neal Gompa wrote:
distribution testing of packages as they're updated has been almost entirely automated. Rather than spending ridiculous amounts of time having*people* test it
....
There's a non-trivial amount of "that's the way we've always done it" in many of the arguments against moving to CentOS Stream. In my opinion, the more we do to push back against the idea that humans should be involved in the QA process itself, and the more we do to expand our trust in automated tests (which we also continue to expand and improve), the better we become as a community and an industry.
A good reason it is the way we've always done it is because it worked and very few of us have the time to reinvent the wheel in our infrastructure. Most of our time is being spent keeping aging infrastructure that you have said needed to be replaced for years... but well it still works and replacing costs more money this year than last year. Another part of our time is spent dealing with a lot meetings, trouble tickets from users to bosses, the latest fire of the week (OMG the billing system died again), etc etc. Having a set of tools which work reliably like they have for years and for the budget that you have been given ($0) is about all you can ask for. Many of us are not trained as system administrators and there is always the knowledge that we will become obsolete at the drop of a release.
All of that is going to make a lot of people really afraid. They are afraid that they won't have enough time as they know it took them 4 years to get approval to get off of EL6 and their management isn't going to spend any more money than they have in the past. They are afraid that they are going to have to babysit servers when they didn't have to before. They are afraid that they are going to have to work even more overtime to keep up with weekly with whatever 'upgrade' got put into CentOS Stream. And there is the fear that the tool they have been using is going ot become even more of a knock-off than it was before. Yes it was the Craftsman of handtools but now its going to be used by the manufacturer for prerelease work versus post.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Mon, Dec 21, 2020 at 4:18 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On the other hand, it is theoretically possible that bugs will be found through real-world testing after release. At this point, I think it's all but explicit that some of the most vocal objectors view all software as a "beta" until someone else has used it for some period of time. For those people, the value of the point releases is that they believe they can reasonably expect to wait for an arbitrary period after a point release to let other users work out the bugs. Rather than testing software themselves, they're relying on the broader community to deploy software and report bugs. To be clear, though, there's no evidence that doing so is necessary or effective. If it were, we'd expect to see a flurry of fixes for bugs that were introduced BY the point release after each point release.
I can't speak for all vocal objectors - but that's not it for me.
I am happy to test. That's why I use Fedora as my home system, and even in certain production use cases, such as containers that run particular applications. We catch problems all the time. CentOS 8 Stream is a great opportunity to test.
But what I'm testing with, is not what I send out to thousands of machines that are being used for production purposes to build product.
If minor releases cease to exist, we will be forced to develop our own in-house minor releases. Which is not an insurmountable problem - we already do to it a degree. But, the bigger problem is in agreement on what those minor releases are. Without milestones to build against, and certify against, there is also no option except to upgrade to latest. For example, if Vendor X builds something in Jan, 2021, what happens if some daily update comes down the pipeline that breaks the Vendor X build, and our production use case stops working? Now, imagine Vendor Y builds something in Jan, 2022 that requires latest. But, we can't upgrade to latest because this breaks Vendor X. Now multiple this by 1000 different tools and tool versions.
We have internal product releases that were last built in 2016, or 2008, or even 1999, that would take a substantial amount of money to update to work on latest releases. Today, we can do this by running older releases. Without these baselines, this becomes "not possible". It's fundamentally the same problem as the buildroot problem for 100% perfect reproduction of RPM, but extended beyond RPM to using EL as a development platform. We encourage users to use versioned tool chains, and containers to control this, and buildroots have been used in the past - but the existence of tools such as "gcc" and "make" in the PATH basically have the natural consequence that somewhere in the build process it is relying on the native tools, and upgrades break builds.
I can appreciate that you are not aware of this problem - or perhaps you don't agree with the problem, and think that it should not exist. But, it's not correct to over-simplify the problem space. Versioning of products exists for important reasons. Whether a distro can or should be versioned, including minor versions, depends upon the requirements. Clearly, your requirements don't align with mine - and you are ok with results that I am not allowed to be ok with.
On 12/21/20 2:27 PM, Mark Mielke wrote:
We have internal product releases that were last built in 2016, or 2008, or even 1999, that would take a substantial amount of money to update to work on latest releases. Today, we can do this by running older releases. Without these baselines, this becomes "not possible". It's fundamentally the same problem as the buildroot problem for 100% perfect reproduction of RPM, but extended beyond RPM to using EL as a development platform.
I'm aware. I work for a vendor that produces binary software for GNU/Linux systems. This is the forward-compatibility problem that I've mentioned previously.
Because RHEL isn't necessarily forward compatible with future releases, we may need to build our applications on the oldest release that we support deployments on. That's difficult to do with CentOS Stream. I can think of a couple of possible solutions, but generally, it will be difficult to use CentOS Stream as a build root for an RHEL target.
I've been meaning to ask Red Hat if there are programs for vendors that need to build software for RHEL, but I haven't yet. I believe that EPEL has used CenOS as its build root in the past, so I assume this is something that they'll need to solve for. For now, that's one of the exceptions that stands out when I say that I expect CentOS Stream to be better for most purposes.
I can appreciate that you are not aware of this problem
Dude. I'm trying to be civil. I'm not really sure if there's a polite way to point out that a whole lot of your arguments seem to conclude and rest on your perception of yourself as the keeper of secret knowledge. You're not, and it's kind of rude.
On Mon, Dec 21, 2020 at 7:53 PM Gordon Messmer gordon.messmer@gmail.com wrote:
I can appreciate that you are not aware of this problem
Dude. I'm trying to be civil. I'm not really sure if there's a polite way to point out that a whole lot of your arguments seem to conclude and rest on your perception of yourself as the keeper of secret knowledge. You're not, and it's kind of rude.
Of course you conveniently snipped where I said "or you don't agree it should be a problem."
It's hard to conclude else, when you only seem to remember actual problems after I raise them. If from the start you had said "I totally get why you need minor versions, here is proof", then we wouldn't be in an argument. Instead, you over simplify and say things like "it's merely semantic versioning" or "maybe people can't change". These are statements that somebody who does not understand would make.
So, I'll take you at your word that you understand, and you knew about this information - but then please, start allowing for it earlier in the discussion.
On Mon, Dec 21, 2020 at 04:53:02PM -0800, Gordon Messmer wrote:
I've been meaning to ask Red Hat if there are programs for vendors that need to build software for RHEL, but I haven't yet. I believe that EPEL has used CenOS as its build root in the past, so I assume this is something that they'll need to solve for. For now, that's one of the exceptions that stands out when I say that I expect CentOS Stream to be better for most purposes.
Please do ask -- either through your channels or through the centos-questions@redhat.com address. This is definitely one of the areas being looked at.
On Tue, Dec 22, 2020 at 2:01 PM Matthew Miller mattdm@mattdm.org wrote:
On Mon, Dec 21, 2020 at 04:53:02PM -0800, Gordon Messmer wrote:
I've been meaning to ask Red Hat if there are programs for vendors that need to build software for RHEL, but I haven't yet. I believe that EPEL has used CenOS as its build root in the past, so I assume this is something that they'll need to solve for. For now, that's one of the exceptions that stands out when I say that I expect CentOS Stream to be better for most purposes.
Please do ask -- either through your channels or through the centos-questions@redhat.com address. This is definitely one of the areas being looked at.
If the goal is purely for qualifying a solution on Red Hat Enterprise Linux, they have these:
https://access.redhat.com/solutions/5181471
These "Not-For-Resale" subscriptions can be used for specific use cases related to "customer test". The problem is that the criteria around these subscriptions is very specific, and to ensure ethical and legal compliance, it basically means sectioning off a set of machines to be for RHEL qualification only. It does not address the rest of the environment, and the subscription model remains a problem for the rest of the environment, which leads to the following result:
1. RHEL for production use cases that require RHEL, with no real incentive for for expanding this beyond the absolute minimum.
2. Something else for primary development, particularly including large workstations and build / test farms. CentOS? CentOS Stream? Oracle Linux? Rocky? Ubuntu? ...
3. NFR as a sectioned off part of the environment that is specifically for the "customer test" / "qualification" scenarios that the NFR subscription permits.
The part that makes me sad is #2. If I sound angry, or frustrated in my posts - it is because Red Hat literally does not want *some* money from #2, and this is beyond comprehension for me. They have an all-or-nothing approach which is contrary to other popular distributions, which consequently makes other distributions more appealing. Oracle Linux and Ubuntu make the software releases free, but the support optional. This then opens the door for negotiations on support, so that you can individually subscribe systems, or you can come to a deal on "all systems". Red Hat is not willing to make a deal for "all systems". This makes it really difficult to sell the company on why we should choose RHEL for #2. Red Hat wants to *sell* us free software, including free software that we contributed to. They are not selling access to their incredible people that work for them - they are selling software. That's what the subscription model boils down to, and that is why I say their subscription model is fundamentally broken.
"Red Hat" is a big company. It's not fair to attribute every motivation to every person. There are a huge number of people in Red Hat that are struggling with what is happening, and trying to reconcile the situation in their own heads. Things like "well, maybe CentOS 8 Stream is better!" is an example of this reconciling in action. But, just look at the quotes in this article that are quite believable, that explain what is really going on here:
https://www.zdnet.com/article/why-red-hat-dumped-centos-for-centos-stream/
In particular:
"A former Red Hat executive confided, "CentOS was gutting sales. The customer perception was 'it's from Red Hat and it's a clone of RHEL, so it's good to go!' It's not. It's a second-rate copy." From where, this person sits, "This is 100% defensive to stave off more losses to CentOS."
Still another ex-Red Hat official said. If it wasn't for CentOS, Red Hat would have been a 10-billion dollar company before Red Hat became a billion-dollar business."
This is about business models and money. It may be that CentOS 8 Stream was a successful negotiation to prevent Red Hat Exec from completely eliminating CentOS, but the driving force that started this all was Red Hat taking over CentOS by becoming a principal owner of it, and then cancelling it.
I don't want Red Hat to be free. However, I don't accept that Red Hat can define whatever price they wish, and we should automatically consider it a bargain. Just as Red Hat is a business, making business decisions, the "users" that they disdain are also part of businesses, making business decisions:
"Red Hat hasn't been talking about this aspect much at all, but Mike McGrath, Red Hat's VP of Linux Engineering, let the cat out of the bag in an interview with Christine Hall in ITPro Today. "I would say the big one for us was that CentOS itself was not actually providing that much usefulness to Red Hat. Most of the communities we set up, Fedora, for example, do have a lot of bidirectional community involvement. Unfortunately, CentOS was never like that. It was always a community of users, so that contribution model was mostly one way.""
On Sun, 20 Dec 2020 at 22:19, Mark Mielke mark.mielke@gmail.com wrote:
On Sun, Dec 20, 2020 at 6:34 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 8:27 PM, Nico Kadel-Garcia wrote:
On Sat, Dec 19, 2020 at 12:29 PM Matthew Miller mattdm@mattdm.org
wrote:
It's important to note that the CentOS Linux rebuild never actually
had
this. RHEL minor releases are actually branches, and you can stay at
a minor
release and still get security updates.
Are you saying the CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer
that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
We came up with the phrase "bug-for-bug" compatible during EL5 as a GOAL to aim for. CentOS was NEVER bug-for-bug compatible. We aimed for it like a target to get to but we also had to release the software eventually and don't have the extensive testing mechanisms to prove 'bug-for-bug' compatibility. Sometimes CentOS shipped packages which did not have a particular bug because we could not exactly duplicate the build environment and other times we added new bugs because our build environment is not exactly the same.
Over the years as software has grown in greater complexity this divergence has become more likely. Modularity, software like rust, go, etc all require long build chains where any 'divergence' can introduce or remove bugs without a quick way to test.
At best, CentOS has been "good-enough" compatible for a set of years. The only way to be bug-for-bug compatible would require a reproducible build and test environment.
On Mon, Dec 21, 2020 at 7:59 AM Stephen John Smoogen smooge@gmail.com wrote:
On Sun, 20 Dec 2020 at 22:19, Mark Mielke mark.mielke@gmail.com wrote:
CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not offer that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
We came up with the phrase "bug-for-bug" compatible during EL5 as a GOAL to aim for. CentOS was NEVER bug-for-bug compatible. We aimed for it like a target to get to but we also had to release the software eventually and don't have the extensive testing mechanisms to prove 'bug-for-bug' compatibility. Sometimes CentOS shipped packages which did not have a particular bug because we could not exactly duplicate the build environment and other times we added new bugs because our build environment is not exactly the same.
This is a good point of clarification. But, to be clear in both directions, please let us know if you agree that these points are correct:
1. CentOS was intended to be "bug-for-bug" compatible with RHEL as a GOAL. 2. This was especially difficult when Red Hat was not contributing to the project, as it involved a lot of guesswork and trial and error. 3. This became easier when Red Hat began contributing, including staging the source for build, and providing the build engines. 4. It's still not perfect, however, in almost every way there is currently a 1:1 relationship between the exact .spec file and patch set being used to build CentOS releases, in parallel (or a short time later) to RHEL minor release. 5. CentOS 8 Stream is a departure from the above. What is built in CentOS 8 Stream is *not* intended to be "bug-for-bug" compatible, but is instead a moving target that represents the *next* RHEL minor release. At any given point in time, it is not necessarily aligned with either RHEL X.N, or RHEL X.N+1, but is something in between, or may even be something beyond, such as RHEL X.N+2 depending upon RHEL internal processes.
Over the years as software has grown in greater complexity this divergence has become more likely. Modularity, software like rust, go, etc all require long build chains where any 'divergence' can introduce or remove bugs without a quick way to test.
Yep. The reproducibility problem inherent to RHEL has become more complex, rather than less. Dropping CentOS 8 essentially abandons this problem from a Red Hat perspective. Changing the subscription terms for RHEL to allow the RHEL binaries to be directly used would solve this problem from a Red Hat perspective (but possibly introduce others, mostly relating to coming up with a better financially sustainable model). Documenting (possible as configuration and code) the build requirements for packages may be a cleaner solution.
At best, CentOS has been "good-enough" compatible for a set of years. The only way to be bug-for-bug compatible would require a reproducible build and test environment.
I think "good-enough" is in the 99.9%+ today, depending upon what the test is. With CentOS 8 Stream, it drops to something less, also depending upon what the test is.
I think the people who understand the delta best, may be under-representing the value they provided. The people who understand it the least, never really appreciated the effort. :-)
On Mon, 21 Dec 2020 at 14:31, Mark Mielke mark.mielke@gmail.com wrote:
On Mon, Dec 21, 2020 at 7:59 AM Stephen John Smoogen smooge@gmail.com wrote:
On Sun, 20 Dec 2020 at 22:19, Mark Mielke mark.mielke@gmail.com wrote:
CentOS point releases do *not* match as closely as possible the corresponding RHEL point release?
No, no one is saying that. Matthew said that you can stay at a minor release of RHEL and still get security updates. CentOS does not
offer that.
This is not correct. Please stop saying it. CentOS is bug-for-bug compatible with RHEL for *active* releases.
We came up with the phrase "bug-for-bug" compatible during EL5 as a GOAL
to aim for. CentOS was NEVER bug-for-bug compatible. We aimed for it like a target to get to but we also had to release the software eventually and don't have the extensive testing mechanisms to prove 'bug-for-bug' compatibility. Sometimes CentOS shipped packages which did not have a particular bug because we could not exactly duplicate the build environment and other times we added new bugs because our build environment is not exactly the same.
This is a good point of clarification. But, to be clear in both directions, please let us know if you agree that these points are correct:
- CentOS was intended to be "bug-for-bug" compatible with RHEL as a GOAL.
- This was especially difficult when Red Hat was not contributing to
the project, as it involved a lot of guesswork and trial and error. 3. This became easier when Red Hat began contributing, including staging the source for build, and providing the build engines.
The methods for building the source to solution changed from dropping .src.rpm to git pushes in EL7 time frame. The build engine changed from running a highly modified version of plague? (reimzul) to koji. This did not make things easier or worse overall. Some things became easier and some things became harder.
What became easier was that Red Hat provided a lot of hardware and steady paychecks for 5 or 6 people who had been doing the builds in their spare time and out of pocket expenses.
4. It's still not perfect, however, in almost every way there is
currently a 1:1 relationship between the exact .spec file and patch set being used to build CentOS releases, in parallel (or a short time later) to RHEL minor release.
A spec file is the tiniest part of making something match. Build order, hardware, etc have a larger effect.
- CentOS 8 Stream is a departure from the above. What is built in
CentOS 8 Stream is *not* intended to be "bug-for-bug" compatible, but
CentOS-8 has not been bug-for-bug compatible in the first place. We would still be working the initial release if we were trying that. So please let us rephrase this
5. CentOS 8 Stream is not intended to have the goal of being "bug-for-bug" compatible. It is recognizing this is not possible and has not been possible since probably EL5. At any given time it is not necessarily aligned with RHEL X.Y or RHEL X.Y+1 but may be something in between or beyond depending on when the release cycle is happening.
[That is the best over-simplification I can come up with this for.]
On Mon, Dec 21, 2020 at 2:50 PM Stephen John Smoogen smooge@gmail.com wrote:
On Mon, 21 Dec 2020 at 14:31, Mark Mielke mark.mielke@gmail.com wrote:
- It's still not perfect, however, in almost every way there is
currently a 1:1 relationship between the exact .spec file and patch set being used to build CentOS releases, in parallel (or a short time later) to RHEL minor release.
A spec file is the tiniest part of making something match. Build order, hardware, etc have a larger effect.
I think your definition of match is overly pedantic, and I love it. :-)
When it comes to behaviour exhibited by the program - my own experience is that source bundle and the patch set primary defines the visible and user-impacting behaviours. When an issue is raised on bugzilla.redhat.com - I know that there are some issues that have been addressed through package re-building, or changing the order of the build (and yes, Modularity makes this much harder), but the vast majority of issues by quantity relate to problems in the software, that are only addressed through source changes, that make it into builds. The definition of which source bundle is used, and which patches are applied on top, is the .spec file. Basically every software fix on bugzilla.redhat.com turns into a .spec file change. So, I do believe that .spec file is the biggest part of making things match from a visible and user-impacting behaviouors perspective.
However, .spec file alone is not sufficient. As you say, the build context matters as well, and this is the next level of reproducibility that has been difficult to achieve (although, I believe it comes "close enough" that most people cannot tell the difference even if they were looking for one). If the .spec file were to embed the "build requirements" not just as package dependencies, but also things like Modularity configuration, and build baseline, this would improve things. But, today - everything is sort of built on latest, and the only time something is done to fix this, is when a problem is found. CentOS 8 Stream accepts this, and makes it official. Dropping CentOS 8, means dropping any attempt to continue to solve this.
- CentOS 8 Stream is a departure from the above. What is built in
CentOS 8 Stream is *not* intended to be "bug-for-bug" compatible, but
CentOS-8 has not been bug-for-bug compatible in the first place. We would still be working the initial release if we were trying that. So please let us rephrase this 5. CentOS 8 Stream is not intended to have the goal of being "bug-for-bug" compatible. It is recognizing this is not possible and has not been possible since probably EL5. At any given time it is not necessarily aligned with RHEL X.Y or RHEL X.Y+1 but may be something in between or beyond depending on when the release cycle is happening. [That is the best over-simplification I can come up with this for.]
With the understanding that we are talking about "perfect reproduction" and not "reproduction that would pass most inspections", I would agree. :-)
On 12/21/20 3:07 PM, Mark Mielke wrote:
On Mon, Dec 21, 2020 at 2:50 PM Stephen John Smoogen smooge@gmail.com wrote:
On Mon, 21 Dec 2020 at 14:31, Mark Mielke mark.mielke@gmail.com wrote:
- It's still not perfect, however, in almost every way there is
currently a 1:1 relationship between the exact .spec file and patch set being used to build CentOS releases, in parallel (or a short time later) to RHEL minor release.
A spec file is the tiniest part of making something match. Build order, hardware, etc have a larger effect.
I think your definition of match is overly pedantic, and I love it. :-) When it comes to behaviour exhibited by the program - my own experience is that source bundle and the patch set primary defines the visible and user-impacting behaviours. ... With the understanding that we are talking about "perfect reproduction" and not "reproduction that would pass most inspections", I would agree. :-)
Perfect reproduction requires: 1.) Same source (easy) 2.) Same spec (easy) 3.) Same patches (easy) 4.) Same binary buildroot (HARD; same versions of every package in upstream buildroot, where each of those packages has been built in a perfectly reproduced environment; the exact contents of any given package's buildroot can be dramatically different from another distributed package's buildroot; Modularity greatly complicates this) 5.) Same .rpmmacro setup (not impossible, but not easy)
While item 4 often gets oversimplified as 'build order' (I've said it that way recently myself) it actually is beyond that, since each particular buildroot might not contain the very latest built packages but earlier ones (I don't know how the RHEL buildroots are populated). The many tools that have been developed over the years to try to make this happen are better now than they were at the beginning (mach, anyone?), but it still sometimes requires guesswork.
While I've not been doing this at the same scale as you and others, I have done this before; you and I have the same favorite package, PostgreSQL.... :-)
On 12/21/20 11:49 AM, Stephen John Smoogen wrote:
- CentOS 8 Stream is not intended to have the goal of being
"bug-for-bug" compatible. It is recognizing this is not possible and has not been possible since probably EL5. At any given time it is not necessarily aligned with RHEL X.Y or RHEL X.Y+1 but may be something in between or beyond depending on when the release cycle is happening.
I recall talking about this with you and others in 2013, when we were trying to define "what IS CentOS Linux" compared to what people thought it was. Red Hat didn't want to change the project for the worse, and that required us having a baseline idea of what the project really did.
The reason this matters is that in 2014, it was clear that "bug-for-bug compatible" was the EL distro equivalent of "all natural ingredients." It was a lofty goal that became a comforting phrase.
Some people argued for having that goal phrasing changed for the project, because it was an impossible goal and setup an impossible expectation with the users. But frankly we also didn't want to worry users by making that change, and it became one of a many things about the CentOS brand-as-a-sponge that is hard to change.
One thing about this whole experience, it is bringing a lot of spoken and unpsoken expectations about the brand directly into conversation with the people at Red Hat who know and understand how RHEL actually gets made. I appreciate all the clever detective work in this thread, but much of it is the exercise of trying to describe an object in the dark by feel when the object is an elephant.
Let's bring the elephant out of the room into the sunlight, I'm getting tired of building better flashlights.
Kind regards,
- Karsten
On 12/19/20 1:34 AM, Mark Mielke wrote:
However, these are significant reasons why CentOS Linux is superior to CentOS Stream:
- Bug-for-bug compatibility with RHEL. This is important or a variety
of reasons, particularly including reproducibility. 2. Minor release milestones to stabilize branches.
Reproducibility *is* important. Minor releases, however, are not necessary to achieve reproducibility, nor are they sufficient. Red Hat does fix bugs within minor releases:
https://access.redhat.com/errata/#/?q=&p=1&sort=portal_publication_d...
Therefore, if you operate an environment where reproducibility in your test environment is a critical feature, you should be testing and deploying on the *same* underlying OS. Maybe that means you test and then deploy gold images, or containers. Immutable infrastructure is popular for good reason.
If you have mutable systems, there's no mechanism to keep CentOS and RHEL packages perfectly in sync. You're not actually getting reproducible builds out of that setup, even if you think you are. It doesn't make a good argument in favor of keeping CentOS.
I don't agree with you that CentOS cannot be two things. It's quite normal for most projects to have an "upstream" and a "LTS" branch.
Fedora is the upstream branch. CentOS Stream is the LTS branch. RHEL is the LTS branch with point releases (semantic versioning) and vendor support.
On Sat, Dec 19, 2020 at 3:28 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/19/20 1:34 AM, Mark Mielke wrote:
However, these are significant reasons why CentOS Linux is superior to CentOS Stream:
- Bug-for-bug compatibility with RHEL. This is important or a variety
of reasons, particularly including reproducibility. 2. Minor release milestones to stabilize branches.
Reproducibility *is* important. Minor releases, however, are not necessary to achieve reproducibility, nor are they sufficient. Red Hat does fix bugs within minor releases:
https://access.redhat.com/errata/#/?q=&p=1&sort=portal_publication_d...
Therefore, if you operate an environment where reproducibility in your test environment is a critical feature, you should be testing and deploying on the *same* underlying OS. Maybe that means you test and then deploy gold images, or containers. Immutable infrastructure is popular for good reason.
This will be *much* harder with CentOS Stream only. The point releases, much like RHEL point release mdeida which they effectively mirrored, provide a stable release tag to work from From the description, this is being discarded entirely.
If you have mutable systems, there's no mechanism to keep CentOS and RHEL packages perfectly in sync. You're not actually getting
They never were, and they never will be unless Red Hat returns to what it did before RHEL and publishes precisely the same packages available to the public without an extra release team.
reproducible builds out of that setup, even if you think you are. It
The point releases were a solid starting part. There has been a push from Red Hat to get customers onto a "streaming release" model, and especially to encourage people to pay for RHN to do host-by-host package installation tuning. I don't know anyone who likes it. I've gotten years of consulting work for big companies keeping *away* from spending all their time hand-tuninging individual package-by-package tuning with RHN.
doesn't make a good argument in favor of keeping CentOS.
Same RPM tree, defined package list, defined build material, golden images for media or PXE installation and for generating build environments from scratch. This predated mock: I deployed approximately 13,000 hosts in one month this way.
I don't agree with you that CentOS cannot be two things. It's quite normal for most projects to have an "upstream" and a "LTS" branch.
Fedora is the upstream branch. CentOS Stream is the LTS branch. RHEL is the LTS branch with point releases (semantic versioning) and vendor support.
I'm staring at the announcement at https://centos.org/distro-faq/, which blatantly contradicts:
Q5: Does this mean that CentOS Stream is the RHEL BETA test platform now?
A: No. CentOS Stream will be getting fixes and features ahead of RHEL.
There's a Greek word for that. It's spelled "B-E-T-A". I soo no assurance that anything in CentOS Stream will ever make it to RHEL, and I expect such discarded or retuned binaries to be a real source of pain. I expect to be particularly awful when a client-driven patch for commercial RHEL is split-brain with ongoing patches or development in CentOs Stream. If you think tha tcan't happen, I'll point you to the kernel patches for ext2 optimization in the days when I worked for CDN companies.
On Sat, Dec 19, 2020 at 1:44 AM Karsten Wade kwade@redhat.com wrote:
I wrote a blog post to share with you: https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platfo...
Lots of posts later - I think everybody knows where everybody stands, and there isn't much point in debating further. It sounds like it is a done deal that CentOS 8 is gone, that CentOS 8 Stream is live, and that if we don't like it we need to look elsewhere.
I have the same requirements I did in 2016 when we first started to consider options, and ask Red Hat for options. At the time, I was a fan of Red Hat, and I had looked to Red Hat to provide some sort of acceptable compromise on subscription that would allow for a win/win. The offer provided was not appreciably different from list price of subscribing every single machine in the company no matter whether it needed support or not, and this made other options attrative. This forced us to (sadly) remove Red Hat Enterprise Linux from our systems, and replace it with something else. We want to stay a part of the community, but we are not likely to be able to use CentOS 8 Stream. As much as there is assurances from people on this list that CentOS 8 Stream is similar to, or better than CentOS 8 for production purposes, CentOS 8 Stream is not likely to meet our requirements.
Please let us know what alternate options Red Hat has for CentOS 8. Otherwise, we will be looking elsewhere.
Thanks,
-
Brendan Conoboy -
Gordon Messmer -
Jean-Marc Liger -
Karsten Wade -
Konstantin Boyandin -
Lamar Owen -
Leon Fauster -
Ljubomir Ljubojevic -
Mark Mielke -
Matthew Miller -
Mike McGrath -
Neal Gompa -
Nico Kadel-Garcia -
Peter Georg -
Phil Perry -
Simon Matter -
Stephen John Smoogen -
Tetsuo Handa -
Trevor Hemsley