As we get closer to the CentOS Linux 8 EOL, we're getting more and more questions about how the EOL is actually going to work. While there tends to be a great deal of "everybody just knows how it works" among the people actually on this list, we're getting attention from a much larger audience now, and they don't know.
To that end, I want to make sure we have clear documentation, prominently displayed, that sets expectations as we get closer to that December 31 date,
So ... as I understand, our process is basically:
1. Copy old data to vault.centos.org 2. Update web pages and links to say it is now on vault.centos.org 3. Turn off mirrorlist to that release name (aka all requests to CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
Is there more to it than this? Will we do anything different this time because CL8 is a special case? Am I missing any steps, or have I misunderstood any of the steps? Is this already documented somewhere that I simply haven't found yet?
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
Hi,
On 7/8/21 7:10 PM, Rich Bowen wrote:
- Turn off mirrorlist to that release name (aka all requests to
CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
...
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
I'm not sure what the timelines for this whole process are, but I'd like to advocate for not deleting everything from the main mirrors on the 31st. Having the C8 content there for an extra month or so would be nice, particularly given the holiday period.
Cheers, Alex
On 7/8/21 12:44 PM, Alex Iribarren wrote:
Hi,
On 7/8/21 7:10 PM, Rich Bowen wrote:
- Turn off mirrorlist to that release name (aka all requests to
CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
...
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
I'm not sure what the timelines for this whole process are, but I'd like to advocate for not deleting everything from the main mirrors on the 31st. Having the C8 content there for an extra month or so would be nice, particularly given the holiday period.
Of course the problem with that is, we will not be adding any security updates past that point.
I doubt it will happen exactly on Dec 31 as our people will not be working because of the holiday on that date as well.
We are still working on the absolute details of this process as far as exact dates go. Our goal is, so long as RHEL 8.5 releases before 31 DEC 2021, we will get the files from 8.5 released before we remove CentOS Linux 8 from the mirrors. We will not be adding any updates from RHEL source code released after 31 DEC 2021 to CentOS Linux 8.
This release will be put into at least vault.centos.org/8.5.xxxx/ (where xxxx is the date). Of course, if the RHEL 8.5 release happens after 01 Jan 2022, we would not be doing that release in CentOS Linux 8.
New items (to CentOS Stream 8) will be being built and still going into CentOS Stream 8 after 01 JAN 2022 until CentOS Stream 8 EOLs 5 years after the RHEL 8 release (EOL is 31 May 2024).
We do need to decide exactly when this will take place and get that info out on this list. But it likely needs to be soon after we stop publishing updates as the moment a security update comes out in 2022, you are at risk with CentOS 8 Linux from that point on.
You should absolutely have migrated well before this date to something else. That something else could be RHEL (7 or 8) .. OR
As fully free options, CentOS Linux 7 until 30 Jun 2024 OR CentOS Stream 8 OR one of the other downstream builds that exist.
We will post here what the exact removal date goal is.
Thanks, Johnny Hughes
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
1. Change mirrorlist to respond to 8 repo requests with 8-stream repos at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
2. Same as 1, but wait some period of time after the EOL date, such as 1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
3. Don't change mirrorlist to respond to 8 repo requests with 8-stream repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
[0] https://twitter.com/mattdm/status/1411012254812852225
On Thu, Jul 8, 2021 at 12:10 PM Rich Bowen rbowen@redhat.com wrote:
As we get closer to the CentOS Linux 8 EOL, we're getting more and more questions about how the EOL is actually going to work. While there tends to be a great deal of "everybody just knows how it works" among the people actually on this list, we're getting attention from a much larger audience now, and they don't know.
To that end, I want to make sure we have clear documentation, prominently displayed, that sets expectations as we get closer to that December 31 date,
So ... as I understand, our process is basically:
- Copy old data to vault.centos.org
- Update web pages and links to say it is now on vault.centos.org
- Turn off mirrorlist to that release name (aka all requests to
CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
Is there more to it than this? Will we do anything different this time because CL8 is a special case? Am I missing any steps, or have I misunderstood any of the steps? Is this already documented somewhere that I simply haven't found yet?
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Thu, Jul 8, 2021 at 3:40 PM Carl George carl@redhat.com wrote:
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I would prefer to do this change 1-2 months after December 31, though I'm quite fine with doing it on the EOL date. Having run CentOS Stream 8 for two years now, I feel that it's a solid experience and most people aren't going to have issues with it after upgrading from classic CentOS 8. And frankly, it's nice having people respond to my bug reports and actually *do* something about it. Strictly speaking, that's an upgrade from the experience with classic CentOS 8.
- Don't change mirrorlist to respond to 8 repo requests with 8-stream
repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
I think this is a terrible idea and does a disservice to the large "silent majority" of CentOS systems that would otherwise be fine with the upgrade.
On 7/8/21 4:08 PM, Neal Gompa wrote:
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I would prefer to do this change 1-2 months after December 31, though I'm quite fine with doing it on the EOL date. Having run CentOS Stream 8 for two years now, I feel that it's a solid experience and most people aren't going to have issues with it after upgrading from classic CentOS 8. And frankly, it's nice having people respond to my bug reports and actually*do* something about it. Strictly speaking, that's an upgrade from the experience with classic CentOS 8.
While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.
Surprising users seldom goes well, even if it's an overall positive surprise.
On Thu, 2021-07-08 at 16:22 -0400, Rich Bowen wrote:
While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.
Surprising users seldom goes well, even if it's an overall positive surprise.
What are our options for making it less of a surprise? We could post a blog, etc. about it of course (and we should, if we decide to go down this path), but that won't reach folks that don't follow the community closely. Maybe we should push an update to the default MOTD to let people know this is coming? Although, from personal experience, people tend to react pretty poorly to those kind of things as well sometimes. Any other ideas?
I do agree that upgrading to 8-stream is strictly better than leaving systems unsupported, and that for the vast majority of users it would be a net positive.
Cheers Davide
On Thu, Jul 8, 2021 at 5:07 PM Davide Cavalca via CentOS-devel < centos-devel@centos.org> wrote:
On Thu, 2021-07-08 at 16:22 -0400, Rich Bowen wrote:
Surprising users seldom goes well, even if it's an overall positive surprise.
What are our options for making it less of a surprise? We could post a blog, etc. about it of course (and we should, if we decide to go down this path), but that won't reach folks that don't follow the community closely. Maybe we should push an update to the default MOTD to let people know this is coming? Although, from personal experience, people tend to react pretty poorly to those kind of things as well sometimes. Any other ideas?
I do agree that upgrading to 8-stream is strictly better than leaving systems unsupported, and that for the vast majority of users it would be a net positive.
What about moving to a security update only mode?
Nothing else really changes but at least you don't leave 450k+ systems idling on the internet just waiting to be scooped up into a bot net.
Personally I feel that's what a good netizen would do.
Leif.
On Fri, Jul 9, 2021 at 1:03 AM Leif Madsen leif@redhat.com wrote:
On Thu, Jul 8, 2021 at 5:07 PM Davide Cavalca via CentOS-devel centos-devel@centos.org wrote:
On Thu, 2021-07-08 at 16:22 -0400, Rich Bowen wrote:
Surprising users seldom goes well, even if it's an overall positive surprise.
What are our options for making it less of a surprise? We could post a blog, etc. about it of course (and we should, if we decide to go down this path), but that won't reach folks that don't follow the community closely. Maybe we should push an update to the default MOTD to let people know this is coming? Although, from personal experience, people tend to react pretty poorly to those kind of things as well sometimes. Any other ideas?
I do agree that upgrading to 8-stream is strictly better than leaving systems unsupported, and that for the vast majority of users it would be a net positive.
What about moving to a security update only mode?
It's not predictable if applied without testing. I remember when a security update broke curl, and required a curl-libs update to go with it because the security update broke it but didn't list the updated curl-libs as an RPM requirement. It's precisely why people appreciated the point releases: they could point to a base OS, apply updates on top of that, and schedule an upgrade to the next system with an organized set of updates. People will spend time emulating this by making local snapshots, just as I've done for RHEL internally and at far less cost than setting up spacewalk or RHN. I taught several departments of the BBC how to do that: some of my old tools are still at https://github.com/nkadel/nkadel-rsync-scripts .
On 08/07/2021 21:22, Rich Bowen wrote:
On 7/8/21 4:08 PM, Neal Gompa wrote:
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I would prefer to do this change 1-2 months after December 31, though I'm quite fine with doing it on the EOL date. Having run CentOS Stream 8 for two years now, I feel that it's a solid experience and most people aren't going to have issues with it after upgrading from classic CentOS 8. And frankly, it's nice having people respond to my bug reports and actually*do* something about it. Strictly speaking, that's an upgrade from the experience with classic CentOS 8.
While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.
Surprising users seldom goes well, even if it's an overall positive surprise.
Sorry for the slow reply - I've been away for a few days.
Whilst I completely understand the desire to take this approach, I would respectfully request you don't as it's already creating extra support load for us with users switching to the Stream kernel and finding out their el8 kmod packages no longer work. If you switch all C8 users en mass at EOL, you run the risk of creating a potentially huge support burden that we are simply not in a position to manage.
--Phil
On Sun, Jul 11, 2021 at 7:25 PM Phil Perry pperry@elrepo.org wrote:
On 08/07/2021 21:22, Rich Bowen wrote:
On 7/8/21 4:08 PM, Neal Gompa wrote:
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I would prefer to do this change 1-2 months after December 31, though I'm quite fine with doing it on the EOL date. Having run CentOS Stream 8 for two years now, I feel that it's a solid experience and most people aren't going to have issues with it after upgrading from classic CentOS 8. And frankly, it's nice having people respond to my bug reports and actually*do* something about it. Strictly speaking, that's an upgrade from the experience with classic CentOS 8.
While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.
Surprising users seldom goes well, even if it's an overall positive surprise.
Sorry for the slow reply - I've been away for a few days.
Whilst I completely understand the desire to take this approach, I would respectfully request you don't as it's already creating extra support load for us with users switching to the Stream kernel and finding out their el8 kmod packages no longer work. If you switch all C8 users en mass at EOL, you run the risk of creating a potentially huge support burden that we are simply not in a position to manage.
--Phil
It's a particular issue for VirtualBox, which relies on kmod.
On Sun, Jul 11, 2021 at 7:25 PM Phil Perry pperry@elrepo.org wrote:
On 08/07/2021 21:22, Rich Bowen wrote:
On 7/8/21 4:08 PM, Neal Gompa wrote:
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I would prefer to do this change 1-2 months after December 31, though I'm quite fine with doing it on the EOL date. Having run CentOS Stream 8 for two years now, I feel that it's a solid experience and most people aren't going to have issues with it after upgrading from classic CentOS 8. And frankly, it's nice having people respond to my bug reports and actually*do* something about it. Strictly speaking, that's an upgrade from the experience with classic CentOS 8.
While I agree with you, that it's an upgrade, I anticipate that moving people from CentOS Linux 8 to CentOS Stream 8 automatically will result in a lot of backlash from people who continue to believe that Stream is a alpha/beta/testing/buggy/unstable/[pick your favorite complaint] distribution. Y'know, once they noticed that it had happened.
Surprising users seldom goes well, even if it's an overall positive surprise.
Sorry for the slow reply - I've been away for a few days.
Whilst I completely understand the desire to take this approach, I would respectfully request you don't as it's already creating extra support load for us with users switching to the Stream kernel and finding out their el8 kmod packages no longer work. If you switch all C8 users en mass at EOL, you run the risk of creating a potentially huge support burden that we are simply not in a position to manage.
I think this is a good point. Transition to CentOS Stream should be an opt-in choice made by the users. There are too many situations like the kernel, or internal policy compliance, or other reasons make automated migration problematic. We should encourage and advocate for people to switch to CentOS Stream, and focus on making tools and documentation for that easy to use and easily accessible, but we should not be doing that kind of migration unilaterally.
I would advocate for a form of Carl's option 3. Release the 8.5.0 content when it is ready, give people a grace period for one last update, and then either return 404s for mirror requests or simply do nothing else.
josh
On 08.07.21 22:08, Neal Gompa wrote:
On Thu, Jul 8, 2021 at 3:40 PM Carl George carl@redhat.com wrote:
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I would prefer to do this change 1-2 months after December 31, though I'm quite fine with doing it on the EOL date. Having run CentOS Stream 8 for two years now, I feel that it's a solid experience and most people aren't going to have issues with it after upgrading from classic CentOS 8. And frankly, it's nice having people respond to my bug reports and actually *do* something about it. Strictly speaking, that's an upgrade from the experience with classic CentOS 8.
- Don't change mirrorlist to respond to 8 repo requests with 8-stream
repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
I think this is a terrible idea and does a disservice to the large "silent majority" of CentOS systems that would otherwise be fine with the upgrade.
Due the nature of this situation (it never had to be managed), I would prefer not seeing such repo change. It should be a active opt-in of the operator (dnf swap ...). Personally I will switch to stream but as mentioned elsewhere, forcing this change will backfire.
-- Leon
On Thu, 8 Jul 2021 at 15:40, Carl George carl@redhat.com wrote:
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
I think both of these fall into the surprise category which ends up with lawsuits and very very angry people. Forced upgrades rarely work well and let us just say CentOS is being used in enough life/safety places that failed updates that have turned into upgrades will not be pretty. [Whether or not the hospital/chemical plant/bank should have used CentOS here is not going to matter and won't be mentioned until Page 6 on the NY Times.]
I don't see either 1 or 2 ending well no matter how well it is messaged or done.
- Don't change mirrorlist to respond to 8 repo requests with 8-stream
repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
4. Have mirrorlist return a 410 versus a 404 for requests to centos repositories.
There have been 150k CentOS systems added since the announcement last year. [There have been about 8000 Alma Linux and 3000 Rocky Linux in that time.]The growth rate is leveling off but not at a rate that the other rebuilds are 'catching' up anytime soon. While many of the current 450k CentOS 8 systems may function well on CentOS Stream, we don't know which ones are just web servers in some advertising farm and which ones are controlling the flow rates on a dam or petroleum flows at a refinery in Texas.
On 08/07/2021 21:39, Carl George wrote:
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
- Don't change mirrorlist to respond to 8 repo requests with 8-stream
repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
My personal take on this was initially to go with 1, as that would mean that for majority of users not following what happened, they'd still get updates on regular basis, not even noticing the difference, (as Stream 8 will continue to have 8.6.x content and so on, just in advance)
But then, by reading some comments, I realized that it's "sensitive" enough to view it from another angle.
So in fact, the political question to answer (and then one choice or the other would make sense) : do we consider that CentOS 8 is dead (and so applying the same principle/method we used for *all* EOL CentOS releases, so what Rich described) *or* we consider that Stream 8 is the replacement for CentOS 8, and so it would just make sense to apply method 1 (so redirecting 8 to 8-stream)
Let's consider two kind of 'users' :
* The ones who followed the December 2020 announcement, and so with one year to decide what to do : these ones I'd trust them to have already a plan for the conversion to Stream (like we did for centos infra) or have moved elsewhere already (and some are migrating to AlmaLinux, Rocky or Oracle EL, etc). For that category, we can apply both methods as they *shouldn't* be impacted
* The ones that still don't realize what was communicated, and so would continue to deploy it and use it : I admit that if you deploy CentOS, a good sysadmin would be up2date with what happens in the distro land *but* also by looking at the number of mirrorlist requests even for CentOS 5, I can tell you for sure that some don't ..... (ouch). So for that category, I'd suggest that we move to 8-stream as that would mean at least getting some updates , as the same users would come back to complain that 8 isn't available on mirrors etc if we just remove it (we always have that on each EOL distro)
On Fri, Jul 09, 2021 at 08:09:22AM +0200, Julien Pivotto wrote:
Could you please configure your MUA to send text/plain or similar to the CentOS lists?
John
On 09 Jul 02:45, John R. Dennison wrote:
On Fri, Jul 09, 2021 at 08:09:22AM +0200, Julien Pivotto wrote:
Could you please configure your MUA to send text/plain or similar to the CentOS lists?
Sorry, the one and only time I sent a mail from my phone :(
Providing stream as a continuity of 8 is the best thing to do, to show that we are confident and that the whole "stream fits most of the CentOS use case". It also has the side effect of better protecting the internet.
Note that it will put a lot of pressure and bring a lot of users on stream. It is a giant opportunity for stream, probably a unique opportunity.
I think that it will not backfire if we announce it soon.
John-- If we are to be a great democracy, we must all take an active role in our democracy. We must do democracy. That goes far beyond simply casting your vote. We must all actively champion the causes that ensure the common good.
-- Martin Luther King III (1957-), human rights advocate and community activist, first son of civil rights leader Martin Luther King, Jr., Speech at the Democratic Convention (28 August 2008)
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Fri, Jul 9, 2021 at 5:45 AM Julien Pivotto roidelapluie@inuits.eu wrote:
On 09 Jul 02:45, John R. Dennison wrote:
On Fri, Jul 09, 2021 at 08:09:22AM +0200, Julien Pivotto wrote:
Could you please configure your MUA to send text/plain or similar to the CentOS lists?
Sorry, the one and only time I sent a mail from my phone :(
Providing stream as a continuity of 8 is the best thing to do, to show that we are confident and that the whole "stream fits most of the CentOS use case". It also has the side effect of better protecting the internet.
Note that it will put a lot of pressure and bring a lot of users on stream. It is a giant opportunity for stream, probably a unique opportunity.
Stream is driving users and companies away from CentOS. Many are reviewing their previous usage and support. Many are questioning the wisdom of a migration to RHEL and reviewing other options, such as non RHEL based operating systems or a different supported operating system. Amazon Linux 2, for example, has picked up a lot of clients by porting more contemporary versions of system components like Python 3.7 to their RHEL 7 based operating system.
The backfire already happened. Having a solid end-date for the point releases puts a timetable on migration decisions, which is useful.
On Fri, 9 Jul 2021 at 07:03, Julien Pivotto roidelapluie@inuits.eu wrote:
So it is better for CentOS to keep the users rather than letting them go elsewhere.
Not always. It is a disservice to try and keep users who 'value' things a project no longer offers to them. Many people value CentOS as a drop-in-replacement for RHEL. They don't have to think about it, and they don't have to have any sort of 'relationship' with either CentOS or anyone else. Many of the people I have known in CentOS over the years have been proud of the fact that it is an 'anti-social' community. [Several put it is that is the reason why the CentOS logo has all its arrows going out. (It isn't actually but it was seen by them as such)]
CentOS Stream is about building a co-operative relationship between the consumers and the distribution where what the distribution builds is evaluated and feedback is given. If a consumer is expecting never to have problems, to never have to file/track a bugzilla or ever even check to see what is being delivered.. then CentOS Stream is not the distribution for them. Because even if it doesn't look like it, there is a very very large gap between 'never' and 'once in a while' that might happen with Stream. There is a need for co-operation between the consumer of stream and the makers to get things right. If you do not have time, energy, or want to do that, then Stream is going to be a constant thorn. The consumer in that case would be better off with another rebuild.
Le 9 juil. 2021 12:27, Nico Kadel-Garcia nkadel@gmail.com a écrit :
On Fri, Jul 9, 2021 at 5:45 AM Julien Pivotto roidelapluie@inuits.eu wrote:
On 09 Jul 02:45, John R. Dennison wrote:
On Fri, Jul 09, 2021 at 08:09:22AM +0200, Julien Pivotto wrote:
Could you please configure your MUA to send text/plain or similar to the CentOS lists?
Sorry, the one and only time I sent a mail from my phone :(
Providing stream as a continuity of 8 is the best thing to do, to show that we are confident and that the whole "stream fits most of the CentOS use case". It also has the side effect of better protecting the internet.
Note that it will put a lot of pressure and bring a lot of users on stream. It is a giant opportunity for stream, probably a unique opportunity.
Stream is driving users and companies away from CentOS. Many are reviewing their previous usage and support. Many are questioning the wisdom of a migration to RHEL and reviewing other options, such as non RHEL based operating systems or a different supported operating system. Amazon Linux 2, for example, has picked up a lot of clients by porting more contemporary versions of system components like Python 3.7 to their RHEL 7 based operating system.
The backfire already happened. Having a solid end-date for the point releases puts a timetable on migration decisions, which is useful. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
I disagree with the suggestion that participation is required to use CentOS Stream. It's certainly recommended, but I have had multiple people tell me personally that they switched from CL8 to CS8 and forgot they did it and didn't notice a difference. They didn't file bugs, they didn't participate in IRC or the mailing list, they just used it and it was fine. CS8 hasn't been a constant thorn for them. I'm not claiming it's been perfect, there have certainly been regressions, but they are fixed faster than they ever were in CL8 (or previous major versions) and I strongly feel that most users will be best served by getting switched to CS8 at or just after the CL8 EOL.
On Fri, Jul 9, 2021 at 6:23 AM Stephen John Smoogen smooge@gmail.com wrote:
On Fri, 9 Jul 2021 at 07:03, Julien Pivotto roidelapluie@inuits.eu wrote:
So it is better for CentOS to keep the users rather than letting them go elsewhere.
Not always. It is a disservice to try and keep users who 'value' things a project no longer offers to them. Many people value CentOS as a drop-in-replacement for RHEL. They don't have to think about it, and they don't have to have any sort of 'relationship' with either CentOS or anyone else. Many of the people I have known in CentOS over the years have been proud of the fact that it is an 'anti-social' community. [Several put it is that is the reason why the CentOS logo has all its arrows going out. (It isn't actually but it was seen by them as such)]
CentOS Stream is about building a co-operative relationship between the consumers and the distribution where what the distribution builds is evaluated and feedback is given. If a consumer is expecting never to have problems, to never have to file/track a bugzilla or ever even check to see what is being delivered.. then CentOS Stream is not the distribution for them. Because even if it doesn't look like it, there is a very very large gap between 'never' and 'once in a while' that might happen with Stream. There is a need for co-operation between the consumer of stream and the makers to get things right. If you do not have time, energy, or want to do that, then Stream is going to be a constant thorn. The consumer in that case would be better off with another rebuild.
Le 9 juil. 2021 12:27, Nico Kadel-Garcia nkadel@gmail.com a écrit :
On Fri, Jul 9, 2021 at 5:45 AM Julien Pivotto roidelapluie@inuits.eu wrote:
On 09 Jul 02:45, John R. Dennison wrote:
On Fri, Jul 09, 2021 at 08:09:22AM +0200, Julien Pivotto wrote:
Could you please configure your MUA to send text/plain or similar to the CentOS lists?
Sorry, the one and only time I sent a mail from my phone :(
Providing stream as a continuity of 8 is the best thing to do, to show that we are confident and that the whole "stream fits most of the CentOS use case". It also has the side effect of better protecting the internet.
Note that it will put a lot of pressure and bring a lot of users on stream. It is a giant opportunity for stream, probably a unique opportunity.
Stream is driving users and companies away from CentOS. Many are reviewing their previous usage and support. Many are questioning the wisdom of a migration to RHEL and reviewing other options, such as non RHEL based operating systems or a different supported operating system. Amazon Linux 2, for example, has picked up a lot of clients by porting more contemporary versions of system components like Python 3.7 to their RHEL 7 based operating system.
The backfire already happened. Having a solid end-date for the point releases puts a timetable on migration decisions, which is useful. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Stephen J Smoogen. I've seen things you people wouldn't believe. Flame wars in sci.astro.orion. I have seen SPAM filters overload because of Godwin's Law. All those moments will be lost in time... like posts on BBS... time to reboot. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 09.07.21 17:41, Carl George wrote:
I disagree with the suggestion that participation is required to use CentOS Stream. It's certainly recommended, but I have had multiple people tell me personally that they switched from CL8 to CS8 and forgot they did it and didn't notice a difference. They didn't file bugs, they didn't participate in IRC or the mailing list, they just used it and it was fine. CS8 hasn't been a constant thorn for them. I'm not claiming it's been perfect, there have certainly been regressions, but they are fixed faster than they ever were in CL8 (or previous major versions) and I strongly feel that most users will be best served by getting switched to CS8 at or just after the CL8 EOL.
Hands on - I tried to switch some workstations to CS8 like it would be done at the end of the year. The next monday would bring me angry users into my "virtual" office because there applications does not run anymore. What happens? Applications from 3rd party repos like RPM Fusion that link against "qt5-qtbase(x86-64) = 5.12.5" do not have any providers anymore (CS8 got an upgrade to 5.15.2. So, like EPEL it seems that everything else needs also a -next branch (just kidding). What would be new in 8.6? That will be state of CS8 at the end of the year?
I expect that such upgrades will happen in RHEL productions phase. When this phase ends then CS8 is also EOL, and that point would be the only dist upgrade situation where no surprises would happen ...
The bottom line here is that a slightly more complex setup/configuration of a system would lead into a higher risk that the mentioned upgrade path would fail. Complex means here every things that uses more then just CentOS artifacts (ISV/proprietary software, 3rd/custom repos, etc.). As mentioned elsewhere because RHEL has stripped a lot of packages out of the compose, people are forced to compensate this with there own builds, what leads to the above mentioned system configuration.
-- Leon
Dear all,
On Sat, Jul 10, 2021 at 02:18:48AM +0200, Leon Fauster via CentOS-devel wrote:
On 09.07.21 17:41, Carl George wrote:
I disagree with the suggestion that participation is required to use CentOS Stream. It's certainly recommended, but I have had multiple people tell me personally that they switched from CL8 to CS8 and forgot they did it and didn't notice a difference. They didn't file bugs, they didn't participate in IRC or the mailing list, they just used it and it was fine. CS8 hasn't been a constant thorn for them. I'm not claiming it's been perfect, there have certainly been regressions, but they are fixed faster than they ever were in CL8 (or previous major versions) and I strongly feel that most users will be best served by getting switched to CS8 at or just after the CL8 EOL.
Hands on - I tried to switch some workstations to CS8 like it would be done at the end of the year. The next monday would bring me angry users into my "virtual" office because there applications does not run anymore. What happens? Applications from 3rd party repos like RPM Fusion that link against "qt5-qtbase(x86-64) = 5.12.5" do not have any providers anymore (CS8 got an upgrade to 5.15.2. So, like EPEL it seems that everything else needs also a -next branch (just kidding). What would be new in 8.6? That will be state of CS8 at the end of the year?
This seems to be worth discussing with RPM Fusion developers.
With CentOS 8 going EOL at the end of the year, it's likely that a substantial portion of the userbase will end up on CentOS Stream.
What would it take to have a `-next` overlay repo for packages that need to be rebuilt for Stream for rpmfusion free and nonfree?
Best regards,
On 7/9/21 7:18 PM, Leon Fauster via CentOS-devel wrote:
On 09.07.21 17:41, Carl George wrote:
I disagree with the suggestion that participation is required to use CentOS Stream. It's certainly recommended, but I have had multiple people tell me personally that they switched from CL8 to CS8 and forgot they did it and didn't notice a difference. They didn't file bugs, they didn't participate in IRC or the mailing list, they just used it and it was fine. CS8 hasn't been a constant thorn for them. I'm not claiming it's been perfect, there have certainly been regressions, but they are fixed faster than they ever were in CL8 (or previous major versions) and I strongly feel that most users will be best served by getting switched to CS8 at or just after the CL8 EOL.
Hands on - I tried to switch some workstations to CS8 like it would be done at the end of the year. The next monday would bring me angry users into my "virtual" office because there applications does not run anymore. What happens? Applications from 3rd party repos like RPM Fusion that link against "qt5-qtbase(x86-64) = 5.12.5" do not have any providers anymore (CS8 got an upgrade to 5.15.2. So, like EPEL it seems that everything else needs also a -next branch (just kidding). What would be new in 8.6? That will be state of CS8 at the end of the year?
That is actually true .. people will need to support stream, IF they want to support stream.
I expect that such upgrades will happen in RHEL productions phase. When this phase ends then CS8 is also EOL, and that point would be the only dist upgrade situation where no surprises would happen ...
Yes .. this same error will happen on the switch to RHEL 8.5, as that is the version that will be in 8.5. So it happened a little earlier, but it would have also happened in both RHEL and CentOS 8.5.
Other than the kernels, I'm sorry but you would be hard pressed to find something in Stream that doesn't also happen in CentOS and even RHEL. It just happens a month or two earlier in Stream. BUT, so do bugfixes :)
The bottom line here is that a slightly more complex setup/configuration of a system would lead into a higher risk that the mentioned upgrade path would fail. Complex means here every things that uses more then just CentOS artifacts (ISV/proprietary software, 3rd/custom repos, etc.). As mentioned elsewhere because RHEL has stripped a lot of packages out of the compose, people are forced to compensate this with there own builds, what leads to the above mentioned system configuration.
On Thu, Jul 15, 2021 at 4:36 PM Johnny Hughes johnny@centos.org wrote:
Other than the kernels, I'm sorry but you would be hard pressed to find something in Stream that doesn't also happen in CentOS and even RHEL. It just happens a month or two earlier in Stream. BUT, so do bugfixes :)
I hope you understand that we expect both gross and subtle failures of the beta testing in CentOS 8 Stream. I expect it with python modules, when the author of one module fails to restrict versions of other modules, especially modules brought in by dependencies, to versions that are compatible because they cannot *predict* that the next version will be compatible. It can be very difficult to anticipate those: ansible is a great example right now, since the latest ansible release requires python 3.8 and the panoply of necessary modules are not yet available anywhere.
It's one of the dangers of the "streaming" model, when unanticipated dependencies are discovered in the field. It's why I expect people to use rsync or reposync tools to generate internal mirrors with locked snapshots, which they used to do with CentOS point releases.
On Thu, Jul 15, 2021 at 7:25 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Jul 15, 2021 at 4:36 PM Johnny Hughes johnny@centos.org wrote:
Other than the kernels, I'm sorry but you would be hard pressed to find something in Stream that doesn't also happen in CentOS and even RHEL. It just happens a month or two earlier in Stream. BUT, so do bugfixes :)
I hope you understand that we expect both gross and subtle failures of the beta testing in CentOS 8 Stream. I expect it with python modules, when the author of one module fails to restrict versions of other modules, especially modules brought in by dependencies, to versions that are compatible because they cannot *predict* that the next version will be compatible. It can be very difficult to anticipate those: ansible is a great example right now, since the latest ansible release requires python 3.8 and the panoply of necessary modules are not yet available anywhere.
I hope you understand that we've actually taken care to deal with issues like that. The CentOS/RHEL platform inherits all the work the Fedora Python team and myself have done over the years to simplify, automate, and rationalize Python packaging. The Python dependency generator *already* deals with most of those problems for us. The modularity tooling makes it relatively trivial for us to rebootstrap the world of Python modules for a new Python version if we so choose. Now, Red Hat has not currently elected to do this, but that doesn't mean it isn't possible.
It's one of the dangers of the "streaming" model, when unanticipated dependencies are discovered in the field. It's why I expect people to use rsync or reposync tools to generate internal mirrors with locked snapshots, which they used to do with CentOS point releases.
You mean like how people *already* did it because they thought regular CentOS updates were "too dangerous"? Frankly, I don't buy what you're selling here. To make matters worse, the previous model gave you *zero* opportunity to resolve issues with updates if they were buggy. They just stayed broken for months or years. At least now there's a chance of them getting fixed in a reasonable time window.
Nico, you spend far too much time looking at CentOS Stream the wrong way. You should be looking at this as an opportunity to bring value to the folks you work with leveraging the CentOS platform and to the CentOS ecosystem as a whole. You can validate things early and often, identify specific issues, contribute fixes, and be part of the process to raise the quality bar for Enterprise Linux as a whole. And that's the bare minimum! You could also go so much further if you wanted by developing features that are useful to you and your organization in Fedora and bringing them to RHEL/CentOS through the RHEL Feature Proposal process handled by the CentOS Stream Feature SIG[1]. This allows folks in the CentOS community to lobby for improvements in the platform in an avenue that was never available to them before as non-customers.
The whole point of CentOS Stream is continuous improvement. In a very real way, CentOS Stream allows CentOS to truly earn its name as the "Community Enterprise Operating System". We, the community, now have the power to contribute, develop, and deliver *the* community enterprise Linux platform. We've never had that before. We are the folks with the opportunity to literally *set the standard* for Enterprise Linux. Don't waste it!
And I'm putting my money where my mouth is. I've been actively contributing to the platform since it launched. Even before this whole self-inflicted disaster happened, I had already switched my CentOS machines to CentOS Stream. I've been working in the Hyperscale SIG[2] to develop features that I hope will be part of RHEL one day. And I'm already looking forward to CentOS Stream 9.
[1]: https://wiki.centos.org/SpecialInterestGroup/StreamFeatureRequest [2]: https://wiki.centos.org/SpecialInterestGroup/Hyperscale
-- 真実はいつも一つ！/ Always, there's only one truth!
Hello,
(sorry diverging from the OT)
On 7/16/21 9:43 AM, Neal Gompa wrote:
It's one of the dangers of the "streaming" model, when unanticipated dependencies are discovered in the field. It's why I expect people to use rsync or reposync tools to generate internal mirrors with locked snapshots, which they used to do with CentOS point releases.
You mean like how people *already* did it because they thought regular CentOS updates were "too dangerous"? Frankly, I don't buy what you're selling here. To make matters worse, the previous model gave you *zero* opportunity to resolve issues with updates if they were buggy. They just stayed broken for months or years. At least now there's a chance of them getting fixed in a reasonable time window.
While I agree with this in theory, in practice it doesn't work out quite that nicely. We are currently affected by two[1][2] different issues in CS8 and the only way we can mitigate them somewhat is by snapshotting and tweaking the packages we distribute to our internal users. Sure, we can report the issues to Red Hat/CentOS, but then we still have to wait until they do their testing and decide they're ready to publish the fixes. This can take a really long time, and in the meantime there may be security fixes[3] that you *have* to publish, so you have to be able to keep some updates back and promote others, independently of what Red Hat/CentOS decides.
In fairness, the same thing happens in the "non-streaming" model, but just saying "but now you can contribute!" doesn't really help much in practice.
Anyway, sorry for the rant.
Cheers, Alex
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1966712: CentOS decided to upgrade to a release candidate version of mdadm which is unable to verify it's own checksums, so we can't install machines with software RAID. Fix was sent upstream, and we're stuck waiting until they acknowledge it. *We the community* can't push the fix to Stream 8, so how is our contribution useful? [2] https://bugzilla.redhat.com/show_bug.cgi?id=1972278 Apparently waiting to pass Red Hat's gating, a black-box process. *We the community* can do nothing but wait, it will be ready when it's ready. [3] CVE-2021-3560
On Fri, Jul 16, 2021 at 3:44 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Jul 15, 2021 at 7:25 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Jul 15, 2021 at 4:36 PM Johnny Hughes johnny@centos.org wrote:
Other than the kernels, I'm sorry but you would be hard pressed to find something in Stream that doesn't also happen in CentOS and even RHEL. It just happens a month or two earlier in Stream. BUT, so do bugfixes :)
I hope you understand that we expect both gross and subtle failures of the beta testing in CentOS 8 Stream. I expect it with python modules, when the author of one module fails to restrict versions of other modules, especially modules brought in by dependencies, to versions that are compatible because they cannot *predict* that the next version will be compatible. It can be very difficult to anticipate those: ansible is a great example right now, since the latest ansible release requires python 3.8 and the panoply of necessary modules are not yet available anywhere.
I hope you understand that we've actually taken care to deal with issues like that. The CentOS/RHEL platform inherits all the work the Fedora Python team and myself have done over the years to simplify, automate, and rationalize Python packaging. The Python dependency
I see the efforts. I've run into and tried to resolve them myself myself, especially with python sphinx and the dependency chains. The python generator has difficulty with dependencies, and dependencies, are updated and the original module authors did not know and did not publish requirements of modules *less* than an incompatible version. It's an underlying python architecture issue, one that I and others rely on RPM's to keep consistent. It's tricky.
generator *already* deals with most of those problems for us. The modularity tooling makes it relatively trivial for us to rebootstrap the world of Python modules for a new Python version if we so choose. Now, Red Hat has not currently elected to do this, but that doesn't mean it isn't possible.
Many things are possible. Enacting modularity for python module resolution would destabilize them further, modularity is *not* woking well.
It's one of the dangers of the "streaming" model, when unanticipated dependencies are discovered in the field. It's why I expect people to use rsync or reposync tools to generate internal mirrors with locked snapshots, which they used to do with CentOS point releases.
You mean like how people *already* did it because they thought regular CentOS updates were "too dangerous"? Frankly, I don't buy what you're
It's familiar turf. Many used the point releases for creating new hosts and disabled the update channels at system build time, precisely to avoid randomized out-of-band updates. They did not run "yum update" at build time or after deployment, they did only "yum update --security", and switched their configs to pint to the new point releases only after testing and review.
selling here. To make matters worse, the previous model gave you *zero* opportunity to resolve issues with updates if they were buggy.
It's a trade-off. Bugs created by unplanned vendor updates are very bad indeed and preventable, and most preventable by locking down the update process. Red Hat tried to sell spacewalk setups or RHN to provide that kind of update management, but it turned out to be far too micro-managed and expensive for most environments. The update channels were available: they simply are not, and should not be, turned on by default for high stability, production grade, systems. The idea that updates will not introduce bugs is not a historically well-founded one, especially for updates that have not had yet gone through beta testing, and that instead are part of the beta testing cycle as CentOS 8 Stream is now designed for.
They just stayed broken for months or years. At least now there's a chance of them getting fixed in a reasonable time window.
Nico, you spend far too much time looking at CentOS Stream the wrong way. You should be looking at this as an opportunity to bring value to the folks you work with leveraging the CentOS platform and to the CentOS ecosystem as a whole. You can validate things early and often,
I do so, because I'm aggressive about catching issues before the people who sign my paychecks. I'm not looking forward to expanding my mock setups to include CentOs 8 Stream as well as CentOS 8. The fiasco is very familiar from Red Hat 9, which tried the same "we do not have point releases" stunt, and was abandoned with RHEL. I anticipate that CentOS 9 or CentOS 10 will abandon most of this replayed debate, or people will abandon CentOS in favor of one of the alternatives. I remember switching from White Box Linux to CenTOS, and a dalliance with Scientific Linux when CentOS took a really, really long time to release CentOS 6.
identify specific issues, contribute fixes, and be part of the process to raise the quality bar for Enterprise Linux as a whole. And that's the bare minimum! You could also go so much further if you wanted by developing features that are useful to you and your organization in Fedora and bringing them to RHEL/CentOS through the RHEL Feature Proposal process handled by the CentOS Stream Feature SIG[1]. This
I publish. I've had repeated problems with the Fedora registration process.
allows folks in the CentOS community to lobby for improvements in the platform in an avenue that was never available to them before as non-customers.
The whole point of CentOS Stream is continuous improvement. In a very real way, CentOS Stream allows CentOS to truly earn its name as the "Community Enterprise Operating System". We, the community, now have the power to contribute, develop, and deliver *the* community enterprise Linux platform. We've never had that before. We are the folks with the opportunity to literally *set the standard* for Enterprise Linux. Don't waste it!
IWe had nearly all of that flexibility with EPEL (for components not in CentOS), with RPMforge (which admittedly seems to have dried up as Dag Weiers found other work to do, I took him drinking about the time he stepped back from that), with RPMfusion, and with IUS (which I find handy for nignx, haproxy, and git updates). It's not been as well integrated as I'd hope that CentOS 8 Steam might be, but they've certainly existed as development and testing environments.
And I'm putting my money where my mouth is. I've been actively contributing to the platform since it launched. Even before this whole self-inflicted disaster happened, I had already switched my CentOS machines to CentOS Stream. I've been working in the Hyperscale SIG[2] to develop features that I hope will be part of RHEL one day. And I'm already looking forward to CentOS Stream 9.
Unfortunately, we weren't asked. It's not what was on the menu, and we're not being allowed to send it back to finish baking. We're having to set up our own ovens at our own tables. I *publish* some of those oven tools, thank you for reminding me that I need to update them to include CentOS 8 Stream.
On Fri, Jul 16, 2021 at 8:13 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Fri, Jul 16, 2021 at 3:44 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Jul 15, 2021 at 7:25 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Jul 15, 2021 at 4:36 PM Johnny Hughes johnny@centos.org wrote:
Reviewing this, I realized I had a lot of grammar issues. Forgive me, please, a new cat is trying to colonize my keyboard, and I should have reviewed the message.
On Fri, Jul 16, 2021 at 9:12 AM Rich Bowen rbowen@redhat.com wrote:
On 7/16/21 9:03 AM, Nico Kadel-Garcia wrote:
Reviewing this, I realized I had a lot of grammar issues. Forgive me, please, a new cat is trying to colonize my keyboard, and I should have reviewed the message.
By the laws of The Internet, you now owe us pictures.
On a text-only group? Frm work by Felix Lee:
_,'| _.-''``-...___..--';) /_ '. __..-' , ,--...--''' <\ .`--''' ` /' `-';' ; ; ; __...--'' ___...--_..' .;.' (,__....----''' (,..--'' Felix Lee flee@cse.psu.edu
On Fri, 2021-07-16 at 09:51 -0400, Nico Kadel-Garcia wrote:
On Fri, Jul 16, 2021 at 9:12 AM Rich Bowen rbowen@redhat.com wrote:
On 7/16/21 9:03 AM, Nico Kadel-Garcia wrote:
Reviewing this, I realized I had a lot of grammar issues. Forgive me, please, a new cat is trying to colonize my keyboard, and I should have reviewed the message.
By the laws of The Internet, you now owe us pictures.
On a text-only group? Frm work by Felix Lee:
_,'| _.-''``-...___..--';) /_ '. __..-' , ,--...--''' <\ .`--''' ` /' `-';' ; ; ; __...--'' ___...--_..' .;.' (,__....----''' (,..--'' Felix Lee flee@cse.psu.edu
I celebrate all things cat!
Pat
On Fri, Jul 16, 2021, 09:18 Patrick Riehecky riehecky@fnal.gov wrote:
Reviewing this, I realized I had a lot of grammar issues. Forgive me, please, a new cat is trying to colonize my keyboard, and I should have reviewed the message.
By the laws of The Internet, you now owe us pictures.
On a text-only group? Frm work by Felix Lee:
_,'| _.-''``-...___..--';) /_ \'. __..-' , ,--...--''' <\ .`--''' ` /' `-';' ; ; ;__...--'' ___...--_..' .;.' (,__....----''' (,..--'' Felix Lee flee@cse.psu.edu
I celebrate all things cat!
Technically, a url that leads to a picture of the cat conforms to the text only rule...
I'm really torn on these ideas.
I guess I'm trying to sort out who we'd be helping most and who we'd be harming least with each of these ideas.
Is there a CentOS user community of folks who apply nightly updates but don't respond to errors during that process? Since we don't enable automatic updates by default (sad face), I'm trying to figure out which workflows would be impacted by each approach and what options are the most transparent and predictable for each one.
Pat
On Thu, 2021-07-08 at 14:39 -0500, Carl George wrote:
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
- Don't change mirrorlist to respond to 8 repo requests with 8-stream
repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
[0] https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_mattdm_stat...
On Thu, Jul 8, 2021 at 12:10 PM Rich Bowen rbowen@redhat.com wrote:
As we get closer to the CentOS Linux 8 EOL, we're getting more and more questions about how the EOL is actually going to work. While there tends to be a great deal of "everybody just knows how it works" among the people actually on this list, we're getting attention from a much larger audience now, and they don't know.
To that end, I want to make sure we have clear documentation, prominently displayed, that sets expectations as we get closer to that December 31 date,
So ... as I understand, our process is basically:
- Copy old data to vault.centos.org
- Update web pages and links to say it is now on vault.centos.org
- Turn off mirrorlist to that release name (aka all requests to
CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
Is there more to it than this? Will we do anything different this time because CL8 is a special case? Am I missing any steps, or have I misunderstood any of the steps? Is this already documented somewhere that I simply haven't found yet?
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
CentOS-devel mailing list CentOS-devel@centos.org https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.centos.org_mailma...
On 7/9/21 7:32 PM, Patrick Riehecky wrote:
I'm really torn on these ideas.
I guess I'm trying to sort out who we'd be helping most and who we'd be harming least with each of these ideas.
Is there a CentOS user community of folks who apply nightly updates but don't respond to errors during that process?
yes, there is. I know of installations ( not mine :) ) which have a "yum -y update " run from cron.daily
Since we don't enable automatic updates by default (sad face), I'm trying to figure out which workflows would be impacted by each approach and what options are the most transparent and predictable for each one.
Pat
On Thu, 2021-07-08 at 14:39 -0500, Carl George wrote:
One idea I've been noodling on is whether or not at some point after the EOL we should have mirrorlist.centos.org respond to requests for 8 repos with 8-stream repos, which effectively converts any remaining CentOS Linux 8 systems to CentOS Stream 8. I see this as a natural extension of how CentOS has never supported (to the extent CentOS "supports" anything) staying on old previous minor releases of a major release. CentOS Stream is still CentOS. 8-stream isn't that different from 8. CentOS Stream 8 is the latest CentOS 8. (Note: These are facts that I'm not interested in arguing with people about. If you feel the need to reply to those particular points, don't expect a reply from me.)
I can envision three possible scenarios.
- Change mirrorlist to respond to 8 repo requests with 8-stream repos
at the EOL date. This ensures that CentOS Linux 8 systems continue to get security updates by way of converting them to CentOS Stream 8. It also reinforces that CentOS Stream 8 is the latest CentOS 8. Obviously people who are not fans of the new project direction will view this negatively and lash out about Red Hat for supposedly forcing them to use CentOS Stream 8.
- Same as 1, but wait some period of time after the EOL date, such as
1-3 months. Waiting much longer than that would result in effectively updating from 8.5 directly to 8.7, which is less appealing.
- Don't change mirrorlist to respond to 8 repo requests with 8-stream
repos, ever. This approach gives users the maximum amount of choice to migrate to either CentOS Stream 8, RHEL 8, or another RHEL 8 rebuild, when and only when they are ready. The downside is that many systems will be left without security updates for long periods of time, until users decide to take action. CentOS Linux 8 hits to EPEL are still growing [0], which makes me believe that many people still are unaware of the EOL change.
[0] https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_mattdm_stat...
On Thu, Jul 8, 2021 at 12:10 PM Rich Bowen rbowen@redhat.com wrote:
As we get closer to the CentOS Linux 8 EOL, we're getting more and more questions about how the EOL is actually going to work. While there tends to be a great deal of "everybody just knows how it works" among the people actually on this list, we're getting attention from a much larger audience now, and they don't know.
To that end, I want to make sure we have clear documentation, prominently displayed, that sets expectations as we get closer to that December 31 date,
So ... as I understand, our process is basically:
- Copy old data to vault.centos.org
- Update web pages and links to say it is now on vault.centos.org
- Turn off mirrorlist to that release name (aka all requests to
CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
Is there more to it than this? Will we do anything different this time because CL8 is a special case? Am I missing any steps, or have I misunderstood any of the steps? Is this already documented somewhere that I simply haven't found yet?
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
On Fri, 2021-07-09 at 23:14 +0300, Manuel Wolfshant wrote:
On 7/9/21 7:32 PM, Patrick Riehecky wrote:
I'm really torn on these ideas.
I guess I'm trying to sort out who we'd be helping most and who we'd be harming least with each of these ideas.
Is there a CentOS user community of folks who apply nightly updates but don't respond to errors during that process?
yes, there is. I know of installations ( not mine :) ) which have a "yum -y update " run from cron.daily
I guess it is a workflow issue, but most of my hosts apply updates nightly and when they fail to apply I (or another team member) look into why.
To my mind this is compounded by AppStream/Modularity where some of your streams may make some of your updates fail in new/interesting ways.
I guess I'm wondering how far off the Best Practices we want to support in Jan 2022...
"Set it and forget it" nodes that aren't "managed" by anyone are always a problem. But are those folks likely to blame us for issues vs folks with managed systems who are "reading" the error reports from dnf- automatic (etc) and aren't aware of Stream8?
I guess I'm not sure upgrading folks to Stream8 is in the 'predictable' end of life workflow.
Pat
On 7/9/21 11:44 PM, Patrick Riehecky wrote:
On Fri, 2021-07-09 at 23:14 +0300, Manuel Wolfshant wrote:
On 7/9/21 7:32 PM, Patrick Riehecky wrote:
I'm really torn on these ideas.
I guess I'm trying to sort out who we'd be helping most and who we'd be harming least with each of these ideas.
Is there a CentOS user community of folks who apply nightly updates but don't respond to errors during that process?
yes, there is. I know of installations ( not mine :) ) which have a "yum -y update " run from cron.daily
I guess it is a workflow issue,
Yes, it is.
but most of my hosts apply updates nightly and when they fail to apply I (or another team member) look into why.
Because you are part of an organization with proper infrastructure, procedures and qualified people.
Now think about a small shop with 1-2-3 systems, no IT budget and no qualified personnel (*). They got help initially and installed/configured the machines. Knowing that they have no admin or external contractor, the person that did the installation tried to help and left the machines in a state which more or less keeps them updated automatically ( it's good to stay updated, right ? ). Maybe the failures are even mailed to someone.. but that someone does not read the mails because a) (s)he has actual but completely different work to do as $DAYTASKS, b) there were never important automatic mails so reading those is a pure loss of time c) does not even understand what (s)he reads in those mails.
Not to mention the more interesting aspects generated by botched updates ( see RHSA-2020:3216 ) which might lead to d) the systems hung so no way to read the mails anyway.
To my mind this is compounded by AppStream/Modularity where some of your streams may make some of your updates fail in new/interesting ways.
I guess I'm wondering how far off the Best Practices we want to support in Jan 2022...
_THAT_ is a very good question. I will not hold my breath waiting for a convenient answer.
"Set it and forget it" nodes that aren't "managed" by anyone are always a problem.
Yes, they are. You are 1000 % right here. But yet they exist and leaving them behind because "we do not care" is neither polite nor indicated ( from my point of view ).
But are those folks likely to blame us for issues
Again, yes, they are. " RH did it again, it worked yesterday but last night it suddenly stopped working. And when I looked, guess what ? It was not the OS I installed !!!!!11111"
vs folks with managed systems who are "reading" the error reports from dnf- automatic (etc) and aren't aware of Stream8?
The Universe is large and it has many aspects :)
I guess I'm not sure upgrading folks to Stream8 is in the 'predictable' end of life workflow.
from my point of view , it is not. I cannot speak for CS8 but in C6 and C7 I had ( rare ) issues when proprietary apps stopped working after upgrading a lib or another. Or the kernel ( see Cisco's vpnclient whose kernel module could no longer be compiled after a certain update of the kernel ). I live in the world of EDA tools for VLSI development and support for said tools is already problematic when running on more "classic" OSes . A moving target like CS8 is not even envisioned. I asked a developer from Synopsys a few years ago "Why do you always support your application only on OSes which are at least 2 minor releases behind the current one ?". His answer ? "We wait 6 months for the release to stabilize and we need another 6 months for internal testing"
(*) I know, they should use an embedded router and externalize all IT, maybe host everything in cloud. Reread the "no IT budget" part.
On Fri, Jul 9, 2021 at 4:14 PM Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
On 7/9/21 7:32 PM, Patrick Riehecky wrote:
I'm really torn on these ideas.
I guess I'm trying to sort out who we'd be helping most and who we'd be harming least with each of these ideas.
Is there a CentOS user community of folks who apply nightly updates but don't respond to errors during that process?
yes, there is. I know of installations ( not mine :) ) which have a "yum -y update " run from cron.daily
Against CentOS 8 Stream? I'd be surprised. I'd consider it on a test server or cluster with some regression tests to keep an eye on what is coming down the pike, but I normally wouldn't do it to any host or container I used for anything like a production environment.
I've generally been *paid* for high reliability environments. Regression issues are too frequent, especially if somebody has been doing "pip install" or "gem install" as a root user, or does things like multiple JDK installations. There's nothing like an update to the openjdk 8 to overwrite the default settings of the openjdk 11 on the same machine to make people throw tools at you, even if you warned them.
On Thu, Jul 8, 2021 at 7:10 PM Rich Bowen rbowen@redhat.com wrote:
As we get closer to the CentOS Linux 8 EOL, we're getting more and more questions about how the EOL is actually going to work. While there tends to be a great deal of "everybody just knows how it works" among the people actually on this list, we're getting attention from a much larger audience now, and they don't know.
To that end, I want to make sure we have clear documentation, prominently displayed, that sets expectations as we get closer to that December 31 date,
So ... as I understand, our process is basically:
- Copy old data to vault.centos.org
- Update web pages and links to say it is now on vault.centos.org
- Turn off mirrorlist to that release name (aka all requests to
CentOS-8 will 404) 4. Remove the data from the main mirrors with a file saying look at vault.centos.org 5. Mirrors pick all that up in 4-8 hours.
Just for confirmation, this applies to the SIGs content under centos/8/?
Note that some SIGs are not providing repos under 8-stream and just keep using the repos under 8/ fos stream as those packages are compatible with CS8. Will those repos be archived too?, that would require them to, at least, cross-tagging the desired builds in 8s tags and update the release rpms in cs8 estras repos.
Is there more to it than this? Will we do anything different this time because CL8 is a special case? Am I missing any steps, or have I misunderstood any of the steps? Is this already documented somewhere that I simply haven't found yet?
Are there any changes to this process that we *want* to advocate for, given the special nature of this particular EOL? (No, I absolutely cannot promise anything, but I can ask.)
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-
Alex Iribarren -
Alfredo Moralejo Alonso -
Carl George -
Davide Cavalca -
Fabian Arrotin -
John R. Dennison -
Johnny Hughes -
Josh Boyer -
Julien Pivotto -
Leif Madsen -
Leon Fauster -
Manuel Wolfshant -
Michel Alexandre Salim -
Neal Gompa -
Nico Kadel-Garcia -
Patrick Riehecky -
Phil Perry -
Rich Bowen -
Stephen John Smoogen -
Wynona Stacy Lockwood