Hello All,
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
What else I can do (as user or developer) for fixing this bug?
P.S.
https://centos.org/distro-faq/
Q5: Does this mean that CentOS Stream is the RHEL BETA test platform now?
A: No. CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release
==================================
But currently I hit the bug, which is absent in CentOS 8.3 but present in CentOS Stream 8. Looks like CentOS Stream 8 is really the beta version for future RHEL minor release, despite the CentOS FAQ. Sorry, but this is true.
On Thu, Jan 14, 2021, at 14:50, Gena Makhomed wrote:
Hello All,
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
What else I can do (as user or developer) for fixing this bug?
P.S.
https://centos.org/distro-faq/
Q5: Does this mean that CentOS Stream is the RHEL BETA test platform now?
A: No. CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release
==================================
But currently I hit the bug, which is absent in CentOS 8.3 but present in CentOS Stream 8. Looks like CentOS Stream 8 is really the beta version for future RHEL minor release, despite the CentOS FAQ. Sorry, but this is true.
-- Best regards, Gena _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
We should have a new kernel out in the next couple of days, next step would be to see if that bug is fixed in this build: https://koji.mbox.centos.org/koji/buildinfo?buildID=15499
One thing we might be able to update about the documentation if you're looking into the sources is to add a link here: https://wiki.centos.org/Sources (see the section there describing the get_sources.sh script). We store large sources in a separate lookaside, not directly in the dist-git trees.
Cheers! --Brian
On 14.01.2021 23:06, Brian Stinson wrote:
On Thu, Jan 14, 2021, at 14:50, Gena Makhomed wrote:
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
What else I can do (as user or developer) for fixing this bug?
We should have a new kernel out in the next couple of days, next step would be to see if that bug is fixed in this build: https://koji.mbox.centos.org/koji/buildinfo?buildID=15499
I check latest CentOS Stream 8 kernel version 4.18.0-269.el8.x86_64 - this regression with broken nspawn containers still present here.
Does any chance exist that this kernel regression will be fixed in CentOS Stream 8 before CentOS 8.4 / RHEL 8.4 will be released?
One thing we might be able to update about the documentation if you're looking into the sources is to add a link here: https://wiki.centos.org/Sources (see the section there describing the get_sources.sh script). We store large sources in a separate lookaside, not directly in the dist-git trees.
Thank you! Now I can download source - but all patches are merged into one file linux-4.18.0-269.el8.tar.xz - I have no chances to find which patch to the kernel source tree causes this regression with broken systemd-nspawn containers. As I understand, only Red Hat kernel developers now can find the root cause of regression and fix it.
And all I can do is file a bug https://bugzilla.redhat.com/show_bug.cgi?id=1913806 and hope for the best?
On 1/25/21 10:21 AM, Gena Makhomed wrote:
On 14.01.2021 23:06, Brian Stinson wrote:
On Thu, Jan 14, 2021, at 14:50, Gena Makhomed wrote:
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
What else I can do (as user or developer) for fixing this bug?
We should have a new kernel out in the next couple of days, next step would be to see if that bug is fixed in this build: https://koji.mbox.centos.org/koji/buildinfo?buildID=15499
I check latest CentOS Stream 8 kernel version 4.18.0-269.el8.x86_64
- this regression with broken nspawn containers still present here.
Does any chance exist that this kernel regression will be fixed in CentOS Stream 8 before CentOS 8.4 / RHEL 8.4 will be released?
One thing we might be able to update about the documentation if you're looking into the sources is to add a link here: https://wiki.centos.org/Sources (see the section there describing the get_sources.sh script). We store large sources in a separate lookaside, not directly in the dist-git trees.
Thank you! Now I can download source - but all patches are merged into one file linux-4.18.0-269.el8.tar.xz - I have no chances to find which patch to the kernel source tree causes this regression with broken systemd-nspawn containers. As I understand, only Red Hat kernel developers now can find the root cause of regression and fix it.
This is how Red Hat releases all kernel sources.
And all I can do is file a bug https://bugzilla.redhat.com/show_bug.cgi?id=1913806 and hope for the best?
On Tuesday, January 26, 2021 9:32 AM, Johnny Hughes johnny@centos.org wrote:
On 1/25/21 10:21 AM, Gena Makhomed wrote:
On 14.01.2021 23:06, Brian Stinson wrote:
On Thu, Jan 14, 2021, at 14:50, Gena Makhomed wrote:
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806 After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself. This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster. What else I can do (as user or developer) for fixing this bug?
We should have a new kernel out in the next couple of days, next step would be to see if that bug is fixed in this build: https://koji.mbox.centos.org/koji/buildinfo?buildID=15499
I check latest CentOS Stream 8 kernel version 4.18.0-269.el8.x86_64
- this regression with broken nspawn containers still present here.
Does any chance exist that this kernel regression will be fixed in CentOS Stream 8 before CentOS 8.4 / RHEL 8.4 will be released?
One thing we might be able to update about the documentation if you're looking into the sources is to add a link here: https://wiki.centos.org/Sources (see the section there describing the get_sources.sh script). We store large sources in a separate lookaside, not directly in the dist-git trees.
Thank you! Now I can download source - but all patches are merged into one file linux-4.18.0-269.el8.tar.xz - I have no chances to find which patch to the kernel source tree causes this regression with broken systemd-nspawn containers. As I understand, only Red Hat kernel developers now can find the root cause of regression and fix it.
This is how Red Hat releases all kernel sources.
And all I can do is file a bug https://bugzilla.redhat.com/show_bug.cgi?id=1913806 and hope for the best?
It is true that this has been how Red Hat releases all it's kernel sources and the issue has been outside of CentOS' control.
The immediate answer would be to file a bug and hope for the best (I share your frustration).
However, this issue has been heard and is being addressed already. Check the thread "RFC: Stream Kernel SIG Proposal." On Jan 20th, Mike McGrath indicates part of the move of Stream into gitlab in a few weeks includes being able to see individual git commits.
At this point I think he would have committed to a more solid date if he could have. My best guess is there are issues going with with the gitlab transition outside of his control. Maybe we will learn more from the CentOS Dojo @FOSDEM.
Greetings,
----- Original Message -----
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
What else I can do (as user or developer) for fixing this bug?
P.S.
https://centos.org/distro-faq/
Q5: Does this mean that CentOS Stream is the RHEL BETA test platform now?
A: No. CentOS Stream will be getting fixes and features ahead of RHEL. Generally speaking we expect CentOS Stream to have fewer bugs and more runtime features as it moves forward in time but always giving direct indication of what is going into a RHEL release
==================================
But currently I hit the bug, which is absent in CentOS 8.3 but present in CentOS Stream 8. Looks like CentOS Stream 8 is really the beta version for future RHEL minor release, despite the CentOS FAQ. Sorry, but this is true.
-- Best regards, Gena
You do realize that Red Hat doesn't really support every function available in every program, right? I wonder how many users are using systemd-nspawn-based containers? My point is that they can't test every use case that exists and have no bugs or regressions. I realize that feature is very important to you... but no RHEL release has ever been perfect. CentOS Linux, as a downstream of RHEL... being 100% compatible... prides|d itself in having all the same bugs as RHEL. :) While Stream wants to have less bugs than RHEL, that doesn't mean there won't be any bugs that appear first in Stream. No matter how much testing is done, the real world reveals more.
There *WILL* be bugs in Stream. There are bugs in RHEL. The more lines-of-code there are, potentially the more bugs... and no one wants to nor can fix all of them. They fix the most important ones using someone's determination. I only speak for myself. Thanks for reporting the bug. It definitely has a greater chance of getting fixed if it is reported.
I tried to check out your bugzilla link but even after logging in, it was blocked. :(
TYL,
Greetings,
----- Original Message -----
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
Ok, I got added to the bug and could see it... and in the reproduce steps, disabling SELinux is a step? Does Red Hat support the system if SElinux has been disabled? (even if it isn't a factor) Is it just in permissive mode or is it completely disabled? My mind is blown. I guess it doesn't take much.
TYL,
On 14/01/2021 20:50, Gena Makhomed wrote:
Hello All,
I found regression in the CentOS Stream 8 kernels, and create bug report https://bugzilla.redhat.com/show_bug.cgi?id=1913806
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
What else I can do (as user or developer) for fixing this bug?
Not a lot really, because until Red Hat / CentOS provide a proper Git repository of all kernel commits it will be impossible for you to bisect the code to identify the commit that introduced the bug, let alone offer a fix.
About all you can do is file a bug and hope for the best.
On 1/14/21 12:50 PM, Gena Makhomed wrote:
After reading https://wiki.centos.org/Contribute/CentOSStream I try to download kernel sources and look for root cause of bug, but dist-git and source-git repos does not contain kernel sources, even if I try to execute %prep phase. For this reason I can't even try to find root cause of this bug by myself.
See Brian Stinson's message today, re: "Questions regarding CS8"
https://composes.centos.org/CentOS-Stream-8-20210108.n.2/compose/BaseOS/sour...
https://composes.centos.org/CentOS-Stream-8-20201210.n.0/compose/BaseOS/sour...
Hi,
Am 14.01.21 um 20:50 schrieb Gena Makhomed:
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
There will be no CentOS 8.4 anymore, right?
Rainer
On 26/01/2021 19:44, Rainer Traut wrote:
Hi,
Am 14.01.21 um 20:50 schrieb Gena Makhomed:
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
There will be no CentOS 8.4 anymore, right?
RHEL8 is supposed to be on a 6 month release schedule for minor releases, so I'd expect RHEL8.4 April/May and RHEL8.5 around beginning of Nov. Given CentOS 8 continues until end of 2021, I'd expect CentOS 8.4 and even 8.5 depending when RHEL8.5 ships and how quickly the team can turn around that release.
On 1/26/21 9:50 PM, Phil Perry wrote:
On 26/01/2021 19:44, Rainer Traut wrote:
Hi,
Am 14.01.21 um 20:50 schrieb Gena Makhomed:
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
There will be no CentOS 8.4 anymore, right?
RHEL8 is supposed to be on a 6 month release schedule for minor releases, so I'd expect RHEL8.4 April/May and RHEL8.5 around beginning of Nov. Given CentOS 8 continues until end of 2021, I'd expect CentOS 8.4 and even 8.5 depending when RHEL8.5 ships and how quickly the team can turn around that release.
Based on past experience, I am willing to bet on no CentOS 8.5. Even the last updates for EL6 were not properly distributed before completely retiring CentOS 6 despite being much easier to build and push and no one would have been hurt if mirrors kept C6 another week.
I am 99% sure that even if RH releases 8.5 in nov, CentOS 8.5 will not be ready before CentOS 8 being EOLed. /M//aybe/, just /maybe/ we might see a CentOS 8.5 in Dec if RHEL 8.5 sees the light in Oct. But I doubt that, too.
@GenaMakhomed: If possible, you'd better take advantage of the new subscriptions from RH and switch to using RHEL proper. CentOS 8 is dead, it's just kept on life support for another 11 months.
On Tuesday, January 26, 2021 2:56 PM, Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
On 1/26/21 9:50 PM, Phil Perry wrote:
On 26/01/2021 19:44, Rainer Traut wrote:
Hi,
Am 14.01.21 um 20:50 schrieb Gena Makhomed:
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
There will be no CentOS 8.4 anymore, right?
RHEL8 is supposed to be on a 6 month release schedule for minor releases, so I'd expect RHEL8.4 April/May and RHEL8.5 around beginning of Nov. Given CentOS 8 continues until end of 2021, I'd expect CentOS 8.4 and even 8.5 depending when RHEL8.5 ships and how quickly the team can turn around that release.
Based on past experience, I am willing to bet on no CentOS 8.5. Even the last updates for EL6 were not properly distributed before completely retiring CentOS 6 despite being much easier to build and push and no one would have been hurt if mirrors kept C6 another week.
I am 99% sure that even if RH releases 8.5 in nov, CentOS 8.5 will not be ready before CentOS 8 being EOLed. Maybe, just maybe we might see a CentOS 8.5 in Dec if RHEL 8.5 sees the light in Oct. But I doubt that, too.
@GenaMakhomed: If possible, you'd better take advantage of the new subscriptions from RH and switch to using RHEL proper. CentOS 8 is dead, it's just kept on life support for another 11 months.
The legacy C6 updates are still up on vault.centos.org. Several of the final updates were released November 9th. Terminating the mirrors and replacing them with the readme provided a strong message that people needed to migrate off C6.
In terms of there possibly never being a CentOS 8.5, I kind of agree and disagree. I think it is very likely there will be no install media update for 8.5. But I think CR will get the critical package updates fromn RHEL 8.5.
On 1/26/21 5:04 PM, redbaronbrowser via CentOS-devel wrote:
On Tuesday, January 26, 2021 2:56 PM, Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
On 1/26/21 9:50 PM, Phil Perry wrote:
On 26/01/2021 19:44, Rainer Traut wrote:
Hi,
Am 14.01.21 um 20:50 schrieb Gena Makhomed:
This bug is critical for me, because currently I try to use systemd-nspawn for containers virtualization in production. If future CentOS 8.4 will contain this bug - it will be disaster.
There will be no CentOS 8.4 anymore, right?
RHEL8 is supposed to be on a 6 month release schedule for minor releases, so I'd expect RHEL8.4 April/May and RHEL8.5 around beginning of Nov. Given CentOS 8 continues until end of 2021, I'd expect CentOS 8.4 and even 8.5 depending when RHEL8.5 ships and how quickly the team can turn around that release.
Based on past experience, I am willing to bet on no CentOS 8.5. Even the last updates for EL6 were not properly distributed before completely retiring CentOS 6 despite being much easier to build and push and no one would have been hurt if mirrors kept C6 another week.
I am 99% sure that even if RH releases 8.5 in nov, CentOS 8.5 will not be ready before CentOS 8 being EOLed. Maybe, just maybe we might see a CentOS 8.5 in Dec if RHEL 8.5 sees the light in Oct. But I doubt that, too.
@GenaMakhomed: If possible, you'd better take advantage of the new subscriptions from RH and switch to using RHEL proper. CentOS 8 is dead, it's just kept on life support for another 11 months.
The legacy C6 updates are still up on vault.centos.org. Several of the final updates were released November 9th. Terminating the mirrors and replacing them with the readme provided a strong message that people needed to migrate off C6.
In terms of there possibly never being a CentOS 8.5, I kind of agree and disagree. I think it is very likely there will be no install media update for 8.5. But I think CR will get the critical package updates fromn RHEL 8.5.
I will work on 8.5 as long as i am allowed to do so. Certainly and 8.4 will be done.
-
Brian Stinson -
Gena Makhomed -
Gordon Messmer -
Johnny Hughes -
Manuel Wolfshant -
Phil Perry -
Rainer Traut -
redbaronbrowser -
Scott Dowdle