Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel patches will be available via yum updates as of RHEL 7.7. Is this carried over to CentOS 7.7.1908?
--
*Matt Phelps*
*Information Technology Specialist, Systems Administrator*
(Computation Facility, Smithsonian Astrophysical Observatory)
Center for Astrophysics | Harvard & Smithsonian
60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps@cfa.harvard.edu
cfa.harvard.edu | Facebook http://cfa.harvard.edu/facebook | Twitter http://cfa.harvard.edu/twitter | YouTube http://cfa.harvard.edu/youtube | Newsletter http://cfa.harvard.edu/newsletter
On 10/3/19 1:32 PM, Phelps, Matthew wrote:
Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel patches will be available via yum updates as of RHEL 7.7. Is this carried over to CentOS 7.7.1908?
The functionality should be available, but we don't provide patches in this way, no.
On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 1:32 PM, Phelps, Matthew wrote:
Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel
patches
will be available via yum updates as of RHEL 7.7. Is this carried over to CentOS 7.7.1908?
The functionality should be available, but we don't provide patches in this way, no.
-- Jim Perrin The CentOS Project | http://www.centos.org twitter: @BitIntegrity | GPG Key: FA09AD77 _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
What would it take to make this happen? This would be a huge help to those of us running servers. Not to mention it would make the world a more secure place :)
Is it an upstream issue? No SRPMS available? Etc?
Just trying to understand. I don't follow the centos-devel list. Has this been discussed there, or elsewhere?
On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 1:32 PM, Phelps, Matthew wrote:
Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel
patches
will be available via yum updates as of RHEL 7.7. Is this carried over to CentOS 7.7.1908?
The functionality should be available, but we don't provide patches in this way, no.
What would it take to make this happen? This would be a huge help to those of us running servers. Not to mention it would make the world a more secure place :)
Is it an upstream issue? No SRPMS available? Etc?
Just trying to understand. I don't follow the centos-devel list. Has this been discussed there, or elsewhere?
There is a lot to go into making a correct kpatch. You have to determine that you have a working kpatch (you can have one which works on 1% and corrupts 80% and crashes 19%), you have to determine that the patch fixes the problem (you can build patches which should do the right thing but don't), and you have to determine that it doesn't add in some sort of long term corruption of memory/disk/etc. That takes specialized kernel expertise, a large amount of varied hardware to test the patch on, some amount of time, and a very large test suite.
You can also only live patch a system so many times and in only certain places. There are just some parts of the kernel which have to be rebooted and others you can put in a patch which works but your performance is going to be 25% of what it was before. There are other places that if you patch.. that is it.. try another and you hardlock. As much as some sites like to call it some sort of panacea for never having to reboot again.. it is really meant to be a tourniquet to air chopter the crash victim to a hospital. They may still not make it... you are just giving them a chance.
On Thu, 3 Oct 2019 21:35:15 -0400 Stephen John Smoogen wrote:
There is a lot to go into making a correct kpatch.
Thank you ever so much for your very clear description of what's needed to accomplish something like this.
I, for one, had no idea. And I feel much smarter now than before!
On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 1:32 PM, Phelps, Matthew wrote:
Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel
patches
will be available via yum updates as of RHEL 7.7. Is this carried over to CentOS 7.7.1908?
The functionality should be available, but we don't provide patches in this way, no.
What would it take to make this happen? This would be a huge help to those of us running servers. Not to mention it would make the world a more secure place :)
The short answer is "a team of kernel engineers, which we don't have". Smooge's overview which I've left below is great at explaining some of this:
Is it an upstream issue? No SRPMS available? Etc?
It's quite a bit more work than just SRPM (re) building. This is one of those things where if your workflow requires this functionality rather than the occasional reboot you should really just pay for RHEL. They put far more people and testing behind this feature than the team building CentOS is able to.
(DISCLAIMER: I work for RH, so that may not sound as true as it is)
Just trying to understand. I don't follow the centos-devel list. Has this been discussed there, or elsewhere?
There is a lot to go into making a correct kpatch. You have to determine that you have a working kpatch (you can have one which works on 1% and corrupts 80% and crashes 19%), you have to determine that the patch fixes the problem (you can build patches which should do the right thing but don't), and you have to determine that it doesn't add in some sort of long term corruption of memory/disk/etc. That takes specialized kernel expertise, a large amount of varied hardware to test the patch on, some amount of time, and a very large test suite.
You can also only live patch a system so many times and in only certain places. There are just some parts of the kernel which have to be rebooted and others you can put in a patch which works but your performance is going to be 25% of what it was before. There are other places that if you patch.. that is it.. try another and you hardlock. As much as some sites like to call it some sort of panacea for never having to reboot again.. it is really meant to be a tourniquet to air chopter the crash victim to a hospital. They may still not make it... you are just giving them a chance.
On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew mphelps@cfa.harvard.edu
wrote:
On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 1:32 PM, Phelps, Matthew wrote:
Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel
patches
will be available via yum updates as of RHEL 7.7. Is this carried
over to
CentOS 7.7.1908?
The functionality should be available, but we don't provide patches in this way, no.
What would it take to make this happen? This would be a huge help to
those
of us running servers. Not to mention it would make the world a more
secure
place :)
The short answer is "a team of kernel engineers, which we don't have". Smooge's overview which I've left below is great at explaining some of this:
I don't understand. If RHEL is putting out patches, and CentOS is a recompile of RHEL, hasn't that "team of kernel engineers " already done the work?
I fully realize this is not a panacea for never rebooting again, but if we can patch a critical kernel bug immediately, then schedule less disruptive reboots in a week or three, this would help tremendously.
Is it an upstream issue? No SRPMS available? Etc?
It's quite a bit more work than just SRPM (re) building. This is one of those things where if your workflow requires this functionality rather than the occasional reboot you should really just pay for RHEL. They put far more people and testing behind this feature than the team building CentOS is able to.
(DISCLAIMER: I work for RH, so that may not sound as true as it is)
I knew someone was going to say that. :) In our case, as I'm sure is the case for many other environments, we are a noncommercial CentOS shop that can't afford the resources to have a mixed environment, not to mention the RHEL licenses. Not all of the machines I'm thinking of are critical infrastructure. We have many researchers running simulations that take weeks, sometimes months, to finish, and avoiding the occasional forced immediate reboot for a critical kernel bug would help expand Human Knowledge :).
Anyway, I saw the functionality for live kernel patching in the RHEL 7.7 release notes, which the CentOS 7.7.1908 release notes pointed to, and assumed (hoped?) that it would be available for us as well. If it won't ever be provided, then I suggest the CentOS documentation be updated to explicitly state so.
Just trying to understand. I don't follow the centos-devel list. Has
this
been discussed there, or elsewhere?
There is a lot to go into making a correct kpatch. You have to determine that you have a working kpatch (you can have one which works on 1% and corrupts 80% and crashes 19%), you have to determine that the patch fixes the problem (you can build patches which should do the right thing but don't), and you have to determine that it doesn't add in some sort of long term corruption of memory/disk/etc. That takes specialized kernel expertise, a large amount of varied hardware to test the patch on, some amount of time, and a very large test suite.
You can also only live patch a system so many times and in only certain places. There are just some parts of the kernel which have to be rebooted and others you can put in a patch which works but your performance is going to be 25% of what it was before. There are other places that if you patch.. that is it.. try another and you hardlock. As much as some sites like to call it some sort of panacea for never having to reboot again.. it is really meant to be a tourniquet to air chopter the crash victim to a hospital. They may still not make it... you are just giving them a chance.
-- Jim Perrin The CentOS Project | http://www.centos.org twitter: @BitIntegrity | GPG Key: FA09AD77 _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Fri, 4 Oct 2019 at 08:18, Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew mphelps@cfa.harvard.edu
wrote:
On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 1:32 PM, Phelps, Matthew wrote:
Forgive me if this has been answered before and I've missed it.
This https://access.redhat.com/solutions/2206511 says live kernel
patches
will be available via yum updates as of RHEL 7.7. Is this carried
over to
CentOS 7.7.1908?
The functionality should be available, but we don't provide patches in this way, no.
What would it take to make this happen? This would be a huge help to
those
of us running servers. Not to mention it would make the world a more
secure
place :)
The short answer is "a team of kernel engineers, which we don't have". Smooge's overview which I've left below is great at explaining some of this:
I don't understand. If RHEL is putting out patches, and CentOS is a recompile of RHEL, hasn't that "team of kernel engineers " already done the work?
No. because most of the work on making a patch is after the kernel is compiled and working. Thus even though you have the same source code, similar compilers etc.. there are going to be differences which have to be looked at to make sure it is really working. A CentOS kernel is not exactly the same as a RHEL kernel is not the same as a Oracle kernel is not the same as the one you recompiled locally. From most operational points they seem the same, but kernel patching is where those differences really show up.
Yes it would be easy to set up some automated tool which 'made' kpatches.. and I expect they may 'work' for most systems. But I also expect that they would also eat babies more times than people would like. If sites really need them, they can set up the tooling themselves and make them work when they know they want it. Trying to make it a general purpose answer for something which may corrupt data 5 or 20% or 40% of the time.. is just waiting to be on Slashdot daily (wait do we do Slashdot anymore.. Reddit? nope the kids aren't there anymore either.. ok someplace daily) in a bad way.
On Fri, Oct 4, 2019 at 9:24 AM Stephen John Smoogen smooge@gmail.com wrote:
On Fri, 4 Oct 2019 at 08:18, Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Fri, Oct 4, 2019 at 6:33 AM Jim Perrin jperrin@centos.org wrote:
On 10/3/19 9:35 PM, Stephen John Smoogen wrote:
On Thu, 3 Oct 2019 at 13:52, Phelps, Matthew <
mphelps@cfa.harvard.edu>
wrote:
On Thu, Oct 3, 2019 at 1:42 PM Jim Perrin jperrin@centos.org
wrote:
On 10/3/19 1:32 PM, Phelps, Matthew wrote: > Forgive me if this has been answered before and I've missed it. > > This https://access.redhat.com/solutions/2206511 says live kernel patches > will be available via yum updates as of RHEL 7.7. Is this carried
over to
> CentOS 7.7.1908? >
The functionality should be available, but we don't provide
patches in
this way, no.
What would it take to make this happen? This would be a huge help to
those
of us running servers. Not to mention it would make the world a more
secure
place :)
The short answer is "a team of kernel engineers, which we don't have". Smooge's overview which I've left below is great at explaining some of this:
I don't understand. If RHEL is putting out patches, and CentOS is a recompile of RHEL, hasn't that "team of kernel engineers " already done
the
work?
No. because most of the work on making a patch is after the kernel is compiled and working. Thus even though you have the same source code, similar compilers etc.. there are going to be differences which have to be looked at to make sure it is really working. A CentOS kernel is not exactly the same as a RHEL kernel is not the same as a Oracle kernel is not the same as the one you recompiled locally. From most operational points they seem the same, but kernel patching is where those differences really show up.
Yes it would be easy to set up some automated tool which 'made' kpatches.. and I expect they may 'work' for most systems. But I also expect that they would also eat babies more times than people would like. If sites really need them, they can set up the tooling themselves and make them work when they know they want it. Trying to make it a general purpose answer for something which may corrupt data 5 or 20% or 40% of the time.. is just waiting to be on Slashdot daily (wait do we do Slashdot anymore.. Reddit? nope the kids aren't there anymore either.. ok someplace daily) in a bad way.
Thanks for the explanation(s).
I'm still puzzled why RedHat is doing it then, and making it more generally available (to paying customers even), if it's so dire a proposition that it will fail so badly, so often. That seems counter-intuitive to me.
Anyway, I again point out that the CentOS documentation should be made clear that this functionality won't ever be coming to CentOS.
-Matt
Am 04.10.19 um 15:35 schrieb Phelps, Matthew:
I'm still puzzled why RedHat is doing it then, and making it more generally available (to paying customers even), if it's so dire a proposition that it will fail so badly, so often. That seems counter-intuitive to me.
I've been using kernel live patching on an Ubuntu Machine for several years now without any problems (Ubuntu offers them for registered users for free (three machines per account)) and haven't noticed any downsides so far.
On Fri, 4 Oct 2019 at 09:36, Phelps, Matthew mphelps@cfa.harvard.edu wrote:
On Fri, Oct 4, 2019 at 9:24 AM Stephen John Smoogen smooge@gmail.com wrote:
Thanks for the explanation(s).
I'm still puzzled why RedHat is doing it then, and making it more generally available (to paying customers even), if it's so dire a proposition that it will fail so badly, so often. That seems counter-intuitive to me.
Because they have kernel developers, qa, and other staff dedicated to making that kpatch work? They have a large set of servers to test different workloads? They have some time before the kernel is built internally and when it is made available externally to do all this and hand tune any problems found? Because big companies are paying a large amount of money to make it work and so the extra labour is profitable?
In the past, all of this would be a challenge for people to come together and show that they can also do it themselves... or improve on something to make it so less labour intensive at parts. If that happens, I am happy to have laid out the challenge :).
Anyway, I again point out that the CentOS documentation should be made clear that this functionality won't ever be coming to CentOS.
-Matt
--
*Matt Phelps*
*Information Technology Specialist, Systems Administrator*
(Computation Facility, Smithsonian Astrophysical Observatory)
Center for Astrophysics | Harvard & Smithsonian
60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps@cfa.harvard.edu
cfa.harvard.edu | Facebook http://cfa.harvard.edu/facebook | Twitter http://cfa.harvard.edu/twitter | YouTube http://cfa.harvard.edu/youtube | Newsletter http://cfa.harvard.edu/newsletter _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 10/4/19 9:35 AM, Phelps, Matthew wrote:
... I'm still puzzled why RedHat is doing it then, and making it more generally available (to paying customers even), if it's so dire a proposition that it will fail so badly, so often. That seems counter-intuitive to me.
It would likely boil down to a risk-benefit analysis; for RHEL RH is willing to take the risks associated with it due to the added benefits of offering it. And, well, the elephant in the room is that it is one of the things that make an RHEL subscription more attractive, whether that's an intended effect or not. Ubuntu/Canonical apparently made a different analysis, per another poster in-thread.
Of course, I'm in a similar situation to you in that we're a non-profit and don't have the budget for RHEL subscriptions. So what I've done here is to stay on top of what the kernel issues are, and schedule reboots accordingly, and take those long-running analysis job machines and temporarily suspend general Internet accessibility until a reboot is possible if the kernel issue warrants that. I likely don't have anywhere near as many of those jobs running as you, but I still can sympathize!
Anyway, I again point out that the CentOS documentation should be made clear that this functionality won't ever be coming to CentOS.
I would suggest the team, rather than a blanket statement that it's 'never' coming to CentOS would articulate (Smooge's posts are a great start!) what it would take from the community to make it happen, thus leaving the question open-ended.
-
Frank Cox -
Jim Perrin -
Kai Bojens -
Lamar Owen -
Phelps, Matthew -
Stephen John Smoogen