[CentOS-gsoc] kpatch irc log 2015-03-16
Corey Henderson
corman at cormander.com
Tue Mar 17 05:11:23 UTC 2015
Here's to hoping the copy/paste out of ChatZilla doesn't get screwed up
in email :)
cormander huzzah, I haven't been on IRC in years...
kragniz cormander: \o
cormander howdy
kragniz cormander: you should spend more time here!
cormander let's just say, I have a hard enough time managing
distractions and social anxiety without being in a chat room all day long
kragniz maybe I will learn something similar one day
kragniz anyway, kpatch stuff!
cormander pardon my ignorance - but do I mentor one student, or many?
cormander and if you don't know the answer, that's fine
kragniz just the one is what people say
cormander well so far it's you and one other person that's contacted me
directly
cormander though someone who is closer to my timezone is probably
better suited
kragniz you'll need to pick one, I'm pretty sure - the actual selection
takes place in a few weeks, though
cormander okay so remind me - was kpatch your first choice? one of you
said it wasn't your first choice ...
kragniz you should talk with the centos gsoc admins, though
cormander yeppers will do
kragniz now you ask that question, I guess it is
cormander so I have some recommended reading for you regarding kernel
patching
kragniz I was weighing it and tinykdump equally in my head previously
cormander which is, in some ways, a shameless self promotion...
kragniz fire away!
cormander er, I would have sworn I posted about kpatch on my blog
cormander okay plan B, my older post about ksplice
kragniz heh
cormander holy crap that was 4 years ago. I feel old now.
cormander
http://cormander.com/2011/08/how-to-use-the-ksplice-raw-utilities/
kragniz I'm guessing this one?
http://cormander.com/2011/08/how-to-use-the-ksplice-raw-utilities/
kragniz hehe
cormander not to be a copycat or anything, but that's roughly the same
kind of process I'd like to work with
cormander the kpatch utilities (as of about 6 months ago - haven't
looked at them since) worked similar to it
kragniz yup, they do
cormander each tree needs to be treated like it's own line of code,
because not all patches are applicable to the older ones, and not all
fixes go into each version exactly the same way
kragniz (as of last week)
cormander and by tree, I mean version/release of the kernel
cormander so one patch may go into 4 different versions exactly the
same way, but there will come a point where it needs to be modified to
work with a newer version
cormander and vice versa
cormander originally I wanted to maintain one big patch for each
version and just update it each time a new version came out
cormander but, as I started actually working on it, I discovered that
until kpatch solves some technical problems with kernel functions that
are always on the stack
cormander the bigger the patch, the more risky it is that once you
remove it, it becomes difficult to re-insert it again
kragniz okay, that's interesting
cormander we don't want to hit a patch with a heavy kernel thread and
fight the battle of yanking it / re-probing it every time
cormander because most patches don't run that risk
cormander but that creates a different techincal problem - tracking
which functions are being touched
cormander not a hard problem to solve if you stay consistent, but then
you run into issues with being incompatible with other kpatch efforts,
should there ever be any, if everyone doesn't agree on an API for it
kragniz couldn't that be solved by adding that to upstream kpatch-build?
cormander ie; my project tpe-lkm modifies live kernel code, and does it
outside of kpatch
cormander https://github.com/cormander/tpe-lkm/
kragniz clicks
cormander tpe-lkm is better at supporting a wide range of kernel
versions, but it's not as fine-grained as kpatch
cormander and thus, is limited in what kinds of patching it can do
kragniz as an aside, that project is really cool!
cormander thanks
cormander now, we're not going to be able to solve compatibility issues
with non-kpatch projects like that
cormander but it's worth thinking about
kragniz but getting something that keeps in mind other systems would be
good?
cormander yeah
kragniz okay
cormander as an aside, my primary motivation into learning kpatch was
to apply portions of grsecurity via kpatch
cormander just like how I apply some of them via tpe-lkm,
cormander but kpatch can do many more of them
kragniz interesting
cormander the bump I hit was, which derailed my efforts, that problem
with functions in heavy use
cormander it's probably solvable, but let's just say I only have so
much motivation to solve certain problems
kragniz hehe
kragniz have you played with the livepatching in linux 4.0?
cormander no
cormander I've barely scratched the surface with kpatch on EL7
kragniz (just changing the topic slightly)
kragniz okay
cormander I went hardcore into the kpatch code and submitted patches
and all that to https://github.com/dynup/kpatch
cormander for a few weeks
cormander then life happened again and knocked me off the wagon
kragniz yeah, I had a look at them
kragniz life is a sad thing
cormander have you read this?
cormander https://github.com/dynup/kpatch#limitations
kragniz yup
cormander we need a strategy to handle patches that touch those limitations
cormander which may just end up being "sorry folks, you actually have
to reboot now"
cormander but I'm leaving that as a last resort
cormander for example, kpatch uses something called shodows
cormander and then you can redirect the use of statically allocated
data and use the shadows instead
cormander *shadows
kragniz shadows being new copies of the old data?
cormander yes
cormander one very important thing we need, moreso than the API to
marry all our different patches together (so they don't stomp on each other)
cormander is some kind of automated QA testing
kragniz CI on each patch?
cormander yes
kragniz okay!
cormander something that boots a VM, applies each patch in sequence,
and runs some sort of kernel test suite to make sure it doesn't go kablooey
cormander and then do it for each kernel version we support
kragniz doing that in a VM is okay?
cormander yes
kragniz cool
kragniz that sort of thing is fun
kragniz although, setting that up may be a project in itself
cormander the good news is, the "kernel test suite" would only need to
be limited to calling the functions that are patched
cormander so initially, the test suite is small, and we simply write
more tests as patches touch more symbols
cormander I agree, and I need to specifically talk to kbsingh about that
kragniz yup, so you could collect the functions being patched an
automatically build a test suite?
cormander if you think that could be done, go for it
cormander I was thinking along the lines of just small c code a.out
executables that make certain syscalls
kragniz that sounds good, also
kragniz simple is better, most of the time
cormander occam's razor
cormander so, do you think this stuff is up your alley?
kragniz totally!
cormander full disclosure: I'm not as skilled with kernel C code as you
might think, considering I've written quote a lot of kernel module code
myself
cormander I shoot myself in the foot more than I care to admit
kragniz I warn you, though, I'm still learning kernel level stuff
kragniz we can learn together!
cormander great! we're both good enough to be dangerous
kragniz I've done a fair amount of userland C, though python is my pet
cormander perl here
kragniz shakes his fist
cormander alright so - what timezone are you in? I hope this isn't too
late in the evening for you
kragniz normally utc, but I'm in est at the moment
cormander ahh! 5am utc
cormander too early?
kragniz I work on openstack, so I'm used to late night irc meetings!
kragniz so, more about the general scope of the project:
cormander I'll be as accommodating as I can - although I do get up at
4am from time to time myself
kragniz should this be a system which is only designed to run on
centos? anything rpm based? anything running linux?
cormander good questions,
cormander for now, only designed for EL7
cormander yes RPM based
cormander needs to be compatible with yum
cormander the API of what function(s) each of our kpatches use could be
as simple as
cormander making an rpm "virtual" dependancy
cormander kernel-kpatch-function-X
cormander where X is the functions it touches
kragniz hmm, that would be interesting
cormander I forget what I'm thinking of is called
cormander where you specify this is the only package that can touch this
cormander installing another with it will get rejected
kragniz conflicts?
cormander yes!
cormander wow it's been a while
kragniz hehe
cormander there will be situations where two patches touch the same
functions
cormander they need to get rolled together into an update for that
particular patch
cormander so, occationally, an already released kpatch rpm will get an
update
cormander but mostly, it's just installing new kpatch rpms
kragniz okay, so when that new package comes along, the other patches
will be removed
cormander updated
cormander and by being updated, the old one will get rmmod'd, and
re-insmod with the new one
cormander or however kpatch handles that
kragniz if it conflicts, shouldn't the whole conflicting package be
uninstalled?
cormander if it conflicts, and it's from our repository, that means we
didn't do a good QA job
cormander I'll explain it like this;
cormander we have patches A B C D E F G
cormander all touching different things
cormander patch H comes out, and touches what A does
cormander instead of releasing H as a new rpm
cormander we update A to include H
kragniz right, I get you now!
cormander then, patch J comes out, it touches what D and G do
kragniz I thought you meant bundle As patch with H and remove A
cormander we update D to include G and J, and have it "depricate" G
cormander or whatever term rpm uses for depricate ... where it
uninstalls it upon install
kragniz yup, that makes sense
cormander and this, my friend, is where the QA matters most
kragniz indeed!
cormander like I said, if an rpm conflicts, we f***ed up
kragniz yup
cormander so, chew on that for now. I actually have to get up early
tomorrow
cormander see you tomorrow around this time?
kragniz alrighty
kragniz sounds good!
kragniz thanks for the thoughts!
cormander goes "away" - if there is such a thing in IRC
kragniz goes to bed
More information about the CentOS-gsoc
mailing list