[CentOS-gsoc] kpatch irc log 2015-03-16

Tue Mar 17 05:11:23 UTC 2015
Corey Henderson <corman at cormander.com>

Here's to hoping the copy/paste out of ChatZilla doesn't get screwed up 
in email :)

	cormander	huzzah, I haven't been on IRC in years...
	kragniz	cormander: \o
	cormander	howdy
	kragniz	cormander: you should spend more time here!
	cormander	let's just say, I have a hard enough time managing 
distractions and social anxiety without being in a chat room all day long
	kragniz	maybe I will learn something similar one day
	kragniz	anyway, kpatch stuff!
	cormander	pardon my ignorance - but do I mentor one student, or many?
	cormander	and if you don't know the answer, that's fine
	kragniz	just the one is what people say
	cormander	well so far it's you and one other person that's contacted me 
directly
	cormander	though someone who is closer to my timezone is probably 
better suited
	kragniz	you'll need to pick one, I'm pretty sure - the actual selection 
takes place in a few weeks, though
	cormander	okay so remind me - was kpatch your first choice? one of you 
said it wasn't your first choice ...
	kragniz	you should talk with the centos gsoc admins, though
	cormander	yeppers will do
	kragniz	now you ask that question, I guess it is
	cormander	so I have some recommended reading for you regarding kernel 
patching
	kragniz	I was weighing it and tinykdump equally in my head previously
	cormander	which is, in some ways, a shameless self promotion...
	kragniz	fire away!
	cormander	er, I would have sworn I posted about kpatch on my blog
	cormander	okay plan B, my older post about ksplice
	kragniz	heh
	cormander	holy crap that was 4 years ago. I feel old now.
	cormander 
http://cormander.com/2011/08/how-to-use-the-ksplice-raw-utilities/
	kragniz	I'm guessing this one? 
http://cormander.com/2011/08/how-to-use-the-ksplice-raw-utilities/
	kragniz	hehe
	cormander	not to be a copycat or anything, but that's roughly the same 
kind of process I'd like to work with
	cormander	the kpatch utilities (as of about 6 months ago - haven't 
looked at them since) worked similar to it
	kragniz	yup, they do
	cormander	each tree needs to be treated like it's own line of code, 
because not all patches are applicable to the older ones, and not all 
fixes go into each version exactly the same way
	kragniz	(as of last week)
	cormander	and by tree, I mean version/release of the kernel
	cormander	so one patch may go into 4 different versions exactly the 
same way, but there will come a point where it needs to be modified to 
work with a newer version
	cormander	and vice versa
	cormander	originally I wanted to maintain one big patch for each 
version and just update it each time a new version came out
	cormander	but, as I started actually working on it, I discovered that 
until kpatch solves some technical problems with kernel functions that 
are always on the stack
	cormander	the bigger the patch, the more risky it is that once you 
remove it, it becomes difficult to re-insert it again
	kragniz	okay, that's interesting
	cormander	we don't want to hit a patch with a heavy kernel thread and 
fight the battle of yanking it / re-probing it every time
	cormander	because most patches don't run that risk
	cormander	but that creates a different techincal problem - tracking 
which functions are being touched
	cormander	not a hard problem to solve if you stay consistent, but then 
you run into issues with being incompatible with other kpatch efforts, 
should there ever be any, if everyone doesn't agree on an API for it
	kragniz	couldn't that be solved by adding that to upstream kpatch-build?
	cormander	ie; my project tpe-lkm modifies live kernel code, and does it 
outside of kpatch
	cormander	https://github.com/cormander/tpe-lkm/
	kragniz	clicks
	cormander	tpe-lkm is better at supporting a wide range of kernel 
versions, but it's not as fine-grained as kpatch
	cormander	and thus, is limited in what kinds of patching it can do
	kragniz	as an aside, that project is really cool!
	cormander	thanks
	cormander	now, we're not going to be able to solve compatibility issues 
with non-kpatch projects like that
	cormander	but it's worth thinking about
	kragniz	but getting something that keeps in mind other systems would be 
good?
	cormander	yeah
	kragniz	okay
	cormander	as an aside, my primary motivation into learning kpatch was 
to apply portions of grsecurity via kpatch
	cormander	just like how I apply some of them via tpe-lkm,
	cormander	but kpatch can do many more of them
	kragniz	interesting
	cormander	the bump I hit was, which derailed my efforts, that problem 
with functions in heavy use
	cormander	it's probably solvable, but let's just say I only have so 
much motivation to solve certain problems
	kragniz	hehe
	kragniz	have you played with the livepatching in linux 4.0?
	cormander	no
	cormander	I've barely scratched the surface with kpatch on EL7
	kragniz	(just changing the topic slightly)
	kragniz	okay
	cormander	I went hardcore into the kpatch code and submitted patches 
and all that to https://github.com/dynup/kpatch
	cormander	for a few weeks
	cormander	then life happened again and knocked me off the wagon
	kragniz	yeah, I had a look at them
	kragniz	life is a sad thing
	cormander	have you read this?
	cormander	https://github.com/dynup/kpatch#limitations
	kragniz	yup
	cormander	we need a strategy to handle patches that touch those limitations
	cormander	which may just end up being "sorry folks, you actually have 
to reboot now"
	cormander	but I'm leaving that as a last resort
	cormander	for example, kpatch uses something called shodows
	cormander	and then you can redirect the use of statically allocated 
data and use the shadows instead
	cormander	*shadows
	kragniz	shadows being new copies of the old data?
	cormander	yes
	cormander	one very important thing we need, moreso than the API to 
marry all our different patches together (so they don't stomp on each other)
	cormander	is some kind of automated QA testing
	kragniz	CI on each patch?
	cormander	yes
	kragniz	okay!
	cormander	something that boots a VM, applies each patch in sequence, 
and runs some sort of kernel test suite to make sure it doesn't go kablooey
	cormander	and then do it for each kernel version we support
	kragniz	doing that in a VM is okay?
	cormander	yes
	kragniz	cool
	kragniz	that sort of thing is fun
	kragniz	although, setting that up may be a project in itself
	cormander	the good news is, the "kernel test suite" would only need to 
be limited to calling the functions that are patched
	cormander	so initially, the test suite is small, and we simply write 
more tests as patches touch more symbols
	cormander	I agree, and I need to specifically talk to kbsingh about that
	kragniz	yup, so you could collect the functions being patched an 
automatically build a test suite?
	cormander	if you think that could be done, go for it
	cormander	I was thinking along the lines of just small c code a.out 
executables that make certain syscalls
	kragniz	that sounds good, also
	kragniz	simple is better, most of the time
	cormander	occam's razor
	cormander	so, do you think this stuff is up your alley?
	kragniz	totally!
	cormander	full disclosure: I'm not as skilled with kernel C code as you 
might think, considering I've written quote a lot of kernel module code 
myself
	cormander	I shoot myself in the foot more than I care to admit
	kragniz	I warn you, though, I'm still learning kernel level stuff
	kragniz	we can learn together!
	cormander	great! we're both good enough to be dangerous
	kragniz	I've done a fair amount of userland C, though python is my pet
	cormander	perl here
	kragniz	shakes his fist
	cormander	alright so - what timezone are you in? I hope this isn't too 
late in the evening for you
	kragniz	normally utc, but I'm in est at the moment
	cormander	ahh! 5am utc
	cormander	too early?
	kragniz	I work on openstack, so I'm used to late night irc meetings!
	kragniz	so, more about the general scope of the project:
	cormander	I'll be as accommodating as I can - although I do get up at 
4am from time to time myself
	kragniz	should this be a system which is only designed to run on 
centos? anything rpm based? anything running linux?
	cormander	good questions,
	cormander	for now, only designed for EL7
	cormander	yes RPM based
	cormander	needs to be compatible with yum
	cormander	the API of what function(s) each of our kpatches use could be 
as simple as
	cormander	making an rpm "virtual" dependancy
	cormander	kernel-kpatch-function-X
	cormander	where X is the functions it touches
	kragniz	hmm, that would be interesting
	cormander	I forget what I'm thinking of is called
	cormander	where you specify this is the only package that can touch this
	cormander	installing another with it will get rejected
	kragniz	conflicts?
	cormander	yes!
	cormander	wow it's been a while
	kragniz	hehe
	cormander	there will be situations where two patches touch the same 
functions
	cormander	they need to get rolled together into an update for that 
particular patch
	cormander	so, occationally, an already released kpatch rpm will get an 
update
	cormander	but mostly, it's just installing new kpatch rpms
	kragniz	okay, so when that new package comes along, the other patches 
will be removed
	cormander	updated
	cormander	and by being updated, the old one will get rmmod'd, and 
re-insmod with the new one
	cormander	or however kpatch handles that
	kragniz	if it conflicts, shouldn't the whole conflicting package be 
uninstalled?
	cormander	if it conflicts, and it's from our repository, that means we 
didn't do a good QA job
	cormander	I'll explain it like this;
	cormander	we have patches A B C D E F G
	cormander	all touching different things
	cormander	patch H comes out, and touches what A does
	cormander	instead of releasing H as a new rpm
	cormander	we update A to include H
	kragniz	right, I get you now!
	cormander	then, patch J comes out, it touches what D and G do
	kragniz	I thought you meant bundle As patch with H and remove A
	cormander	we update D to include G and J, and have it "depricate" G
	cormander	or whatever term rpm uses for depricate ... where it 
uninstalls it upon install
	kragniz	yup, that makes sense
	cormander	and this, my friend, is where the QA matters most
	kragniz	indeed!
	cormander	like I said, if an rpm conflicts, we f***ed up
	kragniz	yup
	cormander	so, chew on that for now. I actually have to get up early 
tomorrow
	cormander	see you tomorrow around this time?
	kragniz	alrighty
	kragniz	sounds good!
	kragniz	thanks for the thoughts!
	cormander	goes "away" - if there is such a thing in IRC
	kragniz	goes to bed