[CentOS-devel] openafs status?

Thu Aug 20 08:33:24 UTC 2015

Hi All,

Maintaining Openafs kmod for SLC{5,6} clients at work for the last 4 
years in production (and more recently for CentOS 7), I think the 
problem is not that easy to solve in an automated way for any kmod. 
(dkms is not a solution ; you don't want thousand of client to recompile 
code at each kernel upgrade and open tickets :)

I'll give you my take and how we do it for production.

There is 2 use cases :

1/ Upgrade Openafs to a new version.

Each time we want to upgrade openafs we recompile the kmod for the last 
3 kernel versions and simply update the client rpms. Why ? because you 
always have people that need/want/don't upgrade kernels in a large user 
base.
What if there is a security issue ? you need to protect most users.

There are many workaround in RH knowledge base that suggest to use older 
kernel.

We have a simple script which wrap around mock and allow us to do the 
build : 
https://github.com/alphacc/rpms-tools/blob/master/build/bsbuildmodules

-> It's a manual process we do launch when we have a new openafs version.

2/ Kernel upgrade to newer version.
- Rebuild openafs kmod for the newer kernel.

-> It's a manual process we do launch when we have a new kernel

* On top of that we need to be able to provide the kmod for each 
installed kernel, because user do switch between kernels sometimes. 
(again some issues take time to be fixed in the kernel...)

For this as already mentioned in past discussion, Troy Dawson wrote a 
yum plugin we still maintain : 
http://linuxsoft.cern.ch/cern/slc6X/SRPMS/yum-kernel-module-1-5.slc6.cern.src.rpm

* Using koji to build anything else (i.e kernels) we didn't find an easy 
way to integrate this process nicely yet. At CentOS we would have to 
deal with another build system on top of that.

* Whether the kmod could be built only for the first version of the 
minor release, we need to have feedback for Openafs developers, the 
uname being part of the kmod package name, some work will need to be 
done on the spec file.

At least this approach allow us to be kernel/openafs issue free (peace 
of mind and less tickets :) and is not impossible to automate.

thanks for listening,
Thomas.

On 08/19/2015 02:46 AM, Jason Edgecombe wrote:
> On 08/18/2015 01:28 PM, Ned Slider wrote:
>>
>> On 18/08/15 01:09, Jason Edgecombe wrote:
>>> On 08/15/2015 07:52 PM, Ned Slider wrote:
>>>> On 15/08/15 22:41, Jason Edgecombe wrote:
>>>>> On 08/15/2015 10:56 AM, Ned Slider wrote:
>>>>>> On 15/08/15 14:27, Jonathan Billings wrote:
>>>>>>> On Aug 15, 2015, at 9:10 AM, Jason Edgecombe
>>>>>>> <jason at rampaginggeek.com>  wrote:
>>>>>>>> What's the plan for fixing CBS to accommodate this? If I had a
>>>>>>>> spare
>>>>>>>> afternoon, what could I do to move this forward?
>>>>>>> Really, the end goal is to have the kmod-openafs packages built for
>>>>>>> every kernel release as it comes out, without removing the existing
>>>>>>> kmod-openafs from the repo.  This is why I separated it out from the
>>>>>>> openafs spec/srpm, because it will need to be rebuilt every time
>>>>>>> there is a new kernel.  However, OpenAFS users want to have
>>>>>>> historical kmods available.
>>>>>>>
>>>>>> Given that you really don't seemed to have moved forward in the last
>>>>>> year, I would reiterate what I said a year ago:
>>>>>>
>>>>>> http://lists.centos.org/pipermail/centos-devel/2014-September/012047.html
>>>>>>
>>>>>>
>>>>>>
>>>>>> I really don't understand why you would not want to use the framework
>>>>>> Red Hat provides to solve exactly this problem. You are trying to
>>>>>> solve
>>>>>> a problem that is largely already solved.
>>>>>>
>>>>>> Worst case scenario is you'd need to rebuild kmod-openafs for major
>>>>>> point releases every 9-12 months (e.g, 6.5, 6.6, 6.7; and/or 7.0,
>>>>>> 7.1,
>>>>>> ...) where the kABI is broken for non-whitelisted symbols used by
>>>>>> OpenAFS. That has to be a preferable solution for OpenAFS users than
>>>>>> rebuilding against each and every kernel release.
>>>>>>
>>>>> Does yum check and prevent or warn you when installing a kABI
>>>>> module for
>>>>> a non-matching kernel? How do I know which kmods match my kernel?
>>>>>
>>>> No, yum will only warn on symbols that may be missing, for example on
>>>> older kernels.
>>>>
>>>> Look to see if the module weak links against the kernel in the
>>>> corresponding weak-updates directory.
>>>>
>>>> Sometime a module can weak link but still not be compatible but it's
>>>> uncommon and only likely to happen when going from one point release to
>>>> another (e.g, 6.6 to 6.7) when Red Hat have changed the kABI of a
>>>> non-whitelisted symbol used by your module. Pre-release testing will
>>>> pick this up. Red Hat generally do not change the kABI within a point
>>>> release.
>>> A few folks in the OpenAFS community have been burned by the kABI and
>>> don't trust it, even for point RH releases. One gentleman was even
>>> burned by elrepo.
>>>
>> People tend to not trust what they don't understand. kABI does exactly
>> what it says on the tin - nothing more, nothing less. IMHO "burned"
>> seems a bit strong but lets look at that.
>>
>> People tend to get "burned" when their expectations do not aline with
>> reality. In reality kABI tracking kmods work seamlessly when the driver
>> uses only symbols on the whitelist (and in my experience that is
>> actually very few to none).
>>
>> As I understand, OpenAFS uses some non-whitelisted symbols so the
>> reality is that openafs may break at point releases. It should work
>> seamlessly for kernel updates between point releases. In other (your)
>> words, you *may* get burned at a point release.
>>
>> Compare that to compiling against every single kernel release. That
>> guarantees the existing module(s) will break for every new kernel
>> release, so now instead of getting "burned" every 9 months at a point
>> release, users now get "burned" at every kernel update.
>>
>> Lets look at an example. Every couple weeks RH release a kernel update
>> but I can't update because my matching kmod-openafs isn't available yet.
>> First I have to wait for CentOS to rebuild it and release the kernel
>> update (lag). Then I have to wait for the SIG to rebuild their
>> kmod-openafs against the newly built CentOS kernel (more lag). If I
>> accidentally update my kernel and reboot - poof, I'm burned. Every time,
>> guaranteed. If I don't update my kernel then compliance are on my case
>> because I'm vulnerable to the latest CVE.
>>
>> Personally, if I'm gonna get burned I'd rather be burned every 9 months
>> that every single time upstream releases a new kernel update.
>>
>> Of course in the real world, where people are using filesystem drivers
>> in production, I'm sure they test any new releases before they put them
>> into production so they never actually get burned. What they do get is a
>> heads up when there is an issue, they file a bug, the driver gets
>> rebuilt, the issue is solved and the bug is closed. It's that simple.
>>
>> I don't use OpenAFS so I really don't care if you build against every
>> kernel or use kABI tracking kmods. As I've said before, there is no
>> precedent in RHEL to build kmods against each kernel - kABI tracking
>> kmods is the RHEL way. Given that you don't seem to have progressed from
>> a year ago and kABI tracking kmods do not suffer from the problem you
>> have made for yourselves, it just seemed like the obvious solution. A
>> few folks in the OpenAFS community having a bad experience doesn't seem
>> like a valid reason for throwing out the established way of doing things
>> and adopting something that has the potential to cause far more issues
>> and has no precedence in the RHEL ecosystem. If you want to deviate from
>> the established method then I really think you need to (1) present a
>> stronger case why you believe it doesn't work, and (2) present a viable
>> alternative that significantly improves on the currently established
>> methods.
>>
>> TBH it all seems fairly irrelevant now as I would imagine any users
>> waiting for a solution are long gone by now.
> Is there a yum repo or easy way to download RPMs from CBS if I want to
> try them for myself?
>
> I'm just an OpenAFS community member, not one of the main packagers. It
> seems that I'm hard-pressed to convince either side to switch.
>
> Jason
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> http://lists.centos.org/mailman/listinfo/centos-devel