Is anyone on the list using kerberized-nfs on any kind of scale?
I've been fighting with this for years. In general, when we have
issues with this system, they are random and/or not repeatable. I've
had very little luck with community support. I hope I don't offend by
saying that! Rather, my belief is that these problems are very
niche/esoteric, and so beyond the scope of typical community support.
But I'd be delighted to be proven wrong!
So this is more of a "meta" question: anyone out there have any
general recommendations for how to get support on what I presume are
niche problems specific to our environment? How is paid upstream
support?
Just to give a little insight into our issues: we have an
in-house-developed compute job dispatching system. Say a user has
100s of analysis jobs he wants to run, he submits them to a central
master process, which in turn dispatches them to a "farm" of >100
compute nodes. All these nodes have two different krb5p NFS mounts,
to which the jobs will read and write. So while the users can
technically log in directly to the compute nodes, in practice they
never do. The logins are only "implicit" when the job dispatching
system does a behind-the-scenes ssh to kick off these processes.
Just to give some "flavor" to the kinds of issues we're facing, what
tends to crop up are one of three things:
(1) Random crashes. These are full-on kernel trace dumps followed
by an automatic reboot. This was really bad under CentOS 5. A random
kernel upgrade magically fixed it. It happens almost never under
CentOS 6. But happens fairly frequently under CentOS 7. (We're
completely off CentOS 5 now, BTW.)
(2) Permission denied issues. I have user Kerberos tickets
configured for 70 days. But there is clearly some kind of
undocumented kernel caching going on. Looking at the Kerberos server
logs, it looks like it "could" be a performance issue, as I see 100s
of ticket requests within the same second when someone tries to launch
a lot of jobs. Many of these will fail with "permission denied" but
if they immediately re-try, it works. Related to this, I have been
unable to figure out what creates and deletes the
/tmp/krb5cc_uid_random files.
(3) Kerberized NFS shares getting "stuck" for one or more users.
We have another monitoring app (in-house developed) that, among other
things, makes periodic checks of these NFS mounts. It does so by
forking and doing a simple "ls" command. This is to ensure that these
mounts are alive and well. Sometimes, the "ls" command gets stuck to
the point where it can't even be killed via "kill -9". Only a reboot
fixes it. But the mount is only stuck for the user running the
monitoring app. Or sometimes the monitoring app is fine, but an
actual user's processes will get stuck in "D" state (in top, means
waiting on IO), but everyone else's jobs (and access to the kerberizes
nfs shares) are OK.
This is actually blocking us from upgrading to CentOS 7. But my
colleagues and I are at a loss how to solve this. So this post is
really more of a semi-desperate plea for any kind of advice. What
other resources might we consider? Paid support is not out of the
question (within reason). Are there any "super specialist"
consultants out there who deal in Kerberized NFS?
Thanks!
Matt