[CentOS] weird slowdowns with recent CentOS 6 update

I just spent a while trying to debug some weird slowdown issues, and I thought I’d let the mailing list know in case it helps anyone else, or someone has a cleaner solution (or better ways of debugging the real issue).

Just by way of context, we have a cluster using Infiniband and CentOS 6 (up to the latest patches as of a couple weeks ago, so 6.9+anything newer) and OpenMPI 3.1.0.  Most of the nodes are "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz” or "Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz”, currently with kernel .6.32-696.30.1.el6.x86_64

Recently (I believe when we upgraded to that kernel, but not 100% sure that's really when it started), certain parallel floating-point-compute intensive jobs would sometimes (maybe 1 time in 20) run very slowly.  The overall speed was reduced by about a factor of 2, but the slowdown appeared to be localized to specific routines, which would go from taking ~1s to ~50-100s.  I first assumed it had to do with communication, but as I made the timing information more fine grained, it became clear that nearly trivial loops like
      DO I=1,GRIDC%RL%NP
        DWORK1(I,ISP)= REAL( DWORK(I,ISP) ,KIND=q)
      ENDDO
were a large part of the problem.  The only thing I could find after poking around that was at all odd is that when this happened, khugepaged would be taking 100% of cpu for tens of seconds on the affected nodes.  It appears, although it’s not completely certain yet (since the problem appears rarely enough to that it’s hard to definitively rule out), that disabling THP by putting “never” in /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/defrag fixes the problem. 

I guess I am also still wondering what exactly khugepaged does, and what kind of problems I should worry about because I disabled it.  The documentation I’ve found for it doesn’t really explain what the transparent huge pages are good for.

									Noam