[CentOS] system slowdown

Thu May 18 16:42:41 UTC 2017
Larry Martell <larry.martell at gmail.com>

I have a CentOS 6 system that has suddenly starting significantly
slowing down. It runs a django app with an Apache server and MySQL
server. There is plenty of disk space and no process seems to be
hogging the memory or CPU. But operations that used to take 5 minutes
are now taking hours and hours.

Coinciding with this slow down I see these messages in
/var/log/messages (this is hand typed, as I can't copy/paste or send
email from this system, although for a change I do have remote access
via 2 VPN hops):

kernel: BUG: soft lockup - CPU#5 stuck for 67s! [khungtaskd]
kernel: BUG: soft lockup - CPU#7 stuck for 67s! [khugepaged]

These messages started appearing today around 7 AM, which is when
users started reporting the slowdown. They are still occurring
periodically and the system is still slow. Are these messages caused
by the slowdown or are they the reason for it? Are they useful in
tracking down the problem?