[CentOS] Strange performance issue on CentOS 6.7 server

Wed Feb 3 20:30:21 UTC 2016
Alfred von Campe <alfred at von-campe.com>

I’m running CentOS 6.7 on my build servers, and on one of the servers the builds are taking almost an order of magnitude longer than usual.  There are no runaway processes and there is plenty of free memory.  So I suspected that file I/O might be slow, and sure enough, that appears to be the case.  I ran a simple dd test and compared the results to a “normal” build server (412 MB/s vs. 31.7 MB/s).  Both are on similarly configured VGs on identical hardware.  What could cause this performance degradation and are there any other tests I can run before I reboot the server to see if things improve?  There is nothing in /var/log messages - are there other logs I should check?