I had a rather strange problem last week with one of our 8 core servers. The users complained the performance was "slow" so I checked the basic things, processes on top, vmstat for memory and context switching, i/o stats for internal disk I/O, netstat for any network issues and other things like network through put by copying a large file (1gb file across the network).
It turned out I had an NMI related issue on the processor. I figured this out by checking the /var/log/messages but it was a real mystery for be at first. My question, is there a way to detect or benchmark a system and all of its processors to make sure I don't bypass this type of error again? I am not necessary looking for monitoring tools but more of techniques like, run a while loop on all processors/cores to make sure they all give a constant time?
TIA