[CentOS] Server packet loss and becomes unresponsive

I am at a lost with this server of mine.

I have replaced everything except just replacing the entire server.  The
server is running at a low load but every so often it starts to have high
packet loss / latency(the average ms for me is 80 but it jumps to 4000
during this period) and eventually becomes unresponsive.

All traffic seems normal, no unusual activity. But once or twice a month it
starts to get packet loss and after an hour or two it crashes.

The network itself is fine. All other servers on the same subnet work fine.

Any ideas what could be causing this kind of behavior?

Thanks,

Tim