[CentOS] CentOS 5.x 32bit VPS Ping Issue

I just got an inexpensive VPS too have an outside server to play/test
with.  I am pinging it every 5 minutes and graphing with MRTG on
another CentOS box.  This works fine to all servers but the VPS.  The
first ping to the VPS is always crud and following ones are fine.

[root at ns1 scripts]# ping X
PING X 56(84) bytes of data.
64 bytes from X: icmp_seq=1 ttl=55 time=773 ms
64 bytes from X: icmp_seq=2 ttl=55 time=42.4 ms
64 bytes from X: icmp_seq=3 ttl=55 time=42.8 ms
64 bytes from X: icmp_seq=4 ttl=55 time=43.1 ms

Makes my graph for that one look real bad because it looks at first
ping only.  Thought about modifying my script to send a warm up ping
first but I just don't think I should have to do that or is this a
normal thing for a VPS that is pretty much idle?