[CentOS] Possibly OT: intermittently long response times when connecting to host

I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and
drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it -
2 hosts. The problem is that when I pull up a web page after not
having accessed it for ~15+ minutes it takes between 10-12s to load
fully. The 2nd page, 3rd page - and sometimes more than that - take
between 8-12 second. After that - decent performance, though not great
(0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes
back to slow. While it's been most apparent with httpd, I've noticed
similar slowness with the initial ssh session, etc.

I know what you're thinking. DNS problem, right? Yeah, me too. But for
the life of me I can't find a problem. This is at a colocation site
where we have other servers - all of those are just fine. We've got an
almost identical box internally - working just fine. I can do an
nslookup off of each of our different internal name servers and get
identical and immediate responses.

I've done tcpdumps (both on the client and the server). It shows the
request gets to the server immediately. I've started httpd using
strace and not seen anything glaringly wrong (though I'm not well
versed in the output I have enough C under my belt to have an idea
what's going on). Thinking it was code that the developer working on
the project I pulled all the custom drupal modules out and it's still
slow. Even created another virtual host with a stock drupal install
and it's slow (identical results). I moved the mysql server from local
to our central DB server. Load avg never gets above ~0.10 so I don't
think the box is stressed. Watching the ESX load average it looks good
too. I've tried a hell of a lot of other things and tried to include
all the relevant information so apologies in advance for anything I
might have left out.


Thanks,
Dave

Relevant info:

uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP
Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux

relevant portion of httpd.conf:
NameVirtualHost *:80

<VirtualHost *:80>
   DocumentRoot /path/to/drupal-current
   ServerName publicname.domainname.com
   ErrorLog /path/to/logserror_log
   CustomLog /path/to/logs/access_log common
</VirtualHost>

/etc/hosts:
127.0.0.1 localhost.localdomain localhost internal-name
12.34.56.78 publicname.domainname.com
12.34.56.79 db-server.domainname.com db-server