I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it - 2 hosts. The problem is that when I pull up a web page after not having accessed it for ~15+ minutes it takes between 10-12s to load fully. The 2nd page, 3rd page - and sometimes more than that - take between 8-12 second. After that - decent performance, though not great (0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes back to slow. While it's been most apparent with httpd, I've noticed similar slowness with the initial ssh session, etc.
I know what you're thinking. DNS problem, right? Yeah, me too. But for the life of me I can't find a problem. This is at a colocation site where we have other servers - all of those are just fine. We've got an almost identical box internally - working just fine. I can do an nslookup off of each of our different internal name servers and get identical and immediate responses.
I've done tcpdumps (both on the client and the server). It shows the request gets to the server immediately. I've started httpd using strace and not seen anything glaringly wrong (though I'm not well versed in the output I have enough C under my belt to have an idea what's going on). Thinking it was code that the developer working on the project I pulled all the custom drupal modules out and it's still slow. Even created another virtual host with a stock drupal install and it's slow (identical results). I moved the mysql server from local to our central DB server. Load avg never gets above ~0.10 so I don't think the box is stressed. Watching the ESX load average it looks good too. I've tried a hell of a lot of other things and tried to include all the relevant information so apologies in advance for anything I might have left out.
Thanks, Dave
Relevant info:
uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux
relevant portion of httpd.conf: NameVirtualHost *:80
<VirtualHost *:80> DocumentRoot /path/to/drupal-current ServerName publicname.domainname.com ErrorLog /path/to/logserror_log CustomLog /path/to/logs/access_log common </VirtualHost>
/etc/hosts: 127.0.0.1 localhost.localdomain localhost internal-name 12.34.56.78 publicname.domainname.com 12.34.56.79 db-server.domainname.com db-server
On Tue, 2008-10-28 at 16:30 -0500, David Miller wrote:
I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it - 2 hosts. The problem is that when I pull up a web page after not having accessed it for ~15+ minutes it takes between 10-12s to load fully. The 2nd page, 3rd page - and sometimes more than that - take between 8-12 second. After that - decent performance, though not great (0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes back to slow. While it's been most apparent with httpd, I've noticed similar slowness with the initial ssh session, etc.
I know what you're thinking. DNS problem, right? Yeah, me too. But for the life of me I can't find a problem. This is at a colocation site where we have other servers - all of those are just fine. We've got an almost identical box internally - working just fine. I can do an nslookup off of each of our different internal name servers and get identical and immediate responses.
I've done tcpdumps (both on the client and the server). It shows the request gets to the server immediately. I've started httpd using strace and not seen anything glaringly wrong (though I'm not well versed in the output I have enough C under my belt to have an idea what's going on). Thinking it was code that the developer working on the project I pulled all the custom drupal modules out and it's still slow. Even created another virtual host with a stock drupal install and it's slow (identical results). I moved the mysql server from local to our central DB server. Load avg never gets above ~0.10 so I don't think the box is stressed. Watching the ESX load average it looks good too. I've tried a hell of a lot of other things and tried to include all the relevant information so apologies in advance for anything I might have left out.
Thanks, Dave
Relevant info:
uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux
relevant portion of httpd.conf: NameVirtualHost *:80
<VirtualHost *:80> DocumentRoot /path/to/drupal-current ServerName publicname.domainname.com ErrorLog /path/to/logserror_log CustomLog /path/to/logs/access_log common
</VirtualHost>
/etc/hosts: 127.0.0.1 localhost.localdomain localhost internal-name 12.34.56.78 publicname.domainname.com 12.34.56.79 db-server.domainname.com db-server
---- sounds sleepy - perhaps the mysql connections drop after inactivity and then it takes that long to connect
You might just want to set up a cron script that does a curl or wget of a standard page every 2 minutes or so just to keep things active.
Craig
--snip--
sounds sleepy - perhaps the mysql connections drop after inactivity and then it takes that long to connect
You might just want to set up a cron script that does a curl or wget of a standard page every 2 minutes or so just to keep things active.
Craig
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
hmmm.... you may be onto something there, Craig. Since it was the same environment (we set up lots of LAMP servers and use a common MySQL backend) I gave it the benefit of the doubt when I was initially troubleshooting (the connection was there and everything looked sane). I'll take another look at that.
Thanks. Dave
David Miller wrote:
--snip--
sounds sleepy - perhaps the mysql connections drop after inactivity and then it takes that long to connect
You might just want to set up a cron script that does a curl or wget of a standard page every 2 minutes or so just to keep things active.
Craig
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
hmmm.... you may be onto something there, Craig. Since it was the same environment (we set up lots of LAMP servers and use a common MySQL backend) I gave it the benefit of the doubt when I was initially troubleshooting (the connection was there and everything looked sane). I'll take another look at that.
Also, take a good look at the logs /var/log/messages and dmesg for hard drive errors. Lots of times a bad drive will cause controller bus resets that happen intermittently ... that can cause a 5-10 second delay and then things seem to work fine till the next time.
--snip--
Also, take a good look at the logs /var/log/messages and dmesg for hard drive errors. Lots of times a bad drive will cause controller bus resets that happen intermittently ... that can cause a 5-10 second delay and then things seem to work fine till the next time.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I should have mentioned that.... /var/log/messages, /var/log/secure, /var/log/dmesg, /var/log/audit/audit.log.... well, any log I could think of - none show anything out of the ordinary.
David Miller wrote:
--snip--
sounds sleepy - perhaps the mysql connections drop after inactivity and then it takes that long to connect
You might just want to set up a cron script that does a curl or wget of a standard page every 2 minutes or so just to keep things active.
Craig
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
hmmm.... you may be onto something there, Craig. Since it was the same environment (we set up lots of LAMP servers and use a common MySQL backend) I gave it the benefit of the doubt when I was initially troubleshooting (the connection was there and everything looked sane). I'll take another look at that.
Also, make sure that this is not set in apache (the default is Off ... but things like that can take massive amounts of time):
HostnameLookups On
--snip--
sounds sleepy - perhaps the mysql connections drop after inactivity and then it takes that long to connect
You might just want to set up a cron script that does a curl or wget of a standard page every 2 minutes or so just to keep things active.
Craig
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
The server hosting the web page looks to be creating the connecting immediately to the MySQL server (verified by mytop and tcpdump). Troubleshooting MySQL connections isn't my forte - to now the connections have always "just worked" so there hasn't been much of a need. Another thing that has led me away from thinking database problem is that this existed when the database was hosted locally and when it was on our production database server.
Sorry for the delay in posting. 2-3 attempts before the problem "resolves" itself makes troubleshooting slow.
Thanks to all that have replied.
Dave
-
Craig White -
David Miller -
Johnny Hughes