I’ve held off reporting this since I thought it might just be some kind of fluke, but I’ve seen it now on three different boxes.
The symptom is that the stock configuration of Apache only listens for IPv6 connections:
$ netstat -na | grep :80.*LISTEN tcp6 0 0 :::80 :::* LISTEN
You should see a second line there for IPv4, but you don't:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
The stock httpd.conf file just says “Listen 80” which is documented as listening on both IPv4 and IPv6. [1] You’re supposed to need to go out of your way to get it to listen on just one or the other, but somehow CentOS 7’s Apache manages it.
Since I only need IPv4, I’ve managed to hack it into working by changing that line in /etc/httpd/conf/httpd.conf to:
Listen 0.0.0.0:80
Why do I need to do this?
[1] https://httpd.apache.org/docs/2.4/mod/mpm_common.html#listen
Am 11.12.2014 um 04:48 schrieb Warren Young:
I’ve held off reporting this since I thought it might just be some kind of fluke, but I’ve seen it now on three different boxes.
The symptom is that the stock configuration of Apache only listens for IPv6 connections:
$ netstat -na | grep :80.*LISTEN tcp6 0 0 :::80 :::* LISTEN
No, that's just the way it is displayed for apache. In fact the service listens on IPv4 as well (given we speak about the default configuration with `Listen 80').
Easy to verify.
You should see a second line there for IPv4, but you don't:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
The stock httpd.conf file just says “Listen 80” which is documented as listening on both IPv4 and IPv6. [1] You’re supposed to need to go out of your way to get it to listen on just one or the other, but somehow CentOS 7’s Apache manages it.
Since I only need IPv4, I’ve managed to hack it into working by changing that line in /etc/httpd/conf/httpd.conf to:
Listen 0.0.0.0:80
Why do I need to do this?
[1] https://httpd.apache.org/docs/2.4/mod/mpm_common.html#listen
Alexander
On Dec 11, 2014, at 3:10 AM, Alexander Dalloz ad+lists@uni-x.org wrote:
Am 11.12.2014 um 04:48 schrieb Warren Young:
the stock configuration of Apache only listens for IPv6 connections:
No, that's just the way it is displayed for apache. In fact the service listens on IPv4 as well (given we speak about the default configuration with `Listen 80').
Easy to verify.
We noticed this problem when web browsers would refuse to connect to the server. *Then* we discovered the netstat oddity, and *then* we found that changing the Listen line in httpd.conf fixed it.
That leaves me still wanting an explanation for what happened.
New guess: there’s a difference between the IPv4 and v6 firewalls, so that changing the Listen line caused Apache to avoid the problem on the v6 side.
I guess we’ll just keep an eye out when we build our next EL7 box...
On 12/11/2014 09:35 AM, Warren Young wrote:
Am 11.12.2014 um 04:48 schrieb Warren Young:
the stock configuration of Apache only listens for IPv6 connections:
As per RFC 3493 (Sections 3.7 and 5.3) an IPv6 socket will accept connections from IPv4 hosts, which will be mapped into the IPv6 address space.
We noticed this problem when web browsers would refuse to connect to the server. *Then* we discovered the netstat oddity, and *then* we found that changing the Listen line in httpd.conf fixed it.
That leaves me still wanting an explanation for what happened.
New guess: there’s a difference between the IPv4 and v6 firewalls, so that changing the Listen line caused Apache to avoid the problem on the v6 side.
I don't have a good guess, there. If the client was actually connecting from an IPv4 address, then the IPv4 firewall rules should have applied. At least, that's what my testing indicates.
What I can say for sure is that a CentOS 7 system will accept IPv4 http connections if the IPv4 firewall allows that port, or if the firewall is disabled. Whatever problem you faced was caused by post-install configuration. Try making your standard changes incrementally and testing as you go until you can locate the steps where the problem occurs.
On Thu, Dec 11, 2014 at 12:35 PM, Warren Young wyml@etr-usa.com wrote:
We noticed this problem when web browsers would refuse to connect to the server. *Then* we discovered the netstat oddity, and *then* we found that changing the Listen line in httpd.conf fixed it.
That leaves me still wanting an explanation for what happened.
was the fix the config change or restarting of the service? did netstat show client connections to port 80 in other states(FIN_WAIT/etc)? high load apache websites can often run out of connections due to the limits in httpd.conf "<IfModule prefork.c>" section. these limits can be increased if clients can no longer connect due to http child processes being stuck in other tcp states.
On Dec 12, 2014, at 6:32 AM, Steven Tardy sjt5atra@gmail.com wrote:
On Thu, Dec 11, 2014 at 12:35 PM, Warren Young wyml@etr-usa.com wrote:
We noticed this problem when web browsers would refuse to connect to the server. *Then* we discovered the netstat oddity, and *then* we found that changing the Listen line in httpd.conf fixed it.
was the fix the config change or restarting of the service?
I'm going to need to see this happen on the next server build to test for such details.
The only reason I restarted it is because that’s the only way to be certain the configuration changes take effect. Theoretically, “service reload” is supposed to be equivalent to “service restart”, but too many times I’ve run into services that don’t reload *everything*. I got out of the habit of using “reload” years ago.
As a result of your question, I realized that systemd might open up yet another new possibility for confusion in CentOS 7. What if the service is down from the start?
In CentOS 6 and earlier, “restart” on a downed service typically gets you an error, since the init script usually implements this in terms of “stop-then-start”, and the “stop” step fails.
CentOS 7 is a new ballgame, though, because systemd can just figure out what you meant, and silently start it. I tested this, and my guess was correct: this is exactly systemd does. Grrr.
I’ve already ruled that out as the problem in this particular case with the netstat test. I wouldn’t have seen the IPv6 port 80 listener if it was down from the start.
Nevertheless, it’s a good thing to keep in mind going forward.
did netstat show client connections to port 80 in other states(FIN_WAIT/etc)?
Since the problem servers never completed one single connection until I modified the Listen line, I don’t see how any WAIT states could occur, much less still be in the process of timing out.
high load apache websites can often run out of connections
These are freshly-built boxes on a quiet private LAN, being tested with single connections by a single user. They are not running out of resources.
-
Alexander Dalloz -
Gordon Messmer -
Steven Tardy -
Warren Young