Hi!
I have a really weird problem with some of my servers, namely all the ones running Centos5
When I try to download a file from the server to a machine outside our Cisco 6500 router/firewall, the download hangs about half the times (15 out of 40) when less than half a megabyte into the transfer (varied from 76 kb to 496 kb).
One server has a portchannel (Cisco speak for ethernet bundle), others do not, some use e1000, some use broadcom, I've tried httpd and scp transfers, I've tried from three different clients,
Here's the really annoying part: the problem only occurs when downloading from outside the firewall, not when transferring files internally in the serverroom! And the switch is the firewall is a modular chassis, so the data comes over the same backplane regardless.
And there's 42 files in /proc/net/sys/ipv4 which differ between EL4 and Centos5, so I'm a little lost here
Anybody got some ideas?
with kind regards, Bent
PS Crossposted from Centos Forum, in case sombody on the list, who doesn't read the forum, can help me
I have a really weird problem with some of my servers, namely all the ones running Centos5
When I try to download a file from the server to a machine outside our Cisco 6500 router/firewall, the download hangs about half the times (15 out of 40) when less than half a megabyte into the transfer (varied from 76 kb to 496 kb).
One server has a portchannel (Cisco speak for ethernet bundle), others do not, some use e1000, some use broadcom, I've tried httpd and scp transfers, I've tried from three different clients,
Here's the really annoying part: the problem only occurs when downloading from outside the firewall, not when transferring files internally in the serverroom! And the switch is the firewall is a modular chassis, so the data comes over the same backplane regardless.
And there's 42 files in /proc/net/sys/ipv4 which differ between EL4 and Centos5, so I'm a little lost here
I don't know if this would help, but I remember reading a post about a problem with tcp window scaling. To disable tcp window scaling add the line
net.ipv4.tcp_window_scaling = 0
to /etc/sysctl.conf and then run sysctl -p to activate your change.
Barry
I have a really weird problem with some of my servers, namely all the ones running Centos5
When I try to download a file from the server to a machine outside our Cisco 6500 router/firewall, the download hangs about half the times (15 out of 40) when less than half a megabyte into the transfer (varied from 76 kb to 496 kb).
One server has a portchannel (Cisco speak for ethernet bundle), others do not, some use e1000, some use broadcom, I've tried httpd and scp transfers, I've tried from three different clients,
Here's the really annoying part: the problem only occurs when downloading from outside the firewall, not when transferring files internally in the serverroom! And the switch is the firewall is a modular chassis, so the data comes over the same backplane regardless.
And there's 42 files in /proc/net/sys/ipv4 which differ between EL4 and Centos5, so I'm a little lost here
Anybody got some ideas?
Just to state the obvious (well, obvious to me), which you don't seem to have mentioned above: The filtering part of the 6500 is dropping the traffic, and is dropping it because of something that Centos 5 is doing differently from EL4. I think there was a post to this list just last week about something similar; I don't have time to search the archive, but it is something to do with a TCP option/extension which is on in Centos 5, but can be turned off via a setting in /proc somewhere; the extension should be acceptable to all firewalls/routers (uses a previously unused few bits in the TCP header), but some decide it's not valid and drop packets/connections.
Craig Miskell ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
Miskell, Craig wrote:
I have a really weird problem with some of my servers, namely all the ones running Centos5
When I try to download a file from the server to a machine outside our Cisco 6500 router/firewall, the download hangs about half the times (15 out of 40) when less than half a megabyte into the transfer (varied from 76 kb to 496 kb).
One server has a portchannel (Cisco speak for ethernet bundle), others do not, some use e1000, some use broadcom, I've tried httpd and scp transfers, I've tried from three different clients,
Here's the really annoying part: the problem only occurs when downloading from outside the firewall, not when transferring files internally in the serverroom! And the switch is the firewall is a modular chassis, so the data comes over the same backplane regardless.
And there's 42 files in /proc/net/sys/ipv4 which differ between EL4 and Centos5, so I'm a little lost here
Anybody got some ideas?
Just to state the obvious (well, obvious to me), which you don't seem to have mentioned above: The filtering part of the 6500 is dropping the traffic, and is dropping it because of something that Centos 5 is doing differently from EL4. I think there was a post to this list just last week about something similar; I don't have time to search the archive, but it is something to do with a TCP option/extension which is on in Centos 5, but can be turned off via a setting in /proc somewhere; the extension should be acceptable to all firewalls/routers (uses a previously unused few bits in the TCP header), but some decide it's not valid and drop packets/connections.
Craig Miskell
<snip>
You might be thinking of the thread that climaxed about here: http://marc.info/?l=centos&m=119033374928629&w=2 The entire thread makes interesting reading, esp as a post-mortem.
Something like that probably, but apparently NOT the tcp window scaling. Aty least changing it doesn't help :-(
I'm gonna try do some sniffing while downloading, as the regular error logs don't show anything.
/B
The only thing which shows up is that the client start sending duplicate ACK's, getting "Destination unreachable" as reply from the server (not from the Cisco). This happened 220 KB into the transfer in this case, but that figure varies quite a bit.
As it turns out, the problem goes away if I use old-fashioned iptables, that is without connection tracking. Go figure!
Take home lesson is do not use connection tracking iptables behind a Cisco FireWall Service Module.
Is this just to be accepted as canon, or can somebody actually explain to me WHY?
best regards, Bent
On 10/8/07, Bent Terp bent@nagstrup.dk wrote:
The only thing which shows up is that the client start sending duplicate ACK's, getting "Destination unreachable" as reply from the server (not from the Cisco). This happened 220 KB into the transfer in this case, but that figure varies quite a bit.
-
Barry Brimer -
Bent Terp -
Miskell, Craig -
Robert