I have a strange problem, where some clients see the website on my server and some do not. It is not about the iptables, and seems to be not about tcp wrapper. Still it is something within the box.
More details: - the problem is only with some clients, with no geographical connection between them; other clients see the website just fine - the problem-clients get timeout with their browser - they get timeout also when they try a numerical ip address - but they see another machine in the same subnet just fine (when they browse by ip number), so the problem has to be inside this webserver box, right? - port 80 (not ssl)
Switching off iptables does not help. The files hosts.allow and hosts.deny are empty, so I guess it's not the tcp wrapper.
I am out of things to test. Any ideas?
- Jussi Hirvi
Is one of your dns servers broken?
On Thu, May 06, 2010 at 09:31:22PM +0300, Jussi Hirvi wrote:
I have a strange problem, where some clients see the website on my server and some do not. It is not about the iptables, and seems to be not about tcp wrapper. Still it is something within the box.
More details:
- the problem is only with some clients, with no geographical connection
between them; other clients see the website just fine
- the problem-clients get timeout with their browser
- they get timeout also when they try a numerical ip address
- but they see another machine in the same subnet just fine (when they
browse by ip number), so the problem has to be inside this webserver box, right?
- port 80 (not ssl)
Switching off iptables does not help. The files hosts.allow and hosts.deny are empty, so I guess it's not the tcp wrapper.
I am out of things to test. Any ideas?
- Jussi Hirvi
On 5/6/2010 2:35 PM, Gavin Carr wrote:
Is one of your dns servers broken?
On Thu, May 06, 2010 at 09:31:22PM +0300, Jussi Hirvi wrote:
I have a strange problem, where some clients see the website on my server and some do not. It is not about the iptables, and seems to be not about tcp wrapper. Still it is something within the box.
More details:
- the problem is only with some clients, with no geographical connection
between them; other clients see the website just fine
- the problem-clients get timeout with their browser
*- they get timeout also when they try a numerical ip address*
- but they see another machine in the same subnet just fine (when they
browse by ip number), so the problem has to be inside this webserver box, right?
- port 80 (not ssl)
Switching off iptables does not help. The files hosts.allow and hosts.deny are empty, so I guess it's not the tcp wrapper.
Notice the op posted they get timeouts even when going directly to a numerical address (if the apache server is configured to respond to *:80 it should at least display something)
Try using telnet from a client machine that can not connect.
e.g. telnet host.name.here 80
or
telnet xx.xxx.xxx.xxx 80
Try a few times and see if you're getting a timeout or if it connects every time. Run tcpdump on the apache server while sending the connection requests and see if the connection attempts show up at all. If they do not, then it's a network problem.
On 05/06/2010 11:42 AM, Ryan Manikowski wrote:
Notice the op posted they get timeouts even when going directly to a numerical address (if the apache server is configured to respond to *:80 it should at least display something)
Try using telnet from a client machine that can not connect.
e.g. telnet host.name.here 80
or
telnet xx.xxx.xxx.xxx 80
Try a few times and see if you're getting a timeout or if it connects every time. Run tcpdump on the apache server while sending the connection requests and see if the connection attempts show up at all. If they do not, then it's a network problem.
Try running 'ab' (the apache bench tool - see 'man ab' for how to use it) against your server and see if you can provoke the timeouts. If you can, then you are probably not configured to handle many quick connections and should check (1) httpd.conf to make sure you don't have an excessively low setting for 'MaxClients' or (2) a too low setting for max open filehandles. Look in /etc/security/limits.conf - you should have a line reading something similar to:
* - nofile 64000
somewhere in it to raise the max number of open files. Busy web servers need lots of filehandles.
Ok, thanks for ideas - many new things to test. So far no luck.
Too bad i don't have first-hand access to any of the client machines who *do* have this problem.
Next, I will go and switch the ethernet cable to a different slot on the router - kind of desperate, I know.
Some more details: - this web server is a xen virtual guest system, with CentOS 5.4 - the problem surfaced yesterday morning (6th of May), after I had migrated all these web sites from an old Fedora box to this new CentOS system
Does the problem affect other xen systems on the same box? I haven't tested this yet (I cannot reproduce the error).
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
- Jussi
On 6.5.2010 22.00, Benjamin Franz wrote:
On 05/06/2010 11:42 AM, Ryan Manikowski wrote:
Notice the op posted they get timeouts even when going directly to a numerical address (if the apache server is configured to respond to *:80 it should at least display something)
Try using telnet from a client machine that can not connect.
e.g. telnet host.name.here 80
or
telnet xx.xxx.xxx.xxx 80
Try a few times and see if you're getting a timeout or if it connects every time. Run tcpdump on the apache server while sending the connection requests and see if the connection attempts show up at all. If they do not, then it's a network problem.
Try running 'ab' (the apache bench tool - see 'man ab' for how to use it) against your server and see if you can provoke the timeouts. If you can, then you are probably not configured to handle many quick connections and should check (1) httpd.conf to make sure you don't have an excessively low setting for 'MaxClients' or (2) a too low setting for max open filehandles. Look in /etc/security/limits.conf - you should have a line reading something similar to:
- nofile 64000
somewhere in it to raise the max number of open files. Busy web servers need lots of filehandles.
-- Benjamin Franz
-- Benjamin Franz
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Le Fri, 07 May 2010 07:38:45 +0300, Jussi Hirvi listmember@greenspot.fi a écrit :
... You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
It is the case from 147.99.7.1, and not only for port 80 :
$ ping -c 10 62.236.221.71 PING 62.236.221.71 (62.236.221.71) 56(84) bytes of data.
--- 62.236.221.71 ping statistics --- 10 packets transmitted, 0 received, 100% packet loss, time 8998ms
$ ping -c 1 62.236.221.78 PING 62.236.221.78 (62.236.221.78) 56(84) bytes of data. 64 bytes from 62.236.221.78: icmp_seq=1 ttl=46 time=58.9 ms
--- 62.236.221.78 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 58.975/58.975/58.975/0.000 ms
Hi,
Philippe Naudin sent a missive on 2010-05-07:
Le Fri, 07 May 2010 07:38:45 +0300, Jussi Hirvi a écrit :
... You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
It is the case from 147.99.7.1, and not only for port 80 :
$ ping -c 10 62.236.221.71 PING 62.236.221.71 (62.236.221.71) 56(84) bytes of data.
--- 62.236.221.71 ping statistics --- 10 packets transmitted, 0 received, 100% packet loss, time 8998ms
$ ping -c 1 62.236.221.78 PING 62.236.221.78 (62.236.221.78) 56(84) bytes of data. 64 bytes from 62.236.221.78: icmp_seq=1 ttl=46 time=58.9 ms
--- 62.236.221.78 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 58.975/58.975/58.975/0.000 ms
Can you confirm the routing on the two boxes - is there anything different? I would also check the routing on the upstream routers - it is possible that one of your ingress/egress routers has a static entry that is causing issues. I would check all the routers that are inside the 62.236.0.0/15 subnet (BGP thinks that these addresses are part of that subnet).
Simon.
Le Fri, 7 May 2010 09:01:17 +0100, "Simon Billis" simon@houxou.com a écrit :
Can you confirm the routing on the two boxes - is there anything different? I would also check the routing on the upstream routers - it is possible that one of your ingress/egress routers has a static entry that is causing issues. I would check all the routers that are inside the 62.236.0.0/15 subnet (BGP thinks that these addresses are part of that subnet).
$ traceroute -T 62.236.221.71 traceroute to 62.236.221.71 (62.236.221.71), 30 hops max, 40 byte packets 1 cc-campus.supagro.inra.fr (147.99.0.20) 0.231 ms 0.186 ms 0.185 ms 2 cc-dmz1.supagro.inra.fr (147.99.75.1) 0.406 ms 0.392 ms 0.373 ms 3 (195.220.89.181) 22.530 ms 22.517 ms 22.843 ms 4 193.51.241.145 (193.51.241.145) 6.910 ms 6.806 ms 7.637 ms 5 * * * 6 te1-2-marseille-rtr-021.noc.renater.fr (193.51.189.21) 9.527 ms 9.756 ms 9.976 ms 7 te0-0-0-0-lyon1-rtr-001.noc.renater.fr (193.51.189.17) 10.801 ms 10.786 ms 10.767 ms 8 xe-8-0-0.edge5.Paris1.Level3.net (212.73.207.173) 18.686 ms 17.010 ms 16.981 ms 9 ae-33-51.ebr1.Paris1.Level3.net (4.69.139.193) 16.548 ms 20.324 ms 20.076 ms 10 ae-47-47.ebr1.London1.Level3.net (4.69.143.109) 22.232 ms ae-48-48.ebr1.London1.Level3.net (4.69.143.113) 22.659 ms 22.723 ms 11 ae-1-51.edge3.London1.Level3.net (4.69.139.73) 22.949 ms 22.260 ms 22.547 ms 12 tdcdenmark-level3-xe.london1.Level3.net (4.68.63.90) 22.949 ms 22.611 ms 22.695 ms 13 atm1-0-5.psl-gw3.hel.fi.ip.tdc.net (62.236.1.26) 55.654 ms 55.624 ms 55.806 ms 14 proequal-cpe1.hel.fi.sn.net (62.236.27.110) 70.389 ms 71.992 ms 69.084 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
$ traceroute -T 62.236.221.78 traceroute to 62.236.221.78 (62.236.221.78), 30 hops max, 40 byte packets 1 cc-campus.supagro.inra.fr (147.99.0.20) 0.256 ms 0.185 ms 0.182 ms 2 cc-dmz1.supagro.inra.fr (147.99.75.1) 0.283 ms 0.267 ms 0.256 ms 3 (195.220.89.181) 1150.194 ms 1150.189 ms 1150.165 ms 4 193.51.241.145 (193.51.241.145) 1.050 ms 0.947 ms 0.910 ms 5 * * * 6 te1-2-marseille-rtr-021.noc.renater.fr (193.51.189.21) 8.441 ms 8.389 ms 8.646 ms 7 te0-0-0-0-lyon1-rtr-001.noc.renater.fr (193.51.189.17) 10.117 ms 10.090 ms 10.065 ms 8 xe-8-0-0.edge5.Paris1.Level3.net (212.73.207.173) 15.203 ms 17.176 ms 17.279 ms 9 ae-33-51.ebr1.Paris1.Level3.net (4.69.139.193) 17.261 ms 15.151 ms 15.124 ms 10 ae-47-47.ebr1.London1.Level3.net (4.69.143.109) 22.346 ms ae-48-48.ebr1.London1.Level3.net (4.69.143.113) 22.200 ms 22.164 ms 11 ae-1-51.edge3.London1.Level3.net (4.69.139.73) 22.625 ms 22.504 ms 22.582 ms 12 tdcdenmark-level3-xe.london1.Level3.net (4.68.63.90) 22.247 ms 22.714 ms 22.815 ms 13 atm1-0-5.psl-gw3.hel.fi.ip.tdc.net (62.236.1.26) 55.513 ms 55.065 ms 55.150 ms 14 proequal-cpe1.hel.fi.sn.net (62.236.27.110) 60.118 ms 60.908 ms 60.062 ms 15 ns2.greenspot.fi (62.236.221.78) 62.618 ms 63.832 ms 64.659 ms
Thanks everyone for feedback.
This could be something weird with the xen network-interface bridging. This problematic server-system is the only xen guest that shares *both* network cards on the machine.
I asked my upstream ISP to check things from their side.
I hope I will soon get a ssh account on some client where the error is reproducible. Then I could start testing.
- Jussi
On Fri, 2010-05-07 at 07:38 +0300, Jussi Hirvi wrote:
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
- Jussi
Tested: I cannot reach the first one but can see the second one. The first one does not respond to ping either.
Calin
Key fingerprint = 37B8 0DA5 9B2A 8554 FB2B 4145 5DC1 15DD A3EF E857
================================================= Inara: "Mal, you don't have to die alone." Mal: "Everybody dies alone." --Episode #8, "Out of Gas"
On Friday 07 May 2010 05:38:45 Jussi Hirvi wrote:
Ok, thanks for ideas - many new things to test. So far no luck.
Too bad i don't have first-hand access to any of the client machines who *do* have this problem.
Next, I will go and switch the ethernet cable to a different slot on the router - kind of desperate, I know.
Some more details:
- this web server is a xen virtual guest system, with CentOS 5.4
- the problem surfaced yesterday morning (6th of May), after I had
migrated all these web sites from an old Fedora box to this new CentOS system
Does the problem affect other xen systems on the same box? I haven't tested this yet (I cannot reproduce the error).
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
- Jussi
OK I can see the second one but not the first.
I can also ping the second one but not the first.
Tony
On 6.5.2010 22.00, Benjamin Franz wrote:
On 05/06/2010 11:42 AM, Ryan Manikowski wrote:
Notice the op posted they get timeouts even when going directly to a numerical address (if the apache server is configured to respond to *:80 it should at least display something)
Try using telnet from a client machine that can not connect.
e.g. telnet host.name.here 80
or
telnet xx.xxx.xxx.xxx 80
Try a few times and see if you're getting a timeout or if it connects every time. Run tcpdump on the apache server while sending the connection requests and see if the connection attempts show up at all. If they do not, then it's a network problem.
Try running 'ab' (the apache bench tool - see 'man ab' for how to use it) against your server and see if you can provoke the timeouts. If you can, then you are probably not configured to handle many quick connections and should check (1) httpd.conf to make sure you don't have an excessively low setting for 'MaxClients' or (2) a too low setting for max open filehandles. Look in /etc/security/limits.conf - you should have a line reading something similar to:
- nofile 64000
somewhere in it to raise the max number of open files. Busy web servers need lots of filehandles.
-- Benjamin Franz
-- Benjamin Franz
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, 2010-05-07 at 09:14 +0100, Tony Molloy wrote:
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
- Jussi
OK I can see the second one but not the first.
I can also ping the second one but not the first.
+ 1
Ok, I have now ssh account with which I can reproduce the errors. The error is now narrowed down to inside the box: tcpdump shows that data is coming in, but nothing is leaving.
The box is a xen system with 2 if-cards which are shared with xen guests. The error is connected to eth0 (not eth1) and affects both the host and one guest system. However, guest4, which uses eth0 only, works quite ok!
Here is a list of the host and guest, their if cards, and errors:
xen host: eth0 (produces the error with some clients) ...and eth1 (default gateway; works ok) guest2: eth1 (ok) guest3: eth1 (ok) guest4: eth0 (ok) guest5: eth0 (errors), eth1 (ok)
Below is some more data. I would still need ideas about what to test.
Again I made sure that the firewall (iptables) does not cause the error. Not tcpwrapper either: /etc/hosts.allow and /etc/hosts.deny are both empty.
- Jussi
Physical if cards:
Intel Corporation 82567LM-3 Gigabit Network Connection (eth0; on motherboard)
Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (eth1; very old card)
ifconfig output:
eth0 Link encap:Ethernet HWaddr 00:1C:C0:D7:A6:5B inet addr:62.236.221.67 Bcast:62.236.221.79 Mask:255.255.255.240 inet6 addr: fe80::21c:c0ff:fed7:a65b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1470 errors:0 dropped:0 overruns:0 frame:0 TX packets:37 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:121445 (118.5 KiB) TX bytes:7584 (7.4 KiB)
eth1 Link encap:Ethernet HWaddr 00:02:44:97:95:50 inet addr:62.220.237.104 Bcast:62.220.237.127 Mask:255.255.255.224 inet6 addr: fe80::202:44ff:fe97:9550/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:22601 errors:0 dropped:0 overruns:0 frame:0 TX packets:3371 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1644700 (1.5 MiB) TX bytes:598979 (584.9 KiB)
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:168 errors:0 dropped:0 overruns:0 frame:0 TX packets:168 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11574 (11.3 KiB) TX bytes:11574 (11.3 KiB)
peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:49343 errors:0 dropped:0 overruns:0 frame:0 TX packets:35975 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:5661018 (5.3 MiB) TX bytes:5200943 (4.9 MiB) Memory:d0600000-d0620000
peth1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:115801 errors:0 dropped:0 overruns:0 frame:0 TX packets:125786 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:23407678 (22.3 MiB) TX bytes:83301169 (79.4 MiB) Interrupt:19 Base address:0xa100
vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:37 errors:0 dropped:0 overruns:0 frame:0 TX packets:1470 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:7584 (7.4 KiB) TX bytes:121445 (118.5 KiB)
vif0.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:3391 errors:0 dropped:0 overruns:0 frame:0 TX packets:22614 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:603387 (589.2 KiB) TX bytes:1645558 (1.5 MiB)
vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:5877 errors:0 dropped:0 overruns:0 frame:0 TX packets:23727 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:379075 (370.1 KiB) TX bytes:2115170 (2.0 MiB)
vif3.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:28011 errors:0 dropped:0 overruns:0 frame:0 TX packets:50422 errors:0 dropped:22 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:16503337 (15.7 MiB) TX bytes:12688396 (12.1 MiB)
vif4.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:32753 errors:0 dropped:0 overruns:0 frame:0 TX packets:34011 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:4416529 (4.2 MiB) TX bytes:4253292 (4.0 MiB)
vif5.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:3302 errors:0 dropped:0 overruns:0 frame:0 TX packets:16735 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:280451 (273.8 KiB) TX bytes:1521618 (1.4 MiB)
vif5.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:59898 errors:0 dropped:0 overruns:0 frame:0 TX packets:71340 errors:0 dropped:74 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:62709103 (59.8 MiB) TX bytes:10451893 (9.9 MiB)
virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:27 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:6484 (6.3 KiB)
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:1135 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:82476 (80.5 KiB) TX bytes:0 (0.0 b)
xenbr1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:17410 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:819653 (800.4 KiB) TX bytes:0 (0.0 b)
On Fri, May 7, 2010 at 7:17 AM, Didi Hoffmann ribalba@gmail.com wrote:
On Fri, 2010-05-07 at 09:14 +0100, Tony Molloy wrote:
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
- Jussi
OK I can see the second one but not the first.
I can also ping the second one but not the first.
- 1
Same from this side of the world
On Fri, May 7, 2010 at 11:47 AM, Eduardo Grosclaude eduardo.grosclaude@gmail.com wrote:
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
Sure your network masks are OK?
On 05/06/2010 09:38 PM, Jussi Hirvi wrote:
Does the problem affect other xen systems on the same box? I haven't tested this yet (I cannot reproduce the error).
You could test yourself if you can see http://62.236.221.71 (the problem system) http://62.236.221.78 (another guest on the same xen host)
If someone *cannot* see the 1st one, then it would be interesting to know if (s)he can see the 2nd one or not.
Interesting. I can ping and reach the second address, but not the first one. A thought: Double-check your routes are good, your interfaces are actually all up, and that no other machine has accidentally got the IP address up as well. Post the results for 'route -n', 'ifconfig', and 'arping -D 62.236.221.71' on the machine.
On 7.5.2010 16.40, Benjamin Franz wrote:
Post the results for 'route -n', 'ifconfig', and 'arping -D 62.236.221.71' on the machine.
The values in the previous message and below are from the xen host (62.236.221.67/62.220.237.104), which displays just the same errors as the xen guest (62.236.221.71).
ifconfig is on the list already.
[root@farm1 log]# ip route show 62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 169.254.0.0/16 dev eth1 scope link default via 62.220.237.126 dev eth1
[root@farm1 log]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 62.236.221.64 0.0.0.0 255.255.255.240 U 0 0 0 eth0 62.220.237.96 0.0.0.0 255.255.255.224 U 0 0 0 eth1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1 0.0.0.0 62.220.237.126 0.0.0.0 UG 0 0 0 eth1
A meaningful response to arping I cannot deliver - I don't have a machine which displays this error *and* has arping installed.
- Jussi
On 05/08/2010 12:26 AM, Jussi Hirvi wrote:
On 7.5.2010 16.40, Benjamin Franz wrote:
Post the results for 'route -n', 'ifconfig', and 'arping -D 62.236.221.71' on the machine.
The values in the previous message and below are from the xen host (62.236.221.67/62.220.237.104), which displays just the same errors as the xen guest (62.236.221.71).
Hmmm have you got more than one bridge on your network? If so you need to make sure you have STP turned ON on all your bridges. If you have any services that require network at start up (nfs), you'll need set you network start up delay to more than 10 seconds as well, so STP has some time to settle.
I encountered similar problems when I plugged a _second_ virtualisation host into my network.
Kal
On 8.5.2010 4.31, Kahlil Hodgson wrote:
Hmmm have you got more than one bridge on your network? If so you need to make sure you have STP turned ON on all your bridges. If you have any services that require network at start up (nfs), you'll need set you network start up delay to more than 10 seconds as well, so STP has some time to settle.
I encountered similar problems when I plugged a _second_ virtualisation host into my network.
Turning on stp sounds promising (I have to confess that I never heard about stp before). Stp is indeed off for both bridges:
[root@farm1 scripts]# brctl show bridge name bridge id STP enabled interfaces virbr0 8000.000000000000 yes xenbr0 8000.feffffffffff no vif5.0 vif4.0 peth0 vif0.0 xenbr1 8000.feffffffffff no vif5.1 vif3.0 vif2.0 peth1 vif0.1
How can I turn stp on? In my /etc/xen/scripts/xen-network-common.sh there is a section:
# Don't create the bridge if it already exists. if [ ! -e "/sys/class/net/${bridge}/bridge" ]; then brctl addbr ${bridge} brctl stp ${bridge} off brctl setfd ${bridge} 0 sysctl -w "net.bridge.bridge-nf-call-arptables=0" sysctl -w "net.bridge.bridge-nf-call-ip6tables=0" sysctl -w "net.bridge.bridge-nf-call-iptables=0" ip link set ${bridge} arp off ip link set ${bridge} multicast off fi
Is if safe to turn stp "on" there (instead of "off"? (Requires xend restart at least, I suppose.) Or is there a better way to turn stp on permanently?
The box has 2 physical if cards, and both of them are used for bridges (xenbr0 and xenbr1).
- Jussi
On 05/08/2010 05:38 PM, Jussi Hirvi wrote:
How can I turn stp on? In my /etc/xen/scripts/xen-network-common.sh there is a section:
# Don't create the bridge if it already exists. if [ ! -e "/sys/class/net/${bridge}/bridge" ]; then brctl addbr ${bridge} brctl stp ${bridge} off brctl setfd ${bridge} 0 sysctl -w "net.bridge.bridge-nf-call-arptables=0" sysctl -w "net.bridge.bridge-nf-call-ip6tables=0" sysctl -w "net.bridge.bridge-nf-call-iptables=0" ip link set ${bridge} arp off ip link set ${bridge} multicast off fi
Is if safe to turn stp "on" there (instead of "off"? (Requires xend restart at least, I suppose.) Or is there a better way to turn stp on permanently?
STP is safe to turn on, but there is a small start up and tiny performance hit - that's why its off by default. All the bridges on your network have to establish relationships with each other, which can take 10-15 seconds depending on you network. Also, its not just the bridges on that box that you have to worry about: any other bridges on other boxes that are on the same network also need STP turned on. Your old Fedora box may be a potential culprit.
I've never used Xen, so I can't give any firm advice. That looks like the place where the bridge is created, so at a guess, that's where you want to turn it on. Not to sure about turning ARP or MULTICAST off though -- that might interfere with STP.
The box has 2 physical if cards, and both of them are used for bridges (xenbr0 and xenbr1).
Yeah. Thinking you definitely need STP. You can turn it on temporarily with
brctl stp xenbr0 on brctl stp xenbr1 on
wait a few seconds and run
brctrl showstp xenbr0
to see what's going on, and also see if it fixes your problem.
Hope this helps
Kal
On 8.5.2010 11.56, Kahlil Hodgson wrote:
Is if safe to turn stp "on" there (instead of "off"? (Requires xend restart at least, I suppose.) Or is there a better way to turn stp on permanently?
STP is safe to turn on, but there is a small start up and tiny performance hit - that's why its off by default. All the bridges on your network have to establish relationships with each other, which can take 10-15 seconds depending on you network. Also, its not just the bridges on that box that you have to worry about: any other bridges on other boxes that are on the same network also need STP turned on. Your old Fedora box may be a potential culprit.
I've never used Xen, so I can't give any firm advice. That looks like the place where the bridge is created, so at a guess, that's where you want to turn it on. Not to sure about turning ARP or MULTICAST off though -- that might interfere with STP.
The box has 2 physical if cards, and both of them are used for bridges (xenbr0 and xenbr1).
Yeah. Thinking you definitely need STP. You can turn it on temporarily with
brctl stp xenbr0 on brctl stp xenbr1 on
wait a few seconds and run
brctrl showstp xenbr0
to see what's going on, and also see if it fixes your problem.
Hope this helps
Kal
Thanks, it does (though the problem still persists).
I turned stp on (for both bridges). I found another virbridge on another machine which has 2 if-cards: "virbr0", created by CentOS 5 by default I guess, for dhcp network, which I never even thought of. I brought this bridge down with icfonfig - btw, how can I disable it so that it stays off through reboots?
So far the problem persists - I guess that I will have to start modifying routing tables.
I guess it's natural that this kind of problem is weird. :-)
For example, it is kind of natural that I can access these problematic 62.236.221.xx addresses (on the xen box) from other boxes in the same 62.236.221.xx network segment.
But I can *also* access those ip addresses from the network 62.220.237.xx. Why? No idea. (the other if-card on the xen box is configured to this network segment, but I don't see why this would explain this.)
Also seen from my home computer at 84.20.154.60 everything seems normal - no idea why!
These (62.236.221.xx, 62.220.237.xx, 84.20.154.58/xx) are the only known clients from which the problematic addresses (62.236.221.67, 62.236.221.71) on the xen box are visible. :-/
- Jussi
On Sat, 2010-05-08 at 15:00 +0300, Jussi Hirvi wrote:
But I can *also* access those ip addresses from the network 62.220.237.xx. Why? No idea. (the other if-card on the xen box is configured to this network segment, but I don't see why this would explain this.)
Also seen from my home computer at 84.20.154.60 everything seems normal
- no idea why!
These (62.236.221.xx, 62.220.237.xx, 84.20.154.58/xx) are the only known clients from which the problematic addresses (62.236.221.67, 62.236.221.71) on the xen box are visible. :-/
---
You only see them from your home pc because your on the same address block/dom/ip carrier! Look at your routing and dns. Sporadic dns issues and routing? BTW some addys get blocked from certain countries also. If I were you I would start from scratch and go step by step and set it up.
John
On 8.5.2010 16.28, JohnS wrote:
You only see them from your home pc because your on the same address block/dom/ip carrier! Look at your routing and dns. Sporadic dns issues and routing? BTW some addys get blocked from certain countries also. If I were you I would start from scratch and go step by step and set it up.
Just for the record: dns has been ruled out as a cause of these errors, since the error persists even when connecting by ip address.
The subject of the thread still seems appropriate: it *seems* just like firewall issue (my home, and the local network segments get access, everybody else not), but it isn't (iptables stop does not help). Instead, the cause of the malady is most probably some routing tangle in the xen box itself.
- Jussi
On Sat, 2010-05-08 at 21:28 +0300, Jussi Hirvi wrote:
On 8.5.2010 16.28, JohnS wrote:
You only see them from your home pc because your on the same address block/dom/ip carrier! Look at your routing and dns. Sporadic dns issues and routing? BTW some addys get blocked from certain countries also. If I were you I would start from scratch and go step by step and set it up.
Just for the record: dns has been ruled out as a cause of these errors, since the error persists even when connecting by ip address.
The subject of the thread still seems appropriate: it *seems* just like firewall issue (my home, and the local network segments get access, everybody else not), but it isn't (iptables stop does not help). Instead, the cause of the malady is most probably some routing tangle in the xen box itself.
--- Since you really only have access on the address blocks allocated to (your name here) then maybe talk to the Up Stream provider from yours since you have access with in the *greenspot domain allocation. For the record I cant ping any one of the two IPs from my home or any where else.
Do you have inbound access to other servers on the same block and subnet? Do you have the right sub net mask for the ip block your using? 62.236.221.64 - 62.236.221.79
Out of that block I can only access these if that helps you any. This should give you a big clue to what is going on.
62.236.221.70: https://webmail.greenspot.fi/cgi-bin/openwebmail/openwebmail.pl
62.236.221.76 http://www.danskecapital.fi/
John
On 05/08/2010 11:28 PM, JohnS wrote:
If I were you I would start from scratch and go step by step and set it up.
John
I'm in agreement with John here. Your set up looks complex and may be starting from scratch is the way to go. Looking back though the thread, your set up might also be unnecessarily complex. I'm not entirely sure why you have two xen bridges and two vlans on the xen host, but that may just be the way the xen setup scripts work, and, well, I've never used xen. (I use libvirt with KVM, plus virtio to give me decent performance, and I hardwire my bridges in my network-scripts so I can ensure they are up and running before anything tries to use them).
I think the problem is either ARP or routing (or both) or an overly complex setup (or all three) ;-) Lets start with ARP.
ARP is pretty reliable and usually 'just works' on most networks -- so much so that it rarely causes any problems and is easy to forget about. The one situation that is guaranteed to cause a problem is having two host with the same IP address on the same network. (I had a situation once where a technician reset a problematic UPS to its defaults -- giving it the same IP address as one of our core routers and causing us no end of problems till we figured it out). This situation is tricky to troubleshoot because some parts of the network may be quite stable depending on the layout of bridges. (I had another situation once where a wireless router was reset to defaults and took on the IP address of the main gateway -- one bunch of Linux machines behind one switch was stable whereas another bunch of Windows machines behind a different switch was unstable with arp entries flip-flopping every 10 minutes or so). I mention this case in detail because: 1) you're migrating from an existing setup and there is a small chance that the old host is still on the network and has the problematic IP address, or there is some static APR entry lying around somewhere, and double checking may save you some sanity; and 2) you're using virtualization and bridges which means ARP has to flow and (in complex set ups) is why turning on STP can help.
For completeness (and the archives) detecting the 'multiple hosts with the same IP' case is pretty easy: just install arpwatch on a host that receives traffic from all hosts on the network (router, dns server, dhcp server). You get a nice little email every time it thinks an IP address is flip-flopping. You also get a nice little email every time someone plugs a new device into your network (say, the cleaner's laptop at 2am). If you don't have arpwatch, then you could try:
while true; do arp -n >> arp.log; sleep 5; done
for 10-15 minutes (longer on a quiet network) and then do
sort -u arp.log
If the same IP address pops up twice, you have a problem. Tracking down the culprit is a little harder, but you can use the fact that the leading portion of the HW address is usually the same for the same NIC manufacturer. If the offending HW address is similar to those used by, say, known SUN machines, then that rules out any, say, Dell desktops. The arp.log file you just created can be handy for making these comparisons.
The above problem is rare. I've only experienced on 'real' hardware 3 times in the passed 10 years, but I've seen it a lot more since I started playing with virtualization. A more common issue is ARP requests not flowing around your network properly. Bridges allow you to join networks together by passing along ARP queries. If you have multiple bridges, you can get loops, and this can prevent the ARP query reaching the right bridge, so you need STP to ensure bridges cooperate appropriately. (You can get by without STP, but only if you are 100% sure there are no loops). Firewalls can stop ARP flowing as well. On Linux this is arptables and ebtables. I note that your xen startup scripts turn this off, so you should be good, provided turning them off doesn't block ARP forwarding. Not sure about that. (I notice they are on on my systems, but I like to restrict traffic on the bridge, with the iptables/arptable/ebtables configuration managed by shorewall).
I troubleshoot the above ARP issue the same way as for routing and firewalls, so lets move onto that. For newbies and the archives, I'll detail my approach.
Firstly, you want to map out whether or not traffic can pass between various IP addresses. These flows may or may not be bi-directional (A to B, is not necessarily the same as B to A) depending on stateful firewalls and convoluted routing. If there are more than 2 addresses involved, things can get complicated and its easy miss important clues, so its good to be organised. I usually grab a scrap piece of paper and draw up a NxN matrix of the problematic IP addresses so I can note which flows pass and which flows fail. I also add a few good IP addresses (your desktop, routers, dhcp server, dns server, xen host, other guests, home desktop, are good candidates) so I can verify my methodology and potentially spot related issues -- these may provide clues as to the real problem. This may seem excessive, but at least you know you are getting a complete picture.
I test flows with a combination of ping (or hping) and tcpdump. I normally layout 4 terminals in a 2x2 grid like so:
host A: tcpdump host A: ping host B host B: tcpdump host B: ping host A
For testing routes and ARP my tcpdump invocation is something like
tcpdump -i ethX icmp or arp
where ethX is the interface with the IP address I'm testing. For testing firewall rules I use something like
tcpdump -i ethX port YYYY
and invoke hping like so
hping -S -p YYYY aaa.bbb.ccc.ddd
where YYYY is the port you are testing and aaa.bbb.ccc.ddd the IP address. There are lots of other tcpdump options that you can use to refine what you see and cut out any irrelevant traffic (like your ssh session). Note that I only run one ping test at a time so I can watch packets flowing from both sides. The extra terminal is just to make it a little easier to be systematic in tracing reverse flows.
Now if you do the above tests on a pair of IP addresses where routing and ARP is good, you should see nice little ICMP request/reply pairs and nice little ARP who-has/reply pairs. If APR isn't working, you'll be getting a strings of ARP who-has packets with no replies. If routing is broken, you get strings of ICMP requests without replies (yeah obviously) and by comparing the outputs in the three terminals, you may be able to figure out where its broken. If you fill out your flow matrix, you'll have a pretty clear picture of the situation and may have some more clues as to were the problem lies. You'll also have a nice framework for testing potential fixes.
Jussi, if this is all old news to you, please forgive my rambling diatribe. From bitter experience, I just like to make sure everyone's on the same page with this stuff:-)
Now lets look at some details.
Physical if cards: Intel Corporation 82567LM-3 Gigabit Network Connection (eth0; on motherboard)
Recently I've had some onboard NICs slowly die over a period of about 2 weeks. When it happens you can get some fairly inexplicable errors. Probably not your problem, but thought I might point out the possibility.
ip route output on the xen host:
62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104 default via 62.220.237.126 dev eth1
Your 'good' interface eth1 is also the default route. I gather 62.220.237.126 is your border router. Where does your 'bad' interface eth0 go then? The most common problem with routing on multi-homed systems is having packets leaving one interface and responses coming back on another.
Really need to see what 'ip route' says on each of you guests.
Hmmm... ifconfig on you xen host:
vif5.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF .... vif5.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF ... xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
Normally an interface with a 'dot' suffix is on a specific VLAN. So you've got two VLANs running here? This complicated things. Also your bridge has NOARP set which does not make much sense to me.
brctl show
bridge name bridge id STP enabled interfaces virbr0 8000.000000000000 yes xenbr0 8000.feffffffffff no vif5.0 vif4.0 peth0 vif0.0 xenbr1 8000.feffffffffff no vif5.1 vif3.0 vif2.0 peth1 vif0.1
Okay, that makes my head hurt. Why two VLANs? What's you mapping between virtual interfaces and guests? And which guest is the bad one? Hmmm ... I could interpret this as meaning vif5.0 and vif5.1 are two interfaces on guest5 which you want to directly tie to to different physical cards. Perhaps some source based routing on your xen host would serve you better. This may be part of the overly complex setup which is causing problems.
Are these guests all just web servers?
Sorry if that is a too long ramble. I'm a bit confused by your set up, and as John suggested, you may need to rebuild things from scratch to sort things out.
Hope this helps!
Kal
On 9.5.2010 14.03, Kahlil Hodgson wrote:
Okay, that makes my head hurt. Why two VLANs? What's you mapping between virtual interfaces and guests? And which guest is the bad one?
Ok, Kal, thank you for very useful ramblings!
This box is already in production, but I think the most useful approach here is to reconsider my setup.
I have two public networks here, 62.220.237.x and 62.236.221.x. I want to build a xen system, where some guests connect to one network, some guest to the other one, and some to both. To reduce cabling, I would like to do this with only two nics.
My solution now is two virtual bridges (I can post nearer details, if needes). And I have now landed into routing difficulties.
Are there some simpler or otherwise better approaches?
- Jussi
On 05/10/2010 05:34 PM, Jussi Hirvi wrote:
This box is already in production, but I think the most useful approach here is to reconsider my setup.
I have two public networks here, 62.220.237.x and 62.236.221.x. I want to build a xen system, where some guests connect to one network, some guest to the other one, and some to both. To reduce cabling, I would like to do this with only two nics.
My solution now is two virtual bridges (I can post nearer details, if needes). And I have now landed into routing difficulties.
Are there some simpler or otherwise better approaches?
I'd opt for NAT and policy-based routing. I'll get back to you with details after I've had my diner ;-)
Cheers!
Kal
On 10.5.2010 12.50, Kahlil Hodgson wrote:
I'd opt for NAT and policy-based routing. I'll get back to you with details after I've had my diner ;-)
Cheers!
Kal
Hm, NAT might be difficult, because there are common ports to the guest systems. Below is more detail:
If we say network A = 62.220.237.x and B = 62.236.221.x
My guest systems are: - name server (port 53) (network B) - mail server (80,443,25,465,995.993,563,636) (network A) - secondary mail server to a mail server in another box (25,465) (preferably network A AND B, for maximum availability) - a test system, can be in either network (but port 22 required)
Of course I could rearrange, for example set up another xen box for one of these mail servers.
- Jussi
On 05/10/2010 08:09 PM, Jussi Hirvi wrote:
Hm, NAT might be difficult, because there are common ports to the guest systems.
Yeah, but they're going to have different IP addresses and we could do NAT around that. My personal preference is to put a router between external interfaces (with non-RFC1918 addresses) and my internal network (even a DMZ). I'd then use NAT to map the external ipaddr:port pairs to internal ipaddr:port pairs. If your hosts need SSL, I'd add dnsmaq to the mix so internally your guests FQDNs resolve to internal addresses, whereas externally their FQDNs resolve to the external addresses.
This gives me a very clean and clear separation between inside my network and outside, and no one outside my network is going to see my RFC1918 address space. I'm using shorewall to configure iptables to do the NAT so everything is clean and clear and my firewalls can be very restrictive. Haven't got a spare router? Well your xen-host _can_ be the router.
Below is more detail:
If we say network A = 62.220.237.x and B = 62.236.221.x
My guest systems are:
- name server (port 53) (network B)
- mail server (80,443,25,465,995.993,563,636) (network A)
- secondary mail server to a mail server in another box (25,465) (preferably network A AND B, for maximum availability)
- a test system, can be in either network (but port 22 required)
Of course I could rearrange, for example set up another xen box for one of these mail servers.
Okay, and you name server 62.236.221.71 (?) is the one causing problems?
If I was starting from scratch, I'd be giving you guests IP addresses 192.168.0.1-5 and put them all on the same bridge. I'd then use NAT on the xen-host to map the external IPs to internal and forward them to the bridge. Not strictly necessary but cleaner to my mind.
My guess is requests coming in on network B and responses leaving by network A. This will confuse the heck out of the recipient 'cause it mixes up the SRC and DST headers. The solution to this is policy-based routing. Basically we set things up to use different routing tables depending on the source address of the packet. Here's a pointer to some reading that should get you up to speed.
http://www.policyrouting.org/PolicyRoutingBook/ONLINE/TOC.html
Lots of good stuff in there and well work the read.
Hope this helps,
Kal
On 10.5.2010 16.20, Kahlil Hodgson wrote: Here's a pointer to some
reading that should get you up to speed.
http://www.policyrouting.org/PolicyRoutingBook/ONLINE/TOC.html
Lots of good stuff in there and well work the read.
Thanks Kal, the nat approach starts to sound good. I will read that doc, then let's see.
- Jussi
On 05/10/2010 06:20 AM, Kahlil Hodgson wrote:
This gives me a very clean and clear separation between inside my network and outside, and no one outside my network is going to see my RFC1918 address space.
I weep every time I see someone advocate NAT for security reasons. It's ridiculous.
Routing policy is definitely required for a multi-homed system such as Jussi presented, but NAT is totally superfluous. It adds an extra layer of complexity that makes the system more difficult to diagnose and configure, and contributes nothing of value in return.
John Pierce's advice was simple and correct. If you don't want to set up ifup-post scripts of your own, you can use shorewall. Shorewall is actually more complex, but you don't have to understand much about the "ip" tool to use it.
For shorewall, you'd need the following files:
interfaces: inet eth0 - norfc1918,nosmurfs,tcpflags inet eth1 - norfc1918,nosmurfs,tcpflags lan virbr0 - dhcp
zones: fw firewall inet ipv4 lan ipv4
policy: $FW all ACCEPT inet inet DROP all inet ACCEPT all all REJECT info
providers: isp0 1 1 main eth0 62.236.221.78 track,balance isp1 2 2 main eth1 62.220.237.126 track,balance
route_rules: lo - isp0 11000 eth0 - isp0 11000 eth1 - isp1 11000 virbr0 - isp1 11000
On 11/05/10 10:40, Gordon Messmer wrote:
On 05/10/2010 06:20 AM, Kahlil Hodgson wrote:
This gives me a very clean and clear separation between inside my network and outside, and no one outside my network is going to see my RFC1918 address space.
I weep every time I see someone advocate NAT for security reasons. It's ridiculous.
I agree 100% and was not the intent of the comment. This is more an organisational approach based on what I've seen and done in the past. Being organised does contribute to security ;-)
Routing policy is definitely required for a multi-homed system such as Jussi presented, but NAT is totally superfluous. It adds an extra layer of complexity that makes the system more difficult to diagnose and configure, and contributes nothing of value in return.
Now that I understand Jussi's set up a little better (all the components seem to be fully internet facing) I also agree 100% with that.
John Pierce's advice was simple and correct. If you don't want to set up ifup-post scripts of your own, you can use shorewall. Shorewall is actually more complex, but you don't have to understand much about the "ip" tool to use it.
Understanding a bit about the "ip" tool and policy-based routing (and routing in general) will help to understand the configuration you have suggested, and mapping it to his needs.
interfaces: inet eth0 - norfc1918,nosmurfs,tcpflags inet eth1 - norfc1918,nosmurfs,tcpflags lan virbr0 - dhcp
zones: fw firewall inet ipv4 lan ipv4
policy: $FW all ACCEPT inet inet DROP all inet ACCEPT all all REJECT info
providers: isp0 1 1 main eth0 62.236.221.78 track,balance isp1 2 2 main eth1 62.220.237.126 track,balance
route_rules: lo - isp0 11000 eth0 - isp0 11000 eth1 - isp1 11000 virbr0 - isp1 11000
Nice use of providers and route_rules. Only one bridge, the one that libvirtd brings up by default. Shorewall probably has to start after libvirtd for this to work. All Jussi has to do is to mod this and his xen-startup scripts to use the same bridge.
Kal
On 11.5.2010 3.40, Gordon Messmer wrote:
Routing policy is definitely required for a multi-homed system such as Jussi presented, but NAT is totally superfluous. It adds an extra layer of complexity that makes the system more difficult to diagnose and configure, and contributes nothing of value in return.
Funny, this morning I came to the same conclusion after some googling. A xen box with two bridges should be considered normal, and it should not break anything inside or outside the box.
There are good instructions on the net for installing 2 virtual bridges on a xen box. But I have found no mention of this specific dual-bridge problem I have: that ip traffic goes in ok through any physical nic to the dom0 or domUs, but all replies are routed to only one nic (the default gateway). (I verified this with tcpdump.)
John Pierce's advice was simple and correct. If you don't want to set up ifup-post scripts of your own, you can use shorewall. Shorewall is actually more complex, but you don't have to understand much about the "ip" tool to use it.
I am going to try this first without Shorewall (simpler, I hope).
John, could you elaborate a little on this (I never had to adjust routing before):
On 10.5.2010 21.15, John R Pierce wrote:
something like...
[after setting up network 1 the 'normal' way, we add these rules for network 2...]
NET2=xxx.yyy.zzz.www/26 NET2GWY=xxx.yyy.zzz.wwx ip rule add from $NET2 table 200 ip route add default via $NET2GWY dev eth1 table 200 ip route flush cacheso... any packet thats 'from' the subnet $NET2 is tagged to use ip routing table '200' (quite arbitrary), and in turn route table 200 specifies a different default gateway.
Where should I put that script? network-scripts/ifup-post? What would your "table 200" look like, and where should I put that?
- Jussi
Jussi Hirvi wrote:
On 11.5.2010 3.40, Gordon Messmer wrote:
Routing policy is definitely required for a multi-homed system such as Jussi presented, but NAT is totally superfluous. It adds an extra layer of complexity that makes the system more difficult to diagnose and configure, and contributes nothing of value in return.
Funny, this morning I came to the same conclusion after some googling. A xen box with two bridges should be considered normal, and it should not break anything inside or outside the box.
There are good instructions on the net for installing 2 virtual bridges on a xen box. But I have found no mention of this specific dual-bridge problem I have: that ip traffic goes in ok through any physical nic to the dom0 or domUs, but all replies are routed to only one nic (the default gateway). (I verified this with tcpdump.)
That's not xen or bridge related. Unless you do policy-based routing, packets always follow the destination route regardless of where the input was received. That's a feature, not a bug.
Jussi Hirvi wrote:
But I have found no mention of this specific dual-bridge problem I have: that ip traffic goes in ok through any physical nic to the dom0 or domUs, but all replies are routed to only one nic (the default gateway). (I verified this with tcpdump.)
On 11.5.2010 16.08, Les Mikesell wrote:
That's not xen or bridge related. Unless you do policy-based routing, packets always follow the destination route regardless of where the input was received. That's a feature, not a bug.
Ok. But this error does not occur on my other CentOS 5 box (mailserver, non-xen) which also has 2 nics for 2 public ip segments. There input-nic is always = outputnic. And I have done nothing special to achieve this (pure "linux magic"). That's why I "blame" bridges - they are the most notable difference between these two machines.
- Jussi
On 05/11/2010 06:32 AM, Jussi Hirvi wrote:
Ok. But this error does not occur on my other CentOS 5 box (mailserver, non-xen) which also has 2 nics for 2 public ip segments. There input-nic is always = outputnic. And I have done nothing special to achieve this (pure "linux magic"). That's why I "blame" bridges - they are the most notable difference between these two machines.
That's odd. Is there any output on that host from "ip rule show"? What about:
# ip rule show # ip rule show | awk '{print $NF}' | sort | uniq | \ while read table ; do echo ; echo " $table" ; ip route show table "$table" ; done
On 11.5.2010 18.36, Gordon Messmer wrote:
That's odd. Is there any output on that host from "ip rule show"? What about:
# ip rule show # ip rule show | awk '{print $NF}' | sort | uniq | \ while read table ; do echo ; echo " $table" ; ip route show table "$table" ; done
Interesting commands, and revealing, it seems to me.
Here's the results, first from a "healthy" (non-xen) host ("ordinary" (?) CentOS 5.4 with two nics, each connecting to their own public network segment:
[root@mail ~]# ip rule show 0: from all lookup 255 500: from 62.236.221.70 lookup 2 600: from 62.220.237.110 lookup 1 32766: from all lookup main 32767: from all lookup default
[root@mail ~]# ip rule show | awk '{print $NF}' | sort | uniq | \
while read table ; do echo ; echo " $table" ; ip route show table "$table" ; done
1 default via 62.220.237.126 dev eth0
2 default via 62.236.221.65 dev eth1
255 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 local 62.236.221.70 dev eth1 proto kernel scope host src 62.236.221.70 broadcast 62.220.237.127 dev eth0 proto kernel scope link src 62.220.237.110 broadcast 62.236.221.64 dev eth1 proto kernel scope link src 62.236.221.70 local 62.220.237.110 dev eth0 proto kernel scope host src 62.220.237.110 local 192.168.122.1 dev virbr0 proto kernel scope host src 192.168.122.1 broadcast 62.236.221.79 dev eth1 proto kernel scope link src 62.236.221.70 broadcast 62.220.237.96 dev eth0 proto kernel scope link src 62.220.237.110 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
default
main 62.236.221.64/28 dev eth1 proto kernel scope link src 62.236.221.70 62.220.237.96/27 dev eth0 proto kernel scope link src 62.220.237.110 169.254.0.0/16 dev eth1 scope link default via 62.236.221.65 dev eth1 [root@mail ~]#
Now the "sick" host, the CentOS 5.4 xen box (dom0) with two nics, each connecting to their own public network segment (there should be something more in "ip rule show", right?):
[root@farm1 ~]# ip rule show 0: from all lookup 255 32766: from all lookup main 32767: from all lookup default
[root@farm1 ~]# ip rule show | awk '{print $NF}' | sort | uniq | \
while read table ; do echo ; echo " $table" ; ip route show table "$table" ; done
255 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 local 62.220.237.104 dev eth1 proto kernel scope host src 62.220.237.104 broadcast 62.220.237.127 dev eth1 proto kernel scope link src 62.220.237.104 broadcast 62.236.221.64 dev eth0 proto kernel scope link src 62.236.221.67 local 192.168.122.1 dev virbr0 proto kernel scope host src 192.168.122.1 local 62.236.221.67 dev eth0 proto kernel scope host src 62.236.221.67 broadcast 192.168.122.0 dev virbr0 proto kernel scope link src 192.168.122.1 broadcast 62.236.221.79 dev eth0 proto kernel scope link src 62.236.221.67 broadcast 62.220.237.96 dev eth1 proto kernel scope link src 62.220.237.104 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 broadcast 192.168.122.255 dev virbr0 proto kernel scope link src 192.168.122.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
default
main 62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 169.254.0.0/16 dev eth1 scope link default via 62.220.237.126 dev eth1 [root@farm1 ~]#
- Jussi
On 05/11/2010 10:21 AM, Jussi Hirvi wrote:
Interesting commands, and revealing, it seems to me.
Well, there you go. Something set up policy routing on the working host. Do you have any files like /etc/sysconfig/network-scripts/route-* or /etc/sysconfig/network-scripts/rule-* ?
On 12.5.2010 3.25, Gordon Messmer wrote:
On 05/11/2010 10:21 AM, Jussi Hirvi wrote:
Interesting commands, and revealing, it seems to me.
Well, there you go. Something set up policy routing on the working host. Do you have any files like /etc/sysconfig/network-scripts/route-* or /etc/sysconfig/network-scripts/rule-* ?
None. But I found these (standard CentOS files):
[root@farm1 network-scripts]# grep -rl "ip rule" . ./ifdown-routes ./ifup-routes
- Jussi
On 05/11/2010 10:21 PM, Jussi Hirvi wrote:
On 12.5.2010 3.25, Gordon Messmer wrote:
On 05/11/2010 10:21 AM, Jussi Hirvi wrote:
Interesting commands, and revealing, it seems to me.
Well, there you go. Something set up policy routing on the working host. Do you have any files like /etc/sysconfig/network-scripts/route-* or /etc/sysconfig/network-scripts/rule-* ?
None. But I found these (standard CentOS files):
[root@farm1 network-scripts]# grep -rl "ip rule" . ./ifdown-routes ./ifup-routes
Yes, those scripts will run "ip rule" to process the contents of the "rule-*" files. The company I work for uses shorewall on all of their multi-homed systems, so I'm not sure how systems without it behave. That said, I don't see any magic in the init scripts to handle this without your input. I'm inclined to believe that something on your system was manually configured to set up the routing policy that you see.
Find it harder: find /etc/ -type f -print0 | xargs -0 grep "ip rule"
Gordon wrote:
On 05/11/2010 10:21 PM, Jussi Hirvi wrote:
On 12.5.2010 3.25, Gordon Messmer wrote:
On 05/11/2010 10:21 AM, Jussi Hirvi wrote:
<snip>
Find it harder: find /etc/ -type f -print0 | xargs -0 grep "ip rule"
Or, since modern find's default to -print, you could do find /etc -type f -exec grep -l "ip rule" {} ;
mark "of all the ways you *can* do it in *Nix, how would you *like* to do it?"
On 05/13/2010 12:47 PM, m.roth@5-cent.us wrote:
Gordon wrote:
Find it harder: find /etc/ -type f -print0 | xargs -0 grep "ip rule"
Or, since modern find's default to -print,
Yes, they do, but I have no idea what that has to do with your suggestion to use -exec. If you had suggested eliminating the use of -print as redundant, your suggestion would be merely inefficient rather than a non sequitur.
you could do find /etc -type f -exec grep -l "ip rule" {} ;
You could, but that would run "grep" once for each file where xargs will run grep the minimum number of times. Using xargs is substantially faster.
Having said that, I see that find has an option of which I was previously unaware. If you use the '+' character instead of the ';', it will behave more or less the same way that xargs does:
find /etc/ -type f -exec grep "ip rule" {} +
[root@farm1 network-scripts]# grep -rl "ip rule" . ./ifdown-routes ./ifup-routes
On 13.5.2010 21.36, Gordon Messmer wrote:
Yes, those scripts will run "ip rule" to process the contents of the "rule-*" files. The company I work for uses shorewall on all of their multi-homed systems, so I'm not sure how systems without it behave. That said, I don't see any magic in the init scripts to handle this without your input. I'm inclined to believe that something on your system was manually configured to set up the routing policy that you see.
Find it harder: find /etc/ -type f -print0 | xargs -0 grep "ip rule"
Ok, rc.d/routes is probably it (on the "healthy" machine I previously used as a reference). I will have to study the ip command and routing a bit, then make a fix on the "non-healthy" (xen) box.
[root@mail ~]# find /etc -type f -exec grep -l "ip rule" {} ; /etc/udev/rules.d/50-udev.rules.rpmorig /etc/udev/rules.d/50-udev.rules /etc/rc.d/routes /etc/sysconfig/network-scripts/ifdown-routes.rpmorig /etc/sysconfig/network-scripts/ifdown-routes /etc/sysconfig/network-scripts/ifup-routes.rpmorig /etc/sysconfig/network-scripts/ifup-routes
[root@mail rc.d]# cat routes
/sbin/ip address add 62.220.237.110/27 dev eth0 /sbin/ip route add default via 62.220.237.126 tab 1 /sbin/ip route add default via 62.236.221.65 tab 2 /sbin/ip rule add from 62.236.221.70 tab 2 prio 500 /sbin/ip rule add from 62.220.237.110 tab 1 prio 600 /sbin/ip route flush cache
- Jussi
On 05/14/2010 12:10 AM, Jussi Hirvi wrote:
Ok, rc.d/routes is probably it
Looks that way. I find that relatively reassuring. No "linux magic" involved. But then, if you didn't set that up, who did?
(on the "healthy" machine I previously used as a reference). I will have to study the ip command and routing a bit, then make a fix on the "non-healthy" (xen) box.
I'd recommend either setting the rules up in a "rules-eth0" or such file in /etc/sysconfig/network-scripts, or using shorewall. Inventing your own system is workable, but as you've found, they tend not to be documented well which leads future admins (or even future you) to wonder how things work. Use the facilities available rather than fighting them.
On 05/11/2010 10:21 AM, Jussi Hirvi wrote:
Interesting commands, and revealing, it seems to me.
On 12.5.2010 3.25, Gordon Messmer wrote:
Well, there you go. Something set up policy routing on the working host. Do you have any files like /etc/sysconfig/network-scripts/route-* or /etc/sysconfig/network-scripts/rule-* ?
BTW, the same grep result both on the working and non-working (xen) machine.
- Jussi
On 5/11/2010 8:32 AM, Jussi Hirvi wrote:
Jussi Hirvi wrote:
But I have found no mention of this specific dual-bridge problem I have: that ip traffic goes in ok through any physical nic to the dom0 or domUs, but all replies are routed to only one nic (the default gateway). (I verified this with tcpdump.)
On 11.5.2010 16.08, Les Mikesell wrote:
That's not xen or bridge related. Unless you do policy-based routing, packets always follow the destination route regardless of where the input was received. That's a feature, not a bug.
Ok. But this error does not occur on my other CentOS 5 box (mailserver, non-xen) which also has 2 nics for 2 public ip segments. There input-nic is always = outputnic. And I have done nothing special to achieve this (pure "linux magic"). That's why I "blame" bridges - they are the most notable difference between these two machines.
That doesn't make much (any?) sense. IP traffic is always destination-routed unless you do something unusual. On the other hand, even if you send out to the 'wrong' internet gateway following your default route, any internet connection should be able to deliver to any internet destination. Asymmetrical routing is both permitted and normal, although not necessarily desirable and it may not make it through stateful firewalls.
Jussi Hirvi wrote:
On 9.5.2010 14.03, Kahlil Hodgson wrote:
Okay, that makes my head hurt. Why two VLANs? What's you mapping between virtual interfaces and guests? And which guest is the bad one?
Ok, Kal, thank you for very useful ramblings!
This box is already in production, but I think the most useful approach here is to reconsider my setup.
I have two public networks here, 62.220.237.x and 62.236.221.x. I want to build a xen system, where some guests connect to one network, some guest to the other one, and some to both. To reduce cabling, I would like to do this with only two nics.
My solution now is two virtual bridges (I can post nearer details, if needes). And I have now landed into routing difficulties.
Are there some simpler or otherwise better approaches?
How do you handle the default route on the 'connect to both' guests? Normally you only want one default gateway and it should be the same one where the connections are coming in. Otherwise you have to do some very tricky things to make return packets go back the same path they came in, although asymmetrical routes are supposed to work if you don't have NAT or stateful firewalls in the way.
I have two public networks here, 62.220.237.x and 62.236.221.x. I want to build a xen system, where some guests connect to one network, some guest to the other one, and some to both. To reduce cabling, I would like to do this with only two nics.
On 10.5.2010 15.48, Les Mikesell wrote:
How do you handle the default route on the 'connect to both' guests? Normally you only want one default gateway and it should be the same one where the connections are coming in. Otherwise you have to do some very tricky things to make return packets go back the same path they came in, although asymmetrical routes are supposed to work if you don't have NAT or stateful firewalls in the way.
On that dual-network xen-guest, I don't handle the routing in any special way. Now only one nw connection works (because of these routing problems), but if they would both work, packets still might leave from only one interface (default route). I don't see why this would be a problem, though, even if it may not be very elegant.
Here is "ip route show" from that host:
62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.71 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.111 169.254.0.0/16 dev eth1 scope link default via 62.220.237.126 dev eth1
- Jussi
Jussi Hirvi wrote:
On 10.5.2010 15.48, Les Mikesell wrote:
How do you handle the default route on the 'connect to both' guests? Normally you only want one default gateway and it should be the same one where the connections are coming in. Otherwise you have to do some very tricky things to make return packets go back the same path they came in, although asymmetrical routes are supposed to work if you don't have NAT or stateful firewalls in the way.
On that dual-network xen-guest, I don't handle the routing in any special way. Now only one nw connection works (because of these routing problems), but if they would both work, packets still might leave from only one interface (default route). I don't see why this would be a problem, though, even if it may not be very elegant.
A) it could saturate the outbound on one link while leaving the other empty
B) the ISP on link 1 might not forwarding outbound packets that are 'from' an IP on a different subnet
NAT'ing two different blocks is semi-ugly, and requires diving into `ip rule add` and `ip route add`... something like...
[after setting up network 1 the 'normal' way, we add these rules for network 2...]
NET2=xxx.yyy.zzz.www/26 NET2GWY=xxx.yyy.zzz.wwx
ip rule add from $NET2 table 200 ip route add default via $NET2GWY dev eth1 table 200 ip route flush cache
so... any packet thats 'from' the subnet $NET2 is tagged to use ip routing table '200' (quite arbitrary), and in turn route table 200 specifies a different default gateway.
I dunno any better way to do this. Also, if you have DMZ hosts you specifically want to bind to the $NET2, you can add source rules for their NAT IP to force them to use the 2nd interface.
On 05/10/2010 11:03 PM, Jussi Hirvi wrote:
On 10.5.2010 15.48, Les Mikesell wrote:
How do you handle the default route on the 'connect to both' guests? Normally you only want one default gateway and it should be the same one where the connections are coming in. Otherwise you have to do some very tricky things to make return packets go back the same path they came in, although asymmetrical routes are supposed to work if you don't have NAT or stateful firewalls in the way.
On that dual-network xen-guest, I don't handle the routing in any special way. Now only one nw connection works (because of these routing problems), but if they would both work, packets still might leave from only one interface (default route). I don't see why this would be a problem, though, even if it may not be very elegant.
Here is "ip route show" from that host:
62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.71 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.111 169.254.0.0/16 dev eth1 scope link default via 62.220.237.126 dev eth1
You've also got two bridges (xenbr0 and xenbr1) and you've enslaved eth0 to the first and eth1 to the second. From your ifconfig output, none of you're bridges or virtual interfaces seem to have IP addresses or networks. Okay, its early in the morning and I had a few beers while watching the footy last night, so I could be completely wrong here, but I'm not entirely sure your routing table is having any direct impact on the network flows at all. My guess is traffic from guests on network A is going straight out eth0 to whatever switch it is connected to and not touching your xen-host routing table at all; likewise traffic from guest on network B and eth1 (other list readers feel free to correct me here).
I have to shower and head off to work but the shorewall documentation about bridging and routers might help clear things up:
http://shorewall.net/Documentation.html
and specifically
http://shorewall.net/bridge-Shorewall-perl.html
Hope this helps,
Kal
On 05/07/2010 07:26 AM, Jussi Hirvi wrote:
[root@farm1 log]# ip route show 62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 169.254.0.0/16 dev eth1 scope link default via 62.220.237.126 dev eth1
Yeah, so you have two interfaces on different IP networks. When someone connects to 62.236.221.67, the reply packets will still head out through 62.220.237.126 on eth1. That router probably filters the reply packets since they're from a non-local IP network.
I'm not sure if there's a simpler way to do this: When I have multi-homed servers I usually just use Shorewall to create two routing tables: one with a default route through each outbound router. Packets are marked based on their source address and routed based on those marks.
On Thu, 6 May 2010, Jussi Hirvi wrote:
I have a strange problem, where some clients see the website on my server and some do not. It is not about the iptables, and seems to be not about tcp wrapper. Still it is something within the box.
More details:
- the problem is only with some clients, with no geographical connection
between them; other clients see the website just fine
A while back, I remember there was a problem with TCP window scaling that would impact only some clients in a way that you describe:
http://lwn.net/Articles/92727/
Am 06.05.10 20:43, schrieb Paul Heinlein:
A while back, I remember there was a problem with TCP window scaling that would impact only some clients in a way that you describe:
Thank you, I was searching for that a few weeks ago, but didn't find it. Bookmarked.
Ralph
-
Benjamin Franz -
Didi Hoffmann -
Eduardo Grosclaude -
Gavin Carr -
Gordon Messmer -
John R Pierce -
JohnS -
Jussi Hirvi -
Jussi Hirvi -
Kahlil Hodgson -
kalinix -
Les Mikesell -
m.roth＠5-cent.us -
Paul Heinlein -
Philippe Naudin -
Ralph Angenendt -
Rob Kampen -
Ryan Manikowski -
Simon Billis -
Tony Molloy