What can I do to UNDERSTAND why, periodically, I can't reach www.centos.org (but everyone else can)?
Every few months, http://www.centos.org just drops off for me and for me only. It's not the web site, because I can reach it via a proxy (e.g., TOR browser bundle) and other tests show it to be alive (e.g., http://www.downforeveryoneorjustme.com/centos.org or http://traceroute.monitis.com).
But, for me, I just can't reach it by any means.
Mike Easter kindly determined "centos.org does not echo ICMP or UDP pings, only TCP, and its associated netblock level3 doesn't echo UDP, but does echo ICMP and TCP."
But, for me, this is what a traceroute returned just now on Centos:
$ traceroute www.centos.org ... early hops removed ... 16 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 211.850 ms ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 205.221 ms ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 207.392 ms 17 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 128.453 ms ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) 160.451 ms ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 160.437 ms 18 ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 165.923 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 165.920 ms 178.414 ms 19 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 241.537 ms 244.324 ms ae-3-80.edge3.Dallas1.Level3.net (4.69.145.136) 244.318 ms 20 * LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 244.291 ms 244.250 ms 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
Here is a screenshot (modified to remove early hops) using mtr: $ mtr www.centos.org http://www2.picturepush.com/photo/a/12303130/img/12303130.png
My related questions are: 1. What can I do (on Centos) to UNDERSTAND this problem? 2. Why does this problem happen every few months to me (and to just me)? 3. How can the net be this arbitrary and still work?
Rock wrote:
What can I do to UNDERSTAND why, periodically, I can't reach www.centos.org (but everyone else can)?
Every few months, http://www.centos.org just drops off for me and for me only. It's not the web site, because I can reach it via a proxy (e.g., TOR browser bundle) and other tests show it to be alive (e.g., http://www.downforeveryoneorjustme.com/centos.org or http://traceroute.monitis.com).
But, for me, I just can't reach it by any means.
Could you define "me"? Are you using this at home, or at work? What kind of connection? If this is work, are you paying for a fix IP?
Mike Easter kindly determined "centos.org does not echo ICMP or UDP pings, only TCP, and its associated netblock level3 doesn't echo UDP, but does echo ICMP and TCP."
But, for me, this is what a traceroute returned just now on Centos:
$ traceroute www.centos.org ... early hops removed ... 16 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 211.850 ms ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 205.221 ms ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 207.392 ms 17 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 128.453 ms ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) 160.451 ms ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 160.437 ms 18 ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 165.923 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 165.920 ms 178.414 ms 19 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 241.537 ms 244.324 ms ae-3-80.edge3.Dallas1.Level3.net (4.69.145.136) 244.318 ms 20 * LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 244.291 ms 244.250 ms
Interesting. I get to 20 (except for me it's 15), and www.centos.org is the next hop (16). Now, I *do* have fat pipes and good connections (this is a US fed gov't site), but still.... <snip>
My related questions are:
- What can I do (on Centos) to UNDERSTAND this problem?
- Why does this problem happen every few months to me (and to just me)?
- How can the net be this arbitrary and still work?
Have you spoken to your ISP? When you try the traceroute, and it fails, have you tried other well-known locations, like google.com, or slashdot.org?
Also, have you compared the contents of /etc/resolv.conf now, and again when centos.org isn't visible?
Sounds like either a firewall, spam, or nameserver issue.
mark
On Thu, 28 Feb 2013 13:40:55 -0500, m.roth-x6lchVBUigD1P9xLtpHBDw wrote:
Could you define "me"? Are you using this at home, or at work? What kind of connection? If this is work, are you paying for a fixed IP?
I have a fixed IP address at home.
I get to 20 (except for me it's 15), and www.centos.org is the next hop (16).
After hashing this out on alt.comp.freeware all morning, someone surmised that my static public IP address might have been added to a blocklist in the past few days, either by the penultimate hop or by centos.org itself.
So, the question morphs to whether or not we have tools at our disposal which can pinpoint which entity is blocking my IP address?
Note: I left a phone message at the DNS registrant for the penultimate hop and I left an email for the centos.org webmaster.
have you compared the contents of /etc/resolv.conf now, and again when centos.org isn't visible?
Well, I must admit I never even knew that file existed. Here is what it has in it:
$ cat /etc/resolv.conf # Generated by NetworkManager nameserver 192.168.1.1
Rock wrote:
On Thu, 28 Feb 2013 13:40:55 -0500, m.roth-x6lchVBUigD1P9xLtpHBDw wrote:
Could you define "me"? Are you using this at home, or at work? What kind of connection? If this is work, are you paying for a fixed IP?
I have a fixed IP address at home.
I get to 20 (except for me it's 15), and www.centos.org is the next hop (16).
After hashing this out on alt.comp.freeware all morning, someone surmised that my static public IP address might have been added to a blocklist in the past few days, either by the penultimate hop or by centos.org itself.
Oh. great. Have any bouncing mail when centos.org disappears? I have this happen a number of times a year, and yell about it every time. They use dnsorbs, who I say use a blacklisting methodology that worked 15 years ago, but is a complete disaster now, when one hosting co (as I have now), or ISP (say, roadrunner in Chicago, which provides one third or one half of *all* of Chicago's net access), and both have mail go through one mailserver address... and when a few idiots get their systems or sites compromised, dnsorbs instead of blocking each domain, blocs ALL the hundreds of thousands of folks innocently emailing through their service provider or hosting site. They don't seem to parse headers, and go *past* the mailserver to the culprit. Thank you, I prefer fail2ban's methodology.
So, the question morphs to whether or not we have tools at our disposal which can pinpoint which entity is blocking my IP address?
Note: I left a phone message at the DNS registrant for the penultimate hop and I left an email for the centos.org webmaster.
have you compared the contents of /etc/resolv.conf now, and again when centos.org isn't visible?
Well, I must admit I never even knew that file existed. Here is what it has in it:
$ cat /etc/resolv.conf # Generated by NetworkManager nameserver 192.168.1.1
Right, that's given you by your cable modem or router. Can you see what that is on that ISP-provided box? Alternatively, add well-known nameservers to that file, which will be overwritten when it's working again. For example, you could add nameserver 8.8.8.8 nameserver 8.8.4.4 (see https://developers.google.com/speed/public-dns/docs/using)
mark
On 2/28/2013 1:08 PM, m.roth@5-cent.us wrote:
They don't seem to parse headers, and go*past* the mailserver to the culprit.
you can't parse the headers until you read them, and you can't read the headers until you accept the incoming message. once you've accepted it, you can't bounce it back to the sending server via refusing the connection, and if you try and bounce it to the 'from' address you'll be spamming a lot of innocent parties who's email addresses have been forged on said headers.
so, if you use header parsing, all you can do is quietly drop the message.
On Thu, Feb 28, 2013 at 4:52 PM, John R Pierce pierce@hogranch.com wrote:
They don't seem to parse headers, and go*past* the mailserver to the culprit.
you can't parse the headers until you read them, and you can't read the headers until you accept the incoming message. once you've accepted it, you can't bounce it back to the sending server via refusing the connection, and if you try and bounce it to the 'from' address you'll be spamming a lot of innocent parties who's email addresses have been forged on said headers.
so, if you use header parsing, all you can do is quietly drop the message.
That's not true - there are several phases to smtp delivery and you can reject at most of them. You just have to have a mailer where you can control the operations at the right place. Using sendmail with MimeDefang as a milter gives you about as much control as possible if you want to parse/scan headers and content and react accordingly.
Am 28.02.2013 23:52, schrieb John R Pierce:
you can't parse the headers until you read them, and you can't read the headers until you accept the incoming message.
Not true. You can read the entire mail in the SMTP DATA phase and still reject it after the terminating single dot. Works perfectly fine on several MIMEDefang installations I set up to reject incoming mails containing malware or exceeding a certain SpamAssassin score.
HTH T.
Tilman Schmidt wrote:
Am 28.02.2013 23:52, schrieb John R Pierce:
you can't parse the headers until you read them, and you can't read the headers until you accept the incoming message.
Not true. You can read the entire mail in the SMTP DATA phase and still reject it after the terminating single dot. Works perfectly fine on several MIMEDefang installations I set up to reject incoming mails containing malware or exceeding a certain SpamAssassin score.
Right, I meant to respond to that, but forgot before I got home.
Look, somehow, someone, somewhere, has to decide they're receiving spam from an address... and the question is, *what* address. By trying to block what are allegedly "open relays", they're *also* blocking very large hosting and service providers, *all* of whose mail goes through that gateway. What *should* be reported to be blocked is the domain that's sending the spam.
Blocking an open relay should be done *only* on human investigation, to see whether that's the majority of what's coming out of there, and consideration of what the "relay" is, whether it's a known source, or an innocent large provider.
mark
On Fri, Mar 1, 2013 at 10:24 AM, m.roth@5-cent.us wrote:
Blocking an open relay should be done *only* on human investigation, to see whether that's the majority of what's coming out of there, and consideration of what the "relay" is, whether it's a known source, or an innocent large provider.
The argument on the other side of this is that the blacklist maintainers don't really block anything, they just make a lookup service available that corresponds to certain reported or potential problems. It is the mail delivery service provider that makes the decision to reject mail (or not) on behalf of the recipient and no one forces them to use the blacklists. If the recipient doesn't want this, they can use some other mail service or (sometimes) set up whitelists. A couple of decades ago I would have agreed that no one should reject but I think we've learned that you can't trust everyone with an IP address.
On 2/28/2013 9:12 AM, Rock wrote:
$ traceroutewww.centos.org ... early hops removed ... 16 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 211.850 ms ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 205.221 ms ...
thats a rather high hop count to get that far. I see...
$ traceroute www.centos.org traceroute to www.centos.org (72.232.194.162), 30 hops max, 40 byte packets 1 ge-0-0-1-414.er2.sjc1.got.net (207.111.214.242) 0.511 ms 0.440 ms 0.462 ms 2 ge-0-0-1-10.cr1.scz1.got.net (207.111.198.49) 3.961 ms 3.956 ms 4.193 ms 3 ge-1-3-0-745.cr1.sfo1.got.net (207.111.219.131) 7.796 ms 7.784 ms 7.765 ms 4 vlan114.car2.SanFrancisco1.Level3.net (4.53.130.89) 28.162 ms 28.165 ms 29.320 ms 5 ae-2-4.bar2.SanFrancisco1.Level3.net (4.69.133.158) 8.636 ms 8.621 ms 8.594 ms 6 ae-6-6.ebr2.SanJose1.Level3.net (4.69.140.154) 50.823 ms 50.664 ms 50.917 ms 7 ae-2-2.ebr2.SanJose5.Level3.net (4.69.148.141) 50.709 ms 50.545 ms 50.533 ms 8 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 50.560 ms 50.566 ms 50.543 ms 9 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 50.524 ms 50.418 ms 50.407 ms 10 ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 50.373 ms 50.310 ms ae-63-63.csw1.Dallas1.Level3.net (4.69.151.133) 56.827 ms 11 ae-4-90.edge3.Dallas1.Level3.net (4.69.145.200) 50.268 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 50.282 ms ae-2-70.edge3.Dallas1.Level3.net (4.69.145.72) 50.335 ms 12 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 50.386 ms 52.124 ms 51.458 ms 13 www.centos.org (72.232.194.162) 51.202 ms !X 51.118 ms !X 51.113 ms !X
which is only 8 hops to that same LosAngeles1 gateway.
anyways, I never have any problems seeing it, so my guess is, your problem is in the first 15 hops you didn't show.
btw, the traceroute worked from my host using both -I and -T (icmp and tcp respectively) traces.
On Thu, 28 Feb 2013 11:06:35 -0800, John R Pierce wrote:
thats a rather high hop count to get that far.
Assuming you're in Santa Cruz, I'm not more than 20 miles from you, but, I'm using a mountain WISP (with radio relays on all the hilltops), so that's likely why I have many more hops than you do.
Here is my traceroute with a bit more detail (my local stuff is redacted, for privacy).
$ date Thu Feb 28 12:54:47 PST 2013
$ traceroute www.centos.org traceroute to www.centos.org (72.232.194.162), 30 hops max, 60 byte packets 1 router (192.168.1.1) 27.845 ms 27.807 ms 27.787 ms 2 REDACTED STATIC IP ADDRESS 27.766 ms 27.745 ms 27.725 ms 3 10.10.0.1 (10.10.0.1) 45.639 ms 51.532 ms 51.523 ms 4 10.52.0.1 (10.52.0.1) 51.505 ms 53.765 ms 53.759 ms 5 10.30.0.1 (10.30.0.1) 59.289 ms 62.681 ms 62.672 ms 6 10.0.0.1 (10.0.0.1) 65.632 ms 114.790 ms 124.655 ms 7 REDACTED WISP 127.280 ms 127.282 ms 127.266 ms 8 REDACTED WISP backbone 127.247 ms 121.642 ms 48.416 ms 9 REDACTED WISP backbone 51.977 ms 55.483 ms 55.473 ms 10 REDACTED WISP backbone 64.521 ms 67.369 ms 67.352 ms 11 xe-9-0-0.edge1.SanJose3.Level3.net (4.68.110.49) 64.463 ms 67.261 ms xe-11-1-0.edge1.SanJose3.Level3.net (4.68.111.189) 69.552 ms 12 vlan60.csw1.SanJose1.Level3.net (4.69.152.62) 88.431 ms vlan70.csw2.SanJose1.Level3.net (4.69.152.126) 91.063 ms vlan90.csw4.SanJose1.Level3.net (4.69.152.254) 90.976 ms 13 ae-62-62.ebr2.SanJose1.Level3.net (4.69.153.17) 96.160 ms ae-61-61.ebr1.SanJose1.Level3.net (4.69.153.1) 90.955 ms ae-82-82.ebr2.SanJose1.Level3.net (4.69.153.25) 90.896 ms 14 ae-5-5.ebr1.SanJose5.Level3.net (4.69.148.137) 76.111 ms ae-2-2.ebr2.SanJose5.Level3.net (4.69.148.141) 89.501 ms ae-5-5.ebr1.SanJose5.Level3.net (4.69.148.137) 76.081 ms 15 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 76.597 ms 76.026 ms 82.326 ms 16 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 82.317 ms ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 95.557 ms ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 95.517 ms 17 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 95.466 ms 97.991 ms 95.449 ms 18 ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 97.945 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 97.927 ms ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 95.342 ms 19 ae-4-90.edge3.Dallas1.Level3.net (4.69.145.200) 97.833 ms LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 97.845 ms ae-3-80.edge3.Dallas1.Level3.net (4.69.145.136) 62.875 ms 20 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 77.207 ms 77.174 ms * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
What I now know (that I had not known prior), is that the hop you see at 20 is the penultimate hop.
So either hop 20 is not forwarding the packets, or the final destination is not acknowledging them.
Someone suggested I might be on a blacklist, somehow, and that, at least, would make technical sense.
On 01/03/13 08:01, Rock wrote:
So either hop 20 is not forwarding the packets, or the final destination is not acknowledging them.
Do you always have the same number of hops? Wondering if the WISPs you bounce through might change. Perhaps one of those is problematic.
Someone suggested I might be on a blacklist, somehow, and that, at least, would make technical sense.
You could try checking something like
to rule out blacklisting as the issue.
K
On Thu, Feb 28, 2013 at 4:01 PM, Kahlil Hodgson kahlil.hodgson@dealmax.com.au wrote:
Do you always have the same number of hops? Wondering if the WISPs you bounce through might change. Perhaps one of those is problematic.
Someone suggested I might be on a blacklist, somehow, and that, at least, would make technical sense.
You could try checking something like
http://multirbl.valli.org/to rule out blacklisting as the issue.
The only blacklists I know about involve mail and require the receiving machines to actively choose to do lookups and reject based on the result. Nothing should be dropping/blocking other types of traffic. There are an assortment of 'looking glass' sites on the internet where you can originate traceroutes and check that BGP routes for source and targets are being propagated. This seems to be a directory of those sites: http://www.lookinglass.org/
On Thu, 28 Feb 2013 16:56:44 -0600, Les Mikesell wrote:
This seems to be a directory of those sites: http://www.lookinglass.org/
Thanks. I had never heard of this. Going there, it wasn't obvious what to do, so, I'll hunt and peck a bit.
On 01/03/13 09:56, Les Mikesell wrote:
Someone suggested I might be on a blacklist, somehow, and that, at least, would make technical sense.
You could try checking something like
http://multirbl.valli.org/to rule out blacklisting as the issue.
The only blacklists I know about involve mail and require the receiving machines to actively choose to do lookups and reject based on the result. Nothing should be dropping/blocking other types of traffic.
Very true. I thinking that if he _was_ on an email blacklist, then his network might _also_ have been flagged as bad (by something else). Because I didn't know about ...
There are an assortment of 'looking glass' sites on the internet where you can originate traceroutes and check that BGP routes for source and targets are being propagated. This seems to be a directory of those sites: http://www.lookinglass.org/
Hey, that is very cool. Did not know such a thing existed.
Thanks for pointing that out ;-)
K
On Fri, 01 Mar 2013 09:01:03 +1100, Kahlil Hodgson wrote:
Do you always have the same number of hops? Wondering if the WISPs you bounce through might change. Perhaps one of those is problematic.
Dunno the exact number of hops but it's always long but I only have 1 WISP so I don't have a choice.
It goes from my laptop to my home broadband router to my radio & antenna on the roof to the WISP antenna and at that point it bounces a while within the WISP's network before exiting on some backbone on it's merry way to centos.
The ONE STOP prior to centos.org is the last stop; but all other web sites work. So, my "assumption" is that Centos.org is blocking me. I didn't think about this when I first posted this thread, but then someone told me that they ran a traceroute and there was only ONE STOP after where it stopped for me, so, that's when the light bulb lit.
Now, as to WHY they're blocking me - that I asked their webmaster by mail, earlier today.
You could try checking something like http://multirbl.valli.org/
Nice site. I see blacklists, whitelists, Yellowlists, neutralized, and even not listed as reports, and ALL (even the not listed) came up with zero.
On 03/01/2013 02:09 AM, Rock wrote:
On Fri, 01 Mar 2013 09:01:03 +1100, Kahlil Hodgson wrote:
Do you always have the same number of hops? Wondering if the WISPs you bounce through might change. Perhaps one of those is problematic.
Dunno the exact number of hops but it's always long but I only have 1 WISP so I don't have a choice.
It goes from my laptop to my home broadband router to my radio & antenna on the roof to the WISP antenna and at that point it bounces a while within the WISP's network before exiting on some backbone on it's merry way to centos.
The ONE STOP prior to centos.org is the last stop; but all other web sites work. So, my "assumption" is that Centos.org is blocking me. I didn't think about this when I first posted this thread, but then someone told me that they ran a traceroute and there was only ONE STOP after where it stopped for me, so, that's when the light bulb lit.
Now, as to WHY they're blocking me - that I asked their webmaster by mail, earlier today.
You could try checking something like http://multirbl.valli.org/
Nice site. I see blacklists, whitelists, Yellowlists, neutralized, and even not listed as reports, and ALL (even the not listed) came up with zero.
It is worth noting that www.centos.org was inaccessible from Serbia and http://www.downforeveryoneorjustme.com/centos.org on 27.01.2013 around 16:30, for about 1-2 hours.
On Fri, 01 Mar 2013 13:57:26 +0100, Ljubomir Ljubojevic wrote:
It is worth noting that www.centos.org was inaccessible from Serbia and http://www.downforeveryoneorjustme.com/centos.org on 27.01.2013 around 16:30, for about 1-2 hours.
I can see pretty bad connectivity for Centos.org over here: http://www.isitdownrightnow.com/centos.org.html
Here's a screenshot of the results: http://www1.picturepush.com/photo/a/12306664/img/12306664.png
I'm not on any of these blacklists: http://multirbl.valli.org http://www.lookinglass.org
In addition, I had asked some friends, who reported back: APPLE:
For what it is worth I can't get to centos.org from Apple's network either. The traceroute finally gets through after 42 hops but the servers are dropping the connection immediately. Me thinks it is a centros.org problem and not an ISP issue.
MAXIM:
Ha - just checked. I can't from Maxim either.
GOOGLE:
Works OK for me from google, as long as I spell it right this time!
A neighborly friend tested similar situations:
While it is possible that the laptop had a temporary problem wherein it had set the TimeToLive (TTL) of the packets to 19, that also seems like a long shot.
However, trying a reboot if/when it happens again to rule out laptop fatigue is another test to try. Also, finding a site that has more than 20 hops would also be a test of that. It turns out it isn't all that easy to find a site with that many hops. Traceroute deliberately plays with the TTL number, so it would seem like a long shot if traceroute also showed the problem, as it does in this case.
Nonetheless, if it happens again, try a traceroute to tempotv.com.tr, which takes 23 hops from my computer.
I can get to Iran in 24 hops: traceroute www.gu.ac.ir. On the Internet, centos.org is as far away as Pakistan: traceroute www.gu.edu.pk.
Some routers drop ICMP packets when they get busy, or rate-limit the replies. You could have been trying to get to centos.org while they were being hit by a DDOS attack. Routers would automatically drop whole blocks of IPs for a short while to cope with the problem.
...and just for fun, let's try Malaysia:
C:\temp>tracert -h 255 www.tourism.gov.my
Tracing route to www.tourism.gov.my [168.63.252.50] over a maximum of 255 hops:
try a traceroute to tempotv.com.tr, which takes 23 hops from my computer.
For the record, I was unable to get to any of these long-hop destinations: $ traceroute tempotv.com.tr ==> died on the 23rd hop $ traceroute -I www.centos.org ==> died on the 19th (penultimate) hop $ traceroute www.gu.ac.ir ==> Iran, died on the 22nd hop $ traceroute www.gu.edu.pk ==> Pakistan, made it only to the 19th hop $ traceroute -m 255 www.tourism.gov.my ==> Malaysia died on the 19th hop
All the above died before reaching their destinations.
Rock wrote:
try a traceroute to tempotv.com.tr, which takes 23 hops from my computer.
For the record, I was unable to get to any of these long-hop destinations: $ traceroute tempotv.com.tr ==> died on the 23rd hop $ traceroute -I www.centos.org ==> died on the 19th (penultimate) hop $ traceroute www.gu.ac.ir ==> Iran, died on the 22nd hop $ traceroute www.gu.edu.pk ==> Pakistan, made it only to the 19th hop $ traceroute -m 255 www.tourism.gov.my ==> Malaysia died on the 19th hop
All the above died before reaching their destinations.
Back to the question: have you spoken to level two, at least, tech support of your provider of network access, to see if they have some explanation?
mark
On Fri, 01 Mar 2013 11:26:27 -0500, m.roth-x6lchVBUigD1P9xLtpHBDw wrote:
Back to the question: have you spoken to level two, at least, tech support of your provider of network access, to see if they have some explanation?
Hi Mark,
My WISP provider is as perplexed as I because I'm the first subscriber to have this problem, he says.
He thinks it's on the Centos side of things.
On Fri, Mar 1, 2013 at 10:38 AM, Rock RockSockDoc@gmail.com wrote:
Back to the question: have you spoken to level two, at least, tech support of your provider of network access, to see if they have some explanation?
My WISP provider is as perplexed as I because I'm the first subscriber to have this problem, he says.
He thinks it's on the Centos side of things.
Some of the Centos mirrors sites are insanely fast - maybe when they are syncing stuff out it overloads some intermediate connections.
Rock wrote:
On Fri, 01 Mar 2013 11:26:27 -0500, m.roth-x6lchVBUigD1P9xLtpHBDw wrote:
Back to the question: have you spoken to level two, at least, tech support of your provider of network access, to see if they have some explanation?
My WISP provider is as perplexed as I because I'm the first subscriber to have this problem, he says.
He thinks it's on the Centos side of things.
Does tech support have the same problem?
I mean, here on our internal network, some idiot blocked the BBC; after I put in a ticket, they unblocked it... then contacted me, and I found the main page was unblocked... but *every* article link was still blocked, and the guy I was talking to told me he could see them... but someone he was working with couldn't.... They did, finally fix it, and claimed it had happened to to a phishing attack....
mark
Am 01.03.2013 17:38, schrieb Rock:
My WISP provider is as perplexed as I because I'm the first subscriber to have this problem, he says.
Yeah, providers would say that. Several times in a row to different subscribers, if necessary.
He thinks it's on the Centos side of things.
Indeed. I know at least three big providers hereabout for whom I routinely warn my customers if they have to call support that the first answer will invariably be: "Everything's ok at our side, the problem must be with your equipment/your communication partner/anywhere but us."
Good support engineers don't make statements about *where* the problem is before they have determined *what* the problem is.
SCNR
On Fri, Mar 1, 2013 at 10:19 AM, Rock RockSockDoc@gmail.com wrote:
try a traceroute to tempotv.com.tr, which takes 23 hops from my computer.
For the record, I was unable to get to any of these long-hop destinations: $ traceroute tempotv.com.tr ==> died on the 23rd hop $ traceroute -I www.centos.org ==> died on the 19th (penultimate) hop $ traceroute www.gu.ac.ir ==> Iran, died on the 22nd hop $ traceroute www.gu.edu.pk ==> Pakistan, made it only to the 19th hop $ traceroute -m 255 www.tourism.gov.my ==> Malaysia died on the 19th hop
All the above died before reaching their destinations.
Note that traceroute requires a round-trip for you to see success. A failure is almost as likely to mean that the router where it stops does not have a route back to the source as that it can't reach the next forward hop. Those 'looking glass' sites can show you if the bgp routes are propagating to various locations.
On Fri, 01 Mar 2013 10:27:12 -0600, Les Mikesell wrote:
Those 'looking glass' sites can show you if the bgp routes are propagating to various locations.
I saw the reference to the lookingglass site, e.g., http://www.lookinglass.org But, I'm sorry ... I'm totally clueless as to how to properly USE them.
May I ask: Q: Which of the IP addresses below do I put into lookingglass to gather data?
Given that these all died as follows: $ traceroute tempotv.com.tr ==> died on the 23rd hop 19 82.222.13.145 (82.222.13.145) 262.456 ms 82.222.3.117 (82.222.3.117) 270.129 ms 273.278 ms 20 82.222.13.106 (82.222.13.106) 276.814 ms 276.812 ms 276.797 ms 21 82.222.253.82 (82.222.253.82) 273.131 ms 264.450 ms 247.432 ms 22 82.222.254.210 (82.222.254.210) 247.354 ms 247.362 ms 82.222.224.234 (82.222.224.234) 309.858 ms 23 asy60.asy53.tellcom.com.tr (92.45.53.60) 326.022 ms 329.493 ms 329.483 ms 24 * * * (and so on)
$ traceroute -I www.centos.org ==> died on the 19th (penultimate) hop 15 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 60.128 ms 62.055 ms 63.873 ms 16 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 64.999 ms 56.867 ms 59.059 ms 17 ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) 60.437 ms 62.928 ms 57.975 ms 18 ae-2-70.edge3.Dallas1.Level3.net (4.69.145.72) 59.828 ms 55.071 ms 57.122 ms 19 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 56.049 ms 65.838 ms 58.644 ms 20 * * * (and so on)
$ traceroute www.gu.ac.ir ==> Iran, died on the 22nd hop 14 rostelecom-ic-300487-ffm-b11.c.telia.net (213.248.96.202) 197.545 ms 197.545 ms 284.625 ms 15 95.167.92.186 (95.167.92.186) 324.490 ms 324.486 ms 324.468 ms 16 95.167.14.198 (95.167.14.198) 378.393 ms 378.393 ms 378.378 ms 17 * * * 18 195.146.33.29 (195.146.33.29) 369.322 ms 369.306 ms 300.357 ms 19 78.38.245.228 (78.38.245.228) 300.263 ms 300.264 ms 306.080 ms 20 78.38.244.250 (78.38.244.250) 290.148 ms 300.201 ms 300.198 ms 21 89.221.87.25 (89.221.87.25) 338.022 ms 338.042 ms 337.997 ms 22 95.38.120.254 (95.38.120.254) 330.673 ms 330.681 ms 330.669 ms 23 * * * (and so on)
$ traceroute www.gu.edu.pk ==> Pakistan, died on the 19th hop 10 charter-peer.sv1.layer42.net (69.36.225.18) 45.929 ms 48.200 ms 48.191 ms 11 bbr02snjsca-bue-6.snjs.ca.charter.com (96.34.3.2) 48.173 ms 50.328 ms 52.805 ms 12 bbr01dnvrco-bue-2.dnvr.co.charter.com (96.34.0.3) 81.440 ms 81.442 ms 81.423 ms 13 bbr01olvemo-bue-1.olve.mo.charter.com (96.34.0.145) 112.975 ms 112.900 ms 112.801 ms 14 bbr01blvlil-tge-0-15-0-4.blvl.il.charter.com (96.34.0.111) 129.811 ms 129.811 ms 93.669 ms 15 bbr01atlnga-tge-0-0-0-9.atln.ga.charter.com (96.34.0.120) 122.469 ms 122.468 ms 146.517 ms 16 crr01sghlga-bue-3.sghl.ga.charter.com (96.34.2.71) 123.937 ms 123.943 ms 127.972 ms 17 acr02atlnga-tge-0-0-0-0.atln.ga.charter.com (96.34.72.179) 130.189 ms 132.137 ms 132.136 ms 18 75-131-187-34.static.gwnt.ga.charter.com (75.131.187.34) 129.996 ms 129.998 ms 129.978 ms 19 67.23.161.143 (67.23.161.143) 129.894 ms 129.873 ms 129.859 ms 20 * * * (and so on)
$ traceroute -m 255 www.tourism.gov.my ==> Malaysia died on the 19th hop 11 gi9-0-0.cr2.nrt1.asianetcom.net (202.147.50.134) 244.797 ms 244.804 ms 244.797 ms 12 ge-0-0-0-0.gw3.nrt4.asianetcom.net (202.147.0.226) 215.347 ms 215.348 ms 220.710 ms 13 61.8.59.22 (61.8.59.22) 267.994 ms 268.004 ms 267.960 ms 14 ge-2-3-0-0.sin-64cbe-1a.ntwk.msn.net (207.46.41.75) 325.436 ms 259.629 ms 259.605 ms 15 10.22.212.213 (10.22.212.213) 217.562 ms 10.22.212.209 (10.22.212.209) 273.695 ms ge-5-2-0-0.sin-64cbe-1b.ntwk.msn.net (207.46.41.26) 265.336 ms 16 10.175.56.241 (10.175.56.241) 265.273 ms * 10.175.56.49 (10.175.56.49) 272.494 ms 17 10.175.61.5 (10.175.61.5) 282.798 ms 10.175.60.7 (10.175.60.7) 248.975 ms 10.241.90.205 (10.241.90.205) 258.597 ms 18 10.78.236.2 (10.78.236.2) 258.541 ms 10.78.220.2 (10.78.220.2) 211.475 ms 10.241.88.59 (10.241.88.59) 220.578 ms 19 * * 10.78.246.2 (10.78.246.2) 239.054 ms 20 * * * (and so on)
On Fri, Mar 1, 2013 at 10:47 AM, Rock RockSockDoc@gmail.com wrote:
Those 'looking glass' sites can show you if the bgp routes are propagating to various locations.
I saw the reference to the lookingglass site, e.g., http://www.lookinglass.org But, I'm sorry ... I'm totally clueless as to how to properly USE them.
May I ask: Q: Which of the IP addresses below do I put into lookingglass to gather data?
If you thing the problem is on your side, try ping/traceroute to your own IP from some other locations.
Given that these all died as follows: $ traceroute tempotv.com.tr ==> died on the 23rd hop 19 82.222.13.145 (82.222.13.145) 262.456 ms 82.222.3.117 (82.222.3.117) 270.129 ms 273.278 ms 20 82.222.13.106 (82.222.13.106) 276.814 ms 276.812 ms 276.797 ms 21 82.222.253.82 (82.222.253.82) 273.131 ms 264.450 ms 247.432 ms 22 82.222.254.210 (82.222.254.210) 247.354 ms 247.362 ms 82.222.224.234 (82.222.224.234) 309.858 ms 23 asy60.asy53.tellcom.com.tr (92.45.53.60) 326.022 ms 329.493 ms 329.483 ms 24 * * * (and so on)
It is very common for organizations to block ICMP at their private border routers and just let a few port/address combinations through for the public services they provide, so not reaching a target does not mean there is a problem. Sometimes you can reach the target with tcptracroute (or traceroute -T -p port), but some intermediate hops may not show because their ICMP replies are blocked.
Organizations that are multi-homed and do their own BGP routing will have fairly specific routes showing up in the bgp tables that you can query from the looking glass tools and if they fall completely off the grid the routes will disappear.
On 03/01/2013 10:47 AM, Rock wrote:
On Fri, 01 Mar 2013 10:27:12 -0600, Les Mikesell wrote:
Those 'looking glass' sites can show you if the bgp routes are propagating to various locations.
I saw the reference to the lookingglass site, e.g., http://www.lookinglass.org But, I'm sorry ... I'm totally clueless as to how to properly USE them.
May I ask: Q: Which of the IP addresses below do I put into lookingglass to gather data?
Given that these all died as follows: $ traceroute tempotv.com.tr ==> died on the 23rd hop 19 82.222.13.145 (82.222.13.145) 262.456 ms 82.222.3.117 (82.222.3.117) 270.129 ms 273.278 ms 20 82.222.13.106 (82.222.13.106) 276.814 ms 276.812 ms 276.797 ms 21 82.222.253.82 (82.222.253.82) 273.131 ms 264.450 ms 247.432 ms 22 82.222.254.210 (82.222.254.210) 247.354 ms 247.362 ms 82.222.224.234 (82.222.224.234) 309.858 ms 23 asy60.asy53.tellcom.com.tr (92.45.53.60) 326.022 ms 329.493 ms 329.483 ms 24 * * * (and so on)
$ traceroute -I www.centos.org ==> died on the 19th (penultimate) hop 15 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 60.128 ms 62.055 ms 63.873 ms 16 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 64.999 ms 56.867 ms 59.059 ms 17 ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) 60.437 ms 62.928 ms 57.975 ms 18 ae-2-70.edge3.Dallas1.Level3.net (4.69.145.72) 59.828 ms 55.071 ms 57.122 ms 19 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 56.049 ms 65.838 ms 58.644 ms 20 * * * (and so on)
$ traceroute www.gu.ac.ir ==> Iran, died on the 22nd hop 14 rostelecom-ic-300487-ffm-b11.c.telia.net (213.248.96.202) 197.545 ms 197.545 ms 284.625 ms 15 95.167.92.186 (95.167.92.186) 324.490 ms 324.486 ms 324.468 ms 16 95.167.14.198 (95.167.14.198) 378.393 ms 378.393 ms 378.378 ms 17 * * * 18 195.146.33.29 (195.146.33.29) 369.322 ms 369.306 ms 300.357 ms 19 78.38.245.228 (78.38.245.228) 300.263 ms 300.264 ms 306.080 ms 20 78.38.244.250 (78.38.244.250) 290.148 ms 300.201 ms 300.198 ms 21 89.221.87.25 (89.221.87.25) 338.022 ms 338.042 ms 337.997 ms 22 95.38.120.254 (95.38.120.254) 330.673 ms 330.681 ms 330.669 ms 23 * * * (and so on)
$ traceroute www.gu.edu.pk ==> Pakistan, died on the 19th hop 10 charter-peer.sv1.layer42.net (69.36.225.18) 45.929 ms 48.200 ms 48.191 ms 11 bbr02snjsca-bue-6.snjs.ca.charter.com (96.34.3.2) 48.173 ms 50.328 ms 52.805 ms 12 bbr01dnvrco-bue-2.dnvr.co.charter.com (96.34.0.3) 81.440 ms 81.442 ms 81.423 ms 13 bbr01olvemo-bue-1.olve.mo.charter.com (96.34.0.145) 112.975 ms 112.900 ms 112.801 ms 14 bbr01blvlil-tge-0-15-0-4.blvl.il.charter.com (96.34.0.111) 129.811 ms 129.811 ms 93.669 ms 15 bbr01atlnga-tge-0-0-0-9.atln.ga.charter.com (96.34.0.120) 122.469 ms 122.468 ms 146.517 ms 16 crr01sghlga-bue-3.sghl.ga.charter.com (96.34.2.71) 123.937 ms 123.943 ms 127.972 ms 17 acr02atlnga-tge-0-0-0-0.atln.ga.charter.com (96.34.72.179) 130.189 ms 132.137 ms 132.136 ms 18 75-131-187-34.static.gwnt.ga.charter.com (75.131.187.34) 129.996 ms 129.998 ms 129.978 ms 19 67.23.161.143 (67.23.161.143) 129.894 ms 129.873 ms 129.859 ms 20 * * * (and so on)
$ traceroute -m 255 www.tourism.gov.my ==> Malaysia died on the 19th hop 11 gi9-0-0.cr2.nrt1.asianetcom.net (202.147.50.134) 244.797 ms 244.804 ms 244.797 ms 12 ge-0-0-0-0.gw3.nrt4.asianetcom.net (202.147.0.226) 215.347 ms 215.348 ms 220.710 ms 13 61.8.59.22 (61.8.59.22) 267.994 ms 268.004 ms 267.960 ms 14 ge-2-3-0-0.sin-64cbe-1a.ntwk.msn.net (207.46.41.75) 325.436 ms 259.629 ms 259.605 ms 15 10.22.212.213 (10.22.212.213) 217.562 ms 10.22.212.209 (10.22.212.209) 273.695 ms ge-5-2-0-0.sin-64cbe-1b.ntwk.msn.net (207.46.41.26) 265.336 ms 16 10.175.56.241 (10.175.56.241) 265.273 ms * 10.175.56.49 (10.175.56.49) 272.494 ms 17 10.175.61.5 (10.175.61.5) 282.798 ms 10.175.60.7 (10.175.60.7) 248.975 ms 10.241.90.205 (10.241.90.205) 258.597 ms 18 10.78.236.2 (10.78.236.2) 258.541 ms 10.78.220.2 (10.78.220.2) 211.475 ms 10.241.88.59 (10.241.88.59) 220.578 ms 19 * * 10.78.246.2 (10.78.246.2) 239.054 ms 20 * * * (and so on)
We have a dynamic iptables blocking script on the site that adds IPs that do lots of nmap scans, try to do xss, spam the forums, etc. If you e-mail me your IP at johnny@centos.org off list, I'll take a look and see if you are being blocked by that.
Thanks, Johnny Hughes
Rock wrote:
For the record, I was unable to get to any of these long-hop destinations: $ traceroute tempotv.com.tr ==> died on the 23rd hop $ traceroute -I www.centos.org ==> died on the 19th (penultimate) hop $ traceroute www.gu.ac.ir ==> Iran, died on the 22nd hop $ traceroute www.gu.edu.pk ==> Pakistan, made it only to the 19th hop $ traceroute -m 255 www.tourism.gov.my ==> Malaysia died on the 19th hop
All the above died before reaching their destinations.
Your traceroute is UDP. None of those answer UDP. All but the last one will answer ICMP.
If you use your mtr on them in default ICMP mode, all but the last will ICMP traceroute. The last one needs tcptraceroute.
traceroute is not the best tool for some purposes. All but the last of those will ping ICMP.
-
John R Pierce -
Johnny Hughes -
Kahlil Hodgson -
Les Mikesell -
Ljubomir Ljubojevic -
m.roth＠5-cent.us -
Mike Easter -
Rock -
Tilman Schmidt