I agree, BGP is important to route the IP's, but I've been exploring this same option with a different thought. I'd like to hear what others say about this!
Here is my plan (although not implemented or tested) for Web Services.
At our main data center we have the primary DNS server and our primary web server. The remote location houses the secondary DNS server and our secondary web server. Also at that second location is "hidden" master DNS server. Your registrar name records only point to the primary and secondary, therefore, the internet knows nothing of the hidden master. Then, the hidden master contains a similar set of DNS records that point to the secondary site. Here's the trick! The secondary DNS server syncs with the primary DNS server every x minutes. If the secondary DNS server cannot communicate with the primary DNS server, it then looks at the "hidden" master DNS server and syncs the records (which is pointing at your secondary site). Then, when your data center site comes back up, the secondary tries to communicate with the true master DNS server...it can...therefore it updates its records.
That is the theory in a nutshell. I've read that this is possible, but I haven't had a chance to test it.
What do others think about this? This is no substitution for BGP, but for those that don't run BGP or need to re-route the IP networks, this may work.
--Todd
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Bryan J. Smith Sent: Thursday, January 05, 2006 12:22 AM To: CentOS mailing list Subject: Re: [CentOS] High Availability using 2 sites
Tim Edwards tim@registriesltd.com.au wrote:
We currently have a backup site at a different location to our main site. This backup site mirrors (as closely as possible) our main services, particularly web serving. Is there a way to have the backup site act as a failover for the main site using something like Linux-HA? They are on seperate internet connections with different IP ranges.
Yes and no.
Yes in that you have a couple of options -- one common, one pretty much a hack.
The common one is to have your own autonomous system number and run BGP. That way you control your IP assignments, failover, etc... in ways that are efficient and quickly propogated.
The hack is to put routers and/or 1-to-1 NAT devices at each site, which can redirect traffic to the other site. That is less efficient and can cause some headaches.
No in the fact that there's really no "software" or "service" facility to deal with this. Round robin DNS does nothing to solve this. Name propogation is always an issue.
So it's something you can only address at the IP-level -- either by having your own, Internet-recognized autonomous system number, or redirecting IPs from each site to the other when servers/sites go down.
I agree, BGP is important to route the IP's, but I've been exploring this same option with a different thought. I'd like to hear what others say about this!
Here is my plan (although not implemented or tested) for Web Services.
At our main data center we have the primary DNS server and our primary web server. The remote location houses the secondary DNS server and our secondary web server. Also at that second location is "hidden" master DNS server. Your registrar name records only point to the primary and secondary, therefore, the internet knows nothing of the hidden master. Then, the hidden master contains a similar set of DNS records that point to the secondary site. Here's the trick! The secondary DNS server syncs with the primary DNS server every x minutes. If the secondary DNS server cannot communicate with the primary DNS server, it then looks at the "hidden" master DNS server and syncs the records (which is pointing at your secondary site). Then, when your data center site comes back up, the secondary tries to communicate with the true master DNS server...it can...therefore it updates its records.
That is the theory in a nutshell. I've read that this is possible, but I haven't had a chance to test it.
What do others think about this? This is no substitution for BGP, but for those that don't run BGP or need to re-route the IP networks, this may work.
propagation
Tom Brown tom.brown@goodtechnology.com wrote:
propagation
Damn me for taking the time to even explain it and list options. In 1 word, I think Tom Brown nails it on the head. Touche! ;->
Again, just to build upon Tom's word, because of "propogation," you're either going to have to setup your networks as part of the same, autonomous system (works absolutely), or use 1-to-1 NAT to point to the other network when the servers/facilites go down (works only when the NAT devices at the failed site are still up).
Most people don't realize that's the reality of the Internet. Otherwise, if DNS was real-time to the real authority everytime, the Internet would be a pretty slow place. ;->
On Thu, 5 Jan 2006, Bryan J. Smith wrote:
Tom Brown tom.brown@goodtechnology.com wrote:
propagation
Most people don't realize that's the reality of the Internet. Otherwise, if DNS was real-time to the real authority everytime, the Internet would be a pretty slow place. ;->
Some organizations set very low TTLs on the DNS records for their public hosts. Google's are only five minutes. That wouldn't allow real-time failover, but it's fairly quick as those things go.
Some organizations set very low TTLs on the DNS records for their public hosts. Google's are only five minutes. That wouldn't allow real-time failover, but it's fairly quick as those things go.
we do this BUT there are TONS of DNS servers out there that seem to ignore TTL's and don't get me started on web cache's
thanks
Tom Brown tom.brown@goodtechnology.com wrote:
we do this BUT there are TONS of DNS servers out there that seem to ignore TTL's and don't get me started on web
cache's
Exactomundo. Even Google has to accomodate such. That's why their model is piecemeal and localizes as much as they can.
But even Google has an ASN, AS15169, when it comes to their combined presence.
On Thu, 2006-01-05 at 12:18, Bryan J. Smith wrote:
Exactomundo. Even Google has to accomodate such. That's why their model is piecemeal and localizes as much as they can.
But even Google has an ASN, AS15169, when it comes to their combined presence.
I think they know better than to try to flap BGP routing around to accomodate a failed computer at one site or another, though. Is that what you are suggesting?
BGP would normally be used to handle routing over multiple paths to a fixed location and would change in response to the route availability. You can play tricks by shuffling a route to a completely new destination if a whole site fails but the minimum you could move would be a whole class C at a time, and some bad things will happen during the switch as different machines with the old IP's become visible.
Les Mikesell lesmikesell@gmail.com wrote:
I think they know better than to try to flap BGP routing around to accomodate a failed computer at one site or another, though. Is that what you are suggesting?
Did you even stop to take the *2* paragraphs I wrote into context? First paragraph ...
"Exactomundo. Even Google has to accomodate such. That's why their model is piecemeal and localizes as much as they can."
They localize ports of the Internet to localized servers. I'm sure it's a 3 or even 4-tier model, where 2 or even 3-tiers are in the same, physical layer-3 space.
If at all possible, you want to minimize flap BGP routing. I would NOT suggest you do otherwise. I'm purposely trying to point someone in the correct direction on obtaining an ASN and deploying external BGP, without assuming everyone on this list as the equivalent understanding of a CCDP or similar.
BGP would normally be used to handle routing over multiple paths to a fixed location and would change in response to the route availability.
Again, I'm talking more than just BGP, but an AS number assigned. I wasn't trying to get into the entire issue of internal v. external BGP, but the fact that AS numbers and BGP interaction between different AS systems is how you address redundant sites as if coming from the same network.
That's why I said ...
"But even Google has an ASN, AS15169, when it comes to their combined presence."
You can play tricks by shuffling a route to a completely
new
destination if a whole site fails but the minimum you could move would be a whole class C at a time, and some bad
things
will happen during the switch as different machines with
the
old IP's become visible.
Totally missed my points _entirely_!
Sigh, I'm not going to dissect how the Internet works with ASNs for everyone here to tell you that you're taking my comments and applying them _incorrectly_.
You can 1-to-1 NAT sites back to another, and that does the job. But that also requires you have equipment on the failed site that can do that. That's the problem if the site is taken off-line.
With an ASN, you don't need to localize equipment to do it. Damn me for even mentioning BGP, because you seem to have put your entire focus on BGP as the dynamic routing protocol, and not the concept of an AS. ;->
I think they know better than to try to flap BGP routing around to accomodate a failed computer at one site or another, though. Is that what you are suggesting?
BGP would normally be used to handle routing over multiple paths to a fixed location and would change in response to the route availability. You can play tricks by shuffling a route to a completely new destination if a whole site fails but the minimum you could move would be a whole class C at a time, and some bad things will happen during the switch as different machines with the old IP's become visible.
I'd venture to guess that Google's public IP's have absolutely nothing to do with their internal IP's, with the machine you actually reach being probably based on some hash function of your IP, your country, location, ISP, the time, the currently working google-servers, load on the servers, etc. With all this NAT happening on the edge of their network space. Very, very difficult to set up correctly, but once it's working it's VERY powerful. Add on to this BGP changing when entire data-centers fail with possible help from DNS updates and you have a pretty robust system.
Cheers, MaZe.
Maciej ¯enczykowski maze@cela.pl wrote:
and you have a pretty robust system.
That's exactly what I was talking about -- it's a combination of things. I _never_ said BGP solves them. I just said that the proper way is to get an AS (which involves deploying external BGP).
To cut to the chance, far too many people think you can do some server-side configurations and all of the sudden the Internet will change -- maybe because they understand simple NAT/PAT for their home or small office network. It's far, far, _far_ more complicated when you start talking about the Internet targetting _you_ as a "destination" on the Internet. ;->
And that means how you appear to the rest of the Internet as a routed, _layer-3_ destination (i.e., IP).
Hello Les,
Forgive me for this sidestep, but you are saying that Windows/IE actually ignores bad IP addresses if a site lists multiple IP's in a DNS lookup? I tried this approach for some redundancy a couple years ago and it didn't seem to work as you suggest. If it has indeed changed to work that way, this will help one of my clients immensely.
Mickael Maddison centos@silverservers.com wrote:
Hello Les, Forgive me for this sidestep, but you are saying that Windows/IE actually ignores bad IP addresses if a site
lists
multiple IP's in a DNS lookup? I tried this approach for some redundancy a couple years ago and it didn't seem to
work
as you suggest. If it has indeed changed to work that way, this will help one of my clients immensely.
No. The Windows resolver does not work so well. It's quite the opposite. The resolver and IP stack actually "holds down" names and IPs depending on availability -- for the _entire_ name. It also varies based whether it is DNS or ADS-integrated DNS.
There's about 2 dozen factors that I really don't want to get into on a Linux -- most specifically RHEL/CentOS -- list. You can really hack around the registry and change the attributes of the Windows resolver/stack, but at _no_ point have I _ever_ seen the resolver, stack or apps like MS IE ignore some IPs and use others, from DNS.
The only time I've seen MS IE do some interesting resolution is for ADS -- using RPC services, _not_ DNS (even if it is seemingly hitting a DNS server, it's ADS-integrated DNS and works very differently, long story).
On Thu, 2006-01-05 at 13:33, Mickael Maddison wrote:
Forgive me for this sidestep, but you are saying that Windows/IE actually ignores bad IP addresses if a site lists multiple IP's in a DNS lookup? I tried this approach for some redundancy a couple years ago and it didn't seem to work as you suggest. If it has indeed changed to work that way, this will help one of my clients immensely.
The versions of IE I've tried have been very good at this although it doesn't work for other stock windows apps. You may need to make sure that ICMP 'unreachable' packets are not firewalled so down hosts are detected quickly. You can test it easily by setting up a dummy DNS name with a bunch of A records where some point to working servers and some don't.
Les Mikesell lesmikesell@gmail.com wrote:
The versions of IE I've tried have been very good at this although it doesn't work for other stock windows apps. You may need to make sure that ICMP 'unreachable' packets are
not
firewalled so down hosts are detected quickly.
Okay, now you're getting somewhere. Before, all you were talking about was some arbitrary logic. If MS IE expects an ICMP unreachable to be returned before it attempts to use another host, then that would be doable. But even in that case, there is a lot of space between you and the end. And the terms in which a router or other device sends back that ICMP packet is a matter for yet further discussion.
In other words, it's still "arbitrary" compared to the absoluteness of a service at an IP. ;->
You can test it easily by setting up a dummy DNS name with
a
bunch of A records where some point to working servers and some don't.
That isn't a "real test" until you are connecting across many ISPs, through many layers of firewalls, routers, etc... As I said, much of this can be done when you control many aspects. You define the timeouts, authority, etc...
But for the arbitrary Internet, the terms in which even some ICMP messages are being returned are getting mighty arbitrary.
On Thursday 05 January 2006 13:51, Les Mikesell wrote:
BGP would normally be used to handle routing over multiple paths to a fixed location and would change in response to the route availability. You can play tricks by shuffling a route to a completely new destination if a whole site fails but the minimum you could move would be a whole class C at a time, and some bad things will happen during the switch as different machines with the old IP's become visible.
I don't know; BGP4 could do this, even though it is not the smoothest solution.
However, let me correct one point. On today's Internet, there is no such thing as a class C. Now, as to whether prefixes longer than /24 will be propagated or not, well, more than likely you won't get a prefix longer than /20 propagated through most AS's BGP tables these days. So route-flapping smaller CIDR blocks than /20 is futile unless you have good solid agreements with all the BGP-speaking AS's your traffic needs to traverse; that is, you'd need to make sure the AS's you care about propagate your prefixes in all their glory. Man, that would fit right in on NANOG, no?
Gepgraphically diverse HA solutions are fascinating to me; getting a working CentOS solution that is architecture agnostic would be fantastic.
Todd Reed treed@astate.edu wrote:
I agree, BGP is important to route the IP's, but I've been exploring this same option with a different thought.
I guess you missed my point. It's _not_ just a matter of using BGP for your dynamic routing. It's a matter of getting an assigned, autonomous system number so the Internet addressing your multiple networks as the same network.
[ There's a lot more to the Internet than just IPs ;-]
That's the proper way to do it.
I'd like to hear what others say about this!
I also made the suggestion to enable 1-to-1 NAT at each facility. Should the servers on one site go down, your 1-to-1 NAT devices would redirect requests to servers at the other site.
That doesn't require an additional, "external" registration/administration. Of course it means packets are now routed to your first site first, then your second site, so if the first site is wiped out (with no equipment), that doesn't help you.
Here is my plan (although not implemented or tested) for Web Services. At our main data center we have the primary DNS server and our primary web server. The remote location houses the secondary DNS server and our secondary web server. Also at that second location is "hidden" master DNS server. .. cut ... That is the theory in a nutshell. I've read that this is possible, but I haven't had a chance to test it.
The problem with the theory is that names are cached all over the Internet. That's why DNS server/name changes don't do squat when it comes to failover.
Now you could _consider_ setting a very low time-to-live (TTL) on your servers -- like 5 minutes. But that doesn't always work either.
What do others think about this? This is no substitution for BGP, but for those that don't run BGP or need to
re-route
the IP networks, this may work.
Again, it's _more_ than just BGP. ;->
You have to modify how the Internet sees you. Not just what you provide to the Internet. ;->
That's a key distinction that most people don't consider.
-
Bryan J. Smith -
Lamar Owen -
Les Mikesell -
Maciej Żenczykowski -
Mickael Maddison -
Paul Heinlein -
Todd Reed -
Tom Brown