Karabir suggested we start getting together a list of steps that need completion to deliver an IPv6-preferenced or IPv6-only list of mirrors for IPv6-only clients using CentOS. Here's what I've got so far. Feel free to add to it:
Steps are in order of priority:
- Establish at least two authoritative name servers for centos.org with v6 IPs. - Determine if centos.org registrar supports v6 glue, transfer to a registrar that supports it if needed, then add the glue. - Augment the current mirror database with a field for IPv6 support and IPv4 support, filling in IPv4 as enabled for all existing mirrors. - Build a list of centos mirrors which have AAAA records and/or consult with mirror admins to identify the mirrors which are prepared to provide preferred IPv6 connectivity to IPv6 clients. - Prepare a separate monitoring system for IPv6 content to confirm that mirrors are serving the content correctly over both v4 and v6. - Modify the mirror list on mirrorlist.centos.org to provide IPv6 mirrors with a higher priority than IPv4 mirrors when the client accesses it via its AAAA address rather than A. - Add AAAA for mirrorlist.centos.org
Thoughts?
On Mon, Aug 01, 2011 at 02:16:30PM -0500, Kevin Stange wrote:
Karabir suggested we start getting together a list of steps that need completion to deliver an IPv6-preferenced or IPv6-only list of mirrors for IPv6-only clients using CentOS.
FWIW, Fedora hasn't had a significant need to IPv6-preferenced of IPv6-only mirrors. We do manage preference by ASN, netblocks (IPv4 and IPv6), internet2 (and related networks) and country. Fedora does have nameservers advertised on AAAA records, as well as A records. The inbound web proxies are reachable via both AAAA and A records, so MirrorManager does see a client's IPv6 or IPv4 address.
MirrorManager replies to client requests with DNS names. Mirrors may themselves advertise a given name with an AAAA or A record.
Here's what I've got so far. Feel free to add to it:
Steps are in order of priority:
- Establish at least two authoritative name servers for centos.org with
v6 IPs.
- Determine if centos.org registrar supports v6 glue, transfer to a
registrar that supports it if needed, then add the glue.
- Augment the current mirror database with a field for IPv6 support and
IPv4 support, filling in IPv4 as enabled for all existing mirrors.
- Build a list of centos mirrors which have AAAA records and/or consult
with mirror admins to identify the mirrors which are prepared to provide preferred IPv6 connectivity to IPv6 clients.
- Prepare a separate monitoring system for IPv6 content to confirm that
mirrors are serving the content correctly over both v4 and v6.
Right now Fedora's MM crawler runs on machines that can only speak IPv4. I trust mirror admins that if they're advertising both AAAA and A records, that both methods are equivalent and if one works, the other works. I haven't had a significant problem with this in practice.
- Modify the mirror list on mirrorlist.centos.org to provide IPv6
mirrors with a higher priority than IPv4 mirrors when the client accesses it via its AAAA address rather than A.
- Add AAAA for mirrorlist.centos.org
Thoughts?
Are there a bunch of mirrors for whom IPv6 bandwidth is "free", compared to IPv4? That's the only reason I can see going to this effort. I understand that bandwidth via Internet2 would be cheaper than commercial links, which is why MM sends I2-capable clients to I2-capable servers first. Otherwise, seems like a lot of work for not necessarily any gain.
Thanks, Matt
On 08/01/2011 03:37 PM, Matt Domsch wrote:
On Mon, Aug 01, 2011 at 02:16:30PM -0500, Kevin Stange wrote:
Karabir suggested we start getting together a list of steps that need completion to deliver an IPv6-preferenced or IPv6-only list of mirrors for IPv6-only clients using CentOS.
FWIW, Fedora hasn't had a significant need to IPv6-preferenced of IPv6-only mirrors. We do manage preference by ASN, netblocks (IPv4 and IPv6), internet2 (and related networks) and country. Fedora does have nameservers advertised on AAAA records, as well as A records. The inbound web proxies are reachable via both AAAA and A records, so MirrorManager does see a client's IPv6 or IPv4 address.
MirrorManager replies to client requests with DNS names. Mirrors may themselves advertise a given name with an AAAA or A record.
The idea would be to provide DNS names we know have AAAA records to provide better quality of service to a yum client that can't use A records. Otherwise, it's a lot of poking randomly at hosts until one with AAAA is found.
It's also just a good idea at this point to at least build the mirror infrastructure to be IPv6 capable from end to end so that an IPv6-only client at least has a shot at finding a mirror at all.
- Prepare a separate monitoring system for IPv6 content to confirm that
mirrors are serving the content correctly over both v4 and v6.
Right now Fedora's MM crawler runs on machines that can only speak IPv4. I trust mirror admins that if they're advertising both AAAA and A records, that both methods are equivalent and if one works, the other works. I haven't had a significant problem with this in practice.
I haven't seen it a lot in practice, but it happens. Currently, admins sometimes forget their IPv6 records when they change IPs or their network administrator assumes no one is really using IPv6 so the IPv6 network goes down more often or breaks in a way that no one really notices for a while.
I've also seen a few people just map an AAAA to a server and not have the web server configured to apply the VirtualHost to the IPv6 address.
Thoughts?
Are there a bunch of mirrors for whom IPv6 bandwidth is "free", compared to IPv4? That's the only reason I can see going to this effort. I understand that bandwidth via Internet2 would be cheaper than commercial links, which is why MM sends I2-capable clients to I2-capable servers first. Otherwise, seems like a lot of work for not necessarily any gain.
My mirror is substantially cheaper if we send traffic over IPv6, but for the most part, if you do have IPv6 and happen to hit my mirror, you will end up preferring it anyway. This change would actually drive more IPv6 traffic to my mirror rather than to IPv4 only mirrors, but I have no real objection to this because the cost is very low and I like anything that increases the IPv6 traffic numbers on the Internet for statistical reasons.
The techniques used for driving traffic to I2 could well be used for IPv6 as well.
On Mon, 1 Aug 2011, Kevin Stange wrote:
On 08/01/2011 03:37 PM, Matt Domsch wrote:
On Mon, Aug 01, 2011 at 02:16:30PM -0500, Kevin Stange wrote:
Karabir suggested we start getting together a list of steps that need completion to deliver an IPv6-preferenced or IPv6-only list of mirrors for IPv6-only clients using CentOS.
FWIW, Fedora hasn't had a significant need to IPv6-preferenced of IPv6-only mirrors. We do manage preference by ASN, netblocks (IPv4 and IPv6), internet2 (and related networks) and country. Fedora does have nameservers advertised on AAAA records, as well as A records. The inbound web proxies are reachable via both AAAA and A records, so MirrorManager does see a client's IPv6 or IPv4 address.
MirrorManager replies to client requests with DNS names. Mirrors may themselves advertise a given name with an AAAA or A record.
The idea would be to provide DNS names we know have AAAA records to provide better quality of service to a yum client that can't use A records. Otherwise, it's a lot of poking randomly at hosts until one with AAAA is found.
It's also just a good idea at this point to at least build the mirror infrastructure to be IPv6 capable from end to end so that an IPv6-only client at least has a shot at finding a mirror at all.
The question is: are we optimizing our service for ipv6-only clients, or for ipv6-capable (dual-stack) clients?
Returning only a list of v6-capable mirrors to a client will help the case of a v6-only client, but could hurt a v6-capable client.
One example problem: what do you do when there are no v6 mirrors in a country (since the mirror system today is country-based)? Do you return v4 mirrors from that country, or v6 mirrors from other countries? Remember that in some countries international bandwidth can be very expensive.
I am a big proponent of ipv6 (my mirror has had a v6 address for years). However, running a client v6-only ignores the fact the most of the internet is still v4-only (let alone dual-stack). It's an interesting experiment, but a v6-only client is not very useful on today's internet (and probably won't be for several years). I don't think we should cater much to people who choose to make their own lives difficult.
DR
On 08/01/2011 05:16 PM, David Richardson wrote:
On Mon, 1 Aug 2011, Kevin Stange wrote:
On 08/01/2011 03:37 PM, Matt Domsch wrote:
On Mon, Aug 01, 2011 at 02:16:30PM -0500, Kevin Stange wrote:
Karabir suggested we start getting together a list of steps that need completion to deliver an IPv6-preferenced or IPv6-only list of mirrors for IPv6-only clients using CentOS.
FWIW, Fedora hasn't had a significant need to IPv6-preferenced of IPv6-only mirrors. We do manage preference by ASN, netblocks (IPv4 and IPv6), internet2 (and related networks) and country. Fedora does have nameservers advertised on AAAA records, as well as A records. The inbound web proxies are reachable via both AAAA and A records, so MirrorManager does see a client's IPv6 or IPv4 address.
MirrorManager replies to client requests with DNS names. Mirrors may themselves advertise a given name with an AAAA or A record.
The idea would be to provide DNS names we know have AAAA records to provide better quality of service to a yum client that can't use A records. Otherwise, it's a lot of poking randomly at hosts until one with AAAA is found.
It's also just a good idea at this point to at least build the mirror infrastructure to be IPv6 capable from end to end so that an IPv6-only client at least has a shot at finding a mirror at all.
The question is: are we optimizing our service for ipv6-only clients, or for ipv6-capable (dual-stack) clients?
I would say we're talking about just making it possible for v6 only clients to get the mirror list. Right now a v6 only client cannot obtain the mirror list at all. We'd be catering to v6-capable (dual stack) clients by offering mirror choices with v6 ahead of v4 if the client seems to prefer v6.
Returning only a list of v6-capable mirrors to a client will help the case of a v6-only client, but could hurt a v6-capable client.
I would prefer to return v4 with lower priority after the v6 list for this reason.
One example problem: what do you do when there are no v6 mirrors in a country (since the mirror system today is country-based)? Do you return v4 mirrors from that country, or v6 mirrors from other countries? Remember that in some countries international bandwidth can be very expensive.
This is an interesting question. If the user connects via v6 we have no way to know if they're v4-enabled unless we do something really weird. Right now, we can reasonably assume that they do have v4 because it's sort of ridiculous not to, then perhaps we want to give v4 addresses for the local country first, followed by v6 addresses for other countries in the region?
Making this assumption permanently is not a good idea though, so it seems wise to me that we are flexible with how things are set up.
I am a big proponent of ipv6 (my mirror has had a v6 address for years). However, running a client v6-only ignores the fact the most of the internet is still v4-only (let alone dual-stack). It's an interesting experiment, but a v6-only client is not very useful on today's internet (and probably won't be for several years). I don't think we should cater much to people who choose to make their own lives difficult.
Part of my goal is to ready for the idea that someone that does run a v6 only system can actually find a mirror at all (and that means making centos reachable via v6), but, I recognize for now that it's unlikely.
In the meantime, if the client prefers IPv6, I think it's better to offer him IPv6 mirrors as the top priority, followed by other regional v4 only mirrors after that.
On Mon, Aug 01, 2011 at 05:45:16PM -0500, Kevin Stange wrote:
It's also just a good idea at this point to at least build the mirror infrastructure to be IPv6 capable from end to end so that an IPv6-only client at least has a shot at finding a mirror at all.
agreed, IPv6-enabling the mirror infrastructure is a worthwhile goal in and of itself.
The question is: are we optimizing our service for ipv6-only clients, or for ipv6-capable (dual-stack) clients?
I would say we're talking about just making it possible for v6 only clients to get the mirror list. Right now a v6 only client cannot obtain the mirror list at all.
I'm with you to here. :-)
We'd be catering to v6-capable (dual stack) clients by offering mirror choices with v6 ahead of v4 if the client seems to prefer v6.
Here's where I start to lose you. We don't presently have a way for a client to indicate to anyone anywhere that it "prefers" v6 instead of v4, or vice versa. From the server side, it sees a connection, either on v6 or on v4, and sees an address. We would have to add a flag into the yum mirrorlist request URL to give the client an opportunity to note its preference. Not hard technically, but the obvious implementation would require manual configuration by each end user, and that's nearly never done.
How would you envision a client recognizing that it should prefer v6 over v4, automatically? Solving that cleanly, the rest is straightforward.
Returning only a list of v6-capable mirrors to a client will help the case of a v6-only client, but could hurt a v6-capable client.
I would prefer to return v4 with lower priority after the v6 list for this reason.
One example problem: what do you do when there are no v6 mirrors in a country (since the mirror system today is country-based)? Do you return v4 mirrors from that country, or v6 mirrors from other countries? Remember that in some countries international bandwidth can be very expensive.
This is an interesting question. If the user connects via v6 we have no way to know if they're v4-enabled unless we do something really weird. Right now, we can reasonably assume that they do have v4 because it's sort of ridiculous not to, then perhaps we want to give v4 addresses for the local country first, followed by v6 addresses for other countries in the region?
Making this assumption permanently is not a good idea though, so it seems wise to me that we are flexible with how things are set up.
I am a big proponent of ipv6 (my mirror has had a v6 address for years). However, running a client v6-only ignores the fact the most of the internet is still v4-only (let alone dual-stack). It's an interesting experiment, but a v6-only client is not very useful on today's internet (and probably won't be for several years). I don't think we should cater much to people who choose to make their own lives difficult.
Part of my goal is to ready for the idea that someone that does run a v6 only system can actually find a mirror at all (and that means making centos reachable via v6), but, I recognize for now that it's unlikely.
In the meantime, if the client prefers IPv6, I think it's better to offer him IPv6 mirrors as the top priority, followed by other regional v4 only mirrors after that.
You're asking for a different sort order to the results list, based on the presence of a "prefers_v6=1' flag in the request URL. Not difficult really, MirrorManager could add that. Right now it doesn't do A or AAAA lookups for its mirrors, it just returns the URLs that the mirror admins put in the database, and the clients then do their own DNS lookups, and then connect whichever way they choose. So MM would need to add DNS lookups and cache that info, and use it in the sort order.
MM would also need to add a flag for each mirror Host, indicating "prefers_v6_clients", because I expect for most hosts, they really don't care. What does that do to the sort order?
Protocol is orthogonal to network distance, I don't really like mixing the two in the ordered list:
(same ASN) + (in netblock) + (in peer ASN) + (client_prefers_v6 && host_prefers_v6) + (client_prefers_v6) + (rest of the list in geographic order) ?
(the above assumes a MM implementation, which CentOS isn't using at the moment, I'm simply projecting. That's another discussion to be had when time allows.)
Thanks, Matt
On 08/02/2011 10:10 AM, Matt Domsch wrote:
On Mon, Aug 01, 2011 at 05:45:16PM -0500, Kevin Stange wrote:
We'd be catering to v6-capable (dual stack) clients by offering mirror choices with v6 ahead of v4 if the client seems to prefer v6.
Here's where I start to lose you. We don't presently have a way for a client to indicate to anyone anywhere that it "prefers" v6 instead of v4, or vice versa. From the server side, it sees a connection, either on v6 or on v4, and sees an address. We would have to add a flag into the yum mirrorlist request URL to give the client an opportunity to note its preference. Not hard technically, but the obvious implementation would require manual configuration by each end user, and that's nearly never done.
How would you envision a client recognizing that it should prefer v6 over v4, automatically? Solving that cleanly, the rest is straightforward.
We can tell if a client prefers v6 because it will access mirrorlist.centos.org using its IPv6 address. The client doesn't need to tell us explicitly because its decision to access the v6 address means it's decided to try the AAAA first. This is the default behavior in most software, but can generally be changed manually if the user recognizes they have very poor IPv6 connectivity.
In the meantime, if the client prefers IPv6, I think it's better to offer him IPv6 mirrors as the top priority, followed by other regional v4 only mirrors after that.
You're asking for a different sort order to the results list, based on the presence of a "prefers_v6=1' flag in the request URL. Not difficult really, MirrorManager could add that. Right now it doesn't do A or AAAA lookups for its mirrors, it just returns the URLs that the mirror admins put in the database, and the clients then do their own DNS lookups, and then connect whichever way they choose. So MM would need to add DNS lookups and cache that info, and use it in the sort order.
I suppose we could compile a list of prefers_v6 mirrors, but my thoughts didn't extend past concluding that if the client prefers v6 by evidence we just offer it the v6 mirrors sorted first so it can use the preferred protocol.
Do you think it's likely that mirrors will prefer avoiding v6 if they have enabled it?
On Tue, Aug 02, 2011 at 11:42:47AM -0500, Kevin Stange wrote:
On 08/02/2011 10:10 AM, Matt Domsch wrote:
On Mon, Aug 01, 2011 at 05:45:16PM -0500, Kevin Stange wrote:
We'd be catering to v6-capable (dual stack) clients by offering mirror choices with v6 ahead of v4 if the client seems to prefer v6.
Here's where I start to lose you. We don't presently have a way for a client to indicate to anyone anywhere that it "prefers" v6 instead of v4, or vice versa. From the server side, it sees a connection, either on v6 or on v4, and sees an address. We would have to add a flag into the yum mirrorlist request URL to give the client an opportunity to note its preference. Not hard technically, but the obvious implementation would require manual configuration by each end user, and that's nearly never done.
How would you envision a client recognizing that it should prefer v6 over v4, automatically? Solving that cleanly, the rest is straightforward.
We can tell if a client prefers v6 because it will access mirrorlist.centos.org using its IPv6 address. The client doesn't need to tell us explicitly because its decision to access the v6 address means it's decided to try the AAAA first.
With all due respect, I disagree. Apps on a dual-stack system will use IPv6 if they can, then IPv4, at a system level, provided an IPv6 address is provisioned to the system. It has nothing to do with connectivity between the system and any other mirror server on the internet, which is what mirrors are trying to solve.
I suppose we could compile a list of prefers_v6 mirrors, but my thoughts didn't extend past concluding that if the client prefers v6 by evidence we just offer it the v6 mirrors sorted first so it can use the preferred protocol.
I understand, but I don't think that's a good indicator for preference, and I think specifying one protocol is preferred over another doesn't buy us much.
Do you think it's likely that mirrors will prefer avoiding v6 if they have enabled it?
No. If they advertise both, they'll accept both. There's no way to specify a mirror's preference for one or the other, just as there is no way to specify a client's preference. If they want to avoid v6, they simply don't advertise an AAAA record. :-)
MirrorManager (and the respected MirrorBrain) aim to solve for the "best" mirror on the assumption that closer mirrors network-wise are preferred over mirrors that are further away, irrespective of a client's IPv4 or IPv6 address. Both offer ways to direct clients to preferred servers based on network characteristics (BGP ASNs, network blocks) between client and potential servers, and only then falling back to GeoIP, all of which are protocol agnostic (both v4 and v6 are supported).
Thanks, Matt
Hi.
I would agree. IPv6 has nothing to do with mirror manager. It is a protocol wich client uses or not. If you have AAAA records and are on dual stack you will prefer to go thru IPv6. Otherwise if there is also A/CNAME record you will go over IPv4.
Regards, Marko On Tue, 2 Aug 2011, Matt Domsch wrote:
On Tue, Aug 02, 2011 at 11:42:47AM -0500, Kevin Stange wrote:
On 08/02/2011 10:10 AM, Matt Domsch wrote:
On Mon, Aug 01, 2011 at 05:45:16PM -0500, Kevin Stange wrote:
We'd be catering to v6-capable (dual stack) clients by offering mirror choices with v6 ahead of v4 if the client seems to prefer v6.
Here's where I start to lose you. We don't presently have a way for a client to indicate to anyone anywhere that it "prefers" v6 instead of v4, or vice versa. From the server side, it sees a connection, either on v6 or on v4, and sees an address. We would have to add a flag into the yum mirrorlist request URL to give the client an opportunity to note its preference. Not hard technically, but the obvious implementation would require manual configuration by each end user, and that's nearly never done.
How would you envision a client recognizing that it should prefer v6 over v4, automatically? Solving that cleanly, the rest is straightforward.
We can tell if a client prefers v6 because it will access mirrorlist.centos.org using its IPv6 address. The client doesn't need to tell us explicitly because its decision to access the v6 address means it's decided to try the AAAA first.
With all due respect, I disagree. Apps on a dual-stack system will use IPv6 if they can, then IPv4, at a system level, provided an IPv6 address is provisioned to the system. It has nothing to do with connectivity between the system and any other mirror server on the internet, which is what mirrors are trying to solve.
I suppose we could compile a list of prefers_v6 mirrors, but my thoughts didn't extend past concluding that if the client prefers v6 by evidence we just offer it the v6 mirrors sorted first so it can use the preferred protocol.
I understand, but I don't think that's a good indicator for preference, and I think specifying one protocol is preferred over another doesn't buy us much.
Do you think it's likely that mirrors will prefer avoiding v6 if they have enabled it?
No. If they advertise both, they'll accept both. There's no way to specify a mirror's preference for one or the other, just as there is no way to specify a client's preference. If they want to avoid v6, they simply don't advertise an AAAA record. :-)
MirrorManager (and the respected MirrorBrain) aim to solve for the "best" mirror on the assumption that closer mirrors network-wise are preferred over mirrors that are further away, irrespective of a client's IPv4 or IPv6 address. Both offer ways to direct clients to preferred servers based on network characteristics (BGP ASNs, network blocks) between client and potential servers, and only then falling back to GeoIP, all of which are protocol agnostic (both v4 and v6 are supported).
Thanks, Matt
-
David Richardson -
Kevin Stange -
Marko Bevc -
Matt Domsch