On Tue, 17 Apr 2007, Maciej Zenczykowski wrote:
also need to keep in mind that if its going to hit .centos.org machines, its going to come atleast a few days later than the .torrent - since we tend to get hit real hard by people using yum and doing net installs as well.
So ideally the best solution here is to have some sort of DNS resolving in place so that mirror.centos.org always resolves to your closest in-sync mirror -- I was under the impression that we already have this for yum???
But --- does jigdo have any kind of indirection in place , eg to get a list of mirrors and use them ??
Or could yum be used as a wrapper around jigdo maybe to pull stuff down using fastestmirror or maybe even dags stuff ??
Otherwise it gets messy because mirrors have different file structures, so just using CNAME's wont work.
The trouble I see is that if a user doesnt have a local mirror or local copy of a set if isos then downloading the packages individually from a mirror may be less efficient than downloading an iso / set of isos, and certainly without mirror redirect and fastestmirror it is a non starter ...
The fedora guys have beern working on stuff that we also ought to look at , and we should look at metalinks while we are at it.
Regards Lance
-- uklinux.net - The ISP of choice for the discerning Linux user.
But --- does jigdo have any kind of indirection in place , eg to get a list of mirrors and use them ??
It does use a list of mirros for the debian. I haven't really looked into it too deeply. Furthermore the jigdo-lite program is really just a shell script - should be easy to get it to behave however we want it (it just runs wget).
Or could yum be used as a wrapper around jigdo maybe to pull stuff down using fastestmirror or maybe even dags stuff ??
I would really like to see dynamic DNS resolution based on source IP giving out the address of the nearest centos compatible mirror - then it would 'just work'.
Otherwise it gets messy because mirrors have different file structures, so just using CNAME's wont work.
And that's the problem... But we could possibly work with them to get this working - at least for http this is relatively trivial (just setup a virtual domain for whatever global name we would decide on like mirror.centos.org).
The trouble I see is that if a user doesnt have a local mirror or local copy of a set if isos then downloading the packages individually from a mirror may be less efficient than downloading an iso / set of isos, and certainly without mirror redirect and fastestmirror it is a non starter ...
There aren't that many files on a cd, so the extra time spent opening closing connections shouldn't really be significant - unless you have an extremely big pipe. But - yes - we would want each user to use the closest possible mirror.
The fedora guys have beern working on stuff that we also ought to look at , and we should look at metalinks while we are at it.
No idea what this is... :-)
Cheers, Maciej.
On Tue, 17 Apr 2007, Maciej ¯enczykowski wrote:
But --- does jigdo have any kind of indirection in place , eg to get a list of mirrors and use them ??
It does use a list of mirros for the debian. I haven't really looked into it too deeply. Furthermore the jigdo-lite program is really just a shell script - should be easy to get it to behave however we want it (it just runs wget).
Or could yum be used as a wrapper around jigdo maybe to pull stuff down using fastestmirror or maybe even dags stuff ??
I would really like to see dynamic DNS resolution based on source IP giving out the address of the nearest centos compatible mirror - then it would 'just work'.
see below re cnames ...
we really need to push the bandwidth out to external mirrors - and 302's are messy ...
Otherwise it gets messy because mirrors have different file structures, so just using CNAME's wont work.
And that's the problem... But we could possibly work with them to get this working - at least for http this is relatively trivial (just setup a virtual domain for whatever global name we would decide on like mirror.centos.org).
Except that is not what we do for mirror.centos.org - all those are our servers.
The trouble I see is that if a user doesnt have a local mirror or local copy of a set if isos then downloading the packages individually from a mirror may be less efficient than downloading an iso / set of isos, and certainly without mirror redirect and fastestmirror it is a non starter ...
There aren't that many files on a cd, so the extra time spent opening closing connections shouldn't really be significant - unless you have an extremely big pipe. But - yes - we would want each user to use the closest possible mirror.
The fedora guys have beern working on stuff that we also ought to look at , and we should look at metalinks while we are at it.
No idea what this is... :-)
https://hosted.fedoraproject.org/projects/revisor
I dont really think it is the same thing ...
and http://www.metalinker.org/
eg http://www.metalinker.org/samples/CentOS-4.4-i386-binDVD.iso.metalink
Regards Lance
-- uklinux.net - The ISP of choice for the discerning Linux user.
arrgh - sorry for the cross posting - I didnt notice the cc to CentOS list :(
Lets try to keep this on -devel now shall we ??
Cheers
Lance
-- uklinux.net - The ISP of choice for the discerning Linux user.
Trimming centos@ from reply list.
It does use a list of mirros for the debian. I haven't really looked into it too deeply. Furthermore the jigdo-lite program is really just a shell script - should be easy to get it to behave however we want it (it just runs wget).
I would really like to see dynamic DNS resolution based on source IP giving out the address of the nearest centos compatible mirror - then it would 'just work'.
see below re cnames ...
I know, and yet... this would still be the most useful way. it would make everything just work.
we really need to push the bandwidth out to external mirrors - and 302's are messy ...
agreed. We can definitely do this, I'll have to take a look at how this is done for debian (and how it chooses the closest mirror). It'd be nice not to have to reinvent the wheel.
Do we have a canonical list of mirrors? How do we decide on a best mirror for a given user? (based on his IP? some sort of lookup? some sort of ping times? some sort of cgi applet on the centos servers which returns a customized to requestor ip list of mirrors?) etc...
Otherwise it gets messy because mirrors have different file structures, so just using CNAME's wont work.
And that's the problem... But we could possibly work with them to get this working - at least for http this is relatively trivial (just setup a virtual domain for whatever global name we would decide on like mirror.centos.org).
Except that is not what we do for mirror.centos.org - all those are our servers.
oh, didn't realize - I thought these were participating primary mirrors. you have a lot of these machines then :-)
and http://www.metalinker.org/
eg http://www.metalinker.org/samples/CentOS-4.4-i386-binDVD.iso.metalink
This is more like a auto-mirror system (a way to provide many alternative url's for a file), then a way to stitch files from pieces.
Jigdo here offers the bonus of not having to store the CDs and DVDs on the servers, and not having to download them over and over again if you already have either a local mirror, or local CDs or DVDs and want the opposite - and it decreases bandwidth when issuing an update, respin, etc. A fair number of packages will be upgraded with 4.5 - but many many will remain the same, I'd guess a jigdo'ed release of 4.5 could get and burn the DVD/CDs while downloading about 20-30% of the image size of data only.
Cheers, Maciej
Maciej ¯enczykowski wrote:
But --- does jigdo have any kind of indirection in place , eg to get a list of mirrors and use them ??
It does use a list of mirros for the debian. I haven't really looked into it too deeply. Furthermore the jigdo-lite program is really just a shell script - should be easy to get it to behave however we want it (it just runs wget).
Or could yum be used as a wrapper around jigdo maybe to pull stuff down using fastestmirror or maybe even dags stuff ??
I would really like to see dynamic DNS resolution based on source IP giving out the address of the nearest centos compatible mirror - then it would 'just work'.
This "just works" very badly for me. My best mirrors are local to me, but as they won't serve the world (I believe they pay for both incoming and outgoing traffic, that's how the US wanted it anyway), CentOS won't list them.
A tool, run compulsorily, to configure a list of mirrors _would_ work well.
It needn't be complicated; debian configures a default list for apt-get in two or three questions, starting with "which country?"
"Which country" gives me a list of mirrors, and the obvious choice for me in Perth, Western Australia is ftp.wa.au.debian.org
It also allows me to name my own mirror.
And no scheme you're likely to come up with is likely to find this:
[summer@bilby ~]$ lynx -dump http://rhel.demo.lan/
Index of /
Icon [1]Name [2]Last modified [3]Size [4]Descriptio n ____ etc
The trouble I see is that if a user doesnt have a local mirror or local copy of a set if isos then downloading the packages individually from a mirror may be less efficient than downloading an iso / set of isos,
With jigdo, I could have used a local mirror containing the beta packages to see the new ISOs (assuming that CentOS didn't needlessly rebuild them).
With Jigdo, I could download one set of images _or_ the tree, and using the templates and jigdo files, I could build whichever ISOs I want.
When U1 comes around, I only need the updates files and the new jigdo and templates.
and certainly without mirror redirect and fastestmirror it is a non starter ...
There aren't that many files on a cd, so the extra time spent opening closing connections shouldn't really be significant - unless you have an extremely big pipe. But - yes - we would want each user to use the closest possible mirror.
The fastest isn't necessarily the closest, but the closest is likely faster than most users' connexion. Everyone who can send faster than I can receive is equal, so far as speed is concerned.
-
John Summerfield -
Lance Davis -
Maciej Żenczykowski