Hi,
what do you think about using jigdo for distributing iso Images?
http://en.wikipedia.org/wiki/Jigdo http://atterer.net/jigdo/
advantages: - save space at mirror because data are now redandant (as rpm's and "inside" the iso images - the webserver does handle small files more efficient than big files - can save bandwith if a download of an iso image aborts and have to be restarted
disadvantage: - maybe more requests, for downloading all rpm's to build the iso at client site
Happy discussing! Uwe
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
On 10/29/2009 2:51 PM, Uwe Kiewel wrote:
Hi,
what do you think about using jigdo for distributing iso Images?
http://en.wikipedia.org/wiki/Jigdo http://atterer.net/jigdo/
advantages:
- save space at mirror because data are now redandant (as rpm's and
"inside" the iso images
- the webserver does handle small files more efficient than big files
- can save bandwith if a download of an iso image aborts and have to be restarted
disadvantage:
- maybe more requests, for downloading all rpm's to build the iso
at client site
Happy discussing! Uwe _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
Nick Olsen schrieb:
Addition of jigdo yes. Replace ISO's no.
Does it make sense? If someone whants to have a iso image, he will download the iso file and will not use jigdo...
More sense makes to keep the boot.iso and replace the DVD and CD images by jigdo
Educating joe sixpack on using something new, well I like to use altercation avoidance.
On 10/29/2009 2:51 PM, Uwe Kiewel wrote:
Hi,
what do you think about using jigdo for distributing iso Images?
http://en.wikipedia.org/wiki/Jigdo http://atterer.net/jigdo/
advantages:
- save space at mirror because data are now redandant (as rpm's and
"inside" the iso images
- the webserver does handle small files more efficient than big files
- can save bandwith if a download of an iso image aborts and have to be restarted
disadvantage:
- maybe more requests, for downloading all rpm's to build the iso
at client site
Happy discussing! Uwe _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
Need to give people that option though. I don't really think its worth the hassle, IMO
On 10/29/2009 3:34 PM, Uwe Kiewel wrote:
Nick Olsen schrieb:
Addition of jigdo yes. Replace ISO's no.
Does it make sense? If someone whants to have a iso image, he will download the iso file and will not use jigdo...
More sense makes to keep the boot.iso and replace the DVD and CD images by jigdo
Educating joe sixpack on using something new, well I like to use altercation avoidance.
On 10/29/2009 2:51 PM, Uwe Kiewel wrote:
Hi,
what do you think about using jigdo for distributing iso Images?
http://en.wikipedia.org/wiki/Jigdo http://atterer.net/jigdo/
advantages:
- save space at mirror because data are now redandant (as rpm's and
"inside" the iso images
- the webserver does handle small files more efficient than big files
- can save bandwith if a download of an iso image aborts and have to be restarted
disadvantage:
- maybe more requests, for downloading all rpm's to build the iso
at client site
Happy discussing! Uwe _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
On Thu, Oct 29, 2009 at 02:59:22PM -0400, Nick Olsen wrote:
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
please no top posting ;)
There was a thread about jidgo in the archives and it boiled down to: - someone has to maintain the jidgo package in the CentOS tree (how would a CentOS user for C3/4/5 use jidgo?) - document it in the wiki (how to use, at least) - someone needs to make it work from the current setup - how much burden will it put on the mirrors? centos.org ones and public ones? (I have no experience on using it) - no one volonteered. - one more thing to check for the QA release process.
back to you :)
Cheers,
Tru (in the todo list, we need to move from a pull mode into a push mode at least on the centos.org infra, add more dvd-mirrors for the public mirrors, maybe merge with/without dvd,....)
Tru Huynh schrieb:
On Thu, Oct 29, 2009 at 02:59:22PM -0400, Nick Olsen wrote:
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
please no top posting ;)
There was a thread about jidgo in the archives and it boiled down to:
- someone has to maintain the jidgo package in the CentOS tree
(how would a CentOS user for C3/4/5 use jidgo?)
- document it in the wiki (how to use, at least)
- someone needs to make it work from the current setup
- how much burden will it put on the mirrors? centos.org ones and public ones?
(I have no experience on using it)
- no one volonteered.
- one more thing to check for the QA release process.
back to you :)
Ok. Understood.
It was just an idea because Fedora do it successfuly since Fedora 6 and Debian do so as well.
Thanks, Uwe
Uwe Kiewel wrote:
Tru Huynh schrieb:
On Thu, Oct 29, 2009 at 02:59:22PM -0400, Nick Olsen wrote:
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
please no top posting ;)
There was a thread about jidgo in the archives and it boiled down to:
- someone has to maintain the jidgo package in the CentOS tree
(how would a CentOS user for C3/4/5 use jidgo?)
- document it in the wiki (how to use, at least)
- someone needs to make it work from the current setup
- how much burden will it put on the mirrors? centos.org ones and public ones?
(I have no experience on using it)
- no one volonteered.
- one more thing to check for the QA release process.
back to you :)
Ok. Understood.
It was just an idea because Fedora do it successfuly since Fedora 6 and Debian do so as well.
Ok back that monkey truck up slightly here. Fedora has *ONLY* been doing it for the Fedora Spins stuff which, as you can imagine, an *INCREDIBLY* low volume set of accesses. Debian is going to be likewise, and I wouldn't exactly call it a popular thing from them.
Speaking as a mirror here are my thoughts:
- Cutting down on the working data set is a good thing, though I do have some serious reservations about this on a larger scale.
- Claiming a webserver doesn't handle large files is a bogus statement, if your on Linux you have send_file() and that is darned fast and efficient. It more or less doesn't matter what your file size is for that.
- If your on a client, or a server, and it doesn't support http restarts you really have to ask why? I can understand how *PAINFUL* that is to a mirror to do a random seek into the middle of a file, but once the download has started it's effectively no additional overhead beyond that.
- Speaking to the apache module that auto-generates the iso on the fly: any mirror of any reasonable size will shoot this down in a heartbeat. We already have an I/O problem on the systems, ram issues, etc. Adding something into apache that's going to thrash about and magically generate this as it's requested is *WORSE* than the wasted disk space. Again send_file() is your friend.
My thoughts ------------
Honestly if Centos is actively looking to eliminate the ISOs I would tentatively support this, but Jigdo (at least the last time I used it) is *ANYTHING* but userfriendly. It would *HAVE* to be as simple as download a script, program, etc you get a download box and *poof* your dvd comes out, no user interaction unless a lot of advanced options are selected somewhere, and last time I used it it wasn't that simple.
Furthermore I think jigdo is likely going to be a lot of work, with little payoff. From my gut reaction I think moving to more of a universal network installer (ala http://boot.kernel.org w/ it's network installers, which happen to include Centos)[Disclaimer: I'm one of the devs & the primary admin for http://boot.kernel.org] is a *LOT* more intuitive to a user and a lot simpler to get them to use than Jigdo ever will be, and honestly it gets a user moving sooner and it can take less time anyway depending on what a user selects, has all of the advantages of Jigdo and with significantly fewer downsides.
Just my $0.02
- John 'Warthog9' Hawley Chief Kernel.org Administrator
On Oct 29, 2009, at 10:52 PM, "J.H." warthog9@kernel.org wrote:
Uwe Kiewel wrote:
Tru Huynh schrieb:
On Thu, Oct 29, 2009 at 02:59:22PM -0400, Nick Olsen wrote:
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
please no top posting ;)
There was a thread about jidgo in the archives and it boiled down to:
- someone has to maintain the jidgo package in the CentOS tree
(how would a CentOS user for C3/4/5 use jidgo?)
- document it in the wiki (how to use, at least)
- someone needs to make it work from the current setup
- how much burden will it put on the mirrors? centos.org ones and
public ones? (I have no experience on using it)
- no one volonteered.
- one more thing to check for the QA release process.
back to you :)
Ok. Understood.
It was just an idea because Fedora do it successfuly since Fedora 6 and Debian do so as well.
Ok back that monkey truck up slightly here. Fedora has *ONLY* been doing it for the Fedora Spins stuff which, as you can imagine, an *INCREDIBLY* low volume set of accesses. Debian is going to be likewise, and I wouldn't exactly call it a popular thing from them.
Speaking as a mirror here are my thoughts:
- Cutting down on the working data set is a good thing, though I do
have some serious reservations about this on a larger scale.
- Claiming a webserver doesn't handle large files is a bogus
statement, if your on Linux you have send_file() and that is darned fast and efficient. It more or less doesn't matter what your file size is for that.
- If your on a client, or a server, and it doesn't support http
restarts you really have to ask why? I can understand how *PAINFUL* that is to a mirror to do a random seek into the middle of a file, but once the download has started it's effectively no additional overhead beyond that.
- Speaking to the apache module that auto-generates the iso on the
fly: any mirror of any reasonable size will shoot this down in a heartbeat. We already have an I/O problem on the systems, ram issues, etc. Adding something into apache that's going to thrash about and magically generate this as it's requested is *WORSE* than the wasted disk space. Again send_file() is your friend.
My thoughts
Honestly if Centos is actively looking to eliminate the ISOs I would tentatively support this, but Jigdo (at least the last time I used it) is *ANYTHING* but userfriendly. It would *HAVE* to be as simple as download a script, program, etc you get a download box and *poof* your dvd comes out, no user interaction unless a lot of advanced options are selected somewhere, and last time I used it it wasn't that simple.
Furthermore I think jigdo is likely going to be a lot of work, with little payoff. From my gut reaction I think moving to more of a universal network installer (ala http://boot.kernel.org w/ it's network installers, which happen to include Centos)[Disclaimer: I'm one of the devs & the primary admin for http://boot.kernel.org] is a *LOT* more intuitive to a user and a lot simpler to get them to use than Jigdo ever will be, and honestly it gets a user moving sooner and it can take less time anyway depending on what a user selects, has all of the advantages of Jigdo and with significantly fewer downsides.
Just my $0.02
- John 'Warthog9' Hawley
Chief Kernel.org Administrator _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
I agree with the apache module not being feasable... On top of the fact you have additional resource usage from it you are making a huge assumtion that the mirror is running apache. As great as apache is, it is resource hog. Something that only serves high volume static files is rarely running apache.
My 2 cents on that :) - Billy
Billy Vierra wrote:
On Oct 29, 2009, at 10:52 PM, "J.H." warthog9@kernel.org wrote:
Uwe Kiewel wrote:
Tru Huynh schrieb:
On Thu, Oct 29, 2009 at 02:59:22PM -0400, Nick Olsen wrote:
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
please no top posting ;)
There was a thread about jidgo in the archives and it boiled down to:
- someone has to maintain the jidgo package in the CentOS tree
(how would a CentOS user for C3/4/5 use jidgo?)
- document it in the wiki (how to use, at least)
- someone needs to make it work from the current setup
- how much burden will it put on the mirrors? centos.org ones and
public ones? (I have no experience on using it)
- no one volonteered.
- one more thing to check for the QA release process.
back to you :)
Ok. Understood.
It was just an idea because Fedora do it successfuly since Fedora 6 and Debian do so as well.
Ok back that monkey truck up slightly here. Fedora has *ONLY* been doing it for the Fedora Spins stuff which, as you can imagine, an *INCREDIBLY* low volume set of accesses. Debian is going to be likewise, and I wouldn't exactly call it a popular thing from them.
Speaking as a mirror here are my thoughts:
- Cutting down on the working data set is a good thing, though I do
have some serious reservations about this on a larger scale.
- Claiming a webserver doesn't handle large files is a bogus
statement, if your on Linux you have send_file() and that is darned fast and efficient. It more or less doesn't matter what your file size is for that.
- If your on a client, or a server, and it doesn't support http
restarts you really have to ask why? I can understand how *PAINFUL* that is to a mirror to do a random seek into the middle of a file, but once the download has started it's effectively no additional overhead beyond that.
- Speaking to the apache module that auto-generates the iso on the
fly: any mirror of any reasonable size will shoot this down in a heartbeat. We already have an I/O problem on the systems, ram issues, etc. Adding something into apache that's going to thrash about and magically generate this as it's requested is *WORSE* than the wasted disk space. Again send_file() is your friend.
My thoughts
Honestly if Centos is actively looking to eliminate the ISOs I would tentatively support this, but Jigdo (at least the last time I used it) is *ANYTHING* but userfriendly. It would *HAVE* to be as simple as download a script, program, etc you get a download box and *poof* your dvd comes out, no user interaction unless a lot of advanced options are selected somewhere, and last time I used it it wasn't that simple.
Furthermore I think jigdo is likely going to be a lot of work, with little payoff. From my gut reaction I think moving to more of a universal network installer (ala http://boot.kernel.org w/ it's network installers, which happen to include Centos)[Disclaimer: I'm one of the devs & the primary admin for http://boot.kernel.org] is a *LOT* more intuitive to a user and a lot simpler to get them to use than Jigdo ever will be, and honestly it gets a user moving sooner and it can take less time anyway depending on what a user selects, has all of the advantages of Jigdo and with significantly fewer downsides.
Just my $0.02
- John 'Warthog9' Hawley
Chief Kernel.org Administrator _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
I agree with the apache module not being feasable... On top of the fact you have additional resource usage from it you are making a huge assumtion that the mirror is running apache. As great as apache is, it is resource hog. Something that only serves high volume static files is rarely running apache.
I dunno, I run Apache quite successfully and I would argue I'm a high volume content provider.
- John 'Warthog9' Hawley Chief Kernel.org Administrator
Am 30.10.09 10:00, schrieb Billy Vierra:
On 10/30/2009 1:22 AM, J.H. wrote:
I dunno, I run Apache quite successfully and I would argue I'm a high volume content provider.
Oh I am sure that you do and that you are :). However in my past experience for static files, there tend to be better options than apache.
We were not talking about static files here, but some module which generates isos on the fly.
Ralph
Seems this didn't get through, sorry if its a dup...
[...]
- Speaking to the apache module that auto-generates the iso on the
fly: any mirror of any reasonable size will shoot this down in a heartbeat. We already have an I/O problem on the systems, ram issues, etc. Adding something into apache that's going to thrash about and magically generate this as it's requested is *WORSE* than the wasted disk space. Again send_file() is your friend.
Possibly, perhaps even probably.
It would very likely use (a lot?) more CPU, but your working set would be smaller. So depending on how smart/memmory effective you could get the generation, you might be able to a lot higher cache hit rate, which would reduce the I/O problem.
One similar idea that might work is using loop mounted iso, if the io layer is smart enough (I have no idea whatever it is or not).
All depending on your setup of course, and where your bottleneck is.
I agree with the apache module not being feasable... On top of the fact you have additional resource usage from it you are making a huge assumtion that the mirror is running apache.
Thats why I wrote """ Or even better (for those running linux at least) - do it in the VFS layer. That way other services like rsync would benefit from it too. """
Regards, Emil
Any apache hackers among us?
What would be _realy_ cool is an apache module / cgi / ... to build the iso on the fly, using for example jigdo as a backend tool.
Or even better (for those running linux at least) - do it in the VFS layer. That way other services like rsync would benefit from it too.
Oh well, if only time was a bit more... stretchable:-)
// Emil
Addition of jigdo yes. Replace ISO's no. Educating joe sixpack on using something new, well I like to use altercation avoidance.
On 10/29/2009 2:51 PM, Uwe Kiewel wrote:
Hi,
what do you think about using jigdo for distributing iso Images?
http://en.wikipedia.org/wiki/Jigdo http://atterer.net/jigdo/
advantages:
- save space at mirror because data are now redandant (as rpm's
and "inside" the iso images
- the webserver does handle small files more efficient than big files
- can save bandwith if a download of an iso image aborts and have to be restarted
disadvantage:
- maybe more requests, for downloading all rpm's to build the
iso at client site
Happy discussing! Uwe _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
On Thu, 29 Oct 2009, Uwe Kiewel wrote:
what do you think about using jigdo for distributing iso Images?
http://en.wikipedia.org/wiki/Jigdo http://atterer.net/jigdo/
advantages:
- save space at mirror because data are now redandant (as rpm's and
"inside" the iso images
- the webserver does handle small files more efficient than big files
- can save bandwith if a download of an iso image aborts and have to be restarted
disadvantage:
- maybe more requests, for downloading all rpm's to build the iso
at client site
It'll take just as much time and bandwidth for Jigdo to download the individual pieces as it would to just download the ISO. Modern web clients support resuming an aborted download, so the savings there wouldn't be that much (and is shrinking as clients get smarter).
It doesn't save server load: finding-opening-closing a file is not free. It may not hurt my load much, but it certainly will not help it at all.
It doesn't make the user's life easier: too many people won't (or worse, can't) install something to get the ISO. Who want to install yet another specialty downloader?
We'd have a lot of people unhappy if we made Jigdo mandatory. I won't support that idea. And if Jigdo isn't mandatory, it won't save any disk space.
If someone wants to maintain the Jigdo templates and evangelise using it, fine, but it looks to me like a lot of effort for not much return.
If we're looking to reduce disk space used on the mirrors, dropping the CD images (in favor of DVDs) would be a better plan. (If you're disk-space constrained, you can already loop-back mount the DVD image to get the expanded tree.)
DR
-
Billy Vierra -
David Richardson -
Emil -
Emil Pedersen
-
J.H. -
Nick Olsen -
Ralph Angenendt -
Tru Huynh -
Uwe Kiewel