Reposting original message from CentOS mailing list for further discussion with developers and QA deep (per instructions). ---
I've gone through the trouble (after some 10 rounds I finally have something I'm happy with) and generated jigdo/template files for CentOS 5.0 iso's (CD/DVD i386/x86_64) (and CentOS 4.4 while I was at it: CD/DVD/ServerCD SRPMS/i386/x86_64).
(and I started on this before the thread about creating the DVD's from the CD's started ;-) )
They are available at: http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/
(I skipped the LiveCD since jigdo offers no size improvement here -- already for the ServerCD the template file is 125 MBs because of the different .img files inside the iso)
These jigdo/template files should allow you to generate any of the listed cd/dvd isos from the publically available http://mirror.centos.org mirror site (which is DNS-aliased to something close to you). Jigdo will also allow using a local source of files (ie. an existing CD, etc...)
You will need to install jigdo-file and jigdo-lite.
It is available: - in package jigdo from dag's repository (rpmforge) for CentOS 4 (and others) - in package jigdo-file for debian/ubuntu (just apt-get it) - from the source website (both sources, and windows binaries, and statically linked linux binaries) http://atterer.net/jigdo/
Once installed usage is (...should be...) trivial:
jigdo-lite --noask http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/%.jigdo
which will generate the requested (replace the % with what you want, see the http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/ listing) iso file in the directory in which you run it.
[If you have a local source of files you can skip the --noask and provide it when asked]
Next step: get this included on the centos mirrors ;-) [hint hint]
Afterwards: it would be nice to drop the iso files from the mirrors in their entirety and generate them on-the-fly using the JTE extensions package. Should be significant space-savings. [requires perl cgi in the hosting webserver, but should be very safe since it's all read-only access]
Even later: it would be awesome if we could generate the jigdo/template during mkisofs (requires using a patched mkisofs). This would skip the slightly painful jigdo generation step (painful - because I have to download a local mirror of both the files and the CD/DVD images, much less painful to do it directly on the centos source/build system/mirrors)
Both later steps are 95% there (the JTE jigit website/extensions/mkisofs patches/etc) and mostly require putting together rpms for EL4/5. (I'm going to be trying to put together a mkisofs-jte rpm which doesn't override the normal mkisofs rpm, but instead provides two files - /usr/bin/mkisofs-jte and it's manpage)
This should also allow more customized images (like the rolled up updates) without increasing mirror space usage.
Jigdo (combined with hardlinking identical files, which I believe is already done) should enable drastic reductions in centos HDD footprint.
Comments?
Cheers, Maciej Zenczykowski _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Maciej Żenczykowski wrote:
You will need to install jigdo-file and jigdo-lite.
It is available:
- in package jigdo from dag's repository (rpmforge) for CentOS 4 (and
others)
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
Looks like this should go into extras then. I'll take a look at it and see if it "just builds" on CentOS 5.
As soon as I find a plague-client for c5 :)
Cheers,
Ralph
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
I'd seen that in dag's logs as well... was wondering what happened to it. So the package was dropped? oh...
Looks like this should go into extras then. I'll take a look at it and see if it "just builds" on CentOS 5.
This as in w3c-libwww-* I assume :-)
That'd be cool, Maciej
Maciej Żenczykowski wrote:
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
I'd seen that in dag's logs as well... was wondering what happened to it. So the package was dropped? oh...
Yes.
Looks like this should go into extras then. I'll take a look at it and see if it "just builds" on CentOS 5.
This as in w3c-libwww-* I assume :-)
Yes. And this does build without problems, I haven't found the time to build jigdo yet. Probably something for tomorrow. I'll try out your guide, then.
Cheers,
Ralph
Ralph Angenendt wrote:
Maciej Żenczykowski wrote:
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
I'd seen that in dag's logs as well... was wondering what happened to it. So the package was dropped? oh...
Jigdo 0.7.3 does work on CentOS 5. But: Trying to rebuild CentOS-5.0-x86_64-bin-1of7, jigdo fails with
"Aaargh - 9 files could not be downloaded. This should not happen! Depending on the problem, it may help to retry downloading the missing files."
Trying to redownload I see it copying several release notes files - and then it dies again with the same error ...
Error in the template file or in the jigdo file?
Ralph
Could you post which files it's failing for? Maybe one of the mirrors is missing something...
On 4/18/07, Ralph Angenendt ra+centos@br-online.de wrote:
Ralph Angenendt wrote:
Maciej Żenczykowski wrote:
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
I'd seen that in dag's logs as well... was wondering what happened to it. So the package was dropped? oh...
Jigdo 0.7.3 does work on CentOS 5. But: Trying to rebuild CentOS-5.0-x86_64-bin-1of7, jigdo fails with
"Aaargh - 9 files could not be downloaded. This should not happen! Depending on the problem, it may help to retry downloading the missing files."
Trying to redownload I see it copying several release notes files - and then it dies again with the same error ...
Error in the template file or in the jigdo file?
Ralph
CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
So I've run this locally, and the following files are failing:
http://mirror.centos.org/centos/5.0/os/x86_64/RELEASE-NOTES-%7Bit,ru,pt,fr,n...
be15c179b9b5b10d4935d4624116905a RELEASE-NOTES-cz.html 286abb64487f0fc54218aac5f311aa2a RELEASE-NOTES-de.html fd6bed45eccf4f762afada0a7f3eea6a RELEASE-NOTES-es.html df5ef6acd90d3eb8e11bbc9b86bf6afb RELEASE-NOTES-fr.html c5d3ebf54884c103f96c5a2250539c2b RELEASE-NOTES-it.html 0dcea4e5d2c8b27301ab26cdc8c79511 RELEASE-NOTES-ja.html de38b686c18f72f0f8b24b4a13034efc RELEASE-NOTES-nl.html 5992ff754791c7c7d2b46ee4961f0de6 RELEASE-NOTES-pt.html fec20c8e6087fd11d5d4b35edd3a21df RELEASE-NOTES-ru.html
They all have newer timestamps on the main mirror, then the remaining files in that directory. Apparently someone has gone and changed them after the fact. And most importantly after I generated the jigdo files.
Does this mean the main directories contents can actually change after release? If so... then that's bad.
On 4/18/07, Maciej Żenczykowski zenczykowski@gmail.com wrote:
Could you post which files it's failing for? Maybe one of the mirrors is missing something...
On 4/18/07, Ralph Angenendt ra+centos@br-online.de wrote:
Ralph Angenendt wrote:
Maciej Żenczykowski wrote:
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
I'd seen that in dag's logs as well... was wondering what happened to it. So the package was dropped? oh...
Jigdo 0.7.3 does work on CentOS 5. But: Trying to rebuild CentOS-5.0-x86_64-bin-1of7, jigdo fails with
"Aaargh - 9 files could not be downloaded. This should not happen! Depending on the problem, it may help to retry downloading the missing files."
Trying to redownload I see it copying several release notes files - and then it dies again with the same error ...
Error in the template file or in the jigdo file?
Ralph
CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
(Obviously I can regenerate the jigdo's with the new contents of the mirrors... but it seems like much pointless work...)
[I'll do the regenerate anyway, but we should still have some policy in place to prevent such spurious changes]
On 4/18/07, Maciej Żenczykowski zenczykowski@gmail.com wrote:
So I've run this locally, and the following files are failing:
http://mirror.centos.org/centos/5.0/os/x86_64/RELEASE-NOTES-%7Bit,ru,pt,fr,n...
be15c179b9b5b10d4935d4624116905a RELEASE-NOTES-cz.html 286abb64487f0fc54218aac5f311aa2a RELEASE-NOTES-de.html fd6bed45eccf4f762afada0a7f3eea6a RELEASE-NOTES-es.html df5ef6acd90d3eb8e11bbc9b86bf6afb RELEASE-NOTES-fr.html c5d3ebf54884c103f96c5a2250539c2b RELEASE-NOTES-it.html 0dcea4e5d2c8b27301ab26cdc8c79511 RELEASE-NOTES-ja.html de38b686c18f72f0f8b24b4a13034efc RELEASE-NOTES-nl.html 5992ff754791c7c7d2b46ee4961f0de6 RELEASE-NOTES-pt.html fec20c8e6087fd11d5d4b35edd3a21df RELEASE-NOTES-ru.html
They all have newer timestamps on the main mirror, then the remaining files in that directory. Apparently someone has gone and changed them after the fact. And most importantly after I generated the jigdo files.
Does this mean the main directories contents can actually change after release? If so... then that's bad.
On 4/18/07, Maciej Żenczykowski zenczykowski@gmail.com wrote:
Could you post which files it's failing for? Maybe one of the mirrors is missing something...
On 4/18/07, Ralph Angenendt ra+centos@br-online.de wrote:
Ralph Angenendt wrote:
Maciej Żenczykowski wrote:
Not CentOS 5, as I have found out. Looks like upstream dropped w3c-libwww-* which is needed for jigdo ...
I'd seen that in dag's logs as well... was wondering what happened to it. So the package was dropped? oh...
Jigdo 0.7.3 does work on CentOS 5. But: Trying to rebuild CentOS-5.0-x86_64-bin-1of7, jigdo fails with
"Aaargh - 9 files could not be downloaded. This should not happen! Depending on the problem, it may help to retry downloading the missing files."
Trying to redownload I see it copying several release notes files - and then it dies again with the same error ...
Error in the template file or in the jigdo file?
Ralph
CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
Maciej Żenczykowski wrote:
So I've run this locally, and the following files are failing:
http://mirror.centos.org/centos/5.0/os/x86_64/RELEASE-NOTES-%7Bit,ru,pt,fr,n...
Yes, those are the files.
They all have newer timestamps on the main mirror, then the remaining files in that directory. Apparently someone has gone and changed them after the fact. And most importantly after I generated the jigdo files.
Does this mean the main directories contents can actually change after release? If so... then that's bad.
They shouldn't. These seem to have changed because of the charset fuckup with the release notes on the ISOs.
But normally stuff in os/ shouldn't change after release.
On another note: I was able to rebuild all other x86_64 CD isos and they do have the same checksums as the ISOs on the mirrors.
And I asked dag to rebuild jigdo-0.7.3 for rpmforge, so it will be available for CentOS 5 also.
More testing from me in the next few days, I have to go through the documentation first and that needs a little time.
Cheers and thanks,
Ralph
Regenerated and uploaded the new jigdoes, old ones renamed with a trailing '_'.
CentOS-5.0-i386-bin-1of6.jigdo CentOS-5.0-i386-bin-1of6.template CentOS-5.0-i386-bin-DVD.jigdo CentOS-5.0-i386-bin-DVD.template CentOS-5.0-x86_64-bin-1of7.jigdo CentOS-5.0-x86_64-bin-1of7.template CentOS-5.0-x86_64-bin-DVD.jigdo CentOS-5.0-x86_64-bin-DVD.template md5sum.txt sha1sum.txt
(basically both the first CD and the DVD contain the release notes, so they all needed a rebuild, I verified all the others didn't change/still work (at least on my local mirror)).
They shouldn't. These seem to have changed because of the charset fuckup with the release notes on the ISOs.
Hmm. Well, in the future we should either not make such changes (I'm assuming we want to keep valid jigdoes around) or we'll have to regenerate the jigdo/template files in such a situation. I think sticking the new fixed files under a different name would also be a valid solution (you can add new files, what you can't do is you can't change existing files required by jigdo). We could also possibly run jigdo with a minimum file size requirement closer to 200KB which would allow changes to the smaller (ie. text) files.
However I would vote for keeping the os tree totally frozen from the point of release.
Although increasing the min file size would possibly also with only a slight template file size increase reduce the number of files needing downloading...
Here's a list of the current number of external (ie. in mirror) files in each jigdo/template: CentOS-4.4-SRPMS-src1of4.jigdo: 204 CentOS-4.4-SRPMS-src2of4.jigdo: 205 CentOS-4.4-SRPMS-src3of4.jigdo: 208 CentOS-4.4-SRPMS-src4of4.jigdo: 208 CentOS-4.4-i386-ServerCD.jigdo: 606 CentOS-4.4-i386-bin1of4.jigdo: 2138 CentOS-4.4-i386-bin2of4.jigdo: 255 CentOS-4.4-i386-bin3of4.jigdo: 381 CentOS-4.4-i386-bin4of4.jigdo: 340 CentOS-4.4-i386-binDVD.jigdo: 3104 CentOS-4.4-x86_64-bin1of4.jigdo: 2368 CentOS-4.4-x86_64-bin2of4.jigdo: 256 CentOS-4.4-x86_64-bin3of4.jigdo: 411 CentOS-4.4-x86_64-bin4of4.jigdo: 474 CentOS-4.4-x86_64-binDVD.jigdo: 3508 CentOS-5.0-i386-bin-1of6.jigdo: 559 CentOS-5.0-i386-bin-2of6.jigdo: 418 CentOS-5.0-i386-bin-3of6.jigdo: 421 CentOS-5.0-i386-bin-4of6.jigdo: 357 CentOS-5.0-i386-bin-5of6.jigdo: 422 CentOS-5.0-i386-bin-6of6.jigdo: 214 CentOS-5.0-i386-bin-DVD.jigdo: 2386 CentOS-5.0-x86_64-bin-1of7.jigdo: 575 CentOS-5.0-x86_64-bin-2of7.jigdo: 555 CentOS-5.0-x86_64-bin-3of7.jigdo: 520 CentOS-5.0-x86_64-bin-4of7.jigdo: 398 CentOS-5.0-x86_64-bin-5of7.jigdo: 492 CentOS-5.0-x86_64-bin-6of7.jigdo: 348 CentOS-5.0-x86_64-bin-7of7.jigdo: 219 CentOS-5.0-x86_64-bin-DVD.jigdo: 3100
To me it doesn't look to bad, seems average file size is easily above 1MB, so the number of required files (ie. connection establishment/handshake/etc) shouldn't drastically impact performance.
(ie. downloading via jigdo shouldn't be much slower than downloading the iso directly - assuming you're downloading from the same mirror).
Jigdo also has the added benefit of reducing the amount of data mirrors need to sync, and allowing us to release stuff like rolled up update cds with almost no mirror space/bandwidth overhead (same thing for anything else that uses in-tree files, live ServerCD's, but not LiveCD's [which are totally recompressed and thus have no common parts to the os tree].
On another note: I was able to rebuild all other x86_64 CD isos and they do have the same checksums as the ISOs on the mirrors.
Yes :) I did verify this at home.
And I asked dag to rebuild jigdo-0.7.3 for rpmforge, so it will be available for CentOS 5 also.
Cool.
More testing from me in the next few days, I have to go through the documentation first and that needs a little time.
What we need to figure out is a better way to specify the source mirror, possibly hack around in the jigdo-lite script. It might also make sense to see if we can improve (parallelize) jigdo-lite download performance to make it work better on bigger pipes (and possibly not require quite so much temporary space).
Maciej Żenczykowski wrote:
More testing from me in the next few days, I have to go through the documentation first and that needs a little time.
What we need to figure out is a better way to specify the source mirror, possibly hack around in the jigdo-lite script.
That should be fairly easy - just do this as an option to jigdo-lite: If there is an --mirror http://mirror.example.com/linux/centos, then use that, if not then use the ones from the mirrors variable in the jigdo-lite script. Which will only work if CentOS distributes jigdo, we cannot force Dries to hardwire CentOS requirements into the rpmforge package.
It might also make sense to see if we can improve (parallelize) jigdo-lite download performance to make it work better on bigger pipes (and possibly not require quite so much temporary space).
One step after the other?
That's why I said "I need some time to go through the documentation", because at the moment I only have a vague idea of how jigdo works =:)
But I'll keep in touch.
Cheers,
Ralph
Ralph Angenendt wrote:
Maciej Żenczykowski wrote:
More testing from me in the next few days, I have to go through the documentation first and that needs a little time.
What we need to figure out is a better way to specify the source mirror, possibly hack around in the jigdo-lite script.
That should be fairly easy - just do this as an option to jigdo-lite: If there is an --mirror http://mirror.example.com/linux/centos, then use that, if not then use the ones from the mirrors variable in the jigdo-lite script. Which will only work if CentOS distributes jigdo, we cannot force Dries to hardwire CentOS requirements into the rpmforge package.
jigdo needs more than one mirror; I could have 4.3 here in a local mirror and wish to use that first, then another the other side of my modem, and maybe another - perhaps one's broken.
Repeating --mirror is probably simplest, the script could tuck the names away in an array.
?? How does Debian do this? Obviously doesn't use apt-get config files, I've run it on CentOS 4.
<looks at a config file> [summer@bilby ~]$ ssh cdm cat /home/summer/.jigdo-lite jigdo='' debianMirror='http://ftp.wa.au.debian.org/debian/' nonusMirror='http://ftp.wa.au.debian.org/debian-non-US/' tmpDir='.' jigdoOpts='--cache jigdo-file-cache.db' wgetOpts=' --continue --timeout=300' scanMenu='//misc/I1'
Oh, it doesn't really. Making those *Mirror lines into an array should work well, for us and for Debian.
It might also make sense to see if we can improve (parallelize) jigdo-lite download performance to make it work better on bigger pipes (and possibly not require quite so much temporary space).
One step after the other?
There's no reason jigdo can't download more than the default, and I have it doing that.
Years ago, I had some patches to wget 1.5.3 (just for ftp) that allowed it to run user-specified programs/scripts at various points: 1. Just before fetching a file 2. Just after fetching a file. 3. I forget
I used the first to test whether I had internet connectivity, and waited until I had. The second could be used to signal something that "here is a file to poke into the iso" in the particular case of jigdo.
I'm generally not keen on running concurrent downloads from a remote site. I figure if the sender's faster than me, I should already be running close to capacity. If I'm faster than the sender, it would be pretty unfriendly to further swamp him.
ftp sites generally limit the number of connexions, and some (rsync at least) sites around threaten to block your IP for 24 hours if you run more than two connexions. I don't wish to find out whether they're serious.
On Thursday April 19 2007, Ralph Angenendt wrote:
Maciej Żenczykowski wrote:
More testing from me in the next few days, I have to go through the documentation first and that needs a little time.
What we need to figure out is a better way to specify the source mirror, possibly hack around in the jigdo-lite script.
That should be fairly easy - just do this as an option to jigdo-lite: If there is an --mirror http://mirror.example.com/linux/centos, then use that, if not then use the ones from the mirrors variable in the jigdo-lite script. Which will only work if CentOS distributes jigdo, we cannot force Dries to hardwire CentOS requirements into the rpmforge package.
I don't know if it's possible with jigdo, but maybe we could add a separate jigdo-centos package containing some scripts (like /usr/bin/jigdo-centos) and/or config files which simply use the normal jigdo package?
kind regards, Dries
-
Dries Verachtert -
John Summerfield -
Maciej Żenczykowski -
Ralph Angenendt