Shad,
----- "Shad L. Lords" slords@lordsfam.net wrote:
Scott Dowdle wrote:
Shad,
----- "Shad L. Lords" slords@lordsfam.net wrote:
From the os directory something like this should produce jigdo files/templates for all associated iso files.
for arch in i386 x86_64 do rsync -a --delete --delete-excluded \ --exclude 'repodata/' --exclude 'EULA' \ --exclude 'GPL' --exclude 'NOTES/' \ --exclude 'RELEASE-NOTES*' --exclude 'RPM-GPG-KEY*' \ --link-dest=../$arch/ $arch/ ${arch}.jigdo/
for iso in $(find ../isos/i386/ -name '*.iso' -size +100000) do jigdo-file make-template --force -i $iso \ -j ${iso%.iso}.jigdo -t ${iso%.iso}.template \ --label CentOS=$arch.jigdo/ \ --uriCentOS=http://mirror.centos.org/centos/5/os/$arch/
\ ${arch}.jigdo// done done
After the files/templates are created they can be moved to a different/better location as well as updating the Servers section if desired.
Thanks for the recipe. Worked for me. My only complaint is that the .template files are about 2/3rds the size of the .iso they are for... so you end up downloading 2/3rds of an ISO as a template, then all of the files that make up the media... and then have to process it for your own .iso.
If you have a local copy of a the repos it pulls from that is handy... but if not, it seems to be significantly more downloading for the end user than just the .iso media... and the only transfer savings for the provider of the .jigdo (rather than providing the .isos) is that 1/3rd-ish savings over the .iso.
Is that how it is supposed to work?
I don't have the DVD iso so I'm not sure about that one but when I ran it for the CD iso's I got a template of about 15MB for disc1 and < 1MB for each of the other discs. If you remove the exclude for NOTES/ from the rsync then the template drops to < 5MB (but you have to download all the release notes).
Are you sure you had a complete rsync of the os directory without excluding anything prior to running the above command? This was really more a recipe for the CentOS folks to as an example of how easy it would be to create jigdo/template files from their master area that could then be pushed to the mirrors for everyone to enjoy.
Yes, I had complete os/ trees. I basically mounted the DVD iso and copied everything from it. Here's the results I get after running your script:
[root@img i386]# pwd /var/www/html/linux/centos/5/isos/i386 [root@img i386]# ls -lha total 13G drwxr-xr-x 2 root root 4.0K May 6 14:37 . drwxr-xr-x 4 root root 4.0K Mar 27 11:58 .. -rw-r--r-- 1 root root 624M Mar 21 10:17 CentOS-5.3-i386-bin-1of6.iso -rw-r--r-- 1 root root 13K May 6 14:48 CentOS-5.3-i386-bin-1of6.jigdo -rw-r--r-- 1 root root 482M May 6 14:48 CentOS-5.3-i386-bin-1of6.template -rw-r--r-- 1 root root 637M Mar 21 10:18 CentOS-5.3-i386-bin-2of6.iso -rw-r--r-- 1 root root 13K May 6 15:22 CentOS-5.3-i386-bin-2of6.jigdo -rw-r--r-- 1 root root 411M May 6 15:22 CentOS-5.3-i386-bin-2of6.template -rw-r--r-- 1 root root 635M Mar 21 10:19 CentOS-5.3-i386-bin-3of6.iso -rw-r--r-- 1 root root 25K May 6 14:43 CentOS-5.3-i386-bin-3of6.jigdo -rw-r--r-- 1 root root 388M May 6 14:43 CentOS-5.3-i386-bin-3of6.template -rw-r--r-- 1 root root 632M Mar 21 10:20 CentOS-5.3-i386-bin-4of6.iso -rw-r--r-- 1 root root 14K May 6 15:12 CentOS-5.3-i386-bin-4of6.jigdo -rw-r--r-- 1 root root 354M May 6 15:12 CentOS-5.3-i386-bin-4of6.template -rw-r--r-- 1 root root 638M Mar 21 10:21 CentOS-5.3-i386-bin-5of6.iso -rw-r--r-- 1 root root 5.1K May 6 15:07 CentOS-5.3-i386-bin-5of6.jigdo -rw-r--r-- 1 root root 383M May 6 15:07 CentOS-5.3-i386-bin-5of6.template -rw-r--r-- 1 root root 625M Mar 21 10:22 CentOS-5.3-i386-bin-6of6.iso -rw-r--r-- 1 root root 3.3K May 6 15:16 CentOS-5.3-i386-bin-6of6.jigdo -rw-r--r-- 1 root root 599M May 6 15:16 CentOS-5.3-i386-bin-6of6.template -rw-r--r-- 1 root root 3.7G Mar 31 08:38 CentOS-5.3-i386-bin-DVD.iso -rw-r--r-- 1 root root 71K May 6 15:04 CentOS-5.3-i386-bin-DVD.jigdo -rw-r--r-- 1 root root 2.6G May 6 15:04 CentOS-5.3-i386-bin-DVD.template -rw-r--r-- 1 root root 8.3M Mar 21 09:05 CentOS-5.3-i386-netinstall.iso -rw-r--r-- 1 root root 505 Mar 31 08:33 md5sum.txt -rw-r--r-- 1 root root 741 Mar 31 08:33 md5sum.txt.asc -rw-r--r-- 1 root root 569 Mar 31 08:33 sha1sum.txt -rw-r--r-- 1 root root 805 Mar 31 08:33 sha1sum.txt.asc
As you can see, my .template files are about 2/3rds the size of the .iso they are for. What's up with that?
When I ran your script I was inside of the os/ dir... and there are i386 and x86_64 dirs there... and it as part of the process created i386.jigdo/ and x86_64.jigdo/ directories... and copied everything from the i386 and the x86_64 dirs into the .jigdo directories... so maybe I'm doing something wrong?
TYL,
Scott Dowdle wrote:
Are you sure you had a complete rsync of the os directory without excluding anything prior to running the above command? This was really more a recipe for the CentOS folks to as an example of how easy it would be to create jigdo/template files from their master area that could then be pushed to the mirrors for everyone to enjoy.
Yes, I had complete os/ trees. I basically mounted the DVD iso and copied everything from it. Here's the results I get after running your script:
I'd guess that your basic os tree you created isn't in the same format as the mirrors then. I'm running it against the non-dvd mirror that I have and my results are below.
[root@img i386]# pwd /var/www/html/linux/centos/5/isos/i386 [root@img i386]# ls -lha total 13G drwxr-xr-x 2 root root 4.0K May 6 14:37 . drwxr-xr-x 4 root root 4.0K Mar 27 11:58 .. -rw-r--r-- 1 root root 624M Mar 21 10:17 CentOS-5.3-i386-bin-1of6.iso -rw-r--r-- 1 root root 13K May 6 14:48 CentOS-5.3-i386-bin-1of6.jigdo -rw-r--r-- 1 root root 482M May 6 14:48 CentOS-5.3-i386-bin-1of6.template -rw-r--r-- 1 root root 637M Mar 21 10:18 CentOS-5.3-i386-bin-2of6.iso -rw-r--r-- 1 root root 13K May 6 15:22 CentOS-5.3-i386-bin-2of6.jigdo -rw-r--r-- 1 root root 411M May 6 15:22 CentOS-5.3-i386-bin-2of6.template -rw-r--r-- 1 root root 635M Mar 21 10:19 CentOS-5.3-i386-bin-3of6.iso -rw-r--r-- 1 root root 25K May 6 14:43 CentOS-5.3-i386-bin-3of6.jigdo -rw-r--r-- 1 root root 388M May 6 14:43 CentOS-5.3-i386-bin-3of6.template -rw-r--r-- 1 root root 632M Mar 21 10:20 CentOS-5.3-i386-bin-4of6.iso -rw-r--r-- 1 root root 14K May 6 15:12 CentOS-5.3-i386-bin-4of6.jigdo -rw-r--r-- 1 root root 354M May 6 15:12 CentOS-5.3-i386-bin-4of6.template -rw-r--r-- 1 root root 638M Mar 21 10:21 CentOS-5.3-i386-bin-5of6.iso -rw-r--r-- 1 root root 5.1K May 6 15:07 CentOS-5.3-i386-bin-5of6.jigdo -rw-r--r-- 1 root root 383M May 6 15:07 CentOS-5.3-i386-bin-5of6.template -rw-r--r-- 1 root root 625M Mar 21 10:22 CentOS-5.3-i386-bin-6of6.iso -rw-r--r-- 1 root root 3.3K May 6 15:16 CentOS-5.3-i386-bin-6of6.jigdo -rw-r--r-- 1 root root 599M May 6 15:16 CentOS-5.3-i386-bin-6of6.template -rw-r--r-- 1 root root 3.7G Mar 31 08:38 CentOS-5.3-i386-bin-DVD.iso -rw-r--r-- 1 root root 71K May 6 15:04 CentOS-5.3-i386-bin-DVD.jigdo -rw-r--r-- 1 root root 2.6G May 6 15:04 CentOS-5.3-i386-bin-DVD.template -rw-r--r-- 1 root root 8.3M Mar 21 09:05 CentOS-5.3-i386-netinstall.iso -rw-r--r-- 1 root root 505 Mar 31 08:33 md5sum.txt -rw-r--r-- 1 root root 741 Mar 31 08:33 md5sum.txt.asc -rw-r--r-- 1 root root 569 Mar 31 08:33 sha1sum.txt -rw-r--r-- 1 root root 805 Mar 31 08:33 sha1sum.txt.asc
As you can see, my .template files are about 2/3rds the size of the .iso they are for. What's up with that?
No idea what is up with yours but here is mine after running the command:
[root@server i386]# pwd /mirrors/centos/5.3/isos/i386 [root@server i386]# ls -lha total 3.8G drwxr-sr-x 2 root 4.0K May 7 11:25 . drwxr-sr-x 4 root 4.0K Mar 18 03:58 .. -rw-r--r-- 2 root 624M Mar 21 10:17 CentOS-5.3-i386-bin-1of6.iso -rw-r--r-- 1 root 34K May 7 11:32 CentOS-5.3-i386-bin-1of6.jigdo -rw-r--r-- 1 root 15M May 7 11:32 CentOS-5.3-i386-bin-1of6.template -rw-r--r-- 1 root 297K Mar 31 17:18 CentOS-5.3-i386-bin-1to6.torrent -rw-r--r-- 2 root 637M Mar 21 10:18 CentOS-5.3-i386-bin-2of6.iso -rw-r--r-- 1 root 34K May 7 11:35 CentOS-5.3-i386-bin-2of6.jigdo -rw-r--r-- 1 root 70K May 7 11:35 CentOS-5.3-i386-bin-2of6.template -rw-r--r-- 2 root 635M Mar 21 10:19 CentOS-5.3-i386-bin-3of6.iso -rw-r--r-- 1 root 48K May 7 11:39 CentOS-5.3-i386-bin-3of6.jigdo -rw-r--r-- 1 root 92K May 7 11:39 CentOS-5.3-i386-bin-3of6.template -rw-r--r-- 2 root 632M Mar 21 10:20 CentOS-5.3-i386-bin-4of6.iso -rw-r--r-- 1 root 37K May 7 11:43 CentOS-5.3-i386-bin-4of6.jigdo -rw-r--r-- 1 root 73K May 7 11:43 CentOS-5.3-i386-bin-4of6.template -rw-r--r-- 2 root 638M Mar 21 10:21 CentOS-5.3-i386-bin-5of6.iso -rw-r--r-- 1 root 20K May 7 11:46 CentOS-5.3-i386-bin-5of6.jigdo -rw-r--r-- 1 root 43K May 7 11:46 CentOS-5.3-i386-bin-5of6.template -rw-r--r-- 2 root 625M Mar 21 10:22 CentOS-5.3-i386-bin-6of6.iso -rw-r--r-- 1 root 11K May 7 11:48 CentOS-5.3-i386-bin-6of6.jigdo -rw-r--r-- 1 root 26K May 7 11:48 CentOS-5.3-i386-bin-6of6.template -rw-r--r-- 1 root 297K Mar 31 17:18 CentOS-5.3-i386-bin-DVD.torrent -rw-r--r-- 2 root 8.3M Mar 21 09:05 CentOS-5.3-i386-netinstall.iso -rw-r--r-- 1 root 505 Mar 31 04:26 md5sum.txt -rw-r--r-- 1 root 741 Mar 31 04:26 md5sum.txt.asc -rw-r--r-- 1 root 569 Mar 31 04:26 sha1sum.txt -rw-r--r-- 1 root 805 Mar 31 04:26 sha1sum.txt.asc
You can see that the only template over 100K is the one for disc 1. I did notice that there is one place on my script that hardcodes i386. If you replace that with $arch then it may/should fix your issue.
-Shad
Shad,
----- "Shad L. Lords" slords@lordsfam.net wrote:
I'd guess that your basic os tree you created isn't in the same format as the mirrors then. I'm running it against the non-dvd mirror that I have and my results are below.
I don't know how it isn't in the "same format". I mount the i386 DVD iso to os/i386/ and the x86_64 DVD iso to os/x86_64/. I've been using those directories for my [base] repositories and they work fine from yum to install packages and package groups.
Look for yourself here:
http://img.cs.montana.edu/linux/centos/5/os/i386/
and
http://img.cs.montana.edu/linux/centos/5/os/x86_64/
Those map out to /var/www/html/linux/centos/5/os/{arch} on the filesystem and I run your jigdo script from /var/www/html/linux/centos/5/os/. Is that correct?
As you can see, my .template files are about 2/3rds the size of the
.iso they are for. What's up with that?
No idea what is up with yours but here is mine after running the command:
[root@server i386]# pwd /mirrors/centos/5.3/isos/i386 [root@server i386]# ls -lha total 3.8G drwxr-sr-x 2 root 4.0K May 7 11:25 . drwxr-sr-x 4 root 4.0K Mar 18 03:58 .. -rw-r--r-- 2 root 624M Mar 21 10:17 CentOS-5.3-i386-bin-1of6.iso -rw-r--r-- 1 root 34K May 7 11:32 CentOS-5.3-i386-bin-1of6.jigdo -rw-r--r-- 1 root 15M May 7 11:32 CentOS-5.3-i386-bin-1of6.template
[...]
I'm glad to see yours are coming out correctly. I wonder what I'm doing wrong?
TYL,
Shad,
----- "Shad L. Lords" slords@lordsfam.net wrote:
I'd guess that your basic os tree you created isn't in the same format as the mirrors then. I'm running it against the non-dvd mirror that I have and my results are below.
Ok, I think I got it working. I removed the rsync step and had the last item to jigdo-file be ${arch}//. I guess you wanted it a little different because you were excluding some things. That seems to work.
TYL,
Shad,
----- Scott Dowdle wrote:
I'd guess that your basic os tree you created isn't in the same format as the mirrors then. I'm running it against the non-dvd mirror that I have and my results are below.
Ok, I think I got it working. I removed the rsync step and had the last item to jigdo-file be ${arch}//. I guess you wanted it a little different because you were excluding some things. That seems to work.
Well, it worked in the sense that it generated reasonably sized .jigdo and .template files but they didn't work. When I download the .jigdo file on a client machine and run jigdo-lite against it...it doesn't seem to know where to download anything from and just looks on the local filesystem for all files, finds none, and gives up.
It certainly isn't your job to teach me jigdo-file. I'll figure it out. I'm just glad you prompted me to give it a try.
TYL,
Shad L. Lords wrote:
I'd guess that your basic os tree you created isn't in the same format as the mirrors then. I'm running it against the non-dvd mirror that I have and my results are below.
-rw-r--r-- 1 root 34K May 7 11:32 CentOS-5.3-i386-bin-1of6.jigdo
Can you mail that file around? The first 50 or so lines would be enough. I still seem to have a problem with label and URI generation in the .jigdo files.
This is the file I get with the following command (which is missing the --uri section):
jigdo-file make-template \ --image=/home/admin/iso/CentOS-5.3-i386-bin-DVD.iso \ /local/mirror/linux/centos/5/os/i386/ \ --label CentOS5-i386=/local/mirror/linux/centos/5/os/i386/ \ --jigdo=/home/admin/iso/result/CentOS5-i386.jigdo \ --template=/home/admin/iso/result/CentOS5-i386.template
[Parts] GFckbEI56dspBqCKRYqrwA=A:local/mirror/linux/centos/5/os/i386/CentOS/Cluster_Administr ation-as-IN-5.2-1.el5.centos.noarch.rpm 68-TbW5iS6FcKlD_RWzOJQ=A:local/mirror/linux/centos/5/os/i386/CentOS/Cluster_Administr ation-bn-IN-5.2-1.el5.centos.noarch.rpm iqQceDWNEj-0pWNDYajgQw=A:local/mirror/linux/centos/5/os/i386/CentOS/Cluster_Administr ation-de-DE-5.2-1.el5.centos.noarch.rpm
Cheers.
Ralph
-
Ralph Angenendt -
Scott Dowdle -
Shad L. Lords