Hi all, since I'm installing several copies of CentOS I was thinking of using some system to save bandwidth.
I started looking at mrepo, but I noticed that, if I let him sync with updates and rpmforge repos, I'll end with a _lot_ of unneeded files. Also I have to modify the various yum repos configuration files for it to work.
What I would like to achieve is just download the same file only once: does anyone know if there is something like a "yum proxy-cache" or if it is possible to instruct squid to caches at least the rpm's and to serve them regardless of the server (by default yum uses a server list, so it would be nice to have the system working without having to fiddle too much with config files)
tia
Lorenzo
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Lorenzo wrote:
since I'm installing several copies of CentOS I was thinking of using some system to save bandwidth.
I started looking at mrepo, but I noticed that, if I let him sync with updates and rpmforge repos, I'll end with a _lot_ of unneeded files. Also I have to modify the various yum repos configuration files for it to work.
I'm not sure if this answers what you want or not, but I just use rsync and point it to a lower level mirror as per the CentOS folks wishes.
Put it in a bash script, enter it into cron, and then it syncs every night with a mirror of choice. There's nothing to worry about configuring, except an exclude file.
For example:
/usr/bin/rsync -v -r -t -p --delete -l \ - --exclude-from=/usr/local/etc/centos.exclude \ rsync://one.of.the.mirrors/centos \ /srv/centos/updates 2>&1 >> /srv/centos/logs/rsync.log
My exclude file then skips all x86_64, SRPMS, ia64, isos, ppc, dvds, etc. and just pulls the rpms and whatever else I only need. Whatever you want to skip you can just put in here. Of course, this gets just the base CentOS stuff. You can even skip by release if you wanted to only get 4.4 stuff.
Then, I have a similar rsync going then to pull things from rpmforge that I want, because I don't pull the entire repo. I only pull, again, what I need to save some bandwidth.
What I would like to achieve is just download the same file only once:
I don't know exactly what you mean here, but the initial rsync will be large and consume a lot of time, but then after that rsync only grabs the files that have changed. So only the rpms that are new and changed will be downloaded in subsequent syncs.
I hope this helps you a little. This is how I create a local yum mirror without the hassle of configuring a lot of things. If I recall, the CentOS folks just don't want you pulling from say their main mirror, mirror.centos.org, but instead pick a public mirror.
Hope this helps you. Let me know if you want to see the exclude file.
Regards, Max
On 5/17/07, Max Hetrick btmanmeh@verizon.net wrote:
Lorenzo wrote:
since I'm installing several copies of CentOS I was thinking of using some system to save bandwidth.
I started looking at mrepo, but I noticed that, if I let him sync with updates and rpmforge repos, I'll end with a _lot_ of unneeded files. Also I have to modify the various yum repos configuration files for it to work.
I'm not sure if this answers what you want or not, but I just use rsync and point it to a lower level mirror as per the CentOS folks wishes.
<snip>
To take Max's suggestion a bit farther if you do the local mirror you can then use the boot.iso file found under /$release_number$/os/i386/images/ and create a bootable CD from that. It will give you the option to choose a location for the source files. I have my local yum mirror under http://server/centos/$release_number/ and I can enter the server name and then the path /centos/4/os/i386 and it will then install over the wire the rest of the OS instead of swapping disks. Then after I log in I go to the /etc/yum.repos.d/ folder and rename the existing CentOS-Base.repo to CentOS-Base.old and wget my edited CentOS-Base.repo file from that same web server. Now when I do a yum update it will use the new version and pull all the updates right from the local server.
I also have to use the centosplus repo for the kernel so that is already changed in the .repo file on the web server.
It does take a little bit of time to set up the initial mirror and to do the initial rsync but now when new updates come down they were synced over night when the internet connection is far less taxed and I can do the individual updates much faster.
If you have a whole lot you could even look at making a kickstart file and putting it up on the webserver to use in combination with the boot.iso CD and add a script to the end of the kickstart to do the rename and download of the new .repo file.
Rob
<snip>
Put it in a bash script, enter it into cron, and then it syncs every night with a mirror of choice. There's nothing to worry about configuring, except an exclude file.
For example:
/usr/bin/rsync -v -r -t -p --delete -l \
- --exclude-from=/usr/local/etc/centos.exclude \
rsync://one.of.the.mirrors/centos \ /srv/centos/updates 2>&1 >> /srv/centos/logs/rsync.log
My exclude file then skips all x86_64, SRPMS, ia64, isos, ppc, dvds, etc. and just pulls the rpms and whatever else I only need. Whatever you want to skip you can just put in here. Of course, this gets just the base CentOS stuff. You can even skip by release if you wanted to only get 4.4 stuff.
<snip>
FYI - I used to do this too, then one day the mirror changed their config or something and *bam!* - all my repo was nuked! The cause? - " --delete" worked exactly as expected. However from my reading of man rsync there is a flag (--max-delete) to ensure that not too many files are deleted at once, so that someone elses mirror config errors dont do this to your precious 5G of repo. My advice would be that if your bandwidth is not tooooo abundant, lookup this switch: --max-delete=NUM
Oh well *sigh* - bandwidth is cheap i guess.
MrKiwi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
MrKiwi wrote:
My advice would be that if your bandwidth is not tooooo abundant, lookup this switch: --max-delete=NUM
Oh, ok. Yeah, I'll definitely check into this switch for rsync. I guess I wouldn't have really thought about that until happened. Thanks!!
Max
Max Hetrick ha scritto:
MrKiwi wrote:
My advice would be that if your bandwidth is not tooooo abundant, lookup this switch: --max-delete=NUM
Oh, ok. Yeah, I'll definitely check into this switch for rsync. I guess I wouldn't have really thought about that until happened. Thanks!!
Max
Thank all of you guys for the help, but the thing I would like to avoid is really the first rsync (mrepo just doest that and is nicely configurable by architecture and so on).
What I was looking for is something like debian's apt-cache or approxy for yum.
Lorenzo
Lorenzo wrote:
Max Hetrick ha scritto:
MrKiwi wrote:
My advice would be that if your bandwidth is not tooooo abundant, lookup this switch: --max-delete=NUM
Oh, ok. Yeah, I'll definitely check into this switch for rsync. I guess I wouldn't have really thought about that until happened. Thanks!!
Max
Thank all of you guys for the help, but the thing I would like to avoid is really the first rsync (mrepo just doest that and is nicely configurable by architecture and so on).
What I was looking for is something like debian's apt-cache or approxy for yum.
Lorenzo _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I seeded mine from the DVD iso - not sure if i did it right though :)
MrKiwi
Miguel Medalha spake the following on 5/17/2007 12:46 PM:
CentOS 4.5 is here:
http://mirror.chpc.utah.edu/pub/centos/4.5/
The DVD ISO is there too :-)
I'm sure there will be an official announcement as soon as more mirrors are in sync.
On 5/17/07, Miguel Medalha miguelmedalha@sapo.pt wrote:
CentOS 4.5 is here:
Okay, I know you guys are eager for releases.. but do you think you could at least let US be the ones to announce the release? Pretty please? :-P
Jim Perrin wrote:
On 5/17/07, Miguel Medalha miguelmedalha@sapo.pt wrote:
CentOS 4.5 is here:
Okay, I know you guys are eager for releases.. but do you think you could at least let US be the ones to announce the release? Pretty please? :-P
You cant break a habit that easily Jim - you know we're like vultures circling a carcass on *every* release :) *background music of "food, glorious food" from Ice Age* Damn you slow mirrors - stop breathing so we can start eatin'!
I bet MS would have wished that kind of eagerness for vista.
MrKiwi.
On 5/17/07, MrKiwi mrkiwi@gmail.com wrote:
I bet MS would have wished that kind of eagerness for vista.
Please watch your language here!
;^)
CentOS 4.5 is here:
Only partially there (or only partially upstream). There have packages in updates but no SRPMS.
-Shad
I finished downloading "CentOS-4.5-i386-binDVD.iso" about one hour ago...
What do you mean by "no RPMS"?
Are you referring to something other than this?
Miguel Medalha spake the following on 5/17/2007 3:21 PM:
What do you mean by "no RPMS"?
Are you referring to something other than this?
You can usually see when the mirror is complete when the 4 link points to 4.5 ( it is still pointing to 4.4 as of 15:28 PDT)
What do you mean by "no SRPMS"?
Are you referring to something other than this?
Not quite. I'm refering to there being packages in here:
http://mirror.chpc.utah.edu/pub/centos/4.5/updates/i386/RPMS/
Not no SRPMS in here:
http://mirror.chpc.utah.edu/pub/centos/4.5/updates/SRPMS/
-Shad
-
Jim Perrin -
Lorenzo -
Mark Hull-Richter -
Max Hetrick -
Miguel Medalha -
MrKiwi -
Rob Lines -
Scott Silva -
Shad L. Lords