Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
James A. Peltier wrote:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
Hi you can create your own repository pointing yum on your CentOS machines to that.
Createrepo is the main command you need
James A. Peltier ha scritto:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
I asked the same thing some time ago, but I wasn't able to came with a solution. I set up a local mirror of updates with mrepo: is a waste of bandwidth (I am mirroring tons of apps I don't need, laguage packs and so on), but is the "best" solution I found. At this time I guess we should ask the people that develop yum: does anyone know how to get in touch with them?
Regards
Lorenzo Quatrini
On Wed, 10 Oct 2007, Lorenzo wrote:
James A. Peltier ha scritto:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
I asked the same thing some time ago, but I wasn't able to came with a solution. I set up a local mirror of updates with mrepo: is a waste of bandwidth (I am mirroring tons of apps I don't need, laguage packs and so on), but is the "best" solution I found. At this time I guess we should ask the people that develop yum: does anyone know how to get in touch with them?
Install squid, configure it so it only caches those files from a certain location or with a certain extension and then give it a large enough pool of diskspace so it can cache everything you require.
Then use that as a proxy for all your apt/yum depsolvers.
There is really not that much to it.
Dag Wieers ha scritto:
On Wed, 10 Oct 2007, Lorenzo wrote:
James A. Peltier ha scritto:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
I asked the same thing some time ago, but I wasn't able to came with a solution. I set up a local mirror of updates with mrepo: is a waste of bandwidth (I am mirroring tons of apps I don't need, laguage packs and so on), but is the "best" solution I found. At this time I guess we should ask the people that develop yum: does anyone know how to get in touch with them?
Install squid, configure it so it only caches those files from a certain location or with a certain extension and then give it a large enough pool of diskspace so it can cache everything you require.
Then use that as a proxy for all your apt/yum depsolvers.
There is really not that much to it.
I had troubles with this setup with corrupted metadata (or similar, I don't remember); also you have to disable the mirrors, otherwise squid would download the same file for each mirror. I agree that configuring local repos or squid could be of some help, but I don't think that there is a real solution, but just a work-around.
On Wed, 10 Oct 2007, Lorenzo wrote:
Dag Wieers ha scritto:
On Wed, 10 Oct 2007, Lorenzo wrote:
James A. Peltier ha scritto:
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
I asked the same thing some time ago, but I wasn't able to came with a solution. I set up a local mirror of updates with mrepo: is a waste of bandwidth (I am mirroring tons of apps I don't need, laguage packs and so on), but is the "best" solution I found. At this time I guess we should ask the people that develop yum: does anyone know how to get in touch with them?
Install squid, configure it so it only caches those files from a certain location or with a certain extension and then give it a large enough pool of diskspace so it can cache everything you require.
Then use that as a proxy for all your apt/yum depsolvers.
There is really not that much to it.
I had troubles with this setup with corrupted metadata (or similar, I don't remember); also you have to disable the mirrors, otherwise squid would download the same file for each mirror.
The metadata mention seems very fuzzy. I am not saying you did something wrong, but at least that seems very fixable if you know what is going on.
The mirror issues seems evident and inherent to the solution. Although I would not be surprised if there was a smart fix for linking different location with the same content to avoid cache pollution.
I agree that configuring local repos or squid could be of some help, but I don't think that there is a real solution, but just a work-around.
Then tell us what a real solution looks like ? I am confident that writing a simple proxy server that does exactly what you want cannot be hard to write yourself.
Dag Wieers ha scritto:
The metadata mention seems very fuzzy. I am not saying you did something wrong, but at least that seems very fixable if you know what is going on.
I'm sorry, but I can't recall exactly what the error was... I remember yum complaining about metadata not matching checksums, and that there was a similar issue with other proxy services (maybe from mirrorservice.co.uk?)
The mirror issues seems evident and inherent to the solution. Although I would not be surprised if there was a smart fix for linking different location with the same content to avoid cache pollution.
If there is, then it isn't an "easy" or well documented one, since I wasn't able to do find it
I agree that configuring local repos or squid could be of some help, but I don't think that there is a real solution, but just a work-around.
Then tell us what a real solution looks like ? I am confident that writing a simple proxy server that does exactly what you want cannot be hard to write yourself.
I may have chosen the wrong words; I meant that there is nothing that does for yum and .rpm what apt-cacher does for apt and .deb, so the "real solution" meaning. And I wish I would able to write such a program, but my programming skills are really really poor.
Hey, it may sound strange to you guys out there, but here in Italy there are still people that don't learn English at school... so please, be indulgent on me ;)
Regards
Lorenzo
On 10/10/07, James A. Peltier jpeltier@cs.sfu.ca wrote:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
-- James A. Peltier Technical Director, RHCE SCIRF | GrUVi @ Simon Fraser University - Burnaby Campus Phone : 778-782-3610 Fax : 778-782-3045 Mobile : 778-840-6434 E-Mail : jpeltier@cs.sfu.ca Website : http://gruvi.cs.sfu.ca | http://scirf.cs.sfu.ca MSN : subatomic_spam@hotmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I dont know about apt-cacher, but if its something to do with apt-get you could you --download-only. Like apt-get upgrade --download-only
Tronn
Tronn Wærdahl wrote:
I dont know about apt-cacher, but if its something to do with apt-get you could you --download-only. Like apt-get upgrade --download-only
Tronn
apt-cacher allows you to point all clients to a centralized location, when one client performs, say and apt-get dist-upgrade, the apt-cache server caches all of the packages to it's local apt directory. It's very convienent and easy to setup, whereas the squid solution proposed by Dag is a fair bit more conviluted.
A possible solution would be to possibly extend create-repo to add functionality to mirror updates and packages for selected architectures. Perhaps add the proxy/caching functionality that is required to perform these operations?
James A. Peltier wrote:
Tronn Wærdahl wrote:
I dont know about apt-cacher, but if its something to do
with apt-get
you could you --download-only. Like apt-get upgrade --download-only
Tronn
apt-cacher allows you to point all clients to a centralized location, when one client performs, say and apt-get dist-upgrade, the apt-cache server caches all of the packages to it's local apt directory. It's very convienent and easy to setup, whereas the squid solution proposed by Dag is a fair bit more conviluted.
A possible solution would be to possibly extend create-repo to add functionality to mirror updates and packages for selected architectures. Perhaps add the proxy/caching functionality that is required to perform these operations?
I just have a central server mirror the updates repo with wget and have either apache or iis serve them up to clients.
Then in the yum repo files, comment out mirrorlist=, uncomment baseurl=, add the url to your update server first for baseurl= and set the failovermethod=priority.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
It looks like mrepo would be up to your task. http://dag.wieers.com/home-made/mrepo/ I'm surprised Dag didn't mention this. I haven't tried it myself.
Where I work we rsync down from a centos mirror, excluding the bits we don't need like isos-dvd/ and s390/. This keeps the total size within reason, but still need nearly 30G for 4.5 and 5.0 combined.
Mark D. Foster wrote:
It looks like mrepo would be up to your task. http://dag.wieers.com/home-made/mrepo/ I'm surprised Dag didn't mention this. I haven't tried it myself.
Where I work we rsync down from a centos mirror, excluding the bits we don't need like isos-dvd/ and s390/. This keeps the total size within reason, but still need nearly 30G for 4.5 and 5.0 combined.
I just found mrepo and it looks like it might do it OK. I'll give it a try and see if it works for me.
James A. Peltier ha scritto:
Mark D. Foster wrote:
It looks like mrepo would be up to your task. http://dag.wieers.com/home-made/mrepo/ I'm surprised Dag didn't mention this. I haven't tried it myself.
Maybe he is a bit shy? I think he deserves a big Thanks for all the work he does for us :) (and isn't the only one)
Where I work we rsync down from a centos mirror, excluding the bits we don't need like isos-dvd/ and s390/. This keeps the total size within reason, but still need nearly 30G for 4.5 and 5.0 combined.
I just found mrepo and it looks like it might do it OK. I'll give it a try and see if it works for me.
I can confirm that mrepo does an excellent job on making a local mirror of wathever repo you wish. The only downside is, as I told before, that yum may end downloading and storing files you'll never need; but it is really easy to configure the server and you just need to modify the repos on the various clients to point to your server as it was just another mirror.
There is only a thing that doesn't seem to work correctly (at least for me): if I do "yum list groups" it shows only a few groups. Did I miss some option in configuration file?
Regards
Lorenzo Quatrini
On Wed, 10 Oct 2007, James A. Peltier wrote:
Tronn Wærdahl wrote:
I dont know about apt-cacher, but if its something to do with apt-get you could you --download-only. Like apt-get upgrade --download-only
apt-cacher allows you to point all clients to a centralized location, when one client performs, say and apt-get dist-upgrade, the apt-cache server caches all of the packages to it's local apt directory. It's very convienent and easy to setup, whereas the squid solution proposed by Dag is a fair bit more conviluted.
I think it is fairly easy to write a proxy-server that forwards requests to repositories and caches it locally. A very simple implementation only needs to hold a list of known mirrors, only caches RPM information (not the metadata, although adding that could be done at some later stage).
The information cached should be indexed based on filename and md5 so that for storing you never end up caching the same file on disk (even when you list of known mirrors is wrong) and so that retrieval is fast.
And you get some bonus-points if you can log every attempt of downloading/caching a file that you already have stored and use that information to automatically update your list of known mirrors.
Implementing something like that would automatically cause a list of known mirrors to be build and improved for every download made.
Who wants to take this up ? :)
PS the difference between this scheme and a squid setup is basically that you do not pollute your cache/bandwidth for retrieval of mirrored files.
James A. Peltier wrote:
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
The Upstream have a Satellite program that provides local copies for their distribution/management network. Don't know if that is something that could be re-packaged?
On Thu, 11 Oct 2007, Morten Torstensen wrote:
James A. Peltier wrote:
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
The Upstream have a Satellite program that provides local copies for their distribution/management network. Don't know if that is something that could be re-packaged?
I think as part of CentOS we should have a wiki page that provides a few of the already discussed solutions. RHN Proxy or RHN Satellite are at this moment not Open Source and cannot be packaged.
Is anyone interested in writing this content and maintaining it in the CentOS wiki ? I am very interested to contribute to it myself, but have other responsibilities already.
On Thu, Oct 11, 2007 at 12:34:45PM +0200, Dag Wieers enlightened us:
James A. Peltier wrote:
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
The Upstream have a Satellite program that provides local copies for their distribution/management network. Don't know if that is something that could be re-packaged?
I think as part of CentOS we should have a wiki page that provides a few of the already discussed solutions. RHN Proxy or RHN Satellite are at this moment not Open Source and cannot be packaged.
Is anyone interested in writing this content and maintaining it in the CentOS wiki ? I am very interested to contribute to it myself, but have other responsibilities already.
This is discussed a little bit already on http://wiki.centos.org/HowTos/PackageManagement/YumOnRHEL but it could probably have some of that content split out to a separate page on managing repos, and let YumOnRHEL concentrate on just that - Yum on RHEL. Especially since that is a moot point on RHEL5.
Matt
James A. Peltier ha scritto:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
Hi, after a lot of googling and searching I think I've found something that looks really interesting:
http://freshmeat.net/projects/http-replicator
I'm testing it right now, it looks really promising.
Regards
Lorenzo Quatrini
yOn Thu, 11 Oct 2007, Lorenzo Quatrini wrote:
James A. Peltier ha scritto:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
after a lot of googling and searching I think I've found something that looks really interesting:
http://freshmeat.net/projects/http-replicator
I'm testing it right now, it looks really promising.
If it understands the concept of mirrors, than yes, it is exactly as I envisioned it :)
I just packaged it and you can provide it with a list of mirrors although it could have handled it transparantly instead of requiring a list upfront.
You can find a temporary package here:
http://dag.wieers.com/attic/RPMS/http-replicator-3.0-1.el5.rf.noarch.rpm
but it will be available soon on the mirrors. Even though it is tagged el5, it should work fine on EL4 or EL3.
Let me know if you have any problems.
Dag Wieers ha scritto:
yOn Thu, 11 Oct 2007, Lorenzo Quatrini wrote:
James A. Peltier ha scritto:
Hi All,
Is there something similar to apt-cacher available for CentOS? I would like to ensure that an application is only downloaded once to the network and not hundreds of times.
If there isn't an equivalent, would someone please point me in the direction of how I might accomplish this with CentOS?
after a lot of googling and searching I think I've found something that looks really interesting:
http://freshmeat.net/projects/http-replicator
I'm testing it right now, it looks really promising.
If it understands the concept of mirrors, than yes, it is exactly as I envisioned it :)
I just packaged it and you can provide it with a list of mirrors although it could have handled it transparantly instead of requiring a list upfront.
You can find a temporary package here:
http://dag.wieers.com/attic/RPMS/http-replicator-3.0-1.el5.rf.noarch.rpm
but it will be available soon on the mirrors. Even though it is tagged el5, it should work fine on EL4 or EL3.
Let me know if you have any problems.
Thanks Dag for the packaging; I'm on my way of testing, and I find that dealing with mirror lists isn't straight forward at all.
I will try to understand if is possible to modify the source so it only caches rpm files on a flat format and it compares only the file name and not the full path.
Also I have to understand if it is possible to add ftp support.
I'll let you know if I have news
Regards Lorenzo Quatrini
-
Clint Dilks -
Dag Wieers -
James A. Peltier -
Lorenzo -
Lorenzo Quatrini
-
Mark D. Foster -
Matt Hyclak -
Morten Torstensen -
Ross S. W. Walker -
Tronn Wærdahl