Something I posted on the blog, and I'd appreciate comments from this list about:
--------------------------------------------------
Has anyone given though to doing mirror updates via a 'push' rather than a 'pull' mechanism ?
Basically, when new packages are available the machines that get updates would do a rsync to the machines down the order, pushing the updates out. It would allow us to get more updates, out faster and also not need to waste cpu and i/o on repeated rsyncs that dont need to be run.
Yes, we could/would/should still leave in place some mechanism for people to setup new mirrors and also to do pull based rsync's as and when they need.
Just something to think about at the moment, and comments would be very welcome.
- KB
Karanbir Singh wrote:
Something I posted on the blog, and I'd appreciate comments from this list about:
Has anyone given though to doing mirror updates via a 'push' rather than a 'pull' mechanism ?
Basically, when new packages are available the machines that get updates would do a rsync to the machines down the order, pushing the updates out. It would allow us to get more updates, out faster and also not need to waste cpu and i/o on repeated rsyncs that dont need to be run.
Yes, we could/would/should still leave in place some mechanism for people to setup new mirrors and also to do pull based rsync's as and when they need.
Just something to think about at the moment, and comments would be very welcome.
The idea itself is nice, but some mirrors need to do the "pull" at specific moments in time. For instance on the mirrors I take care of we have interceded the downloads from several well known repos with high-traffic, such as debian / ubuntu / fedora / centos / opensuse. Overlapping downloads would max out our bandwidth and trigger the ISPs bw limiters. We've done that once and I do not want to repeat the experience... one of our hosters lowered the bw after that and kept the lower limit in place for two weeks even after we lowered the bw consumption. And this hurt, pretty bad actually.
manuel
On 05/14/2009 11:17 AM, Manuel Wolfshant wrote:
Has anyone given though to doing mirror updates via a 'push' rather than a 'pull' mechanism ?
The idea itself is nice, but some mirrors need to do the "pull" atspecific moments in time.
Absolutely, I dont see the process as its now, going away in a hurry. However, if we did setup a mechanism for mirrors to sign up for an update push, when there was an update due, and then ask those mirrors to reduce their regular pull-mirroring down to perhaps once every 96 hrs, would many people opt in ?
It address's the issue of just-in-time-pushing and saving excessive b/w and cpu loads caused on the upstream mirros by regular rsyncs. Plenty of people hit the main msync targets 4 - 6 sometimes 8 times a day. The load on this caused by 200+ mirrors is non-trivial. Also, there is an issue that sometimes critical updates dont make it out for hours, since we must wait for mirrors to pull and finish. Push, should, in theory atleast address some of these issues.
Then there is -c issue. Would be nice if some portions of the tree's were run with the '-c' option set, its not something we can even look at with the pull model.
- KB
-
Karanbir Singh -
Manuel Wolfshant