Hi,
I today have a problem as well with metadata being newer than what is available on disk. I wonder whether we should require from mirrors to use rsync with --delay-updates as problems with repository metadata was exactly why we got this feature added to rsync a couple of years ago.
We could enforce it from the CentOS mirrors by checking the rsync version and/or options provided to the server. Is something like this feasible ?
---- [root@moria ~]# apt-get update; apt-get upgrade Get:1 http://centos.mirrors.skynet.be centos/5/os/x86_64 repomd.xml [1140B] Get:2 http://centos.mirrors.skynet.be centos/5/updates/x86_64 repomd.xml [951B] Get:3 http://centos.mirrors.skynet.be centos/5/extras/x86_64 repomd.xml [951B] Get:4 http://apt.sw.be redhat/el5/en/x86_64/rpmforge repomd.xml [1095B] Get:5 http://rpm.guifications.org centos/5/x86_64/ repomd.xml [951B] Get:6 http://rpm.pidgin.im centos/5/x86_64/ repomd.xml [951B] Fetched 6039B in 0s (8752B/s) Hit http://centos.mirrors.skynet.be centos/5/os/x86_64/ primary.xml Hit http://centos.mirrors.skynet.be centos/5/os/x86_64/ filelists.xml Get:1 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ primary.xml [110kB] Hit http://apt.sw.be redhat/el5/en/x86_64/rpmforge/ primary.xml Hit http://apt.sw.be redhat/el5/en/x86_64/rpmforge/ filelists.xml Get:2 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ filelists.xml [674kB] Hit http://rpm.guifications.org centos/5/x86_64/ primary.xml Hit http://rpm.pidgin.im centos/5/x86_64/ primary.xml Hit http://rpm.guifications.org centos/5/x86_64/ filelists.xml Hit http://rpm.pidgin.im centos/5/x86_64/ filelists.xml Hit http://centos.mirrors.skynet.be centos/5/extras/x86_64/ primary.xml Hit http://centos.mirrors.skynet.be centos/5/extras/x86_64/ filelists.xml Fetched 784kB in 0s (936kB/s) Reading Package Lists... Done Building Dependency Tree... Done Reading Package Lists... Done Building Dependency Tree... Done The following packages will be upgraded NetworkManager (0.7.0-3.el5 => 0.7.0-4.el5_3) NetworkManager-glib (0.7.0-3.el5 => 0.7.0-4.el5_3) NetworkManager-glib.32bit (0.7.0-3.el5 => 0.7.0-4.el5_3) NetworkManager-gnome (0.7.0-3.el5 => 0.7.0-4.el5_3) NetworkManager.32bit (0.7.0-3.el5 => 0.7.0-4.el5_3) firefox (3.0.6-1.el5.centos => 3.0.7-1.el5.centos) firefox.32bit (3.0.6-1.el5.centos => 3.0.7-1.el5.centos) ntp (4.2.2p1-9.el5.centos => 4.2.2p1-9.el5.centos.1) xen (3.0.3-80.el5 => 3.0.3-80.el5_3.2) xen-libs (3.0.3-80.el5 => 3.0.3-80.el5_3.2) xen-libs.32bit (3.0.3-80.el5 => 3.0.3-80.el5_3.2) xulrunner (1.9.0.6-1.el5 => 1.9.0.7-3.el5) xulrunner.32bit (1.9.0.6-1.el5 => 1.9.0.7-3.el5) 13 upgraded, 0 newly installed, 0 removed and 0 not upgraded. Need to get 52.5MB of archives. After unpacking 3968kB disk space will be freed. Do you want to continue? [Y/n] Get:1 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ NetworkManager-gnome 1:0.7.0-4.el5_3 [344kB] Get:2 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ NetworkManager-glib.32bit 1:0.7.0-4.el5_3 [82.1kB] Get:3 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ NetworkManager.32bit 1:0.7.0-4.el5_3 [1093kB] Get:4 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ NetworkManager 1:0.7.0-4.el5_3 [1096kB] Get:5 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ NetworkManager-glib 1:0.7.0-4.el5_3 [83.4kB] Err http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xulrunner 1.9.0.7-3.el5 404 Not Found Err http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xulrunner.32bit 1.9.0.7-3.el5 404 Not Found Get:6 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ firefox 3.0.7-1.el5.centos [12.5MB] Get:7 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ firefox.32bit 3.0.7-1.el5.centos [12.5MB] Err http://centos.mirrors.skynet.be centos/5/updates/x86_64/ ntp 4.2.2p1-9.el5.centos.1 404 Not Found Get:8 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xen-libs 3.0.3-80.el5_3.2 [145kB] Get:9 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xen 3.0.3-80.el5_3.2 [1845kB] Get:10 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xen-libs.32bit 3.0.3-80.el5_3.2 [148kB] Fetched 29.8MB in 24s (1209kB/s) Failed to fetch http://centos.mirrors.skynet.be/pub//centos/5/updates/x86_64/RPMS/xulrunner-... 404 Not Found Failed to fetch http://centos.mirrors.skynet.be/pub//centos/5/updates/x86_64/RPMS/xulrunner-... 404Not Found Failed to fetch http://centos.mirrors.skynet.be/pub//centos/5/updates/x86_64/RPMS/ntp-4.2.2p... 404 Not Found E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? ----
On Thu, Apr 09, 2009 at 11:53:03PM +0200, Dag Wieers wrote:
Hi,
I today have a problem as well with metadata being newer than what is available on disk. I wonder whether we should require from mirrors to use rsync with --delay-updates as problems with repository metadata was exactly why we got this feature added to rsync a couple of years ago.
we could gently ask them, require might be a little strong (imho)
We could enforce it from the CentOS mirrors by checking the rsync version and/or options provided to the server. Is something like this feasible ?
one would need to have all the mirrors with >= c5, unless you also provide the needed version on rpmforge for c3 and c4.
[root@moria ~]# apt-get update; apt-get upgrade
...
Get:5 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ NetworkManager-glib 1:0.7.0-4.el5_3 [83.4kB] Err http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xulrunner 1.9.0.7-3.el5 404 Not Found Err http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xulrunner.32bit 1.9.0.7-3.el5 404 Not Found Get:6 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ firefox 3.0.7-1.el5.centos [12.5MB] Get:7 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ firefox.32bit 3.0.7-1.el5.centos [12.5MB] Err http://centos.mirrors.skynet.be centos/5/updates/x86_64/ ntp 4.2.2p1-9.el5.centos.1 404 Not Found Get:8 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xen-libs 3.0.3-80.el5_3.2 [145kB] Get:9 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xen 3.0.3-80.el5_3.2 [1845kB] Get:10 http://centos.mirrors.skynet.be centos/5/updates/x86_64/ xen-libs.32bit 3.0.3-80.el5_3.2 [148kB] Fetched 29.8MB in 24s (1209kB/s) Failed to fetch http://centos.mirrors.skynet.be/pub//centos/5/updates/x86_64/RPMS/xulrunner-... 404 Not Found Failed to fetch http://centos.mirrors.skynet.be/pub//centos/5/updates/x86_64/RPMS/xulrunner-... 404Not Found Failed to fetch http://centos.mirrors.skynet.be/pub//centos/5/updates/x86_64/RPMS/ntp-4.2.2p... 404 Not Found E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
shouldn't apt-get do as yum and try another mirror?
mirmon as currently setup has noticed that skynet.be is not uptodate :( but one can not parse all the centos tree..
Cheers,
Tru
mirmon as currently setup has noticed that skynet.be is not uptodate :( but one can not parse all the centos tree..
Hello,
For the mirrors who offer rsync access, you can also use "rsync -n" to compare your own mirror against different "upstream" ones.
I've found several quirks in mirror servers this way who e.g. have a certain policy of excluding certain files or just had their setup screwed.
greetings,
Florian La Roche
Florian La Roche wrote:
For the mirrors who offer rsync access, you can also use "rsync -n" to compare your own mirror against different "upstream" ones.
Thats a good idea, we should plumb that into the checking scripts.
Also, internally all the centos machines run 2 rounds of rsync, the first one runnging with an --exclude='repodata/' and without the --delete. The second one has --delete, but no excludes, so that way we try and make sure the time that a machine might not be in a 'good' state is reduced quite a bit.
On Thursday 09 April 2009 17:53:03 Dag Wieers wrote:
Hi,
I today have a problem as well with metadata being newer than what is available on disk. I wonder whether we should require from mirrors to use rsync with --delay-updates as problems with repository metadata was exactly why we got this feature added to rsync a couple of years ago.
We could enforce it from the CentOS mirrors by checking the rsync version and/or options provided to the server. Is something like this feasible ?
See https://lists.ubuntu.com/archives/ubuntu-mirrors-announce/2006-August/000002... for how 'the other half lives.' Different files; still good advice.
Lamar Owen wrote:
See https://lists.ubuntu.com/archives/ubuntu-mirrors-announce/2006-August/000002... for how 'the other half lives.' Different files; still good advice.
Thats about how we have been running our mirrors for years now.
-
Dag Wieers -
Florian La Roche -
Karanbir Singh -
Lamar Owen -
Tru Huynh