mirror April 2024

mirror@lists.centos.org

6 participants
7 discussions

Centos Stream mirror recreation
by Hostico 26 Apr '24

26 Apr '24

Hello, we had some hardware issues with out mirror server (https://mirrors.hostico.ro) which caused it to be offline for about 24hours and then took some time to resync everything. In the meantime our epel (https://mirrors.hostico.ro/fedora/epel/) and centos-stream (http://mirrors.hostico.ro/centos-stream/) mirrors were taken offline due to crawl errors. I updated the host from the mirror managed admin panel a couple of days ago but since the links didn't reappear online and the server was not crawled i tried deleting and readding the categories. To my surprise i was only able to readd the "Fedora EPEL" category . Is there a way to readd the Centos-stream one or should i just stop syncing it ? Thank you. -- ------ Regards, Sebastian Bobriuc https://hostico.ro

3 2

Mirror removal
by Eric K. Miller 07 Apr '24

07 Apr '24

Please remove mirror.genesishosting.com from the mirror list. Thank you! Eric

1 0

Re: [CentOS-mirror] Abusive repodata downloads
by Carsten Otto 03 Apr '24

03 Apr '24

On Wed, Apr 03, 2024 at 02:14:07PM -0400, Stephen John Smoogen wrote: > My expectation is that traffic is going to be higher in the next 6 months > as various systems are getting moved from CentOS 7 to some newer release. > This upgrade process will require a system to be first updated to the > latest '7' and many have had updates turned off for years.. then there will > be a download and update to EL8 or EL9 depending on what they are doing > which will also create a spike in traffic. Even better, that sounds like useful traffic and I have around 15 GBit/sec to spare for that :) -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/

1 0

Re: [CentOS-mirror] Abusive repodata downloads
by Carsten Otto 03 Apr '24

03 Apr '24

On Wed, Apr 03, 2024 at 12:54:10PM -0500, Dennis Gilmore wrote: > An exception to this would be people building containers. > Theytypically remove the metadata at the end of each build. So do not > have it cached. However, the described behaviour would seem to be > more widespread than some people building containers, which may > indicate some other reason that a group of people are repeatedly > grabbing the same data. It would be interesting to know if they are > only getting repodata or also downloading rpms and other things. Interesting thought! I just grepped for a few of the "abusive" hosts. These hosts are downloading packages, including some "basic" ones like perl, curl, python setuptools. I'll relax the fail2ban rules a bit and see how things evolve over time. Maybe everyone needs to rebuild everything once now, which obviously causes a bit of traffic. -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/

1 0

Re: [CentOS-mirror] Abusive repodata downloads
by Carsten Otto 03 Apr '24

03 Apr '24

On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote: > In general clients should pull the repomd.xml file and check if it's > changed, if not, use their cached versions. Does this also apply to this file? /fedora-epel/7/x86_64/repodata/f25bccaa54b08e731162aec431dd82887a5ed1abce8eb6f892ad41454f07e1f3-primary.sqlite.bz2 I have five hosts which each downloaded 4 GByte of this file in the span of 24 hours. The delay between two downloads varies, from a few seconds to a few minutes. -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/

3 6

Re: [CentOS-mirror] Abusive repodata downloads
by Carsten Otto 03 Apr '24

03 Apr '24

Hi Kevin, On Tue, Apr 02, 2024 at 05:08:33PM -0700, Kevin Fenzi wrote: > How are you determining this? From IP address? Yes. > Those could be a large number of different machines behind a nat proxy > no? Possibly. > Recent versions of fedora do not even download the filelists unless a > user requests/looks up something that needs it. For normal install or > updates, it shouldn't be pulled. Can you tell what OS/versions this is > happening with? Great! No, I just see basic file download information. Blocking those who do it much too often shouldn't be a concern then. > In general clients should pull the repomd.xml file and check if it's > changed, if not, use their cached versions. That's not what I see for quite a lot of peers. > This should normally already be the case, so I wonder what is happening > here, could be a bug in some version/os or otherwise something > unexpected. That's my guess too. Bye Carsten -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/

2 1

Abusive repodata downloads
by Carsten Otto 02 Apr '24

02 Apr '24

Hi, I'm one of the admins of ftp.halifax.rwth-aachen.de, which offers fedora and fedora-epel among other distributions and projects. I've recently noticed that many Fedora users (including EPEL, CentOS, and BlackArch) frequently re-download "repodata" files that haven't been updated since the previous request. As some of those files are rather large ("filelists") and others are hit extremely often, I added fail2ban rules to deny users access to our service for some time. Please adjust your software so that file mirrors like ours are not (ab)used like this. Files that haven't been changed usually shouldn't be downloaded. Thanks Carsten -- Dr. Carsten Otto http://verify.rwth-aachen.de/otto/

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

mirror April 2024