New mirror

List overview All Threads
Download

newer

older

Update of Mirror Site IP for...

Add additional IPs / rsync white...

Ryan Nix

31 Aug 2015 31 Aug '15

7:27 p.m.

HTTP: http://mirror.sesp.northwestern.edu/centos/

Sync schedule: Every four hours Bandwidth: Internet2 Location: Evanston, Illinois Sponsor: Northwestern University School of Education and Social Policy Sponsor URL: www.sesp.northwestern.edu IP to authorize: 129.105.244.4 Email contact: r-nix@northwestern.edu

Attachments:

attachment.html (text/html — 927 bytes)

Show replies by date

Anssi Johansson

31 Aug 31 Aug

7:55 p.m.

31.8.2015, 22.27, Ryan Nix kirjoitti:

...

HTTP: http://mirror.sesp.northwestern.edu/centos/

Hi, I seem to be unable to connect to this URL. Perhaps a firewall problem?

$ telnet mirror.sesp.northwestern.edu 80 Trying 129.105.244.5... telnet: connect to address 129.105.244.5: No route to host

Ryan Nix

9:37 p.m.

My apologies, the firewall was indeed blocking. Please try it now.

On Mon, Aug 31, 2015 at 2:55 PM, Anssi Johansson centos@miuku.net wrote:

...

31.8.2015, 22.27, Ryan Nix kirjoitti:

...
HTTP: http://mirror.sesp.northwestern.edu/centos/

Hi, I seem to be unable to connect to this URL. Perhaps a firewall problem?

$ telnet mirror.sesp.northwestern.edu 80 Trying 129.105.244.5... telnet: connect to address 129.105.244.5: No route to host

CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

Anssi Johansson

11:43 p.m.

1.9.2015, 0.37, Ryan Nix kirjoitti:

...

My apologies, the firewall was indeed blocking. Please try it now.

Yes, it is now reachable. However, I get the impression that there's some rate limiting involved. Here's what I get when I tried to run my (unofficial) manual mirror checking script:

If I retry the test, the failing URLs will be different for each run. If I'm lucky, all the downloads will succeed. Most often at least one file will fail to download.

This rate limiting may affect your mirror's visibility to the official mirror checker (causing your mirror to be marked as unreachable), and it may also cause problems for people who download updates from your mirror or do a netinstall with your mirror as the source.

I would suggest increasing the maximum connection rates, if you want to keep rate limiting.

Ryan Nix

1 Sep 1 Sep

1:39 a.m.

There shouldn't be any bandwidth restrictions. I wonder if there was some kind of misconfiguration in the mount point where the data are stored. I made some adjustments thenI tried downloading from home and was getting 3 MBs per second.

On Mon, Aug 31, 2015 at 6:43 PM, Anssi Johansson centos@miuku.net wrote:

...

1.9.2015, 0.37, Ryan Nix kirjoitti:

...
My apologies, the firewall was indeed blocking. Please try it now.

Yes, it is now reachable. However, I get the impression that there's some rate limiting involved. Here's what I get when I tried to run my (unofficial) manual mirror checking script:

$ perl checkcentosmirror_v4.pl http://mirror.sesp.northwestern.edu/centos/ Checking http://mirror.sesp.northwestern.edu/centos/ IPV4: 129.105.244.5

http://mirror.sesp.northwestern.edu/centos/5/updates/i386/repodata/repomd.xm...

http://mirror.sesp.northwestern.edu/centos/5/fasttrack/i386/repodata/repomd.... http://mirror.sesp.northwestern.edu/centos/5/cr/i386/repodata/repomd.xml repomd.xml failed at http://mirror.sesp.northwestern.edu/centos/5/cr/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/5/centosplus/i386/repodata/repomd... http://mirror.sesp.northwestern.edu/centos/5/os/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/5/extras/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/5/contrib/i386/repodata/repomd.xm...

http://mirror.sesp.northwestern.edu/centos/5/addons/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/6/updates/i386/repodata/repomd.xm...

http://mirror.sesp.northwestern.edu/centos/6/fasttrack/i386/repodata/repomd.... repomd.xml failed at http://mirror.sesp.northwestern.edu/centos/6/fasttrack/i386/repodata/repomd.... http://mirror.sesp.northwestern.edu/centos/6/cr/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/6/centosplus/i386/repodata/repomd... http://mirror.sesp.northwestern.edu/centos/6/os/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/6/extras/i386/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/6/contrib/i386/repodata/repomd.xm...

http://mirror.sesp.northwestern.edu/centos/5/updates/x86_64/repodata/repomd....

http://mirror.sesp.northwestern.edu/centos/5/fasttrack/x86_64/repodata/repom... http://mirror.sesp.northwestern.edu/centos/5/cr/x86_64/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/5/centosplus/x86_64/repodata/repo... http://mirror.sesp.northwestern.edu/centos/5/os/x86_64/repodata/repomd.xml repomd.xml failed at http://mirror.sesp.northwestern.edu/centos/5/os/x86_64/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/5/extras/x86_64/repodata/repomd.x...

http://mirror.sesp.northwestern.edu/centos/5/contrib/x86_64/repodata/repomd....

http://mirror.sesp.northwestern.edu/centos/5/addons/x86_64/repodata/repomd.x...

http://mirror.sesp.northwestern.edu/centos/6/updates/x86_64/repodata/repomd....

http://mirror.sesp.northwestern.edu/centos/6/fasttrack/x86_64/repodata/repom... http://mirror.sesp.northwestern.edu/centos/6/cr/x86_64/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/6/centosplus/x86_64/repodata/repo... repomd.xml failed at http://mirror.sesp.northwestern.edu/centos/6/centosplus/x86_64/repodata/repo... http://mirror.sesp.northwestern.edu/centos/6/os/x86_64/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/6/extras/x86_64/repodata/repomd.x...

http://mirror.sesp.northwestern.edu/centos/6/contrib/x86_64/repodata/repomd....

http://mirror.sesp.northwestern.edu/centos/7/centosplus/x86_64/repodata/repo... http://mirror.sesp.northwestern.edu/centos/7/cr/x86_64/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/7/extras/x86_64/repodata/repomd.x...

http://mirror.sesp.northwestern.edu/centos/7/fasttrack/x86_64/repodata/repom... repomd.xml failed at http://mirror.sesp.northwestern.edu/centos/7/fasttrack/x86_64/repodata/repom... http://mirror.sesp.northwestern.edu/centos/7/os/x86_64/repodata/repomd.xml

http://mirror.sesp.northwestern.edu/centos/7/updates/x86_64/repodata/repomd.... http://mirror.sesp.northwestern.edu/centos/ : FAIL

If I retry the test, the failing URLs will be different for each run. If I'm lucky, all the downloads will succeed. Most often at least one file will fail to download.

This rate limiting may affect your mirror's visibility to the official mirror checker (causing your mirror to be marked as unreachable), and it may also cause problems for people who download updates from your mirror or do a netinstall with your mirror as the source.

I would suggest increasing the maximum connection rates, if you want to keep rate limiting.

CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

Anssi Johansson

1:59 a.m.

1.9.2015, 4.39, Ryan Nix kirjoitti:

...

There shouldn't be any bandwidth restrictions. I wonder if there was some kind of misconfiguration in the mount point where the data are stored. I made some adjustments thenI tried downloading from home and was getting 3 MBs per second.

Oh, I'm sure the bandwidth isn't the problem. My concern was the HTTP requests per second rate, not bytes per second rate.

The way to test this is to run something like this: for i in {1..50}; do curl http://mirror.sesp.northwestern.edu/centos/TIME; done

Oddly this seems to work fine from one U.S. host I have access to, but doing that from .fi, .fr or .uk hosts seems to fail every now and then.

Ryan Nix

11:07 a.m.

Very strange. Our campus does have an intrusion detection system, however, I don't believe it's filtering http traffic.

I ran your time test. How do I interpret these numbers?

1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541 1441026541

On Mon, Aug 31, 2015 at 8:59 PM, Anssi Johansson centos@miuku.net wrote:

...

1.9.2015, 4.39, Ryan Nix kirjoitti:

...
There shouldn't be any bandwidth restrictions. I wonder if there was some kind of misconfiguration in the mount point where the data are stored. I made some adjustments thenI tried downloading from home and was getting 3 MBs per second.

Oh, I'm sure the bandwidth isn't the problem. My concern was the HTTP requests per second rate, not bytes per second rate.

The way to test this is to run something like this: for i in {1..50}; do curl http://mirror.sesp.northwestern.edu/centos/TIME; done

Oddly this seems to work fine from one U.S. host I have access to, but doing that from .fi, .fr or .uk hosts seems to fail every now and then.

CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

Fabian Arrotin

11:23 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 01/09/15 13:07, Ryan Nix wrote:

...

Very strange. Our campus does have an intrusion detection system, however, I don't believe it's filtering http traffic.

I ran your time test. How do I interpret these numbers?

1441026541

Compare with `date +%s' (aka seconds since 1970-01-01 00:00:00 UTC) ;-)

- -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlXliswACgkQnVkHo1a+xU4FegCdEIavXtqBgUoKs/ywieiZJpJk tMwAnjjza0AzpBnQdaaQOJInQ/zsotcs =tIFG -----END PGP SIGNATURE-----

Anssi Johansson

12:24 p.m.

1.9.2015, 14.07, Ryan Nix kirjoitti:

...

Very strange. Our campus does have an intrusion detection system, however, I don't believe it's filtering http traffic.

I ran your time test. How do I interpret these numbers?

1441026541 1441026541 1441026541 ... 1441026541 1441026541 1441026541

The TIME file is just a timestamp. If you get 50 times that timestamp without any delays in between, then the test passed.

For me, I got that line 33 times, then the output stopped for a few minutes before a "curl: (56) Failure when receiving data from the peer".

In any case, looks like it's behaving better now. If you get complaints from someone else using your mirror, please check your intrusion detection system settings and/or logs.

Ryan Nix

5:29 p.m.

I will definitely keep an eye on things. I have the system monitored through New Relic.

On another note, I'm setting up a new cron job and I can't get sync'd with the main mirror. Any ideas?

rsync -aqzH --delete us-msync.centos.org::CentOS /var/www/html/centos >> /root/cronsync.txt @ERROR: Unknown module 'CentOS'

On Tue, Sep 1, 2015 at 7:24 AM, Anssi Johansson centos@miuku.net wrote:

...

1.9.2015, 14.07, Ryan Nix kirjoitti:

...
Very strange. Our campus does have an intrusion detection system, however, I don't believe it's filtering http traffic.

I ran your time test. How do I interpret these numbers?

1441026541 1441026541 1441026541 ... 1441026541 1441026541 1441026541

The TIME file is just a timestamp. If you get 50 times that timestamp without any delays in between, then the test passed.

For me, I got that line 33 times, then the output stopped for a few minutes before a "curl: (56) Failure when receiving data from the peer".

In any case, looks like it's behaving better now. If you get complaints from someone else using your mirror, please check your intrusion detection system settings and/or logs.

CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

Fabian Arrotin

5:33 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 01/09/15 19:29, Ryan Nix wrote:

...

I will definitely keep an eye on things. I have the system monitored through New Relic.

On another note, I'm setting up a new cron job and I can't get sync'd with the main mirror. Any ideas?

rsync -aqzH --delete us-msync.centos.org::CentOS /var/www/html/centos >> /root/cronsync.txt @ERROR: Unknown module 'CentOS'

That means that your outgoing IP isn't the one you mentioned in your previous mail : 129.105.244.4 is the IP currently allowed. Can you verify on your side ? Are there multiple IPs on that node ? behind a Firewall/GW doing NAT, etc ? `curl -4 http://icanhazip.com%60 from your server can give pointers, but the NAT GW can also do SNAT using multiple IPs from a pool and also depending on destination ports.

Cheers,

- -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlXl4XUACgkQnVkHo1a+xU7+aQCePl1+byVWmJ9j3Kmn4TjMZZV6 3JwAnjEPLKot33XJoerKAZDfvkr9ZL6R =w62K -----END PGP SIGNATURE-----

Ryan Nix

5:40 p.m.

My apologies, the IP is actually 129.105.244.5

dhcp-10-105-12-35:~ rtn361$ dig mirror.sesp.northwestern.edu

; <<>> DiG 9.8.3-P1 <<>> mirror.sesp.northwestern.edu ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61490 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;mirror.sesp.northwestern.edu. IN A

;; ANSWER SECTION: mirror.sesp.northwestern.edu. 14400 IN A 129.105.244.5

;; Query time: 6 msec ;; SERVER: 129.105.49.1#53(129.105.49.1) ;; WHEN: Tue Sep 1 12:39:06 2015 ;; MSG SIZE rcvd: 62

On Tue, Sep 1, 2015 at 12:33 PM, Fabian Arrotin arrfab@centos.org wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 01/09/15 19:29, Ryan Nix wrote:

...
I will definitely keep an eye on things. I have the system monitored through New Relic.

On another note, I'm setting up a new cron job and I can't get sync'd with the main mirror. Any ideas?

rsync -aqzH --delete us-msync.centos.org::CentOS /var/www/html/centos >> /root/cronsync.txt @ERROR: Unknown module 'CentOS'

That means that your outgoing IP isn't the one you mentioned in your previous mail : 129.105.244.4 is the IP currently allowed. Can you verify on your side ? Are there multiple IPs on that node ? behind a Firewall/GW doing NAT, etc ? `curl -4 http://icanhazip.com%60 from your server can give pointers, but the NAT GW can also do SNAT using multiple IPs from a pool and also depending on destination ports.

Cheers,

Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlXl4XUACgkQnVkHo1a+xU7+aQCePl1+byVWmJ9j3Kmn4TjMZZV6 3JwAnjEPLKot33XJoerKAZDfvkr9ZL6R =w62K -----END PGP SIGNATURE----- _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

Fabian Arrotin

6:49 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 01/09/15 19:40, Ryan Nix wrote:

...

My apologies, the IP is actually 129.105.244.5

Updated and pushed, so will be modified on all msync nodes in the next 30 minutes (max).

Cheers, - -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlXl8z0ACgkQnVkHo1a+xU5FbwCdGvqZl2Po2XiwF60pdUUHyPqf b80AoJFO//CzTgN/a9XSOF1qd28ddy9X =OXLR -----END PGP SIGNATURE-----

Fabian Arrotin

6:18 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 31/08/15 21:27, Ryan Nix wrote:

...

HTTP: http://mirror.sesp.northwestern.edu/centos/

Sync schedule: Every four hours Bandwidth: Internet2 Location: Evanston, Illinois Sponsor: Northwestern University School of Education and Social Policy Sponsor URL: www.sesp.northwestern.edu http://www.sesp.northwestern.edu IP to authorize: 129.105.244.4 Email contact: r-nix@northwestern.edu mailto:r-nix@northwestern.edu

Thanks : it has been added to the mirrors DB and will normally be listed as a public mirror (and on http://mirror-status.centos.org / http://centos.org/download/mirrors/ ) in the following minutes/hours. Your IP has been whitelisted (check that it's working in the following hour)

As I see that the host is on Internet2, I guess you have very good connectivity there, but probably not to outside ? What would be the available bandwidth for the rest of the world ? I guess that can explain also the issues Anssi saw when trying to verify your mirror from .fi and .uk.

Kind Regards,

- -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlXlQyIACgkQnVkHo1a+xU4d4wCfbXGi3nEBsZ3sEmzHYARp1FXe 1dUAmgJSWibSsK2+MPeV0Nv9VCXCOHUY =6bc1 -----END PGP SIGNATURE-----

Ryan Nix

11:04 a.m.

I'm not capping the speed in anyway so I'm not sure I can give an answer. The server is on a gigabit port too.

On Tue, Sep 1, 2015 at 1:18 AM, Fabian Arrotin arrfab@centos.org wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 31/08/15 21:27, Ryan Nix wrote:

...
HTTP: http://mirror.sesp.northwestern.edu/centos/

Sync schedule: Every four hours Bandwidth: Internet2 Location: Evanston, Illinois Sponsor: Northwestern University School of Education and Social Policy Sponsor URL: www.sesp.northwestern.edu http://www.sesp.northwestern.edu IP to authorize: 129.105.244.4 Email contact: r-nix@northwestern.edu mailto:r-nix@northwestern.edu

Thanks : it has been added to the mirrors DB and will normally be listed as a public mirror (and on http://mirror-status.centos.org / http://centos.org/download/mirrors/ ) in the following minutes/hours. Your IP has been whitelisted (check that it's working in the following hour)

As I see that the host is on Internet2, I guess you have very good connectivity there, but probably not to outside ? What would be the available bandwidth for the rest of the world ? I guess that can explain also the issues Anssi saw when trying to verify your mirror from .fi and .uk.

Kind Regards,

Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlXlQyIACgkQnVkHo1a+xU4d4wCfbXGi3nEBsZ3sEmzHYARp1FXe 1dUAmgJSWibSsK2+MPeV0Nv9VCXCOHUY =6bc1 -----END PGP SIGNATURE----- _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

3411

Age (days ago)

3412

Last active (days ago)

mirror@lists.centos.org

14 comments

3 participants

tags (0)

participants (3)

Anssi Johansson
Fabian Arrotin
Ryan Nix