On 04/20/2016 05:41 PM, Judit Flo Gaya wrote:
Hi,
On 4/20/16 14:23, Anssi Johansson wrote:
20.4.2016, 20.43, Judit Flo Gaya kirjoitti:
Hi,
For quite some time now, we've not been receiving updates on our public mirror (http://mirrors.seas.harvard.edu/centos). It seems like we no longer have access to the information.
receiving file list ... done ./ TIME dir_sizes filelist.gz timestamp.txt io timeout after 603 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(200) [receiver=3.0.6] rsync: connection unexpectedly closed (4153066 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]
Could you please verify that we are allowed to rsync, the ip is nat one: 140.247.173.13.
This kind of looks like the problem could be at the msync.centos.org side. If your IP address wasn't allowed, you would not be able to sync even those timestamp files.
When you sync, you should be shown the server's name, such as centoso6: "msync.CentOS.org rsync service (centoso6)". Please tell us the server name that gave you the error message you mentioned.
I hope you are syncing fro us-msync.centos.org instead of some specific server. msync.c.o, eu-msync.c.o and us-msync.c.o point to a number of msync servers, and that list is updated periodically to reflect any new or removed servers. _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror
I thought that too, but... I am syncing to the us-msync, yes. Please find attached the logs from today, the names of the servers where: msync.CentOS.org rsync service (centosq4) msync.CentOS.org rsync service (centosi6) msync.CentOS.org rsync service (centosv)
This is cron # CentOS gets synced 4 times/day. Debuginfo hits a different server, so # it's okay to do it simultaneously. 30 */6 * * * mirror /srv/mirror-scripts/bin/mirror centos > /dev/null 30 */6 * * * mirror /srv/mirror-scripts/bin/mirror centos-debuginfo > /dev/null
Producing: /usr/bin/rsync -vrltH --partial --delay-updates --delete-after --delete-excluded --stats --timeout=600 --exclude=.snapshot/ rsync://us-msync.centos.org/CentOS /misc/mirrors-rw/centos
Thanks for your help, j
If it's helpful, here are some more complaining servers, same error always: msync.CentOS.org rsync service (centosc5) msync.CentOS.org rsync service (centosm7) msync.CentOS.org rsync service (centosq4) msync.CentOS.org rsync service (centosy6) msync.CentOS.org rsync service (centosl7) msync.CentOS.org rsync service (centosd7)
On 21/04/16 18:01, Judit Flo Gaya wrote:
On 04/20/2016 05:41 PM, Judit Flo Gaya wrote:
Hi,
On 4/20/16 14:23, Anssi Johansson wrote:
20.4.2016, 20.43, Judit Flo Gaya kirjoitti:
Hi,
For quite some time now, we've not been receiving updates on our public mirror (http://mirrors.seas.harvard.edu/centos). It seems like we no longer have access to the information.
receiving file list ... done ./ TIME dir_sizes filelist.gz timestamp.txt io timeout after 603 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(200) [receiver=3.0.6] rsync: connection unexpectedly closed (4153066 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6]
Could you please verify that we are allowed to rsync, the ip is nat one: 140.247.173.13.
This kind of looks like the problem could be at the msync.centos.org side. If your IP address wasn't allowed, you would not be able to sync even those timestamp files.
When you sync, you should be shown the server's name, such as centoso6: "msync.CentOS.org rsync service (centoso6)". Please tell us the server name that gave you the error message you mentioned.
I hope you are syncing fro us-msync.centos.org instead of some specific server. msync.c.o, eu-msync.c.o and us-msync.c.o point to a number of msync servers, and that list is updated periodically to reflect any new or removed servers. _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror
I thought that too, but... I am syncing to the us-msync, yes. Please find attached the logs from today, the names of the servers where: msync.CentOS.org rsync service (centosq4) msync.CentOS.org rsync service (centosi6) msync.CentOS.org rsync service (centosv)
This is cron # CentOS gets synced 4 times/day. Debuginfo hits a different server, so # it's okay to do it simultaneously. 30 */6 * * * mirror /srv/mirror-scripts/bin/mirror centos > /dev/null 30 */6 * * * mirror /srv/mirror-scripts/bin/mirror centos-debuginfo > /dev/null
Producing: /usr/bin/rsync -vrltH --partial --delay-updates --delete-after --delete-excluded --stats --timeout=600 --exclude=.snapshot/ rsync://us-msync.centos.org/CentOS /misc/mirrors-rw/centos
Thanks for your help, j
If it's helpful, here are some more complaining servers, same error always: msync.CentOS.org rsync service (centosc5) msync.CentOS.org rsync service (centosm7) msync.CentOS.org rsync service (centosq4) msync.CentOS.org rsync service (centosy6) msync.CentOS.org rsync service (centosl7) msync.CentOS.org rsync service (centosd7)
Well, can you investigate also at your side if something it saturating the IO subsystem ? Not sure if the " io timeout after 603 seconds -- exiting" is coming from your node or from one of our nodes (as that would seem strange if that was the case for all of them, as then all external mirrors will be invalidated too because they'd not be able to sync from us either) One interesting thing to investigate at your firewall/gw is the outgoing IP, as I see this ; connect from pool-computing-servers.seas.harvard.edu (140.247.173.57) (while you mention 140.247.173.13) so a network setting at the firewall level stopping network connections that seems idle (during the "building file list step", last thing I see in our logs)
21.4.2016, 19.01, Judit Flo Gaya kirjoitti:
Producing: /usr/bin/rsync -vrltH --partial --delay-updates --delete-after --delete-excluded --stats --timeout=600 --exclude=.snapshot/ rsync://us-msync.centos.org/CentOS /misc/mirrors-rw/centos
You could try --delete-delay instead of --delete-after. See "man rsync" for details.
21.4.2016, 20.22, Anssi Johansson kirjoitti:
21.4.2016, 19.01, Judit Flo Gaya kirjoitti:
Producing: /usr/bin/rsync -vrltH --partial --delay-updates --delete-after --delete-excluded --stats --timeout=600 --exclude=.snapshot/ rsync://us-msync.centos.org/CentOS /misc/mirrors-rw/centos
You could try --delete-delay instead of --delete-after. See "man rsync" for details.
"Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after."
You could also try without --delay-updates, which also triggers this requirement to know the full file list in advance.
You could also try spreading the load by changing the centos sync to start at xx:27, and centos-debuginfo sync at xx:33 or so.
You could also try without --delay-updates, which also triggers this requirement to know the full file list in advance.
But not using --delay-updates means that the yum repo could be in an inconsistent/nonfuctional state until the sync finishes. That isn't good for a public mirror.
22.4.2016, 0.48, Chuck Anderson kirjoitti:
You could also try without --delay-updates, which also triggers this requirement to know the full file list in advance.
But not using --delay-updates means that the yum repo could be in an inconsistent/nonfuctional state until the sync finishes. That isn't good for a public mirror.
If also using --delete-delay, the files to be deleted will be deleted only at the end of sync, reducing the chances of anything breaking. Although I see your point, I believe the base repositories (os, updates, extras, fasttrack etc) would end up getting synced in such a sequence that yum won't get confused. The repodata directory tends to get synced last, and it is not harmful if there are .rpm files on the mirror that are not yet referenced in the repodata.
For the directories that have a more exotic directory layout, such as "atomic", it might indeed be useful to have --delete-delay in use.
However, for this particular case, testing one sync without that --delay-updates and --delete-delay instead of --delete-after might show if the problem is related to rsync needing to gather the full filelist.
On the other hand, I might be entirely on the wrong track here.. rsync did transfer a few files before stopping, which doesn't quite fit well with my theory. Oh well. Perhaps it's a firewall problem then.
On Fri, Apr 22, 2016 at 04:04:17AM +0300, Anssi Johansson wrote:
22.4.2016, 0.48, Chuck Anderson kirjoitti:
You could also try without --delay-updates, which also triggers this requirement to know the full file list in advance.
But not using --delay-updates means that the yum repo could be in an inconsistent/nonfuctional state until the sync finishes. That isn't good for a public mirror.
If also using --delete-delay, the files to be deleted will be deleted only at the end of sync, reducing the chances of anything breaking. Although I see your point, I believe the base repositories (os, updates, extras, fasttrack etc) would end up getting synced in such a sequence that yum won't get confused. The repodata directory tends to get synced last, and it is not harmful if there are .rpm files on the mirror that are not yet referenced in the repodata.
"tends to get synced last" isn't something to rely on.
Instead, you could sync everything except the repodata, then do a 2nd sync of just the repodata.
-
Anssi Johansson -
Chuck Anderson -
Fabian Arrotin -
Judit Flo Gaya