[CentOS-mirror] FTP mirrors & SIZE command

[Francois Petillon]

archive-admin at mirrorservice.org wrote:
> Ok is there any way for mirrors to check there status ie why its failing
> so they can do something about it...

First of all, I am not part of CentOs team. I have just given a bit of 
my time to patch the monitoring script.

Right now, HTTP seems to be nicely managed but the perl module used 
(LWP) have unrequested side effects on FTP which make unusable. Thus, 
the script is now using Net::FTP for FTP. For FTP, we mainly need MDTM 
and SIZE commands.
The last part will concern HTTP mirrors redirecting to FTP server (a 
patch has been made today but not tested yet).

> The "SIZE" command is not standardised. Check RFC 959 for a list
> of standard FTP commands.

Yes, I know but, on the few FTP mirrors I checked by hand, only two of 
them refused (none with an command unknown). After a quick brainstorming 
(but I was helped by mirrorservice who was answering "504 Can only do 
SIZE in image mode"), I just have realized some servers needed to be in 
type I to return size (even belnet whose answer was "550 SIZE: Not
owner" returns the size when type is I).

OK, using type I is now giving right size on all servers. A few of them 
seem to have problems with timestamp and/or timezone.

> Most FTP servers provide the size of the upcoming transfer in the
> 150-response to RETR, and libcurl attempts to figure out the size from
> that if the SIZE command didn't work.  You could then abort.

Starting a download was the problem of LWP. There were side effects on 
somes servers which make LWP fails sooner or later (sometimes the script 
can get info for a few file, sometimes it fails after the first one). 
Thus, I wanted to avoid at all cost this workaround.

François