Greetings,
I have written a script using lftp to mirror one directory from a ftp site.
I do not have access to the server except for being a humble user.
This is running on a fully updated Centos 5.3 box.
I am using whatever is provided by the repositories.
I noticed that lftp consumes almost all of cpu times
I found this while searching for a solution.
http://www.mail-archive.com/lftp%40uniyar.ac.ru/msg02101.html
Could it be this reason? if its not has anybody else observed what I have described? What are the possible ways of getting around .
I am using --parallel=10 in the mirror sub-command of lftp
TIA
Rajagopal
On Tue, 2009-05-12 at 04:07 +0000, Rajagopal Swaminathan wrote:
Greetings,
I have written a script using lftp to mirror one directory from a ftp site.
I do not have access to the server except for being a humble user.
This is running on a fully updated Centos 5.3 box.
I am using whatever is provided by the repositories.
I noticed that lftp consumes almost all of cpu times
I found this while searching for a solution.
http://www.mail-archive.com/lftp%40uniyar.ac.ru/msg02101.html
Could it be this reason? if its not has anybody else observed what I have described? What are the possible ways of getting around .
I am using --parallel=10 in the mirror sub-command of lftp
TIA
Rajagopal
--- What does uname -a say? Post It. Post the complete lftp command also your using.
JohnS <jses27@...> writes:
What does uname -a say? Post It.
uname -a Linux localhost.localdomain 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:19:18 EDT 2009 i686 i686 i386 GNU/Linux
Post the complete lftp command also your using.
lftp -u 'username' ftp://ftpsite.com -d -e "set xfer:log true; set ftp:ssl-allow false;mirror -vc --log=/home/dnld/logs/lftp.log --parallel=10 /sourcedir/ /home/dnld/Dnld;exit"
For obvious reasons, I have had to hide exact details
Is there a method to log the filenames that are downloaded?
Regards
Rajagopal
Rajagopal Swaminathan <raju.rajsand@...> writes:
JohnS <jses27@...> writes:
I tried switching over to wget but inspite of having 2mbps link which gives us a download speed of around 200kBytes/sec otherwise, wget manages only 15-20kBytes/s even after using --limit-rate=150k
Nothing interesting in the /etc/wgetrc and no ~/.wgetrc either
Regards
Rajagopal
On Tue, 2009-05-12 at 04:52 +0000, Rajagopal Swaminathan wrote:
Rajagopal Swaminathan <raju.rajsand@...> writes:
JohnS <jses27@...> writes:
I tried switching over to wget but inspite of having 2mbps link which gives us a download speed of around 200kBytes/sec otherwise, wget manages only 15-20kBytes/s even after using --limit-rate=150k
Nothing interesting in the /etc/wgetrc and no ~/.wgetrc either
--- May be a server issue. You may want to try another server. If you have a local LAN try to wget a large file for comparing.
JohnS <jses27@...> writes:
May be a server issue. You may want to try another server. If you have a local LAN try to wget a large file for comparing.
But from the same server, on a windows box with DAP such speeds are available with multiple files downloading. I want to switch them over to centos so it can be decently cron-nable job.
The aggregate speed I have observed vary from 190kBytes/sec to peak of 230kBytes/sec -- On a Winbox -- with AV and other thrash running...
Regards and thanks
Rajagopal
On Tue, 2009-05-12 at 06:13 +0000, Rajagopal Swaminathan wrote:
JohnS <jses27@...> writes:
May be a server issue. You may want to try another server. If you have a local LAN try to wget a large file for comparing.
But from the same server, on a windows box with DAP such speeds are available with multiple files downloading. I want to switch them over to centos so it can be decently cron-nable job.
The aggregate speed I have observed vary from 190kBytes/sec to peak of 230kBytes/sec -- On a Winbox -- with AV and other thrash running...
--- Check out you networking stack. Like NIC Card settings with ethtool and your dns like namserver settings in resolve.conf. If it is getting an address by dhcp sometimes it want pull in the actual real dns servers. All will get is like a 192.168.0.x from the modem/router. Ifconfig ethX will show you the amount of packets dropped also. Possibly a driver issue with your nic? Could be many things you just have to go step by step...
JohnS <jses27@...> writes:
Check out you networking stack. Like NIC Card settings with ethtool and your dns like namserver settings in resolve.conf. If it is getting an address by dhcp sometimes it want pull in the actual real dns servers.
ethtool eth0
spake thus:
Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000033 (51) Link detected: yes
They are reasonably fine. it is a static IP.
All will get is like a 192.168.0.x from the modem/router. Ifconfig ethX will show you the amount of packets dropped also. Possibly a driver issue with your nic? Could be many things you just have to go step by step...
ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:16:E6:96:CD:A8 inet addr:192.168.2.220 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::216:e6ff:fe96:cda8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3434000 errors:0 dropped:0 overruns:0 frame:0 TX packets:1879546 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2928267020 (2.7 GiB) TX bytes:550552465 (525.0 MiB) Interrupt:177 Base address:0xe000
But then yum install/update etc gives me reasonable speeds in the range of 100-120 KBytes/second and our network load is that much usually. 15-20Kbytes is ridiculous
We have a DNS server (an AD server)
Is it that something that wget puts out in the network that is not liked by our firewall?
Regards
Rajagopal
On Tue, 2009-05-12 at 08:04 +0000, Rajagopal Swaminathan wrote:
JohnS <jses27@...> writes:
Check out you networking stack. Like NIC Card settings with ethtool and your dns like namserver settings in resolve.conf. If it is getting an address by dhcp sometimes it want pull in the actual real dns servers.
ethtool eth0
spake thus:
Settings for eth0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000033 (51) Link detected: yes
They are reasonably fine. it is a static IP.
All will get is like a 192.168.0.x from the modem/router. Ifconfig ethX will show you the amount of packets dropped also. Possibly a driver issue with your nic? Could be many things you just have to go step by step...
ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:16:E6:96:CD:A8 inet addr:192.168.2.220 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::216:e6ff:fe96:cda8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3434000 errors:0 dropped:0 overruns:0 frame:0 TX packets:1879546 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2928267020 (2.7 GiB) TX bytes:550552465 (525.0 MiB) Interrupt:177 Base address:0xe000
But then yum install/update etc gives me reasonable speeds in the range of 100-120 KBytes/second and our network load is that much usually. 15-20Kbytes is ridiculous
We have a DNS server (an AD server)
Is it that something that wget puts out in the network that is not liked by our firewall?
--- That is possible and it could be Bandwidth Throttling the connection much like ISPs do to peer to peer networks.
There is a wget for windows. Maybe you could try that and compare the difference.
On Tue, 2009-05-12 at 04:48 +0000, Rajagopal Swaminathan wrote:
JohnS <jses27@...> writes:
What does uname -a say? Post It.
uname -a Linux localhost.localdomain 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:19:18 EDT 2009 i686 i686 i386 GNU/Linux
Post the complete lftp command also your using.
lftp -u 'username' ftp://ftpsite.com -d -e "set xfer:log true; set ftp:ssl-allow false;mirror -vc --log=/home/dnld/logs/lftp.log --parallel=10 /sourcedir/ /home/dnld/Dnld;exit"
For obvious reasons, I have had to hide exact details
Is there a method to log the filenames that are downloaded?
Hmm prolly so, I believe for sure wget can do that and also pull files from a server based on a filelist.
Regards
Rajagopal
--- Try this: "lftp -e 'open ftp://ftpsite.com/pub/centos/5/updates && \ mirror -c --delete x86_64 && exit' "
Where do you get all those other options because there is no Man Page and lftp --help shows very little. Anyway that is what I use to mirror CentOS 5 4 and 2.1 with and no problems with high CPU usage.
JohnStanley
JohnS <jses27@...> writes:
Is there a method to log the filenames that are downloaded?
Hmm prolly so, I believe for sure wget can do that and also pull files from a server based on a filelist.
Where do you get all those other options because there is no Man Page and lftp --help shows very little. Anyway that is what I use to mirror CentOS 5 4 and 2.1 with and no problems with high CPU usage.
just type lftp. It drops you into a shell and help is your friend. set -a gives you all kinds of variables that can be set
Regards
Rajagopal
On Tue, 2009-05-12 at 06:17 +0000, Rajagopal Swaminathan wrote:
JohnS <jses27@...> writes:
Is there a method to log the filenames that are downloaded?
Hmm prolly so, I believe for sure wget can do that and also pull files from a server based on a filelist.
Where do you get all those other options because there is no Man Page and lftp --help shows very little. Anyway that is what I use to mirror CentOS 5 4 and 2.1 with and no problems with high CPU usage.
just type lftp. It drops you into a shell and help is your friend. set -a gives you all kinds of variables that can be set
--- I never actually use the shell like that way. Will check the set -a variable later on.
Rajagopal Swaminathan <raju.rajsand@...> writes:
It seems yum -y update does not update lftp. probably lftp in standard repos are older version -- 3.5.x .
I included the rpmforge repo and did yum -y update lftp -- 3.7.x.
So far it seems to be ok.
I am yet to find the solution to get wget to use full bandwith
Thanks and Regards
Rajagopal
-
JohnS -
Rajagopal Swaminathan