I have an SSH server that was set up for a client, and every time we try to upload large files via SFTP or scp, the transfers speed quickly slows to zero and gives a - stalled - status message, then disconnects. Here is an example:
ftp> put iTunesSetup.exe iTunesSetup.exe Uploading iTunesSetup.exe to /home/scarolan/iTunesSetup.exe iTunesSetup.exe 0% 704KB 0.0KB/s - stalled -debug1: channel 0: free: client-session, nchannels 1 Read from remote host ftp.authoria.net: Connection reset by peer debug1: Transferred: stdin 0, stdout 0, stderr 66 bytes in 203.9 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.3 debug1: Exit status -1 Connection closed
I have no idea why this is happening. Can anyone point me in the right direction?
I met the same as you, but always due to the bad network connection.
On 12/21/09, Sean Carolan scarolan@gmail.com wrote:
I have an SSH server that was set up for a client, and every time we try to upload large files via SFTP or scp, the transfers speed quickly slows to zero and gives a - stalled - status message, then disconnects. Here is an example:
ftp> put iTunesSetup.exe iTunesSetup.exe Uploading iTunesSetup.exe to /home/scarolan/iTunesSetup.exe iTunesSetup.exe 0% 704KB 0.0KB/s - stalled -debug1: channel 0: free: client-session, nchannels 1 Read from remote host ftp.authoria.net: Connection reset by peer debug1: Transferred: stdin 0, stdout 0, stderr 66 bytes in 203.9 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.3 debug1: Exit status -1 Connection closed
I have no idea why this is happening. Can anyone point me in the right direction? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Mon, Dec 21, 2009 at 7:06 PM, 唐建伟 myhnet@gmail.com wrote:
I met the same as you, but always due to the bad network connection.
I should probably provide some more information, the server is a VMware guest running CentOS 5.3. It's using the vmxnet driver for the eth0 connection. IPv6 is disabled.
Sean Carolan wrote:
On Mon, Dec 21, 2009 at 7:06 PM, 唐建伟 <myhnet@gmail.com mailto:myhnet@gmail.com> wrote:
I met the same as you, but always due to the bad network connection.I should probably provide some more information, the server is a VMware guest running CentOS 5.3. It's using the vmxnet driver for the eth0 connection. IPv6 is disabled.
I'm not sure what would cause that, but I'd use rsync over ssh instead of sftp anyway - and use the -P option to permit restarting.
I'm not sure what would cause that, but I'd use rsync over ssh instead of sftp anyway - and use the -P option to permit restarting.
If it were up to me, we'd take that route. The software the client is using is WinSCP which does have a restart feature, however it's not working for us. I'm wondering if this is somehow caused by the vmware network driver?
Sean Carolan wrote on Tue, 22 Dec 2009 03:08:53 -0600:
The software the client is using is WinSCP which does have a restart feature, however it's not working for us.
Tell him to switch WinSCP to SCP mode.
Kai
Tell him to switch WinSCP to SCP mode.
Kai
Tried that, it still fails the same way. Here's the short list of what I've tried to troubleshoot this:
Used SCP via the gui and command line Used SFTP via the gui and command line Ran yum update to bring all packages up to date Tried stock CentOS sshd daemon (version 4.3), as well as sshd built from source (version 5.3) Adjusted MTU settings Reinstalled virtual network card Updated vmware tools and network card driver Tried vmxnet as well as e1000 drivers
At this point I don't know what else to try. I'm thinking that it's either a problem with VMWare, or perhaps our load balancer that is routing the packets back and forth. Hopefully one of the vendors will be able to help solve the problem. In the meantime we are building out a physical server to test whether vmware is the issue or not.
If anyone else has seen this problem before or has suggestions please post them here. Thanks.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Sean Carolan Sent: Tuesday, December 22, 2009 6:13 AM To: CentOS mailing list Subject: Re: [CentOS] SFTP - stalled - on large files
Tell him to switch WinSCP to SCP mode.
Kai
Tried that, it still fails the same way. Here's the short list of what I've tried to troubleshoot this:
Used SCP via the gui and command line Used SFTP via the gui and command line Ran yum update to bring all packages up to date Tried stock CentOS sshd daemon (version 4.3), as well as sshd built from source (version 5.3) Adjusted MTU settings Reinstalled virtual network card Updated vmware tools and network card driver Tried vmxnet as well as e1000 drivers
At this point I don't know what else to try. I'm thinking that it's either a problem with VMWare, or perhaps our load balancer that is routing the packets back and forth. Hopefully one of the vendors will be able to help solve the problem. In the meantime we are building out a physical server to test whether vmware is the issue or not.
If anyone else has seen this problem before or has suggestions please post them here. Thanks.
---
Just an idea or thought on it. You never said what the file size was or did you? My idea is that is, there not a file size limitation on transfer to and from the server? I thought there was? Check you vsftpd.conf out or what ever ftp server your running for the size limitation. Maybe some help or maybe not?
John
Just an idea or thought on it. You never said what the file size was or did you? My idea is that is, there not a file size limitation on transfer to and from the server? I thought there was? Check you vsftpd.conf out or what ever ftp server your running for the size limitation. Maybe some help or maybe not?
The problem is with SFTP, so I'm afraid that vsftpd.conf isn't the culprit here.
Sean Carolan wrote on Tue, 22 Dec 2009 05:12:52 -0600:
Here's the short list of what I've tried to troubleshoot this:
which means it doesn't only fail for your client from outside but also for you from within your network?
Kai
Sean Carolan wrote: <snip>
At this point I don't know what else to try. I'm thinking that it's either a problem with VMWare, or perhaps our load balancer that is routing the packets back and forth. Hopefully one of the vendors will
Load balancer... is that set up to maintain connections, or will it, like IBM's WebSeal, go to whichever server is next/least used in the middle of a connection?
mark
Load balancer... is that set up to maintain connections, or will it, like IBM's WebSeal, go to whichever server is next/least used in the middle of a connection?
It's set to use "least connection" but there is only one server behind the virtual IP at the moment.
I'm reasonably sure at this point that the Netscaler is causing the problem, because file transfers inside the LAN work fine, and we see this same issue on both physical and virtual servers. I just tested with a physical box to verify, and the same thing happens, transfer speed quickly drops to zero and stalls.
I've got a ticket open with Citrix to hopefully get to the bottom of this. It wouldn't be the first time we've seen the Netscaler muck up a TCP connection from a client. The last time I dealt with this it was sending unwanted FIN packets to mail servers. Fun stuff.
We had a similar problem copying files between servers on two of our campuses via SCP. After a while the connection just stalled out and hung. The problem turned out to be SCP and SFTP interacting a bug in the SACK (Selective Acknowledgment) algorithm used in Linux. We turned it off on the two endpoints using the following addition to /etc/sysctl.conf:
# Turn off SACK net.ipv4.tcp_sack = 0
and execute "sysctl -p" to apply it. You can also use "sysctl -w net.ipv4.tcp_sack=0" to turn it off temporarily. Our file transfers worked just fine after the change.
I realize there are differences our situation and yours and this might not work in your case. Given the length of this thread, though, it might be worth a try!
# Turn off SACK net.ipv4.tcp_sack = 0
and execute "sysctl -p" to apply it. You can also use "sysctl -w net.ipv4.tcp_sack=0" to turn it off temporarily. Our file transfers worked just fine after the change.
I realize there are differences our situation and yours and this might not work in your case. Given the length of this thread, though, it might be worth a try!
It appears that the Netscaler load balancer was the problem. We turned off TCP buffering (TCPB) on the netscaler for this particular virtual server, and I was immediately able to transfer a 95MB file with no issues. Citrix has acknowledged that there may be some issues with the tcp stack on this device, which they think have been resolved in more recent versions of the Netscaler OS.
Hopefully if anyone else experiences this issue, they'll be able to Google it via the CentOS list archives.
-
Jay Leafey -
JS -
Kai Schaetzl -
Les Mikesell -
mark -
Sean Carolan -
唐建伟