Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
On Mon, Feb 12, 2018 at 6:25 PM H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
There are usually 2 different reasons for this: 1. The VPN is UDP and times out/drops keeps alives so that they no longer function properly. [The UDP connection will make it look like you have a new SSH connection which of course the system will drop because that would allow for security problems.]
2. A firewall in the chain of things (system you are on, the system you are going to, or somewhere in between) has session flushing issues. If you have the firewall set up to only accept NEW port 22 connections and then just looks to see if the ESTABLISHED, RELATED tables are accepted elsewhere then if the session somehow ages out or is flushed due to usage, the ssh connection can get dropped.
The solution to one is to see if a TCP VPN fixes the problem. The second one is to either make the iptables kernel tables larger or to have all port 22 accepted even if it is not ESTABLISHED.
These aren’t the only ways the problem you see can occur but they are some of the most common I have run into.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 02/12/2018 06:34 PM, Stephen John Smoogen wrote:
On Mon, Feb 12, 2018 at 6:25 PM H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
There are usually 2 different reasons for this:
- The VPN is UDP and times out/drops keeps alives so that they no longer
function properly. [The UDP connection will make it look like you have a new SSH connection which of course the system will drop because that would allow for security problems.]
- A firewall in the chain of things (system you are on, the system you are
going to, or somewhere in between) has session flushing issues. If you have the firewall set up to only accept NEW port 22 connections and then just looks to see if the ESTABLISHED, RELATED tables are accepted elsewhere then if the session somehow ages out or is flushed due to usage, the ssh connection can get dropped.
The solution to one is to see if a TCP VPN fixes the problem. The second one is to either make the iptables kernel tables larger or to have all port 22 accepted even if it is not ESTABLISHED.
These aren’t the only ways the problem you see can occur but they are some of the most common I have run into.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Not sure if I am reading your reply correctly but I should clarify that I have problems when running naked ssh to the server, when I run ssh to the same server but over the VPN connection (that goes via third server) everything is flawless.
I should also explain that:
- I am on a workstation (located in the US), ssh-ing into server 1 (located in the US).
- From server 1 I use scp to transfer large files from server 2 (located in Europe) to server 1 (in the US).
The above randomly disconnects.
However, when:
- I use a VPN connection to server 3 (also located in Europe).
- From the same workstation as above, do exactly as above, connections are rock-solid.
On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote: ...
Not sure if I am reading your reply correctly but I should clarify that I have problems when running naked ssh to the server, when I run ssh to the same server but over the VPN connection (that goes via third server) everything is flawless.
I should also explain that:
- I am on a workstation (located in the US), ssh-ing into server 1
(located in the US).
- From server 1 I use scp to transfer large files from server 2
(located in Europe) to server 1 (in the US).
The above randomly disconnects.
You have two ssh sessions here. AFAICT, it's the first one (workstation->server 1) that randomly disconnects, right?
However, when:
I use a VPN connection to server 3 (also located in Europe).
From the same workstation as above, do exactly as above,
connections are rock-solid.
It would seem to me that something between your workstation and server 1 is possibly mishandling TCP options (being it sack, timestamps or both). Is there a router or a firewall in between?
Try doing a traffic capture between both and see why it hangs. If you're hesitant to post the binary traffic capture, post the text version of it with anonymized IP addresses. Just please be sure to disable ssh protocol so it would include TCP details on it then.
Marcelo
On 02/19/2018 10:09 PM, Marcelo Ricardo Leitner wrote:
On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote: ...
Not sure if I am reading your reply correctly but I should clarify that I have problems when running naked ssh to the server, when I run ssh to the same server but over the VPN connection (that goes via third server) everything is flawless.
I should also explain that:
- I am on a workstation (located in the US), ssh-ing into server 1
(located in the US).
- From server 1 I use scp to transfer large files from server 2
(located in Europe) to server 1 (in the US).
The above randomly disconnects.
You have two ssh sessions here. AFAICT, it's the first one (workstation->server 1) that randomly disconnects, right?
However, when:
I use a VPN connection to server 3 (also located in Europe).
From the same workstation as above, do exactly as above,
connections are rock-solid.
It would seem to me that something between your workstation and server 1 is possibly mishandling TCP options (being it sack, timestamps or both). Is there a router or a firewall in between?
Try doing a traffic capture between both and see why it hangs. If you're hesitant to post the binary traffic capture, post the text version of it with anonymized IP addresses. Just please be sure to disable ssh protocol so it would include TCP details on it then.
Marcelo
Let's reduce the problem to a simpler case: the workstation running CentOS 7 is behind a router, behind the same router is the server running CentOS 6. For various reasons, I access the server via its external IP address, not the local one. Access is via ssh.
When using straight ssh, the connection times out after random times, always less than 20 minutes or so, though, with the error message "packet_write_wait: broken pipe", not the common "write failed: broken pipe".
However, when I am in a ssh session on a VPN connection (which terminates in an outside server far away from my network), the ssh connection does /not/ time out.
Further, accessing the same server as first mentioned, in the identical fashion, from another computer running CentOS 6, I never encounter the "packet_write_wait" error and disconnection.
My conclusion is thus that this is not a router issue but could conceivable be a bug in CentOS 7. I am also ruling out a cable issue or a physical network card issue since the VPN connection is stable.
On 02/20/2018 09:33 AM, H wrote:
On 02/19/2018 10:09 PM, Marcelo Ricardo Leitner wrote:
On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote: ...
Not sure if I am reading your reply correctly but I should clarify that I have problems when running naked ssh to the server, when I run ssh to the same server but over the VPN connection (that goes via third server) everything is flawless.
I should also explain that:
- I am on a workstation (located in the US), ssh-ing into server 1
(located in the US).
- From server 1 I use scp to transfer large files from server 2
(located in Europe) to server 1 (in the US).
The above randomly disconnects.
You have two ssh sessions here. AFAICT, it's the first one (workstation->server 1) that randomly disconnects, right?
However, when:
I use a VPN connection to server 3 (also located in Europe).
From the same workstation as above, do exactly as above,
connections are rock-solid.
It would seem to me that something between your workstation and server 1 is possibly mishandling TCP options (being it sack, timestamps or both). Is there a router or a firewall in between?
Try doing a traffic capture between both and see why it hangs. If you're hesitant to post the binary traffic capture, post the text version of it with anonymized IP addresses. Just please be sure to disable ssh protocol so it would include TCP details on it then.
Marcelo
Let's reduce the problem to a simpler case: the workstation running CentOS 7 is behind a router, behind the same router is the server running CentOS 6. For various reasons, I access the server via its external IP address, not the local one. Access is via ssh.
When using straight ssh, the connection times out after random times, always less than 20 minutes or so, though, with the error message "packet_write_wait: broken pipe", not the common "write failed: broken pipe".
However, when I am in a ssh session on a VPN connection (which terminates in an outside server far away from my network), the ssh connection does /not/ time out.
Further, accessing the same server as first mentioned, in the identical fashion, from another computer running CentOS 6, I never encounter the "packet_write_wait" error and disconnection.
My conclusion is thus that this is not a router issue but could conceivable be a bug in CentOS 7. I am also ruling out a cable issue or a physical network card issue since the VPN connection is stable.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
No messages in secure or message logs.
On 2018-02-12, H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.
On 02/12/2018 07:24 PM, Liam O'Toole wrote:
On 2018-02-12, H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.
Not that this happens while I do large scp file transfers that may take more than half an hour, simply restarting an ssh session is not going to help since I will lose the file transfer.
On Mon, 2018-02-12 at 20:13 -0500, H wrote:
On 02/12/2018 07:24 PM, Liam O'Toole wrote:
On 2018-02-12, H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.
Not that this happens while I do large scp file transfers that may take more than half an hour, simply restarting an ssh session is not going to help since I will lose the file transfer.
I don't know if this would help but I had a similar issue and it turned out that there was a custom script in /etc/profile.d/ that contain TMOUT 900.
You can also check in /etc/profile, usually, the security logs has something about the disconnects, of you can use wireshark or a similar tool to capture and analyse the packets.
On 02/12/2018 09:12 PM, Earl Ramirez wrote:
On Mon, 2018-02-12 at 20:13 -0500, H wrote:
On 02/12/2018 07:24 PM, Liam O'Toole wrote:
On 2018-02-12, H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.
Not that this happens while I do large scp file transfers that may take more than half an hour, simply restarting an ssh session is not going to help since I will lose the file transfer.
I don't know if this would help but I had a similar issue and it turned out that there was a custom script in /etc/profile.d/ that contain TMOUT 900.
You can also check in /etc/profile, usually, the security logs has something about the disconnects, of you can use wireshark or a similar tool to capture and analyse the packets.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
I think that I forgot to mention that the problem is not the common "write failed: broken pipe" but "packet_write_wait: broken pipe".
I don't think the problem is router related since another computer, albeit running CentOS 6, does not have similar issues.
Does the information above suggest any other reason for my problem?
On 2018-02-13, H agents@meddatainc.com wrote:
On 02/12/2018 07:24 PM, Liam O'Toole wrote:
On 2018-02-12, H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.
Not that this happens while I do large scp file transfers that may take more than half an hour, simply restarting an ssh session is not going to help since I will lose the file transfer.
In that case sftp with the '-a' option would be a better choice.
On 02/13/2018 11:54 AM, Liam O'Toole wrote:
On 2018-02-13, H agents@meddatainc.com wrote:
On 02/12/2018 07:24 PM, Liam O'Toole wrote:
On 2018-02-12, H agents@meddatainc.com wrote:
Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:
Host * TCPKeepAlive yes ServerAliveInterval 30 ServerAliveCountMax 300
and sshd_config on the server contains:
TCPKeepAlive yes ClientAliveInterval 60 ClientAliveCountMax 300
Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.
Thanks.
Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.
Not that this happens while I do large scp file transfers that may take more than half an hour, simply restarting an ssh session is not going to help since I will lose the file transfer.
In that case sftp with the '-a' option would be a better choice.
or rsync --partial
-
Earl Ramirez -
H -
Kay Diederichs -
Liam O'Toole -
Marcelo Ricardo Leitner -
Stephen John Smoogen