I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
These servers are all over the country on different ISPs in Tier1 datacenters. Some are in our office, so they are on the local lan. We have a mix of RHEL 3, 4 & 5 and CentOS 4 & 5 on the servers. If I'm using a windows computer with putty or SecureCRT this never happens, it only happens when I'm using any of our linux desktops or laptops. It doesn't matter if I'm in the office or at home (on comcast) or over at a friend's house (verizon dsl). This problem has been going on for at least two years and I'm finally fed up to the point where I might switch back to windows since 99% of my job is working while ssh'ed into servers.
Anyone had similar problems?
-matt
I should add that the hang occurs after an unknown amount of time.
-matt
On 7/12/07, Matt Shields mattboston@gmail.com wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
These servers are all over the country on different ISPs in Tier1 datacenters. Some are in our office, so they are on the local lan. We have a mix of RHEL 3, 4 & 5 and CentOS 4 & 5 on the servers. If I'm using a windows computer with putty or SecureCRT this never happens, it only happens when I'm using any of our linux desktops or laptops. It doesn't matter if I'm in the office or at home (on comcast) or over at a friend's house (verizon dsl). This problem has been going on for at least two years and I'm finally fed up to the point where I might switch back to windows since 99% of my job is working while ssh'ed into servers.
Anyone had similar problems?
-matt
I should add that the hang occurs after an unknown amount of time.
Are we talking days, hours, or minutes here? I use ssh all the time to log into the systems I manage, but they are all on the LAN. I often log into my home CentOS system, and keep the connection up for an entire work day (8+ hours).
I currently have some ssh sessions that have been running for at least a few weeks. When the connection hangs, are you still able to disconnect it using "~." (make sure you hit return before you try that escape sequence).
Alfred
time of day doesn't matter. sometimes I can maintain a connection all day (very rare), but other times it hangs quite a few times a day. Most of the time it usually hangs when a screen full of data is being requested like 'ps -auxwww' or 'cat file', but other times it hangs when I'm typing on the command line.
all machines are hard wired (no wifi). I've replace all CAT6 cables at least twice. It doesn't seem to disconnect because if I open a new terminal and ssh again I can see the connection on the server, but on my side it doesn't respond again.
there's no real common piece that i've been able to track down. like I said this has been going on for a few years. It happens on numerous type of hardware (dell laptops and desktops, hp desktops), various flavors of RedHat based linux. Various servers (dell, hp, whitebox), various internet connections.
-matt
On 7/12/07, Alfred von Campe alfred@von-campe.com wrote:
I should add that the hang occurs after an unknown amount of time.
Are we talking days, hours, or minutes here? I use ssh all the time to log into the systems I manage, but they are all on the LAN. I often log into my home CentOS system, and keep the connection up for an entire work day (8+ hours).
I currently have some ssh sessions that have been running for at least a few weeks. When the connection hangs, are you still able to disconnect it using "~." (make sure you hit return before you try that escape sequence).
Alfred
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Thu, Jul 12, 2007 at 09:46:00AM -0400, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
ssh -vvv might give a glue selinux enforced on your client machines?
Tru
selinux is turned off on both servers and desktops
On 7/12/07, Tru Huynh tru@centos.org wrote:
On Thu, Jul 12, 2007 at 09:46:00AM -0400, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
ssh -vvv might give a glue selinux enforced on your client machines?
Tru
Tru Huynh (CentOS-3 i386/x86_64 Package Maintenance) http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
most problem in this case is DNS!!
does the session comes ca. 30sec later?
problem can be:
1) the server has wrong dns-server in /etc/resolv.conf 2) the client ip is a private ip, and not in /etc/hosts 3) the server ip is not in clients /etc/hosts
point 2) is the problem i have most time
bg, paul
Am Thu, 12 Jul 2007 10:45:50 -0400 "Matt Shields" mattboston@gmail.com schrieb:
selinux is turned off on both servers and desktops
On 7/12/07, Tru Huynh tru@centos.org wrote:
On Thu, Jul 12, 2007 at 09:46:00AM -0400, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
ssh -vvv might give a glue selinux enforced on your client machines?
Tru
Tru Huynh (CentOS-3 i386/x86_64 Package Maintenance) http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
It shouldn't be dns because the session is already established and it now IP based. I don't believe ssh tries to keep resolving the IP again and again.
No session doesn't come back ever. It just hangs permanently.
-matt
On 7/12/07, Paul Valentin pvalentin@roadrunner.at wrote:
most problem in this case is DNS!!
does the session comes ca. 30sec later?
problem can be:
- the server has wrong dns-server in /etc/resolv.conf
- the client ip is a private ip, and not in /etc/hosts
- the server ip is not in clients /etc/hosts
point 2) is the problem i have most time
bg, paul
Am Thu, 12 Jul 2007 10:45:50 -0400 "Matt Shields" mattboston@gmail.com schrieb:
selinux is turned off on both servers and desktops
On 7/12/07, Tru Huynh tru@centos.org wrote:
On Thu, Jul 12, 2007 at 09:46:00AM -0400, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
ssh -vvv might give a glue selinux enforced on your client machines?
Tru
Tru Huynh (CentOS-3 i386/x86_64 Package Maintenance) http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 7/12/07, Matt Shields mattboston@gmail.com wrote:
No session doesn't come back ever. It just hangs permanently.
Curiouser and curiouser.
My first thought would be a firewall/router/switch timing out, but that typically happens for idle sessions, and you said somewhere that this is happening during activity. And of course you said this happens from multiple source locations and multiple destinations, that does make it harder to think a network element would be the cause.
I have active ssh sessions between various Linux systems that have up for weeks (or months), with no issues.
Have you ever tried sniffing for network traffic related to a hung session?
Up till now I haven't had the time or energy to put into sniffing or debugging at all. But it's getting to be a pain in the @ss, that's why I'm trying to see if anyone else has these issues.
Here are the locations and hardware:
Home: comcast with linksys cable/dsl router, hard wired (no wifi), brand new cables, tried multiple cable/dsl routers
Work (over a month ago): over 3 bonded T1 ATM circuits (from local ISP), using linux hardware firewall, 10/100/1000 Dell poweredge switches, some servers locally, some servers at remote Tier1 datacenters
Work (recently): upgraded connection to 10MBit fractional DS3 from Verizon Business (MCI circuit)
Friends house: dlink cable/dsl router, hard wired
-matt
On 7/12/07, Dave K davek08054@gmail.com wrote:
On 7/12/07, Matt Shields mattboston@gmail.com wrote:
No session doesn't come back ever. It just hangs permanently.
Curiouser and curiouser.
My first thought would be a firewall/router/switch timing out, but that typically happens for idle sessions, and you said somewhere that this is happening during activity. And of course you said this happens from multiple source locations and multiple destinations, that does make it harder to think a network element would be the cause.
I have active ssh sessions between various Linux systems that have up for weeks (or months), with no issues.
Have you ever tried sniffing for network traffic related to a hung session?
-- Dave K Unix Systems & Network Administrator Mount Laurel NJ _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Matt Shields wrote:
It shouldn't be dns because the session is already established and it now IP based. I don't believe ssh tries to keep resolving the IP again and again.
No session doesn't come back ever. It just hangs permanently.
-matt
We were seeing something similar to this a while back, SSH sessions to or from outside our network were dropping after some period of no activity (which may not be your problem). Eventually we found that the Cisco PIX on our perimeter was set to kill idle sessions sessions after a certain period.
We were able to resolve this by editing /etc/ssh/sshd_config and setting the ClientAliveInterval to a non-zero value. In our case we set it to 240, which caused a "ClientAlive" request packet to be sent every 4 minutes over the encrypted channel as the idle threshold on the PIX was set to 5 minutes. This resolved our issues, perhaps it might help with yours.
Just a thought!
Yeah I've set that on a few machines just to test. But these connections aren't inactive. I'm actively working in that session when it happens.
-matt
On 7/15/07, Jay Leafey jay.leafey@mindless.com wrote:
Matt Shields wrote:
It shouldn't be dns because the session is already established and it now IP based. I don't believe ssh tries to keep resolving the IP again and again.
No session doesn't come back ever. It just hangs permanently.
-matt
We were seeing something similar to this a while back, SSH sessions to or from outside our network were dropping after some period of no activity (which may not be your problem). Eventually we found that the Cisco PIX on our perimeter was set to kill idle sessions sessions after a certain period.
We were able to resolve this by editing /etc/ssh/sshd_config and setting the ClientAliveInterval to a non-zero value. In our case we set it to 240, which caused a "ClientAlive" request packet to be sent every 4 minutes over the encrypted channel as the idle threshold on the PIX was set to 5 minutes. This resolved our issues, perhaps it might help with yours.
Just a thought!
Jay Leafey - Memphis, TN jay.leafey@mindless.com
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I've seen a similar issue when I've had an ssh connection open to a remote site over a VPN connection (not a VPN client on my PC, but a VPN connection between my site and the remote site, between Cisco routers). Catting a large file would hang the connection, and I had to kill and re-establish the connection. The issue was that the maximum packet size over the VPN link was something like 1460. The VPN connection added 40 extra bytes of payload. When I would cat a large file, my computer would send 1500 byte packets with the do-not- fragment bit set. The routers couldn't pass the 1500 byte packet because it was too large, and the do-not-fragment bit prevented them from fragmenting the packet, so it would get dropped, and my connection would die. I don't remember the exact details, as it occurred a couple years ago, but the gist was that the MTU on the servers I had to connect to had to be reduced to something like 1460. I don't think this is exactly your situation, but it's just an idea of something to consider.
-Tim
On Jul 12, 2007, at 9:46 AM, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
These servers are all over the country on different ISPs in Tier1 datacenters. Some are in our office, so they are on the local lan. We have a mix of RHEL 3, 4 & 5 and CentOS 4 & 5 on the servers. If I'm using a windows computer with putty or SecureCRT this never happens, it only happens when I'm using any of our linux desktops or laptops. It doesn't matter if I'm in the office or at home (on comcast) or over at a friend's house (verizon dsl). This problem has been going on for at least two years and I'm finally fed up to the point where I might switch back to windows since 99% of my job is working while ssh'ed into servers.
Anyone had similar problems?
-matt _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Interesting, we use OpenVPN heavily, but we also have a lot not behind the vpn. That will be something to test out, I haven't kept track of which servers it happens with.
-matt
On 7/15/07, Tim Meanor timspam@meanor.net wrote:
I've seen a similar issue when I've had an ssh connection open to a remote site over a VPN connection (not a VPN client on my PC, but a VPN connection between my site and the remote site, between Cisco routers). Catting a large file would hang the connection, and I had to kill and re-establish the connection. The issue was that the maximum packet size over the VPN link was something like 1460. The VPN connection added 40 extra bytes of payload. When I would cat a large file, my computer would send 1500 byte packets with the do-not- fragment bit set. The routers couldn't pass the 1500 byte packet because it was too large, and the do-not-fragment bit prevented them from fragmenting the packet, so it would get dropped, and my connection would die. I don't remember the exact details, as it occurred a couple years ago, but the gist was that the MTU on the servers I had to connect to had to be reduced to something like 1460. I don't think this is exactly your situation, but it's just an idea of something to consider.
-Tim
On Jul 12, 2007, at 9:46 AM, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
These servers are all over the country on different ISPs in Tier1 datacenters. Some are in our office, so they are on the local lan. We have a mix of RHEL 3, 4 & 5 and CentOS 4 & 5 on the servers. If I'm using a windows computer with putty or SecureCRT this never happens, it only happens when I'm using any of our linux desktops or laptops. It doesn't matter if I'm in the office or at home (on comcast) or over at a friend's house (verizon dsl). This problem has been going on for at least two years and I'm finally fed up to the point where I might switch back to windows since 99% of my job is working while ssh'ed into servers.
Anyone had similar problems?
-matt _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I'd say the majority of the time this is a due to an unreliable internet connection or inconsistent equipment (routers, etc.) on the connection. My router at home is a piece of crap (aren't all consumer grade routers?) and if I leave an SSH connection open for several hours it will eventually hang.
Devin
On 7/15/07, Matt Shields mattboston@gmail.com wrote:
Interesting, we use OpenVPN heavily, but we also have a lot not behind the vpn. That will be something to test out, I haven't kept track of which servers it happens with.
-matt
On 7/15/07, Tim Meanor timspam@meanor.net wrote:
I've seen a similar issue when I've had an ssh connection open to a remote site over a VPN connection (not a VPN client on my PC, but a VPN connection between my site and the remote site, between Cisco routers). Catting a large file would hang the connection, and I had to kill and re-establish the connection. The issue was that the maximum packet size over the VPN link was something like 1460. The VPN connection added 40 extra bytes of payload. When I would cat a large file, my computer would send 1500 byte packets with the do-not- fragment bit set. The routers couldn't pass the 1500 byte packet because it was too large, and the do-not-fragment bit prevented them from fragmenting the packet, so it would get dropped, and my connection would die. I don't remember the exact details, as it occurred a couple years ago, but the gist was that the MTU on the servers I had to connect to had to be reduced to something like 1460. I don't think this is exactly your situation, but it's just an idea of something to consider.
-Tim
On Jul 12, 2007, at 9:46 AM, Matt Shields wrote:
I've noticed this in CentOS 4 & 5 and Fedora 5 & 6. If I'm in Gnome desktop and using any of the terminal programs and I ssh into any server, the connection just hangs. Not drops, it just hangs and doesn't recover.
These servers are all over the country on different ISPs in Tier1 datacenters. Some are in our office, so they are on the local lan. We have a mix of RHEL 3, 4 & 5 and CentOS 4 & 5 on the servers. If I'm using a windows computer with putty or SecureCRT this never happens, it only happens when I'm using any of our linux desktops or laptops. It doesn't matter if I'm in the office or at home (on comcast) or over at a friend's house (verizon dsl). This problem has been going on for at least two years and I'm finally fed up to the point where I might switch back to windows since 99% of my job is working while ssh'ed into servers.
Anyone had similar problems?
-matt _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Alfred von Campe -
Dave K -
Devin Henderson -
Jay Leafey -
Matt Shields -
Paul Valentin -
Tim Meanor -
Tru Huynh