Hi.
For a long time I am puzzled why, when internet connection is gone (beyond first router, outside of subnet) SSH (and telnet?) connection is drastically slowed.
Same behavior, but less impacted is observed when system boots without working internet connection, and I am not sure, but I think even access to SAMBA fileshares can be/is affected. Affected systems are mostly production servers, and too spaced apart in time to be able to point a finger in something I am doing wrong.
I have not had time to gather more facts, I always forget/don;t have time, so I will only focus on definite SSH issue on CentOS 6.3, although I have seen same with 5.x.
1 server in question is set in following maner: [root@chiron ~]# cat /etc/networks default 0.0.0.0 loopback 127.0.0.0 link-local 169.254.0.0
[root@chiron ~]# cat /etc/sysconfig/network NETWORKING=yes NTPSERVERARGS=iburst HOSTNAME=chiron.example.com
[root@chiron ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 chiron.example.com chiron ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 chiron.example.com chiron
Does it have anything to do with /etc/hosts file not having row with used IP address, or is problem somewhere else?
Searching the net gives too much irrelevant links, and all I need it to be pointed in the right direction.
Ljubomir Ljubojevic wrote:
Hi.
For a long time I am puzzled why, when internet connection is gone (beyond first router, outside of subnet) SSH (and telnet?) connection is drastically slowed.
Same behavior, but less impacted is observed when system boots without working internet connection, and I am not sure, but I think even access to SAMBA fileshares can be/is affected. Affected systems are mostly production servers, and too spaced apart in time to be able to point a finger in something I am doing wrong.
I have not had time to gather more facts, I always forget/don;t have time, so I will only focus on definite SSH issue on CentOS 6.3, although I have seen same with 5.x.
<snip> Have you tried traceroute to a well-known location, like google.com, and seen where the delay is?
mark
On 11/20/2012 07:34 PM, m.roth@5-cent.us wrote:
Ljubomir Ljubojevic wrote:
Hi.
For a long time I am puzzled why, when internet connection is gone (beyond first router, outside of subnet) SSH (and telnet?) connection is drastically slowed.
Same behavior, but less impacted is observed when system boots without working internet connection, and I am not sure, but I think even access to SAMBA fileshares can be/is affected. Affected systems are mostly production servers, and too spaced apart in time to be able to point a finger in something I am doing wrong.
I have not had time to gather more facts, I always forget/don;t have time, so I will only focus on definite SSH issue on CentOS 6.3, although I have seen same with 5.x.
<snip> Have you tried traceroute to a well-known location, like google.com, and seen where the delay is?
mark
Let me make it more clear.
I have a small WISP. I get my internet connection vai several wireless links/towers. My main server is located in my house. My own PC and the Server in question (in the example) are separated by a router in my house (different subnets). Server is fresh install of CentOS 6.3, all updated.
Now, when there was power failure (works on the transformer there), I lost my internet connection, but router and wireless routers for the rest of my towers were accessible. SSH logins to any of them is instant.
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
Past experience was, separate from this example, that a CentOS 5.x server in one company slows access, temporary timeout to SAMBA file server when wireless router connecting them to internet is turned off, although they are in local lan with static IP's.
On Tue, Nov 20, 2012 at 07:48:40PM +0100, Ljubomir Ljubojevic wrote:
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
DNS. Your host is configured to resolved IP addresses and is not able to because of lack of external DNS, so it's timing out.
On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic office@plnet.rs wrote:
Now, when there was power failure (works on the transformer there), I lost my internet connection, but router and wireless routers for the rest of my towers were accessible. SSH logins to any of them is instant.
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
Most server apps will do a reverse-DNS lookup, if only to log the name for the connection, some will try an ident query for the user at the other end of the socket. A 30+ second delay is a pretty sure sign that one or more of the DNS servers in your resolv.conf did not respond. Running a local nameserver with a dummy local domain is one way to fix it, but just putting all your local systems in the /etc/hosts file will work too.
On 11/20/2012 09:25 PM, Les Mikesell wrote:
On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic office@plnet.rs wrote:
Now, when there was power failure (works on the transformer there), I lost my internet connection, but router and wireless routers for the rest of my towers were accessible. SSH logins to any of them is instant.
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
Most server apps will do a reverse-DNS lookup, if only to log the name for the connection, some will try an ident query for the user at the other end of the socket. A 30+ second delay is a pretty sure sign that one or more of the DNS servers in your resolv.conf did not respond. Running a local nameserver with a dummy local domain is one way to fix it, but just putting all your local systems in the /etc/hosts file will work too.
OK, that is what crossed my mind, but what I was hopping for is some elegant solution that would decrease the timeout. My server already has DNS server running and "nameserver 127.0.0.1" as first in /etc/resolv.conf.
So the question is: "is there a setting that will reduce that DNS timeout for all running services, maybe like a ping-watchdog that would recognize the problem and skip the reverse-DNS lookup if DNS servers are not reachable?"
Adding and maintaining 30+ subnets in /etc/hosts is not really a good solution, and booting the server without reachable DNS server in some cases can be really frustrating, like if I boot Lap-top on the silo when internet connection is down (It was happening to me when I ran RHEL 6 beta I think on each opening of the terminal, but I can not say I have seen this lately).
Ljubomir Ljubojevic wrote:
On 11/20/2012 09:25 PM, Les Mikesell wrote:
On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic office@plnet.rs wrote:
<snip>
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
Most server apps will do a reverse-DNS lookup, if only to log the name for the connection, some will try an ident query for the user at the other end of the socket. A 30+ second delay is a pretty sure sign that one or more of the DNS servers in your resolv.conf did not respond. Running a local nameserver with a dummy local domain is one way to fix it, but just putting all your local systems in the /etc/hosts file will work too.
<snip>
So the question is: "is there a setting that will reduce that DNS timeout for all running services, maybe like a ping-watchdog that would recognize the problem and skip the reverse-DNS lookup if DNS servers are not reachable?"
<snip> What does it say in /etc/nsswitch: is it dns files, or files dns?
mark
On 11/20/2012 09:47 PM, m.roth@5-cent.us wrote:
Ljubomir Ljubojevic wrote:
On 11/20/2012 09:25 PM, Les Mikesell wrote:
On Tue, Nov 20, 2012 at 12:48 PM, Ljubomir Ljubojevic office@plnet.rs wrote:
<snip> >>> But when I tried to login to my server, it was not instantenous, and I >>> think it was 15+, maybe even 30+ seconds (I forgot to time it) from >>> start of ssh command to password prompt. It is in-house connection, so >>> there is nothing to traceroute. >> >> Most server apps will do a reverse-DNS lookup, if only to log the name >> for the connection, some will try an ident query for the user at the >> other end of the socket. A 30+ second delay is a pretty sure sign >> that one or more of the DNS servers in your resolv.conf did not >> respond. Running a local nameserver with a dummy local domain is one >> way to fix it, but just putting all your local systems in the >> /etc/hosts file will work too. <snip> > So the question is: "is there a setting that will reduce that DNS > timeout for all running services, maybe like a ping-watchdog that would > recognize the problem and skip the reverse-DNS lookup if DNS servers are > not reachable?" <snip> What does it say in /etc/nsswitch: is it dns files, or files dns?
"files dns"
On Tue, Nov 20, 2012 at 2:45 PM, Ljubomir Ljubojevic office@plnet.rs wrote:
Most server apps will do a reverse-DNS lookup, if only to log the name for the connection, some will try an ident query for the user at the other end of the socket. A 30+ second delay is a pretty sure sign that one or more of the DNS servers in your resolv.conf did not respond. Running a local nameserver with a dummy local domain is one way to fix it, but just putting all your local systems in the /etc/hosts file will work too.
OK, that is what crossed my mind, but what I was hopping for is some elegant solution that would decrease the timeout. My server already has DNS server running and "nameserver 127.0.0.1" as first in /etc/resolv.conf.
OK, but does it answer for your local network or do those queries float up to the roots to get an answer?
So the question is: "is there a setting that will reduce that DNS timeout for all running services, maybe like a ping-watchdog that would recognize the problem and skip the reverse-DNS lookup if DNS servers are not reachable?"
How about adding zone files for the things that belong to you? I've always been surprised that distributions that contain name servers don't include dummy reverse zones for all the private IP ranges. It must make a huge load on the DNS infrastructure when everyone passes those queries upstream where they will always fail (but quickly...). You'd get the same answer even faster if you had an empty zone file yourself.
Hi Ljubomir,
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
are you using an external DNS server that is reachable via the internet only?
If so, edit /etc/ssh/sshd_config and change this
#UseDNS yes
to
UseDNS no
Then restart sshd and see whether it still happens. sshd tries to look up its counterpart's host name using DNS in the default setting, and if DNS is not reachable it waits for the request to time out.
Best regards,
Peter.
On 11/21/2012 09:44 AM, Peter Eckel wrote:
Hi Ljubomir,
But when I tried to login to my server, it was not instantenous, and I think it was 15+, maybe even 30+ seconds (I forgot to time it) from start of ssh command to password prompt. It is in-house connection, so there is nothing to traceroute.
are you using an external DNS server that is reachable via the internet only?
If so, edit /etc/ssh/sshd_config and change this
#UseDNS yes
to
UseDNS no
Then restart sshd and see whether it still happens. sshd tries to look up its counterpart's host name using DNS in the default setting, and if DNS is not reachable it waits for the request to time out.
Best regards,
Peter. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
"UseDns no" doesn't solve the problem because GSS Api needs reverse lookup.
If (do not need GSS Authentication) then put "GSSAPIAuthentication no" in the host /etc/ssh/sshd_config else put your server ip and server name in the client /etc/hosts "xxx.xxx.xxxx.xxx yourserver.name" :)
H.can
Hi Hakan,
"UseDns no" doesn't solve the problem because GSS Api needs reverse lookup.
good point. I disable the GSS API along with some other things in routine initial server hardening anyway so this did not occur to me.
If (do not need GSS Authentication) then put "GSSAPIAuthentication no" in the host /etc/ssh/sshd_config else put your server ip and server name in the client /etc/hosts "xxx.xxx.xxxx.xxx yourserver.name" :)
I'd rather put the client IP address in the server's /etc/hosts ...
Best regards,
Peter.
Ljubomir Ljubojevic wrote:
Hi.
For a long time I am puzzled why, when internet connection is gone (beyond first router, outside of subnet) SSH (and telnet?) connection is drastically slowed.
Same behavior, but less impacted is observed when system boots without working internet connection, and I am not sure, but I think even access to SAMBA fileshares can be/is affected. Affected systems are mostly production servers, and too spaced apart in time to be able to point a finger in something I am doing wrong.
I have not had time to gather more facts, I always forget/don;t have time, so I will only focus on definite SSH issue on CentOS 6.3, although I have seen same with 5.x.
1 server in question is set in following maner: [root@chiron ~]# cat /etc/networks default 0.0.0.0 loopback 127.0.0.0 link-local 169.254.0.0
[root@chiron ~]# cat /etc/sysconfig/network NETWORKING=yes NTPSERVERARGS=iburst HOSTNAME=chiron.example.com
[root@chiron ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 chiron.example.com chiron ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 chiron.example.com chiron
Does it have anything to do with /etc/hosts file not having row with used IP address, or is problem somewhere else?
Searching the net gives too much irrelevant links, and all I need it to be pointed in the right direction.
perhaps due to DNS being down?
try starting sshd in debug mode (-d up to -ddd) on the server and connecting while the internet connection is down and see what is being logged.
-
Hakan Can -
Les Mikesell -
Ljubomir Ljubojevic -
m.roth＠5-cent.us -
Nicolas Thierry-Mieg -
Peter Eckel -
Stephen Harris