Hello,
Is anyone else having problems with the ssh session output stalling when ssh'd to another system and the command is running via sudo and a pager is being used (e.g., more)?
Too highly reproducible :( to date, all known occurrences have involved Centos 4.3 servers; makes no difference whether client is RHEL WS 4.3, Centos 4.3 or FC 5 or WinDoze. Makes no difference if terminal emulator is Gnome's stock issue or Konsole or Putty. Not sure about SecureCRT.
Sometimes one can break loose the logjam; if so: . terminal prompts are stacked left to right vs. vertically . keyboard input is not echo'd . in this event, terminal clear/reset doesn't end if so.
Any thoughts?
rgds/ldv
[vaden@mx1 ~]$ ps auxw | grep -i pts/0 vaden 2475 0.0 0.1 37444 3064 ? S 04:53 0:00 sshd: vaden@pts/0 vaden 2476 0.0 0.0 53964 1480 pts/0 Ss 04:53 0:00 -bash root 2511 0.0 0.0 71284 1312 pts/0 S 04:53 0:00 su - root 2512 0.0 0.0 53968 1492 pts/0 S 04:53 0:00 -bash root 3426 0.0 0.0 2376 288 pts/0 S+ 13:57 0:00 sesh /bin/grep -i unknown /var/log/maillog root 3427 0.1 0.0 51128 784 pts/0 S+ 13:57 0:00 /bin/grep -i unknown /var/log/maillog vaden 3434 0.0 0.0 51072 540 pts/1 R+ 13:58 0:00 grep -i pts/0
Is anyone else having problems with the ssh session output stalling when ssh'd to another system and the command is running via sudo and a pager is being used (e.g., more)?
Are you sure you aren't accidentally hitting ctrl-s or scroll lock? Try hitting ctrl-q next time it happens.
Too highly reproducible
What are the steps to reproduce?
j
-- Jason Bradley Nance What would you cry for? aitrus@tresgeek.net Swallow your pride for? Tres Geek What would you go wild for? http://tresgeek.net/ -Poe, "Wild" --
On Sun, 2006-06-25 at 14:20 -0500, Larry Vaden wrote:
Hello,
Is anyone else having problems with the ssh session output stalling when ssh'd to another system and the command is running via sudo and a pager is being used (e.g., more)?
Too highly reproducible :( to date, all known occurrences have involved Centos 4.3 servers; makes no difference whether client is RHEL WS 4.3, Centos 4.3 or FC 5 or WinDoze. Makes no difference if terminal emulator is Gnome's stock issue or Konsole or Putty. Not sure about SecureCRT.
If sudo is removed from the mix, does it still happen?
Sometimes one can break loose the logjam; if so: . terminal prompts are stacked left to right vs. vertically . keyboard input is not echo'd
These two tell us that end-of-line and other terminal settings (usually NL on *IX systems, but I don't know about ssh) are getting lost before the pager is seeing them. Possibly doing stty against the pseudo-terms involved will give a clue?
. in this event, terminal clear/reset doesn't end if so.
Any thoughts?
Generally, having a pager involved when there are interactive prompts ("terminal prompts are stacked...") is a bad idea. Commands often issue prompts with no EOL so that you see on one line something like
Enter reply [Y/N/Q] -
and the cursor is after the dash. The pager usually works on *lines* not *characters*, although some used to have timeouts that could be applied to force output flushing (don't know if this applies anymore).
Wish I had more to offer than just a couple Qs that might get you going.
rgds/ldv
[vaden@mx1 ~]$ ps auxw | grep -i pts/0 vaden 2475 0.0 0.1 37444 3064 ? S 04:53 0:00 sshd: vaden@pts/0 vaden 2476 0.0 0.0 53964 1480 pts/0 Ss 04:53 0:00 -bash root 2511 0.0 0.0 71284 1312 pts/0 S 04:53 0:00 su - root 2512 0.0 0.0 53968 1492 pts/0 S 04:53 0:00 -bash root 3426 0.0 0.0 2376 288 pts/0 S+ 13:57 0:00 sesh /bin/grep -i unknown /var/log/maillog root 3427 0.1 0.0 51128 784 pts/0 S+ 13:57 0:00 /bin/grep -i unknown /var/log/maillog vaden 3434 0.0 0.0 51072 540 pts/1 R+ 13:58 0:00 grep -i pts/0
<snip sig stuff>
On 6/26/06, William L. Maltby BillsCentOS@triad.rr.com wrote:
If sudo is removed from the mix, does it still happen?
Dunno, but I'll try removing sudo from the mix.
Generally, having a pager involved when there are interactive prompts ("terminal prompts are stacked...") is a bad idea.
The original example published _inadvertently_ didn't have a pager in the pipe.
To my knowledge, the hangs have all been while watching logs which don't exhibit interactive prompts (other than paging more, and more is not always involved); examples are:
sudo grep -i unknown /var/log/maillog | more sudo tail -F /var/log/maillog | grep -i "yada1 yada2"
rgds/ldv
On Mon, 2006-06-26 at 09:30 -0500, Larry Vaden wrote:
On 6/26/06, William L. Maltby BillsCentOS@triad.rr.com wrote:
If sudo is removed from the mix, does it still happen?
Dunno, but I'll try removing sudo from the mix.
<snip>
To my knowledge, the hangs have all been while watching logs which don't exhibit interactive prompts (other than paging more, and more is not always involved); examples are:
sudo grep -i unknown /var/log/maillog | more sudo tail -F /var/log/maillog | grep -i "yada1 yada2"
Sounds like isolation of the problem source is in order. Since a pager is not always involved, maybe a monitor of the involved processes to see which one becomes blocked and what it is waiting for. Ah! Some things just came to mind.
Don't know if it is related, but we recently had a discussion of swapping here. One of the symptoms when it got excessive was almost non- existent response (in some cases). Have you run a top or other process to see if swap usage has jumped up when the problems occur? If "less" is one of the potential culprits, the -B option might help. Seems unlikely since you said a pager is not always involved.
From rom "man more"
-l more usually treats ^L (form feed) as a special character, and will pause after any line that contains a form feed. The -l option will prevent this behavior.
Again, seems unlikely.
Regardless, with the right field displays enabled in top (or other monitoring processes), maybe a blocked process will appear.
Another possibility. Whenever there is a "pipe", there is (by default?) some buffering, IIRC. Someone correct me if I'm wrong here! Further, more and tail normally operate on lines. Your mention of "stacked prompts" made me think of CR/LF problems. Even if there are no stacked prompts, could it be there are missing or deferred CR/LF characters? This would be common if a response to an error message was expected by one of the applications or waiting for some input, like more or less. If the physical connection losing characters? E.g. dial-up modem without flow control on one or both ends? Ssh connection vi UDP, which is "connectionless", advertised as "unreliable" and not the usual mode?
If you put the outputs into a log file and then tail or more them, what are the results? Have you tried running the commands without SSH involvement?
rgds/ldv
<snip sig stuff>
HTH
On 26/06/06, William L. Maltby BillsCentOS@triad.rr.com wrote:
If you put the outputs into a log file and then tail or more them, what are the results? Have you tried running the commands without SSH involvement?
It might also be worth considering the use of Screen under these circumstances. He'll be able to reconnect and resume the session, assuming it is the SSH connection that's dropping and not the local process locking up somehow.
https://savannah.gnu.org/projects/screen/
http://gentoo-wiki.com/TIP_Using_screen
(For more see: http://gentoo-wiki.com/TIP_Using_screen )
Will.
On 26/06/06, Will McDonald wmcdonald@gmail.com wrote:
On 26/06/06, William L. Maltby BillsCentOS@triad.rr.com wrote:
If you put the outputs into a log file and then tail or more them, what are the results? Have you tried running the commands without SSH involvement?
It might also be worth considering the use of Screen under these circumstances. He'll be able to reconnect and resume the session, assuming it is the SSH connection that's dropping and not the local process locking up somehow.
https://savannah.gnu.org/projects/screen/
http://gentoo-wiki.com/TIP_Using_screen
(For more see: http://gentoo-wiki.com/TIP_Using_screen )
Oops, 2nd link should've been...
http://www.google.co.uk/search?hl=en&q=using+screen&btnG=Search&...
On Mon, 2006-06-26 at 16:12 +0100, Will McDonald wrote:
On 26/06/06, Will McDonald wmcdonald@gmail.com wrote:
On 26/06/06, William L. Maltby BillsCentOS@triad.rr.com wrote:
If you put the outputs into a log file and then tail or more them, what are the results? Have you tried running the commands without SSH involvement?
It might also be worth considering the use of Screen under these circumstances. He'll be able to reconnect and resume the session, assuming it is the SSH connection that's dropping and not the local process locking up somehow.
Good idea. I was even going to suggest using something besides SH just to see what happened. Regardless, I see there is also this param to SSH (also in the config file) that may have some bearing?
-o ConnectTimeout
That made me wonder if network congestion might have a role? If results are inconsistent, as indicated IIRC, then congestion of *some* sort seems a likely culprit.
https://savannah.gnu.org/projects/screen/
http://gentoo-wiki.com/TIP_Using_screen
(For more see: http://gentoo-wiki.com/TIP_Using_screen )
Oops, 2nd link should've been...
http://www.google.co.uk/search?hl=en&q=using+screen&btnG=Search&... _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 6/26/06, Will McDonald wmcdonald@gmail.com wrote:
It might also be worth considering the use of Screen under these circumstances. He'll be able to reconnect and resume the session, assuming it is the SSH connection that's dropping and not the local process locking up somehow.
Thanks for the suggestion.
The SOP has been changed to incorporate the suggestion as follows:
1. SSH to the destination 2. At the destination, invoke screen
The failure modality should be interesting.
rgds/ldv
-
Jason Bradley Nance -
Larry Vaden -
Will McDonald -
William L. Maltby