Simpler Navigation for Servers and Operating Systems - Please Update Your Bookmarks
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
If you have bookmarked forums or discussion boards in Servers and Operating Systems, we suggest you check and update them as needed.
Networking
cancel
Showing results for 
Search instead for 
Did you mean: 

ssh session hangs after successful execution of remote command

akm_1
Occasional Advisor

ssh session hangs after successful execution of remote command

Hi,

I need your help in resolving hung ssh sessions in the hp 11.23 after successful execution of the remote command.

Below is the command used

/usr/bin/ssh -n user@hostname "date"

Below is the stack trace of the above process

-------------------------------- lwpid : 6371279 -------------------------------

0: 60000000c0392210 : _select_sys() + 0x30 (/usr/lib/hpux32/libc.so.1)
1: 60000000c03a6570 : _select() + 0xd0 (/usr/lib/hpux32/libc.so.1)
2: 00000000040d5b00 : client_register_global_confirm() + 0xca0 (/usr/bin/ssh)
3: 00000000040dbb40 : client_loop() + 0x1120 (/usr/bin/ssh)
4: 00000000040c5d00 : main() + 0x65d0 (/usr/bin/ssh)
5: 00000000040c4f70 : main() + 0x5840 (/usr/bin/ssh)
6: 60000000c003c7c0 : main_opd_entry() + 0x50 (/usr/lib/hpux32/dld.so)


Below are the details of the SSH version that we use

$ssh -V
OpenSSH_5.2p1+sftpfilecontrol-v1.3-hpn13v5, OpenSSL 0.9.8k 25 Mar 2009
HP-UX Secure Shell-A.05.20.014, HP-UX Secure Shell version

Any information to fix this is very much appreciated.

Thanks,
13 REPLIES
Shibin_2
Honored Contributor

Re: ssh session hangs after successful execution of remote command

I think, you are using wrong options.

As per man page,

-n Redirects stdin from /dev/null

Why can't you try simply

ssh user@remothost "date" ??

or else use with -l option.
Regards
Shibin
Dennis Handly
Acclaimed Contributor

Re: ssh session hangs after successful execution of remote command

>Below is the stack trace of the above process

Do you have one for the hostname side?

>Shibin: you are using wrong options.
>-n Redirects stdin from /dev/null

-n seems appropriate for a command like date(1) that takes no input.
akm_1
Occasional Advisor

Re: ssh session hangs after successful execution of remote command

Since the process is executed successfully in the remote server, I won't be able to get the stack trace for remote server process.
akm_1
Occasional Advisor

Re: ssh session hangs after successful execution of remote command

I have placed the verbose log of both good and bad ssh. Looks like in the bad log, we are not receiving "rcvd close" from the remote server. Do you have any idea why we didn't receive "close" from remote?

Good
===================
debug1: Sending command: date^M
debug2: channel 0: request exec confirm 1^M
debug2: fd 3 setting TCP_NODELAY^M
debug2: callback done^M
debug2: channel 0: open confirm rwindow 0 rmax 32768^M
debug2: channel 0: rcvd adjust 65536^M
debug2: channel_input_status_confirm: type 99 id 0^M
debug2: exec request accepted on channel 0^M
debug2: channel 0: read<=0 rfd 4 len 0^M
debug2: channel 0: read failed^M
debug2: channel 0: close_read^M
debug2: channel 0: input open -> drain^M
debug2: channel 0: ibuf empty^M
debug2: channel 0: send eof^M
debug2: channel 0: input drain -> closed^M
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0^M
debug2: channel 0: rcvd eof^M
debug2: channel 0: output open -> drain^M
debug2: channel 0: obuf empty^M
debug2: channel 0: close_write^M
debug2: channel 0: output drain -> closed^M
debug2: channel 0: rcvd close^M
debug3: channel 0: will not send data after close^M
debug2: channel 0: almost dead^M
debug2: channel 0: gc: notify user^M
debug2: channel 0: gc: user detached^M
debug2: channel 0: send close^M
debug2: channel 0: is dead^M
debug2: channel 0: garbage collecting^M
debug1: channel 0: free: client-session, nchannels 1^M
debug3: channel 0: status: The following connections are open:^M


Bad
===================
debug1: Sending command: date^M
debug2: channel 0: request exec confirm 1^M
debug2: fd 3 setting TCP_NODELAY^M
debug2: callback done^M
debug2: channel 0: open confirm rwindow 0 rmax 32768^M
debug2: channel 0: rcvd adjust 65536^M
debug2: channel_input_status_confirm: type 99 id 0^M
debug2: exec request accepted on channel 0^M
debug2: channel 0: read<=0 rfd 4 len 0^M
debug2: channel 0: read failed^M
debug2: channel 0: close_read^M
debug2: channel 0: input open -> drain^M
debug2: channel 0: ibuf empty^M
debug2: channel 0: send eof^M
debug2: channel 0: input drain -> closed^M
debug2: channel 0: rcvd eof^M
debug2: channel 0: output open -> drain^M
debug2: channel 0: obuf empty^M
debug2: channel 0: close_write^M
debug2: channel 0: output drain -> closed^M
Doug O'Leary
Honored Contributor

Re: ssh session hangs after successful execution of remote command

Hey;

Just to be clear: have you verified you can actually run the date command on the remote box? it's not aliased to something else - something that may be looking for input?

Second: the ssh "hangs" typically appear when you're trying to run an interactive program (hence the first question) or an X11 gui such as sam. You can try running

ssh -i user@hostname date

to see if anything is displayed on the terminal. the -i tells ssh to set upa psuedo tty for command interaction.

Hope this helps.

Doug O'Leary

------
Senior UNIX Admin
O'Leary Computers Inc
linkedin: http://www.linkedin.com/dkoleary
Resume: http://www.olearycomputers.com/resume.html
Dennis Handly
Acclaimed Contributor

Re: ssh session hangs after successful execution of remote command

>I won't be able to get the stack trace for remote server process.

No sshd process hanging around?
akm_1
Occasional Advisor

Re: ssh session hangs after successful execution of remote command

-i option in ssh is used along with identity file


the sshd version we use in the local server is OpenSSH_5.2 and in remote server is OpenSSH_5.1


root 5522 24595 0 16:05:05 ? 0:00 sshd: netops [priv]
netops 5699 5522 0 16:05:06 ? 0:00 sshd: netops@notty
netops 5713 5699 3 16:05:06 ? 0:00


$ pstack 5699
attach: Permission denied

$ pstack 5713
pstat_getprocvm: Error 0
could not find (static?) executable, cant unwind
attach: Permission denied

akm_1
Occasional Advisor

Re: ssh session hangs after successful execution of remote command

stack trace in remote server

root 5522 24595 0 16:05:05 ? 0:00 sshd: netops [priv]
netops 5699 5522 0 16:05:06 ? 0:00 sshd: netops@notty
netops 5713 5699 3 16:05:06 ? 0:00


$ pstack 5699
attach: Permission denied

$ pstack 5713
pstat_getprocvm: Error 0
could not find (static?) executable, cant unwind
attach: Permission denied

Laurent Menase
Honored Contributor

Re: ssh session hangs after successful execution of remote command

check what version of xport/streams patch you have.
Dennis Handly
Acclaimed Contributor

Re: ssh session hangs after successful execution of remote command

>stack trace in remote server
>attach: Permission denied

You'll need to be root or netops.
akm_1
Occasional Advisor

Re: ssh session hangs after successful execution of remote command

please find below the stream patches in local server

$show_patches -a -s | grep -i stream
PHCO_31677 Japanese streams manpages
PHNE_35482 Streams Pty cumulative patch
PHNE_37258 Cumulative STREAMS Patch
PHNE_32201 Cumulative STREAMS Patch
PHNE_33732 Cumulative STREAMS Patch
PHNE_33899 Streams Pty cumulative patch
PHNE_36575 Cumulative STREAMS Patch


Please find below the stream patches in remote server

$ show_patches -a -s | grep -i stream
PHCO_31677 Japanese streams manpages
PHNE_35482 Streams Pty cumulative patch
PHNE_36575 Cumulative STREAMS Patch
PHNE_32201 Cumulative STREAMS Patch
PHNE_33732 Cumulative STREAMS Patch
PHNE_33899 Streams Pty cumulative patch


If this is not what you expected, please let me know the command I need to execute
akm_1
Occasional Advisor

Re: ssh session hangs after successful execution of remote command

Stack trace in remote server

root 1152 1 0 Jun 8 ? 13:59 /opt/ssh/sbin/sshd
root 29435 1152 0 22:02:07 ? 0:00 sshd: netops [priv]
netops 29601 29435 0 22:02:08 ? 0:00 sshd: netops@notty
netops 29616 29601 3 22:02:08 ? 0:00



root$
# pstack 29616
pstat_getprocvm: Error 0
could not find (static?) executable, cant unwind
attach: Permission denied
29616:


root$
# pstack 29601
29601: /opt/ssh/sbin/sshd

-------------------------------- lwpid : 3505344 -------------------------------

0: c00000000036aff0 : _select_sys() + 0x30 (/usr/lib/hpux64/libc.so.1)
1: c00000000037dca0 : _select() + 0xc0 (/usr/lib/hpux64/libc.so.1)
2: 40000000000f2bd0 : dump_config() + 0x2f50 (/opt/ssh/sbin/sshd)
3: 40000000000f66c0 : server_loop2() + 0x470 (/opt/ssh/sbin/sshd)
4: 4000000000106b30 : do_authenticated() + 0x1e0 (/opt/ssh/sbin/sshd)
5: 4000000000106a80 : do_authenticated() + 0x130 (/opt/ssh/sbin/sshd)
6: 40000000000db460 : main() + 0x5940 (/opt/ssh/sbin/sshd)
7: c000000000042450 : main_opd_entry() + 0x50 (/usr/lib/hpux64/dld.so)
Dennis Handly
Acclaimed Contributor

Re: ssh session hangs after successful execution of remote command

>netops 29616 29601 3 22:02:08

It looks like there is a sshd zombie left over.