topic Re: ssh session hangs after successful execution of remote command in Operating System - HP-UX

ssh session hangs after successful execution of remote command

akm_1 — Mon, 03 Jan 2011 23:23:20 GMT

Hi,

I need your help in resolving hung ssh sessions in the hp 11.23 after successful execution of the remote command.

Below is the command used

/usr/bin/ssh -n user@hostname "date"

Below is the stack trace of the above process

-------------------------------- lwpid : 6371279 -------------------------------

0: 60000000c0392210 : _select_sys() + 0x30 (/usr/lib/hpux32/libc.so.1)
1: 60000000c03a6570 : _select() + 0xd0 (/usr/lib/hpux32/libc.so.1)
2: 00000000040d5b00 : client_register_global_confirm() + 0xca0 (/usr/bin/ssh)
3: 00000000040dbb40 : client_loop() + 0x1120 (/usr/bin/ssh)
4: 00000000040c5d00 : main() + 0x65d0 (/usr/bin/ssh)
5: 00000000040c4f70 : main() + 0x5840 (/usr/bin/ssh)
6: 60000000c003c7c0 : main_opd_entry() + 0x50 (/usr/lib/hpux32/dld.so)

Below are the details of the SSH version that we use

$ssh -V
OpenSSH_5.2p1+sftpfilecontrol-v1.3-hpn13v5, OpenSSL 0.9.8k 25 Mar 2009
HP-UX Secure Shell-A.05.20.014, HP-UX Secure Shell version

Any information to fix this is very much appreciated.

Thanks,

Re: ssh session hangs after successful execution of remote command

Shibin_2 — Tue, 04 Jan 2011 01:13:05 GMT

I think, you are using wrong options.

As per man page,

-n Redirects stdin from /dev/null

Why can't you try simply

ssh user@remothost "date" ??

or else use with -l option.

Re: ssh session hangs after successful execution of remote command

Dennis Handly — Tue, 04 Jan 2011 13:26:56 GMT

>Below is the stack trace of the above process

Do you have one for the hostname side?

>Shibin: you are using wrong options.
>-n Redirects stdin from /dev/null

-n seems appropriate for a command like date(1) that takes no input.

Re: ssh session hangs after successful execution of remote command

akm_1 — Tue, 04 Jan 2011 16:56:18 GMT

Since the process is executed successfully in the remote server, I won't be able to get the stack trace for remote server process.

Re: ssh session hangs after successful execution of remote command

akm_1 — Tue, 04 Jan 2011 17:14:30 GMT

I have placed the verbose log of both good and bad ssh. Looks like in the bad log, we are not receiving "rcvd close" from the remote server. Do you have any idea why we didn't receive "close" from remote?

Good
===================
debug1: Sending command: date^M
debug2: channel 0: request exec confirm 1^M
debug2: fd 3 setting TCP_NODELAY^M
debug2: callback done^M
debug2: channel 0: open confirm rwindow 0 rmax 32768^M
debug2: channel 0: rcvd adjust 65536^M
debug2: channel_input_status_confirm: type 99 id 0^M
debug2: exec request accepted on channel 0^M
debug2: channel 0: read<=0 rfd 4 len 0^M
debug2: channel 0: read failed^M
debug2: channel 0: close_read^M
debug2: channel 0: input open -> drain^M
debug2: channel 0: ibuf empty^M
debug2: channel 0: send eof^M
debug2: channel 0: input drain -> closed^M
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0^M
debug2: channel 0: rcvd eof^M
debug2: channel 0: output open -> drain^M
debug2: channel 0: obuf empty^M
debug2: channel 0: close_write^M
debug2: channel 0: output drain -> closed^M
debug2: channel 0: rcvd close^M
debug3: channel 0: will not send data after close^M
debug2: channel 0: almost dead^M
debug2: channel 0: gc: notify user^M
debug2: channel 0: gc: user detached^M
debug2: channel 0: send close^M
debug2: channel 0: is dead^M
debug2: channel 0: garbage collecting^M
debug1: channel 0: free: client-session, nchannels 1^M
debug3: channel 0: status: The following connections are open:^M

Bad
===================
debug1: Sending command: date^M
debug2: channel 0: request exec confirm 1^M
debug2: fd 3 setting TCP_NODELAY^M
debug2: callback done^M
debug2: channel 0: open confirm rwindow 0 rmax 32768^M
debug2: channel 0: rcvd adjust 65536^M
debug2: channel_input_status_confirm: type 99 id 0^M
debug2: exec request accepted on channel 0^M
debug2: channel 0: read<=0 rfd 4 len 0^M
debug2: channel 0: read failed^M
debug2: channel 0: close_read^M
debug2: channel 0: input open -> drain^M
debug2: channel 0: ibuf empty^M
debug2: channel 0: send eof^M
debug2: channel 0: input drain -> closed^M
debug2: channel 0: rcvd eof^M
debug2: channel 0: output open -> drain^M
debug2: channel 0: obuf empty^M
debug2: channel 0: close_write^M
debug2: channel 0: output drain -> closed^M

Re: ssh session hangs after successful execution of remote command

Doug O'Leary — Tue, 04 Jan 2011 17:52:32 GMT

Hey;

Just to be clear: have you verified you can actually run the date command on the remote box? it's not aliased to something else - something that may be looking for input?

Second: the ssh "hangs" typically appear when you're trying to run an interactive program (hence the first question) or an X11 gui such as sam. You can try running

ssh -i user@hostname date

to see if anything is displayed on the terminal. the -i tells ssh to set upa psuedo tty for command interaction.

Hope this helps.

Doug O'Leary

Re: ssh session hangs after successful execution of remote command

Dennis Handly — Tue, 04 Jan 2011 20:16:47 GMT

>I won't be able to get the stack trace for remote server process.

No sshd process hanging around?

Re: ssh session hangs after successful execution of remote command

akm_1 — Tue, 04 Jan 2011 21:25:29 GMT

-i option in ssh is used along with identity file

the sshd version we use in the local server is OpenSSH_5.2 and in remote server is OpenSSH_5.1

root 5522 24595 0 16:05:05 ? 0:00 sshd: netops [priv]
netops 5699 5522 0 16:05:06 ? 0:00 sshd: netops@notty
netops 5713 5699 3 16:05:06 ? 0:00

$ pstack 5699
attach: Permission denied

$ pstack 5713
pstat_getprocvm: Error 0
could not find (static?) executable, cant unwind
attach: Permission denied

Re: ssh session hangs after successful execution of remote command

akm_1 — Tue, 04 Jan 2011 21:26:07 GMT

stack trace in remote server

root 5522 24595 0 16:05:05 ? 0:00 sshd: netops [priv]
netops 5699 5522 0 16:05:06 ? 0:00 sshd: netops@notty
netops 5713 5699 3 16:05:06 ? 0:00

$ pstack 5699
attach: Permission denied

$ pstack 5713
pstat_getprocvm: Error 0
could not find (static?) executable, cant unwind
attach: Permission denied

Re: ssh session hangs after successful execution of remote command

Laurent Menase — Wed, 05 Jan 2011 00:57:02 GMT

check what version of xport/streams patch you have.

Re: ssh session hangs after successful execution of remote command

Dennis Handly — Wed, 05 Jan 2011 06:01:40 GMT

>stack trace in remote server
>attach: Permission denied

You'll need to be root or netops.

Re: ssh session hangs after successful execution of remote command

akm_1 — Wed, 05 Jan 2011 16:11:42 GMT

please find below the stream patches in local server

$show_patches -a -s | grep -i stream
PHCO_31677 Japanese streams manpages
PHNE_35482 Streams Pty cumulative patch
PHNE_37258 Cumulative STREAMS Patch
PHNE_32201 Cumulative STREAMS Patch
PHNE_33732 Cumulative STREAMS Patch
PHNE_33899 Streams Pty cumulative patch
PHNE_36575 Cumulative STREAMS Patch

Please find below the stream patches in remote server

$ show_patches -a -s | grep -i stream
PHCO_31677 Japanese streams manpages
PHNE_35482 Streams Pty cumulative patch
PHNE_36575 Cumulative STREAMS Patch
PHNE_32201 Cumulative STREAMS Patch
PHNE_33732 Cumulative STREAMS Patch
PHNE_33899 Streams Pty cumulative patch

If this is not what you expected, please let me know the command I need to execute

Re: ssh session hangs after successful execution of remote command

akm_1 — Wed, 05 Jan 2011 16:40:08 GMT

Stack trace in remote server

root 1152 1 0 Jun 8 ? 13:59 /opt/ssh/sbin/sshd
root 29435 1152 0 22:02:07 ? 0:00 sshd: netops [priv]
netops 29601 29435 0 22:02:08 ? 0:00 sshd: netops@notty
netops 29616 29601 3 22:02:08 ? 0:00

root$
# pstack 29616
pstat_getprocvm: Error 0
could not find (static?) executable, cant unwind
attach: Permission denied
29616:

root$
# pstack 29601
29601: /opt/ssh/sbin/sshd

-------------------------------- lwpid : 3505344 -------------------------------

0: c00000000036aff0 : _select_sys() + 0x30 (/usr/lib/hpux64/libc.so.1)
1: c00000000037dca0 : _select() + 0xc0 (/usr/lib/hpux64/libc.so.1)
2: 40000000000f2bd0 : dump_config() + 0x2f50 (/opt/ssh/sbin/sshd)
3: 40000000000f66c0 : server_loop2() + 0x470 (/opt/ssh/sbin/sshd)
4: 4000000000106b30 : do_authenticated() + 0x1e0 (/opt/ssh/sbin/sshd)
5: 4000000000106a80 : do_authenticated() + 0x130 (/opt/ssh/sbin/sshd)
6: 40000000000db460 : main() + 0x5940 (/opt/ssh/sbin/sshd)
7: c000000000042450 : main_opd_entry() + 0x50 (/usr/lib/hpux64/dld.so)

Re: ssh session hangs after successful execution of remote command

Dennis Handly — Wed, 05 Jan 2011 17:23:41 GMT

>netops 29616 29601 3 22:02:08

It looks like there is a sshd zombie left over.