ssh 3.6.1 leaves 100% CPU sh processes

[Simon_62]

We're running OpenSSH3.6.1 on several HP-UX 11i development machines and although it generally works fine, sometimes (once every couple of days) sshd leaves a parentless sh process that takes 80-100% CPU for eternity (or until you kill it:) (see attachment).
Does anyone have an idea what can be wrong here ?

SSH :
tag T1471AA
software_spec T1471AA,r=A.03.61.002,a=HP-UX_B.11.11_32/64,v=HP
data_model_revision 2.40
instance_id 1
control_directory T1471AA
size 20152906
revision A.03.61.002
title HP-UX Secure Shell
description "HP-UX Secure Shell product, based on OpenSSH"

[Stefan Farrelly]

We run this same version of ssh on many 11i servers and have never ever seen a runaway ssh process like yours. I would thus suggest its something unique to whom/how connects to your hp servers using ssh, theyre possibly not disconnecting properly.

Are all your remote ssh connections from hp servers using the same version, or from pc's - if from pc's what is the version off ssh theyre running - possibly a conflict.

I would investigate who connects in using ssh and watch all ssh connections to see if its the same user/users and find out how they disconnect which could be causing your runaway.

[Simon_62]

all client connections are done from Windows NT4 workstations using Putty 0.53. I don't think most users will gracefully exit the session, but instead just click the close button in Putty to end the connection.

[Stefan Farrelly]

I would certainly think this is where your problem is then - using an old (NT4 !) and cheap (free?) ssh client and not disconnecting properly. There may be some settings on your NT boxes you can change to try to get them to disconnect properly or invest in a better client which when you click on X does close properly, like F-Secure SSH client which we use on Win2K/XP

[Ron Kinner]

There is a known bug in NT's TCP/IP stack where if a one of the FIN packets get lost that NT will not resend. This leaves the otherside stuck in FIN_WAIT or FIN_WAIT_II and the NT box stuck in LAST ACK. netstat -an |grep WAIT will show it on the Unix side. netstat -an on the NT side.

There is a patch from MS but it is not in general release which means you have to beg them for it. I got the patch from them and stuck it on my website if you want to try it. http://www.geocities.com/heardland/9670

It's down at the bottom listed as TCP/IP patch. IT replaces the file tcpip.sys.

You can also adjust the finwaitII timeout in HPUX using ndd.

Ron

[Andrew Cowan]

If you suspect a FIN_WAIT problem then consult "netstat -a". Also try running sshd in debug mode and see what happens when you connect and disconnect from your session.
I would suggest doing it once from another HP-UX box, then from Windows and compare the output.

[James Lynch]

Some of the older versions of the shell have had similar type of problems. I would make sure that you have all of the latest patches regarding the shell.

JL

[Simon_62]

I'm gonna try your suggestions, starting by setting the FIN_WAIT_2 timeout to something not indefinate (=default) as I can't do anything about patching the NT workstations (we're moving to XP in a few months anyway) I've set the timeout to be 60 seconds now on 2 of our servers and will monitor it for the coming days.