SSH Child process hanging and cannot be killed

[Rosli Osman]

Hi experts,
I have a bunch of HP-UX Itanium boxes pre-installed with OpenSSH_3.7 with defaulted sshd_config. Recently my customer complained that when his user account did a batch SSH login every 10 mins, a child process is spawned. However after the account has logged off, the process still exists.even though the ssh login verbose mode shows the exit is clean (Exit Status 0). This caused performance degradation and may eventually lead to probably login limitation through port 22. After a while I fixed the issue by setting parameter UsePrivilegeSeparation in sshd_config to be a no and now the account does a proper logout and its child process left defunct.
Now, I have 1 concern and 1 problem. My concern is by putting UsePrivilegeSeparation's argument to no, there will be no security against corrupted/malicious privilege escalation. Anyone knows what is the risk & mitigating factors?
My problem is I cannot kill the defunct processes from the previous logins. A reboot will clean up the hanging processes but most being Production boxes, I am looking for alternatives.

Some of the problem's synopsis;
$ ps -ef| grep 7747
root 7747 1 0 Apr 15 ? 0:00 sshd: sascoll [priv]
sascoll 7750 7747 0 Apr 15 ? 0:00 sshd: sascoll@notty

This is just one of the many defunct processes which can't be killed even as root.

Finally, could the problem be a bug in the SSH version? Anyone had this problem before?

[Rita C Workman]

Hi Rosli,

I'm hitting your thread, cause there were no responses on this.

I'm getting similar on my 11.11 box(s) with Secure Shell 4.20.004 installed.

Searching patch database and so far not finding any concrete answer..........

Hoping someone else might have some insights on this irritation.

Thanks,
Rita

[Steven E. Protter]

Shalom,

http://software.hp.com
Search: Secure Shell

This could be just bad code. There are security flaws in your ssh version anyway, more than enough reason to update.

SEP

[Rita C Workman]

Hey Stephen,

Well I'm at 4.20 and they're at 3.7. Only seeing our problem on our 7410 PARisc, while they see it on their Itanium.

Still haven't figured out why........

Thanks,
Rita

[John Payne_2]

Rosli,

This UsePrivilegeSeparation setting was introduced with HPUX SSH version 3.10.002. It was done in connection with the security bulletin HPSBUX00195 http://www1.itrc.hp.com/service/cki/docDisplay.do?docId=emr_na-c01001231-1

Having said that, the Sec. Bull. doesn't give you a whole lot to base your decision on, which is normal for these things.

I have seen this happen with ssh, but only with the 'batch' type logins on my machines. The curious thing in your case is that the [priv] process (pid 7747 in your example) has a parent of init. When this happens to me (PA and IA), the parent is always the sshd process itself, not init. I am about to kill the process, either the [priv] process or the no tty process. Is your ssh daemon being restarted on a regular basis or something like that?

If you can get things to start listening to the kill, you can set something up that looks for 'sshd' and 'notty', and kill those off, then toss it into cron.

Of course, you could also open an issue with HP, and complain about the 'feature'.

Hope it helps
John

[John Payne_2]

"I am about to kill the process"

should read

"I am able to kill the process"

Sorry.
John

[Rosli Osman]

Thanks Rita for reigniting this thread.
Yup John, it is strange that the processes are spawned from init (pid 1), instead of sshd itself. This I could not explain.
I could not replicate the issue anymore whether the SSH daemon started everytime a batch login took place (it looks likely though, with all those processes having different pids).
But until today I still could not do a kill to those defunct processes.

[Andrew Young_2]

Hi.

We have had a similar problem on some (but not all) of our servers as well which we battled to solve. In our case though the server was a Red Hat Linux server and the client an HP-UX 11i v1 PA-RISC server. It appeared that the client wasn't terminating properly. We replaced both servers before we resolved the issue, and we are no longer experiencing the problem.

Does netstat give any indication of the status of of the connection (like a long TIME_WAIT2 perhaps?)

Regards

Andrew Y

[John Payne_2]

You aren't running ssh through inetd are you? That's the only case I can think of where sshd itself could be spawned each time the connection occurs. Otherwise, sshd should just be running all the time, listening for connections. (One of the problems with running sshd from inetd is the key generation that occurs on startup)

Anyway, does that mean you can suddenly kill these procs? Did something change to allow this?

[Rosli Osman]

I suspect the problem is more system than application (ssh).
I have fixed the initial problem by setting the parameter UsePrivilegeSeparation to no.
And based on feedback from experts, i ought to upgrade to higher version of SSH. That is already in the pipeline.

This leaves only one irritation, as Rita has correctly indicated.

From my example above, I can send kill signal to "priv" process but since pid 7747 is no longer around anymore, the "notty" process will indicate its ppid as 1. This "notty" process cannot be killed even as root or sascoll and I have 87 of such processes in my machine.

netstat does not show any anomalies and sshd is always running at its full path, not from inetd.

[Rosli Osman]

No solution neither any reply. Closing thread

[Michelle Weiss]

just thought I'd check - any update/resolution on this ssh issue?

[Pierre Pasturel]

I was told that this is most likely related to a known problem when a background process is launched from a ssh login session.

Please see the following links for a description of the known problem and some workarounds.

http://www.openssh.com/faq.html#3.10
http://bugzilla.mindrot.org/show_bug.cgi?id=52

Pierre