topic Re: SSH Child process hanging and cannot be killed in Operating System - HP-UX

SSH Child process hanging and cannot be killed

Rosli Osman — Thu, 26 Apr 2007 19:36:02 GMT

Hi experts,
I have a bunch of HP-UX Itanium boxes pre-installed with OpenSSH_3.7 with defaulted sshd_config. Recently my customer complained that when his user account did a batch SSH login every 10 mins, a child process is spawned. However after the account has logged off, the process still exists.even though the ssh login verbose mode shows the exit is clean (Exit Status 0). This caused performance degradation and may eventually lead to probably login limitation through port 22. After a while I fixed the issue by setting parameter UsePrivilegeSeparation in sshd_config to be a no and now the account does a proper logout and its child process left defunct.
Now, I have 1 concern and 1 problem. My concern is by putting UsePrivilegeSeparation's argument to no, there will be no security against corrupted/malicious privilege escalation. Anyone knows what is the risk & mitigating factors?
My problem is I cannot kill the defunct processes from the previous logins. A reboot will clean up the hanging processes but most being Production boxes, I am looking for alternatives.

Some of the problem's synopsis;
$ ps -ef| grep 7747
root 7747 1 0 Apr 15 ? 0:00 sshd: sascoll [priv]
sascoll 7750 7747 0 Apr 15 ? 0:00 sshd: sascoll@notty

This is just one of the many defunct processes which can't be killed even as root.

Finally, could the problem be a bug in the SSH version? Anyone had this problem before?

Re: SSH Child process hanging and cannot be killed

Rita C Workman — Tue, 22 May 2007 08:15:35 GMT

Hi Rosli,

I'm hitting your thread, cause there were no responses on this.

I'm getting similar on my 11.11 box(s) with Secure Shell 4.20.004 installed.

Searching patch database and so far not finding any concrete answer..........

Hoping someone else might have some insights on this irritation.

Thanks,
Rita

Re: SSH Child process hanging and cannot be killed

Steven E. Protter — Tue, 22 May 2007 08:29:23 GMT

Shalom,

http://software.hp.com
Search: Secure Shell

This could be just bad code. There are security flaws in your ssh version anyway, more than enough reason to update.

SEP

Re: SSH Child process hanging and cannot be killed

Rita C Workman — Tue, 22 May 2007 11:21:02 GMT

Hey Stephen,

Well I'm at 4.20 and they're at 3.7. Only seeing our problem on our 7410 PARisc, while they see it on their Itanium.

Still haven't figured out why........

Thanks,
Rita

Re: SSH Child process hanging and cannot be killed

John Payne_2 — Tue, 22 May 2007 12:47:52 GMT

Rosli,

This UsePrivilegeSeparation setting was introduced with HPUX SSH version 3.10.002. It was done in connection with the security bulletin HPSBUX00195 http://www1.itrc.hp.com/service/cki/docDisplay.do?docId=emr_na-c01001231-1

Having said that, the Sec. Bull. doesn't give you a whole lot to base your decision on, which is normal for these things.

I have seen this happen with ssh, but only with the 'batch' type logins on my machines. The curious thing in your case is that the [priv] process (pid 7747 in your example) has a parent of init. When this happens to me (PA and IA), the parent is always the sshd process itself, not init. I am about to kill the process, either the [priv] process or the no tty process. Is your ssh daemon being restarted on a regular basis or something like that?

If you can get things to start listening to the kill, you can set something up that looks for 'sshd' and 'notty', and kill those off, then toss it into cron.

Of course, you could also open an issue with HP, and complain about the 'feature'.

Hope it helps
John

Re: SSH Child process hanging and cannot be killed

John Payne_2 — Tue, 22 May 2007 12:49:00 GMT

"I am about to kill the process"

should read

"I am able to kill the process"

Sorry.
John

Re: SSH Child process hanging and cannot be killed

Rosli Osman — Wed, 23 May 2007 00:37:20 GMT

Thanks Rita for reigniting this thread.
Yup John, it is strange that the processes are spawned from init (pid 1), instead of sshd itself. This I could not explain.
I could not replicate the issue anymore whether the SSH daemon started everytime a batch login took place (it looks likely though, with all those processes having different pids).
But until today I still could not do a kill to those defunct processes.

Re: SSH Child process hanging and cannot be killed

Andrew Young_2 — Wed, 23 May 2007 01:50:12 GMT

Hi.

We have had a similar problem on some (but not all) of our servers as well which we battled to solve. In our case though the server was a Red Hat Linux server and the client an HP-UX 11i v1 PA-RISC server. It appeared that the client wasn't terminating properly. We replaced both servers before we resolved the issue, and we are no longer experiencing the problem.

Does netstat give any indication of the status of of the connection (like a long TIME_WAIT2 perhaps?)

Regards

Andrew Y

Re: SSH Child process hanging and cannot be killed

John Payne_2 — Wed, 23 May 2007 01:50:52 GMT

You aren't running ssh through inetd are you? That's the only case I can think of where sshd itself could be spawned each time the connection occurs. Otherwise, sshd should just be running all the time, listening for connections. (One of the problems with running sshd from inetd is the key generation that occurs on startup)

Anyway, does that mean you can suddenly kill these procs? Did something change to allow this?

Re: SSH Child process hanging and cannot be killed

Rosli Osman — Wed, 23 May 2007 03:41:55 GMT

I suspect the problem is more system than application (ssh).
I have fixed the initial problem by setting the parameter UsePrivilegeSeparation to no.
And based on feedback from experts, i ought to upgrade to higher version of SSH. That is already in the pipeline.

This leaves only one irritation, as Rita has correctly indicated.

From my example above, I can send kill signal to "priv" process but since pid 7747 is no longer around anymore, the "notty" process will indicate its ppid as 1. This "notty" process cannot be killed even as root or sascoll and I have 87 of such processes in my machine.

netstat does not show any anomalies and sshd is always running at its full path, not from inetd.

Re: SSH Child process hanging and cannot be killed

Rosli Osman — Mon, 18 Jun 2007 12:29:21 GMT

No solution neither any reply. Closing thread

Re: SSH Child process hanging and cannot be killed

Michelle Weiss — Tue, 24 Jul 2007 11:35:58 GMT

just thought I'd check - any update/resolution on this ssh issue?

Re: SSH Child process hanging and cannot be killed

Pierre Pasturel — Tue, 24 Jul 2007 19:25:05 GMT

I was told that this is most likely related to a known problem when a background process is launched from a ssh login session.

Please see the following links for a description of the known problem and some workarounds.

http://www.openssh.com/faq.html#3.10
http://bugzilla.mindrot.org/show_bug.cgi?id=52

Pierre