OpenSSH v.4.5p1 -HP-UX11i - Server refused to allocate pty.

[Sharad Saraswat]

Hello Everyone,

Recently someone had changed the file ownership (recursively) on system critical partitions. Though we got them restored from backup, but since then there has been a problem logging on with SSH. When connecting over putty, I see this message "Server refused to allocate pty". After this message, I see garbled text as profile is executed and then cursor stays without giving me login prompt.

Has anyone seen this issue before. I saw one thread which mentions about kernel parameter nstrpty, which is 60 and there are not many sessions established to the host to reach that count.

Thanks in advance.

Sharad Saraswat

[TY 007]

Hello Sharad,

Increase Kernel Parameter Value of 'npty' - Number of ptys (Pseudo ttys)

Thanks

[TY 007]

Hello Sharad,

Install Latest HP-UX Secure Shell (HP Version):
https://h20293.www2.hp.com/portal/swdepot/try.do?productNumber=T1471AA

Thanks

[J. Bravo]

Hello!

Have you tried with other application that supports SSH (like Xmanager)?, and with telnet?

Could you please send us all of the output error information?

Regards;

J. Bravo.

[Sharad Saraswat]

Here is the error I see in syslog.log for my id when I try to authenticate over SSH. Telnet works okay. I have tried that version of SSH as well but issue still persists.

Oct 1 05:24:17 myhost sshd[7488]: Accepted password for az8793 from 143.5.68.250 port 4782 ssh2
Oct 1 05:24:18 myhost sshd[7488]: error: openpty: Bad file number
Oct 1 05:24:18 myhost sshd[7488]: error: openpty: Bad file number
Oct 1 05:24:18 myhost sshd[7488]: error: session_pty_req: session 0 alloc failed
Oct 1 05:24:18 myhost sshd[7488]: error: session_pty_req: session 0 alloc failed
Oct 1 05:25:01 myhost sshd[7626]: Accepted password for az8793 from 143.5.68.250 port 4784 ssh2

Some more info:
# kmtune | grep -e npty -e nstrpty -e nstrtel
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60

# ps -ef | egrep "pty|pts|tty"
root 1117 1 0 Sep 16 ? 0:00 /usr/sbin/ptydaemon
az8793 11525 11524 0 05:28:44 pts/tb 0:00 -sh
root 13823 11604 3 05:45:47 pts/tb 0:00 egrep pty|pts|tty
root 11524 1485 0 05:28:44 pts/tb 0:00 telnetd -b /etc/issue
root 13822 11604 10 05:45:47 pts/tb 0:00 ps -ef
root 11604 11525 0 05:28:54 pts/tb 0:00 -sh

[J. Bravo]

Hello:

Could you start the sshd daemon in debug mode (-d3)? And then send us the log, please.

Regards;

J. Bravo.

[Sharad Saraswat]

Hello,

Please find attached the debug from client and server sides in zip file. The data was too much to post.

Thanks so much for your help in looking into this.

Best Regards,
Sharad Saraswat

[J. Bravo]

Hello:

Could you check if you have enabled "PAMAuthenticationViaKbdInt yes" in sshd_config?

Regards;

J. Bravo.

[Sharad Saraswat]

It was not enabled so I added the token in sshd_config and bounced it. I got this when I started it:

Deprecated option PAMAuthenticationViaKbdInt

It did not work though. Since (may be) this also used PAM so I enabled PAM as well by using this token UsePAM yes and bounced sshd.

Same results.

[Sharad Saraswat]

It was not enabled so I added the token in sshd_config and bounced it. I got this when I started it.

Deprecated option PAMAuthenticationViaKbdInt

It did not work though. Since (may be) this also used PAM so I enabled PAM as well by using this token UsePAM yes and bounced sshd.

Same results.

[J. Bravo]

Hello:

Do you have checked that all of the /dev/tty* files exist in your server and that the permissions are well?

Regards;

J. Bravo.