topic Re: SSH Weirdness in Operating System - HP-UX

SSH Weirdness

Margaret Sanford — Thu, 02 Apr 2009 17:35:14 GMT

I am running an HP 3440 Server on HP 11i Version 1.

About a week ago our ssh sessions stopped enforcing the inactivity time out. I thought it was weird, but I didn't have a chance to look into it. Then on Wednesday night the system had a catastrophic failure and we had to reboot it from the ILO. HP said the error messages they were seeing were consistent with an ssh failure and recommended upgrading to version 8.

Does this sound familiar to anybody?

Thanks in advance,

Margaret

Re: SSH Weirdness

Steven E. Protter — Thu, 02 Apr 2009 17:52:22 GMT

Shalom,

The details no. The overall situation yes. The system had troubles, showed signs, asked for attention and did not get it.

Please post exact error messages.

seh is constantly being updated due to critical flaws and security holes. Updating it is a very good idea.

SEP

Re: SSH Weirdness

Margaret Sanford — Thu, 02 Apr 2009 17:56:43 GMT

Here's the errors spewing accross the console:

Apr 1 03:30:38 dsbb inetd[1002]: Problem with /var/adm/inetd.sec:
cannot starts
Apr 1 03:30:44 dsbb prngd[19354]: Failed to execl(/usr/bin/ps, ps, -ex, , , ):r

Apr 1 03:31:35 dsbb prngd[19361]: Failed to execl(/usr/bin/who, who, am, i, , r

Apr 1 03:32:26 dsbb prngd[19364]: Failed to execl(/usr/bin/vmstat, vmstat, , ,r

Apr 1 03:33:17 dsbb prngd[19366]: Failed to execl(/usr/bin/tail, tail, -100, /r

Apr 1 03:34:08 dsbb prngd[19373]: Failed to execl(/usr/bin/tail, tail, -100, /r Apr 1 03:34:59 dsbb prngd[19376]: Failed to execl(/usr/bin/tail, tail, -100, /r Apr 1 03:35:51 dsbb prngd[19380]:
Failed to execl(/usr/bin/ls, ls, -alni, /devr Apr 1 03:36:42 dsbb
prngd[19383]: Failed to execl(/usr/bin/ls, ls, -alni, /varr Apr 1
03:37:33 dsbb prngd[19386]: Failed to execl(/usr/bin/ls, ls, -alni, /varr

Apr 1 03:38:24 dsbb prngd[19389]: Failed to execl(/usr/bin/ls, ls, -alni, /usrr Apr 1 03:39:15 dsbb prngd[19391]: Failed to execl(/usr/bin/ls, ls, -alni, /usrr Apr 1 03:40:06 dsbb prngd[19398]:
Failed to execl(/usr/bin/ls, ls, -alni, /tmpr

Here's what was in the syslog when it started:

Apr 1 02:58:12 dsbb sshd[18589]: error: PAM: Authentication failed for blackbrt Apr 1 02:58:20 dsbb above message repeats 2 times Apr 1 02:58:20 dsbb sshd[18613]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 02:58:21 dsbb sshd[18589]: Accepted password for blackbrd from
66.210.592
Apr 1 02:58:21 dsbb sshd[18630]: subsystem request for sftp Apr 1 02:58:40 dsbb sshd[18689]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 02:59:29 dsbb sshd[18727]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 02:59:43 dsbb su: + 0 blackbrd-harco Apr 1 03:00:43 dsbb sshd[18794]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 03:00:51 dsbb su: + 1 blackbrd-harco Apr 1 03:03:35 dsbb sshd[18936]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 03:03:49 dsbb su: + 1 blackbrd-harco Apr 1 03:03:51 dsbb sshd[18969]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:03:58 dsbb sshd[18994]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:04:15 dsbb sshd[19068]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:08:18 dsbb vmunix: msgcnt 1 vxfs: mesg 003: vx_mapbad - /dev/root fild Apr 1 03:08:18 dsbb vmunix: msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/root fi) Apr 1 03:09:42 dsbb vmunix: msgcnt 4 vxfs: mesg 037: vx_metaioerr -
vx_tflush_2
Apr 1 03:09:42 dsbb vmunix: msgcnt 5 vxfs: mesg 031: vx_disable - /dev/root fid Apr 1 03:10:20 dsbb prngd[19234]: Failed to execl(/usr/bin/ls, ls, -alti, /varr Apr 1 03:10:20 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:11:11 dsbb prngd[19238]: Failed to execl(/usr/bin/ls, ls, -alti, /etcr Apr 1 03:11:11 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:12:02 dsbb prngd[19249]: Failed to execl(/usr/bin/ls, ls, -alti, /tmpr Apr 1 03:12:02 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:12:53 dsbb prngd[19255]: Failed to execl(/usr/bin/ls, ls, -alti, /varr Standard input

Re: SSH Weirdness

Kris Knigga — Thu, 02 Apr 2009 18:41:18 GMT

Looks like it might be a disk (or file system) issue. I'm seeing a lot of references to vxfs.

vx_mapbad - /dev/root
vx_nospace - /dev/root
vx_metaioerr
vx_disable - /dev/root

Re: SSH Weirdness

Margaret Sanford — Thu, 02 Apr 2009 18:44:02 GMT

That's what I thought too... but HP said that was not the issue. All the volumes came up cleanly when we rebooted with the FSCK that ran.

Weird.

Re: SSH Weirdness

Kris Knigga — Thu, 02 Apr 2009 18:52:02 GMT

Was a quick fsck done, or a full?

Re: SSH Weirdness

Margaret Sanford — Thu, 02 Apr 2009 18:52:46 GMT

quick

Re: SSH Weirdness

Kris Knigga — Thu, 02 Apr 2009 19:01:21 GMT

I would want to see what `fsck -o full,nolog` says. I've never known good things coming from ignoring vxfs errors like that, especially when the machine crashes shortly after they start.

The only problem is, I'm guessing that since this is the root volume group, you can't unmount anything to try this and I can't think of any way to cause a full fsck on next boot.

Re: SSH Weirdness

Margaret Sanford — Thu, 02 Apr 2009 19:06:27 GMT

Interesting. This server is difficult to bring down for any period of time, so I'll schedule an FSCK asap.

Does anybody have any insight into why the inactive time out period just went away?

I'm not sure where to check if that's still enabled.

-M

Re: SSH Weirdness

Kris Knigga — Thu, 02 Apr 2009 19:16:06 GMT

Looks like you'd have to reboot to single user mode to be able to fsck things like /var properly (while they are unmounted).

I'm leaning toward there being two issues here; one being the SSH stuff and the other being the crash.

How was the timeout set up before? Was it truly a function of SSH, or was it done at the shell level (by setting TMOUT or something)?

Re: SSH Weirdness

Margaret Sanford — Thu, 02 Apr 2009 19:18:11 GMT

I think it was at the shell level. But it was one of those "It was like that when I got here!" things...

But I don't know how to tell. If that parameter is set and it's not timing me out then something in ssh is hosed.

Re: SSH Weirdness

Heironimus — Fri, 03 Apr 2009 14:21:51 GMT

You should make sure you aren't missing any JFS-related patches, but that log really, REALLY looks like your system blew up because it got an I/O error on the root disk.

Re: SSH Weirdness

Margaret Sanford — Fri, 03 Apr 2009 14:24:23 GMT

HP Analyzed the crash log and determined that /tmp and /var/utmp filled up as the result of an update we were running.

I was on the wrong track...

Re: SSH Weirdness

OldSchool — Fri, 03 Apr 2009 17:41:09 GMT

"Apr 1 03:08:18 dsbb vmunix: msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/root fi) "

uh...yeah.