SSH Weirdness

[Margaret Sanford]

I am running an HP 3440 Server on HP 11i Version 1.

About a week ago our ssh sessions stopped enforcing the inactivity time out. I thought it was weird, but I didn't have a chance to look into it. Then on Wednesday night the system had a catastrophic failure and we had to reboot it from the ILO. HP said the error messages they were seeing were consistent with an ssh failure and recommended upgrading to version 8.

Does this sound familiar to anybody?

Thanks in advance,

Margaret

[Steven E. Protter]

Shalom,

The details no. The overall situation yes. The system had troubles, showed signs, asked for attention and did not get it.

Please post exact error messages.

seh is constantly being updated due to critical flaws and security holes. Updating it is a very good idea.

SEP

[Margaret Sanford]

Here's the errors spewing accross the console:

Apr 1 03:30:38 dsbb inetd[1002]: Problem with /var/adm/inetd.sec:
cannot starts
Apr 1 03:30:44 dsbb prngd[19354]: Failed to execl(/usr/bin/ps, ps, -ex, , , ):r



Apr 1 03:31:35 dsbb prngd[19361]: Failed to execl(/usr/bin/who, who, am, i, , r



Apr 1 03:32:26 dsbb prngd[19364]: Failed to execl(/usr/bin/vmstat, vmstat, , ,r


Apr 1 03:33:17 dsbb prngd[19366]: Failed to execl(/usr/bin/tail, tail, -100, /r

Apr 1 03:34:08 dsbb prngd[19373]: Failed to execl(/usr/bin/tail, tail, -100, /r Apr 1 03:34:59 dsbb prngd[19376]: Failed to execl(/usr/bin/tail, tail, -100, /r Apr 1 03:35:51 dsbb prngd[19380]:
Failed to execl(/usr/bin/ls, ls, -alni, /devr Apr 1 03:36:42 dsbb
prngd[19383]: Failed to execl(/usr/bin/ls, ls, -alni, /varr Apr 1
03:37:33 dsbb prngd[19386]: Failed to execl(/usr/bin/ls, ls, -alni, /varr

Apr 1 03:38:24 dsbb prngd[19389]: Failed to execl(/usr/bin/ls, ls, -alni, /usrr Apr 1 03:39:15 dsbb prngd[19391]: Failed to execl(/usr/bin/ls, ls, -alni, /usrr Apr 1 03:40:06 dsbb prngd[19398]:
Failed to execl(/usr/bin/ls, ls, -alni, /tmpr

Here's what was in the syslog when it started:

Apr 1 02:58:12 dsbb sshd[18589]: error: PAM: Authentication failed for blackbrt Apr 1 02:58:20 dsbb above message repeats 2 times Apr 1 02:58:20 dsbb sshd[18613]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 02:58:21 dsbb sshd[18589]: Accepted password for blackbrd from
66.210.592
Apr 1 02:58:21 dsbb sshd[18630]: subsystem request for sftp Apr 1 02:58:40 dsbb sshd[18689]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 02:59:29 dsbb sshd[18727]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 02:59:43 dsbb su: + 0 blackbrd-harco Apr 1 03:00:43 dsbb sshd[18794]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 03:00:51 dsbb su: + 1 blackbrd-harco Apr 1 03:03:35 dsbb sshd[18936]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 03:03:49 dsbb su: + 1 blackbrd-harco Apr 1 03:03:51 dsbb sshd[18969]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:03:58 dsbb sshd[18994]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:04:15 dsbb sshd[19068]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:08:18 dsbb vmunix: msgcnt 1 vxfs: mesg 003: vx_mapbad - /dev/root fild Apr 1 03:08:18 dsbb vmunix: msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/root fi) Apr 1 03:09:42 dsbb vmunix: msgcnt 4 vxfs: mesg 037: vx_metaioerr -
vx_tflush_2
Apr 1 03:09:42 dsbb vmunix: msgcnt 5 vxfs: mesg 031: vx_disable - /dev/root fid Apr 1 03:10:20 dsbb prngd[19234]: Failed to execl(/usr/bin/ls, ls, -alti, /varr Apr 1 03:10:20 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:11:11 dsbb prngd[19238]: Failed to execl(/usr/bin/ls, ls, -alti, /etcr Apr 1 03:11:11 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:12:02 dsbb prngd[19249]: Failed to execl(/usr/bin/ls, ls, -alti, /tmpr Apr 1 03:12:02 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:12:53 dsbb prngd[19255]: Failed to execl(/usr/bin/ls, ls, -alti, /varr Standard input

[Kris Knigga]

Looks like it might be a disk (or file system) issue. I'm seeing a lot of references to vxfs.

vx_mapbad - /dev/root
vx_nospace - /dev/root
vx_metaioerr
vx_disable - /dev/root

[Margaret Sanford]

That's what I thought too... but HP said that was not the issue. All the volumes came up cleanly when we rebooted with the FSCK that ran.

Weird.

[Kris Knigga]

Was a quick fsck done, or a full?

[Margaret Sanford]

quick

[Kris Knigga]

I would want to see what `fsck -o full,nolog` says. I've never known good things coming from ignoring vxfs errors like that, especially when the machine crashes shortly after they start.

The only problem is, I'm guessing that since this is the root volume group, you can't unmount anything to try this and I can't think of any way to cause a full fsck on next boot.

[Margaret Sanford]

Interesting. This server is difficult to bring down for any period of time, so I'll schedule an FSCK asap.

Does anybody have any insight into why the inactive time out period just went away?

I'm not sure where to check if that's still enabled.

-M

[Kris Knigga]

Looks like you'd have to reboot to single user mode to be able to fsck things like /var properly (while they are unmounted).

I'm leaning toward there being two issues here; one being the SSH stuff and the other being the crash.

How was the timeout set up before? Was it truly a function of SSH, or was it done at the shell level (by setting TMOUT or something)?