cancel
Showing results for 
Search instead for 
Did you mean: 

SSH Weirdness

Margaret Sanford
Occasional Advisor

SSH Weirdness

I am running an HP 3440 Server on HP 11i Version 1.

About a week ago our ssh sessions stopped enforcing the inactivity time out. I thought it was weird, but I didn't have a chance to look into it. Then on Wednesday night the system had a catastrophic failure and we had to reboot it from the ILO. HP said the error messages they were seeing were consistent with an ssh failure and recommended upgrading to version 8.

Does this sound familiar to anybody?

Thanks in advance,

Margaret
13 REPLIES
Steven E. Protter
Exalted Contributor

Re: SSH Weirdness

Shalom,

The details no. The overall situation yes. The system had troubles, showed signs, asked for attention and did not get it.

Please post exact error messages.

seh is constantly being updated due to critical flaws and security holes. Updating it is a very good idea.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Margaret Sanford
Occasional Advisor

Re: SSH Weirdness

Here's the errors spewing accross the console:

Apr 1 03:30:38 dsbb inetd[1002]: Problem with /var/adm/inetd.sec:
cannot starts
Apr 1 03:30:44 dsbb prngd[19354]: Failed to execl(/usr/bin/ps, ps, -ex, , , ):r



Apr 1 03:31:35 dsbb prngd[19361]: Failed to execl(/usr/bin/who, who, am, i, , r



Apr 1 03:32:26 dsbb prngd[19364]: Failed to execl(/usr/bin/vmstat, vmstat, , ,r


Apr 1 03:33:17 dsbb prngd[19366]: Failed to execl(/usr/bin/tail, tail, -100, /r

Apr 1 03:34:08 dsbb prngd[19373]: Failed to execl(/usr/bin/tail, tail, -100, /r Apr 1 03:34:59 dsbb prngd[19376]: Failed to execl(/usr/bin/tail, tail, -100, /r Apr 1 03:35:51 dsbb prngd[19380]:
Failed to execl(/usr/bin/ls, ls, -alni, /devr Apr 1 03:36:42 dsbb
prngd[19383]: Failed to execl(/usr/bin/ls, ls, -alni, /varr Apr 1
03:37:33 dsbb prngd[19386]: Failed to execl(/usr/bin/ls, ls, -alni, /varr

Apr 1 03:38:24 dsbb prngd[19389]: Failed to execl(/usr/bin/ls, ls, -alni, /usrr Apr 1 03:39:15 dsbb prngd[19391]: Failed to execl(/usr/bin/ls, ls, -alni, /usrr Apr 1 03:40:06 dsbb prngd[19398]:
Failed to execl(/usr/bin/ls, ls, -alni, /tmpr

Here's what was in the syslog when it started:

Apr 1 02:58:12 dsbb sshd[18589]: error: PAM: Authentication failed for blackbrt Apr 1 02:58:20 dsbb above message repeats 2 times Apr 1 02:58:20 dsbb sshd[18613]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 02:58:21 dsbb sshd[18589]: Accepted password for blackbrd from
66.210.592
Apr 1 02:58:21 dsbb sshd[18630]: subsystem request for sftp Apr 1 02:58:40 dsbb sshd[18689]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 02:59:29 dsbb sshd[18727]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 02:59:43 dsbb su: + 0 blackbrd-harco Apr 1 03:00:43 dsbb sshd[18794]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 03:00:51 dsbb su: + 1 blackbrd-harco Apr 1 03:03:35 dsbb sshd[18936]: Accepted keyboard-interactive/pam for
blackbr2
Apr 1 03:03:49 dsbb su: + 1 blackbrd-harco Apr 1 03:03:51 dsbb sshd[18969]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:03:58 dsbb sshd[18994]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:04:15 dsbb sshd[19068]: Accepted keyboard-interactive/pam for
web1 fr2
Apr 1 03:08:18 dsbb vmunix: msgcnt 1 vxfs: mesg 003: vx_mapbad - /dev/root fild Apr 1 03:08:18 dsbb vmunix: msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/root fi) Apr 1 03:09:42 dsbb vmunix: msgcnt 4 vxfs: mesg 037: vx_metaioerr -
vx_tflush_2
Apr 1 03:09:42 dsbb vmunix: msgcnt 5 vxfs: mesg 031: vx_disable - /dev/root fid Apr 1 03:10:20 dsbb prngd[19234]: Failed to execl(/usr/bin/ls, ls, -alti, /varr Apr 1 03:10:20 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:11:11 dsbb prngd[19238]: Failed to execl(/usr/bin/ls, ls, -alti, /etcr Apr 1 03:11:11 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:12:02 dsbb prngd[19249]: Failed to execl(/usr/bin/ls, ls, -alti, /tmpr Apr 1 03:12:02 dsbb syslogd: /etc/utmp: Bad file number Apr 1 03:12:53 dsbb prngd[19255]: Failed to execl(/usr/bin/ls, ls, -alti, /varr Standard input
Kris Knigga
Advisor

Re: SSH Weirdness

Looks like it might be a disk (or file system) issue. I'm seeing a lot of references to vxfs.

vx_mapbad - /dev/root
vx_nospace - /dev/root
vx_metaioerr
vx_disable - /dev/root

Margaret Sanford
Occasional Advisor

Re: SSH Weirdness

That's what I thought too... but HP said that was not the issue. All the volumes came up cleanly when we rebooted with the FSCK that ran.

Weird.
Kris Knigga
Advisor

Re: SSH Weirdness

Was a quick fsck done, or a full?
Margaret Sanford
Occasional Advisor

Re: SSH Weirdness

quick
Kris Knigga
Advisor

Re: SSH Weirdness

I would want to see what `fsck -o full,nolog` says. I've never known good things coming from ignoring vxfs errors like that, especially when the machine crashes shortly after they start.

The only problem is, I'm guessing that since this is the root volume group, you can't unmount anything to try this and I can't think of any way to cause a full fsck on next boot.
Margaret Sanford
Occasional Advisor

Re: SSH Weirdness

Interesting. This server is difficult to bring down for any period of time, so I'll schedule an FSCK asap.

Does anybody have any insight into why the inactive time out period just went away?

I'm not sure where to check if that's still enabled.

-M
Kris Knigga
Advisor

Re: SSH Weirdness

Looks like you'd have to reboot to single user mode to be able to fsck things like /var properly (while they are unmounted).

I'm leaning toward there being two issues here; one being the SSH stuff and the other being the crash.

How was the timeout set up before? Was it truly a function of SSH, or was it done at the shell level (by setting TMOUT or something)?
Margaret Sanford
Occasional Advisor

Re: SSH Weirdness

I think it was at the shell level. But it was one of those "It was like that when I got here!" things...

But I don't know how to tell. If that parameter is set and it's not timing me out then something in ssh is hosed.
Heironimus
Honored Contributor

Re: SSH Weirdness

You should make sure you aren't missing any JFS-related patches, but that log really, REALLY looks like your system blew up because it got an I/O error on the root disk.
Margaret Sanford
Occasional Advisor

Re: SSH Weirdness

HP Analyzed the crash log and determined that /tmp and /var/utmp filled up as the result of an update we were running.

I was on the wrong track...
OldSchool
Honored Contributor

Re: SSH Weirdness

"Apr 1 03:08:18 dsbb vmunix: msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/root fi) "

uh...yeah.