topic Re: System shutdown unexpectedly in Operating System - HP-UX

System shutdown unexpectedly

Norman Dignard — Mon, 24 Aug 2009 13:18:25 GMT

This morning our prod server was down.
Hit the power switch to bring it back up without a problem. I was trying to find out why the system went down but have only fould a minor clue in the Oldsyslog which stated the following:
Aug 23 01:08:09 nesaweb HP-PRM: [18830]: prmconfig: configuration reset
Aug 23 01:08:43 nesaweb /usr/sbin/envd[1685]: terminated by signal 15
Aug 23 01:08:43 nesaweb diagmond[1682]: Exit due to user requested abort
Aug 23 01:08:44 nesaweb sshd[1119]: Received signal 15; terminating.
Aug 23 01:08:51 nesaweb cimserver[1601]: PGS10031: CIM server HP-UX WBEM Services stopped.
Aug 23 01:08:52 nesaweb cimserver[1600]: exited normally
Aug 23 01:08:59 nesaweb inetd[1192]: Going down on signal 15
Aug 23 01:08:59 nesaweb rpcbind: rpcbind terminating on signal.
Aug 23 01:08:59 nesaweb su: + tty?? root-sfmdb
Aug 23 01:09:07 nesaweb syslogd: going down on signal 15

No cronjob nor power bump triggred it that I can find.

Anyone have any suggestions on tracing down the cause? HPUX 11.31 on RP2660

Re: System shutdown unexpectedly

Mel Burslan — Mon, 24 Aug 2009 13:24:47 GMT

There is nothing in the /etc/shutdownlog ?
Did you check the timestamp of

/var/tombstones/ts99 ? If it is fresh as of the time of reboot, you might want to look into that as well.

I am also assuming, this crash did not generate a crash dump. Do you have a remote console setup, where the serial terminal server device having roll back buffer ? There might be some messages on the console (or console history) that may help you understand the nature of this sudden shutdown.

This is all I can think of in the light of the information given in your question.

Re: System shutdown unexpectedly

Steven E. Protter — Mon, 24 Aug 2009 13:27:04 GMT

Shalom,

Connect into the console. Ctrl-b

Log into it.

SL

Show logs.

It might show a hardware problem you need to deal with.

You may also look for updates to the system in general, a bi-annual QPK, or updates to CIM specifically.

SEP

Re: System shutdown unexpectedly

Torsten. — Mon, 24 Aug 2009 13:35:05 GMT

Looks like a shutdown, not a crash.

All the prcesses are going down, something was initiating this shutdown.

The reason is maybe listed earlier in syslog or in MP event logs too.

Re: System shutdown unexpectedly

Norman Dignard — Mon, 24 Aug 2009 14:34:36 GMT

Still no clue.
checked the var/tombstones dir - its empty
No idea why.. DOes 11.31 work differnently?

Shutdown log - all it has is
07:13 Wed Aug 19, 2009. Reboot: (by nesaweb!root)
01:09 Sun Aug 23, 2009. Halt:

Console logs did not show anything other than a "Cold restart" message around that time.

Checked the UPS logs - no power outage/bumps.

mystified...

Re: System shutdown unexpectedly

Torsten. — Mon, 24 Aug 2009 14:36:31 GMT

Any "overtemp" messages in syslog or MP log?

Re: System shutdown unexpectedly

Norman Dignard — Mon, 24 Aug 2009 14:51:51 GMT

no over temps reported

Re: System shutdown unexpectedly

Ganesan R — Mon, 24 Aug 2009 15:01:10 GMT

Hi,

>>>01:09 Sun Aug 23, 2009. Halt:<<<

This message from shutdownlog shows, it's not a unexpected shutdown. Someone(might be some process or user) halted the system.

In that case you won't find any clue from ts99 or crash dump files or from console logs.

Re: System shutdown unexpectedly

Norman Dignard — Mon, 24 Aug 2009 15:06:12 GMT

Thats just it - there is no one that could have done this. It happened just after midnight on Sunday. No one is in then nor does anyone have remote or phyiscal access to the box.

All that is running on this host is a web app that talks to a db on another server.

Re: System shutdown unexpectedly

Patrick Wallek — Mon, 24 Aug 2009 15:32:14 GMT

Check the output of the 'last' command. See if anyone logged in around that time. A 'last -R' will show hostnames or IP addresses from which the connection originated.

Re: System shutdown unexpectedly

Tingli — Mon, 24 Aug 2009 15:34:15 GMT

Anything from the crontab?

Re: System shutdown unexpectedly

Norman Dignard — Mon, 24 Aug 2009 16:47:49 GMT

No entries in crontab and no logins
between 21 Aug and 24. The shudtwon was on the 23rd

Re: System shutdown unexpectedly

Mel Burslan — Mon, 24 Aug 2009 17:09:27 GMT

All evidence, or lack thereof, points to a shutdown initiated by the EMS, due to a hardware malfunction decision (correct or false)

if you have connected to the console and looked at the system event logs and still have no idea, I am assuming, since this is a production server, it covered by HP service contract and it is in your best interest to have the server looked at. It is an unacceptable behavior of the server/OS pair to deliberately shutdown without leaving a trace behind, explaining what happened on any configuration, let alone on a production grade server.

Re: System shutdown unexpectedly

SUDHAKAR_18 — Tue, 25 Aug 2009 08:03:55 GMT

Can you please check
/var/opt/resmon/log/event.log
/var/opt/resmon/log/rst.log

Re: System shutdown unexpectedly

Norman Dignard — Mon, 31 Aug 2009 16:45:44 GMT

I haven't found out why yet even after searching other logs etc.. I'll monitor the situation