System shutdown unexpectedly

[Norman Dignard]

This morning our prod server was down.
Hit the power switch to bring it back up without a problem. I was trying to find out why the system went down but have only fould a minor clue in the Oldsyslog which stated the following:
Aug 23 01:08:09 nesaweb HP-PRM: [18830]: prmconfig: configuration reset
Aug 23 01:08:43 nesaweb /usr/sbin/envd[1685]: terminated by signal 15
Aug 23 01:08:43 nesaweb diagmond[1682]: Exit due to user requested abort
Aug 23 01:08:44 nesaweb sshd[1119]: Received signal 15; terminating.
Aug 23 01:08:51 nesaweb cimserver[1601]: PGS10031: CIM server HP-UX WBEM Services stopped.
Aug 23 01:08:52 nesaweb cimserver[1600]: exited normally
Aug 23 01:08:59 nesaweb inetd[1192]: Going down on signal 15
Aug 23 01:08:59 nesaweb rpcbind: rpcbind terminating on signal.
Aug 23 01:08:59 nesaweb su: + tty?? root-sfmdb
Aug 23 01:09:07 nesaweb syslogd: going down on signal 15

No cronjob nor power bump triggred it that I can find.

Anyone have any suggestions on tracing down the cause? HPUX 11.31 on RP2660

[Mel Burslan]

There is nothing in the /etc/shutdownlog ?
Did you check the timestamp of

/var/tombstones/ts99 ? If it is fresh as of the time of reboot, you might want to look into that as well.

I am also assuming, this crash did not generate a crash dump. Do you have a remote console setup, where the serial terminal server device having roll back buffer ? There might be some messages on the console (or console history) that may help you understand the nature of this sudden shutdown.

This is all I can think of in the light of the information given in your question.

[Steven E. Protter]

Shalom,

Connect into the console. Ctrl-b

Log into it.

SL

Show logs.

It might show a hardware problem you need to deal with.

You may also look for updates to the system in general, a bi-annual QPK, or updates to CIM specifically.

SEP

[Torsten.]

Looks like a shutdown, not a crash.

All the prcesses are going down, something was initiating this shutdown.

The reason is maybe listed earlier in syslog or in MP event logs too.

[Norman Dignard]

Still no clue.
checked the var/tombstones dir - its empty
No idea why.. DOes 11.31 work differnently?

Shutdown log - all it has is
07:13 Wed Aug 19, 2009. Reboot: (by nesaweb!root)
01:09 Sun Aug 23, 2009. Halt:

Console logs did not show anything other than a "Cold restart" message around that time.

Checked the UPS logs - no power outage/bumps.

mystified...

[Torsten.]

Any "overtemp" messages in syslog or MP log?

[Norman Dignard]

no over temps reported

[Ganesan R]

Hi,

>>>01:09 Sun Aug 23, 2009. Halt:<<<

This message from shutdownlog shows, it's not a unexpected shutdown. Someone(might be some process or user) halted the system.

In that case you won't find any clue from ts99 or crash dump files or from console logs.

[Norman Dignard]

Thats just it - there is no one that could have done this. It happened just after midnight on Sunday. No one is in then nor does anyone have remote or phyiscal access to the box.

All that is running on this host is a web app that talks to a db on another server.

[Patrick Wallek]

Check the output of the 'last' command. See if anyone logged in around that time. A 'last -R' will show hostnames or IP addresses from which the connection originated.

[Tingli]

Anything from the crontab?

[Norman Dignard]

No entries in crontab and no logins
between 21 Aug and 24. The shudtwon was on the 23rd

[Mel Burslan]

All evidence, or lack thereof, points to a shutdown initiated by the EMS, due to a hardware malfunction decision (correct or false)

if you have connected to the console and looked at the system event logs and still have no idea, I am assuming, since this is a production server, it covered by HP service contract and it is in your best interest to have the server looked at. It is an unacceptable behavior of the server/OS pair to deliberately shutdown without leaving a trace behind, explaining what happened on any configuration, let alone on a production grade server.

[SUDHAKAR_18]

Can you please check
/var/opt/resmon/log/event.log
/var/opt/resmon/log/rst.log

[Norman Dignard]

I haven't found out why yet even after searching other logs etc.. I'll monitor the situation