HPUX 11iv1 server rebooted itself

[jaivinder]

Hi Gurus,

We are having a L3000 server in our datacenter. Today it got rebooted at 6 am. I didnt found the info about in the syslog file. Tombstone file is also created after this reboot.
The output of ts99 is
# more ts99
HP-UX windom B.11.11 U 9000/800 1150464646

CPU-ID( Model ) = 0x13

------- Processor 0 HPMC Information - PDC Version: 44.12 ------

* * * No valid timestamp * * *


No HPMC chassis codes logged

General Registers 0 - 31
00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000
20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000
24-27 0000000000000000 0000000000000000 0000000000000000 0000000000000000
28-31 0000000000000000 0000000000000000 0000000000000000 0000000000000000


Control Registers 0 - 31
00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000
20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000
24-27 0000000000000000 0000000000000000 0000000000000000 0000000000000000
28-31 0000000000000000 0000000000000000 0000000000000000 0000000000000000

Space Registers 0 - 7
00-03 00000000 00000000 00000000 00000000
04-07 00000000 00000000 00000000 00000000


IIA Space (back entry) = 0x0000000000000000
IIA Offset (back entry) = 0x0000000000000000
Check Type = 0x00000000
CPU State = 0x00000000
Cache Check = 0x00000000
TLB Check = 0x00000000
Bus Check = 0x00000000
Assists Check = 0x00000000
Assist State = 0x00000000
Path Info = 0x00000000
System Responder Address = 0x0000000000000000
System Requestor Address = 0x0000000000000000


Floating Point Registers 0 - 31
00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000
20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000
24-27 0000000000000000 0000000000000000 0000000000000000 0000000000000000
28-31 0000000000000000 0000000000000000 0000000000000000 0000000000000000


Check Summary = 0x0000000000000000
Available Memory = 0x0000000000000000
CPU Diagnose Register 2 = 0x0000000000000000
CPU Status Register 0 = 0x0000000000000000
CPU Status Register 1 = 0x0000000000000000
SADD LOG = 0x0000000000000000
Read Short LOG = 0x0000000000000000



----------------- DEW 0 HPMC Information - ------

No DEW errors logged


-------------- Memory Error Log Information --------------

Bus 0 Log Information


No errors logged for this bus


Bus 1 Log Information


No errors logged for this bus


------------ I/O Module Error Log Information ------------


No I/O module errors logged


------- Processor 2 HPMC Information - PDC Version: 44.12 ------

* * * No valid timestamp * * *


No HPMC chassis codes logged

General Registers 0 - 31
00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000
20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000
24-27 0000000000000000 0000000000000000 0000000000000000 0000000000000000
28-31 0000000000000000 0000000000000000 0000000000000000 0000000000000000


Control Registers 0 - 31
00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000
20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000
24-27 0000000000000000 0000000000000000 0000000000000000 0000000000000000
28-31 0000000000000000 0000000000000000 0000000000000000 0000000000000000

Space Registers 0 - 7
00-03 00000000 00000000 00000000 00000000
04-07 00000000 00000000 00000000 00000000


IIA Space (back entry) = 0x0000000000000000
IIA Offset (back entry) = 0x0000000000000000
Check Type = 0x00000000
CPU State = 0x00000000
Cache Check = 0x00000000
TLB Check = 0x00000000
Bus Check = 0x00000000
Assists Check = 0x00000000
Assist State = 0x00000000
Path Info = 0x00000000
System Responder Address = 0x0000000000000000
System Requestor Address = 0x0000000000000000


Floating Point Registers 0 - 31
00-03 0000000000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000000000 0000000000000000 0000000000000000 0000000000000000
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000000000000 0000000000000000 0000000000000000 0000000000000000
20-23 0000000000000000 0000000000000000 0000000000000000 0000000000000000
24-27 0000000000000000 0000000000000000 0000000000000000 0000000000000000
28-31 0000000000000000 0000000000000000 0000000000000000 0000000000000000


Check Summary = 0x0000000000000000
Available Memory = 0x0000000000000000
CPU Diagnose Register 2 = 0x0000000000000000
CPU Status Register 0 = 0x0000000000000000
CPU Status Register 1 = 0x0000000000000000
SADD LOG = 0x0000000000000000
Read Short LOG = 0x0000000000000000



----------------- DEW 2 HPMC Information - ------

No DEW errors logged


-------------- Memory Error Log Information --------------

Bus 0 Log Information


No errors logged for this bus


Bus 1 Log Information


No errors logged for this bus


------------ I/O Module Error Log Information ------------


No I/O module errors logged


Module Revision
------ --------
System Board A14302
PA 8700 CPU Module 3.1
PA 8700 CPU Module 3.1

Also find the /etc/shutdownlog o/p

# tail -1 /etc/shutdownlog
06:01 Thu Oct 23, 2008. Reboot:

The Old syslog info is as following
# tail -10 /var/adm/syslog/OLDsyslog.log
Oct 23 05:56:40 windom inetd[26092]: registrar/tcp: Connection from terp (10.8.201.23) at Thu Oct 23 05:56:40 2008
Oct 23 05:58:40 windom inetd[26455]: registrar/tcp: Connection from terp (10.8.201.23) at Thu Oct 23 05:58:40 2008
Oct 23 06:00:03 windom su: + tty?? root-applmgr
Oct 23 06:00:29 windom /usr/sbin/envd[1400]: terminated by signal 15
Oct 23 06:00:29 windom diagmond[1397]: Exit due to user requested abort
Oct 23 06:00:30 windom FontServer[1318]: terminating
Oct 23 06:00:30 windom sshd[19599]: Received signal 15; terminating.
Oct 23 06:00:36 windom inetd[867]: Going down on signal 15
Oct 23 06:00:42 windom rpcbind: terminate: rpcbind terminating on signal. Restart with "rpcbind -w"
Oct 23 06:01:00 windom syslogd: going down on signal 15

[Pete Randall]

Take a look at /etc/shutdownlog. Sometimes you'll find a reason there.


Pete

[Torsten.]

Looks like a "normal" reboot to me - not a crash.

According to the log somebody did connect to the system followed by a "su".

Perhaps somebody initiated this reboot?

[Tim Nelson]

As Torsten pointed out.

>>Oct 23 06:01:00 windom syslogd: going down on signal 15

Looks to me like a normal shutdown was issued. "going down on signal 15".

[Torsten.]

What is

# more /usr/adm/sulog

telling you?

You may also check for a shell history file related to user root-applmgr.

With some luck you find the exact command also in the console history of the GSP:

GSP> cl
(console log)

[Steven E. Protter]

Shalom,

Good news:

No HPMC chassis codes logged


No major hardware failure.

Looks like a software issue.

Logs, logs, enjoy reading the logs.

SEP

[rajv]

Hi,

registrar/tcp is a cluster related bug.

is this server clustered?....

please check with hardware vendor new fix for the cluster.

[Dennis Handly]

>Torsten: You may also check for a shell history file related to user root-applmgr.

root-applmgr is the sulog syntax for root used su(1) to change to applmgr.

[Torsten. · Accepted solution]

Dennis, copy and paste didn't work :-((

Anyway, I think somebody (or something) initiated the reboot anyhow.

You may check the MP logs too (e.g. for overtemp events or failed fan messages - should be also in the OLDsyslog, but earlier than the messages you posted).