N Class Server Crashed

[Senthilkumaran.T]

Hello,
One of the N class server was crashed and it automatically rebooted. The following is the content of the file created in /var/adm/crash/crash.3/INDEX

comment savecrash crash dump INDEX file
version 2
hostname ijmsia02
modelname 9000/800/N4000-55
panic , isr.ior = 0'0.0'0
dumptime 1005204024 Thu Nov 8 15:20:24 SST 2001
savetime 1005205580 Thu Nov 8 15:46:20 SST 2001
release @(#)B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997

memsize 0
chunksize 67108864
warning savecrash: Invalid /etc/savecore.LCK file from previous run.
Dump may be corrupted

module /stand/vmunix vmunix 14026448 2109327043
image image.1.1 0x0000000000000000 0x0000000003ffb000 0x0000000000000000 0x00000000000047bf 1294486519
image image.1.2 0x0000000000000000 0x0000000003ff9000 0x00000000000047c0 0x00000000000087b7 636300134
image image.1.3 0x0000000000000000 0x0000000003ff9000 0x00000000000087b8 0x000000000000c7af 3023565294
image image.1.4 0x0000000000000000 0x0000000003ff8000 0x000000000000c7b0 0x00000000000264a7 342411329
image image.1.5 0x0000000000000000 0x0000000003ff5000 0x00000000000264a8 0x000000000007a26f 488036676
image image.1.6 0x0000000000000000 0x0000000001de9000 0x000000000007a270 0x000000000007ffff 3257218428
image image.2.1 0x0000000000000000 0x0000000000010000 0x0000000000180000 0x00000000001fffff 4215202376

How do I check, what could be the possible cause of this crash.

Thanks
Senthil

[Bill McNAMARA_1]

well,

look at the following:

/etc/shutdownlog

and see when this started happening, perhaps it corresponds to s/w installation..

look at /var/adm/syslog/OLDsyslog
for system messages before the crash.

look at /var/adm/tombstones/ts99 for crash signatures.

man savecrash for configuring your dumps, because there seems to be a problem there..

Your crash dumps will help debug (with q4) the cause of the pb. /var/adm/crash typically.

Later,
Bill

[Bill McNAMARA_1]

you might also consider installing the latest QPK from software.hp.com and XSWHWCR because it's possible a corrected bug.

swlist and
swlist -l product to figure out your patch level

[melvyn burnard]

from your panic message:
panic , isr.ior = 0'0.0'0

This could be either:
A ServiceGuard TOC (if you have ServiceGuard running)
A manual TOC (Someone forced it to happen)
or
An HPMC (hardware problem)
I would get you hardware support people to investigate this if it is the last.

Tail of /etc/shutdownlog could also be a bit more informative here.

[Rainer von Bongartz]

take a look at /var/adm/tombstones/ts99

If you find that the machine had a HPMC (High Priority Machine Check) you are having a HW problem and you should contact technical HW support.

Regards
Rainer

[Senthilkumaran.T]

Hello All,
I did some analysis on Q4 and I got the following output. Could you guess what could be the possible problem. This server is configured with the MC/Service guard as well. The Shutdownlog also has the same information as INDEX file "isr.ior = 0'0.0'0".

Thanks
Senthil


Stack trace from the first crash event:
stack trace for event 0
crash event was a TOC
FUNC PC SP FMP SSP
idle+0x198 0.0x125a48 0.0x0'0d82a1f0 0.0x0'0d82a1e0 n/a
swidle_exit+0x0 0.0x128974 0.0x0'0d82a050 0.0x0'0d82a040 n/a

[Senthilkumaran.T]

Hello All,
I checked the tombstone/ts99 and found that there are no errors.

Senthil

[Roger Baptiste]

<>

This is typically a TOC generated message.
If you TOCed , was it in response to a Hanging
system? You would need go back to the stage
where the system had the problem . What applications were running? Was there heavy , undue load on the system? For instance , was there backups running on the box?
Do you have fibre cards without the latest patches?

Regarding Q4 analysis, i think your best bet would be the HP support folks .

In all probability, you will be installing
a load of patches very soon ;-)

-raj

[Patrick Wessel]

One reason for an N-Class to TOC without user interaction is the Automatic Restart. You are able to configure the GSP that it will TOC the system at a certain alert level. It makes sense to set the alert level to 13 (System hand detected via timer popping). Usually such a TOC would leave a fingerprint in the error logs of the GSP.