K 370 - "Reboot after panic: , isr.ior = 0'1004000b.0'b86e12a0"

[Singar]

Hello gurus,

Today one of our K Class 370 server got rebooted automatically. I checked /etc/shutdownlog file for any entries. The following msg was added before reboot
"Reboot after panic: , isr.ior = 0'1004000b.0'b86e12a0".

Following is the filtered output of /var/tombstones/ts99 file.

HPMC Chassis Codes = 0xcbf0 0x20b3 0x5408 0x5508 0xcbfb
HPMC Chassis Codes = 0xcbf0 0x510b 0x5408 0x5508 0xcbfb

Can anyone tell me what is wrong with my system ? 0x20b3 means what ??

P.S. This thread has been moevd from General to HP 9000. - Hp Forum Moderator

[Adisuria Wangsadinata_1]

Hi,

Need ur help to give us the complete ts99 file (as attachment).

Thanks & Cheers,
AW

[Singar]

Hi,

Attaching ts99 file...

thx
-singaravelu

[Biswajit Tripathy]

If the system paniced, see if you have a
crash dump at /var/adm/crash created at the
time of crash?

- Biswajit

[Isralyn Manalac_1]

Hi,

I'd suggest ringing HP ESC to do a fingerprinting of the ts file. The ts file has indicated a possible system board failure. Do you have files under /var/adm/crash? Can you also please post the OLDsyslog.log?

Regards,

Ira

[Singar]

Hello all,

OLDsyslog.log is of no use. There is no potential message given for the panic.
I followed below procedure to get some details on panic using q4 debugger:

1) # cd /var/adm/crash/crash.0
2) # uncompress /usr/contrib/lib/Q4Lib.tar.Z
3) # tar -xf /usr/contrib/lib/Q4Lib.tar
4) # cp q4lib/sample.q4rc.pl ~/.q4rc.pl
5) # /usr/contrib/bin/gunzip vmunix.gz
6) # /usr/contrib/bin/q4pxdb vmunix
7) # q4 -p .

I got the following o/p when "q4 -p ." was run,
===========================================
@(#) q4 $Revision: 11.X B.11.23d Thu May 6 18:05:11 PST 2003$ 0
q4: (warning) Here are the savecore error messages -
q4: (warning) savecrash: Insufficient space to save full core dump,
q4: (warning) savecrash: Could not completely process dump image area 1 on /dev/
dsk/c0t4d0
Reading kernel symbols ...
Reading data types ...
Error: requested page does not exist on target machine.
q4: (warning) failing page number = 0xbdb status - -2
q4: (error) can not read symbol pdirhash_type from core file
quit
===========================================

The expected behaviour was that q4 shoud have taken me to q4 utility prompt. Looking at the message, I concluded that dump area is lesser size than required. Then I added one more dump device using "crashconf". Below is the o/p of crashconf

===========================================
Crash dump configuration has been changed since boot.

CLASS PAGES INCLUDED IN DUMP DESCRIPTION
-------- ---------- ---------------- -------------------------------------
UNUSED 91611 no, by default unused pages
USERPG 40278 no, by default user process pages
BCACHE 219784 no, by default buffer cache pages
KCODE 2567 no, by default kernel code pages
USTACK 1630 yes, by default user process stacks
FSDATA 684 yes, by default file system metadata
KDDATA 135937 yes, by default kernel dynamic data
KSDATA 31797 yes, by default kernel static data

Total pages on system: 524288
Total pages included in dump: 170048

DEVICE OFFSET(kB) SIZE (kB) LOGICAL VOL. NAME
------------ ---------- ---------- ------------ -------------------------
31:0x004000 314208 4194304 64:0x000002 /dev/vg00/lvol2
31:0x005000 105312 1048576 64:0x010002 /dev/vg01/lvol2 <<< This is the newly added dump device
----------
5242880
==========================================

Again tried q4.. again same error occured.

Any one has ever run q4 to get more details of panic ?

Or coult it be that I haven't configured crashconf to store panic details ?

thx
-velu

[Singar]

finally I got q4 run and got the following two files as described in http://67.176.164.101/ll/hpq4.html#full_doc

- trace.out
- ana.out

and when I grepped

grep HPMC ana.out trace.out

# grep HPMC ana.out trace.out
ana.out:Crash Event 1 (HPMC, struct crash_event_table_struct at 0xa89030):
ana.out:crash event was an HPMC

What is next ? We don't have support contract with HP?

Is there any one who is having K Class Service Manaual ?

[Fabio Ettore]

Hi Singar,

if you don't have a support contract by HP then it is the real problem.
I am pretty sure that the panic was for a bad CPU. I had a very similar problem a few days ago (with isr.ior entry in /etc/shutdownlog and HPMC) and by replacing a CPU the problem was solved.

Best regards,
Fabio

[melvyn burnard]

You have had a hardware problem cause your system panic. You will need to get this analysed to be able to look at what caused the problem.
I suggest you log a suppoprt call with your hardware mainternance supplier, or you could log a call with HP

[Deepak.R]

Hi,

Its a hardware issue and HPMC ( High Priority Machine Check ) occured in your system .

Detailed analysing of ts99 or PIM collected from bch manu is to be done using special decoders to know which part of the system has caused the error.

It could be a mainboard or I/O expansion board issue , need to log a call with HP , they may replace it on a on time payable basis since you do noit have a contract.

You will not find HPMC analysing information in service manuals and this is not a user performable action.


regards
deepak