RP2450 running 11.0 Unexpected HPMC/TOC. Processor HPA FFFFFFFF'FFFA0000

[moonchild]

rp2450 running 11.0 was working fine up to this HPMC. can't boot the system, can't get to BCH, and can't get the CO messages while resetting the system.

any help would be much appreciated
thanks

******* Unexpected HPMC/TOC. Processor HPA FFFFFFFF'FFFA0000 *******


Press Q/q to quit, Enter to continue:
GENERAL REGISTERS:
r00/03 00000000'00000000 00000000'00000000 00000000'00417817 00000000'40103170
r04/07 00000000'C545DC10 00000000'00000003 00000000'C5461A4C
00000000'400E92C8
r08/11 00000000'C546FD1C 00000000'00000003 00000000'4006A4A8
00000000'00007F88
r12/15 00000000'00007F88 00000000'00007F88 00000000'00000000
00000000'00007F88
r16/19 00000000'00000018 00000000'00000020 00000000'00000001
00000000'00000004
r20/23 00000000'FFFFFFFF 00000000'00000180 00000000'00000020 00000000'00000000
r24/27 FFFFFFFF'FFFFFFFC 00000000'C545DC10 00000000'40103170 00000000'4006FEA0
r28/31 00000000'00000000 00000000'000000C6 00000000'7F7F2070 00000000'00000000
CONTROL REGISTERS:
sr0/3 00000000'019F2C00 00000000'00000000 00000000'00000000 00000000'00000000
sr4/7 00000000'019F2C00 00000000'08CB9800 00000000'001E7C00 00000000'001E7C00 pcq = 00000000'019F2C00.00000000'0041A0AF
00000000'019F2C00.00000000'0041A0B3
isr = 00000000'1824004F ior = 40000000'D37F5000 iir = 6BC73F21 rctr =
7FFF4396

pid reg cr8/cr9 000070F7'0000F63B 00000000'0000A3CE
pid reg cr12/cr13 00000000'00000000 00000000'00004E88 ipsw = 000000FF'0006FF1F iva = 00000000'0002C000 sar = 3C ccr = C0
tr0/3 00000000'007E4D40 00000000'C00AF000 00000000'00000000 00000000'7F7A2000
tr4/7 00000000'3D485000 00099591'32204BE0 00000000'40001A70 00000000'03C642B0


Press Q/q to quit, Enter to continue:
eiem = FFFFFFFF'FFFFFFFF eirr = 00000000'00000000 itmr = 00099591'3226540D
cr1/4 00000000'00000000 00000000'00000000 00000000'00000000 00000000'00000000
cr5/7 00000000'00000000 00000000'00000000 00000000'00000000
MACHINE CHECK PARAMETERS:
Check Type = 20000000 CPU STATE = 9E000004 Cache Check = 00000000 TLB Check = 00000000 Bus Check = 00105004 PIM State = ? SIU Status = ????????
Assists = 00000000 Processor = 00000000
Slave Addr = 00000000'00000000 Master Addr = FFFFFFFF'FFFA0000


HPMC, pcsq.pcoq = 0'19f2c00.0'41a0af , isr.ior =
0'1824004f.40000000'd37f5000
@(#)B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997 High-priority machine check: (display==0xd910, flags==0x0)
* CPU State 0x9e000004
* Bus Check 0x105004
* Master Address 0xffffffff'0xfffa0000



*** A system crash has occurred. (See the above messages for details.)

[Michael Steele_2 · Accepted solution]

You lost a core I/O board or since a CPU is referenced, a CPU. Something major. Why you can't access the BCH also points to the GSP. If you succeed in getting into the MP then get the PIMINFO and error logs. And I'd call hp and look for something to canabalize in your lab.

[melvyn burnard]

You have had a serious hardware failure.
I recommend you log a call with your local HP Response Centre

[moonchild]

Michael,

How did you know that it was a CORE IO.
and which CPU number?

System does not even get to BCH level.

the only thing I have is the GSP error logs

thanks

[Michael Steele_2]

I searched the itrc with these patterns:

CPU STATE = 9E000004

isr.ior =
0'1824004f.40000000'd37f5000

HPMC is a High Priority Machine Check seen on every newer HP PA-RISC server. If you look in /var/tombstones/ts99 you'll see a standard template used for recording HPMC's. Since you beast is down you'll have to look on another.

Reference the isr.ior message to HP HW, it means something special to them.

[moonchild]

the following is the output after resetting the server from the GSP:

GSP> co



CO


Leaving Guardian Service Processor Command Interface and entering

Console mode. Type Ctrl-B to reactivate the GSP Command Interface.


********** VIRTUAL FRONT PANEL **********
System Boot detected
*****************************************
LEDs: RUN ATTENTION FAULT REMOTE POWER
OFF FLASH ON OFF ON
LED State: Boot Failed. Non-critical error detected.
Check Chassis and Console Logs for error messages.

platform config 626F
processor test 1142
processor test 1100
processor test 1100
processor test 1100
processor test 1100
processor test 1100
PDH config 322F
PDH test 3149
PDH test 3160
platform test 616A
processor test 1146
processor INIT 1701
processor test 1110
processor test 1111
processor test 1112
processor test 1113
processor test 1114
processor test 1115
processor test 1116
processor test 1117
processor test 1118
processor test 1119
processor test 111A
processor test 111B
processor test 111C
processor test 111D
processor cache test 2111
processor cache test 2112
processor cache test 2113
processor cache test 2121
processor cache test 2122
processor test 1151
processor INIT 1701
processor test 1110
processor test 1142
processor test 1142
PDH test 3158
PDH test 3157
PDH test 316E
PDH test 316E
PDH test 316E
memory config 7210
memory INIT 7702
memory INIT 771D
memory test 7150
memory config 7213
memory config 7213
memory config 7214
memory config 7213
memory config 7214
memory config 7213
memory config 7213
memory config 7213
memory config 7213
memory config 7213
memory config 7213
memory config 7213
memory config 7215
memory config 7218
memory config 72A0
memory test 71A1
memory test 71A2
memory test 71A4
memory test 71A5
memory test 71A6
memory test 71A3
memory test 71A4
memory test 71A4
memory test 71A5
memory test 71A5
memory test 71A6
memory test 71A6
memory config 7210
I/O INIT 8701
I/O test 8118
I/O test 8118
I/O INIT 8701
I/O INIT 8701
I/O INIT 8701
I/O INIT 8701
I/O INIT 8701
memory config 7240
memory INIT 7702
memory config 7242
memory INIT 7745
memory config 72A0
memory test 71A1
memory test 71A2
memory test 71A4
memory test 71A5
memory test 71A6
memory test 71A3
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A4
memory test 71A5
memory test 71A5
memory test 71A5
memory test 71A5

***** EARLY BOOT VFP : SYSTEM ALERT *****
SYSTEM NAME: uninitialized
DATE: 08/16/2006 TIME: 14:28:04
ALERT LEVEL: 7 = reserved

REASON FOR ALERT
SOURCE: 7 = memory
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: FF
PROBLEM DETAIL: 0 = no problem detail

LEDs: RUN ATTENTION FAULT REMOTE POWER
FLASH FLASH ON OFF ON
LED State: Boot Failed. Running non-OS code. Non-critical error
detected.
Check Chassis and Console Logs for error messages.

0x7000007070FFAC34 C3808000 00000000 - type 14 = Problem Detail
0x5800087070FFAC34 00006A07 100E1C04 - type 11 = Timestamp 08/16/2006
14:28:04

this points more to a bad memory and not a bad IO - what do you think?

[Sameer_Nirmal]

Looking at the chassis codes, the culprit looks to be the DIMM in slot 1. The DIMM has failed and is apprently the cause of the HPMC. It makes sense as why you don't get to BCH as you have failed DIMM 1.

The crash event is usually caught by Monarch CPU in the system and it is FFFFFFFF'FFFA0000 in your server. The Monarch CPU does provide all related registers contents at the point of HPMC and that what you see as per your first post. Intepreting those registers and pin-pointing to the cause of problem can only be done by HP in this case.

However, there is decoder inbuilt in GSP firmware which decodes the event chassis codes and give you info in the formatted manner and that's what you see as per in your second post. I assumed that you only got one event.

[moonchild]

Sameer,

thanks for the info.

In fact I got several messages similar to the one I posted but I only posted one.

once again, thank you.