TOC error

[Javier Ortiz Guajardo]

Could MC-Service Guard to cause a TOC Error and reboot the system by a TIMEOUT NODE?

this is the content of TOC
thanks.

******* Unexpected HPMC/TOC. Processor HPA FFFFFFFF'FC1B0000 *******
GENERAL REGISTERS:
r00/03 00000000'00000000 00000000'00847570 00000000'00357414 00000000'0068E7
B8
r04/07 00000000'07FFCA00 00000000'0068D590 00000000'07BF9000 00000000'000000
01
r08/11 00000000'007D0740 00000000'00000004 00000000'00000001 00000000'000008
00
r12/15 00000000'2A41F800 80000001'00108408 00000000'00000133 00000000'2A41F8
00
r16/19 00000000'000001EE 0804001A'00000000 00000000'00000007 00000000'000100
00
r20/23 00000000'0068E758 00000000'00000001 00000000'000000F0 00000000'000100
08
r24/27 00000000'00000202 00000000'00000000 00000000'00000000 00000000'00848D
70
r28/31 00000000'00000000 00000000'00DC4000 00000000'07FFCB80 00000000'5C9800
00
CONTROL REGISTERS:
sr0/3 00000000'00000000 00000000'047EB800 00000000'00000000 00000000'000000
00
sr4/7 00000000'00000000 00000000'047EB800 00000000'0AB09800 00000000'000000
00
pcq = 00000000'00000000.00000000'003B21F8 00000000'00000000.00000000'003B21
FC
isr = 00000000'1434001F ior = 00000000'FF3FCBA8 iir = 0AFD0213 rctr = 7FF25
0F7

pid reg cr8/cr9 0000D3FC'00007973 0000559E'0000A812
pid reg cr12/cr13 000079B0'00007020 00002837'0000448E
ipsw = 00000000'08000008 iva = 00000000'0002C000 sar = 1C ccr = C0
tr0/3 00000000'00F9F378 00000000'0002C000 00000000'07F352B0 80000001'00118E
20
tr4/7 00000000'00000380 00017F29'DA296D27 00000000'0016E2B8 00000000'07FFC3
B0
eiem = 00000007'00000000 eirr = 80000488'00000000 itmr = 00017F29'F25CBC9B

cr1/4 00000000'00000000 00000000'00000000 00000000'00000000 00000000'000000
00
cr5/7 00000000'00000000 00000000'00000000 00000000'00000000
MACHINE CHECK PARAMETERS:
Check Type = 00000000 CPU STATE = 1E000002 Cache Check = 00000000
TLB Check = 00000000 Bus Check = 00000000 PIM State = ? SIU Status = ???????
?
Assists = 00000000 Processor = 00000000
Slave Addr = 00000000'00000000 Master Addr = 00000000'00000000


HPMC, pcsq.pcoq = 0'0.0'3b21f8 , isr.ior = 0'1434001f.0'ff3fcba8
@(#)B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997
High-priority machine check: (display==0xd910, flags==0x0)
Processor 1 TOC: pcsq.pcoq = 0'0.0'34c5bc
isr.ior = 0'340001.0'f53d5d58
Processor 2 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'340001.0'f53d5d58
Processor 3 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'340001.0'f53d5d58
Processor 4 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'340001.0'f53d5d58
Processor 5 TOC: pcsq.pcoq = 0'0.0'34c5b8
isr.ior = 0'340001.0'f53d5d58
Processor 6 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'0.0'0
Processor 7 TOC: pcsq.pcoq = 0'0.0'34c5b0
isr.ior = 0'340001.0'f53d5d58
Processor 8 TOC: pcsq.pcoq = 0'0.0'34c5bc
isr.ior = 0'340001.0'f53d5d58
Processor 9 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'0.0'0
Processor 11 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'0.0'0
Processor 12 TOC: pcsq.pcoq = 0'0.0'34c5bc
isr.ior = 0'340001.0'f53d5d58
Processor 13 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'0.0'0
Processor 14 TOC: pcsq.pcoq = 0'0.0'34c5bc
isr.ior = 0'340001.0'f53d5d58
Processor 15 TOC: pcsq.pcoq = 0'0.0'34c5ac
isr.ior = 0'0.0'0
Processor 16 TOC: pcsq.pcoq = 0'0.0'34c5b4
isr.ior = 0'0.0'0
* CPU State 0x1e000002

[Pramod_4 · Accepted solution]

Yes it will, provided that you have configured the following in the package config:

"NODE_FAIL_FAST_ENABLED YES"

In the event of a package failure on a node, that node will be halted/TOCed.

Thanks,
Pramod

[melvyn burnard]

SG could and does cause a system TOC, if certain conditions occur, but are you sure this was an SG TOC?
looking at what you have provided does not make this clear.
what does the last few lines of the /etc/shutdownlog say?
Also, if htere was a dump, what does the INDEX file of hte dump tell you?
Looking at what you have provided, I am more inclined to think you had an HPMC, or hardware failure,

[Helen French]

Eventhough MC/SG can cause a TOC, I would suspect a hardware error in this case since you had an HPMC error. Check your crash dump files, system PIM informations. Look for HPMC errors and time stamps. Get the help of hardware support if needed.

[Pramod_4]

As your TOC error says, HPMC, have a HP CE look at it to rule out any HW issue. Did you take a look at /var/tombstone/ts99 and /var/adm/crash/INDEX files ?

To find out wheher it is SG TOC, few things to consider are:

+ Was there any broken heartbeat ?
+ Was there any NIC/LAN failure ?
+ Is NOD_FAIL_FAST_ENABLED set to YES ? with services configured ?
+ Is there NIC network switching occured message in OLDsyslog TOCed box ?
+ Was either servers CPU utilization 100% at the time of the TOC ?
..

Pramod

[Javier Ortiz Guajardo]

The shutdownlog show
Reboot after panic: , isr.ior = 0'1434001f.0'ff3fcba8

the question about MS-SG TOC was made because someone told us that maybe it was a TIMEOUT NODE in MS-Service Guard, i am not agree with that, An i didn´t received any message about time out node in syslog.

Could this message bring me an idea about what Hardware has failed?

Thank

[Helen French]

Again, Check your system PIM information and it should have the latest HPMC errors. It will tell you which hardware is failed. You can get to these menus from Boot console menu. You may also run Offline or Online diags to determine the failed hardware type and details.

If there is a problem, call the hardware support.