topic Re: a Problem with Integrity Superdome in Integrity Servers

a Problem with Integrity Superdome

Hu Songyun — Wed, 03 Mar 2010 02:57:51 GMT

a partition in Integrity Superdome reboot suddenly at 23:56 yesterday.

logging on MP,could see the log(SL):

Log Entry 24623: 03/02/2010 16:32:20
Alert level 2: Informational
Keyword: FPAR_VAR_INVALID_PARM
A CPU that passes an invalid parameter to SetVariable() for one of the controlle
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 0
Problem Detail: 0x0000000000000000
0x4e80168500e00b03 0x0000000000000000
0x4b00168500e00b04 0x010000004b8d3d94

Log Entry 24622: 03/02/2010 16:24:09
Alert level 3: Warning
Keyword: HP-UX_DUMP_STATUS
OS dump status (EFxx)
Reporting Entity: HP Unix located in cabinet 0, slot 0, cpu 0
Legacy PA HEX Code: 0xaef00
0x7f80033900e00b01 0x00000000000aef00
0x6b00033900e00b02 0x010000004b8d3ba9

Log Entry 24620: 03/02/2010 16:23:03
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x000000000546b3f0
0x5680007224e00afe 0x000000000546b3f0
0x4b00007224e00aff 0x010000004b8d3b67

Log Entry 24620: 03/02/2010 16:23:03
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x000000000546b3f0
0x5680007224e00afe 0x000000000546b3f0
0x4b00007224e00aff 0x010000004b8d3b67

is this a problem with hardware?(eg.CPU problem)
give me a help!thank you.

Re: a Problem with Integrity Superdome

Sameer_Nirmal — Wed, 03 Mar 2010 03:44:36 GMT

The SL log doesn't look like complete. I would check the all SL error log entries before the time of the system reboot.

Since it looks like that the system crash dump was successful, I would examine the dump to know the cause of the problem.

Also checking the event log from within HP-UX besides OLDsyslog.log, /etc/shutdownlog and /var/tombtones/mca{date} would help.

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:03:48 GMT

i saw the all SL error log entries before the time of the system reboot.
the system reboot time was 23:56 approximately,but the SL log Entry time ended 16:32. No SL log was selected before the time that system reboot.

upload the SL error log before 16:32:

Log Entry 24622: 03/02/2010 16:24:09
Alert level 3: Warning
Keyword: HP-UX_DUMP_STATUS
OS dump status (EFxx)
Reporting Entity: HP Unix located in cabinet 0, slot 0, cpu 0
Legacy PA HEX Code: 0xaef00
0x7f80033900e00b01 0x00000000000aef00
0x6b00033900e00b02 0x010000004b8d3ba9

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24622: 03/02/2010 16:24:09
Alert level 3: Warning
Keyword: HP-UX_DUMP_STATUS
OS dump status (EFxx)
Reporting Entity: HP Unix located in cabinet 0, slot 0, cpu 0
Legacy PA HEX Code: 0xaef00
0x7f80033900e00b01 0x00000000000aef00
0x6b00033900e00b02 0x010000004b8d3ba9

Log Entry 24621: 03/02/2010 16:23:04
Keyword: IPMI Type-02 Event
0x214b8d3b68020b00 0xff0f016f00200300

Log Entry 24620: 03/02/2010 16:23:03
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x000000000546b3f0
0x5680007224e00afe 0x000000000546b3f0
0x4b00007224e00aff 0x010000004b8d3b67

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:05:17 GMT

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24620: 03/02/2010 16:23:03
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x000000000546b3f0
0x5680007224e00afe 0x000000000546b3f0
0x4b00007224e00aff 0x010000004b8d3b67

Log Entry 24619: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0x000000000546b3f0
0x5680007226e00afc 0x000000000546b3f0
0x4b00007226e00afd 0x010000004b8d3b66

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24619: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0x000000000546b3f0
0x5680007226e00afc 0x000000000546b3f0
0x4b00007226e00afd 0x010000004b8d3b66

Log Entry 24618: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0x000000000546b3f0
0x5680007222e00afa 0x000000000546b3f0
0x4b00007222e00afb 0x010000004b8d3b66

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24618: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0x000000000546b3f0
0x5680007222e00afa 0x000000000546b3f0
0x4b00007222e00afb 0x010000004b8d3b66

Log Entry 24617: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 6
Implementation Dependent: 0x000000000546b3f0
0x5680007206e00af8 0x000000000546b3f0
0x4b00007206e00af9 0x010000004b8d3b66

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24617: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 6
Implementation Dependent: 0x000000000546b3f0
0x5680007206e00af8 0x000000000546b3f0
0x4b00007206e00af9 0x010000004b8d3b66

Log Entry 24616: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 4
Implementation Dependent: 0x000000000546b3f0
0x5680007204e00af6 0x000000000546b3f0
0x4b00007204e00af7 0x010000004b8d3b66

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24616: 03/02/2010 16:23:02
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 4
Implementation Dependent: 0x000000000546b3f0
0x5680007204e00af6 0x000000000546b3f0
0x4b00007204e00af7 0x010000004b8d3b66

Log Entry 24615: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
Implementation Dependent: 0x000000000546b3f0
0x5680007220e00af4 0x000000000546b3f0
0x4b00007220e00af5 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24615: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
Implementation Dependent: 0x000000000546b3f0
0x5680007220e00af4 0x000000000546b3f0
0x4b00007220e00af5 0x010000004b8d3b65

Log Entry 24614: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0x00001010086ae01a
0x56800f5926e00af2 0x00001010086ae01a
0x4b000f5926e00af3 0x010000004b8d3b65

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:05:50 GMT

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24614: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0x00001010086ae01a
0x56800f5926e00af2 0x00001010086ae01a
0x4b000f5926e00af3 0x010000004b8d3b65

Log Entry 24613: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0xe000000000697f20
0x56800f5826e00af0 0xe000000000697f20
0x4b000f5826e00af1 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24613: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0xe000000000697f20
0x56800f5826e00af0 0xe000000000697f20
0x4b000f5826e00af1 0x010000004b8d3b65

Log Entry 24612: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
0x4b00023326e00aef 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24612: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
0x4b00023326e00aef 0x010000004b8d3b65

Log Entry 24611: 03/02/2010 16:23:01
Alert level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0x0000000000000016
0xf680007926e00aed 0x0000000000000016
0xeb00007926e00aee 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24611: 03/02/2010 16:23:01
Alert level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 6
Implementation Dependent: 0x0000000000000016
0xf680007926e00aed 0x0000000000000016
0xeb00007926e00aee 0x010000004b8d3b65

Log Entry 24610: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x00001010086ae01a
0x56800f5924e00aeb 0x00001010086ae01a
0x4b000f5924e00aec 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24610: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x00001010086ae01a
0x56800f5924e00aeb 0x00001010086ae01a
0x4b000f5924e00aec 0x010000004b8d3b65

Log Entry 24609: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0xe00000000146a3d0
0x56800f5824e00ae9 0xe00000000146a3d0
0x4b000f5824e00aea 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24609: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0xe00000000146a3d0
0x56800f5824e00ae9 0xe00000000146a3d0
0x4b000f5824e00aea 0x010000004b8d3b65

Log Entry 24608: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
0x4b00023324e00ae8 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24608: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
0x4b00023324e00ae8 0x010000004b8d3b65

Log Entry 24607: 03/02/2010 16:23:01
Alert level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x0000000000000014
0xf680007924e00ae6 0x0000000000000014
0xeb00007924e00ae7 0x010000004b8d3b65

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:06:20 GMT

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24607: 03/02/2010 16:23:01
Alert level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 4
Implementation Dependent: 0x0000000000000014
0xf680007924e00ae6 0x0000000000000014
0xeb00007924e00ae7 0x010000004b8d3b65

Log Entry 24606: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 2
Implementation Dependent: 0x000000000546b3f0
0x5680007202e00ae4 0x000000000546b3f0
0x4b00007202e00ae5 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24606: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: BR_TO_OS_INIT
Branch to OS INIT
Reporting Entity: System Firmware located in cabinet 0, slot 0, cpu 2
Implementation Dependent: 0x000000000546b3f0
0x5680007202e00ae4 0x000000000546b3f0
0x4b00007202e00ae5 0x010000004b8d3b65

Log Entry 24605: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0x00001010086ae01a
0x56800f5922e00ae2 0x00001010086ae01a
0x4b000f5922e00ae3 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24605: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0x00001010086ae01a
0x56800f5922e00ae2 0x00001010086ae01a
0x4b000f5922e00ae3 0x010000004b8d3b65

Log Entry 24604: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0xe000000000697910
0x56800f5822e00ae0 0xe000000000697910
0x4b000f5822e00ae1 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24604: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0xe000000000697910
0x56800f5822e00ae0 0xe000000000697910
0x4b000f5822e00ae1 0x010000004b8d3b65

Log Entry 24603: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
0x4b00023322e00adf 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24603: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
0x4b00023322e00adf 0x010000004b8d3b65

Log Entry 24602: 03/02/2010 16:23:01
Alert level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0x0000000000000012
0xf680007922e00add 0x0000000000000012
0xeb00007922e00ade 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24602: 03/02/2010 16:23:01
Alert level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 2
Implementation Dependent: 0x0000000000000012
0xf680007922e00add 0x0000000000000012
0xeb00007922e00ade 0x010000004b8d3b65

Log Entry 24601: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
Implementation Dependent: 0x00001410086ae01a
0x56800f5920e00adb 0x00001410086ae01a
0x4b000f5920e00adc 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24601: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IPSR
The IPSR value of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
Implementation Dependent: 0x00001410086ae01a
0x56800f5920e00adb 0x00001410086ae01a
0x4b000f5920e00adc 0x010000004b8d3b65

Log Entry 24600: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
Implementation Dependent: 0xe000000000697b00
0x56800f5820e00ad9 0xe000000000697b00
0x4b000f5820e00ada 0x010000004b8d3b65

MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 24600: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_EVENT_IIP
The IIP of the processor during an INIT event.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
Implementation Dependent: 0xe000000000697b00
0x56800f5820e00ad9 0xe000000000697b00
0x4b000f5820e00ada 0x010000004b8d3b65

Log Entry 24599: 03/02/2010 16:23:01
Alert level 2: Informational
Keyword: INIT_CAUSE_UNKNOWN
This indicates that the cause of the INIT is not known.
Reporting Entity: System Firmware located in cabinet 0, slot 2, cpu 0
0x4b00023320e00ad8 0x010000004b8d3b65

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:14:59 GMT

MP-->sl-->D: Dump log starting at current block for capture and analysis:

then get the information:

24575 SFW 0,0,0 2 4e80168500e00aad 0000000000000000 FPAR_VAR_INVALID_PARM
24575 01/12/2010 10:16:03
24576 214b4c4c1a020aaf ff0f066f001f0300 IPMI Type-02 Event
24576 01/12/2010 10:16:58
24577 SFW 0,0,0 2 4e80141000e00ab0 0000000000000009 CMC_DCACHE_SUMMARY
24577 01/20/2010 13:05:29
24578 SFW 0,2,4 2 4e80141024e00ab2 0000000000000009 CMC_DCACHE_SUMMARY
24578 02/10/2010 00:59:36
24579 SFW 0,0,0 *7 f680007900e00ab4 0000000000000000 INIT_INITIATED
24579 03/02/2010 16:23:00
24580 SFW 0,0,0 2 4b00023300e00ab6 010000004b8d3b64 INIT_CAUSE_UNKNOWN
24580 03/02/2010 16:23:00
24581 SFW 0,0,0 2 56800f5800e00ab7 e000000000697930 INIT_EVENT_IIP
24581 03/02/2010 16:23:00
24582 SFW 0,0,0 2 56800f5900e00ab9 00001010086ae01a INIT_EVENT_IPSR
24582 03/02/2010 16:23:00
24583 SFW 0,0,0 2 568016a900e00abb 0000000000000000 INIT_EVENT_MONARCH
24583 03/02/2010 16:23:00
24584 SFW 0,0,0 2 4e80122f00e00abd 0000030100001140 OS_INIT_CHECKSUM_PASSED
24584 03/02/2010 16:23:01
24585 SFW 0,0,0 2 5680007200e00abf 000000000546b3f0 BR_TO_OS_INIT
24585 03/02/2010 16:23:01
24586 SFW 0,0,2 *7 f680007902e00ac1 0000000000000002 INIT_INITIATED
24586 03/02/2010 16:23:01
24587 SFW 0,0,2 2 4b00023302e00ac3 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24587 03/02/2010 16:23:01
24588 SFW 0,0,2 2 56800f5802e00ac4 c000000000225110 INIT_EVENT_IIP
24588 03/02/2010 16:23:01
24589 SFW 0,0,2 2 56800f5902e00ac6 00001013086ae01a INIT_EVENT_IPSR
24589 03/02/2010 16:23:01
24590 SFW 0,0,4 *7 f680007904e00ac8 0000000000000004 INIT_INITIATED
24590 03/02/2010 16:23:01
24591 SFW 0,0,4 2 4b00023304e00aca 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24591 03/02/2010 16:23:01
24592 SFW 0,0,4 2 56800f5804e00acb e000000000698140 INIT_EVENT_IIP
24592 03/02/2010 16:23:01
24593 SFW 0,0,4 2 56800f5904e00acd 00001010086ae01a INIT_EVENT_IPSR
24593 03/02/2010 16:23:01
24594 SFW 0,0,6 *7 f680007906e00acf 0000000000000006 INIT_INITIATED
24594 03/02/2010 16:23:01
24595 SFW 0,0,6 2 4b00023306e00ad1 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24595 03/02/2010 16:23:01
24596 SFW 0,0,6 2 56800f5806e00ad2 e000000000697b40 INIT_EVENT_IIP
24596 03/02/2010 16:23:01
24597 SFW 0,0,6 2 56800f5906e00ad4 00001210086ae01a INIT_EVENT_IPSR
24597 03/02/2010 16:23:01
24598 SFW 0,2,0 *7 f680007920e00ad6 0000000000000010 INIT_INITIATED
24598 03/02/2010 16:23:01
24599 SFW 0,2,0 2 4b00023320e00ad8 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24599 03/02/2010 16:23:01
24600 SFW 0,2,0 2 56800f5820e00ad9 e000000000697b00 INIT_EVENT_IIP
24600 03/02/2010 16:23:01
24601 SFW 0,2,0 2 56800f5920e00adb 00001410086ae01a INIT_EVENT_IPSR
24601 03/02/2010 16:23:01
24602 SFW 0,2,2 *7 f680007922e00add 0000000000000012 INIT_INITIATED
24602 03/02/2010 16:23:01
24603 SFW 0,2,2 2 4b00023322e00adf 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24603 03/02/2010 16:23:01
24604 SFW 0,2,2 2 56800f5822e00ae0 e000000000697910 INIT_EVENT_IIP
24604 03/02/2010 16:23:01
24605 SFW 0,2,2 2 56800f5922e00ae2 00001010086ae01a INIT_EVENT_IPSR
24605 03/02/2010 16:23:01
24606 SFW 0,0,2 2 5680007202e00ae4 000000000546b3f0 BR_TO_OS_INIT
24606 03/02/2010 16:23:01
24607 SFW 0,2,4 *7 f680007924e00ae6 0000000000000014 INIT_INITIATED
24607 03/02/2010 16:23:01
24608 SFW 0,2,4 2 4b00023324e00ae8 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24608 03/02/2010 16:23:01
24609 SFW 0,2,4 2 56800f5824e00ae9 e00000000146a3d0 INIT_EVENT_IIP
24609 03/02/2010 16:23:01
24610 SFW 0,2,4 2 56800f5924e00aeb 00001010086ae01a INIT_EVENT_IPSR
24610 03/02/2010 16:23:01
24611 SFW 0,2,6 *7 f680007926e00aed 0000000000000016 INIT_INITIATED
24611 03/02/2010 16:23:01
24612 SFW 0,2,6 2 4b00023326e00aef 010000004b8d3b65 INIT_CAUSE_UNKNOWN
24612 03/02/2010 16:23:01
24613 SFW 0,2,6 2 56800f5826e00af0 e000000000697f20 INIT_EVENT_IIP
24613 03/02/2010 16:23:01
24614 SFW 0,2,6 2 56800f5926e00af2 00001010086ae01a INIT_EVENT_IPSR
24614 03/02/2010 16:23:01
24615 SFW 0,2,0 2 5680007220e00af4 000000000546b3f0 BR_TO_OS_INIT
24615 03/02/2010 16:23:01
24616 SFW 0,0,4 2 5680007204e00af6 000000000546b3f0 BR_TO_OS_INIT
24616 03/02/2010 16:23:02
24617 SFW 0,0,6 2 5680007206e00af8 000000000546b3f0 BR_TO_OS_INIT
24617 03/02/2010 16:23:02
24618 SFW 0,2,2 2 5680007222e00afa 000000000546b3f0 BR_TO_OS_INIT
24618 03/02/2010 16:23:02
24619 SFW 0,2,6 2 5680007226e00afc 000000000546b3f0 BR_TO_OS_INIT
24619 03/02/2010 16:23:02
24620 SFW 0,2,4 2 5680007224e00afe 000000000546b3f0 BR_TO_OS_INIT
24620 03/02/2010 16:23:03
24621 214b8d3b68020b00 ff0f016f00200300 IPMI Type-02 Event
24621 03/02/2010 16:23:04
24622 HPUX 0,0,0 *3 7f80033900e00b01 00000000000aef00 HP-UX_DUMP_STATUS
24622 03/02/2010 16:24:09
24623 SFW 0,0,0 2 4e80168500e00b03 0000000000000000 FPAR_VAR_INVALID_PARM
24623 03/02/2010 16:32:20
24624 214b8d3dd7020b05 ff0f066f001f0300 IPMI Type-02 Event
24624 03/02/2010 16:33:27

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:31:32 GMT

the OLDsyslog.log information before the partition reboot:

Mar 2 14:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 14:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 14:40:45 erpdb02 above message repeats 19 times
Mar 2 14:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 14:45:10 erpdb02 vmunix: 0/0/10/1/0: Unable to access previously accessed device at nport ID 0x1de00.
Mar 2 14:45:10 erpdb02 avrd[2745]: sctl driver not installed or not configured, using normal open() for scanning
Mar 2 14:45:10 erpdb02 EMS [4517]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/adapters/events/TL_adapter/0_0_10_1_0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 296026124 -r /adapters/events/TL_adapter/0_0_10_1_0 -n 296026114 -a
Mar 2 14:45:00 erpdb02 su: + tty?? root-orarun
Mar 2 14:45:10 erpdb02 above message repeats 4 times
Mar 2 14:46:00 erpdb02 su: + tty?? root-orarun
Mar 2 15:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 15:00:45 erpdb02 above message repeats 14 times
Mar 2 15:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 15:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 15:20:45 erpdb02 above message repeats 19 times
Mar 2 15:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 15:40:01 erpdb02 su: + tty?? root-orarun
Mar 2 15:40:45 erpdb02 above message repeats 19 times
Mar 2 15:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 16:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 16:00:46 erpdb02 above message repeats 19 times
Mar 2 16:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 16:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 16:20:46 erpdb02 above message repeats 19 times
Mar 2 16:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 16:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 16:40:46 erpdb02 above message repeats 19 times
Mar 2 16:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 17:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 17:00:46 erpdb02 above message repeats 19 times
Mar 2 17:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 17:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 17:20:46 erpdb02 above message repeats 19 times
Mar 2 17:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 17:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 17:40:46 erpdb02 above message repeats 19 times
Mar 2 17:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 18:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 18:00:46 erpdb02 above message repeats 19 times
Mar 2 18:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 18:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 18:20:46 erpdb02 above message repeats 19 times
Mar 2 18:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 18:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 18:40:46 erpdb02 above message repeats 19 times
Mar 2 18:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 19:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 19:00:46 erpdb02 above message repeats 19 times
Mar 2 19:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 19:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 19:20:46 erpdb02 above message repeats 19 times
Mar 2 19:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 19:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 19:40:46 erpdb02 above message repeats 19 times
Mar 2 19:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 20:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 20:00:46 erpdb02 above message repeats 23 times
Mar 2 20:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 20:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 20:20:46 erpdb02 above message repeats 19 times
Mar 2 20:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 20:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 20:40:46 erpdb02 above message repeats 19 times
Mar 2 20:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 21:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 21:00:46 erpdb02 above message repeats 19 times
Mar 2 21:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 21:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 21:20:46 erpdb02 above message repeats 19 times
Mar 2 21:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 21:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 21:40:47 erpdb02 above message repeats 19 times
Mar 2 21:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 22:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 22:00:47 erpdb02 above message repeats 19 times
Mar 2 22:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 22:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 22:20:47 erpdb02 above message repeats 19 times
Mar 2 22:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 22:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 22:40:47 erpdb02 above message repeats 19 times
Mar 2 22:41:00 erpdb02 su: + tty?? root-orarun
Mar 2 23:00:00 erpdb02 su: + tty?? root-orarun
Mar 2 23:00:47 erpdb02 above message repeats 20 times
Mar 2 23:01:00 erpdb02 su: + tty?? root-orarun
Mar 2 23:20:00 erpdb02 su: + tty?? root-orarun
Mar 2 23:20:47 erpdb02 above message repeats 19 times
Mar 2 23:21:00 erpdb02 su: + tty?? root-orarun
Mar 2 23:40:00 erpdb02 su: + tty?? root-orarun
Mar 2 23:40:47 erpdb02 above message repeats 19 times
Mar 2 23:41:00 erpdb02 su: + tty?? root-orarun
Mar 3 00:00:00 erpdb02 su: + tty?? root-orarun
Mar 3 00:00:00 erpdb02 above message repeats 19 times
Mar 3 00:00:55 erpdb02 syslog: Oracle CSS family monitor shutting down. 3
Mar 3 00:00:56 erpdb02 su: + tty?? root-orarun
Mar 3 00:00:56 erpdb02 syslog: Cluster Ready Services completed waiting on dependencies.
Mar 3 00:00:57 erpdb02 syslog: Duplicate clsomon found. Restarting.
Mar 3 00:00:57 erpdb02 syslog: Oracle clsomon shutdown successful.
Mar 3 00:00:57 erpdb02 syslog: Oracle CSSD shell script failure. Duplicate CSSD.
Mar 3 00:00:57 erpdb02 su: + tty?? root-orarun
Mar 3 00:00:57 erpdb02 above message repeats 2 times
Mar 3 00:00:57 erpdb02 syslogd: restart
Mar 3 00:00:59 erpdb02 vmunix:
Mar 3 00:00:59 erpdb02 vmunix: Oracle CRS TOC for clusterware integrity...

seemingly,some problem with Oracle caused the system crash.

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 01:34:05 GMT

the information selected from the shutdownlog.log:

14:29 Fri Dec 8, 2006. Reboot: (by gp230-01!root)
14:48 Mon Jun 4, 2007. Reboot:
16:02 Mon Jun 4, 2007. Reboot:
16:08 Mon Jun 4, 2007. Reboot:
16:27 Mon Jun 4, 2007. Reboot: (by erpdb02!root)
16:40 Mon Jun 4, 2007. Reboot: (by SAM)
16:40 Mon Jun 4, 2007. Reboot:
17:39 Tue Jun 12, 2007. Halt: (by erpdb01!root)
16:42 Wed Jun 13, 2007. Reboot: (by erpdb01!root)
10:02 Thu Jun 14, 2007. Reboot: (by erpdb01!root)
15:05 Thu Jun 14, 2007. Reboot: (by erpdb01!root)
17:43 Thu Jun 14, 2007. Reboot: (by erpdb01!root)
16:30 Sat Jun 16, 2007. Reboot: (by erpdb02!root)
16:52 Sat Jun 16, 2007. Reboot: (by erpdb02!root)
17:43 Sat Jun 16, 2007. Reboot: (by erpdb02!root)
18:40 Sat Jun 16, 2007. Reboot: (by erpdb02!root)
20:23 Sat Jun 16, 2007. Reboot: (by erpdb02!root)
21:02 Sat Jun 16, 2007. Reboot: (by erpdb02!root)
09:35 Sun Jun 17, 2007. Reboot: (by erpdb02!root)
12:00 Sun Jun 17, 2007. Reboot: (by erpdb02!root)
13:13 Sun Jun 17, 2007. Reboot: (by erpdb02!root)
17:32 Sun Jun 17, 2007. Reboot: (by erpdb02!root)
10:28 Mon Jun 18, 2007. Reboot: (by erpdb02!root)
13:11 Mon Jun 18, 2007. Reboot: (by erpdb02!root)
13:52 Mon Jun 18, 2007. Reboot:
13:52 Mon Jun 18, 2007. Reboot:
12:33 Tue Jun 19, 2007. Reboot: (by erpdb02!root)
17:52 Thu Jun 28, 2007. Reboot: (by erpdb02!root)
15:24 Sat Sep 8, 2007. Reboot: (by erpdb02!root)
16:41 Sat Nov 3, 2007. Halt: (by erpdb02!root)
16:59 Thu Feb 19, 2009. Halt: (by erpdb02!oper22)
18:02 Thu Jun 11 2009. Reboot after panic: INIT, IIP:0xe000000000697fa0 IFA:0x0000000000000045
07:47 Thu Sep 10 2009. Reboot after panic: INIT, IIP:0xe000000000697fa0 IFA:0x200000004005521c
09:24 Fri Jan 8, 2010. Halt: (by erpdb02!oper)
17:57 Tue Jan 12 2010. Reboot after panic: INIT, IIP:0xe00000000146a3d0 IFA:0x64645f656e747279
00:12 Wed Mar 03 2010. Reboot after panic: INIT, IIP:0xe000000000697930 IFA:0x400000000156cd60

Re: a Problem with Integrity Superdome

Sameer_Nirmal — Thu, 04 Mar 2010 03:27:13 GMT

Yes, it does seem that the reboot was caused by a INIT as a result of Oracle CRS TOC. The entry in the OLDsyslog.log "Oracle CRS TOC for clusterware integrity" indicates that.

You need to engage Oracle support to look into Oracle CRS TOC problem.

Re: a Problem with Integrity Superdome

Hu Songyun — Thu, 04 Mar 2010 07:30:08 GMT

i see. thanks in advance!