System led Flash red on HP-UX 11.23 rx2660 server

[Miguel Silva Rentes]

Hi everyone!

I'm having a problem on a HP-UX 11.23 rx2660 server where the system led started getting flash red and I promptly accessed the MP card to see what was going on. Here is the SL transcript that is worrying me:

Log Entry 0: 23 Jan 2011 10:35:53
Alert Level 7: Fatal
Keyword: Type-02 136f03 1273603
INIT Initiated
Logged by: Redundant w/ an E0 code;
Sensor: Critical Interrupt
Data1: Software NMI
Data2: OEM Code1: 0x3FOEM Code2: 0x00
0xC14D3C0489020010 003FA36F00130300

Log Entry 1: 23 Jan 2011 10:35:53
Alert Level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Logged by: System Firmware 0
Data: Major change in system state
0xF480007900E00020 0000000000000000

Log Entry 3: 23 Jan 2011 10:35:53
Alert Level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Logged by: System Firmware 4
Data: Major change in system state - Housekeeping On
0xF480007904E00050 0000000000000004

Log Entry 5: 23 Jan 2011 10:35:54
Alert Level 7: Fatal
Keyword: INIT_INITIATED
INIT initiated
Logged by: System Firmware 6
Data: Major change in system state - BCH or EFI
0xF480007906E00080 0000000000000006


Log Entry 6: 23 Jan 2011 10:35:55
Alert Level 7: Fatal
Keyword: Type-02 206f01 2125569
OS run-time critical shutdown
Logged by: OS Software Agent;
Sensor: OS Critical Stop
Data1: Run-time Stop
0x214D3C048B0200A0 FF0F016F00200300

Log Entry 7: 23 Jan 2011 10:35:56
Alert Level 5: Critical
Keyword: HP-UX_HEX_FAULT_CODE
OS legacy PA hex fault code (Bxxx)
Logged by: HP-UX Kernel 2
Data: Legacy 20-bit PA HEX chassis code
0xBF80033802E000B0 000000000002B100


Log Entry 8: 23 Jan 2011 10:35:58
Alert Level 3: Warning
Keyword: HP-UX_CRASHDUMP_STARTED
OS crashdump started (D700)
Logged by: HP-UX Kernel 2
Data: Legacy 20-bit PA HEX chassis code
0x7F80033702E000D0 00000000000AD700

Log Entry 9: 23 Jan 2011 10:36:47
Alert Level 2: Informational
Keyword: Type-02 127002 1208322
Soft Reset
Logged by: Baseboard Management Controller;
Sensor: System Event
0x204D3C04BF0200F0 FFFF027000120300


Log Entry 10: 23 Jan 2011 10:36:57
Alert Level 0: Minor Forward Progress
Keyword: BOOT_REBOOT
CPU rebooting
Logged by: System Firmware 0
Data: Major change in system state
0x148002C500E00100 0000000000000000

Log Entry 11: 23 Jan 2011 10:39:09
Alert Level 2: Informational
Keyword: Type-02 1f6f06 2060038
OS Boot complete.
Logged by: OS Software Agent;
Sensor: OS Boot
Data1: boot completed-boot device not specified
0x214D3C054D020120 FF0F066F001F0300

Can you give me some pointers on the possible causes for this problem? Can it be solved by a patch/firmware upgrade?

Any help is great at this point.

Best regards,
Miguel Rentes

[Dennis Handly]

>Keyword: HP-UX_CRASHDUMP_STARTED
>Can you give me some pointers on the possible causes for this problem? Can it be solved by a patch/firmware upgrade?

It appears to be a kernel panic. Have you had HP analyze your crashdump?

How up to date are your patches?

[Miguel Silva Rentes]

Hi Dennis,

Thank you for your quick answer. I'm trying to understand if I can solve this problem independently from HP because that would be the ideal solution. If you can guarantee me that HP help is in fact the only solution available right now, then I'll call them straight ahead. But is it possible to load the dump file and extract more info about the cause of this possible?

Regarding the patches, the server isn't updated frequently. I can list the patches it has installed if that info is necessary.

Best regards

[Steven E. Protter]

Shalom Miguel,

I think getting a system up to date on firmware is good systems administration practice. I believe there is a reasonable chance it will help with this problem.

Your system may also need a battery replacement on its system board. It seems to be having trouble remembering its boot disk path.

SEP

[Dennis Handly]

>I'm trying to understand if I can solve this problem independently from HP because that would be the ideal solution.

If you are paying for HP-UX support, you might as well take advantage of it.

>If you can guarantee me that HP help is in fact the only solution available right now, then I'll call them straight ahead. But is it possible to load the dump file and extract more info about the cause of this possible?

Do you want to download crash analysis tools and learn how to use them? How soon do you want the problem solved?

>the server isn't updated frequently. I can list the patches it has installed if that info is necessary.

You should make sure you have a recent Quality pack installed. You can use SWA to help you with this analysis.
(I doubt anyone wants to go over your list of patches. :-) Better to have SWA analyze your patches.
On the patch database page for HP-UX there is also a web based analysis tool.

[Dennis Handly]

run a patch assessment:
http://www.itrc.hp.com/service/patch/assessSystemsPage.do

[hvhari]

If you have HP support for HPUX, have the crash dump analysed by them to find out the reason for the OS Crash. Suggestion action plans could include patch update, FW upgrade, etc.

[Jorge Pons]

Hi

There is a manual:

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01163996/c01163996.pdf

in page 110 talk about Leds and the meaning.

Regards, Jorge

[Robert_Jewell]

Something you could check is the Console Log from the MP. If you can locate the part of the log where the system crashes, perhaps you will have some idea as to why. The info would be limited, but perhaps useful?

-Bob

[Torsten.]

Based on the provided info it is not really clear IMHO. Either the system panics due to a hung or fatal I/O problem, or somebody does a TOC on the system.