L1000 hangs - ccbootlog shows arcane codes.

[Eric Hagen]

I have a system that is hanging quite often, and it often requires a power down and a unplugging of the power cords to reset it to allow it to boot. This is a remote system so I am not able to manually go through the GSP.
The ccbootlog shows a critcal error 13 - but I see no place that explains the hex codes.

The decoded CE logs do not show anything on the HP-UX error, and points to the I/O system board.

What I am wondering is if it could be the hard drive - Single disk system. Or is it really the system board?


17536 HPUX 0,0,0 *13 0x78e001d41100f000 0x0000000300000000

17536 HPUX 0,0,0 *13 0x58e009d41100f000 0x00006b061814281d 07/24/2007 20:40:29

17537 GSP 0 1 0xf010021166002030 0x0000000000000000

17537 GSP 0 1 0x58100a1166002030 0x0000460001000000 01/01/1970 00:00:00

=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=
Timestamp : 07/24/0x7d7 20:40:29
A Processor Field Replaceable Unit (FRU) has failed.
=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=
Timestamp : 01/01/0x7b2 00:00:00
The Core I/O Field Replaceable Unit (FRU) has failed.
Recommended Action:
The Service Processor is on the Core I/O board. Replace the Core I/O.

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v
Chassis Code Encoded Field : 0xf010021166002030
Alert Level : 0x1 (Service processor alert, no failure detected,
forward progress logs and timer requests only.)
Source FRU : 0x6 (platform entity)
Source FRU Detail : 0x6 (service processor)
Source ID : 0x0 (platform dependent)
Event Detail : 0x1 (selftest result)
Caller Activity : 0x2 (operation)
Caller Subactivity : 0x3 (console)
Activity Status : 0x0 (implementation dependent)
Reporting Entity Type : 0x1 (service processor)
Reporting Entity ID : 0x0 Cabinet 0x0 Cell ! CPU !
Data Type : ! (!)
Message ID : !
Domain Name : !
Chassis Code Data Field : 000000000000000000
Source : 0x0 (TBD)
Reset Type : 0x0 (last reset was a cold reset)

=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=
-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=
Timestamp : 01/01/0x7b2 00:00:00
The Core I/O Field Replaceable Unit (FRU) has failed.
Recommended Action:

The Service Processor is on the Core I/O board. Replace the Core I/O.

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v
Chassis Code Encoded Field : 0xf010051166002030
Alert Level : 0x1 (Service processor alert, no failure detected,
forward progress logs and timer requests only.)
Source FRU : 0x6 (platform entity)
Source FRU Detail : 0x6 (service processor)
Source ID : 0x0 (platform dependent)
Event Detail : 0x1 (selftest result)
Caller Activity : 0x2 (operation)
Caller Subactivity : 0x3 (console)
Activity Status : 0x0 (implementation dependent)
Reporting Entity Type : 0x1 (service processor)
Reporting Entity ID : 0x0 Cabinet 0x0 Cell ! CPU !
Data Type : ! (!)
Message ID : !
Domain Name : !
Chassis Code Data Field : 000000000000000000
Source : 0x0 (TBD)
Reset Type : 0x0 (last reset was a cold reset)

[Steven E. Protter]

Shalom,

GSP has a network interface and can be connected to a network switch with a standard RJ45 cable and then configured on site.

Then, like magic it is network available and you can run GSP from your PC at the remote facility. That would be worth it to see the logs, which give clues to possible hardware issues.

You can see some things by running online diagnostics, mstm cstm or xstm

SEP

[Sameer_Nirmal]

Looking at the chassis code details, it seems that the server's console is getting hung.

I would look for what is connected to console serial port of GSP or GSP & PDC fw version and LED indications on GSP card when the system gets hung which would be useful to know whats going on.

It may be a case like say a mal-funcationing dump terminal/UPS connected to GSP console serial port is causing the port hung? If this is not the case, then you may need to replace the GSP card.

The probability of f/w issue ( even it is not latest ) is low in here as it was working normal before and suddenly the issue pop-up'd yes? However it is always recommended to have latest firmware installed.

[Eric Hagen]

Thanks for the information.

The console is going to a switch box, and none of the other L1000 / L2000 systems are experiencing a problem. Right now I will be in a holding pattern until the next lockup to get details on the front console and the GSP lights. From there I will hopefully be able to identify if it is the GSP or some other usual suspect like the hard drive.

With any luck, it will not hang again. But I'm not very lucky.

[Matti_Kurkela]

Just a thought.

If you get the GSP connected to the network, check the GSP firmware version. It's visible at the top of the GSP's help menu: connect to the GSP, press Ctrl-B to access the GSP menus, then type HE for "help".

Updating a GSP firmware requires a network connection. If your GSP has had no network connection so far, it's likely the firmware has never been updated. The older L class GSPs had some firmware bugs that could make the GSP unresponsive.

Note that the GSP of a L class servers can be either hardware version A or B. The first letter of the firmware version number identifies the GSP hardware version. The L1000 is likely to have a version A GSP, but you should check to be sure.

When HP moved the firmware patches to the Business Support Center, the GSP firmware patches became slightly hard to find.

This thread has the links to the latest version A GSP firmware:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1096548

(Yes, I know it mentions the N class. But the exact same firmware applies to the L class too: read the linked PF_CPREGSPA0112.txt file to confirm it.)

The latest version B firmware should be available through the Business Support Center:
http://www.hp.com/go/support

MK