A Processor Field Replaceable Unit (FRU) has failed.

[Eric Antunes]

I all ITRC addicted, ;)

I'm having serious issues with both "my" HP9000 servers: they crashes almost every day simultaneously!

I ran the logtool on STM and found the following error:

"A Processor Field Replaceable Unit (FRU) has failed.

Recommended Action:

From the Boot Console Handler Main Menu, execute the INfo (in) command.
This brings up the Info Menu. Now execute the PRocessor (pr) command. This
will report what processor is not functioning."

But after doing the recommended action, the PRocessor command shows that the CPU state is OK...

Any idea??

Best Regards,

Eric Antunes

[Eric Antunes]

One more thing:

Since they crash simultaneously, is there a way to check the UPS state/condition besides GSP->PS command?

Best Regards,

Eric Antunes

[Torsten.]

Eric,

if both servers are crashing at the same time, it could be a serious power problem. Have a look into the GSP log for power related messages.
Anyway, most servers have a redundant power supply to prevent this. What model do you have?

Regarding the UPS - it depends on the model. Some are network connected and providing an own interface to see the details.

[Michael Steele_2]

Are these new or old servers?

I'd place a HW call. Gather this information:

Model number:

# model

Check your PDC version in /var/tombstones/ts99

HW serial number which is on your delivery paper work or the outside of the machine.

Here's a thread like yours: http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=805951

[Eric Antunes]

Hi Torsten and Michael,

Thanks for the replies.

In the GSP log there is nothing besides an old 1970 message...

The UPS is a A3589B (5.5 kVA rackmt HP PowerTrust UPS). Is there a way to gather more information about this UPS?

They are 9000/800/L2000-44 and 9000/800/L2000-36 servers.

I don't think it is related with the PDC version since, being connected to the same Power Supply, they both crashes at the same time.

The servers have different PDC versions:

9000/800/L2000-36 has PDC Version: 41.38

9000/800/L2000-44 has PDC Version: 44.28

Best Regards,

Eric Antunes

[Torsten.]

How many power supplies are installed?

GSP> ps

[Torsten.]

unfortunately your UPS doesn't talk too much. There is a possibility to read the data, but this is very cryptical and I don't know all commands.

About the "old" GSP messages: In case of a power loss, the GSP has no actual time. For this reason the timestamp is 1.1.1970. Once the OS is running, it passes to current time to the GSP. So read the "old" messages!

Are there power related entries?

If you have a redundant power supply, try to connect it to a wall outlet and not to the USV.

The L2000 needs 2 PS if the box is fully loaded, but perhaps it is able to run with only 1. The third PS is for redundancy.

[Torsten.]

And BTW, you should bring the other box to PDC 44.28 too - this is not related to this problem, but will prevent others.

[Eric Antunes]

Hi Torsten,

Thanks for the replies.

They are 3 power supplies, all in "Normal" states.

Here's the same "old" message in both server GSP's (they don't seem power related):

"DATE: 01/01/1970 TIME: 00:00:00
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 3 = PDH
SOURCE DETAIL: 6 = interconnect medium SOURCE ID: 0
PROBLEM DETAIL: 3 = non-responding, may need GSP reset.

CALLER ACTIVITY: 2 = operation STATUS: 0
CALLER SUBACTIVITY: 02 = platform internal interconnect
REPORTING ENTITY TYPE: 1 = service processor REPORTING ENTITY ID: 00

0x5810082336002020 00004600 01000000 type 11 = Timestamp 01/01/1970 00:00:00
Type CR for next entry, - CR for previous entry, Q CR to quit."

I didn't understood this one: "If you have a redundant power supply, try to connect it to a wall outlet and not to the USV.". What is a wall outlet??

Best Regards,

Eric Antunes

[Torsten.]

A power outlet/socket in the wall?

Just a direct source for power - not the UPS.

Regarding the logs:

I expect something like

"SOURCE DETAIL: 4 = high voltage DC power SOURCE ID: 2
PROBLEM DETAIL: A = failed or disconnected"

[mits]

Before discussing any troubleshooting, it is better you will clarify what type of "Crash" your systems experienced. Was it software panic? Did you check the shutdown log? Also, it is better you will attach full GSP error log copy for forum members' reference. And I would like you to clarify when (date and time) the system carshed.

As to the power related system rebooting, if we need to suspect some common problem like the AC power failure that would cause multiple server rebooting, you should see the GSP event something like below. This event shows you the GSP selftest result.

Log Entry # 382 :
SYSTEM NAME: C--3-001
DATE: 08/07/2002 TIME: 02:06:48
ALERT LEVEL: 1 = Information only, no action required

SOURCE: 6 = platform
SOURCE DETAIL: 6 = service processor SOURCE ID: 0
PROBLEM DETAIL: 1 = selftest result

CALLER ACTIVITY: 2 = operation STATUS: 0
CALLER SUBACTIVITY: 03 = console
REPORTING ENTITY TYPE: 1 = service processor REPORTING ENTITY ID: 00

0xF010011166002030 00000000 00000000 type 30 = reset type and cause
0x5810091166002030 00006607 07020630 type 11 = Timestamp 08/07/2002 02:06:48

If the data field of the hex code ("00000000 00000000" in this case) is all zero, most suspect is the site AC power interrupt. You would see full system selftest results in the activity log in this case included the blower or fan events. You can check it by pulling all AC input power cables for test or turn off the brunch circuit breaker.

[freddy_21]

Could You provide :
1. GSP revision
from gsp prompt run help command.
2. Do you have /var/adm/crash when your server crash?
3. Could i know what is application run at the server?



Thanks
freddy

[Josiah Henline]

Do a "more /var/tombstones/ts99" and see if there is a valid timestamp.

Also, is there anything logged in /etc/shutdownlog. isr ior will be either a TOC or an HPMC. There may be some other panic message.

[Josiah Henline]

Are the servers clustered? Do they share any common I/O device? If nothing else shows up from this trouble-shooting, moving off of the UPS may be the best "next step" for trouble-shooting.

[Eric Antunes]

Hi all,

As I said in the thread's opening post, both servers crashes simultaneously, so this is a UPS issue. Since this servers are going to be replaced in August/September, the temporary solution I found is to put the UPS in Bypass mode: since then they didn't crash anymore.

Best Regards,

Eric Antunes