rp4440 crashed giving HPMC errors in System event log.

[SANTOSH S. MHASKAR]

Hi,

Yesterday evening one of out rp4440 server with HPUX 11.11 on it crashed giving HPMC errors in System event log. Below is the output of event log.


---------------------------------------------

158 SFW 2 *7 0xE880035C02E009F0 00000000407C07E0 ERR_CHECK_HPMC
17 Jul 2009 11:12:59
159 SFW 2 2 0x57800F7302E00A10 8040004000000000 ERR_CPU_CHECK_SUMMARY
17 Jul 2009 11:12:59
160 SFW 0 *7 0xE880035C00E00A30 0000000000805C9C ERR_CHECK_HPMC
17 Jul 2009 11:12:59
161 SFW 0 2 0x57800F7300E00A50 8040004000000000 ERR_CPU_CHECK_SUMMARY
17 Jul 2009 11:12:59
162 SFW 2 *7 0xF680105E02E00A70 000000003F900000 MC_HPMC_MONARCH_SELECTED
17 Jul 2009 11:12:59
163 SFW 0 2 0x5680109B00E00A90 000000000002A024 MC_BR_TO_OS_HPMC
17 Jul 2009 11:12:59
164 SFW 2 2 0x5680109B02E00AB0 000000000002A024 MC_BR_TO_OS_HPMC
17 Jul 2009 11:13:00
165 SFW 0 *7 0xE880035C00E00AD0 0000000000805C9C ERR_CHECK_HPMC
17 Jul 2009 11:13:00

166 SFW 3 *7 0xE880035C03E00AF0 00000000001777F4 ERR_CHECK_HPMC
17 Jul 2009 11:13:00
167 SFW 0 2 0x57800F7300E00B10 8040004000000000 ERR_CPU_CHECK_SUMMARY
17 Jul 2009 11:13:00
168 SFW 3 2 0x57800F7303E00B30 8040004000000000 ERR_CPU_CHECK_SUMMARY
17 Jul 2009 11:13:00
169 SFW 1 2 0x57800F7301E00B50 2000000000000000 ERR_CPU_CHECK_SUMMARY
17 Jul 2009 11:13:00
170 SFW 0 2 0x5680109B00E00B70 000000000002A024 MC_BR_TO_OS_HPMC
17 Jul 2009 11:13:00
171 SFW 3 2 0x5680109B03E00B90 000000000002A024 MC_BR_TO_OS_HPMC
17 Jul 2009 11:13:00
172 SFW 1 2 0x4380100801E00BB0 00000000002119C8 MC_BR_TO_OS_TOC
17 Jul 2009 11:13:06
173 BMC 2 0x204A605F26020BD0 FFFF027000120300 Type-02 127002 1208322
17 Jul 2009 11:23:18

==============================================

The server rebooted successfully and applications are working fine but still H/W errors indicating in VFP as below

----------------------------------------------

LEDs | LOCATOR | SYSTEM | POWER
-----------------------------------------------------------------------------
| OFF | FLASH RED | ON GREEN
-----------------------------------------------------------------------------
Status | Fatal System State detected. Check System Event Log (SL) for error
| messages.
-----------------------------------------------------------------------------


Pl. guide me what could be problem.

Thanks in advance.

[Andrew Rutter]

hi,

the system suffered a fatal error while trying to process information on the cpu. this in turn forced the system to panic and toc, which caused a reboot.

its difficult to say the exact cause from just this. while it looks loke a hardware problen at first glance it could also bea software application that caused it.

the crash dump files will need to be analysed to find out more info.

as its come back up and is working ok now it could well be s/w problem.

test the cpu in STM by running the exerciser tests.

check if crash is tuned on, and if it is run the Q4 analyser through these.

or, log it with the support centre.

you could also clear the error in the MP by looking at the error logs and reading them all

Andy

[Prashanth.D.S]

Hi Santosh,

Can you attach the ts99 file from /var/tombstones/

Best Regards,
Prashanth

[SANTOSH S. MHASKAR]

Hi Prashanth,

Pl. check ts99 file attached herewith.

Thanks and regards,

-Santosh

[Prashanth.D.S]

Hi Santosh,

TS99 analysis indicate that the system crashed due to a time-out occurred waiting for a programmed I/O,

Possible cause could be in the I/O subsystem.

ts99 points towards rope four, it could be the cards located in slot 3 and 4 or it could be the lba hosted by rope four itself faulty.

My suggestion would be to involve HP to analyze further on this issue.

Best Regards,
Prashanth

[SANTOSH S. MHASKAR]

Hi Prashanth,


Thanks for ur advice.

I have logged a call to HP.


-Santosh