Server's Power Abruptly Cut - Will FPL Logs be Proof Enough?

[Alzhy]

We've a Server in A DC that we do not own. It suffered a clean stoppage of everything ('cept GSP) and what appeared initially as 2 FANS on the I.O Cages failing...

When we were alerted by its unavailability... It shows 2 Fans as failed and the entire Cabinet as w/o AC power.

It neeeded someone to physically power-reset the frame (A 'Dome) and all environments came up clean and the failed fans came back to life...


The FPL logs show the following. CAn this be proof enough that there possibly was an accidental cut off of AC power to the server frame?


MP:VWR (,,+,-,?,F,L,J,D,K,R,T,A,C,U,^B) >
Log Entry 70499: 06/13/2007 02:53:56
Alert level 3: Warning
Keyword: ILLEGAL_BPS_CFG_OR_PHASE_FLT
The BPSs for the cabinet are illegally configured. Data Byte 3 = PDCA number.
Reporting Entity: Power Monitor located in cabinet 0
Physical Location: AC mains cabinet 0
0x64800b0800e00000 0x00ffffff01ffff41
0x6b000b0800e00000 0x01000000466f5c44

Log Entry 70498: 06/13/2007 02:53:56
Alert level 3: Warning
Keyword: AC_DELETED
AC power to the PDCA was removed. Data Byte 3 specifies PDCA number.
Reporting Entity: Power Monitor located in cabinet 0
Physical Location: AC power PDCA cabinet 0 slot 2
0x64800ae200e00000 0x00ffffff0102ff42
0x6b000ae200e00000 0x01000000466f5c44

[Sindhia Naidu]

Hi Nelson,

This is clear enough. First you see if the AC_DELETED message and second you see ILLEGAL_BPS_CFG_OR_PHASE_FLT.

This means that power will not sufficient for the cabinet resulting in switch off. Now as for fans' and why only two, I can not explain. What version of PDC are you running?

But the log entries are enough to point to power. Secondly the GSP will remain up as long as one AC line is life to the cabinet.

[Alzhy]

Sindhia,

Thank you very much for your insight

The server in question is a SuperDome Complex (Full Configuration) with an IOX. The case is in escalation as the Facilities Management team of the data center where the SuperDome is situated would not admit that there was careless disruption of power. There were indications however per our person on the floor that there were power cables coiled up above the raised tiles about 15 ft away. HP is following protocol and has negated this possibility in the meantime and has embarked on a CSI type tests and checks on the BPS. They explained the possibility of "Zinc Whiskers" getting into the the power supply themselves causing the power issue. Why the BPSes went to normal afterwards? Their explanation is that the ZInc Whiskers burned out...

We're supposed to receive the Lab tests/CSI reports tomorrow to give us a conclusion on what happened.

[Sindhia Naidu]

Zinc Whiskers - Blow up power supplies from my experience. But this shouldn't bring down the superdome since it is N+1.

You have some other issue.

Did you have luck by now!