Event Name: Rack power subsystem not redundant (22018)

[Tom McGuigan]

A few things that might help...
1) Rack power is split down the middle so that "redundancy" is maintained by keeping at least 2x the required power supplied to all blades within each half (right and left) at all times.
2) Rebooting a blade generates a spike in power demand during startup (no, I don't know how much, but it is easy to verify by attaching a pc/laptop to the mgmt/serial port of the blade enclosure and watching the power consumption during a reboot).

My guess is that you are running your rack pretty close to the limits of it's power capacity - say 45% of total available power (remember you need to maintain 2x the required power to be redundant). When a blade reboots, it draws just enough additional power that you trip the alert (51% would do it). Since the alert is for the entire 1/2 (left or right) of the whole rack, you get lots of blades all registering the same problem...power supply not redundant. This naturally initiates a panic attack, sending everyone scurrying to try to find the root cause and appease the powers-to-be. Unless the consultants witnessed or staged a blade reboot, they would not see any problem...did they run such a test?

Fixes
The short term fix is...use less power!! Move blades to another rack...if you have one, or swap out high power blades for lower power blades, or add more power supplies, or ignore/disable the alerts (I don't advise this one). The longer term fix is power capacity planning...i.e. get to know how power is supplied and consumed so you don't face these kinds of nasty suprises. Power capacity planning is a topic unto itself, however, and I'm done for tonight! Good luck!

[Juergen.Kopp]

I solved this problem after Blade Enclosure FW update from v2.30 to 2.32 AND Power Enclosure FW update from v2.20 to 2.32.

See also Advisory Board for 1U Power Enclosure: http://h20000.www2.hp.com/bizsupport/TechSupport/SupportTaskIndex.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=459979&taskId=110

[dospavlos]

I upgraded to 2.32 and I'm still getting the event log entries every minute. I'm using 3u power enclosure, not the 1U as listed in the customer advisory on this issue. Anyone else still have the problem after going to 2.32?

[James Kennedy_5]

I can confirm that I am still having this same issue even after upgrading the firmware. I also have the 3U power supplies.

[Tom McGuigan]

2.33 seems to have fixed it for us. 2.32 did not.

[James Kennedy_5]

I have version 2.33 installed on my 3 blade enclosures and 2(3u) power supply enclosures. Interestingly enough, the blades in one of the enclosures are still getting the error, all the other ones seem to be fixed.

[Jim Hanson]

I have upgraded to 2.33, and I am still getting these errors on one Enclosure of BL35p G2s. All are running Linux (RHEL4) and they are spewing them out at about 6 a minute. My other 9 Cabinets seem ok, and the other 4 enclosures of windows blades in the rack with the Linux ones are fine.

[Tom McGuigan]

For those still having redundant power problems with 2.33 of the rack firmware...suggest also checking to ensure all ILO firmware and PSP/Bios on every blade is current. Also, verify that every enclosure and power supply enclosure has latest firmware. Also, reset all management modules (paper clip hole/button). Lastly, verify all server ips and ilo ips are up. We have had to reset a few ilos that got set to 0.0.0.0 for some reason and did not get proper assignment (static ip) until either physically reseated or reconfigured (to use static ip addressing) from dongle+laptop on front port. One or more of the above suggestions has worked to fix all problems we encountered thus far. Good luck!!

[Seany Whetstone]

We have the same problem on one of 6 HP blade racks containing BL25p and BL45p's

The 100 error messages per hour all say

"Event Name: Rack power subsystem not redundant (22018)"

The enclosure firmware is up to date at 2.40

ILO Firmware is up to date at 1.91

PSP on each Blade is 7.91

We are starting to wonder like another poster whether the blades are fully power redundant if all three enclosures are full and heavily utilised.

Any insight greatly received.


Regards Sean Whetstone

Reed - London - UK

[Tom McGuigan]

Have you checked the power utilization via the serial port on the rear of the power supplies? If not, you should. To do so, connect a pc/laptop to the serial port, and launch hyperterminal. The bar at the bottom of the power redundancy screen should not exceed 1/2 of the total available power (asuming your total power capacity is 2x the power used/needed). If you are using over 1/2 the total available power, then you have in fact "lost redundancy"...thus, the error messages are real. If this is your situation, then you probably need to look at moving blades to another rack or adding more power supplies. If you do nothing, you risk an outage if either of the redundant power supplies quits (i.e. they are not redundant).