BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

Event Name: Rack power subsystem not redundant (22018)

Jim Hanson
Advisor

Event Name: Rack power subsystem not redundant (22018)

Hi,
I get these from 1 Blade cabinet whenever one of my blades reboot. All Firmware is up to date, Insight shows no errors on the Rack or Enclosures. Attaching to the mgt port shows PS as redundant with no errors.

Any Ideas?

Thanks
24 REPLIES
sandeep_raman
Honored Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

Hello Jim,

Whats the server blade model?
Power Enclosure -> 1U / 3U?
How many server blade enclosures and power enclosures?
Try changing the bay in the enclosure and see if it makes a difference?

SRH
sandeep_raman
Honored Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

Also whats the OS and Management agent version?

SRH
Jim Hanson
Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

2 3U 3Phase Power
Scalable Bus Bar
5 P class enclosures

mix of 25p, 20p G2&3, xw25p, 35p

one caveat, our linux systems (which is about half the cabinet) do not have agents installed and show as blank spaces in the rack view model. All other servers are win2k3
James Kennedy_4
Trusted Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

We're having pretty much the same issue in our environment. Only our's is 3 enclosure's and only on single phase power. We see this event message on many of our blades, but not all of them.

According to HP's blade documents and the power usage spreadsheet, single phase power supports 3 enclosures.

I've had two HP technicians come in and say that they "think" we're using too much power. They said that we can have 3 enclosures on single phase, but if we fill more than 2 enclosures, it will no longer be redundant.

I have yet to have an official answer on this issue though. If this is the case, then HP needs to change their documentation to mention that you can use this many enlusure's, but you'll lose redundancy.

Management hasn't been too happy about this as HP sold this to us with the understanding that we'd be fully redundant.

As with you, connecting to the management port for the Power Supplies shows everything is all good.

It's my interpretation that if you are using more than 50% of the power in your power enclosure's, you are redundant for the loss of a power supply, but not an entire power enclosure. Makes sense to me, just wish we had known this before.
sandeep_raman
Honored Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

Is there a event log generated as well,id 1160?

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00293064/c00293064.pdf
Pg 101

SRH
James Kennedy_4
Trusted Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

Yes, the 1160 Event is reported in the event logs on the blades. Annoyingly, every minute.

However there are no failed power supplies, and diagnostics show all power supplies are properly working. I've had 2 HP technicians come out and run full diagnostics on all the power supplies, and they say there are no issues.
sandeep_raman
Honored Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

Whats the version of Management agents?

SRH
James Kennedy_4
Trusted Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

7.51 across all baldes,

ILO firmware is at 1.84
Enclosure firmware is 2.30
Jim Hanson
Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

i also get the 1160
Agents at 7.51
ILO Firmware at 1.84
Enclosure Firmware at 2.30.
I have even reflashed all the firmware one at a time.
Tom McGuigan
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

A few things that might help...
1) Rack power is split down the middle so that "redundancy" is maintained by keeping at least 2x the required power supplied to all blades within each half (right and left) at all times.
2) Rebooting a blade generates a spike in power demand during startup (no, I don't know how much, but it is easy to verify by attaching a pc/laptop to the mgmt/serial port of the blade enclosure and watching the power consumption during a reboot).

My guess is that you are running your rack pretty close to the limits of it's power capacity - say 45% of total available power (remember you need to maintain 2x the required power to be redundant). When a blade reboots, it draws just enough additional power that you trip the alert (51% would do it). Since the alert is for the entire 1/2 (left or right) of the whole rack, you get lots of blades all registering the same problem...power supply not redundant. This naturally initiates a panic attack, sending everyone scurrying to try to find the root cause and appease the powers-to-be. Unless the consultants witnessed or staged a blade reboot, they would not see any problem...did they run such a test?

Fixes
The short term fix is...use less power!! Move blades to another rack...if you have one, or swap out high power blades for lower power blades, or add more power supplies, or ignore/disable the alerts (I don't advise this one). The longer term fix is power capacity planning...i.e. get to know how power is supplied and consumed so you don't face these kinds of nasty suprises. Power capacity planning is a topic unto itself, however, and I'm done for tonight! Good luck!
Juergen.Kopp
Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I solved this problem after Blade Enclosure FW update from v2.30 to 2.32 AND Power Enclosure FW update from v2.20 to 2.32.

See also Advisory Board for 1U Power Enclosure: http://h20000.www2.hp.com/bizsupport/TechSupport/SupportTaskIndex.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=459979&taskId=110
dospavlos
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I upgraded to 2.32 and I'm still getting the event log entries every minute. I'm using 3u power enclosure, not the 1U as listed in the customer advisory on this issue. Anyone else still have the problem after going to 2.32?
James Kennedy_5
Regular Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I can confirm that I am still having this same issue even after upgrading the firmware. I also have the 3U power supplies.
Tom McGuigan
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

2.33 seems to have fixed it for us. 2.32 did not.
James Kennedy_5
Regular Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I have version 2.33 installed on my 3 blade enclosures and 2(3u) power supply enclosures. Interestingly enough, the blades in one of the enclosures are still getting the error, all the other ones seem to be fixed.
Jim Hanson
Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I have upgraded to 2.33, and I am still getting these errors on one Enclosure of BL35p G2s. All are running Linux (RHEL4) and they are spewing them out at about 6 a minute. My other 9 Cabinets seem ok, and the other 4 enclosures of windows blades in the rack with the Linux ones are fine.
Tom McGuigan
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

For those still having redundant power problems with 2.33 of the rack firmware...suggest also checking to ensure all ILO firmware and PSP/Bios on every blade is current. Also, verify that every enclosure and power supply enclosure has latest firmware. Also, reset all management modules (paper clip hole/button). Lastly, verify all server ips and ilo ips are up. We have had to reset a few ilos that got set to 0.0.0.0 for some reason and did not get proper assignment (static ip) until either physically reseated or reconfigured (to use static ip addressing) from dongle+laptop on front port. One or more of the above suggestions has worked to fix all problems we encountered thus far. Good luck!!
Seany Whetstone
Occasional Visitor

Re: Event Name: Rack power subsystem not redundant (22018)

We have the same problem on one of 6 HP blade racks containing BL25p and BL45p's

The 100 error messages per hour all say

"Event Name: Rack power subsystem not redundant (22018)"

The enclosure firmware is up to date at 2.40

ILO Firmware is up to date at 1.91

PSP on each Blade is 7.91

We are starting to wonder like another poster whether the blades are fully power redundant if all three enclosures are full and heavily utilised.

Any insight greatly received.


Regards Sean Whetstone

Reed - London - UK
Tom McGuigan
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

Have you checked the power utilization via the serial port on the rear of the power supplies? If not, you should. To do so, connect a pc/laptop to the serial port, and launch hyperterminal. The bar at the bottom of the power redundancy screen should not exceed 1/2 of the total available power (asuming your total power capacity is 2x the power used/needed). If you are using over 1/2 the total available power, then you have in fact "lost redundancy"...thus, the error messages are real. If this is your situation, then you probably need to look at moving blades to another rack or adding more power supplies. If you do nothing, you risk an outage if either of the redundant power supplies quits (i.e. they are not redundant).
Seany Whetstone
Occasional Visitor

Re: Event Name: Rack power subsystem not redundant (22018)

Thanks

My engineer is at the remote data centre today with a laptop and serial lead to check each blade server power output today so I will get him to look at the power supplies also as you suggested.

Regards Sean Whetstone
Bharath Bhushan
Occasional Visitor
Koopa
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I also have racks with 2 x 3u 3 phase power supplies installed, with the load balancing cable installed. I've also got all enclosures running 2.4 firmware, but I still see this issue, and it's on more than just 1 cabinet. Even tho we have 6 PSU's installed in each power enclosures, and never have more than 4 blade enclosures. Any other ideas?
James Kennedy_4
Trusted Contributor

Re: Event Name: Rack power subsystem not redundant (22018)

My resolution was to stop and Disable the "HP ProLiant Rack Infrastructure Interface Service" in services. I did this just on the blades that continued to get this error, and they no longer get it.
Koopa
Frequent Advisor

Re: Event Name: Rack power subsystem not redundant (22018)

I've gone through the service port. I've not looked at these screens before. Can someone look at the attachment, and confirm that the rack is set for redudancy. As I said before we have 2 3u Power enclosures with 6 PSU's installed on 3 phase. We also have the load balancing cable installed and we use scalable bus bars. Shouldn't this be a redudant setup, do we need to do any sort of configuring? please advise