topic Re: Critical Error Redundancy Lost in BladeSystem - General

Critical Error Redundancy Lost

rgol — Wed, 07 Jan 2009 00:25:33 GMT

Hello,

We just started to get a critical error complaining about redundancy lost in the power subsystem on a c7000 chassis. As far as we can tell all the power supplies are working fine.

I've attached a screenshot of what we can see in the OA.

Any ideas on what might be causing this problem?

Thanks!

Re: Critical Error Redundancy Lost

JKytsi — Wed, 07 Jan 2009 11:00:59 GMT

OA firmware, What is the version of it ?

Re: Critical Error Redundancy Lost

rgol — Wed, 07 Jan 2009 13:17:24 GMT

The OA firmware 2.30. We have 2 other c7000 chassis running the same firmware without any problems.

Re: Critical Error Redundancy Lost

Raghuarch — Wed, 07 Jan 2009 15:26:08 GMT

Can you execute show power command from the OA CLI Check the Power Allocation details.
Can you post syslog of the Active OA.

Re: Critical Error Redundancy Lost

rgol — Wed, 07 Jan 2009 15:53:19 GMT

Thanks for your responses guys. We finally contacted HP support and they said that the power is fine and the problem is with the OA administrator firmware. They recommended downgrading to 2.25.

Re: Critical Error Redundancy Lost

Raghuarch — Wed, 07 Jan 2009 16:14:59 GMT

What Downgrade to 2.25!!!!!

Before you downgrade i would suggest, you try upgrading to 2.32 it was better then 2.30 and has some fixes.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=3188465&prodNameId=3188475&swEnvOID=1005&swLang=13&mode=2&taskId=135&swItem=MTX-945bcf9f337b476390b3effdc3

Re: Critical Error Redundancy Lost

rgol — Wed, 07 Jan 2009 16:22:53 GMT

They said that 2.31 and 2.32 have the same problem.

Just because they recommend downgrading doesn't mean we will :-)

We will stay with 2.30 and live with this none-error.

Thanks again for your help.

Re: Critical Error Redundancy Lost

morrisosu — Mon, 26 Jan 2009 22:30:35 GMT

We too are having the same issue. Just out of the blue we received this error, even though all appears to be functioning properly with the PSUs. We are on 2.32 as well.

Re: Critical Error Redundancy Lost

weghn0 — Mon, 09 Mar 2009 12:35:26 GMT

We also have the same problem. (OA firmware 2.32.) I'm having a extra problem now, I can't boot a blade server error: Power Allocation Request Insufficient enclosure power

Re: Critical Error Redundancy Lost

Patrick G. — Thu, 26 Mar 2009 10:11:42 GMT

I have the same error wit OA firmware 2.41. The attached picture shows the allocate power. Only the BL680c in Bay 6 is powered on.

OA 2.41
ILO2 1.70
ROM I17 09/23/2008

Any ideas?

Re: Critical Error Redundancy Lost

Patrick G. — Thu, 26 Mar 2009 12:21:36 GMT

It looks like Bay 6 is the key. If it is the last blade I put into the enclosure, after all other blades are powered on without any failure, the bay 6 blade powered succesfully on. Only the OA got a message like "Redundancy lost" and "Power allocated: 44868 Watt DC".

Maybe it helps

Re: Critical Error Redundancy Lost

Patrick G. — Thu, 26 Mar 2009 12:43:09 GMT

And bay 7 is responsible for that problem too.

Re: Critical Error Redundancy Lost

weghn0 — Thu, 26 Mar 2009 12:53:50 GMT

We solved the "Power Allocation Request Insufficient enclosure power" error by rebooting the bays where the power allocationn was to high. We had more then one bay at about 3000watts after roboot under 200Watts.
hope this helps

Re: Critical Error Redundancy Lost

Patrick G. — Thu, 26 Mar 2009 13:35:45 GMT

unfortunately not

Re: Critical Error Redundancy Lost

darnoc — Thu, 26 Mar 2009 14:42:30 GMT

Hi All,

HP is right, downgrading the OA firmware to 2.25 will solve the Critical Error Redundancy Lost. the OA FW 2.25 solves this issue.

The error also causes power reduction to the servers. Once you shutdown the server,it may not power back on due to less power, and red blinking LED you'll receive instead.

We've applied this solution just now as we experienced the problem this morning.

Re: Critical Error Redundancy Lost

The Brit — Thu, 26 Mar 2009 18:09:56 GMT

Is this issue being actively worked on at HP? and how long (how many versions) does it take to get a resolution.

I am currently at Version 2.32, and I have not seen the problem. I was going to upgrade, however I think I will wait since the problem has been seen to still exist with version 2.41. I dont want to risk activating a problem which currently seems to be dormant at my place.

Dave.

Re: Critical Error Redundancy Lost

saks5th — Thu, 26 Mar 2009 20:20:22 GMT

Hi Everyone,

I just started having the same issue in one of our c-Class enclosures.

OA Firmware: 2.32
iLO2 Firmware: 1.70
ROM: I15 11/02/2008

We are running mostly BL460c. As previously noted, it does not look like any of the power supplies is the root cause of the issue... I would really hate reverting back to 2.25 unless I REALLY have to. Has anyone heard anything from HP?

Thanks.
Jesper

Re: Critical Error Redundancy Lost

John Moorhead_2 — Thu, 26 Mar 2009 21:15:57 GMT

Hi folks. We have a C7000 enclosure loaded with 2 VC 10gb OA blades in the rear, 2 BL680C-G5 in front Bay 1 & 2, 4 BL460C in Bay 3, 4, 11, 12, and BL220C-G5 in Bay 5, 6, 7, 8, 13, 14, 15, 16. Prior to setting up server instances, we performed a full firmware update; both OA blades are at version 2.32, and all server instances are at 1.70. All server blade instances except the BL460C in Bay 12 had been installed and operational running RedHat 4 Linux Nahant 7, kernel rev 2.6.9-78.0.13. For the last few days, we've had degraded status on the BL680C-G5 in Bay 2, showing the same power issues discussed. However, that server has been operational the entire time and performance has not been affected as far as we can tell. We are running continuous ASIC simulation jobs on it with no issues. Our original Bay 12 blade was bad; we just received the replaement. Upon insertion of the replacement, we got an error indicating insufficient power is available, and it refuses to power up at all. We are currently in "Not Redundant" power supply mode, but were previously at "Power Supply Redundant" when the issue first showed up. Our "Power Limit" is 60024 watts, with only 3886 watts â present powerâ . The â enclosure totalâ shows 13,500 watts.

Re: Critical Error Redundancy Lost

saks5th — Thu, 26 Mar 2009 21:25:02 GMT

Hi again,

Just as an FYI, I just opened up an HP case regarding this issue. I'll let you know if/when I hear anything.

Thanks.
Jesper

Re: Critical Error Redundancy Lost

John Moorhead_2 — Fri, 27 Mar 2009 21:00:06 GMT

Update after placing call to HP Response Center:

1) ALL blades MUST be at latest firmware revision. Under Device Bays, click on + next to each numbered Bay, then under Information tab, see ROM Version. Compare this with latest firmware versions as seen on HP's web site.

2) Click on the image of each blade, or under Device Bays, click on each blade instance in turn. Compare figures for "Power Allocated". Look for one blade that shows significantly higher watts (for instance, I had one blade showing 43,850 watts as compared to between 250 to 500 watts for each of the others).

3) Remove that blade from the cabinet. Remove cover, and remove thin circular cache battery. Replace after one minute (make sure polarity stays the same!). Re-insert the blade.

What's going on is that power consumption values got corrupted during a prior firmware upgrade; cache needs to be cleared after the firmware upgrade to present default values at power-up.