Critical Error Redundancy Lost

[John Moorhead_2]

For the record, the blade that I've been having all the problems with is also a BL680c-G5, same as Joshua's. Hmnnn!!!

[saks5th]

Hi Everyone,

I just wanted to give an update to this issue. I opened up a case with HP and basically the response was that you need to install the latest and greatest firmware on your server, iLO2 and OA. In our case we already had the latest installed (BL460c G1; ROM at 11/02/2008, iLO2 at 1.70) with the exception of the OA, which was sitting at 2.32. After updating to the latest OA version (2.41) we were still experiencing the power allocation problems but after rebooting all of the individual blade servers the error went away and we have not seen it re-occur. Hope this helps...

Jesper

[Patrick G.]

ROM Version "I17 02/24/2009" for "ProLiant BL680c G5" solved the problem in our case

[Joshua Oswald]

At this point, I would also agree that ROM 2.24.2009 solved the power redundancy problem as well.

Unfortunately, for us, that has brought a new host of problems.

Apparently, this ROM enumerates physical devices to the OS differently, which means our HP Teaming definitions are no longer valid. I would suspect (although haven't tested) that removing / reinstalling the NCU would resolve this except our blades are running server Core... which doesn't have a mechanism for uninstalling the NCU. The only option is to rebuild the server.

So, since these blades are not yet in production, we bit the bullet and upgraded the ROM and re-installed the OS. Then we discovered that this ROM also breaks dynamic WWN assignment for HBAs... both HBA ports get assigned the same WWN. If we revert back to the previous ROM, the WWNs get assigned correctly, but we (again) have invalid Teams and we're back to the (original) power redundancy error.

I have a case open with HP regarding the WWN assignment problem. The last suggestion was to revert our VC firmware version and recreate the domain.

[The Brit]

I've been following this thread with increasing interest (after my initial panic), and it seems to me that the focus is moving away from OA firmware, and towards Proliant ROM F/W.

Just to clear up a few things, and maybe focus the discussion. The real question is whether the problem is between the OA firmware reading/getting the wrong information from the Power Supplies, or whether the interaction between the Blade and the power supply at power up is causing the PS to report to the OA incorrectly.

1. Does the initial occurance of the problem ALWAYs occur when a blade is being powered-up??
2. Has the problem ever affected a RUNNING server, i.e. caused a crash or shutdown??
3. Have there been any cases where this issue is FIRST SEEN when powering up an Itanium Blade?
4. It is being implied that Proliant ROM F/W 2.24 fixes the problem, so has any Proliant running 2.24 exhibited the problem.

Also, although the emphasis seems to be moving away from OA F/W, and not withstanding that HP is still recommending a downgrade to 2.25 for those experiencing the problem, I am surprised to see in the most recent "Alerts" e-mail, recommendations to upgrade to OA Version >2.25 for what appear to be fairly trivial issues which can be manually resolved via the OA CLI.
Since the cause of the PS issue is not fully resolved, it seems a little reckless to be making this recommendation and risk invoking a much more serious problem (in my opinion).

Makes me wonder if anyone at HP is paying attention to the problems that this PS issue is causing. I am running OA 2.32, and I have not experienced the problem at my site, however I have to admit to a certain level of paranoia, making me fearful of performing many simple functions.

1. I am concerned about power cycling any of my servers since that may initiate the problem in my enclosures.
2. I am holding off with my intended upgrade of the OA firmware (to 2.41) because the problem has also been reported with this FW level, and again, I don't want to risk initiating the problem at my site.

Some official (HP) statement or comment, or reassurance would be appropriate at this point.

Dave.

[The Brit]

Slight modification to my post above, I noticed that the 2.24 version referred to in earlier posts, was for BL680c blades. That version was released on April 2nd this year.

There was a ROM flash released for BL465c G5 blades dated April 3rd (version 2009.03.12) which I hope is the equivalent. The "Fixes" for this release states;

"Resolved an issue where the Low Power Halt State (AMD C1 Clock Ramping) option in the ROM Based Setup Utility (RBSU) does not properly disable this power management feature for AMD Opteron 2300-series processors. Previous revisions of the System ROM would not disable this functionality even when this option was configured for Disabled. This fix is not required for systems configured with older versions of AMD processors or if the Low Power Halt State was enabled (which is the default state)."

Resolved an extremely intermittent issue that could cause the system to encounter a system boot hang with a red screen.

Maybe one of these is the fix we are looking for although neither seems to reference it explicitly.

Dave.

[John Moorhead_2]

My observations addressing Brit's questions:

1) In my case, no, the whacked-out "Power Allocated" figure on the one BL680c-G5 which caused our problem did not occur at power-up, it occured several days later. First indication was an amber alert on the blade with associated "Degraded Status". Once this issue occured and as long as it existed, any blades that had been powered down prior to the event could not be powered up (because the cabinet thinks at that point that there is insufficient power). Any blades that were running fine when the issue occured and were later powered off and then attempted to be powered back on would not go back on for the same reasons. As long as all blades in the cabinet had been running prior to the event, there has been no observed performance degredation or loss of server capabilities as long as they stay powered up. But you're playing with fire here; you are running in a crippled mode and run the risk that a critical production server could go down for some other reason and then you would not be able to bring it back up without powering off the root-cause blade as well.

2) In my experience, it did not actually affect operational performance of a running server at all, as long as that server stayed powered up. As in 1 above though, once the event occurs, you cannot power down another blade in the same cabinet and then power it back up without shutting down/upgrading firmware/resetting cache on the root-cause blade first.

3) My blades are all X86 based so I have no experience on Itanium.

4) I had upgraded my blades to 2.25 and did not see a re-occurence of exactly the same problem. However, with 2.25 I DID have a degraded status event on the BL680c-G5 1.5 weeks after the upgrade, but without the associated "Power Allocated" issue. For this reason I consider this to be a different event for different root causes, as-yet undetermined. Note that with this issue on 2.25, I was able to power-down a blade and back up again without any problems.

I've had very long discussions with the HP Support folks and I know that this issue has been getting a lot of scrutiny both at the Response Center and in the Labs.

[Nikola Mrdja]

There are newer OA fw versions (they appeared in end of May 2009), which should resolve those â status: degradedâ problems. The latest version is 2.51 (May 28 2009) and I'm about to put it in my Blade enclosure soon.

Does anyone tried this fw version already?

[Patrick G.]

In terms of my comment from Apr 16, 2009 08:40:05 GMT

"ROM Version "I17 02/24/2009" for "ProLiant BL680c G5" solved the problem in our case"

solved this ROM the power allocation issue but brings a new failure. Both ports of the HBA get the same WWPN. This ROM was removed by HP.
The power allocation issue occurs in only two BL680c. It doesn't matter in which enclosure they are plugged in.

The new ROM Version "I17 05/10/2009" solved in my case the power allocation issue and gives different WWPN to the HBA.