Updated blade firmware, now device shows "unknown"

[Mike O.]

I have a C7000 chassis with 5 BL465C blades. I was wanting to update the firmware on the blades and the chassis to bring it up to the current levels. Following the various recommendations, my idea was use the firmware deployment tool to update each blade, then I was going to do the virtual connect modules, then the OA.

We are currently at 2.32 on the OA modules. The ILO on the blades is at 1.70


I downloaded the offline blade firmware 1.6 (bbfirmware-1.60-0.zip), mounted the ISO through ILO and restarted the blade.
The install started and seemed to run OK for a while. It then kicked me out of ILO a couple of times (as expected), then it stopped the ILO session and the OA now says that it lost communication with the management processor. The OA GUI shows a big red&white "X" on the device bay. The OA system log says that the "Management processor on blade 5 appears unresponsive"

I waited about 30 minutes and it never came back. I tried reseating the blade, and now it just shows as "unknown". I tried directly connecting a monitor to the I/O connector on the front of the blade, but I'm not getting any display. The power button on the blade just stays orange, it won't power up.

So at this point, is the blade damaged, or is it just that the management processor on the blade is too high of a version for the O/A to communicate? If I update the O/A, will it cause any problems for the other blades?

Any information would be appreciated.
Mike O'Donnell

[JKytsi]

You might have needed to read the _RELEASE NOTES_ from FDT 1.60

It says...

OA FW _MUST_ be 2.52 before using it

[Mike O.]

Thanks for your reply. I thought I had checked the release notes for the update, apparently not..

One thing that was making it confusing was that from other information I had seen, the recommendation was usually update firmware from the inside out; start with the blades, then the interconnects, then the OA. I had seen on the Bladesystem Firmware Compatibility chart that it looked like OA 2.52 needed ILO 1.78 or better, so I thought I needed to do the blade/ILO first.

Now that the blade is updated, if I go ahead and update the OA to 2.52, will the blade start functioning correctly, or is is damaged? And if I do update the OA to 2.52, will my other blades (still on ILO 1.7) and the interconnects (Enet VC 2.01 and FC-VC on 1.3) work OK, at least until I can update them? Or will I run into the same type of thing I did this time, where the OA won't talk to the blades?

[jimenste]

Hi Mike
The management processor error is a synchronization error between the OA and the iLo chip set, what happens is that the firmware process didn't finished correctly, your iLo chip set is now corrupted.

How to fixed it:

There are two ways to do it.

1.If you have another enclosure with OA firmware 2.43, 2.51 or 2.52 please go and install it in that enclosure the blade will power up, after that reinstall the iLo firmware but now using the iLo web Administration and the bin file instead.
When the process finished. reinstall the server in the original enclosure.

2.If you don't have other enclosure: go on site, remove both OAs(you will heard all the fans spinning faster and the leds blinking) without the OAs, remove the affected server, wait 2 minutes while all the bays get the same amount of power,on the server remove the system battery. reinstall the battery and then reinstall the server. power the server up. use a dongle cable keyboard and monitor. get into the os, through the iLo web administration upgrade the iLo firmware using the bin file. finish the process, turn the server off, reinsert the OAs, clear browser cache, login in and check the results.

For all the rest of servers using iLo version 1.70 upgrad them through the web administrator to 1.78. After that go ahead and upgrade both OAs to 2.52.

best regards

[Mike O.]

thanks for your reply.

We do have another enclosure, but it's also at OA 2.32.

based on the information I have, I was thinking about doing the following:

1) Update the ILO to 1.78 on the four working blades
2) Update the OA to 2.52
3) Reflash the firmware on the corrupted blade
4) run the 1.6 firware update on all the blades.

Does this sound like it would work?

[jimenste]

Yes.

Another recommendation. Do not upgrade bios and iLo at the same time if you are using remote console and virtual media. I would recommend upgrading bios, rebooting, booting again from the offline blade firmware 1.6 and then select the rest of devices.

As soon as you finish with the upgrades on the servers, upgrade the OA and then install the affected server and upgrade iLo as I told you before.. using iLo Administrator and the bin file.

let me know.

[Mike O.]

I think I'm getting confused.

You say to boot from the offline firmware 1.6, upgrad the devices, and as soon as I finish upgrading the devices, then upgrade the OA.

Since the 1.6 firmware requires OA at 2.52, shouldn't I upgrade the OA to 2.52 before running the 1.6 offline firmware update?

If I update all the ILO to 1.78 individually first, then shouldn't I be able to install OA 2.52, then complete the full offline 1.6 upgrade?

[jimenste · Accepted solution]

Sorry I thought you were talking about the firmware maintenance CD 1.60.

I this case you are right
update individually iLos to 1.78 and then the OA to 2.52, the idea is to have everything up to date in order to insert the affected server and boot it up.

[Mike O.]

Thank you again for your assistance. I'll start on the updates next week.

I have another issue that has come up, that may also be a firmware issue, and I was wondering if you might have any insights on it.
One of the BL465C blades in our 2nd enclosure died around 1:00 today. No LED, no ILO or anything. I called HP and after a few things they had me try, it was determined to be a bad system board on the blade. They sent a new system board and I replaced it and transferred all the hardware (CPU, RAM, etc.).
I put the blade back in the chassis, and after a few minutes the OA identified the model correctly and I was able to open the ILO. However, no matter what I try, the blade will not power up. I do have the orange light showing on the power button, and the blue UUID comes on when it should, but the blade just ignores any of the power up functions. Also, I noticed that the RAM is not showing correctly. The blade has 34G of RAM (four 8G and two 1G), but the OA shows as six 512M modules.

Could the problem be that the firmware on the replacement board is too new for the OA 2.32 to communicate? It did have ILO 1.70, and I tried updating that to 1.78. The update took correctly, but I still can't power it up.

I tried it in a different chassis, and tried doing a reset of the OA, but no change.

I'm not going to be doing anything on it until next week, and if I can't figure out anything by then, I'll call HP, but I figure they'll probably tell me to update the OA before I do anything else.

[jimenste]

Well
You said you moved all the hardware.
and the oa is reporting incorrect information.
I would recommend you to check the configuration:

Observe the following guidelines when installing additional memory:

Install only ECC PC2-5300 Registered DDR2 SDRAM DIMMs that meet the following specifications:
Supply voltage: 1.8 V
Bus width: 72 bits
Observe the following special conditions when installing memory with a second processor:
Processor 2 can be installed without memory.
Any memory installed into banks for processor 2 can be used only if processor 2 is installed.
DIMMs must always be installed in pairs.
************************
*************
HP recommends installing DIMMs with the greatest capacity in the banks farthest from each populated processor first.
*************
*************************
DIMMs installed in the same memory bank must have the same part number.
DIMMs installed in different banks can be of different sizes.

Also remove the system battery for 3 minutes to clear the non volatile memory saved on the system board.

[Mike O.]

The hardware (CPU, memory, etc.) was all transferred from the failed system board and placed in the same locations on the new board. When I first saw the error, I pulled the blade and re-checked the DIMM locations, and they all appear to be in the correct locations. I didn't try to remove the battery on the new board, though. I guess I thought that on a new board, the system wouldn't have any configuration info.

[jimenste]

what happens is that HP doesn't sent brand new system boards. that is why some times you get a DOA system board or something else. HP tests the hardware and if it seems ok they will use it for spare replacements. so not all the times but sometimes you might get an used part.
That is why I recommended the clear nvram.

[Mike O.]

I was out yesterday, so today was the first time I was able to work with it. I removed the battery, waited about 10 minutes, then replaced the battery and the blade.

It's still not working right. In the OA, the blade first shows as "unknown", then starts to pick up the various modules. It correctly identifies the blade as a BL465G5, lists the fiber channel and ethernet cards, but still comes back with 4G of RAM (it has 34G), and shows the CPU's as Opteron 2600 (they're 2300). When it attempts to power up, I get the green light on the OA (but NOT on the front of the blade), then after a minute or two the OA shows back to orange.

One other interesting thing; the blade name in the device bay in OA originally shows up as "unknown", then "BL465...", then goes to "OP000844AM". Is this some kind of error code?.

At this point, I'm probably going to call HP again...

[Mike O.]

Well, after doing some trial and error Wednesday morning, I figured out that in additon to the original system board problem, one of the CPU's and one of the six DIMMs was bad, too. I called HP and had them send out the parts (asked for a tech this time, in case there was something else..) and everything is working OK now.

[predrag81]

JIMENSTE!!
I must react to your answer bellow:

"Hi Mike
The management processor error is a synchronization error between the OA and the iLo chip set, what happens is that the firmware process didn't finished correctly, your iLo chip set is now corrupted.

How to fixed it:

There are two ways to do it.

1.If you have another enclosure with OA firmware 2.43, 2.51 or 2.52 please go and install it in that enclosure the blade will power up, after that reinstall the iLo firmware but now using the iLo web Administration and the bin file instead.
When the process finished. reinstall the server in the original enclosure.

2.If you don't have other enclosure: go on site, remove both OAs(you will heard all the fans spinning faster and the leds blinking) without the OAs, remove the affected server, wait 2 minutes while all the bays get the same amount of power,on the server remove the system battery. reinstall the battery and then reinstall the server. power the server up. use a dongle cable keyboard and monitor. get into the os, through the iLo web administration upgrade the iLo firmware using the bin file. finish the process, turn the server off, reinsert the OAs, clear browser cache, login in and check the results.

For all the rest of servers using iLo version 1.70 upgrad them through the web administrator to 1.78. After that go ahead and upgrade both OAs to 2.52.

best regards "

As I know, HP recommendations in situation wheh you pull out both OAs is that you don't try insert any modul or Blade server?

Am I right?

and this situation is very close to ILO FW corruption. And when this comes the only safe solution is to replace BD SYS.

If I'm wrong be kind and let me to know.

KR,
Predrag

KR,
Predrag