Contents of Dev Bays disappeard from OA display.

[The Brit]

C7000 with OA firmware 2.60 (upgraded ~3 weeks ago). Enclosure is full with a mix of bl860c (4) and bl465c G1/G5's running OpenVMS, Linux, and Windows (VMWare).

It was noticed this morning that the "Rack Topology" display of the front of the enclosure no longer shows any of the installed blades. The current "Front" display shows only a strip of 6 PW supplies, nothing else.
The "Rear" display is fine.

The Insight display is fine.

Clicking on the + by Device Bays does nothing, it does not expand.

The Device Bay summary shows "no devices to display"

When I go to Enclosure Settings > Configuration Scripts, and select Inventory, the inventory contains all of the information for the installed servers. So clearly, the OA sees them.

This is not causing any operational issues and appears to be just a display/java issue.

Has anyone seen this before?? Any suggestions??

This appears to have happened over the weekend since it seemed ok on friday.

Dave

[JKytsi]

clear browser cache. If no help reboot OA. If no help, reseat the OA module.

[marcus1234]

hmm..
try with browser explorer with vers max 7
what is result ,
are you using firefox i have seen similar issues with firefox , and with explorer 8

[The Brit]

OK.

Attempt 1. Tried reseting OA. Seemed OK, all information returned. Logged out. Logged in again, all information gone again.

Attempt 2. Reset OA again. Only returned information on 3 servers.

Attempt 3. Failed over the OA. When the OA returned, it seemed to be Ok, all information returned. Logged out. Logged in again, all information gone again.

I will be physically in the Data Center tomorrow morning and would like to try removing and re-seating both OA's and the OA tray.

Is there a standard procedure for doing this, and are there any additional steps that I should take. i.e. will my configuration be safe?? and particularly will by IP addressing be safe.

Dave.

[marcus1234]

I would first try
Logging into the OA directly with a laptop and x cable is the issue the same ?

Normally removing both OA'S at the same time poses no issues what so ever fans spin up full speed and a lot of noise but after reinsertion , they resume normal state within 60 sec..

but as there appear to be issues ,
I would physically reseat all interconnects first , then each OA and bay one at a time till fan speed is back to normal state

I would also back up profiles created as a safety measure but have never seen them lost with 2 Onboard admins in use.

But there is always a first.

[JKytsi]

Also the OA sleeve under the OA modules can be re-seated for proper connection.

[The Brit]

Hi Guys,
Just want to let you know that the issue is resolved. After examining the Inventory report, I noticed that the Blade in Bay #14 was showing garbage where the Serial Number is supposed to be displayed. This garbage seems to have been corrupting the information being retrieved by whatever process polls the device bays for system information. Removing blade #14 restored the information on the other blades.

It is not clear why this happened, since this blade has been installed previously without any problems. Now however it seems to have lost its serial #.

I think this qualifies as a BUG since the presence of garbage information on one blade should not result in the loss of information on ALL blades. Anyone know the best way of getting this into the HP system for their consideration???

Dave

[cybercoaster]

I have this same problem with an enclosure. I just did an inventory and the Part Number for Blade #1 is garbage. I will pull it and see if this is causing the issue. To note - the iLo name also looks like garbage.

[kris rombauts]

Those blades with the garbage, what model are they and what BIOS (Proliant) or what iLO/MP and SystemFW (Integrity) revision are they on ?

Kris

[The Brit]

The blade in my case was a bl465c G1, BIOS F/W A13. The problem on this blade has been fixed (for now) by the Proliant Admin manually entering the Serial Number (from the case), into the BIOS (under Advanced Options, I think).

Kris,
To return to my point, can this be escalated to the OA Firmware people to try and stop the loss of ALL information, when one blade returns garbage???

Thanks

Dave.

[marcus1234]

Hmm ..this issues occurs on other proliants too not only blades,,

cast an eye on the solution i provided here


http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1398996

i suspect it is because of a bad rom flash..

[The Brit]

You might be correct, it certainly looks similar.

Right now however, my point is that the OA firmware should be able to handle garbage coming from a single blade, and should not result in the loss of all information on ALL the blades.


Dave.

[marcus1234]

I suspect when a single blade is ,is reporting incorrect info because of a bad flash or similar,

the OA cannot do the reckoning it cannot define resources and assign them correctly
hence looses it ability to administer and this is what actually throws it out of kilter.


I have seen many enclosures where 1 single blade had a firmware that was not compatible with the matrix , and the whole enclosure became unstable. erratic,

In such circumstances no errors were to be found in event logs ilo etc,

Also it appears insight display becomes neutered , and displays green all round,showing no errors after all it only shows issues relayed from the OA.

So the issue that your enclosure had reinforces my belief that a single blade issue can overwhelm the OA and cause erratic issues.
it appears the OA algorithms cannot work outside their preset parameters ..

[The Brit]

See comments above.

I just hope that HP Engineering take this and fix it.

Just one additional problem. After removing the blade from the enclosure, I suspect that one of the admins tried to remove the profile from the empty bay, and now the VCM is hosed. I will be cycling the VC manager this evening.

Dave.