C7000 Strange Hardware Problem!

[SteveCrang]

Hi

We replaced the midplane on one of our C7000 enclosures to try and resolve a problem where none of the Virtual Connect modules were visible via the OA.

However, when the enclosure was powered up, we saw all sorts of errors on Insight Display. Thinking this might be a problem with the replacement midplane, we swapped out that one for yet another midplane.

Now the enclosure is completely unusable! The fans are flashing amber LEDs and running at high speed, there is no Insight Display, two PSU bays are showing errors which persist in the same location even if the two "faulty" PSUs are swapped round.

The VC modules are interconnected in pairs, but not communicating with each other. The storage blades I have will initialise OK, but the server blades don't power up on their own, and I can't manually power them up as they don't respond to the button press

We have swapped out the OA tray, and tried the "minimum config" approach by removing all the VC modules, blades and the standby OA in a bid to eliminate the cause of this, all to no avail.

Not being experts with this equipment, my colleague and I are at a loss as to what's happening.... please help!

[Cederberg]

Hi!

have you tried to reflash your OA firmware since you replaced the Midplane? Somtimes the OA firmware contains some firmware for the underlying components as well.

I'm thinking as you replaced the midplane some firmware inside it is to low to function with your current OA FW?

[Shadders]

Hi, Today - one of our C7000 chassis has done the exact same thing - the Insight Display is blank & all the fans are running at high speed & flashing amber. The chassis is no longer contactable via the OA. How did you fix your issue? My enclosure is fully populated & in a production environment.

Thanks
Steve

[cnb]

Steve,

You'll improve your chances on getting answers if you open and manage your own thread.


Rgds,

[marcus1234]

hi steve if this was a new setup i would , say the chassis is not square and has been squeezed , have seen it a few times,


but if it has already been used,

the remove all interconnects update with one blade at a time add all blades firmware updated and vc modules one at a time , then ad its redundant mate

till we bump into the issue..

[marcus1234]

as for the insight display htat cable is very easy to pinch when reinstalling the side connect circuit boards

also on every blade before firmware updates try this


clear nv ram

blade clear nvranm
Clearing the system configuration
To clear the system configuration:
1. Power down the server blade
2. Remove the server blade
3. Remove the access panellook at the diagram on it it will show the position of the maintenace switch( it clears Nv ram)
4. Change position 6 of the system maintenance switch to on.
5. Install the access panel
6. Install the server blade in the enclosure and power up the server blade.
7. Wait for the POST message that prompts you to change the switch setting:
Maintenance switch detected in the "On" position.
Power off the server and turn switch to the "Off" position.
8. Repeat steps 1 through 3.
9. Change position 6 of the system maintenance switch to off.
10. Repeat steps 5 through 7.


This is how we thank in the forum

http://forums11.itrc.hp.com/service/forums/helptips.do?#33
post feedback on status good luck :)

[marcus1234]

shadders pullout the oa and reseat it
nad ensure your firmwares follow the matrix on the OA firmware compatibility homepage

[Shadders]

Thanks for your help - but Im not sure why this has happened. nothing has changed. OA firmware is all bang up to date & so are the serevrs. One minute the Chassis was fine & then all of a sudden the fans sped up & started flashing & the display on the front went off & is non responsive.

I'm thinking that the midplane has developed a fault.

I'll be speaking to HP today & see what they say. I'll update the thread when I have an answer.

Cheeers
Steve

[Matt Palmer_2]

Hi,

I had this exact same issue, and it was a problem with the Front-to-back assembly that connects the Insight display front panel to to Midplane.

Have you checked to see if there are any problems with the connection from Front to back assy?

Regards

Matt

[Shadders]

Hi Matt, Thanks for your reply - are you referring to the Insight Display pass-through board? Its the board on the left hand side (if you are stood in front) of the enclosure.

[Matt Palmer_2]

Hi,

that may be it, it basically runs the depth of the chassis(behind the metal plate on the left) and the cable from the Insight display connects to it at the front, then the rear part of it connects into the midplane assy.

regards

Matt

[Shadders]

Hi Matt, #

Did HP come out & swap it for you? I have the service manual & its not (according to the guide) a customer replaceable part. As far as I can see - its going to be a major power down to swap it.

Thanks again for your help.

I'll let you know how things progress.

Regards
Steve

[Shadders]

Hi Matt,

Did HP come out & swap it for you? I have the service manual & its not (according to the guide) a customer replaceable part. As far as I can see - its going to be a major power down to swap it.

Thanks again for your help.

I'll let you know how things progress.

Regards
Steve

[Matt Palmer_2]

Hi,

yep the entire chassis has to be powered down. I requested an engineer as I had never replaced it before and wasnt too sure about best practice, but as far as I know you can replace it yourself. You might need someone to help you removing the midplane first so you can see what you are doing.

I think theres only 4 screws to remove along the left hand side to swap the part out. As long as you are happy unplugging Insight display and midplane, you can probably sort yourself.

regards

Matt

[Phil uk]

@SteveCrang,
After you replaced the chassis, did you re-enter via CLI the Product Number, Serial Number of the Chassis, and the power configuration?
I'm no expert with VC, but i think the VC Domain is "linked" to the Chassis Serial Number.
Also, verify the Midplane is "squarely" seated as this can cause problems.
As you know, the Insight Panel slides left to right, check the cable behind the Panel hasn't been pinched, and that you have no bare wires showing as a result of the insulation being damaged as this can "short" the panel.
There is also, as previously mentioned by others, the connection between the Midplane and the PCB board on the side of the chassis which ultimately connects to the Insight display. If this is not "connected" properly, then you will have a blamk panel.
Try re-seating the Midplane a few times.
I think that strictly speaking, the Midplane is not a customer replaceable part, so probably best to get HP involved.
Regards,
Phil

[Shadders]

OK,

Firstly, thanks for all your suggestions.

I replaced the front to back board - no change. I replaced display - no change. I replaced the midplane. Bingo. All back to life.

However.... The serial number has to be entered - if you dont put it in, there is still an option to read the config from the VC modules & commit to NVRAM. As it was 1am I did this.

Then - one of the FC-VC modules refused to communicate with the OA.

It turns out this was faulty. I personally think that this has 'locked' something on the midplane & stopped the OA's from communicating.

HP replaced the module & the whole enclosure is now fully functional.

Thanks again for all your efforts. If I could - I would award points - but it wasnt me who started the thread...

Regards
Steve