OA Module Error : Flashing Fan LED's & No InSight Display

[miramar-tech]

We recently setup our C7000 in a new room, inially all powered up fine and seemed to be no problems until we were alerted of a degraded system. On logging into OA (Web) via the active OA1 we noticed every so often the connection dropped and wouldnt reconnect for a minute or so.. we then realised the entire chassis was actually rebooting each time the connection was dropped.

We assumed OA1 was at fault so switched across to OA2 where everything seemed to be fine once again. After a planned power down 12hrs later the chassis would then not boot up again. The InSight Display no longer powers up and we cannot connect via Ethernet. In an attempt to troubleshoot we have removed OA modules and tray and reseated everything. We have also completely removed OA1 assuming this to be the issue and now OA2 is in the primary slot.

This is what we see when power is provided:

• All PSU's have green light and we believe all to be working with no issues.
• All fans kick in with green lights across the board (initially).
• OA2 (now in left slot) LED's flash green a few times and then the health LED flashes RED - at this point 8 of the fans start flashing AMBER and 2 of the fans LED's show nothing at all.
• There is no InSight Display working at all at any point.
• Fans continue to work at full power.

We can connect to OA2 via serial connection however we seem to be unable to send any commands, however the repeated output below is shown:

HP Onboard Administrator booting...
Oops: machine check, sig: 7 [#1]
**bleep**: 00000740 LR: 07FD55A8 CTR: 00000000
REGS: c0367f50 TRAP: 0202 Not tainted (2.6.17.9-udog)
MSR: 00021000 <ME> CR: 28000028 XER: 20000000
TASK = c030f0f8[0] 'swapper' THREAD: c0338000
GPR00: FFFFFFFF 07FB2A78 C030F0F8 07FFFBFF 07FFA100 07FFA100 00000020 07FFA0FC
GPR08: FFFA70FC 07FB2B54 07FD3000 47FB2A78 01FCA055 00000000 07FFFC00 08053000
GPR16: 00000001 00000001 FFFFFFFF 007FFF00 07FF9B40 00000000 C0310000 00000000
GPR24: C0310000 0000A000 07FD3000 07FB2B54 07FB2B84 F8000F80 07FFFE54 07FB2B38
Call Trace:
Instruction dump:
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
Kernel panic - not syncing: Attempted to kill the idle task!
<0>Rebooting in 180 seconds..

We are now at a complete loss on where to go and what to do now other than ordering new OA modules and tray in the hope that replacements will resolve the issue. We are however hesistant to do this since it sounds suspicious that 2 OA modules would fail within a few hours of each other so assuming there may be more to this than we can guess with our current limited knowledge on these BladeSystems.

Hopefully someone on this board will recognise the issue or understand the output I have pasted above and be able to provide us with some very much needed advice on what to do next!

Thank you

[Robert_Jewell]

Personally I would guess a faulty OA tray.  Let us know how it goes!

-Bob

[Jimmy Vance]

if you physically relocated the chassis it is possible that something came loose.  I'd try to re-seat all the components and make sure none of the backplane connectors are damaged. Maybe try just running with one OA module at a time

[miramar-tech]

Thank you for the response.

All the backplance connectors look to be fine and we've already tried reseating everything and as it is right now it currently only has once module in it so no luck with your suggestions I am afraid.

I am waiting on delivery of 2 new OA modules which I am hoping will resolve the issue however part of me isn't so confident since from my reading up online these OA modules don't fail that often and for me to have issues with 2 within less than 24hrs of each other seems to make me think there is something else at play here.

I'm not sure what firmware version we had installed on these and I rememeber reading somewhere that there were issues relating to a partiuclar firmware on the VC modules that can have adverse affects on the OA module in some circumstances.. I can't quite remember the specific details or if it was relevant to my issue but it is something that is shouting at me in the back of my mind... does anyone have any idea what I am thinking about here and whether it could be related to me issue?

[Jimmy Vance]

Another suggestion is to pull all the I/O modules and servers and see if it goes into orbit with just an OA installed. The OA boot message does appear that there is something corrupt in the firmware image, but shutting it down and moving it really shouldn't have an effect on the firmware load.  I't been to long since I've done any work with a C7000, you might call support and see if there is a way to flash the OA while it is in this state