BL460c G1 and emulex LPe1105e FC link stange problem

[Rodrigo Verissimo]

Hello. We have a very strange problem.
A week ago, one of our blades died. That is the motherboard was diagnosed to be out of order. The motherboard was replaced, and the system worked again but FC connectivity was lost.

Emulex one command manager reports "no link" for BOTH ports connected on the two interconnect brocade switches
and in statistics a several hundred thousand of "loss of sync" and "invalid TX words"
On the brocade switches, the ports show as permanently disabled, when we try to permanently enable it, it says "operation not supported"

We checked that a licence for the port is reserved in the brocade switch manager applet.
There is no zoning
Card is sloted on mezzanine 2

We tried :

Another FC card supplied by HP
Yet another motherboard with the old and new FC card on it
Forced 1, 2, 4 Gbps
Changed the bay on the C3000 to isolate a potential FC internal switch port problem

What i think is going on ?
Well what annoys me is that all ports except 1,3,4,5 show "permanently disabled" so i think a switch factory reset would help.

Any advice would be very appreciated !!

[Matti_Kurkela]

Sounds like a communication problem between the blade and the chassis.

As the FC card is in mezzanine slot 2, one of its FC ports should map to I/O module slot 3, and the other to I/O module slot 4. You don't say that explicitly, but as you say you have two Brocade interconnect switches I guess you've got this covered.

In c-class blade hardware, the OA verifies the mezzanine cards on the blades are compatible with the respective I/O modules before it allows the blades to power up. I would guess it also works the other way: when you plug in a blade that has no mezzanine card and the chassis has an I/O module that corresponds with the blade's empty mezzanine slot, the OA communicates with the I/O module(s) to disable the corresponding port(s).

Your situation almost looks like the existence of the FC mezzanine card is not detected by the OA (so the switch ports are perma-disabled and cannot be enabled because the chassis/switch thinks there is no card), but the blade itself can use the card and is essentially trying to talk to disabled switch ports and is hearing back random noise.

Is the mezzanine card *fully* plugged in? You might want to unplug it and check the connectors carefully: there might be damage or some non-conductive debris on some connector surfaces.

The next question is: what are the firmware levels?
- blade iLO
- blade BIOS
- chassis OA firmware
- Brocade interconnect switch firmware

If your chassis firmware has never been updated, the replacement blade motherboard might have had significantly newer firmware versions than the rest of your chassis has. Some versions of certain firmware elements don't work with all other existing firmware element versions.

Or vice versa: if your chassis is up-to-date but your replacement blade motherboard contained very old firmware, it might be necessary to upgrade the BIOS and blade iLO firmware to levels matching the rest of your chassis.

MK

[Rodrigo Verissimo]

Hello thanks for your detailed answer.

We already cleaned the mezzanine slot and the blade external/chassis connector. No effect.

here are the version of the firmwares on a blade that works :

BIOS : I15 04/26/2010
ILO : 2.01 08/04/2010
FC HBA : 2.72A2

and the versions on the problematic blade :

BIOS : I15 10/25/2010
ILO : 2.05 12/16/2010
FC HBA : 2.80A4

And finally the version of the OA :

2.10

The only "outdated" firmware version is the OA. Maybe it is incompatible with recents versions of ILO on some aspects ?

and you were right, in ILO admin web page, it says it has no FC HBA card on mezzanine 2.

So maybe the best is to update OA ?

Thanks again !