BL685G7 Mezzanine port mapping

[Roberto Fabretti]

Hello

I have a BL685G7 I want to install in a c7000 enclosure.
This enclosure has
two pass trough modules in interconnect bays 3 and 4
two san switches in interconnect bays 5 and 6
an infiniband switch in interconnect bay 7

In which mezzanine slot should I install a fiberchannel HCA and an infiniband HCA in order to connet to the correct switch

If I put infiniband in mezzanine slot 3 it will be connected to the SAN switches
If I connect it to slot 2 it will connect to the passthrough
How it it possible to force the connection to the infiniband switch?

[Cederberg]

Hi. Have a look at the file I attached. it explains the portmapping for half and full height servers in C7000.

With a dual port FC mezz card you should put it in mezz 2 to make them connect to interconnect bay 5 and 6

a dualport infiniband mezz in mezz slot 3 should connect to interconnect 7 and 8

Regards.
//Cederberg

[Roberto Fabretti]

Hello

Thanks for replying but if I put it in Mezz 3 I have the following errors

Nov 24 10:31:32 OA: Blade in bay #6 status changed to OK
Nov 24 10:32:26 OA: Mismatching I/O was detected on Interconnect 5, Port 14.
Nov 24 10:32:26 OA: Mismatching I/O was detected on Blade 6, Mezzanine Card 3, Port 3.
Nov 24 10:32:26 OA: A port mismatch was found with server blade bay 6 and interconnect bay 5
Nov 24 10:32:26 OA: Mismatching I/O was detected on Interconnect 6, Port 14.
Nov 24 10:32:26 OA: Mismatching I/O was detected on Blade 6, Mezzanine Card 3, Port 4.
Nov 24 10:32:26 OA: A port mismatch was found with server blade bay 6 and interconnect bay 6
Nov 24 10:32:26 OA: Blade 6 is reporting nominal health status.
Nov 24 10:32:27 OA: Blade in bay #6 status changed to OK
Nov 24 10:32:27 OA: Blade in bay #6 status changed from OK to Degraded
Nov 24 10:33:16 OA: Blade in bay #6 status changed to Degraded

Best regards

[The Brit]

Is the infiniband HBA a 4-port card?

If it is , then ports 3 and 4 on the card will map to IC Bays 5 & 6. Which would explain your message.

Dave.

[Roberto Fabretti]

Nope it is a 2 port card

[The Brit]

Alternatively, I assume that your new bl685 is in bay 6, and the interconnect modules are as you specified, i.e. FC in 5 & 6, and infiniband in 7 & 8.

If this is true, then the messages indicate that the infiniband HCA is installed on Mezz slot 2. Mezz 2 maps ports 1 & 2 to IC modules 5 & 6. If it is a quad port card, then ports 3 & 4 map to Modules 7 and 8.

For Mezz 3, the opposite is true, i.e. ports 1 & 2 map to modules 7 & 8, and ports 3 & 4 map to modules 5 & 6.

seems complicated, but it is really not.

if in doubt, flip the mezz cards.

Dave.

[Roberto Fabretti]

Here is what happens if I put IB mezz card in mezz slot 2

Strange isn't it?

Nov 24 17:57:57 OA: Blade in bay #6 status changed to OK
Nov 24 17:58:50 OA: Mismatching I/O was detected on Interconnect 5, Port 6.
Nov 24 17:58:50 OA: Mismatching I/O was detected on Blade 6, Mezzanine Card 2, Port 1.
Nov 24 17:58:50 OA: A port mismatch was found with server blade bay 6 and interconnect bay 5
Nov 24 17:58:50 OA: Mismatching I/O was detected on Interconnect 6, Port 6.
Nov 24 17:58:50 OA: Mismatching I/O was detected on Blade 6, Mezzanine Card 2, Port 2.
Nov 24 17:58:50 OA: A port mismatch was found with server blade bay 6 and interconnect bay 6
Nov 24 17:58:51 OA: Blade 6 is reporting nominal health status.
Nov 24 17:58:51 OA: Blade in bay #6 status changed to OK
Nov 24 17:58:51 OA: Blade in bay #6 status changed from OK to Degraded

[The Brit]

That is weird.

So here's a really stupid question, (not meant to sound insulting). Is the Mezz card really an Infiniband card.

On the face of it, this seemed like a simple port mapping issue, however....

This second set of errors clearly indicates that this is not the case.

The first set of errors is indicating a mismatch on Mezz3, Port 3 (which doesnt exist) with Module in bay 5 (which would be possible if Port 3 existed). Similarly, the non-existent Port 4 is complaining of a mismatch with Module 6

At this point I step down. I'm as confused as you.

Good Luck

Dave.

[Cederberg]

Which firmware are you using on the Onboard Administrator cards?

Must be some kind of bug. You need to have at least version 3.11 to support G7 blades. And in an retracted firmware version 3.20 i find what is closest to the problem you are discribing:

"Fixed an issue where extra ports were incorrectly reported for CNA Mezz in slot 1 of any full-height server blade. (1001067813)"

So my advice is to upgrade the enclosure to 3.21 and have a look at http://h18004.www1.hp.com/products/blades/components/c-class.html#tab3_content

so you don't run any unsupported firmware. The order you need to update firmware in an C7000 is Bios of the servers, ILO of the servers and then OA card.

[gregersenj]

Roberto

Post a show all output and attache the file.

How to:
Use a terminal emulator. Ei Putty.
Set it up to log/capture the console output
telnet into the OA.
Login using same credentilas as for the gui.
at the prompt, use the command:
show all
Enter.
Attach the file here.

and I will check it for you.

Quick port mapping over view:
Embedded NICs (LOMs)
are mapped to interconnect 1 and 2.
Mezz 1 is mapped to inteconnect 3 and 4.
mezz 2 is mapped to interconnect 5, 6, 7, and 8.
Full hight servers got a mezz 3. Wich is mapped to 5, 6, 7 and 8 also.

Mezz cards must be put in a mezz slot, where there's a matching interconnect.

If you put a FC HBA in Mezz 1, and you have got ethernet interconnect in IC 3 and 4. Then you got a mismatch.

BR
/jag