2 enclosure VC domain Stacking Links

[r_a]

Hi All,

I have 2 * c7000 enclosures.  Each of them have a flex fabric card in slots 1 and 2, and flex 10 cards in slots 3 and 4.  I've connected the stacking links as I can make out from the documentation, so...

The flex fabric io modules have connections to the adjacent one via the internal ports x7 and x8, and to the other enclosures via x4.  None of these ports are shared.

The flex10 io modules have connections to the adjacent one via the internal ports x7 and x8, and to the other enclosures via the infiniband type connector on x1.  None of these ports are shared.

These are shown below from the terminal connection to the vc, but I also get a stacking failed/degraded error.  Is there a problem with the way that these are connected?  They are on the 3.30 version of firmware...

->show stackinglink
Connection Status : Failed
Redundancy Status : Degraded

===========================================
Link  Speed  Connected From  Connected To
===========================================
1     10Gb   enc0:3:X1       enc1:3:X1
2     10Gb   enc0:3:X7       enc0:4:X7
3     10Gb   enc0:3:X8       enc0:4:X8
4     10Gb   enc0:4:X1       enc1:4:X1
5     10Gb   enc0:1:X4       enc1:1:X4
6     10Gb   enc0:1:X7       enc0:2:X7
7     10Gb   enc0:1:X8       enc0:2:X8
8     10Gb   enc0:2:X4       enc1:2:X4
9     10Gb   enc1:3:X7       enc1:4:X7
10    10Gb   enc1:3:X8       enc1:4:X8
11    10Gb   enc1:1:X7       enc1:2:X7
12    10Gb   enc1:1:X8       enc1:2:X8

[Psychonaut]

Are you using different cables/connections for the FlexFabric vs the Flex-10?

[r_a]

The flexfabric io modules are using the proprietary 10 GBe DAC cables for the stacking links and the flex10 io modules are using the proprietary infiniband type cables for theirs. All connections are there and the LEDs are indicating that they are configured as stacking connections, but VCM seems to think that some VC enet modules are not properly connected via stacking links, so does that mean I need more?

[Psychonaut]

Do you have an SFP in CX-4 on the Flex-10's?

If Flex-10 modules are used the CX-4 port and port X1
are shared, the CX-4 port will be disabled.

[Hongjun Ma]

from show stackinglink, looks like you don't have cables connected between modules in the same enclosure.

slot 1/2: Flexfabric/Flexfabric

slot 3/4: Flex-10/Flex-10

you also need to connect between slot 1 and slot 3, slot 2 and slot4. All Ethernet(including FF) modules in a VC domain need to be connected.

you can try to add 4 cables.

enc0:slot1:X7 <-->enc0:slot3:X7

enc0:slot2:X7 <-->enc0:slot4:X7

enc1:slot1:X7 <-->enc1:slot3:X7

enc1:slot2:X7 <-->enc1:slot4:X7

so we borrow X7 from internal horizontal stacking for vertical stacking.

For more info about stacking, you can check

VC Setup and Installation Guide  6th Edition(8/11): 

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01732252/c01732252.pdf

Page 18 and Page 23.

Your case looks like diagram 1 in page 23. except you have 2 enclosures and your VC modules are in slot 1/2/3/4 instead of 1/2/7/8 in the diagram.

[r_a]

Hello,

Thanks for your reply.  The Flex10 modules were intended on being there to isolate management traffic from the networks present on the FlexFabric modules, so I think this is why the interconnects weren't specified between the two.  Would there be a way of having it configured so that there aren't any interconnects as you suggest between slots 1 and 3 and slots 2 and 4, or is that not possible?  I'm thinking that as VC is complaining about this that perhaps we could have 2 separate VC domains, one having the FlexFabric IO modules, and the other domain containing the Flex10 modules, or is this not possible? :)

Regards..

[Hongjun Ma]

you can't have 2 VC domains inside one c7000 enclosure.

if you really want to have physical mgmt traffic seperation, you can have a dedicated uplink from Flexfabric module for this management traffic.

which kind of mgmt traffic is this? all Virtual connect management traffic is through OA RJ-45 ports. I haven't seen other customers buying a pair of Flex-10 modules(and Mezz cards on servers) just for mgmt seperation.

[Psychonaut]

Doh!, I knew something was there looking me in the face.  We had a discussion awhile back about this sort of scenario -  http://h30499.www3.hp.com/t5/HP-BladeSystem/is-Virtual-Connect-stacking-required/m-p/5390151#M17809  

[r_a]

Hi,

Thanks for the link to the other discussion, it does indeed present a similar scenario to mine. 

As the network on the flex10 side is to be kept physically separate from those that are on the FlexFabric (that is separate Mezz NICs in the servers too at the customer's request), interconnects weren't used between them.  Although I can manage all of the IO modules within the VCM, it still shows a red cross on the stacking link. 

Should I just add the interconnects to keep it quiet about this, or is there a way to tell VCM not to flag this as an error on the management console?

If I do need 4 of these cables, I can see from the Quickspecs that there are a few options for the DAC, one is the JD095B which I think is the right one to use?

[Hongjun Ma]

there is no option to let VCM to discard or hide this stacking link error.

If the DAC part numbers shows in both Flexfabric and Flex-10 quickspec page, you can use it for stacking link.