2 enclosure VC domain Stacking Links

[r_a]

Hi All,

I have 2 * c7000 enclosures.  Each of them have a flex fabric card in slots 1 and 2, and flex 10 cards in slots 3 and 4.  I've connected the stacking links as I can make out from the documentation, so...

The flex fabric io modules have connections to the adjacent one via the internal ports x7 and x8, and to the other enclosures via x4.  None of these ports are shared.

The flex10 io modules have connections to the adjacent one via the internal ports x7 and x8, and to the other enclosures via the infiniband type connector on x1.  None of these ports are shared.

These are shown below from the terminal connection to the vc, but I also get a stacking failed/degraded error.  Is there a problem with the way that these are connected?  They are on the 3.30 version of firmware...

->show stackinglink
Connection Status : Failed
Redundancy Status : Degraded

===========================================
Link  Speed  Connected From  Connected To
===========================================
1     10Gb   enc0:3:X1       enc1:3:X1
2     10Gb   enc0:3:X7       enc0:4:X7
3     10Gb   enc0:3:X8       enc0:4:X8
4     10Gb   enc0:4:X1       enc1:4:X1
5     10Gb   enc0:1:X4       enc1:1:X4
6     10Gb   enc0:1:X7       enc0:2:X7
7     10Gb   enc0:1:X8       enc0:2:X8
8     10Gb   enc0:2:X4       enc1:2:X4
9     10Gb   enc1:3:X7       enc1:4:X7
10    10Gb   enc1:3:X8       enc1:4:X8
11    10Gb   enc1:1:X7       enc1:2:X7
12    10Gb   enc1:1:X8       enc1:2:X8

[Psychonaut]

Are you using different cables/connections for the FlexFabric vs the Flex-10?

[r_a]

The flexfabric io modules are using the proprietary 10 GBe DAC cables for the stacking links and the flex10 io modules are using the proprietary infiniband type cables for theirs. All connections are there and the LEDs are indicating that they are configured as stacking connections, but VCM seems to think that some VC enet modules are not properly connected via stacking links, so does that mean I need more?

[Psychonaut]

Do you have an SFP in CX-4 on the Flex-10's?

If Flex-10 modules are used the CX-4 port and port X1
are shared, the CX-4 port will be disabled.

[Hongjun Ma]

from show stackinglink, looks like you don't have cables connected between modules in the same enclosure.

slot 1/2: Flexfabric/Flexfabric

slot 3/4: Flex-10/Flex-10

you also need to connect between slot 1 and slot 3, slot 2 and slot4. All Ethernet(including FF) modules in a VC domain need to be connected.

you can try to add 4 cables.

enc0:slot1:X7 <-->enc0:slot3:X7

enc0:slot2:X7 <-->enc0:slot4:X7

enc1:slot1:X7 <-->enc1:slot3:X7

enc1:slot2:X7 <-->enc1:slot4:X7

so we borrow X7 from internal horizontal stacking for vertical stacking.

For more info about stacking, you can check

VC Setup and Installation Guide  6th Edition(8/11): 

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01732252/c01732252.pdf

Page 18 and Page 23.

Your case looks like diagram 1 in page 23. except you have 2 enclosures and your VC modules are in slot 1/2/3/4 instead of 1/2/7/8 in the diagram.

[r_a]

Hello,

Thanks for your reply.  The Flex10 modules were intended on being there to isolate management traffic from the networks present on the FlexFabric modules, so I think this is why the interconnects weren't specified between the two.  Would there be a way of having it configured so that there aren't any interconnects as you suggest between slots 1 and 3 and slots 2 and 4, or is that not possible?  I'm thinking that as VC is complaining about this that perhaps we could have 2 separate VC domains, one having the FlexFabric IO modules, and the other domain containing the Flex10 modules, or is this not possible? :)

Regards..

[Hongjun Ma]

you can't have 2 VC domains inside one c7000 enclosure.

if you really want to have physical mgmt traffic seperation, you can have a dedicated uplink from Flexfabric module for this management traffic.

which kind of mgmt traffic is this? all Virtual connect management traffic is through OA RJ-45 ports. I haven't seen other customers buying a pair of Flex-10 modules(and Mezz cards on servers) just for mgmt seperation.

[Psychonaut]

Doh!, I knew something was there looking me in the face.  We had a discussion awhile back about this sort of scenario -  http://h30499.www3.hp.com/t5/HP-BladeSystem/is-Virtual-Connect-stacking-required/m-p/5390151#M17809  

[r_a]

Hi,

Thanks for the link to the other discussion, it does indeed present a similar scenario to mine. 

As the network on the flex10 side is to be kept physically separate from those that are on the FlexFabric (that is separate Mezz NICs in the servers too at the customer's request), interconnects weren't used between them.  Although I can manage all of the IO modules within the VCM, it still shows a red cross on the stacking link. 

Should I just add the interconnects to keep it quiet about this, or is there a way to tell VCM not to flag this as an error on the management console?

If I do need 4 of these cables, I can see from the Quickspecs that there are a few options for the DAC, one is the JD095B which I think is the right one to use?

[Hongjun Ma]

there is no option to let VCM to discard or hide this stacking link error.

If the DAC part numbers shows in both Flexfabric and Flex-10 quickspec page, you can use it for stacking link.

[r_a]

Hi, Both of these part numbers appear in both the FlexFabric and the Flex10 Quickspecs. HP BladeSystem c-Class Small Form-Factor Pluggable .5m 10GbE Copper Cable 487649-B21 HP X240 10G SFP+ SFP+ 0.65m DA Cable JD095B Also I've finally managed to speak with someone at HP - he thinks that it's not possible to stack vertically between FC to F10 in the same enclosure!! He thinks that the configuration is correct, and that the whole thing needs to be resynchronised by reseating the OA modules, and then all the VC modules. I did try to argue against this, but I guess boxes need to be ticked before proceeding further..... Regards,

[chaney56]

I am having a problem with a similar config. I have two c-7000 enclosures. Each enclosure has VC Flex-10 ethernet modules in bays 1,2,5, and 6. I am using the cx-4 connectors for stacking. 

I have created a VC domain with the first enclosure. When I try to "find" the second enclosure I get an error telling me to verify the OA ip address and credentials for the second enclosure. The ip and credentials are correct. The second enclosure has never been part of a VC domain.

I also have an indication within Virtual connect that my stacking links are not redundant.

I could not find any documentation for stacking two enclosures with 4 Flex-10 modules each, but this is how they are currently connected:

Enc1 bay1-internal-bay2

     Bay2-cx-4 cable-bay6

     Bay6-internal-bay5

     Bay5-cx-4 cable-Enc2 bay1

Enc2 bay1-internal-bay2

     Bay2-cx-4 cable-bay6

     Bay6-internal-bay5

     Bay5-cx-4 cable-Enc1 bay1

Basically I have created a large loop using the internal and external connections. Is this configration correct, and could htis be what is preventing "finding" the second enclosure?

Thanks

Mike

[Hongjun Ma]

Hi r_a,

You mentioned "FC to F10" stacking. Not sure if it's typo. You can't stacking any VC-FC(Fibre Channel) module. In your case, you need to stack FF(FlexFabric) to F10. This is supported for sure. please check the link for setup and installation guide I sent out. Page 19 for single enclosure stacking multiple VC modules. there is a diagram showing to connect FF with F10

[Hongjun Ma]

Hi Mike,

i think they are two seperate issues

1) can't import 2nd enclosure.

all import process is done through internal mgmt ethernet to OA and then from OA mgmt port to 2nd enclsoure OA mgmt port. It doesn't use any VC faceplate ports. you may want to open a HP support case to get help. it should work.

2) stacking link cabling.

Your cabling is not 100%, the right cabling is shown in VC setup and install guide, page 23 left diagram. please check the link I sent out in earlier post

[r_a]

Hi Hongjun Ma,

Yes sorry, it was a typo.  It should have said FF to F10.  HP Support have also done a U-Turn and said that stacking links are needed in the same enclosure.

Thanks for your help.  I'll report back once these are in place, but they are taking 2 weeks for delivery...

Regards.

[chaney56]

Thanks Hongjun Ma.

I will take a look at the doc.

Mike

[chaney56]

Hongjun Ma,

That diagram shows multiple connections from the primary enclosure Enet bays 1 and 2. I am only using the X1 cx-4 connections on each Flex-10 module for external cabling stacking. Do I also need to use one of the other ports for stacking(x2, x3,x4 etc)?

Mike

[Hongjun Ma]

yes, Flex-10 has 8 port from X1-X8.  they are pretty much same in regard to speed and feature support. all of ports can do vertical stacking or uplinks.

two things to note:

1) X1 is shared between CX4 and 1st SFP+ port so you can't use them at the same time. In your case, just leave SFP+ port marked X1 empty because you already use CX4 interface.

2) Don't try to use X7 and X8 at the same time for vertical stacking. You need to leave at least one out of these two ports unused so the internal horizontal stacking can use that port. otherwise VCM will complain stacking error.

[chaney56]

Hongjun Ma,

Using the x2 ports for the additional connections, and the stacking links are now good. Talked to HP about the error trying to "find" the second enclosure. They could not figure out why I am getting the error, want me to delete the VC domain and start over. I will try that next week.

Thanks for your help.

Mike