vc-enet and flex-10

[atmorell]

Hello,

We have a single C7000 (1 domain) bladecenter with two vc-enet modules installed in bay 1 and 2. Today we got two flex-10 modules and they have been installed in bay 5 and 6 - Bay 3 and 4 is used for fiberchannel. All the VC modules is running firmware 2.31.

I am getting a stacking error:

The Virtual connect Manager has discovered the VC-Enet module stacking listed below. The Connection Status below indicates whenether all VC-Enet modules are interconnected and accessible.

Connection status: FAILED

from / to
bay1:portx0 -> bay2:portx0
bay5:portx7 -> bay6:portx7
bay5:portx8 -> bay6:portx8

Do I need to cable the vc-enet in bay1 with the flex-10 in 5? - CX4 (and 2 to 7) The stacking error has appeared after the flex-10 modules was installed, so the link from bay1 to bay 2 must be okay.

[The Brit · Accepted solution]

Hi.

Yes, you should run CX4 cables from Bay1 Port X1 to Bay5 Port x1, and from Bay2 Port X1 to Bay6 Port X1.

It is not necessary to link Bay5 to Bay6. They are hard linked across the Midplane via port X0 on each module.

Stacking links are only applied vertically.

HTH

Dave.

[WFHC-WI]

Hi atmorell,

Dave is correct, but this is really an optional step depending on your planned network configuration.

Do you plan on blade NICs 1/2 needing to connect to network resources across the uplinks in the Flex 10 modules? Do you have NICs installed into mezzanine slot 2 of your blades that need to connect through the standard VC-Enet modules?

If either answer is "yes" then you need to use the CX4 cable to attach
bay1:X1 -> bay5:X0
bay2:X1 -> bay6:X0

If the answer to both is "no" then you can safely ignore the stacking error. The three "from/to" links that you listed are all internal and are likely working fine. The page unfortunately still shows "FAILED" as the status if there are any bays not vertically linked.

good luck, points are appreciated if our answers helped

[atmorell]

Hello,

Thank you for answering.

We have both old G5 and new G6 servers in the chassis. I have been told that we need to remove the old PC-ENet modules from bay1/2 and move the flex-10 modules to bay1/2. If we don't do this all the onboard networkcards will run 1Gb, as there is no network cards installed in mezzanine 1/2. This makes sense..

The flex-10 modules will each have 2 links to the core switch. Is it correct that we will have 40Gb for the servers? - 20Gb if one flex-10 module FAILS.

How many nics will be avalible for each blade with this configuration? As I understand it, each blade will support 4 nics/uplink. We need to access minimum 5 networks on each blade (VMware) and would like to have flex-10 doing the VLAN tagging etc. as it is hardware accelerated.

Best regards.
Asbjørn Morell.

[WFHC-WI]

When using Virtual Connect modules, only one of the modules in horizonally-adjacent pairs is active at any time. This means your active bandwidth to the enclosure will always be 20GB.

This also means you will only be able to connect to 4 networks at a time on the G6 servers and 1 network on the G5 servers unless you do one of these two things:
>>Install additional network modules in the enclosure and mezzanine NICs into each blade.
OR
>>Enable trunking on the server rather than the Virtual Connect.

good luck!

[The Brit]

I just want to respond to an earlier reply by "WFH-WI". Maybe I am misunderstanding, however it seems to me that he stated, or at least implied, that there are situations where the vertical stacking links are not required.

I must emphasize that the Virtual Connect Manager (VCM) ONLY runs in either Bay 1 or Bay 2. Without the vertical stacking links, VCM knows nothing about the other VC Ethernet modules in bays 5 & 6. It recognizes that there are other VC ENet modules installed (hence the FAILED messages) however it cannot manage the ports or uplinks on those modules and you will not be able to provision profiles which require reference to the ports on those modules.

As I said at the beginning, I'm probably misunderstanding the statement, but unless you are using your flex10's as some weird kind of passthrough, then I would say that the Vertical Stacking Links are necessary.

Dave.

[WFHC-WI]

It's true that VCM runs only in bay 1 or 2, but their internal stacking link doesn't imply that other interconnect modules go unseen without external stacking. We are running this scenario in several enclosures.

Each onboard and mezzanine NIC is hard-coded to exit the enclosure via a single module. If you want traffic to exit the enclosure via uplinks on a different module, you need stacking. If you have different virtual networks configured for different module pairs (as we do), external stacking links aren't necessary.

-Jamie

[Adrian Clint]

You only need to stack if you want traffic to flow 1/2 <> 5/6. If there is no need/design for this then you can ignore stacking link failures.

For example - if you were to turn on "private" for every network then there is no comms between downlinks within the VC... so no need for stacking.

A good reason to stack is to have a cluster heatbeat network or a vmotion network that could be using different NICs on different servers. Then all the heartbeat or vmotion traffic can run around within the 4 VC modules and never leaves the enclosure.

[The Brit]

Adrian, Jamie,

Thanks for the information, I didn't actually realize that. In retrospect, what you say makes sense, however I am not sure about the advisability of imply to people that it is "OK to run in a degraded state". Ignoring these warning messages is probably habit-forming and will probably bite you later in the game.

Since 0.5m CX4 cables cost <$100, the smart thing would be to fix the problem rather than ignore it.

In addition, adding the stacking links would open up options for pathing and redundency which are not available right now.

Dave.