To stack? or not to stack?

[robshep500]

In the enclosure there is a pair of 1/10GbE VC Enet modules and a pair of VC flex-10 modules. Each are side by side and are stacked horizontally as usual.

the ports X1 in both Enet modules form a Shared uplink set for VNET_A, and are connected to two core switch to provide active/standby.

the ports X1 in both Flex-10 modules form a shared uplink set for VNET_B and are connected to a different core switch, which provides active/standby also.

Although VNET_A and VNET_B carry the same numeric vlan, both LANs may be considered different and external switching in this scenario is an appropriate way for the two VNETs to communicate.

My question is this: do I need to stack the two horizontal rows together?

Each VNET appears to be working as needed.
Each VNET and SUS are independant of the other.

There is an error showing in VCM, but regards total module connectivity and in this case, I don't think it is needed. Is there a way to suppress this message?

Please indicate if you think I have missed something or that the tolopogy stated may fail in certain circumstances.

With thanks

Rob

[The Brit]

Hi Rob.

As you have probably noticed, if you don't have stacking links the VCM attaches a nice big RED "X" and informs you that a critical error has occurred and that your Enclosure is degraded. This is because unstacked VC Modules cannot live in the same VC Domain. (VCM effectively considers the "enclosure" to be the mimimum domain).

I dont know if there is any serious downside to the configuration, however there are many blade-smart people on this forum who will know. The thing that I would be concerned about is the equivalent of "split brain" syndrome (if you are an oracle guy) or cluster partition if you are not.

It would seem that you could find yourself with two independent io subsystems which are not coordinated by VCM. Remember VC Manager resides on Module 1 (or module 2) if there is a failover. It does not and cannot reside anywhere else (at least that is my understanding as of 6 months ago). If this is true, then it is difficult to see how VCM can communicate effectively with the other Ethernet modules if the stacking links are missing.

just my 2 cents worth.

Dave.

[robshep500]

Thanks for your suggestion, this was the kind of thing I'm unsure about.

The reason I ask is that there isn't enough of the right combination of 10GbE link capability in the DC to perform stacking and uplinking.

But, of course, I used VCM (running on a VC ENet module in slot-1) to configure the second SUS, which operates from bay 5&6, and then to assign this to a server blade.

So it appears VCM(Enet 1 and/or 2) can still comunicate with Flex10 5 and 6, without stacking.

if both 1 and 2 then fail, maybe VCM won't be available from 5 or 6, I shall test this failure mode now and report back....

[The Brit]

My comment regarding "communicating" was probably not a good idea, it tends to "fuzzy" the conversation.

You should concentrate your thoughts on the concept of "VC Domain Membership", and what the consequences might be for modules which are NOT included in the VC Domain.

I appreciate your problem with trying to divide the available connections between uplinks and stacking links. Hopefully someone who is more familiar with the VC domain concepts will throw in their comments, (and enlighten us all).

Dave.

[robshep500]

Thanks TheBrit,

An update to this is that although the second (unstacked) shared uplink set and the second Vlan24 VNet states "up and running fine", no communication occurs between servers allocated to this VNet and the external switch.



The topology is shown on the attached JPG.

On server 2 - ethtool reports "link down".
All switchports are enabled and forwarding the particular vlan as expected, but no mac-addresses are seen from the switchports.

Is this a symptom of not stacking?

Apart from the "critical" alarm of loss of total connectivity, there is nothing else (visually, or in the logs) that suggests VNet_B shouldn't be available.

With a great many thanks for any more insight,

Rob

[The Brit]

Rob,
I would expect that your comment about MAC addresses is almost certainly a VC Domain issue, especially if you use Profiles.

VCM (on Module 1) manages MAC addresses for NICs on all of the blades via the VC Domain so if the Flex10 modules are not in the Domain, then it wouldn't surprise me if MAC addresses are not visible from the Network switches. If you are virtualizing the MAC addresses, then the problem is likely to be compounded.

Bottom line is that if you are using VC, then it is not worth the risk of running with some of your VC ENet modules outside of the domain. VCM expects all VC ENet modules to be included in the domain and I suspect you will see many obscure (and probably detrimental) effects of running with your Flex10's outside the domain.

You might as well "bite the bullet" and use the CX4 connector to stack with the 1/10G modules.

Dave.

[robshep500]

Thanks,

The datacentre doesn't have any optical ethernet ports to uplink 10GbE to. So I need the CX4 on the Flex-10s for uplink and not stack.

My solution will thus be to create a "rudimentary" uplink using a copper port on the Ethernet modules and a copper-RJ45 SFP+ on the Flex-10 modules.

This should satisfy the needs of the VCM for total stacked connectivity.

Many thanks

Rob

[The Brit]

That should work.

Dave