VC mode Active/Active : Host unreachable when using virtual connect 2

[james_sr]

Hi everyonne, 

I'm currently facing an issue with our virtual connect. We have a c7000 with two virtual connect modules 10G/24 port module (firmware version 4.01). They are configured in active/active mode in defining 2 Shared uplink set (SUS1_VC1 & SUS2_VC2).  By the way,  we can see "linked-active" for the two sus defined.  

A server profile has been applied to a proliant BL460c Gen8.. 6 NICs then showed up with 2 fcoe adapters.  We installed a windows server 2008 as os . When I was testing the teaming with 2 adapters, I noticed  than the traffic went only through the VC1. 

To be sure, I've broken the teaming and set up only one adpater. On the virtual connect Manager side, assigned to this adpater to use VC2, and I'm now unable to reach the host from outside, can't ping. 

But if I changed it, to use VC1 and I can ping again.  The same network are defined on each virtual connect. 

Thanks for you help, 

[Casper42]

Make sure there are no typos involved in SUS2_VC2 in regards to the individual VLAN you are using on the Server profile (correct VLAN ID?)

Make sure the upstream network switch port for SUS2_VC2 is also configured correctly. (Compare it to the network port for SUS2_VC1)

Lastly, how long did you wait?  Sometimes it takes 5 or 6 seconds before the MAC Address on the upstream switch learns that it needs to use a new port to get to VC Bay 2 for that host.  There is a setting in VC called Fast MAC Cache failover that allows us to adjust that within VC.  Most switches have some kind of timer of their own.

[james_sr]

Hello,

Thanks a lot for you reply. I checked the the vlan ID and it's the same that was set up on both VCs.  I checked also at our switch level and both ports have the same configuration.

And last time I've tried, I've waited for a day and even a full week-end but same issue: ping doesn't respond. 

But this morning, I've tried something else. I kept the same configuration by using only one adapter on OS side, and have the FlexNIC defined to use VC2 on Virtual Connect Manager side. At the switch level,  I shut down the port and as soon as it was effective, I got my ping responding.  Then I re-enabled the port, and lost the ping again.

I think now that I doesn't really have an active/active mode. We were adviced to not use LCAP because we are using two VCs.  And that's what we did. On cisco side, for the port channel, LACP is not used.  I was doubting about the config. but you replied to me in an old thread  "You never Port Channel (LACP) ports together if they are going to DIFFERENT VC modules." (thanks again) .

Any thoughts?

thanks

James.

[Casper42 · Accepted solution]

Its not just LACP, its Port Channeling on the Cisco side of any kind.

If you have 1 physical link going to VC1, and 1 physical link going to VC2, you should not be using Port Channels at all.

The individual ports should be configured as "switchport mode trunk" to carry multiple VLANs, but again, no Port Channel.

Now, if you have 2 physical links attached to VC1 and 2 more attached to VC2, then both links terminating in VC1 can be Port Channeled (including vPC or VSS if you have that licensed and configured).  But here you still need *2* distinct Port Channels, 1 for each VC module.

Basically you can never create a Port Channel ACROSS multiple VC modules.

At least not with any firmware up to and including 4.1, and even then i don't see this on the roadmap for the next 2 versions either.  But that could always change for future versions.

PS: Where are you located roughly?  City & Country?

[james_sr]

Oh ok. Our config. matches the first case : We have 1 physical link going to VC1 and 1 going to VC2. 

But there is something that's puzzling me. Let's say we switch to the ports trunk configuration and remove the port channel on the Cisco switch. The virtual connect modules should thus be in a true active/active mode.  We then team 2 adapters on the OS side.

When this host sends packets to another host, the traffic will go through both links (VC1, VC2).  The issue is that I will have the same "virtual mac address" coming from both uplinks. Spanning tree will then do its role and block one port.  

I think it's normal.  At the result, I will end by having one active link.

Am I wrong? Or Iam I misunderstanding or missing something?

Thanks again for your advices.

Actually, we are located in La Reunion, small island in Indian Ocean.  :)

[james_sr]

Hi Casper, 

Ok. After keeping playing around with it, I think I now understand what you explained to me yesterday. I was worrying about removing the port channel and setting up  2 "standalone" trunk port instead. But in fact, VC has a similar "technology" than spanning tree : It's the " Network Loop Prevention" protection which is enabled by default.  It allows to have only one logical path to the external switch. That's why it's not a problem to not bother about loop or spanning-tree at the cisco level. right?  

My another mistake was related to my understanding of  the concept "active/active" when using multiple VCs.  I thought that when you had an active/active across multiple VC with for example 1 uplink (let's say 10G) each to an external switch, you would end up with a full and usable 20G link aggregation in addition of a redundancy.

It sounds to not work like that.  If you set up your hosts which have 2 adapters (each associated with a different VC) to handle traffic from the same vlan, it will indeed still use only one uplink at the end. And if this uplink failed, it would enable the other one.

 But that doesn't mean that you have a VC (let's say VC2 here) in standby mode because you could it separately to handle another traffic from a different VLAN for example. Both could be used at will except if you tried to team then in order to have  more brandwith.     

Am I right ? or still wrong about it?  I tried to be as clear as possible  :) 

Thanks again for your precious help. 

[Casper42]

Loop Prevention is actually looking for Loops inside the OS on the Blade. Something outside our control but we added that feature to help prevent outages due to bad configs (you have no idea how many people think "Bridge connections" in Windows is the same as Teaming).

Anyway, it goes like this.
Active/Passive, as most people on my team call it, is 1 SUS or VC Network that has Uplink ports from VC1 and VC2. In this mode, since we cannot form an LACP LAG across modules, we automatically use only 1 uplink and leave the other admin down. Similar to Spanning tree.

Active/Active is really a bit of a trick under the hood.
Active/Active is really 2 different SUS (and corresponding vNets in the SUS) and you assign one of them to VC1 and the other to VC2. What happens under the hood is VC actually takes VLAN10_A that you have on VC1/SUS_A and assigns it an internal VLAN ID (you never see this) of something like 1010. Then VLAN10_B that you have on VC2/SUS_B is created and it gets an internal ID of 2010.
So even though the uplink VLAN ID is 10 for both, and the Downlink (to the blade) VLAN ID might be 10 for both, in the middle inside VC they are treated as 2 different VLANs and traffic will never cross from VC1 to VC2 because we are only a L2 device.

A/A has a side effect that if Blade1, Port 1, VLAN 10 is trying to send a packet to Blade5, Port *2*, VLAN10, that packet will actually leave the Enclosure via the VC1 uplink, go across your network switch and then come back down VC2. Because again, VC thinks those are 2 different VLANs and crossing any VLAN boundry has to be done outside.

So you have to think about your traffic flows and optimize for your particular traffic with either A/A or A/P

Now with A/A and an OS with a decent load balancer built in (vSphere vSwitches for example), we can make good use of A/A because the OS will send approx half its VMs/packets through Port 1 which is VC1 and the other half through port 2 which is VC2. So both Uplinks get well utilized even though we don't have a proper Port Channel.

Also, in A/P mode, if you have an uneven amount of bandwdith available, like a 10Gb link on VC1 but only a 1Gb Link on VC2, VC will automatically pick VC1 as the Active path because it has more aggregate bandwdith. Then only if it fails would VC2 get used and bandwidth gets reduced.
Similar with LACP Port Channels.
If you have 4 uplinks, 2+2, then each side has 20Gb but only 1 side is active (A/P). If your Active side loses only 1 link and drops down to 10Gb of bandwidth, we will actually fail over to the other side where there is still 20Gb because it has more aggregate bandwidth.

Hope that helps explain it all.

And wow, I had no idea that Island was even there.
I was going to say if you were in the US somewhere I could find you someone like me on our Team who is closest to you and get you more in depth help to learn VC better. But my team only covers the US geo and then we help other countries as best we can over email and chat. I'm close to Los Angeles myself.