Virtual Connect tunnel mode working with load balancers

[chuckk281]

Nelson was working with a customer regarding Virtual Connect and the various modes of operation:

***********

Hi Experts,

A customer is planning to implement Virtual Connect FlexFabric 20/40 F8 and Tunnel Mode.  They found some Knowledge Brief (KB) mentioned that there is a potential issue to use with a Load Balancer.   If the load balancer enables gratuitous ARP, then Tunnel mode must be disabled in Virtual Connect.  Is the problem still exist on the latest VC firmware? 

My customer cannot disable gratuitous ARP in Load Balancer, but they want to use Tunnel mode in VC.  Is there any solution for them?

The issues found on the web:

C7000 FlexFabric dropping pings intermittently to only some VM guests

Link:

https://communities.vmware.com/message/2412805

http://h20565.www2.hp.com/hpsc/doc/public/display?calledBy=&ac.admitted=1410467226755.876444892.492883150&docId=emr_na-c02684783-2&docLocale=

Explanation:

When VC is in VLAN tunnel mode, it maintains a single MAC Address table for the tunneled VC network even though it encompasses multiple VLANs (this is normal). The result is that when a host (physical or VM) inside the VC Domain sends a broadcast like an ARP, it is sent out the VC uplink on one VLAN, traverses through the load balancer and is broadcast on the load-balanced VLAN. If that VLAN is also sent to the VC uplink port, the MAC address of the host is learned outside of VC. Like any 802.1d bridge, subsequent traffic sent to that host's MAC address and received on the VC uplink is discarded as VC has learned that the MAC address resides outside the domain. The normal MAC address aging time in VC and most other switches is 5 minutes, so this condition will exist until the entry aged out.

Suggested Workaround from the blog/KB:

The only way around this is to use Shared Uplink Sets or if possible disable gratuitous ARP on the load balancers.

*************

A great and loooonnnggg explanation from Herman:

***********

Let me add some explanation, hope it helps…

Gratuitous ARP (request or reply) or Load Balancers themselves are not the issue.  Really any flooded traffic (multicast/broadcast/unknown unicast) can cause connectivity issues with a tunnel mode setup if VLANs are being bridged.  Disabling Gratuitous ARP will not clear up this problem.  The problem VC Tunnel mode has is when 2 or more VLANs are bridged together (i.e. 2 VLANs in one broadcast domain) AND those 2 or more VLANs are included on the same Tunnel Mode network.

If you have a tunnel mode network in VC, you have a single MAC table for that tunnel network regardless of how many VLANs are part of that tunnel.  Let’s make up a scenario, maybe it will help.  Here is a MAC table:

MAC Address                                    Learned on Port               Learned on Network

Blade-1-VLAN10                               d1                                           tunnel

Blade-2-VLAN20                               d2                                           tunnel

outsideHost-VLAN10                      X1                                           tunnel

outsideHost-VLAN20                      X1                                           tunnel

Blade-1 above is the MAC address of a blade inside the enclosure and it has been properly learned on the VC downlink port on the “tunnel” network.  Let’s say that Blade-1 is sending its frames on VLAN10 but because VC tunnels the traffic, VC does not learn VLAN 10, it learns the traffic on the tunnel network.  Blade-2 is sending traffic on VLAN 20 but it is also learned on the tunnel network.  Same thing for the outsideHosts.

Let’s say on the upstream switch connected to our X1 port we have 2 VLANs configured, VLAN 10 and VLAN 20.

Now, if somewhere out on the network VLAN 10 and VLAN 20 are bridged together, we have a problem.  When Blade-1 sends out a broadcast/multicast/unknown unicast, the frame exits the VC uplink on VLAN10, gets bridged over to VLAN 20 out on the network and gets sent back in to the VC uplink on VLAN 20.  Because VC is in tunnel mode, the frame that originated from Blade-1 that has now been sent back to the VC uplink is learned on the tunnel network.  Now our MAC table looks like this:

MAC Address                                    Learned on Port               Learned on Network

Blade-1-VLAN10                               X1                                           tunnel

Blade-2-VLAN20                               d2                                           tunnel

outsideHost-VLAN10                      X1                                           tunnel

outsideHost-VLAN20                      X1                                           tunnel

Blade-1 MAC has now been learned on the uplink port.  This means that any subsequent traffic that is destined for Blade-1 that gets sent to the VC uplink port for delivery will be bridge-filtered or dropped (that’s what bridges do).  That will continue until Blade-1 sends some unicast traffic out which will cause Blade-1 to be relearned on d1.  Hence, you have intermittent connectivity.

So it is not a load-balancer issue specifically.  It is a VLAN bridging issue in combination with VC tunnel mode’s use of a single MAC table.  Some Load Balancers (in my experience, F5) don’t use any type of VLAN bridging and they have no problems working with VC tunnel mode.  Some Load Balancers (in my experience, Cisco) use VLAN bridging which would cause problems in a VC tunnel mode environment.  Other entities like Cisco RSPAN or even a misconfiguration can produce the same type of MAC learning effect.  The problems only happen if 2 or more of the bridged VLANs are configured on the upstream switch where the VC uplink connects.  This scenario can be avoided by architecting the network to prevent configuring both bridged VLANs on the switch port connecting to the VC uplink… or use mapped mode in VC.

*********

Reply from Nelson:

Hi Herman, 

Thanks for the very detailed explanation.   It is very useful for me to explain to the customer. 

I am not quite familiar with load balancer.  For Cisco load balancer, can it work without enabling VLAN bridging?   Or it is a basic feature that can’t be disabled?

From Herman:

The Cisco Load Balancer I had experience with was a Content Service Switch (CSS).  The CSS required VLAN bridging.  I’m not sure how some of their other models/offerings might work.

And from Nelson: 

Thanks.  I will check if the customer’s load balancer requires VLAN bridging or not. 

On the other hand, the customer has another interesting question on MAC address.  In their existing environment, the same MAC address will exist on two servers in different VLANs.   E.g. In both VLAN 101 and 102, there are two servers with the same MAC Address; one server in VLAN 101, one server in VLAN 102. 

These special servers are outside hosts of Virtual Connect, so VC will learn their MAC Address in uplink port.  Will it cause any problem to VC Tunnel Mode or Mapped Mode? 

Herman:

No, that won’t cause a problem.

From Nelson:

Thanks!  Can I explain to the customer like these:

If they deploy VC Tunnel mode, we have a single MAC table for multiple tunnel networks.  The two outside hosts (with same MAC address but in different VLANs) will be treated as a single entry in the MAC table.  As both outside hosts are seen on the same uplink ports of VC.   Single MAC address entry will not cause problem. 

If they deploy VC Mapped mode, we have independent MAC tables for each VLAN.  So even if same MAC address exists in two MAC tables (different VLANs), it will not cause problem. 

Is it correct? 

And from Herman:

Yes, that is a great summary.

**************

Any other comments?