MS NLB & HP Flex10

[Reuv]

We have two NLB clusters, one in Windows 2003 and the other in Windows 2008. All the cluster nodes are VMs, running on 2008 R2 x64 Hyper-V. We do not use Teaming on the physical HP NICs.

We setup a VC Domain between two enclosures and moved one of the physical Hyper-V servers to the second enclosure. This meant, that we had 1 VM in NLB Cluster A running in Enc 1 and another VM in the same NLB Cluster running in another server on Enc 2. Then we started to experience similar results and loss of network connectivity as we used to when we first tried to use Teaming with NLB. Once we moved the physical servers into the same enclosure once again the NLB problem was solved.

Is this explainable?

[Reuv]

Oh, some environment information:

- Blade Servers are BL460 G6
- Enclosure C7000, Firmware 3.2
- Using Flex 10

[Antonio Milanese]

Hello,

could you add more details?

1) are you using unicast or multicast NLB (IGMP)
2) is IGMP snooping enabled on VC domain?
3) VLAN mapping or VLAN tunneling
4) uplinks / cross connect config
5) server mapping / nics config

Best regards,

Antonio

[Reuv]

1. Unicast

2. IGMP snooping is disabled

3. We have selected "Map VLAN Tags" but did not check the box to "Force server connections to use the same VLAN mappings as shared uplink sets".

4. Uplink / X-Connect Config: One 10 GB uplink to the backbone from each VC Flex-10 module

5. Each blade server has a single (and the same) Virtual Connect defined network connected to first NIC (without Network0).

[Antonio Milanese]

Hello Reuv,

>1. Unicast
Are u using Hyper-V Dynamic MAC + MAC Spoofing ?
Are u using VC Domain forged MACs or blade NICS burned-in MACs?
Double check if MAC spoofing is active on vSwitch and try using Static MAC.
VC Domain forged MACs could be problematic as TLB Teaming ones when dealing with Hyper-V/VMware NLB in unicast mode

>4. Uplink / X-Connect Config: One 10 GB uplink to the backbone from each VC Flex-10 module
do you mean that u have 4 SUS to the core switch (or swithes) and no enc uplinks ?
active/active or active/passive SUS?
A diagram could help =)

one note on upstream switches:
VC-FLEX modules technically are not switches and with unicast NLB they flood EACH active uplink port by default with NLB vMAC (as unknown/unregistered mcast by way with IGMP snooping) so upstream switch learn MACs on multiple ports and potentially could mess/block the correct forwarding path since ARP request could not be satisfied without "downgrading" to flooding mode!
In this case you should try mcast+IGMP NLB

>5. Each blade server has a single
Do you mean a single vNIC per server without dedicated NLB vNIC ? single nic unicast NLB is potentially "daungerous" (lot of caveats/registry messing to know of) even without Hyper-V in between!

Regards,

Antonio

[Reuv]

1. Yes, we are using both Hyper-V Dynamic MAC & MAC Spoofing is enabled.

2. We are using Virtual Connect assigned MAC Addresses.

3. When you write "Double check if MAC spoofing is active on vSwitch and try using Static MAC." - there is no option for "Spoofing" on the Hyper-V Network Manager. This option is only available on the Network Adapter within the VM Config.

I don't understand this statement: "VC Domain forged MACs could be problematic as TLB Teaming ones when dealing with Hyper-V/VMware NLB in unicast mode". We are trying in fact to use HP Physical TLB Teaming for a Hyper-V Virtual Network, and have the VM in turn use that adapter. If TLB and VC Domain MACs is a bad combo, what are our options:
a. Switch HP Teaming Mode
b. Switch NLB Mode to Multicast or IGMP Multicast (not sure I understand the difference between the latter two).
c. Using static built-in server MACs is not really an option for us, considering the amount of work it entails (and the fact that we aren't getting a raise any time soon!) :-)

4. We are NOT using SUS. I've attached a diagram since I have no idea what you want from me. :-( Let me know if you need anything else.

5. Back to MultiCast+IGMP NLB - how exactly is this going to improve the situation. Say, for simplicity, I have a single server two adapters that I team into one NIC. That NIC I use on my Hyper-V server. At the end of the day, all packets anyhow are going to need to travel through the two ports which the physical NICs are connected to (although in this case, we have a single Fiber Flex 10 physically connecting the Chasis to the Switch). Maybe I am missing something, or maybe I just need a link to explain what Multicast + IGMP does.

6. If I understood the last comment, our Hyper-V VMs have two NICs, which are essentially connected to the same physical adapter on the blade server, using the same Hyper-V Virtual network, but one vNIC is used for "Management" and the other is used for "NLB", which we have configured for Unicast Mode.

I hope this is clearer. :-)
Thanks!
Reuv

[Reuv]

Before I forget, while on the topic, this has been asked on the HP Forums a lot:

1. Is NLB supported (officially or unofficially) using HP Teaming, at all?

2. Are we wasting our time using software NLB and would an appliance make us sleep better at night? Have any recommendations?

[Antonio Milanese]

Hello Reuv,

well a long post to write tonight =)

>3. When you write "Double check if MAC spoofing is active on vSwitch and try using Static MAC."
yes is per machine setting but spoofed MAC still lives (its "learned") in a per vSwitch context and seams that can't really persist to cluster migration without reboot VM..ah and you need an hotfix if your child partition it's not R2 guest:

http://support.microsoft.com/kb/953828

>I don't understand this statement: "VC Domain forged MACs could be problematic as TLB Teaming
>ones when dealing with Hyper-V/VMware NLB in unicast mode". We are trying in fact to use HP Physical
>TLB Teaming for a Hyper-V Virtual Network, and have the VM in turn use that adapter. If TLB and VC
>Domain MACs is a bad combo, what are our options:
unicast mode relies on 2 facts:
a) your original nic MAC will be rewritten to a "common" (a bit simplified) unicast MAC
b) that MAC will be flooded by L2 switches so each member can handle the NLB cluster traffic

problems with a) :

since the NLB intermediate driver HAVE to rewrite MAC any other "down-stream" driver (i.e. HP TEAMING,hyper-v parent driver,ecc.) must MUST collaborate..
- with advanced teaming configs, TLB f.e., the HP TEAMING driver rewrites itself the nics MAC with a "generated" one so if the driver it's not NLB compliant/mac spoofing aware problem arise
- VC Domain MACs override burned-in MACs on per server profile (BAY/ENC) bay
this is the reason that sometime with teaming you should set LAA on nic driver and use a static MAC on vNIC

problem with b)
generated MACs are bogus one (masked by NLB) to disable L2 switches per port MAC learning (and problems with the same MAC on multiple ports cams) and force flooding but not every switch is happy with that and you must create static MACs or mirror port traffic ecc
and if you use VLAN to avoid flooding all switch ports you should check that you have another hotfix:

http://support.microsoft.com/kb/2286940

ah..and for w2k3 SP level+hotfix matters too

so when there are so many all this moving parts =) down to the wire unicast NLB clusters have hard time to (re)converge without recreate/re-adding members!

>b. Switch NLB Mode to Multicast or IGMP Multicast (not sure I understand the difference between the
>latter two).
multicast may be the solution or may be not..
mcast try to avoid unicast MAC oddities using a "proper" L2 mcast address but..yes but..there are problems dealing with switches since most L3 switches does not like multicast MAC with unicast IP and you have to create statics arp for NLB vip
Multicast/IGMP try to fix the flood of mcasts traffic only on ports that are members of a IGMP registered group..you should have IGMP support on L3 switches
We work mostrly with VMWare setups and mcast is the only reliable solution

>4. We are NOT using SUS. I've attached a diagram since I have no idea what you want from me. :-( Let me
>know if you need anything else.
well this look a strange design to me but this is not the root cause IMHO since VC modules are more hubs that switches =)

Back to your problem..from your setup i'll narrow the problem to switches/teaming and check:
1) wlbs query and wlbs display and WLBS ip2mac to see if it's converged or and registered MACs
2) i'll check whats MACs are touching the switches ports: which one?
3) if ARP requests are satisfied or lost in translation
4) disable teaming and check the above points

Is HP Teaming officially compatible with NLB?
AFAIK yes at least with physical servers and latest PSP but for hyper-v setups i don't know for sure (maybe NFT teaming)

is and HW loadbalancer advisable?
well if you ask me i'm not a big fan of microsoft NLB..yes it's cheap/free but too many oddities,hotfixes,reg hacks, since w2k..yes w2k8r2 it's a lot better..but now we have so much "virtualization" in between that if your SLA dictates and you want to "sleep better at night" =) a couple of relative cheap appliance like KEMP (but you could use F5,Citrix,ecc, or opensource solution) ones are a good investment!

Regards,

Antonio