VC + ESXi with transparent Firewall = package storm?

[Patrick Neuner]

Hi there,

we are just in the process of migrating our stuff to a new C7000 + Virtual Connect (3.51).

What we try to do:

We got 2 uplinks (to the internet) from our DC.

UPL1 goes to VC1, UPL 2 goes to VC2. Booth assigned to vNet "upc_network", connection mode Failover. - this is the "outer" vNet

Also there is a second vNet "futureweb_network" - which should be behind the firewall server for our internal net

On Blade 1 we got a VMWARE ESXi 5 with an virtualized Fortigate Firewall VM in transparent mode.

That's how VC connects to this Server (I explain the deactivated Port 4 later):

Big Image: 1.jpg

UPC network connects to our first virtual ESXi switch:

Our Fortigate VM Firewall Port 1 connects to this switch and transparent filter/forward/scan all the traffic from this Port to Port 2 - which is connected to our vNet "futureweb_network"

All other Blades are connected to the vNet "futureweb_network"

I made a quick drawing of this Topo in MS Paint:

Big Image: vc_topo1.jpg

Everything works quite good - except when I activate PORT 4 for our Firwall Server. As soon as I activate the Port (within "futureweb_network" vNet the whole net goes crazy - loose all pings / everything is dead ... I guess it's causing a package storm. But I can't find WHERE we made the failure in this topo?!? Just want to bring every traffic from the outside through our Firewall into the inner net ...

I hope someone of you can help me on this! :)

Thank you, bye from Austria

Andreas

[Hongjun Ma]

when you assign futureweb vnet to Port 4, you potentially create this loop if it's broadcast/multicast traffic.

VC-1 ---> vswitch0---->Fortigate vm port 1---> Fortigate vm port 2--->vswitch1 vmnic3-->VC-2--->inernal stacking link between VC modules on port X7 and X8----> VC-1

[Patrick Neuner]

mhm - alright - I thought those VC vNet are completely seperated and won't exchange any packets?

But I always saw booth VC Controllers "as one"

What would you suggest to get this topo working? (with connection to booth VC for redundancy)

[Hongjun Ma]

actually what I said last night may not be right. It was very late into night and I didn't have enough time to think over.

I'd like to get more info in order for me to analyze

1) what's your centos blade server profile config? Do they only have one vnet future-web assigned?

2) I see there is a centos vm have two NICs, one to vswitch0 and one to vswitch1. what's the reason for this?

3) what's your vswitch0 and vswitch1 teaming method? are you using  vswitch default based on original port ID?

i won't have time to reply today and likely tomorrow and may have to wait to weekend for some time

[Patrick Neuner]

hehe - alright - no prob! ;-)

1) the centos blade (real server - no VM) is only connected to "futureweb" vNet. (But the storm also happens when this Server is not linked to "Futureweb" (not assigned anywhere) - so there is only the Firewall linked to the Futureweb vNet and nothing other present in the futureweb vNet.

2) centos VM - this was only a test - we replaced the Firewall with a Centos and made a simple bridge there ... tried if the Fortigate is the problem ... but storm also happened there ... this VM is normally DOWN

3) tried it with "original port id", "source mac" and also failover - happened with all three configurations

thx, bye from Austria

Andreas

[Hongjun Ma]

i haven't been able to see the loop as well. are you saying that for vswitch 1, even if you set NIC team and active/standby, you still have the same problem? if so, it's sort of weird. may want to open a support case to track down.

[Patrick Neuner]

yea - at booth vSwitch we tried all 4 settings for teaming ...
- port ID
- IP hash
- source MAC
- failover

also we had the second NIC in standby adapter und "not used" adapters ...

in all those constellations the broadcast storm happens.

[Patrick Neuner]

Just as a follow up, it seems to be an issue with the VMWare VSwitch and Promisious mode that can't as of now really be changed or won't or can't be corrected by VMware. I read at another forum post about a similar problem, that the Cisco VSwitch would be able to handle it, but we never tried that. Unfortunatly I don't have the link handy.

So no real possible solution with out of the box VMWare here when working with a VM Firewall and trying to have HA handled by VMware.  All works fine when using one in/one out LAN connection. 

[Hongjun Ma]

thanks for the update. Great to know the result of this behavior.