Loop detected with Flex-10 and NX500

[ssAPT]

Apologies if theis has been asked and answered already. I could not find anything related.

We have a multi-enclosure Flex-10 environment with ver 3.51 firmware. A 2 port LACP channel is created to a single Nexus 5000 switch from a VC on each enclosure. Enc 1 / VC 1 = Po1 & Enc 2 / VC 1 = Po2

Whenever we remove one cable from a channel and re-insert it the Nexus logs a spanning tree loop detection between port channels and disables mac learning for 3 minutes. During this time the servers are unreachable. After 3 minutes everything goes to normal , the re-inserted port joins the channel and normal operation resumes.

On the VC we have an ethernet network created with the 2 ports assigned. The ENET is in tunnelled mode. On the Nexus the config is as follows:

interface port-channel2

  description ENC2

  switchport mode trunk

  switchport trunk allowed vlan 3,5,8

  spanning-tree port type edge trunk

  interface Ethernet1/5

  description ENC2-UPLINK1

  lacp rate fast

  switchport mode trunk

  switchport trunk allowed vlan 3,5,8

  channel-group 2 mode active

 interface Ethernet1/6

  description ENC2-UPLINK1

  lacp rate fast

  switchport mode trunk

  switchport trunk allowed vlan 3,5,8

  channel-group 2 mode active

Config for PO1 is the same

The nexus Logs:

2012 Apr  3 16:06:23 SANPSFNB1SWI03 %FWM-2-STM_LOOP_DETECT: Loops detected in the network among ports Po2 and Po1 vlan 5 - Disabling dynamic learn notificationsfor 180 seconds 

2012 Apr  3 16:06:26 SANPSFNB1SWI03 %ETHPORT-5-SPEED: Interface Ethernet1/6, operational speed changed to 10 Gbps

2012 Apr  3 16:06:26 SANPSFNB1SWI03 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/6, operational duplex mode changed to Full

2012 Apr  3 16:06:26 SANPSFNB1SWI03 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/6, operational Receive Flow Control state changed to off

2012 Apr  3 16:06:26 SANPSFNB1SWI03 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/6, operational Transmit Flow Control state changed to off

2012 Apr  3 16:06:29 SANPSFNB1SWI03 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel2: Ethernet1/6 is up

2012 Apr  3 16:06:29 SANPSFNB1SWI03 %ETHPORT-5-IF_UP: Interface Ethernet1/6 is up in mode trunk

2012 Apr  3 16:09:23 SANPSFNB1SWI03 %FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all interfaces

Appreciate any help

Regards

[Hongjun Ma]

could you clarify if your enclosures are stacked together?

if not, then what's on ENC1/VC2? you only have one N5K?

I'm still trying to understand how your VC are connected with N5K.

[ssAPT]

Hi Thanks for the reply. the enclosures are stacked and uplinked as follows:

Enclosure 1 - VC 1 - uplinked to NX5k 1

                      VC 2 - NOT uplinked to anything

                      VC 1 and VC 2 crosslinked via ports X7 and X8

Enclosure 2 - VC 1 - NOT uplinked to anything

                       VC 2 - Uplink to NX5k 2

                       VC 1 and VC 2 crosslinked via ports X7 and X8

Enclosures are stacked together via ports X1 and each enclosure connects to a different NX5K

The solution was designed this way to maximise failover using minimal available ports

[Hongjun Ma]

this problem should not happen and your stacking topology is correct.

1) On VC side, do you have 2 tunnel vnets covering 2 uplink bundles or 1 tunnel vnet covering all links? Also, under vnet definitiion, is the connection mode set to auto?

2) what's your VC and nexus version?

this message shows nexus sees some sort of mac flapping. you can use show mac-address dynamic or track down which mac was giving this problem. Also, on each nexus, do you have both po1 and po2 connecting to VC? from the way you described, it looks like you should just have 1 port channel to VC on each N5K.

My suggestion is following

1) remove lacp rate fast from nexus side. This command was for a bug on nexus side when negotiating port-channel with VC but the bug was fixed later so as long as your nexus version is not very old, very likely you don't need this command. please do this at your mantainence window.

2) If possible, try upgrade VC to latest code 3.51 to see. If it's not convenient, then please open support case on Cisco and HP side to do further troubleshooting.