Broadcast storm between family 5400R zl2

[CKPL-KRA]

Hello,

I'm facing some problems with broadcast/multicast storms. I would be grateful if anybody could help me with any suggestion.

Here's a littte description of our infrastructure:
Our network backbone consists of 4 switches from family 5400R zl2. One of them acts as our CORE switch, the rest 3 are connected directly to CORE with two 10Gbps fiber links each (for redundancy) - classic tree topology. These links are configured with MACsec. There are not configured trunks with LACP on these links, so every second link is blocked by STP - that's expected behaviour. Our STP works in MSTI mode. All of 5400Rs (including CORE) have connected at least few access switches (HP 2530). These access switches also are connected via 2 links, 1G each, but configured with LACP. All of switches have the same configuration, with CORE forced to bo root (0 priority - CORE, 2 priority - rest 5400R, default priority - access switches) All of access switches have lastest firmware or one version back. All of 5400R have lastest firmware excluding CORE (it has KB_16_06_0012).

Problem:
Broadcast/multicast storms occurs after (I assume) fail one of interfece on CORE switch, in terms of no bpdu frames, with triggers STP to unblock interface - creating a loop and also triggering MSTI to rebuild/reconfigure the network for STP. From this moment storm occurs and spans across all switches leading to complete network paralyze, excluding one access switch - strange. That one is connected to CORE with two 1Gbps fiber links (trunk LACP) and does not notice any broadcasts storm. Other switches, connected the same way (by fiber or twisted pair) suffer from storms. There aren't any logs that infom about CORE's interface failure. Problem occurs randomly in time and randomly form any 5400R switch. Here are some logs below:

ip address 5400R nr1>	00842 stp:  AM1: CIST starved for a BPDU Rx on port A21 from <CORE's mac addr>
<ip address 5400R nr1>	00842 stp:  AM1: MSTI 1 starved for an MSTI Msg Rx on port A21 from <CORE's mac addr>
<ip address 5400R nr1>	00002 vlan:  AM1: <vlan name> virtual LAN disabled
<ip address 5400R nr1>	00001 vlan:  AM1: <vlan name> virtual LAN enabled
<ip address 5400R nr1>	00435 ports:  AM1: port A21 is Blocked by STP
<ip address 5400R nr1>	00076 ports:  AM1: port B21 is now on-line
<ip address 5400R nr1>	00076 ports:  AM1: port A21 is now on-line
<ip address 5400R nr1>	00328 FFI:  AM1: port A21-Excessive jabbering/fragments. See Help
<ip address 5400R CORE>	00327 FFI:  AM2: port A23-Excessive undersized/giant packets. See help.
<ip address 5400R nr2>	00332 FFI:  AM1: port B21-Excessive Broadcasts. See help.
<ip address 5400R nr3>	00332 FFI:  AM1: port B21-Excessive Broadcasts. See help.
<ip address 5400R nr2>	00332 FFI:  AM1: port A21-Excessive Broadcasts. See help.

Questions:
1. I bet that reconfiguring links between 5400R to LACP mode could resolve problem, but we need MACsec applied to those links. As far I know, LACP with MACsec couldn't be applied together, but maybe there is any walkaround?
2. Why suddenly CORE switch fails to send BPDUs, which leads to STP failure? I would be very grateful for any help and all suggestions.

Best Regards
Kamil Radykowski

[parnassus]

Hi! can you post how the STP (MSTP) is commonly configured? Access switches' edge ports are properly configured (bpdu-protection, loop-protect, admin-edge mode and point-to-point-mac false)? single link uplinks are properly configured (links aggregated links should be OK once well formed)? root protection is configured on Core (STP Root Priority = 0) ports used for downlinking to the other switches?

This article could shed some lights about STP Troubleshooting.

[CKPL-KRA]

Hi,

We use MSTP but only with single instance right now.

Part of configuration:
spanning-tree spanning-tree Trk[1-x] priority 4
spanning-tree config-name "all-vlans"
spanning-tree config-revision 1
spanning-tree instance 1 vlan 2-999
spanning-tree instance 1 priority 0
spanning-tree instance 1 Trk[1-x] priority 4
spanning-tree priority 0

Access switches:
bpdu-protection - disabled
loop-protect - disabled
admin-edge -> auto (default value)
point-to-point-mac -> true (default value)

LACP configuration: trunk A1,B1 trk1 lacp

ASCII art of switch connection:
Access Switch ===2x1G (LACP)=== Distribution Switch ===2x10G (MACSEC) === Core Switch

Regards,
Kamil Radykowski

[CKPL-KRA]

Hello,

Do you have any idea what could cause our problem @parnassus ? Or anyone else?

In the meanwhile, I've implemented some change to our configuration: bpdu-protection (on access switches), loop-protect on all 5400 switches, and root-guard on CORE switch.

Regards,
Kamil Radykowski