spanning tree recalculating

[stieven struyf]

We have (rapid) spanning tree running on a part of our network. Every few hours it recalculates.
Is this normal?
The resulting spanning tree topo is always the same, in the logs i can't find any reason(like plugged/unplugged ports).
I already upgraded to the latest firmware on all switches.
This network part has 2 3400cl's as core and multiple 2848 at the edge.

[Mohieddin Kharnoub]

Hi

Topology changes make RSTP algorithm to restart again.

Usually in RSTP you can use many features to avoid that like assign edge ports where its connected and its by default OFF.

Stop BPDU in some places on the network.

If that was not helpful, then i would ask you to attach the config of the root at least.

Good Luck !!!

[stieven struyf]

all ports are set to edge ports, only connections between switches(gig. ethernet) have spanning tree enabled.
Problem also exists when i remove all redundant links,but keep rstp running.
We already had a consultant looking at it without result.
I also don't see any problem on the uplinks when spanning tree recalculates.

[Mohieddin Kharnoub]

Hi

Usually Spanning-Tree won't recalculate unless you have some topology changes happened

Like if one of your blocked links between 2800 and the core changed its path cost (or speed), that will affect.

I would like to ask you if you can attach the 3400 or the Root show tech, and little explain about your network map.

Good Luck !!!

[Matt Hobbs]

If it's only every few hours I wouldn't lose much sleep over it. If you want to try and track it down though, I would configure syslogging on each switch and see if you can correlate the last topology change to any events that happened at the time.

Otherwise in the 3400 release notes it provides some good information on how to read the output of 'show span detail'

You may also want to look into the new BPDU filtering and detection options available.

[stieven struyf]

i didn't knew the command. Is there a full explanation what every parameter means(and how i can use this to track the problem).
It happens to often to just leave it.
i attached the show tech of the root switch.
I will add a map of the switches in the next reply.

[stieven struyf]

network map

[stieven struyf]

I found some increasing parameters(cfg bpdu) on a port, i am quite sure that this is a hub. Is it possible that a hub causes this behaviour?

I also can't filter bpdu's, i have done this in the past. Does it only works for some stp versions?
On this port i also see that operedgeport is no, while i forced this port to edge.

BESW56KE(config)# show spanning-tree 36 detail

Status and Counters - RSTP Port(s) Detailed Information

Port : 36
Status : Up
BPDU Filtering : No
Role : Designated
State : Forwarding
Priority : 128
Path Cost : 200000
Root Path Cost : 4
Root Bridge ID : 16384:001279-034c00
Designated Bridge ID : 40960:001560-002680
Designated Port ID : 128:36
AdminEdgePort : Yes
OperEdgePort : No
AdminPointToPointMAC : Force-False
OperPointToPointMAC : No
Aged BPDUs Count : 0
Loop-back BPDUs Count : 0
TC Detected : 6
TC Flag Transmitted : 0 TC ACK Flag Transmitted : 22
TC Flag Received : 0 TC ACK Flag Received : 0

RSTP RSTP CFG CFG TCN TCN
BPDUs Tx BPDUs Rx BPDUs Tx BPDUs Rx BPDUs Tx BPDUs Rx
---------- ---------- ---------- ---------- ---------- ----------
868 0 83418 9 0 19

BESW56KE(config)#

[Matt Hobbs]

With the show span detail you really need to capture it off each switch before and after a topology change, and then study the differences. The ones that are most helpful are TC Flag Received, and TC Detected. From the root keep following the port that increases on TC Flag Received until you get to a switch that has a TC Detected.

In your config I noticed that you had single port trunks configured which is not necessary. It shouldn't cause a problem though.

[stieven struyf]

thanks for the tip, but i already did that.
now i have very strange errors.
first i found a port with a lot of tc detected. i disabled it and another port on another switch had the same issue.
this morning i checked again, again another switch and another port.
this are always ports from a switch directly to one of the cores.
This morning it was happening on one of the cores towards a user switch.
what i find strange is that only one side of the link sees a topology change. If there was really a problem i would think both sides would notice.
btw. all switches are connected through cat5e cables on gigabit. Can cable lenghts cause such a problem?

[Jason Scott]

This is an interesting thread. We've suffered with RSTP reconvergances over the last year. At one point they got so bad that it was causing actual application and server problems with noticable impact.

Now we're down to reconvergance every day or so with no real explanation. We have a variety of 3400s, 2800s, 2650s, 5300s and 9300/9400 switches.

[stieven struyf]

it stopped!
it is now stable(last recalculation 4 days ago).
I'm going to replace the links that give problems by fibres(are also installed, just need the gbics).

The latest link that gave problems is still connected though(don't know why it doesn't give problems anymore).

[JHz]

Hi,

I'm also having trouble with RSTP reconvergence every 1-2 seconds!
We have about 230 ProCurve 2626 switches, all with the "same" RSTP-configuration and latest released firmware (H.08.106).
User-ports are set to bpdu-filter and edge-port.
All the ProCurve 2626-switches are connected in rings with about 5-6 switches in each ring. A couple of rings for each distribution-switch.

Topology Change Count : 1,968,976
Time Since Last Change : 1 secs

Our core and dist-routerswitch are using Rapid-PVST (Cisco) and convergence is fine there, no recalcuting every second.

Some Vlans are trunked in all switches, dists and core (management vlan 1 for instance).

I don't know where the problems originate from, I cant find any flapping links etc.

I need help to debug this problem.
Any ideas?

[Mohieddin Kharnoub]

Hi

I recommend you to have a fast look on your topology, where is the place that may affect.

Then enable logging and debug on your switch, and forward it to a Syslog server and keep this running to monitor any strange behavior.

Share us whats happening with you.

Good Luck !!!

[stieven struyf]

jhz,
i found the problem using the
'show span detail' command. This should give you detail about which port flaps the most. If it is a duplicate link that has a high number of topology changes, try removing/replacing it.

[JHz]

We are already syslogging to a syslog-ng.

This is how one "span tree detail" look like.
ALOT of BPDUs are sent/received on port 25 and 26 (uplinks). Strange...


Port : 25
Status : Up
BPDU Filtering : No
Role : Root
State : Forwarding
Priority : 128
Path Cost : 20000
Root Path Cost : 20016
Root Bridge ID : 1:00049b-f76800
Designated Bridge ID : 32768:001560-dbac80
Designated Port ID : 128:26
AdminEdgePort : No
OperEdgePort : No
AdminPointToPointMAC : Force-True
OperPointToPointMAC : Yes
Aged BPDUs Count : 0
Loop-back BPDUs Count : 0
TC Detected : 8
TC Flag Transmitted : 0 TC ACK Flag Transmitted : 0
TC Flag Received : 0 TC ACK Flag Received : 0

RSTP RSTP CFG CFG TCN TCN
BPDUs Tx BPDUs Rx BPDUs Tx BPDUs Rx BPDUs Tx BPDUs Rx
---------- ---------- ---------- ---------- ---------- ----------
144 1986929 0 0 0 0

Port : 26
Status : Up
BPDU Filtering : No
Role : Designated
State : Forwarding
Priority : 128
Path Cost : 20000
Root Path Cost : 40016
Root Bridge ID : 1:00049b-f76800
Designated Bridge ID : 32768:001560-dbd980
Designated Port ID : 128:26
AdminEdgePort : No
OperEdgePort : No
AdminPointToPointMAC : Force-True
OperPointToPointMAC : Yes
Aged BPDUs Count : 0
Loop-back BPDUs Count : 0
TC Detected : 11
TC Flag Transmitted : 0 TC ACK Flag Transmitted : 0
TC Flag Received : 0 TC ACK Flag Received : 0

RSTP RSTP CFG CFG TCN TCN
BPDUs Tx BPDUs Rx BPDUs Tx BPDUs Rx BPDUs Tx BPDUs Rx
---------- ---------- ---------- ---------- ---------- ----------
1986936 34 0 0 0 0

[stieven struyf]

I looked to the port with the highest "TC"
i see that for you it's port 26. Try disconnecting this(if possible) and check for improvement.
I had to do this 3 times to remove all links that gave problems. I still have to replace them btw.

i don't know weather this is the best approach, but it worked for me.

[JHz]

I tried that before, and I tried again now (disable the port).
It still calculates every second.
I wonder why the TC-value so small when "Topology Change Count" is 1,986,777..

Shouldnt "TC detected" on all interfaces be the same thing as "Topology Change Count"?

[stieven struyf]

i had the same.
This is my explanation:
One tc triggers a spanning tree recalculation.
In a complex/large environment it can take a while to stabilize, during which topology changes happen until it is stablized.
I probably don't explain this as it should, but it is more clear with a graph.

[JHz]

I still don't get it. BPDU-filters and edge-ports are enabled on all end-users ports.
Recalculating every second can't be normal.
I will try to debug the BPDU frames and if I can see what device on the network that is forcing this recalculation, but it seems that you cant do "debug spanning-tree" on the HP2626...
I guess i have to connect a pc with ethereal/wireshark or something?

any other ideas how to find the cause?

[mikeynorrish]

I had a similar problem with 5300 and 2650 switches. All spanning tree ports were set as edge and bpdu-filtered.

The problem seemed to be that there were a few ports which didn't have the speed-duplex forced. Also, by turning on debugging (debug dest sess, debug events) I found a few interfaces which weren't stable - one involved swapping a mini-GBIC and the other, which was a gig copper link, just had to be shutdown at both ends. The switches will have to be swapped out for this one. It took me the best part of a day to trace the problems but I think that's done it for now . .