Switches apparently unreachable on a VLAN according to PCM

[EBS-tech]

We're an all-ProCurve shop. Our core is an 8212zl. That's linked to a number of comms cabinets. A cabinet typically contains two 2910 48-port PoE switches and a number of 48-port 2510s. The 2910s each have a fiber span back to the core; each 2510 is connected to both 2910s in the cabinet. The 2910s can be considered to be edge switches. The 2910s have end devices on them (where PoE is needed) and also carry traffic between the 2510s in their cabinet and the core.

We have updated the firmware of every one of our switches recently; we have also replaced several switches. There have been a few network performance problems. I have identified a few misconfigurations after our changes and have corrcetd them, but have discovered something odd in the process.

We have almost 20 VLANs, and the links between the 2510s and their colocated 2910 have (at a first approximation) all VLANs tagged. Other ports are untagged according to what's plugged in to them. One VLAN is used to carry WiFi traffic between the access points and the controllers; WiFi performance is appalling at the moment. This VLAN also happens to be the Primary VLAN.

The PCM network map of the VLAN in question shows the tolopogy as it should be, with tagged links between core and 2910s, and between 2910s and 2510s, but one cabinet, which happens to be where the WiFi problem is worst, shows every single link between 2910 and 2510 to be STP Blocked. Spanning Tree is on at all switches.

How is this possible?

Oh, one other thing. I noticed that the primary VLAN was misconfigured in some switches, and I've now corrected that. It's possible that the management VLAN still has misconfigurations - would this matter (it's the Default VLAN and carries no user traffic)?

[Vince-Whirlwind]

A few things.

1. Creating a management VLAN and putting all my switches into it is one of the things I like to do because it involves only devices under my control so I can do this without anybody else putting the brakes on it. The end result is a network management VLAN that has ONLY my stuff in it. I like!

2. HP very often call Access switches "Edge switches". Let's call them "Access switches", which is the correct terminology.

3. Every Access port should have this on it:
spanning-tree bpdu-protection    (protects you from rogue switches)
spanning-tree admin-edge-port  (saves the host from a 3-second delay when the port comes up)

4. Every Access switch should have this on it:

loop-protect 1-48      (protects you from loops created in any unmanaged remote devices that aren't running spanning-tree)

*** Only apply to Access ports, of course.

5. Every 2910 should have this configured on it:
spanning-tree <switchportID> rootguard

The <switchportID> should be every port that has a 2510 downlinked.

NOT on the uplionks to the core.

6. You need to configure your spanning-tree priorities:

8212 : 0 (0) or 1 (4096)

2910: 6 (24576)

2510: make sure these are all on the default: 8 (32768)

OK, so now things are nice and tidy, what isn't tidy?

Log onto each switch and do a "show lldp neighb". Use this information to sketch a reality-based topology map.

This is a great way to challenge your assumptions and eliminate a few weird surprises.

The next most important thing is to is to gather performance stats on all your switchports: I don't knopw if PCM does this? If not, get hold of some software that does, eg, Solarwinds Engineer's toolkit. (I got it over 10 years ago and I still use it. It's perfect).

What you want is to set the interval to the minimum possible period (longer periods obscure peaks therough averaging).

Gathering stats on your network for one hour could be enough to pinpoint all the bottlenecks and performance.

Gathering stats for a longer period will pick up on sporadic issues.

FInally, do you have a syslong server? Getting all your switchlogs in one place is a great help. I assume PCM does that at least - make sure you use it. switch logs will tell you lots of stuff, both individual errors as well as patterns in more generic log messages.

[TerjeAFK]

Great advice from Vince. Also you should logon to the switches where PCM shows STP blocking and determine which ports are blocked and why:

show spanning-tree inconsistent-ports