PCM+ trafficd on 6200yl flattened entire campus

[André Beck]

Hi,

today I've dropped into a bad nightmare when a newly deployed core/distribution for an entire customer campus went sideways. It started with increasing round trip times of routed traffic and ended with a bunch of 6200yl boxes doing STP yoyo, inducing repeated broadcast storms every 10s or so until we shut down on of the two cores and rebooted one of the distris, then it reduced to a broadcast storm every 30s mostly limited to just one distribution island.

After double checking, I'm now sure the reason is no missing or badly configured STP, broken link or such but is actually the PCM+ Traffic monitoring! As soon as I add the two core and six distribution switches to traffic monitor, the boxes become irresponsive first (high RTTs and packet loss) and then seem to completely break apart, as if SNMP or sFlow overrun their CPUs so they could not do such irrelevant things as STP. Even removing them from trafficd instantly won't help, they go on STP-yoyoing for ever until you either isolate them from redundant paths or just reboot them.

Searching this forum I found

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1039021

which seems to talk of exactly the same issue, just not as bad as it came here when that customers new and supposedly better network wreaked more havoc than we've seen when a Cisco 4908G-L3 went bad there in 2001 which led them to dropping Cisco and replacing the network with ProCurves (3400cl-routed). The yl are running K.11.41 and PCM+ is at the latest patch level. The show cpu statement doesn't make a CPU overload obvious, though.

Consider this a WARNING: Do NOT fire PCM+ trafficd onto any K-Series platform if you like your network up and running. Hopefully HP will fix this issue soon, I don't like the idea of RMAing all these boxes and wiring this customers campus back to how it was...

*Sigh*,
Andre.