Looping packets in 6200yl and/or 2900?

[Peter Eriksson Linköpin]

I'm trying to diagnose a strange network problem where it looks like the a 6200yl is looping some kind of strange ethernet frames.

Setup:

Switch A - HP 6200yl
Switch B - HP 6200yl
Switch C - HP 2900
Switch X - HP 9304 ("Foundry")

Connection: X - A - C - B
(Other swiches hang on A/C/B, X is an external switch not under my control).

Switch X generates an ethernet packet with type "885A" (which as far as I can Google is some kind of "ServerIron Keepalive Packet") on a couple of the VLANs that we receive from them:

Somehow the little A/C/B trio seems to mishandle this in a big way because after a little while a huge number of "885A" packets are being broadcasted around our network and bascially killing things in a big way.

I've "solved" it right now with a strategic:

lockout-mac 00e052-cd8300

in switch A so this packet never enters our
network...

I don't like it though - this is not how things should behave... anyone else seen anything like it? (We have the latest firmware in all the involved switches).

This is how all the packets are looking:

ETHER: ----- Ether Header -----
ETHER:
ETHER: Packet 1 arrived at 15:34:2.19747
ETHER: Packet size = 278 bytes
ETHER: Destination = 0:e0:52:0:0:0,
ETHER: Source = 0:e0:52:cd:83:0,
ETHER: Ethertype = 885A (Unknown)
ETHER:


0: 00e0 5200 0000 00e0 52cd 8300 885a 0101 ..R.....R....Z..
16: 0116 0100 5800 7072 2052 ade8 0000 0080 ....X.pr R......
32: 0000 000a 0000 0000 2052 ae00 201f 78f0 ........ R.. .x.
48: 2052 ae08 201f 78f0 0000 0000 0000 0000 R.. .x.........
64: 2052 ae18 201f 8bec 2052 ae18 201f 7e3c R.. .ì R.. .~<
80: 0000 0000 0180 0000 2052 ae38 201f 8eb4 ........ R.8 ...
96: 2052 afa8 0000 3a92 cee4 1420 0000 0005 R.¨..:.... ....
112: 0000 17c2 0002 2d84 2052 ae68 201f bfe4 ......-. R.h ...
128: 0000 0000 0000 0000 0000 3a92 0000 0000 ..........:.....
144: 2052 ae60 0000 0000 0000 0018 0000 0001 R.`............
160: 2052 ae78 201f 78f0 2052 aec8 201f c694 R.x .x. R.. ...
176: 0000 0001 0000 0000 2052 ae90 201f 7e3c ........ R.. .~<
192: 0000 ae90 0000 0004 0000 0001 0190 0000 ................
208: 2052 aec8 201f 7f24 2052 aef0 0000 007f R.. ..$ R......
224: 2052 aeb8 201f 78f0 2052 aed0 201f 78f0 R.. .x. R.. .x.
240: 2052 aed0 0000 0000 2052 aed0 20e9 504d R...... R.. .PM
256: 2052 aef0 0007 a120 0028 cfcb 20e8 d352 R....¡ .(.. ..R
272: 0000 99f4 2052 .... R

[Mohieddin Kharnoub]

Hi

Apparently you have ServerIron switches in the 9300 Side of the network.

And this MAC is used for heart beat between dual Serveriron load balancers.

I think you did a trick on the solution but what i suggest is to keep any L4-L7 active devices like: Load balancers, Packet Shapers, Cache Boxes .... in a different Vlan so you at least can get rid of any broadcast they do.

Good Luck !!!

[Peter Eriksson Linköpin]

We've done some more testing now and it's the 6200yl switches that are misbehaving. I can now generate this strange type of broadcast storms at will using the free "nemesis" packet injection tool. Some testing reveal some interresting artifacts:

Ethernet type _must_ be 885A
Ethernet destination must be 00:e0:52:00:00:00

If we use any other ethernet frame type then no storm happens. If we use any other destination then no storm happens.

Frame payload makes no difference.

[Peter Eriksson Linköpin]

Ok... Now after some even more network debugging we've narrowed it down to what looks like a problem with the 6200yl switches.

To cause the storm you need:

a) Two 6200yl switches in your network (doesn't have to be connected directly, they just need to see the same VLAN somehow).

b) Both of the 6200yl switches need to have a trunk to the common network (can be either "trunk" or "lacp" - doesn't matter).

When one of the 6200yl switches receive one of the troublesome packets via the trunk then it will send it back out again over the same link that it received it from. And then the other 6200yl switch will see it and send it back again. Repeat forever... And they are pretty quick, and with a couple of more injected packets after a while maaaany packets will be bouncing back and forth...


Note also that using a "lockout-mac" on the 6200yl switches does NOT stop this from happening! It will stop it from being sent *thru* the switch though. But the packets will still be mirrored back out the same trunked port it was received from...


Our 6200yl switches are running firmware K.12.57 btw.

[Matt Hobbs]

Peter, sounds like you've got this reproduced quite nicely. You should open up a case with support to get a fix for it.