HP4000M flooding unicast traffic

[Romeo Chance]

I have been battling an ongoing intermittant
problem with the HP4000M switches where every few minutes we get ALL unicast traffic flooded on every active port.

We have the latest version of the firmware
installed and the problem still persists.

I noticed that there is a similar problem
with the newer switches as well.

basically the switch starts acting like
a hub every few minutes, i see all traffic
passing just as if the switch is a hub.

Monitoring Port is DISABLED, i installed
commview to see the traffic when i noticed
the flood of traffic on all ports.

[Ron Kinner]

Sounds like for some reason the Forwarding Database has gotten corrupted or erased since this behavior is normal for an empty Forwarding Database. Once a MAC is heard from it should be put into the FD along with its port number and then all unicast traffic to that MAC should be limited to its port number. Is there any sign that the swx is rebooting? Maybe in Show Version or Show Boot-history or show logging?

Perhaps a loop is driving the switch crazy? Do you have STP working on all ports to prevent loops?

Ron

[Romeo Chance]

Ron,

Thanks for the reply,

1. No Loops - spanning tree is disabled
2. Only 4 devices on that vlan, all routers
3. Flood is not multicast, it is unicast
traffic (tcp/udp/icmp) between the routers
4. Switch does not reboot (up 32 days)
5. Flood comes every 3 or 4 minutes and
goes away after about 1-2 minutes

It is very strange, the MAC Age Period is
set to 5 Minutes on the switch (default).

When the flood appears, I cannot see anything
weird on the sniffer, all packets are correct
without any checksum errors and all packets
have valid source/dest mac-addresses.

thanks
Romeo

[Ron Kinner]

If it's only a few devices why not try converting them to static MACs. That might stablize the FDB.

If you don't want to do that you might try extending the age timeout and see if the time between floods follows the change.

Is there any chance that two of the devices have the same MAC? Easy to do with HPUX if you clone them.

STP is not your enemy. It's good to have it on if you have more than one switch.

Do you have a router involved here anywhere?

I suspect you may have a software bug tho it seems strange that someone else hasn't noticed it.

Ron

[Ron Kinner]

OOPS just reread your last post. Four routers. What kind? What dynamic routing protocol are you running? RIPV1, RIPV2, OSPF, EIGRP, IGRP?

Is it only this VLAN that has the problem?

Ron

[Romeo Chance]

Ron,

OK 3 linux routers and 1 cisco router
with 3 paths to the internet.

Basically, I know its HP switch related
as I have today put a simple-simon
3com 10/100 switch just for this egress vlan
and it does not leak traffic.

I have another vlan on a seperate switch
which is having the same symptoms.

If the FDB is being corrupted how can
i check this or is there a newer firmware
in the pipeline to fix this problem.

btw, i have also disabled ABC and IGRP
just to see if it helped--- it doesnt.

thanks
Romeo

[Ron Kinner · Accepted solution]

I would change the MAC age time to the max of 1000000 and see if that slows it down:

mac-age-time 10000000

The timer should be reset each time some traffic from a MAC passes through the switch but I suppose they might have messed up and not done that. That way after 5 minutes the whole table would age out about the same time.

There is a mechanism under security to set the the MAC addresses which are allowed on that port. You would think that if you told it that only one MAC was allowed and it was the MAC of the router that it would then not bother sending out stuff to other MACs. You might try that and see if that helps.

Also CDP has a 3 minute timer and it was recently added so you might try turning it off. I doubt you are using it for anything. I don't see it in the manual I downloaded but it was in the release notes. Think it should be something like:
no cdp enable

There is a note in the manual which says that duplicate MACs on different VLANS are not supported and can cause unspecified problems.

There is nothing in the problem reports but HP's site is really weak and only talks about two switches.

Ron

[Mist Lifter]

Hello,

All Ethernet switches, including the HP ProCurve 4000M, flood unicast traffic in the event that it has not yet learned the port number associated with a particular MAC address. Once the unicast is flooded out the ports, the response to that unicast is what the switch uses to learn the MAC address. Since MAC address learning is a relatively rapid process, I wouldn't expect to see the unicast traffic repeatedly flooded unless the switch did not have the opportunity to learn the MAC address from a reply (where the MAC address is listed in the source field), or the MAC address has timed out of the bridging table.

There are some specific instances in which the switch is unable to learn the MAC address. One example of this is when an imaging program is configured to use unicast traffic. Since the system targeted by the program generally can't respond (doesn't have a TCP/IP stack at that point), the switch is unable to learn the MAC address. Another issue with learning of MAC addresses is resolved in switches with updated firmware (C.09.16).

I hope this is helpful. If you believe that your switch is either a) not learning MAC addresses, or b) flooding unicast traffic for a MAC address that the switch knows, I recommend that you phone ProCurve technical support at (970)635-1000, options 3,2,2 for further troubleshooting.

[Romeo Chance]

Ron,

Thanks for the the help, i changed
the mac-address age to 30minutes
and the flood never showed its ugly
face again.

Seems that 5min setting is either
not very good as the 4 routers
in question are constantly
passing traffic all day long
(peering point) so there is no way
in god's green earth any of the 4
mac-addresses could have staled out...
not every 3-4minutes

I contacted support and they said the
5min thing is default and wanted me to
check if any of the 4 mac-addresses were actually disappearing in the port status
fyi, the 4 mac's never removed themselves.

the dumb thing about this whole deal
is that all traffic passed as expected during
the flood, so basically the switch
was flooding away for its own benefit - definitely a bug.

Thanks a million for not callin me crazy :)


ciao
Romeo

[Ardon]

Hi,

I am an HP Networking Engineer located in Europe but now in the US for two weeks and I happen to be working on this "issue".

In Europe we have heard about this Unicast Flooding as well 2 or 3 times. Although I do not know your exact network topology (and on which switch you saw the flooding) I can assure you that it is NOT a bug as the box is behaving as it should (although you can make it flood).
I should have my complete test report completed by Monday December 16th which I will submit to this forum for you and other people's reference.
If you have multiple switched environment you will probably not see the SAME flooding at all switches at a certain point in time because of the following.
Say you have a Switch A and B which are interconnected, both containing one client each (for convenience of explaining).
On Switch B you have also connected an additional client Sniffing.
Now Ping from the Client connected to Switch A to the Client connected to Switch B (NOT the sniffer client of course:-).
Your Sniffer Client should not capture those Pings (and I verified it does NOT).
Next step is to bring DOWN the Client on Switch B and see what happens with you Sniffer. Flooding, right? But why? Well that is pretty logical. At the moment you bring DOWN Client on Switch B that MAC will be flushed immediately from the MAC Table on Switch B but NOT(!) on Switch A (yet). What that means for the Ping is that it will NOT be flooded on Switch A but when it gets to Switch B where the MAC is flushed you WILL get flooding and thereby the Sniffer will capture it. Mind you that this Flooding will continue with this scenario as with every Ping issued the MAC timer on Switch A will be reset which is different from a test where you use a TCP application for example as that will try only a couple of times and then ceases to try (so not resetting the MAC timer as a result). After 300 sec. (Default) also Switch A will Flush the MAC of the Client connected to Switch B.
Most probably you will see Unicast Flooding in your environment only in just 1 direction (from X--->Y but not from Y---->X right?).

I will conduct the same test on a Cisco and/or 3Com Switch to show that this is common behaviour for your reference.

This is just a short explanation and, like promised before, a more extensive report will follow.

Talk to you later.

[Jon Zeeff]

Do HP switches have an option
to disable the flooding
of unicast traffic when they
don't know which port to send
it to?

I find such an option important - we have some high bandwidth flows and we have
some other low (10 mbs) bw links. In no event do we want these high bw flows being sent to the slow links.
We don't find that flooding
is needed to discover the
correct port.

[Ardon]

Hi,

Yes, Cisco boxes have an option to drop unknown Unicast which would otherwise be flooded. In the ProCurve Switches 4000M/8000M there is an option called "Eaves Drop Prevention" that will the same. In other words. Any Frame with a Destination MAC different from the attached MAC will e dropped. Mind you that this of course does not apply to Layer 2 Multicast and Broadcast.

Thanks, Ardon

[Ron Kinner]

I just had a similar case on Cisco 6500 and I solved the problem so I thought I would post here just in case someone else has the problem.

We have a backup process that runs on a central server and stores the backups on a tape drive. Because it was causing major traffic jams through the gateway router, a second router was installed just for the use of the backup traffic.

Shortly after this was done the bottlenecks became worse. MRTG showed that the backup traffic was being sent to all ports on the central 6500 switch. (The backup traffic was on a 100 full duplex link to a powerful server and our link to the Internet is only 10 half so it was totally useless during backups.) After checking the source of the traffic for viruses and finding nothing, I finally hooked up a computer running snort directly to a port on the 6500. This proved the unicast backup traffic was being sent to all ports. I pinged the backup gateway and the problem went away for a while.

I poked around and found that the traffic was being sent to the correct gateway address but the returns were coming back over the old router. This meant that the switch never saw any new traffic from the new gateway so after the ARP table timeout it erased the original MAC entry (created when the system being backed up ARP'd for the backup router's MAC) and then had no idea where to send the traffic so started sending it to all ports. Correcting the route on the other end so traffic was returned the same way it came fixed the problem.


Ron

[nwengineer]

Ardon, 

Good morning! I found your posting today, which matches issue i have right now in the network. I'm wondering what's your test result with 3Com and Cisco switch regarding this unicast flooding issue. And any solution to resolve this? 

Thank you!