The ARP packet rate(89pps) exceeded the rate limit (50pps) on interface...

[MichaelM55]

Hello,

I have hundreds of those messages in my log of one of my switch stacks:

%Oct 10 06:23:16:033 2013 DOTAN1 ARP/4/RATELIMIT: The ARP packet rate(89pps) exceeded the rate limit(50pps) on interface GigabitEthernet9/0/26 in the last 60 seconds.

• on different interfaces
• all the time
• but there´s no loop (loop detection is on, with multiport, per vlan, action, semi shutdown)
• asking myself

- What limit? I didn´t set that one.  So let´s set "arp rate limit disable"?

- What are normal values for "broadcast suppression" (for edge, servers,...)?

- These messages aren´t informational only?

[paulgear]

Hi Martin,

I've been seeing a lot of these on one of our sites.  I haven't had a chance to chase down what is causing it, but I'm guessing it's a feature of recent firmware that is not particularly well-tuned by default.  If anyone has any further info, I'd be keen to know more.

[ccavanna]

I would be interested in that information as well. I started seeing that on our switches after we upgraded the firmware on them so my guess its something new in the firmware. But I haven't had a chance to track down what it is either. 

[Richard Brodie_1]

It's an awful lot of ARP packets, mind you. I'd be inclined to fire up Wireshark, or whatever, and see whether you have some antisocial nodes there. There could be some sort of scan happening (if only a harmless network inventory kind), or maybe some buggy nodes not properly rate-limiting their own ARPs.

[paulgear]

Hi Richard,

Our Ruckus wireless access points do sub-second ARP checking of their default gateway to verify connectivity.  On the site where I'm seeing this, we have 20 access points.  So 40 out of the 50 ARP requests in a second could easily be just those 20 APs.  I'll do a bit of packet capture and see if there's anything else contributing, but it seems to me that this is just much too sensitive a warning, and 50 pps is much too low a level.

What would a conference or exhibition do where they often have 20K users in a building?  If all their wifi APs were on a single 10 GbE backbone you'd see orders of magnitude more ARP than what I have...

[Richard Brodie_1]

Last time I looked the recommendation was to rate limit ARPs to a maximum of 1/second, so I would count the behaviour of your APs as antisocial. However, as an old DECnet guy,  I consider any use of the  broadcast address as antisocial ;)

Coincidentally, this morning I was discussing moving some boxes that don't like excessive broadcasts and lock up onto a private network; here 'excessive' was more than about 10 pps. In the process of investigating, I found a few systems sending ARPs because their subnet mask was incorrectly set.

Anyway, I guess  am trying to make two points:

1. What a reasonable level is for you is way different from what it is for me; interesting to see a different perspective though.
2. Having a good idea of what your baseline load is, and whether wierd stuff is going on is often worth it. Without that, there is no sensible way of answering the part of the original question about "normal" values for broadcast supression.

I like some of the Comware stuff but the "we put so many features in our switches, there's no time to document them all" approach can be a bit trying at times.

[paulgear]

Hi Richard,

I was mistaken about the ARP interval; it averages about 1.1-1.2 seconds; so it sounds like Ruckus have read the same recommendation as you.  :-)

Here's a rather bizarre take on the error message:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?sp4ts.oid=4218345&spf_p.tpst=kbDocDisplay&spf_p.prp_kbDocDisplay=wsrp-navigationalState%3DdocId%253Demr_na-c03661151-1%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

Especially strange given that RSTP is enabled on all the switches on which I'm seeing this log message.

Another strange one here suggesting it's an actual broadcast storm:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?sp4ts.oid=4218345&spf_p.tpst=kbDocDisplay&spf_p.prp_kbDocDisplay=wsrp-navigationalState%3DdocId%253Demr_na-c03784825-1%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

It appears "arp rate-limit rate information <seconds>" can be used to reduce the frequency of this message, and that the 5500 and higher platforms have a tunable rate limit, but on the 5120s and 3100s there doesn't seem to be a command to do it.

[paulgear]

Hi folks,

Has anyone had any further progress on what causes these messages? I'm seeing rates up to 1000pps on the same network, and it's relatively small - certainly less than 300 nodes total.

I've found that 5120s do have some tunable parameters: my current firmware (Version 5.20.99, Release 2220P02) allows "arp rate-limit rate PPS drop" or "arp rate-limit disable" at the interface level, and a number of new "arp rate-limit" and "arp anti-attack" commands at the top level. Anyone experimented with these settings?

[paulgear]

Further to my last message, one of the switches on this site is reporting via syslog:

Jan 29 12:31:39 ... %%10ARP/4/RATELIMIT(l): The ARP packet rate(1000pps) exceeded the rate limit(50pps) on interface GigabitEthernet1/0/24 in the last 60 seconds.
Jan 29 12:32:39 ... %%10ARP/4/RATELIMIT(l): The ARP packet rate(1000pps) exceeded the rate limit(50pps) on interface GigabitEthernet1/0/24 in the last 60 seconds.

Yet over the timeframe in question, the switch reports something very different via SNMP:

This seems to suggest that one or other measurement is not reporting the correct value, or the ARP rate limiter is reporting instantaneous values rather than sustained ones.  My SNMP poller runs every 5 minutes and thus reports the change averaged over the period.  An average of 18 pps between 12:30 and 12:35 seems completely reasonable given there are 20 APs using ARP-based polling to ensure their default gateway is alive.

Would I be better off just disabling this feature?

[MichaelM55]

Well, i disabled it everywhere. a arp packet rate of 1000 pps? How many clients do you have behind that port?

(Btw, which OID is used for broadcast / multicast monitoring?)