MSM760 logging overload

[Adam_Brown]

Short story: My MSM760s are running really high CPU due to massive amounts of RADIUS failures being logged to the controller logs.

Long story: I have two MSM760s. Both manage 200 APs each (mostly 410s with a few 310s and 422s). The APs service thousands of devices across 45 different WAN sites (we are a medium-sized school district). We are currently doing only MAC address authentication with two Windows 2008 R2 NPS RADIUS servers. Everything else is super simple. No VPN, firewalling, etc. No traffic goes through the controllers. I'm pretty much just using the controllers to provision the access points. Here's the problem. Since we are doing only MAC address authentication, the SSID shows up as unsecured and every student, teacher, parent, and random passerby's unregistered phone or other device tries to connect and fails because the device isn't registered on the RADIUS server. I analyzed just one of the log files (I let it log for 30 minutes) and found that of the 121,033 authentication requests, only 436 of them came from registered mac addresses (133 devices). 120,597 of them were from mac addresses not in our system. One mac had over 10,300 requests. There were about 915 unique invalid mac addresses.

All that said, the access points request authentication from RADIUS directly, not through the controllers, and it looks like our RADIUS servers are handling the traffic just fine. The problem is that the access points turn around and log all the failures to the controllers and it's killing them. The controllers are running near 100% utilization throughout the day when school is in session. They are fine in the evening because no one is around. I've disabled LLDP, IGMP Proxy, and other stuff that were known to cause high CPU issues. It gets to the point where I can barely log in to the controllers and loading pages time out.

I figure I can stop broadcasting the SSID or set a WPA key to slow down the unregistered devices from trying to authenticate at all, but doing either would be a huge undertaking as we would have potentially thousands of devices to configure. We've been using the same SSID and authentication for several years. We also have around 400 older Cisco APs that are all unmanaged, so it's a pain to make sweeping changes. Our Cisco access points have an option to stop sending RADIUS requests for x seconds after a client fails to authenticate. Anyone know if there is an HP equivalent? Any other ideas?

Here are a few lines out of one of my controller logs. As you can see, these are only coming in every few minutes. When everyone starts coming to school/work around 8:00AM, there are 10-50 per second.

Oct 31 06:27:32 warning	macauth      TW127C4059 RADIUS Authentication of station (mac-address='BC:67:78:24:E1:B4') was rejected by the RADIUS server.
Oct 31 06:27:22 warning	macauth      TW132C40GX RADIUS Authentication of station (mac-address='10:C6:1F:7D:9B:56') was rejected by the RADIUS server.
Oct 31 06:26:25 warning	macauth      TW127C4059 RADIUS Authentication of station (mac-address='BC:67:78:24:E1:B4') was rejected by the RADIUS server.
Oct 31 06:22:59 warning	macauth      TW132C40MZ RADIUS Authentication of station (mac-address='14:8F:C6:5A:AC:BC') was rejected by the RADIUS server.
Oct 31 06:22:34 warning	macauth      TW132C40GX RADIUS Authentication of station (mac-address='10:C6:1F:7D:9B:56') was rejected by the RADIUS server.

Thanks!

-Adam

[cenk sasmaztin]

Hi Adam
I have two questioons
1-your two controllers connect to be teaming
2-which software version running on controllers

[GertH]

We have had controller reboot because of overload. I disabled "Use controller for authentication" in the config. I als added my access points to the Microsoft AD radius config. This config stopped my controller overload.