msm765 and m422 flapping

[mhc19]

I have a very odd issue that has been going on for about a week. I have one 765 controller that controls approx. 160 msm422 access points. Typically you will see in the gui of the 765 synchronized, detected and configured counts. But I see those numbers just all over the place. Like ill have 150 synchronized access points and then that number will droo to 2. Its as if all my APs are randomly rebooting. I have an open ticket with hp on this but so far I have been disappointed with support. Today I replaced the controller, after upgrading to 5.7.3 SR2 firmware didn't resolve the problem. I upgraded the firmware on the new controller also to 5.7.3 SR2 firmware, issue persisted. I'm not sure if the controller itself is the problem or if all 160 APs are going bad. Has anybody seen or heard of this. This setup has been in production for over 3 years now. No changes were made to the config, this all just started happening all of a sudden.

[JesseR]

"As if all my APs are randomly rebooting." have you verified if they actually ARE rebooting or not? To verify, check the logs of the AP itself or try pinging the IP address of the AP to see if its actually going up/down.

You're seeing this issue with 5.7.3? Odd....

What do your switches report? Any goofiness at the port level? Check the switch event logs and port logs, or even better, if you have a Syslog server setup. Verify you don't have any switch issues with spanning tree or loop protection, etc.

What about the controller when this is occurring, can you actively ping BOTH the LAN port and iPort of the controller without dropped packets even when the problem is happening? Or do you lose some pings to the controller at the same time as the APs are acting up...?

ANY recent changes on the network itself? I doubt all the APs are acting up.

[mhc19]

Hey Jesse,

   thanks for the reply.  I have a case opened with HP and they are looking through my switch logs and aren't finding too much there, and they looked at log on my controller and don't see an issue.  I will do waht you suggested and try pinging the AP's  and check their logs.  But yeah, it's real strange and all my AP's.  I'm running 5.7.3-SR2.  The only potential issue with trying to ping the AP's it is so random, but will give it a try. 

  BTW, I do have loop protection on my switches - all of them.  HP suspected a loop to, but their reasons for suspecting a loop - high mcast count on a port on my switch is valid.

[mhc19]

I'm starting to think I might be under attack. My only tools are the controller, the switch and pcm+. I don't know for sure, but I work for a school district and the flapping essentially stops after hours. My hunch is a student is using something to bring me down. So you know any good tools to help me identify a potential attack and thwart it? A wired attack I could find, seems daunting to find a roving wireless attack.

[mhc19]

I've narrowed this down somewhat to my guest VSC.  I have 3 other VSC's defined.  But through trial and error i was able to determine that the issue ONLY occurs when guest clients are connected.  I have disabled that guest VSC temporarily till I can find a way to find/track/stop any possible threat.  Just gonna seem hard since my guest VSC is access controlled and all traffic is tunneled through the controller.

[JesseR]

I'm curious... but when the problem is occuring, do people on your NON-tunneled VSCs (egressed VSCs, for example) continue to be able to use the wireless network?

Also, when the problem occurs, does simply rebooting the controller fix the issue with ALL the APs?

And.... the APs that go down, they are showing as Pending or Unresponsive, or BOTH?  (in the MSM controllers main webpage).

Last... on your guest VSC that's tunneled, are you using Bandwidth Allocation to throttle their traffic?

If you want to shoot me an email direct, feel free.  I manage MSM controllers for about 25 different schools, so I'm pretty familiar with them and troubleshooting various issues.

JR

[JesseR]

Also, I don't think the issue is with an attack... Nothing done on the guest-tunneled network should ever be able to affect your APs which are on a different network. (I'm assuming you have Attributes in place to prevent access from the guest network to the production network(s) ??)

I'm suspecting a bug/coding issue. I'd be curious for you to try 5.7.2.0 or 5.7.3.0 (without SR2).

Anyways, email me if you like. I could even do a short remote session with you if you want.

Regards,
JR