MSM760's - MSM310 & 410 - Performance Issues

[DntBrnDPig]

I'm looking for some clues as to why my performance is horrible.  I work at a college and every time the students get on campus my wireless proves crappy.  I've went through all the dorms and made sure my coverage is good. 

I have my radios configured to autmatically adjust channels ever 2 hours along with the power.  I have excluded all channels except for 1,6,&11 on all non-n radios.  I dont have any errors on the interfaces on the HP switches that feed these AP's.

When i connect to the AP, I can have full signal.  My pings to my Gateway are horrible.. upwards of 700ms and I continually drop packets.  If I plug in to the switch that the AP's are plugged into, my pings are less than 1ms with no dropped packets.  When I do a spectrum scan - it looks good with only 1,6,and 11 being used with a little overlap.

Where should I be looking for my issue?  Any ideas?

My controllers are running 5.5.4.

[tschaps]

I am sad to say I am seeing some of this already, and the students have not returned to school yet. I have 60 MSM 430's and 466s in a high school, and we've been getting new iPads enrolled in our MDM this week, and until we adjusted our workflow, upwards of 300 iPads were "on" but in sleep mode in the same general area filed in boxes, but they had no useable internet connection. The iPads retain an association with an APs even in sleep mode. At any given time, only a few iPads were doing anything, mainly consisting of using a web-based authentication system to enroll them, no streaming video, but having 150 iPads just associated to one of our 430's completely paralyzed it-- no traffic on a connected iPad, or even to a laptop trying to use a different VSC/VLAN on the AP.

iPads almost always associate to Radio 1 (5 Ghz), even with bandwidth steering off. Despite not planning to ever have this density of iPads on one AP during normal school days, this does not bode well for this system, considering the amount of traffic generated per device will be signficantly higher than right now, and we also have additional devices which will be using the wireless network. I have a growing sense of dread about next week (and I am frankly regretting going with HP with all the problems we've had). 

I was troubled to see that madwifi is used behind the scenes in this HP/colubris equipment, because my last nightmare with an enterprise wireless system in Fall 2008 was using madwifi-based equipment which started doing this same thing with only 20 devices associated. The recommended solution at that time was to turn off WPA and use open or WEP, neither of which were options. The system was still small enough to be dismantled and returned to sender after they sent out an engineer to confirm that madwifi was associating and disassociating with every client when the number of clients reached the 20's, and the overhead of all that processing destroyed bandwidth. The engineer said it did not happen in their simulations with a device which would simulate 60 clients pounding the AP hard. In real life, 25 clients hardly doing anything rendered the APs useless, and at about 30 clients, it would reboot. 

If this is deja-vu all over again, this time with much higher stakes, I think I might lose my sanity. We paid for an expensive site survey, and the vendor partner said 60 of these APs would cover a projected 2,000 total devices in our school (1,500 iPads plus some laptops).  Don't forget the dual MSM765zl's mounted in dual VRRP'd e5406 core switches and new HP edge switches. The whole HP sauce is getting swallowed in this next week. If anyone has any real-world tips to keep the MSM AP's up, working, and stable, it would be REALLY appreciated now. 

[DntBrnDPig]

Ugh...

[JesseR]

Has anyone opened a support ticket with HP on this? Without a ticket, HP isn't going to address the issue. Very surprised tschaps is seeing the issue on the 5.7.0.3 code he's running. Log a case.. squeaky wheel gets the grease so don't be afraid to ask for escalation.

I might be able to test this tomorrow at a customer's environment that has a ton of iPads devices. Have to see if I can wheel in a few carts of them to a given area and test connectivity from a laptop or tow even when those carts of iPads are idle. Not sure if I'll have time to do this, but I'll try.

One thing I'd recommend, if you haven't already, is enable broadcast filtering on your VSCs... for chatty devices, it can sometimes help a lot.

[cenk sasmaztin]

Hi

I have two case for MSM performance on HP support since April

if you want help me... I want some test with you..

please download my test guide..make test and say me result. 

[DntBrnDPig]

I may be tripping down the wrong path, but I have noticed A LOT of multicast traffic.  I have multicast filtering on the lowest setting, but according to a Wireless LAN Analyzer we have it says the majority of the traffic is multicast. 

It's also picked up on CTS DoS attack.  I dont know if this is a false positive or not.

Anyone have any ideas on how to track this down?

[tschaps]

That is actually very interesting - please share more of what you found. Multicast over wireless is essentially a broadcast, and it may be tying up the resources. 

I was also wondering if the CTS/RTS settings could affect these lockups, too. I googled a little bit about CTS DoS attacks, and that's kind of scary. What brand of Wireless LAN analyzer do you have?

[DntBrnDPig]

I have a Fluke Optiview Tablet (that we are still learning how to use it).

So, last night I tried something a little different.  Instead of limiting the multicast traffic to only 6Mbps I opened it up to 48Mbps.  Immediately my pings (to a machine connected wirelessly in a building I am having troubles with) got much more consistent and lower.

I read some articles on Bonjour and believe this is the source of all my broadcasts.  Every student in college has at least 1 apple product trying to announce their presence to the world.  I know Aruba and Aerohive have come up with a way of dealing with it, but I'm not sure what else to do. 

[tschaps]

Very interesting - are you referring to the "Multicast Tx rate" setting on the radio settings page? If so, I would not have guessed that would help from the help page, which says "Use this parameter to set the transmit rate for multicast and broadcast traffic. This is a fixed rate, which means that if a station is too far away to receive traffic at this rate, the multicast is not be seen by the station." Sounds to me like you'd be having more connectivity issues at that setting.

Again, assuming that is the setting to which you were referring, the Multicast Tx rate goes up to 54 mb/s, how did you arrive at the 48 mb/s setting to try? Something you read?

As for Apple devices, the 1,300 iPads I am rolling out starting Tuesday qualify, so I hope some adjustment of this setting works. Thanks for the info.

[DntBrnDPig]

Like I said, I have a laptop connected in one of the dorms I'm having trouble with.  I have a continous ping going to that machine.  I figure my pings are probably a decent indicator of the user experience without actually being there. 

Anyway, so I tried changing that setting to 36Mbps just to see and noticed my pings went down.  Then I set it to 48 Mbps and it got better.  So a couple hours late I changed it to 54Mbps but, again I dont know why, my pings started becoming irratic.  I put it back to 48 Mbps and they settled back down.

Again, I have no clue why this is.  The best I can figure is the radio is geting bombarded with these packets... and by limiting them they tie up the radio cause they are constantly hitting.