MSM760's - MSM310 & 410 - Performance Issues

[DntBrnDPig]

I'm looking for some clues as to why my performance is horrible.  I work at a college and every time the students get on campus my wireless proves crappy.  I've went through all the dorms and made sure my coverage is good. 

I have my radios configured to autmatically adjust channels ever 2 hours along with the power.  I have excluded all channels except for 1,6,&11 on all non-n radios.  I dont have any errors on the interfaces on the HP switches that feed these AP's.

When i connect to the AP, I can have full signal.  My pings to my Gateway are horrible.. upwards of 700ms and I continually drop packets.  If I plug in to the switch that the AP's are plugged into, my pings are less than 1ms with no dropped packets.  When I do a spectrum scan - it looks good with only 1,6,and 11 being used with a little overlap.

Where should I be looking for my issue?  Any ideas?

My controllers are running 5.5.4.

[tschaps]

I am sad to say I am seeing some of this already, and the students have not returned to school yet. I have 60 MSM 430's and 466s in a high school, and we've been getting new iPads enrolled in our MDM this week, and until we adjusted our workflow, upwards of 300 iPads were "on" but in sleep mode in the same general area filed in boxes, but they had no useable internet connection. The iPads retain an association with an APs even in sleep mode. At any given time, only a few iPads were doing anything, mainly consisting of using a web-based authentication system to enroll them, no streaming video, but having 150 iPads just associated to one of our 430's completely paralyzed it-- no traffic on a connected iPad, or even to a laptop trying to use a different VSC/VLAN on the AP.

iPads almost always associate to Radio 1 (5 Ghz), even with bandwidth steering off. Despite not planning to ever have this density of iPads on one AP during normal school days, this does not bode well for this system, considering the amount of traffic generated per device will be signficantly higher than right now, and we also have additional devices which will be using the wireless network. I have a growing sense of dread about next week (and I am frankly regretting going with HP with all the problems we've had). 

I was troubled to see that madwifi is used behind the scenes in this HP/colubris equipment, because my last nightmare with an enterprise wireless system in Fall 2008 was using madwifi-based equipment which started doing this same thing with only 20 devices associated. The recommended solution at that time was to turn off WPA and use open or WEP, neither of which were options. The system was still small enough to be dismantled and returned to sender after they sent out an engineer to confirm that madwifi was associating and disassociating with every client when the number of clients reached the 20's, and the overhead of all that processing destroyed bandwidth. The engineer said it did not happen in their simulations with a device which would simulate 60 clients pounding the AP hard. In real life, 25 clients hardly doing anything rendered the APs useless, and at about 30 clients, it would reboot. 

If this is deja-vu all over again, this time with much higher stakes, I think I might lose my sanity. We paid for an expensive site survey, and the vendor partner said 60 of these APs would cover a projected 2,000 total devices in our school (1,500 iPads plus some laptops).  Don't forget the dual MSM765zl's mounted in dual VRRP'd e5406 core switches and new HP edge switches. The whole HP sauce is getting swallowed in this next week. If anyone has any real-world tips to keep the MSM AP's up, working, and stable, it would be REALLY appreciated now. 

[DntBrnDPig]

Ugh...

[JesseR]

Has anyone opened a support ticket with HP on this? Without a ticket, HP isn't going to address the issue. Very surprised tschaps is seeing the issue on the 5.7.0.3 code he's running. Log a case.. squeaky wheel gets the grease so don't be afraid to ask for escalation.

I might be able to test this tomorrow at a customer's environment that has a ton of iPads devices. Have to see if I can wheel in a few carts of them to a given area and test connectivity from a laptop or tow even when those carts of iPads are idle. Not sure if I'll have time to do this, but I'll try.

One thing I'd recommend, if you haven't already, is enable broadcast filtering on your VSCs... for chatty devices, it can sometimes help a lot.

[cenk sasmaztin]

Hi

I have two case for MSM performance on HP support since April

if you want help me... I want some test with you..

please download my test guide..make test and say me result. 

[DntBrnDPig]

I may be tripping down the wrong path, but I have noticed A LOT of multicast traffic.  I have multicast filtering on the lowest setting, but according to a Wireless LAN Analyzer we have it says the majority of the traffic is multicast. 

It's also picked up on CTS DoS attack.  I dont know if this is a false positive or not.

Anyone have any ideas on how to track this down?

[tschaps]

That is actually very interesting - please share more of what you found. Multicast over wireless is essentially a broadcast, and it may be tying up the resources. 

I was also wondering if the CTS/RTS settings could affect these lockups, too. I googled a little bit about CTS DoS attacks, and that's kind of scary. What brand of Wireless LAN analyzer do you have?

[DntBrnDPig]

I have a Fluke Optiview Tablet (that we are still learning how to use it).

So, last night I tried something a little different.  Instead of limiting the multicast traffic to only 6Mbps I opened it up to 48Mbps.  Immediately my pings (to a machine connected wirelessly in a building I am having troubles with) got much more consistent and lower.

I read some articles on Bonjour and believe this is the source of all my broadcasts.  Every student in college has at least 1 apple product trying to announce their presence to the world.  I know Aruba and Aerohive have come up with a way of dealing with it, but I'm not sure what else to do. 

[tschaps]

Very interesting - are you referring to the "Multicast Tx rate" setting on the radio settings page? If so, I would not have guessed that would help from the help page, which says "Use this parameter to set the transmit rate for multicast and broadcast traffic. This is a fixed rate, which means that if a station is too far away to receive traffic at this rate, the multicast is not be seen by the station." Sounds to me like you'd be having more connectivity issues at that setting.

Again, assuming that is the setting to which you were referring, the Multicast Tx rate goes up to 54 mb/s, how did you arrive at the 48 mb/s setting to try? Something you read?

As for Apple devices, the 1,300 iPads I am rolling out starting Tuesday qualify, so I hope some adjustment of this setting works. Thanks for the info.

[DntBrnDPig]

Like I said, I have a laptop connected in one of the dorms I'm having trouble with.  I have a continous ping going to that machine.  I figure my pings are probably a decent indicator of the user experience without actually being there. 

Anyway, so I tried changing that setting to 36Mbps just to see and noticed my pings went down.  Then I set it to 48 Mbps and it got better.  So a couple hours late I changed it to 54Mbps but, again I dont know why, my pings started becoming irratic.  I put it back to 48 Mbps and they settled back down.

Again, I have no clue why this is.  The best I can figure is the radio is geting bombarded with these packets... and by limiting them they tie up the radio cause they are constantly hitting.  

[cenk sasmaztin]

Hi

can you give me Fluke Optiview Tablet report....

[JesseR]

Just for reference...  I did some testing with 90 iPads on a single MSM460 AP.  All iPads, plus a few other wireless devices connection to Radio 1 of the AP.  I believe there was 102 wireless clients in total on the AP.

I found that I did not experience any sluggist performance while the iPads were just sitting idle/charging in their Apple Cart (they maintain an active wireless connection during the charging process and CAN be pinged as well).  My laptop was still able to get very good ping times (2-3ms) to a server, and even transfer a 250MB file at around 9MB/second. 

The only time things get a bit congested is when the iPads start their call-home functionality for MDM and all that... when that happens, each iPad talks directly to apple.com servers (107.14.38.145 addresses) and this communication was tracked and verified on the firewall.    WHEN those iPads do their call-home stuff, I really have no idea, but it does use quite a bit of bandwidth during that process which seems to last about 10 minutes or so. 

The iPads are running IOS 5.1.1 (9b206), and Model MC769ll.

Also it seems like when iPads go into a sleep mode (which happens after 2-3 minutes when they are NOT charging via the adapter cable in the iPad cart)  they drop their association with the AP and can no longer be pinged.  While they maintain their wireless connection for at least several hours so long as they are plugged-in/charging, eventually they will stop associating with the AP, even while plugged in.  Though, I haven't yet nailed down WHEN they drop their association with the AP when plugged in.  It seems to take more than a few hours before that happens.

So, at least as far as I can tell... Having 3 iPads carts, each with 30 iPads, all charging and sitting idle, does not effect the wireless performance for active wireless clients.  At least, not in this test environment.

JR

Need help with MSM deployments?

http://h30499.www3.hp.com/t5/MSM-Series/MSM-Deployment-Scenario-How-To-Guide/td-p/5717081

[tschaps]

---

@JesseR wrote:

... The only time things get a bit congested is when the iPads start their call-home functionality for MDM and all that... when that happens, each iPad talks directly to apple.com servers (107.14.38.145 addresses) ...

You mean 17.14.38.145, right? I believe Apple owns the whole 17.0.0.0/8 block of addresses. Also, what multicast tx rate are you using on Radio 1?

Thanks

[JesseR]

I could have been wrong with the IP... would have to double check but you're probably right.

I've used multicast rates of the default 6Mb and at some customers I think we're trying 24Mb for multicast rate on Radio 1 (this is what HP has suggested to me to use).   For Radio 2 (2.4Ghz), I've either used the default 1Mb or 11Mb (this is what HP has suggested for me to use).

I know you've had more iPads on an AP than I've been able to test... but I haven't been able to get that many iPads together in one area.   I've also heard some rumblings that after 120 client connections on an AP, there's problems(?).  I haven't gotten to that # yet, but mostly because I generally change the default Max Client Connections on a given Radio from 255 to around 60 or 75 (per radio) in my deployments.

JR

Need help with MSM deployments?

http://h30499.www3.hp.com/t5/MSM-Series/MSM-Deployment-Scenario-How-To-Guide/td-p/5717081

[Evan_ISS]

---

@DntBrnDPig wrote:

I'm looking for some clues as to why my performance is horrible. ...

---

I will play devil's advocate a bit here so bear with me.

To the OP: has a full site survey been performed taking into consideration the type, number and distribution of clients? If not then you might have right there the cause of your issues. Nine times out of ten when we are called in by education customers to troubleshoot wireless deployments, we deal with vanilla installations. If one has been performed then you will have to revisit the assumptions and conclusions and go back to the drawing board if these have been found lacking (and query the vendor/partner and survey agent as to why this is the case).

To user tschaps: As above. You also mention that the density that causes your AP to "paralyse" was not planned. How can such a density be testament to the quality of the installation? If however under normal usage the solution proves inadequate then you have your survey to fall back on and rub it in the face of the partner/vendor.

(Devil's advocate mode off now)

I can't stress enough the importance of a survey (or surveys and subsequent implementation/configuration schedules) that take into consideration all the necessary parameters and intended outcomes.

My organisation manages a fair amount of HP MSM wireless deployments (most of them inherited) and although we had big problems in the past (and still some niggles now), since software 5.7.0.3 (and 5.7.1 now) performance  issues have been minimal and most incidents we encounter are due to bad design.

Minor detail: we almost always manually configure the managed APs we install (that is the point of the survey anyway). We have consistently found that no controller algorithm can better human perception (and common sense).

Hope the above helps.