[Resolved] Slow Performance

We recently changed out our environment from Cisco 1100 and 1200s for the Pepwave AP One. Initially we just pulled the ciscos and put in the AP one APs in a 1-1 ratio. Verified everyone could log in and went home. The next day we were flooded with complaints about poor performance etc. Note that we used the same channels, so the only thing different was the connection method (using WPA2 now instead of a WEP/domain authentication/IAS solution, and the SSID is different) and the hardware itself. We had previously used a wireless survey tool to scope our our building and the previous wireless network was working great.
So purchased a wireless survey tool (TamoGraph) and found that it appeared the signals were overlapping enough that the signal-to-interference ratio was really high. Seemed like a simple fix so we worked that out by reducing the number of APs we are using (down to 9 currently from 11) to reduce the overlap. The problem consistently is slow performance. The tool we are using (Tamograph) will show expected and actual PHY rates of 80-150 Mbps, yet the wireless users are consistently seeing extremely poor performance and off the wall ping times (1 ms then timeouts, then >100 ms, then 5 ms consistently - all in 1 minute).
I know our 2.4 Ghz spectrum is rather clogged, though most of the additional signals are outside of our building, and we were running on the 2.4 Ghz spectrum for the past 5 or so years w/o this problem (the slowness is directly linked to the changeover to the Pepwave product, so its either the product or the config).
The last piece of information to share is I’d really like to run just 802.11n in the 5 Ghz spectrum. That’s almost completely clear, and the performance change between g and n is the main reason we switched to the Pepwave product (we were running 802.11g prior to now).

Config:
APs are controlled by a Balance 380

Wireless Network:
SSID: WWFPPA
VLAN ID: 0
Broadcast SSID is checked
Multicast filter is off
Multicast rate b/g: 54 Mbps (we had this at the default of 1 Mbps but had terrible slowness/disconnects. Upping to this value helped a lot)
Multicast rate n: MCS7
WPA/WPA2
No bandwidth restrictions
No guest protect, no custom subnet etc etc

AP Profile (we have multiple channel profiles, but the rest is the same, so I will just post the other info instead of each channel)
Location: United States
Preferred Frequency: 5 Ghz
5 Ghz protocol: 802.11an
5 Ghz Channel Bonding: 40 Mhz

2.4 Ghz protocol: 802.11n only
2.4 Ghz Channel Bonding: 40 Mhz
Power output: (varies based on area to reduce signal overlap)
Management VLAN ID: 0
802.1x version: V2
Beacon Rate: 11 Mbps
Beacon Interval: 100ms
DTIM: 1
RTS Threshold: 0
Slot Time: 9
ACK Timeout: 48
Frame Aggregation checked
Frame Length: 50000

When I log into the APs, it shows their radio to be 2.4. I don’t know if that matters or not. Seems like they may not display things the same as the management profile.

Also I have the 3.3.1 firmware on all the APs, and am current on firmware on the 380.

Please help - pulling my hair out here.

I don’t recommend setting the multicast rate that high because the clients will only be able to connect at that rate or higher. If you want to eliminate any 802.11b devices you can set it at 6Mbps and the 11n multicast rate can be MCS0.

Also, you might want to change the channel bonding to auto 20/40 and see how it does.

It sounds like you are in a noisy/crowded RF environment, and this can be difficult to troubleshoot.

Thanks for the reply.

I originally turned up the MCS and Multicast rates to that level and it increased the speed. I’m going to try what you suggest later on this evening when I have a maintenance window again.
Also, it was recommended based on other sites I’ve been reading not to use the Auto 20/40 as the overlap reduces speed. Not sure if that’s correct. I can certainly try that as well.

The odd thing is we shouldn’t be in a very congested area. I can see a lot of other networks, but inside our building (which is pretty good sized) we don’t have any wireless networks and limited (if any) wireless devices using the 2.4 or 5 Ghz spectrum.

I did increase the RTS threshold to 500 (from 0) on 3 of the APs and that did seem to help a bit with speed.

One thing I’m still curious about is the statement made on the info sheet for the AP One:

“The AP One family of access points is engineered with today’s increasingly distributed and mobile enterprise in mind. Packed with business-class features and housed in rugged enclosures that withstand vibration and temperatures from -20 to 65°C, AP One access points deliver dependable
and fast performance both indoors and outdoors – even in extreme environments. TruePower RF technology eliminates dead spots and interference wherever you deploy your network, and every channel runs at full power to give you greater Wi-Fi coverage with less equipment,
maintenance, and cost.”

In my situation we’ve needed multiple access points for each part of the building (we are sort of like 3 buildings all connected) as a single one on MAX power can’t cover everyone effectively, but I’m finding that I can’t have 2 APs on MAX in the same area (even if they are on completely different channels, such as 1 and 6) without significant signal and speed degradation My understanding from the quote above is that I should be able to run both APs with different channels w/o interference due to whatever the “TruePower RF technology” is. With our Cisco equipment we had 3 access points (different channels) in each building, running on 1, 6, and 11, and they didn’t seem to interfere near so much as we have now.
Is that just a side effect of the significant signal difference?
At lunch time we played around with 3 APs that are in the same building, and on medium power I get full bars (5) and good speed (~40 Mbps down, ~26 Mbps up), but if I go to High or Max I go down to 2-3 bars and have awful speeds (~1-5 Mbps up and down). We are using a tool that came with the Tamograph wireless survey tool - the Tamograph throughput test.
I’m open to suggestions as to other good throughput/real world speed tests.

Does anyone have any ideas on this? I’m pretty much done with Pepwave if not. I can’t keep decent throughput anywhere in the office.
I spent a week on a small area and finally had it fast enough for 2 days (thursday and friday) that they were happy with it.

Come in today, and everywhere in my office is SUPER SLOW. I have receive error rates that are out of control (over 600k per device across 2 days). I made 0 changes to the wireless or internal network this weekend intentionally due to the problems we have to make sure I didn’t change something that in turn would cause a problem. Yet I suddenly have problems across the entire office. I show users with 150M/80M or better, yet takes 10-40 seconds to open a chart, where for a wired user takes 3-4 seconds.

Frustrated beyond words. The device is to work with up until where it really counts - throughput and usability.

If you haven’t opened a support ticket yet, please do so by visiting:
http://www.pepwave.com/contact/support/

We can have you submit the Diagnostic Reports from the devices and we will take a look at them.

I did change my settings based on what you recommended. I wasn’t able to set the multicast rate for b/g to 6, as my options were 5.5 or 11, so I chose 11. Changed 802.11n to MCS0 and the channel bonding to 20/40.

Spectrum analysis points that the only thing we have strongly on our 2.4/5 Ghz spectrum is the SSID of the pepwave devices. It sees the other ones in some areas but not nearly so strong (in general -80 to -100 dbm, so it shouldn’t be a noisy/crowded environment. We are a quite large facility with a street on two sides and our parking lot on the other two, so somewhat isolated in that fashion.

Checked speeds and they show they should be pretty good, but still causes the application to just drag. Before opening items would take maybe 20-25 seconds, now opening items takes 15-20 seconds. On the prior wireless and wired in we see < 5 sec to open items.

Well I put in a support ticket. The tech emailed back and asked me to turn on the remote support so I did, and emailed him/her a bunch of other information regarding my problem as well. that was Monday - and it is nearly Thursday and haven’t heard a word back.

Fortunately, it appears I have fixed my problem.

I stopped managing the access points.

We were using our Balance 380 with the AP Profiles to manage the access points (what a GREAT feature it was supposed to be) but given my job is (was? I hope …) on the line due to this product, we decided to try something crazy and factory reset the APs and set up each one manually with the settings we had on the Balance 380.

And it worked. Suddenly I have decent speeds throughout the building and my wireless survey shows great RTT and PHY rates. I’m not getting near the performance I was hoping for, mainly because of the RF interference I get when the Peplink APs overlap each other (no one ever did answer why the sales/datasheet says this isn’t a problem, when it actually is), but at least my network is stable again and the peasants have put away their torches and pitchforks.

Man — caveat emptor … I’m REALLY annoyed I can’t send this stuff back. I’m not sure how everyone else gets theirs stuff working so well, but this has been a horrible experience for us.

We are sorry for the delay in response. I’ve just replied back to you via the ticket you created. We will try our best to get the AP Ones to work in your WiFi environment.

I have requested some information from you in the ticket. So if you could reply back to us, we would appreciate it. We will definitely try to get to the bottom of the issue.

Dear Haruki ,

Please post the solution in the forum also , we had the same issue with AP One before and I’m following this topic.

Thank you in advance.

Absolutely, Hootan. I will follow up.

We had moved the AP Ones to a “stand-alone” state, where we were NOT managing them with the Balance 380. Suddenly the throughput was great, and things seemed to be working just fine again (we had them in this state for about 5 days). This was when we started working with Haruki.
Updated the firmware on the Balance 380 to 5.4.7 (we were on 5.4.1 - the “check for updates” apparently wasn’t working). Moved the AP Ones back to a managed state yesterday (not even 24 hrs ago). I set the AP One profiles exactly how we had them in an unmanaged state, and then changed the QoS to how I wanted it on a per-SSID state (a personal network and a corporate network, so giving the corporate network Gold priority) and turned on IGMP snooping (both were set this way in unmanaged mode, but you can’t set them in the profile on the Balance 380). Got everyone connected, and didn’t hear anything good or bad - everyone agreed it was the same.

Today a few hours ago one of the AP Ones went haywire - all users in its range slowed to a crawl, were dropping packets heavily, and a few even just stopped working on the access point (could not connect to the SSID any longer, no wireless connectivity). Then about an hour later, without us doing anything but try and figure out what was going on (logging into Balance 380 to try and look at logs and logging into the AP One and looking at logs) suddenly the computers started working again without problems. A few random other users throughout the building had a similar problem, but it wasn’t consistent in any other area (we have 9 active AP Ones across our facility - same LAN)
We had no users doing anything out of the norm (all were using our EMR product which is not very bandwidth intensive), and we were having no other network problems at the time.

About ready to hop off the managed aspect with the Balance 380 as it would appear I either have defective AP Ones, or a defective Balance 380 in regards to managing the AP Ones. From what I’ve seen I don’t know why a controller holding the config should cause this problem but it seems to be the case.

Hootan - what did you do to prevent this problem? It sounds like from your post you found a solution of some nature. Did you just stop managing them?

Not Actually , We had the same issue and still have it on AP One. in 90% cases it works fine but suddenly the speed and performance is horrible. usually we have <1ms ping tests but suddenly 500ms , 450ms , 390ms and somethings like that. but we don’t have such a problem on AP One 300M access points. the performance on them are great.

I see. Interesting that you have an identical problem to mine. Leads me to believe its a problem with the AP One model. We’ll see what support comes back with. Thanks for the reply.

but something I find is that if you use power boost in a small indoor area with many walls , doors the performance would infected and you have to set the power output level to low or medium and disable the power boost. that would may be because of the high power output and some kind of intensive signal reflex. try that :wink:

That is interesting. I have 1 AP One on “High”, the rest are on “Medium”, so I don’t think that’s what causing my problem.

Well still having problems. I took one of the AP Ones having the biggest problem with sudden area slowness off the controller and that problem isn’t showing up anymore. So apparently having the AP One profile on the Balance 380 causes some problem now and again.

Still have poor performance across the network in general - the crazy thing is the snapshots of performance I’m seeing shows that with 10-15 clients connected we are using < 1 Mbps on the graph, with maybe a spike or two of <3 Mbps every once in a while. So we certainly aren’t saturating the network with wireless traffic. Searching for a good way to track the connection speeds etc, as when a problem happens and we log in to check it doesn’t appear anything is wrong, short of the device having poor RX times (usually seeing 65 M/6M or something - the receive rate (6M) seems really off).

Still working with support but still having extremely poor performance even compared to our 802.11g solution from Cisco we used prior with WEP.

Add to the intrigue: The “AP Info” tab for the wireless is showing that at least once an hour the number of clients for my primary SSID goes to 0, then pops right back up (the graph shows increments of 10 minutes, so somewhere in the 10 minute timespan the # of connections drops to nothing, then pops right back up to where it was). The connected times on the “Connected Clients” tab shows they have been connected the whole time.

So one of the two datapoints is wrong.

Thanks, the support team will continue to follow up via the ticket…

Updated to what appears to be a beta release of the firmware (3.4.01b) and that took care of the wireless drops. Still have slowness for no apparent reason. Ping times of >35 consistently, with lots of drops. The next day things run fine, then a slightly different problem. We have even purchased brand new tablets without this helping.

Very frustrating.

Well we got on beta versions of firmware on the AP Ones and also the Balance. Turned the silly AP Ones upside down so the antennas point down. And got some reprieve for about a week. Now inexplicably the slowness is back and all over the place.

Pulling my hair out again. Reported it back to support. There apparently is no way to monitor the AP Ones in a significant way via SNMP, so you are stuck going in each time and trying to manually check what is going on. No way to really monitor any problems, or get an idea of what is happening. Just blind in the dark hoping thing are working.

Net admin nightmare.