Speedfusion - Packet Loss Issues

Product Discussion Pepwave MAX
1

I’m having some trouble at a number of my sites with Speedfusion bonding and packet loss. I’m deploying Max Transit Duos (FirstNet versions) and while my AT&T Cell links work fine, nearly all my Verizon links are unusable and a fair amount of my residential WiFI links are also unusable I am using the Balance SDX unit on my receive side behind a Palo Alto firewall with port forwarding in place to forward UDP 4500-4510 and TCP 32015-32025 and I’ve confirmed that the firewall is passing everything cleanly. My Speedfusion tunnels are all Layer2 with the SDX in Drop-In Mode as I’m behind a larger firewall that is handling all the local network needs; the SDX is basically acting as a VPN Concentrator only. My receive side is on a 1gig link that is relatively lightly used so it doesn’t appear to be a local congestion issue.

With Verizon, all my sites have good to excellent signals across multiple bands (I’ve seen 2, 4, 13, 66 at differing sites, most seeing many strong bands) but when I add them to a Speedfusion pool, the packet loss jumps by a factor of 10 practically knocking the site offline. Latency tends to be approx. 50 msec and is stable, but when Speedfusion tries to send packets out the Verizon side, it just doesn’t work.

Most of my WiFi links are backed by cable modems (Comcast and WOW mostly) and while not as bad, I’m getting very similar results with them as well causing me to be unable to use a fair amount of those too. It doesn’t appear to be the WiFi though as in the places where I can hard line, the results are identical to what I saw on WiFi.

I have FEC set to high on the Speedfusion configuration and I have tried WAN Smoothing, but that didn’t have any effect; it pretty universally made things worse as it tried to send even more data on the poor networks… I have a latency cutoff of 250msec set, but like I said, it doesn’t seem to be a latency issue, it’s straight packet loss. My antennas are primarily Puma 401 antennas, but I have a few Panorama WMMG-7-27-5SP panels in the mix too in locations that needed some signal help.

The only way I’m able to use these sites in any semblance of good condition is to take the heavy packet loss links out, but that’s completely negating Speedfusion and all the bonding/resiliency benefits available with it. Once I do that though, I’m able to at least maintain and mostly stable (as stable as cellular gets) connection out; only relying on the single AT&T line. If I disable Speedfusion, it seems that my high packet loss connections clean right up and stabilize.

One last bit, I have two sites where I am able to use Verizon because it doesn’t experience the packet loss seen elsewhere. The same applies with my WiFi, I have two sites where WiFi is working well (both backed by Comcast residential like the others). If it were all sites being consistent, I’d be pointing at something in my configurations, but these are all set the same so I’m at a bit of a loss as to what’s going on. On the Verizon side, I have my Rep working with Tech Support trying to find something, but as of yet nothing. I have also cycled through the bands thinking I had hit a congested cell, but I get the same results on all bands.

Has anyone experienced this before? I’m at a bit of a loss as this is spread out across my market, but with such quality signals (and good WiFi in most places too) we shouldn’t be seeing this issue. I’m open to pretty much any idea to try to mitigate this. Thanks all for your help.

2

Sorry to ask the obvious but have you checked your antenna mounting orientation and connections? Have you spoken with Verizon about possible tower maintenance? It seems odd that this just started happening if you haven’t changed anything at your end. Good that you have a ticket open and working with tech support.

3

No apologies necessary, at this point, anything’s an option. I did talk to Verizon Tier 2 looking for just that kind of thing, but they had nothing on the list. They restarted the gateway on the tower in question for one of my sites, but that had no affect either.

These are actually all new sites so I don’t have extended run times yet. The sites where Verizon is unusable, it’s always been unusable. COVID pushed us to deploy about 10 of these over the last 10 months so we had to do a lot of learning and configuration on the fly. I did include a snap of the current Verizon signal status if that helps anyone. Thanks again.

Verizon Signal
Verizon Signal722×163 5.07 KB

One other I forgot to mention, these are all Layer 2 Speedfusion tunnels. I’m going to setup another unit tonight for testing and see if I can replicate the issues using an L3 tunnel instead.

4

You’re never going to believe this, but it’s stupid. Turns out our “unlimited” lines have a hard cap at 22gig and go to 600k after that, which is news to all of us around here. The two Verizon sites that are working well have grandfathered SIMs so they aren’t getting cutoff. Our testing prior to deployment sucked up all the data for the month before the new sites were ever deployed, hence constant Verizon problems on deployment. The packet loss is exactly what we would expect trying to shove normal data through a limited pipe. That’s a wasted week trying to diagnose something that is doing exactly what it should be doing… Thanks for looking at this and for the assistance.

1 Like
5

I realize that you found the problem but I note that your letting Verizon aggregate band 5 in the mix. I have finally turned band 5 off as it dragged the connection down terribly. I aggregate two band 66s and my bandwidth, throughput sky-rocketed with very low latency and minimal packet re-transmission (no loss).

Since your router finds band 66, I would turn off bands 13 and 5 and do some testing. Just MHO.

Very little of Verizon band 5 goes to LTE. It is mostly to service 3G. Also, band 66 is mostly allocated in 20x20 blocks for AT&T, 10x10 for Verizon. Where you can get band 66, it is usually less saturated since most customer’s routers will choose 13 over 66. Anytime a router chooses 66 (superset of 4) over band 4 itself, is a good situation - the flexibility to configure bands with the Peplink is very useful.

Have you tried this?

One more question: What antennas are you using to get such high SINR readings? Nice!

6

Good to know on the Band 66 Front, I will give that a shot. I did cycle through the bands before we found the data throttling but I haven’t tried it post throttling. We’re trying to figure out a long term solution to the data throttling so in the interim, I’m trying to be very stingy with my testing to save as much data as possible for real use. Anything I can do to help provide stability on the links will be very useful though as I’m sending real-time SRT video across the links which can absorb some loss, but not huge amounts of it without super high latency which doesn’t work in my use case.

As for the antenna I’m using for that site, It’s the Puma 401. We bought everything with the expectation of installing these in vehicles, but they got pressed into service as home access points due to COVID. Even inside though, I’ve had great results with them. I just strap them to a tripod and away we go. This specific site has the luxury of being in the 2nd floor of a massive 1880’s vintage home near a window so it probably has nearly line of sight to the local cell transmitter.

7

Thanks for the reply. I have a couple of Puma 401s on the way. I’m 2.5 miles from the AT&T tower, line of sight, and just under 10 miles from the Verizon tower, barely over a ridge. I’m currently using both Panorama directionals, 2x2 MIMO, and Poynting 5G directionals, 2x2 MIMO. I’m quite excited to see if the Pumas will improve my signals, particularly Verizon. I aggregate 2xband 2 on AT&T and 2xband 66 on Verizon. There are a few more antennas around me and closer, but over ridges where it’s difficult to aim the directionals. I’m quite excited to see if the omni Pumas can improve my SINR and RSRQ metrics and give me some other band options (although I’m quite stable with 2 and 66). Although I’m quite geared up to aim antennas without much fuss, since I move my antennas from house to RV, I’ll be happy if the omnis perform well - I can also maintain cellular internet while moving if they do.
Cheers.