Low-Bandwidth connections over speedfusion

Product Discussion Pepwave MAX
1

Hello,

I have the issue that some connections are very slow over speedfusion. Here is the issue:

scp testfile.dat foo@testbed-1:/dev/null
testfile.dat                                       10%  105MB   4.6MB/s   03:20 ETA^

However, on that device, I have reasonable (albeit still very slow) speed with iperf3:

$ iperf3 -c speedtest.init7.net 
Connecting to host speedtest.init7.net, port 5201
[  5] local ... port 60130 connected to ... port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  8.57 MBytes  71.9 Mbits/sec    0    511 KBytes       
[  5]   1.00-2.00   sec  9.06 MBytes  76.0 Mbits/sec   28    326 KBytes       
[  5]   2.00-3.00   sec  7.39 MBytes  62.0 Mbits/sec    0    351 KBytes       
[  5]   3.00-4.00   sec  7.45 MBytes  62.5 Mbits/sec   23    187 KBytes       
[  5]   4.00-5.00   sec  8.93 MBytes  74.9 Mbits/sec    0    214 KBytes       
[  5]   5.00-6.00   sec  8.87 MBytes  74.4 Mbits/sec    9    178 KBytes       
[  5]   6.00-7.00   sec  8.87 MBytes  74.4 Mbits/sec    0    208 KBytes       
[  5]   7.00-8.00   sec  9.61 MBytes  80.6 Mbits/sec    0    237 KBytes       
[  5]   8.00-9.00   sec  8.87 MBytes  74.4 Mbits/sec    0    262 KBytes       
[  5]   9.00-10.00  sec  9.61 MBytes  80.6 Mbits/sec    0    285 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  87.2 MBytes  73.2 Mbits/sec   60             sender
[  5]   0.00-10.01  sec  85.3 MBytes  71.5 Mbits/sec                  receiver

What I’d expect it something closer to this:

backes@bk-loki:~/dev/tmp$ iperf3 -c speedtest.init7.net
Connecting to host speedtest.init7.net, port 5201
[  5] local ... port 35788 connected to ... port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  90.4 MBytes   758 Mbits/sec    0   3.07 MBytes       
[  5]   1.00-2.00   sec   103 MBytes   864 Mbits/sec    0   3.07 MBytes       
[  5]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0   3.07 MBytes       
[  5]   3.00-4.00   sec   109 MBytes   915 Mbits/sec    0   3.07 MBytes       
[  5]   4.00-5.00   sec   110 MBytes   926 Mbits/sec    0   3.07 MBytes       
[  5]   5.00-6.00   sec   103 MBytes   867 Mbits/sec    0   3.07 MBytes       
[  5]   6.00-7.00   sec   107 MBytes   898 Mbits/sec    0   3.07 MBytes       
[  5]   7.00-8.00   sec   108 MBytes   908 Mbits/sec    0   3.07 MBytes       
[  5]   8.00-9.00   sec   110 MBytes   924 Mbits/sec    0   3.07 MBytes       
[  5]   9.00-10.00  sec   104 MBytes   871 Mbits/sec   10   2.35 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.03 GBytes   887 Mbits/sec   10             sender
[  5]   0.00-10.01  sec  1.03 GBytes   885 Mbits/sec                  receiver

iperf Done.

This is done in the same WiFi network than what the peplink connects to.

I have a Peplink MAX BR1 Mini 5G. It is connect via speedfusion to a fusionhub located at the office in a hub-and-spoke setup. I have a NAT that maps the local network 192.168.68.0/24 to some other subnet 10.0.0.0/24. I tested many different VPN settings and all of them fail. These results above were achieved with:

  • No Encryption
  • No Forward Error Correction
  • No Smoothing
  • Bonding mode

I do not have other settings on the peplink. (No QoS or other traffic shaping, no port forwarding, no firewall, etc.). The outbound policy simply routes everything through the VPN.

I tried this with different outbound connections. Cellular only, WiFi only, combinations of WiFi and Cellular. All give me the same result.

In the office. I have a static route to route the traffic of 10.0.0.0/16 to the fusionhub server.

I suspected MTU issues and changed these (on the WiFi profiles) but to no avail. The MTU looks fine:

$ ping -M do -s 1472 ...
PING ... (...) 1472(1500) bytes of data.
1480 bytes from ...: icmp_seq=1 ttl=62 time=27.1 ms
1480 bytes from ...: icmp_seq=2 ttl=62 time=38.0 ms
1480 bytes from ...: icmp_seq=3 ttl=62 time=36.2 ms
1480 bytes from ...: icmp_seq=4 ttl=62 time=33.2 ms

I tested the speeds from the gateway router and the server where I host the fusionhub on, and both don’t show any bandwidth issues.

# iperf3 -c speedtest.init7.net
Connecting to host speedtest.init7.net, port 5201
[  5] local  ... port 35724 connected to ... port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1006 MBytes  8.43 Gbits/sec   32   1.58 MBytes       
[  5]   1.00-2.00   sec  1.07 GBytes  9.21 Gbits/sec    0   1.58 MBytes       
[  5]   2.00-3.00   sec  1.07 GBytes  9.23 Gbits/sec    0   1.59 MBytes       
^C[  5]   3.00-3.15   sec   171 MBytes  9.43 Gbits/sec    0   1.59 MBytes       

$ iperf3 -c speedtest.init7.net 
Connecting to host speedtest.init7.net, port 5201
[  5] local ... port 50826 connected to ... port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    865 KBytes       
[  5]   1.00-2.00   sec   111 MBytes   928 Mbits/sec    0    967 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   935 Mbits/sec    0    967 KBytes       
[  5]   3.00-4.00   sec   113 MBytes   945 Mbits/sec    0    967 KBytes

I also did a WAN Analysis as a client with the fusionhub as server:

  Overall:    165.4738 Mbps         0 retrans /   1688 KB cwnd

and the reverse:

  Overall:    273.0982 Mbps         1 retrans /   1031 KB cwnd

At this point, I am pretty confident that it is a configuration issue (the same happens on other peplinks), but I can’t figure out what the problem is. I found a similar issue where they experience the same 6Mb/s limit: Slow/throttled Speedfusion connection

Does anyone know how to either resolve this or further debug it?

2

Hi…
Have you look at Peplink Router Comparison | 5G, LTE & Multi-WAN for Business & Home - Peplink ?
BR1minihw3 (mini 5g)

for speedfusion

image
image424Ă—194 2.18 KB

Your numbers, under iperf3, going to SFC… I believe… are correct!

2 Likes
3

The issue isn’t the iperf3 result, but it’s that file transfer over scp and rsync doesn’t even reach close to this:

backes@bk-loki:~/dev/tmp$ scp testfile.dat loki@testbed-1:/dev/null
testfile.dat                                       10%  105MB   4.6MB/s   03:20 ETA^

I added the iperf3 result to show that not all of the traffic types is affected for some odd reason

4

Hi, @tbackes

Can you transfer a small file and at support page do a network capture?
Your file, need to be around 5Mb size.

5

Thanks for your answer. What exactly are you looking for? I do not feel comfortable sharing network captures this way, even if they’re from an ssh connection. I have a pcap file from the client and the server and can run any analysis on it that you think is useful.

Attached here is a Stevens Graph as seen from the client that is sending the 5mb dummy file over scp to the device behind the peplink.

image
image1604Ă—828 26.6 KB

I made some more measurements, this time only with the tools that peplink provides. This raises a lot more questions than what it answers:

  • Why does it degrade the latency so much when I do the TCP upload test over speedfusion?
  • Why are the results between the “Speedfusion VPN Test” so different than the WAN analysis? The VPN results are nowhere near the 80Mbps.
  • Why is download and upload not symmetric?
  • Why does UDP download perform worse than TCP download in the WAN analysis? I’d have expected either identical, or UDP to outperform TCP.

VPN speedtest from the peplink. Notice how much the latency degrades?

image
image780Ă—719 91.8 KB

UDP upload has the same bandwidth

image
image780Ă—719 76.7 KB

TCP download performs better than the upload part:

image
image780Ă—719 79.9 KB

UDP download is reaching the requested bandwidth:

image
image780Ă—719 86.6 KB

While this is already strange on its own, here are more strange measurements:
WAN analysis where the peplink is the server, and the fusionhub the client.

TCP upload

image
image793Ă—876 88.6 KB

UDP upload with requested 512.00 Mbps:

image
image793Ă—876 108 KB

UDP upload with 128.00 Mbps:

image
image793Ă—876 97.5 KB

UDP download with 128MBPs

image
image797Ă—866 92.6 KB

TCP download:

image
image797Ă—866 88.1 KB

6

just get a short video from the internet… and transfer it… I just want to understand what is happening at your network. Reasons to not get the full speed that you looking for.

There is no exactly answer. Can be the cpu? Can be something at internet path between both points? I don’t have an answer for you, about this.

Maybe doing a network capture, will help to understand what is change, when do one test and use for other things.

Have you check your internet signature with your ISP?

By default… tcp use more cpu… Strange the your results. Again, maybe a network capture can provide some anwers, maybe not.

I am not a Peplink developer!

Using WiFi for performance tests… Not is a good idea… Best to do this is using cable… WiFi always have some kind of RF issues.

8

So I have some thoughts on what might be happening.

Iperf is showing 75Mbits/s or 9.375 MBytes/s vs scp which is doing 4.6MB/s or 36.9Mbits/s .
Iperf3 will likely start with a big window size expecting a faster connection, and throttle the window to the maximum that is posible which is around 75MB/s (the retrys early will have set the expectations).
SCP being a protocol to go over the internet likely starts with a small window size and with each success it will increase the window size (and amount of data it can send) but with the random latency and maybe a little packet loss you see the window size will likely grow slowly.

Did you let the SCP file copy finish and was the speed the same at the end as it was at 10%?
Did you try running simultaneous SCP file copies, each will have its own TCP session and as such TCP window so you will likely find that 2 together gets close to the iperf3 test.

For the speedtests you were doing on the peplink vs the fusionhub you were likely finding that generating the traffic on the br1 mini which will be CPU bound with 80Mb/s of throughput was meaning it was struggling to create and transmit that much traffic leading to odd results.

2 Likes
9

also regarding the MTU, when you were sending the 1480 packets by ping over the tunnel, the traffic gets fragmented going out of the WAN, you can see this if you turn on the DF bit setting in the tunnel profile. Then when you ping you will see it fail and you will need to reduce the packet size until you fine the largest size that can be sent without fragmentation.
At the same time if the WAN speed is greater than the speedfusion bandwidth (limited by the br1 mini to 80Mb/s) and packet loss is low I would imagine you will see no difference.
If you are seeing a bit of packet loss then enabling FEC on low (or actually adaptive) can really help with SCP and SMB file copies.

2 Likes