Dear All,
the overall speed of my speedfusion tunel is slower than the bonded fastest connection.
So that it is not superior to a hotfailover.
My configuration:
See screenshot. I hope it is self-explaining:
Strange that there are so many abadonned threads on this. I hope someone has a solution idea.
There is a lot going on in that graph.
Lets work our way through it.
Download test #1. Cell1 and Cell2
1.~25MBps download. Cell1 doing most of the work. Cell 2 throughput is low maybe ~5mbps.
2. When combined like this latency is managed and looks reasonable <~100ms
3. Cellular 2 is lossy touching on 50 packets/sec and Cellular1 has a noticeable amount also maybe 25pkt/s
Download Test #2 - Just Cellular 1 (CAT12)
4. Loads of bandwidth ~90+Mbps
5. Latency stays low <100ms
6. Some packet loss but its minimal
Download test #3 just cellular 2 (CAT4)
7. 8-10Mbps?
8. Monster amounts of latency! Over a second! Worse than Satellite.
9. Tiny amount of packetloss at the beginning of the test - then nothing.
Upload Test #1 Combined
10. 30Mbps ish
11. Latency is fairly evenly poor - both connections must be creeping up to 200 - 250ms+
12. So much packet loss per second on both links!
Upload Test #2 Just Cellular 1 (CAT12)
13. Touching 45Mbps of bandwidth which is good
14. Latency isn’t great in the region of 200ms
15. Slight touch of packetloss
Upload Test #3 - Just Cellular2 (CAT4)
16. 30Mbps+ bandwidth
17. Latency is not good >250ms at times.
18. Chunk packet loss.
Some observations / recommendations.
Check antenna positions, make sure you have all antennas connected and separated well. Also what frequencies are each cellular connection using? Are they on the same bands? Are you using splitters or amplifiers maybe?
If you want to continue to work on this please open another new thread (you can copy and paste these two posts into the top). You have duplicate posted your question across a lot of the threads you found which makes things messy.
Thanks a lot for looking into the details.
I understand that if latency of the bonded connection is significantly different than it is not possible to reach higher speed than the faster connection in the bonding.
However celular connections cannot guarantee a healthy performance / latency permanently.
As consequence I see two use cases which I would like to solve:
Use Case 1 - Degradation of one connection: Avoid that the speed of speedfusion tunel gets lower than the faster connection
Use Case 2 - Both connections are healthy: Reach higher speed than the faster connection
I suggest dealing with these two use cases separetly.
Could we please focus on Use Case 1 first?
How could I avoid that the degradated connection couses degradation of the entire speedfusion tunel?
Your feedback has already gave me some ideas about suspending more agressively the degradated connection based on Latency Difference Cutoff. My new settings:
EndPoint:
I will do a fresh re-measurement overnight to exclude load of other users on the tower.
Once Use Case 1 is fixed, I would start to experiment with the radio performance for Use Case 2. Indeed cellular 1 (LTE12) has a much better external antenna than cellular 2 (LTE4).
cellular 1 (LTE12):
cellular 2 (LTE4):
As the SIM cards are from different providers and I do not use any splitters and amplifiers, I do not see how could they interfere on the frequencies … … but happy to look into that if there is any scientific explanation how an interference would be possible - so that I know what anomaly I should look for.
I look forward to seeing the results of your tests. Generally I tend to see the best results when using cellular links with very contrasting latency with latency diff cutoff set to 100-150ms.
With regards interference - keep a time log of your tests, then go to incontrol2 reports and screen grab those time periods for both cellular WANs. Seeing what happens to RSSI and RSRQ on one cellular link whilst you are testing the other will be of interest here I think.
Are we supposed to set cutoff values on the FusionHub? I had gone by this linked post and only set mine on the device. The theory being that the FusionHub only sees 1 connection - “Wan”.
30 ms cutoff was not consistently good - a tried many variations.
So I post here a measurement with 150 ms cutoff as you proposed.
Hub:
EndPoint:
This time I focus only on upload:
Latencies does not seem to be so different when both links are active. Why speeds are not getting added?
Not sure how to troubleshoot the root cause of the packet losses. Could it be that the Fusion Hub itself drops the packets?
Signal Quality - cellular 1 (LTE4):
Signal Quality - cellular 2 (LTE12):
Fusion Hub seems to see both endpoints with separate latency and paket loss values:
For sake of completeness here is a download measurement with 150 ms cutoff:
Please note that with Balance 20X the maximum is 100 MBit not encrypted and these values are close to that - so I would focus on the upload.
That is latency cut off doing its thing.
Latency cut off only affects sent traffic so you only need to do it on the multi-wan end not the fusionhub.
Try setting a hard upload limit on the speedfusion profile of 35Mbps what happens to this upload graph then? Does the upload limit reduce visible packetloss?
Thank you!
Yes, I think the upload limit reduces that packetloss:
I see the key question is where the “limit” which causes the packetloss comes from when there is no limit configured?
The limit will be the backhaul from the tower(s).
This type of packloss is something I see a lot of (but not all the time) and is typically a poorly configured mobile network. Its bufferbloat. You send traffic at a level that saturates the tower you are connected to and instead of just dropping your traffic (which would rate adapt TCP), the tower has a network buffer that it fills. So your latency rises (because your traffic is greatly delayed).
The two cellular connectionss are talking to two different ISPs and to two different towers which are located ca. 5 km from each other. I do not understand how the two could share the same backhaul.
Regarding bufferbloat, I made here a new masurement with cutoff 500 ms to remove the packetloss caused by cutoff-triggered suspension from the test.
It is visible that packetloss occurs in the cellular 1 (LTE12) only case. But there are at least two options for the root cause:
a.) the tunel reaches speed of 50 MBit
b.) LTE12 reaches speed of 50 MBit
How could we figure out if the cause is a.) or b.)?
If it is a.), then it might be tower bufferbloat, then it would be nice to cap the cellular 1 at 45 MBit in the router.
If it is b.), then it has to be other compents like the Peplink 20x it self or the SpeedFusion Solo in AWS.
Just run a test between my LTE12 modem (MikroTik) and my MikroTik server in the same AWS cloud where my PepLink FusionHub Solo is deployed as well:
It shows zero lost packet.
From the facts collected, I assume that the components participating in this recent test are not responsible for the packet loss on the Cellular 1 (LTE12). So neither my LTE12 modem, nor AWS, nor the ISP seem to be responsible for the packet loss. Where else it can come from?
Interesting. Do the same thing using the WAN analysis tools on the Peplink (System > Tools | Wan Analysis) and post the results.
I noticed the test above was conducted using TCP instead of UDP, have you tried running Peplink in “Experimental” TCP mode?
I noticed the test above was conducted using TCP instead of UDP, have you tried running Peplink in “Experimental” TCP mode?
Yes, I will try it in UDP as well.
The “Experimental” TCP mode did not work for me - I just tried. Probably I would need to open a suitable port on the AWS firewall…
The result of WAN Analysis:
Zero packet loss at 50 MBps upload.
I did some further tests:
Lets ask @Steve if he has any idea why the Speedfusion graph shows packetloss when the CAT12 modem is the only WAN in use, but p2p wan analysis using that WAN against the same Fusionhub doesn’t.
Also - the latest firmware (8.1.0RC1) has much better speedfusion graphing and is probably worth installing at both ends. Firmware 8.1.0 RC 1
Thank you! I am usually willing to be the early adopter of any new firmware. The reason why I refraine to do so:
I am looking forward to read ideas from @Steve .
I also moved this finding as a dedicated topic itself: Does Speedfusion overdrive WAN connection and causes packetloss?
On this orginal topic, I will try to make the latencies of the two LTEs more similar by increasing radio performance. I hope that I will see some speed bonding after that.
Just realised I have never asked what Peplink device you are using. What is it?
Balance 20X - I mentioned it earlier in the context that tunel bandwidth is limited at 100 MBps.
