Hot Failover - not failing over fast enough

Hi

I am currently experimenting with a hot failover scenario from a wired 1Gig into the router on WAN1, failing over to two bonded cellulars.

I have not been able to get a quick failover without dropped packets. Anything I try, it takes approximately 8 seconds, and during those 8 seconds, a complete loss of stream.
The Google meet call, do not disconnect, so the session stays persistent, however I lose about 8 seconds before the connection picks back up.

I am running this in an events space, with Google Meet on Google Chromebox.

The test I am running, is spinning up a Google meet call, then monitoring it on the far end with my laptop. As I play a video (yes, yes it is Big Buck Bunny, obviously) I unplug the wired cable in WAN 1, and watch the call stop for 8 seconds, and wait for the SpeedFusion VPN tab in status to show me that the connection has failed over.

Settings:
WAN connection status all in Priority 1 on dashboard (screenshot attached)
WAN 1 - wired 1Gig line
Cellular 1 - set to LTE
Cellular 2 - set to LTE

I have set up SFC with two sub-tunnels.
One tunnel for stream smoothing and another for the failover, which is bonding two cellulars.
Images attached so you can see setup specifics.

Outbound policy setup is attached as a screenshot.
I have a google meet chromebox setup to handle the meet call for remote participants. SFC Tunnel 1 is in priority with tunnel 2 as backup
The rest of the data is routed via WAN1, as I don’t want any other traffic using SFC data.

WAN 1 health check settings screenshot attached. I have tried setting these to 1 second and I get the same result

I have also tried routing the traffic via Speedfusion VPN via the “Send all traffic to” selection and selecting the SFC I have created. Then changing the Speedfusion VPN settings Link Failure Detection time to Extreme, and I still get the same result.

Could someone help to take a look through to see what I may be doing wrong here?

Balance 310 5G

Many thanks,

(Apologies if this is in the wrong channel - let me know if this needs moving)








Move your rule above the grey bar and try again.

Also check the active sessions and confirm the traffic is indeed going out the SFC-LON interface.

Hi Jonathan,

Thanks for the reply

I’ll test this and come back

I currently have the option in Speedfusion VPN de-selected for routing SFC via the VPN, so would this make any difference?

Attached screenshots from the test, after moving SFC Stream above the grey bar - still the same result of approximately 8 seconds to fail over

The transition back to WAN1 when I re-connect is seamless however


on the outbound policy are you using tunnel 1 of the stream smoothing?
If so why do you have cell1 and cell 2 in priority 2 instead of priority 1?

Can you post a screenshot of what you are referring to?

Can you post a screenshot of:
Your new outbould policies?
The rule you configured in detail so we can see all the details?

It seems you haven’t changed the Link Failure Detection Time.

It’s under Advanced → SpeedFusion VPN → SpeedFusion VPN Settings.

By default it’s at 15 seconds, you can raise it up to Extreme, but first try Faster, if it’s fast enough for you. Every step faster will add an overhead to the max possible speed of your SpeedFusion tunnel, so you might lose throughput when set to Extreme.

@Jonathan_Pitts - Yes I believe I am using tunnel 1, I have confirmed this by running the test in the previous screen shots showing WAN1 active with cell 1 & 2 in standby, by looking at the SpeedFusion VPN and seeing data throughput on the connections.

I have attached a screenshot of the selection in Advanced > SpeedFusion VPN > Send all traffic to

The two screenshots I sent initially that show the two outbound policies, should be what you need? Apologies if not, let me know more specifically what you are looking for and ill send here.

Policies;

  1. Default all on WAN
  2. SFC Stream

@Jonathan_Pitts - attached below “the send all traffic to”

@padaco-daniel - thanks for your reply
I have tried enabling send all traffic to + changing the link failure detection time to extreme before, and I still get the same result of approx. 8 seconds.
I will attempt this again and re-establish the tunnels to see if that helps.

Hi all

Unfortunately I am still getting the same result

I selected Send all traffic to - SFC in the SpeedFusion VPN page, and set the link failure detection time to extreme, and was getting 6-8 seconds of failover.

I also tried dropping the SFC stream outbound policy back under the SpeedFusion VPN grey bar, and still the same result.

I am still not able to re-produce a 1 second failover

I wonder if it is due to latency on the cellulars? Not being able to ramp up quick enough?

What peplnk support say ? i used to have a very similar issue with a live stream going trought speefusion , when a link fails / degradated , live stream got chunky , pixelated or blackout

I see you modify wan smothing Maximum Level on the Same Link . leave it default and also go from medium to normal

Hi Salvador

I am still waiting for a response from them

To be honest, I think this may be the best case scenario for hot failover, before bonding which allows for 0 pixelation.
I have seen a comment from Martin describing that the packets are not dropped, however the stream can distort in this way when using hot failover.

Salvador - I have had confirmation that hot failover via SpeedFusion Connect (cloud) is not configurable as it has a different architecture, and runs at 6 seconds.
To get this down to sub 1 second, it is suggested to host FusionHub Solo.

Good to know, so can work around it, but not exactly transparent.

2 Likes

Ah, that’s why. We’ve been using FusionHub Solo for years in Vultr locations close to where the livestream is when we use it for hot fail-over/bonding.

1 Like

Hello, good morning, Harry. I hadn’t seen your message—lots of work on my end.

I hadn’t noticed that you’re creating a SpeedFusion VPN to SpeedFusion Connect instances. In my case, it’s between 310x devices at my client’s premises.

My setup was to enable only WAN Smoothing without FEC, and it has worked wonderfully. The only issue now is that, since there’s packet duplication, the links don’t have enough bandwidth. When one of the two WANs goes down, there’s a degradation in the streaming service. In these cases, we temporarily turn off WAN Smoothing, and the problem is resolved. Once the link is back up, we turn WAN Smoothing back on. This process does not renegotiate the SpeedFusion tunnel.

There’s some confusion here because Hot Failover doesn’t guarantee that the TCP/UDP frame won’t degrade, especially with VoIP/video services.

1 Like

Thanks for sharing Salvador - yes, it seems as though that is the case.

Good to know because I can now work around it, rather than thinking my set up was incorrect.

Hopefully they get the link failure detection down in the future, but my guess it is like this due to additional cost