All tunnels dropped after incontrol applied config

[ThomasV]

Hi Peplink,

It seems that just now, after incontrol applied an update all speedfusion tunnels dropped and do not come back up…

Is this a know issue

[Steve.Taylor]

Hi @ThomasV,

Thank you for the information, we are investigating and will report back shortly.

Steve

[B4BJonathan]

Not only this, but we get large batches of “changes applied” with no audit trail.
I initially suspected it was incontrol pushing new ssl certificates every 76 days, however upon checking the dates don’t tally up (unless it does it quite a bit in advance).
It’s extremely worrying seeing “changes applied” without any record of what is actually happening!!!

[yucelistik]

We also are having the same issue. The problem we’ve seen was central SF termination device’s local ID has been changed by InControl update push, it was “router-name(str) + serial number(str)” and all tunnels are up after changing it back to its original name. May be this would be helpful for investigating the code.

[Lai]

We are investigating on it and will reply yours soon.

[ThomasV]

Nice catch Yucelistik! That solved one of our problems. SF tunnels are back up.

However not all devices show as Online in Incontrol.

[sandor]

[Incident Update] PepVPN / SpeedFusion and InControl (MARS) issues

November 12, 2018 InControl 2 Incident Update - 11/14/2018 We want to provide all customers impacted by Monday’s issues an update on what went wrong. We will provide an additional update to provide you with a detail plan to resolve and prevent these issues from happening again. What Issues Were Seen: Accounts on the mars server cluster were inaccessible to users IC2 mistakenly reconfigured VPN profiles it was not actively managing Some PepVPN/SpeedFusion tunnels were taken offline What Acti…

This issue impacted all our device and customer services. Such a drastic configuration change should never just be automatically pushed by InControl to all devices. We have disabled InControl for all our devices now and will not re-enable again.

[jflanigan]

We are still down. How do we get the PepVPN/SpeedFusion profiles online again?

[Steve.Taylor]

Hi @jflanigan,

Please check this post : > Here < - we’ll update customer via that post.

Thanks,

Steve

[SPeedY]

All all the days, this was the worst possible day for this to happen. A long Holiday, my admins out of the office and how good is redundancy when you guys are going to shutoff our VPN!!!
Ohh well, the manual trick did the job, but took me a ridiculous 4 hours, if you ask how, try doing it from your phone and PepLink InControl overriding your changes and low signal.