[Incident Update] PepVPN / SpeedFusion and InControl (MARS) issues


#41

Removing our Peplinks from InControl seems to have resolved the issue for us. Hopefully Peplink will resolve this issue soon as it is affecting all of their customers that rely on Peplink cloud services. Thank you,


#42

The team is working on restoring the service as we speak. We will post an update in 10-15 mins. We are very sorry for the down time.


#43

We have updated the initial post on this thread with further info and recommendations.


#44

Will this also resolve the issue with nested tunnels?


#45

Please open a ticket with your S/N so we can further investigate.


#46

@Steve.Taylor So if i understand correctly, if we have InControl manage the pepvpn for us then this issue will go away? – given that we can reach the necesary InControl pages.


#47

You can also download the most recent (or a few recent) configs from the IC2 backup, and re-apply that config to restore your sub/nested tunnels.


#48

I tried using IC2 to setup the PepVPNs… it seems like the PepVPNs are staying in starting state…


#49

Engineering has implemented a fix to roll-back the Device IDs that were changed.

This is rolling through the impacted devices and may take around 1 hour. Users will see this update incrementally - many are already updated and fixed.


#50

When are the IC2 performance issues going to be fixed?

I can’t even access my devices!


#51

I have 141 devices in IC2 that are not visible all day long. When is this solved?


#52

We are patching the device IDs first, and then we will roll back IC2 to version 2.7.3, which will restore the performance. We have to do it in this order, unfortunately.

You can continue to refresh pages and will sometimes be able to view/manage.

The device ID update is moving quickly, so hopefully we can start addressing the performance very soon.


#53

Is there anywhere that gives us info on future patches? I am not going to lie this is incredibly concerning to me since we rely on having nearly 100% uptime on IC2.


#54

Can you help me understand by what you mean by rollback? We applied the fix to the PepVPN names and it was working. Now the names were changed back and i has brought down our business and our customers again! Peplink has been extremely reliable up to this point but this is not how I would expect an established business to operate.


#55

We are rolling our InControl platform back - version 2.8 was implemented last night, there was a feature in this update that caused this mess. After cleaning as much of that up as we can, we are now rolling IC2 mars back to the version that was running up to this point - 2.7.3


#56

Ok, and are you’re aware that at this time we don’t have PepVPN connections working? The workaround did allow us to establish a connection, but something during this process broke that fix.


#57

I don’t believe notification was sent out for this change. Additionally, a platform change on a Monday morning never goes well. Can these changes be communicated and planned for Friday night/over the weekend?


#58

Can we get upfront notice on upcoming updates so we can prepare for this? Perhaps a list of what is potentially changing? Any way we can be volunteer to have a device or two in a ‘beta’ channel/org or something?

I am working with a giant client with over a thousand end points and we are working on switching to PepWave, they are not aware of this change, but I will be perfectly honest this heavily concerns me.


#59

Sorry, over a thousand routers, currently Cradlepoints.


#60

You can bet that we will review and implement a much more stringent release mechanism after this is over. We can only win back the trust of customers after proven actions.