[Incident Update] PepVPN / SpeedFusion and InControl (MARS) issues


is there an ETA on MARS?


Jason - we’ll help get any units back up once the roll-back is completed. These will have to be logged into and updated on each router, or download the config backup and update each router that way.

How many of your devices are impacted?


Roll-back is in progress. Latest info I have is 15-30 minutes.


Agreed on Fridays/weekends vs. Mondays, and also very strongly agreed on better notification practices.


Ideally a beta system as well. Unfortunately this caught us in a bad time when we are thinking of buying 1,500 of your PepWave devices to replace Cradlepoints.

This scares me quite a bit and quite candidly has me wondering about pushing the PepWave devices.


No! There are no good times, and for us Friday and Saturday evenings are the worst. We have 350 Pizza restaurants and that is growing fast. So, for us, the very worst time is Friday from 4 PM to 8 PM, across EST/CST/MST/PST (i.e. 4:00 PM EST to 11:00 PM EST).
Changes like this should be done Saturday night into Sunday night, late in the evening.
But more importantly, a method of doing more live testing before full rollout is needed!
Perhaps instead of private IC getting updates last, private IC users (we use both public and private) could be used as beta testers. We do full VM backups every night, so we could easily do a backup, allow the update and roll back the entire VM if it all goes to crap.


Actually we have a couple of IC2 instances in the cloud.

Customers on the MARS (get latest release as it rolls out) instance are currently being impacted.

Customers on the EARTH (get latest release a month later than MARS) instance are not impacted.

Beta customers are on the PLUTO instance.


Also, we have six 24x7 critical customers, including three E-911 centers! I mention that to reiterate that there are no good times for updates. What is dead time for customer A is peak time for customer B.


how do we move to Earth?


How can I switch to the Earth instance?


How do we move to Earth ? (yes, I know this question seems odd written like this…)


We will certainly provide the details on moving to Earth in a day or two.


That means also you did not saw this issue on the Pluto users ?


Good point. I would be happy to move my internal testing devices to Pluto to better find these issues. However my customer units MUST be on Earth. This should be a choice.


Hello Team, curious if whether or not Privately-hosted InControl2 Appliances would in any way have been effected by this update. I had this question come up by a client is why I am asking.


Correct. This ran for 2 weeks on pluto.


No, any privately hosted IC2 is not impacted. Those are ONLY updated by the person hosting it.


The IC2 2.8 version is also not yet released for the self hosted version, so it is not possible to be impacted by this.


This is noted and I agree with you completely.


Still having inconsistencies in InControl2.