[Incident Update] PepVPN / SpeedFusion and InControl (MARS) issues


#81

Is the downgrade complete? I am still not able to view the overview page of IC2.
Also, some pepvpns build via IC2 are still not up yet.


#82

Keith, when you have time, reach out to me about testing with private IC2. We would be happy to spin up a second instance and put enough customers on it to ensure real testing. As you know, we are VERY good beta testers and will put it through it’s paces.
so we could do initial test on small private IC2, then upgrade main private IC2 a week later etc.


#83

i’m still down as well.


#84

Completely down this time, it appears to not find the groups or any devices at all. I think it is currently downgrading, getting a 500 database error from time to time as well.


#85

I now have no devices showing up, but it is showing up with the speed it use to have.


#86

Work is still in progress…


#87

Will the devices then configurable through IC2 again? Luckily we had a backup IPSec tunnel to reconfigure the core devices to serial number again, but what about the mobile (MAX) ones then?


#88

any updates? Still can’t access devices, connect via IC2 based PepVPN, or reconfigure PepVPN on local devices since i can’t reach them.


#89

I need an update as well.


#90

The IC2 team continues to work to resolve an out of memory issue that is preventing Mars from loading properly. I wish I had more information.

The team is still 100% working on this.


#91

@Travis Any updates?


#92

Unfortunately, not much progress yet.


#93

Thanks for the update @Travis.
I am getting worried since we have SLAs and our clients have been down all day.


#94

We have positive signs. Please check your IC2 orgs and report back.


#95

Starting to get remote access to devices.
PepVPNs that were made in IC are still not 100%
I might need help on these.


#96

Yes, I see all the devices and works


#97

Glad to hear. Thanks and obviously very sorry for the prolonged issue.


#98

it seems to be working properly but we will continue monitoring and if we find any additional issue we will notice you guys i would like to know till when you will leave this forum open ?


#99

No plans to close this until we have confidence everyone is working. We will have plenty more information to share with everyone as the dust settles.


#100

@Travis
All devices looking good now!