[Incident Update] IC2/Mars - Actions and Improvements

Announcements
1

November 12, 2018 InControl 2 Incident Update 2 - Actions and Improvements

We want to make sure customers have singular control over their devices and have safeguards to protect this. We also want to provide visibility so you understand what options and controls you have to operate your networks appropriately for each customer. We will be taking the following actions to restore this control:

  • Improving IC2 Device Change Rules -

    • We are reviewing the logic of how and when IC2 makes changes to a device so that only changes that are initiated immediately by a user, or to support a feature enabled by a user.
    • This will also include logging of what triggered each change and also tracking of what change is made.

  • Expansion and Creation of Additional Planets (server clusters)

    • Earth will be expanded with additional instances to allow customers to migrate to this more conservative upgrade schedule
    • We will populate a beta planet with real-world deployments and configurations to expand stress testing capabilities
      • We will invite partners to create networks inside of this planet for better testing and feedback

  • IC2 Device Configuration Lock (Available NOW)

    • There is a current feature at Group level to lock all devices from any IC2 configuration
    • It can be switched on and off to allow for authorized users to make changes and lock any further changes from being made after they are finished
    • This feature was 100% effective in preventing changes during Monday’s event
    • We realize this feature was not well documented or communicated, we have created a FAQ to help you understand and implement this (Click here for instructions)
    • We plan to relocate this feature so it is more visible and user friendly

  • Device Level Configuration Lock

    • We are planning a device firmware level feature that can be enabled to only allow 1 way reporting to IC2
    • This provides absolute certainty that no changes could be made to your most precious devices without a user logging into the router and disabling the feature

We welcome your feedback in shaping how these changes are made, as well as participating in testing them as they become available. Open dialogue and responsiveness has brought us great customers like you - these improvements will only be successful if they are treated with the same approach.

4 Likes
2

Please consider adding an email notification to critical users of your products. This little “improvement” of yours took down our Mobile Emergency Operations Center. We were called out for two missions a Hostage Situation and a missing person. Having a critical asset down due to an unknown issue is aggravating. Thankfully our representatives at Frontier had knowledge of this issue, but it would have been much better if Peplink would have sent us an email immediately so we knew there was an issue. I have additional devices we will now have to check the device names, speedfusion links, and other settings to determine whether or not they were affected. I tried to reply to your other post but it was locked and shown as solved. I can tell you that as of yesterday December 18, 2018 we were still being affected by this problem.

3

Hello @EM724,

We apologize for the interruption that the upgrade caused. To help prevent this from happening again we will be deploying a new IC2 instance named Venus. We have a group of partners and customers that have volunteered to have equipment moved to the Venus instance to aid in more extensive testing. Earth and Mars will only receive updates if no issues are found on Venus.

Steps can also be taken to prevent IC2 from pushing down changes to your devices. We have a post in the forum here that covers how you can enable and disable this feature.

Please let us know if you have any other concerns that you have about this process.

Thanks,
Zach

2 Likes