InControl2 - Remote Web Admin - Results in Error

hcardenas · November 3, 2020, 9:51am

Is it completely solved? Because we are still experiencing some intermittence (we get the message that engineers are solving the problem) and slowness in a few organizations.

Thanks!!
Héctor

Soriam-V · November 3, 2020, 10:56am

Dear Héctor,

We are looking into it at this very moment. Let me get some updates for you in a few minutes.

Thanks,

Soriam

Soriam-V · November 3, 2020, 1:04pm

Root cause analysis : 11/03/2020 approximately 1 pm Central time the database connections came to an alarming state causing DB connections to fail health checks and affecting DDNS, RWA, and Authentication Services on the Mars IC2 branch. Other branches were unaffected.
Fix applied: Services resumed after the restart.
Prevention : Added monitors to check for errors to alert us if this were to happen again.

hcardenas · November 5, 2020, 9:28am

Thanks @Soriam-V!

Soriam-V · November 5, 2020, 1:01pm

Dear Partners and Customers,

In the past 3 days some of you have noticed interruptions to the InControl2 System. Our team has narrowed down the issue to our IAAS provider and we have an open support case with the provider to investigate the cause. As a temporary workaround, we added monitoring and automated services to self restart based on health checks. This allows IC2 service to self-heal in approximately 10 minutes and very rarely needing manual intervention. We are continuing to work with the IaaS team to resolve this issue ASAP. We will keep you posted and thank you for your patience.