Date: 2024-10-09
Time: Since 06:20 UTC
Issue: Devices on IC2 are falsely appearing offline
The IC2 Cloud Engineering team are looking at the issue.
We apologize for any inconvenience caused.
2 Likes
Thanks for the update. I was panicking for a bit there, but I could also see tunnels up on routers/appliances marked as offline in IC2!
The issue has been identified and resolved. Devices have been coming back online gradually since 07:22 UTC. The process will take about 60 minutes additionally.
The root cause was that one of the memory cache servers was intermittently unresponsive. However, it didn’t trigger any monitoring alarms. We will ensure the same issue can be identified and fixed timely in the future.
9 Likes
Date: 2024-10-10
Time: Around 02:26 UTC
Issue: Devices on IC2 are falsely appearing offline
The IC2 Cloud Engineering team is looking at the issue.
Our apologies for any inconvenience caused.
3 Likes
[Update]
The issue is resolved and the devices are back online at 04:09 UTC.
2 Likes