AP's randomly going offline in IC2 affecting many users

Firstly, this issue is having a major impact on our live environment and affecting many users.

We’ve been using Incontrol2 to manage our Peplink devices for some time now. But few days ago, all my Pepwave AP One devices connections to IC2 at one of our clients locations have been randomly going offline & online, while the actual device is up and running physically and can be connected to. So when IC2 shows the AP offline, you can connect to the wireless network on one of the AP’s, but not ping 8.8.8.8. All AP’s are affected by the issue, but not all at the same time. Sometimes there are up to 4/7 of the AP’s offline at once. Some times just one. Rarely are all AP’s showing online in IC2 at the same time.

No changes had been to the network prior to seeing this issue. I did update the firmware for all the AP’s to the latest 3.5.3, but that has only seemed to make the issue worse. Please advise. I called into tech support for a down system and alsoopened a support ticket. Please advise. Thanks.

Please disregard.

We have resolved the issue. Unbeknownst to myself, a change had been made on the network. Once that change had been removed, the issue ceased.

It was hard for me to pin down the issue because not all of the AP’s would go offline at the same time. The behavior of the AP’s was completely random, almost like a rolling power outage. Which made me think it originally might be some type of PoE issue.
Out of curiosity, can anybody think of a reason why a network change would cause such an effect? I can understand if all the AP’s suddenly went offline, but the behavior the AP’s displayed really stumped me.

Not sure what changes have been done on the network. Anyway, Network related changes will cause some of the APs lost communication with InControl2 (Again this is related to the changes have been made). Below is the communication between APs and InControl2.

APs —UDP 5246—> Network Infra —UDP 5246—> Internet —UDP 5246—> InControl2

Hope this help.