Randomly LAN Ports stop passing data/not communicating (Generally every several weeks), Requires Reboot to clear issue

Bit of backstory, We are an MSP that have about 37 Peplink Balance One’s (and Balance One Core’s). Generally install for most of our rural small businesses, SpeedFusion is great for bonding WAN for LTE + Coaxal/Starlink/Fixed Wireless and etc. Products are great, and generally very stable, anytime we have an ISP trying to blame Customers Hardware as the issue, we can pull up the WAN Reports and show exactly high latency points or complete outages and every time we would have the ISP replace their hardware or fix the line and customer stops having issues (besides the occasional outage here or there). So we have continued to addon Peplinks for their reliability.

However we started experiencing strange LAN issues in the past year, and seem to be more frequent lately…

Some Clients had it where it would occur few times a month, while some after about a two months (and some have theirs on for over a year and is still fine), where the LAN Ports just stop passing through data… but then after few times, it may work for half a year without issues.
I had mine do the same thing today (and on weekend) so I could take some time to play around with it and see what the issues could be.

When I say the data stopped passing through, its as if the whole port was just blocked, I cannot ping the Peplink Gateway, or load anything on the internet, I could ping other devices connected (through Unifi Switch), If I unplugged and replugged in devices (they lost their dhcp IP and would not get a new one).

The Peplink did connect to Incontrol2, so on my iPad I could remote into it, and would show 0 Clients, and if going to client page, all the devices were grayed out (likely they are offline). But checking Ethernet ports on Status Page, it was lit green, and on the Peplink itself, all of the ports where lit up like normal…

I have tried to mess with port settings, (have one computer connected to port 4) and Unifi Switch connected to Port 8, tried what someone mentioned was possibly auto negotiate for port wasn’t functioning properly so I set to 1 Gbps Full Duplex and saved settings, it did nothing, then added 100Mbps Half Duplex on port 2 and tried plugging something in, still nothing, then played with another port, and then after saving, everything seemed to pop back up and all the devices on all ports started to reconnect.

However, it seems to still auto negotiate.

Even rebooting it keeps Auto Negotiate on and doesn’t follow what I set under port settings for speed.

Clients that have the issue vs the ones that don’t, don’t really have much difference, they are set up mostly the same. (Most devices are up to date, either 8.4.0 or 8.3.0) but the issue has been ongoing through multiple firmware versions. I thought of temperature, but most places are about the same, and ones with same model and firmware that work fine have been in hotter rooms.

We add on customers and even new ones with clean configuration will have issues, I cannot find any correlation. But here is the other issue, most times when we notice an issue, it’s when the client is using their devices, so we need to reboot and try to get it back up asap and cannot go and investigate it. We may take a few minutes to look around, but that’s about it. It’s rare when we get one to do this when we can take the time to be onsite and try to diagnose the issue, and if we try to open the ticket, likely will be difficult for Peplink team to investigate it when its different random devices that may have issue once a week between all the devices. I would say between those 37 or so Peplink Balance One and One Core’s, this occurs about once a week, where it becomes a real annoyance.

We did start integrating with Peplink API’s to see if device reports 0 clients and auto reboot them, but this isn’t a long-term solution we’re looking for…

I want to see if anyone else had this issue and may know what it may be? Or possibly suggestions on what to look for?

Most of these units are on UPS Battery Backup (so steady power flow), Have a rack mount with airflow holes below and space around so no additional heat from other devices, tried clean resets and resetting up the whole device on different firmware…

So far, the only thing I can reproduce easily, is attempting to change LAN Speeds and they don’t seem to take affect… (If I reboot to 8.3.0 and try changing and saving it, it reboots back to 8.4.0 and doesn’t do anything)

Thanks!