Randomly LAN Ports stop passing data/not communicating (Generally every several weeks), Requires Reboot to clear issue

Bit of backstory, We are an MSP that have about 37 Peplink Balance One’s (and Balance One Core’s). Generally install for most of our rural small businesses, SpeedFusion is great for bonding WAN for LTE + Coaxal/Starlink/Fixed Wireless and etc. Products are great, and generally very stable, anytime we have an ISP trying to blame Customers Hardware as the issue, we can pull up the WAN Reports and show exactly high latency points or complete outages and every time we would have the ISP replace their hardware or fix the line and customer stops having issues (besides the occasional outage here or there). So we have continued to addon Peplinks for their reliability.

However we started experiencing strange LAN issues in the past year, and seem to be more frequent lately…

Some Clients had it where it would occur few times a month, while some after about a two months (and some have theirs on for over a year and is still fine), where the LAN Ports just stop passing through data… but then after few times, it may work for half a year without issues.
I had mine do the same thing today (and on weekend) so I could take some time to play around with it and see what the issues could be.

When I say the data stopped passing through, its as if the whole port was just blocked, I cannot ping the Peplink Gateway, or load anything on the internet, I could ping other devices connected (through Unifi Switch), If I unplugged and replugged in devices (they lost their dhcp IP and would not get a new one).

The Peplink did connect to Incontrol2, so on my iPad I could remote into it, and would show 0 Clients, and if going to client page, all the devices were grayed out (likely they are offline). But checking Ethernet ports on Status Page, it was lit green, and on the Peplink itself, all of the ports where lit up like normal…

I have tried to mess with port settings, (have one computer connected to port 4) and Unifi Switch connected to Port 8, tried what someone mentioned was possibly auto negotiate for port wasn’t functioning properly so I set to 1 Gbps Full Duplex and saved settings, it did nothing, then added 100Mbps Half Duplex on port 2 and tried plugging something in, still nothing, then played with another port, and then after saving, everything seemed to pop back up and all the devices on all ports started to reconnect.

However, it seems to still auto negotiate.

Even rebooting it keeps Auto Negotiate on and doesn’t follow what I set under port settings for speed.

Clients that have the issue vs the ones that don’t, don’t really have much difference, they are set up mostly the same. (Most devices are up to date, either 8.4.0 or 8.3.0) but the issue has been ongoing through multiple firmware versions. I thought of temperature, but most places are about the same, and ones with same model and firmware that work fine have been in hotter rooms.

We add on customers and even new ones with clean configuration will have issues, I cannot find any correlation. But here is the other issue, most times when we notice an issue, it’s when the client is using their devices, so we need to reboot and try to get it back up asap and cannot go and investigate it. We may take a few minutes to look around, but that’s about it. It’s rare when we get one to do this when we can take the time to be onsite and try to diagnose the issue, and if we try to open the ticket, likely will be difficult for Peplink team to investigate it when its different random devices that may have issue once a week between all the devices. I would say between those 37 or so Peplink Balance One and One Core’s, this occurs about once a week, where it becomes a real annoyance.

We did start integrating with Peplink API’s to see if device reports 0 clients and auto reboot them, but this isn’t a long-term solution we’re looking for…

I want to see if anyone else had this issue and may know what it may be? Or possibly suggestions on what to look for?

Most of these units are on UPS Battery Backup (so steady power flow), Have a rack mount with airflow holes below and space around so no additional heat from other devices, tried clean resets and resetting up the whole device on different firmware…

So far, the only thing I can reproduce easily, is attempting to change LAN Speeds and they don’t seem to take affect… (If I reboot to 8.3.0 and try changing and saving it, it reboots back to 8.4.0 and doesn’t do anything)

Thanks!

Hi

I got this problem too. I think it’s a hardware problem.

It happening “every several weeks” and having to do with LAN ports sounds like this issue: DNS Proxy bug in firmware 8.5.1 - #6 by soylentgreen

My experience suggests it’s software, because it didn’t happen for years, but started happening sometime in the 8.x firmware releases.

In any case, I’ve since upgraded from a Balance One to a B One. With the new B One I have seen an apparent DNS proxy bug, but only upon reboot, and that can be fixed by toggling DNS.

Once the B One is working properly, I have not seen the “every few weeks” problem that was happening on the Balance One.

It seems to only really affect units in warmer network rooms. I think its a heat issue. During cooler months they work without issues, summer hits and room is warmer and those units will stop communicating with any LAN Devices, completely 0 clients and unreachable via LAN but will connect to internet and incontrol just fine. We have been swapping them to newer B One units and issues disappeared. I’m leaning to bad cooling design on Balance One’s, as those units do get hot on certain spots on the unit, and only warmer network rooms seem to be the one that had the most trouble. Those rooms, I wouldn’t say get hot enough where I think it should be a problem and we have no problems with any other devices or once swapped out with B One’s. We already got most of them swapped out only few remain.

1 Like