What causes surf soho to drop all 2.4 GHz clients

Product Discussion Pepwave AP & Surf
1

I used to run an Asus router/wifi AP for my setup. There were regular dropouts in the wifi links and on occasion the link would stay dead until reboot. I got rather tired of the situation and decided to throw some money at the issue by purchasing a Peplin/Pepwave SOHO router / AP. While the SOHO is the bottom of the lineup, it is still thought of as a highly reliable router/AP from a company firmly entrenched in the enterprise market in comparison to the Asus that is clearly aimed at the consumer.
Unfortunately, the number of dropouts and reboots required seems to be about the same with this router compared to the Asus router.

A couple of days ago I had some strange issues with a couple of 2.4 GHz wifi clients loosing connectivity. A reboot fixed that.
Today I again experienced a couple of 2.4 GHz wifi clients loosing connectivity followed shortly by EVERY 2.4GHz wifi client getting dropped.
There is no mention in the event log of anything happening, the client list showed broken connections and after the ip lease times expired all 2.4GHz wifi clients disappeared in the clients listing screen. None of those clients are ping-able. There isn’t anything in the web management screens that I have seen that shows an error. Doing a spectrum survey of the 2.4 GHz wifi spectrum, there are only two networks going and they are non-overlapping.
Where in the management system should I see this failure? How could I determine what caused the failure?
Any help locating the issue is appreciated.
This is the 5th (?) case of a reboot required in about 2 months of operation.

1 Like
2

:x:
2.4 GHz radio use - Wikipedia

:white_check_mark:

  • make sure the antennas are snug and angled slightly
  • run Kismet to see if you are getting evil twinned
  • run inSSIDer to see if you are getting evil twinned
  • get a deauth detector
  • migrate all clients to 5Ghz
  • use an AM radio tuned to ~600 kilocycles to look for sources of noise like switchmode power supplies
  • rule out noisy SCR dimmer switches (dimmable lighting)
  • boost transmit power to overcome this
  • power down unused clients
  • discontinue using (1st gen) Bluetooth devices as they compete on 2.4 Ghz near channel 3
1 Like
3

[quote=“happysurfer, post:2, topic:27503”]

  • make sure the antennas are snug and angled slightly

check

  • run Kismet to see if you are getting evil twinned
  • run inSSIDer to see if you are getting evil twinned

Not sure what ‘evil twinned’ refers to
I use LinSSID for my spectrum scanner

  • get a deauth detector

???
Wouldn’t the clients just re authorize? Should I not see something in the logs to say that this happened?

  • migrate all clients to 5Ghz

Not possible. All devices assigned to 2.4 GHz only operate on that band.

  • use an AM radio tuned to ~600 kilocycles to look for sources of noise like switchmode power supplies
  • rule out noisy SCR dimmer switches (dimmable lighting)

There was no SCR dimmer switch running in the house when the clients got dropped.

  • boost transmit power to overcome this

Already at max

  • power down unused clients

Not possible, many of the clients are cameras

  • discontinue using (1st gen) Bluetooth devices as they compete on 2.4 Ghz near channel 3

I use channel 1 and 5 for 2.4 GHz wifi. I do have bluetooth on my laptops but they are only about 2 yrs old so are probably more recent.

I do use a 2.4 GHz dongle for a space mouse but that is nowhere near the router or the devices that failed.

4

Looked into ‘evil twinned’ and deauth attacks … will look into it further although I am somewhat doubtful that these are issues present here. I do not live in an area that has a lot of wifi access points and this has been the first time that I can recall where every 2.4GHz link was taken down. Also, the way I understand it, a deauth attack must be ongoing or else the devices will just reconnect. Well it’s 3 am in the morning and I refuse to believe that somebody is sitting out there continuously sending deauth packets just for the jollies of it. There is no high value target here that would make the effort worthwhile.

5

  • Traditionally, channels 1, 6 and 11 are preferred in 2.4Ghz to avoid interference from overlapping channels. Use your Wi-Fi scanner to select non overlapping channels.

  • In the client list try, to make sure all clients have at least > -70dbm of signal, ideally -65dbm. Centrally locate the router if possible to improve these numbers.

  • Wi-Fi cameras are criminal/hacker magnets. Switch to cabled immediately.

  • the status event log doesn’t show detailed client information. You would have to run a sys log server to try and monitor the situation. I don’t have experience with this.

  • 2020 to-do list: develop proficiency using wireshark to detect deauth attacks and malfunctioning wifi cameras. I don’t have experience with this.

Bonus Points:

  • See if you can update the firmware in the cameras? (and all 2.4Ghz clients)

  • reboot the cameras periodically

  • change camera’s wifi password periodically.

  • move the cameras onto their own SSID / VLAN to help isolate the problem.

  • give each camera its own SSID / VLAN to make it harder to hack

  • separate SSIDs and VLANs for each camera can isolate a malfunctioning camera thus identifying it and making it less likley to knock out the other ones.

  • on the SOHO set 2.4 Ghz AP channel width to 20 Mhz. (AP>Settings)

  • monitor how much bandwidth the wifi cameras are using on the client list.

  • 2.4 Ghz might only be able to support 2-4 wifi cameras depending on their bandwidth utilization

  • upgrade the cameras to 5 Ghz when / as possible (or cabled / ethernet).

1 Like
6

A couple more points of interest:
The clients did not re-connect by themselves.
Power cycling a disconnected device did not result in the rebooted device connecting to the AP.

Based on what I am seeing, it is clear that the issue is not a deauth attack but rather a crashed 2.4 GHz radio or system. A reboot will be required to establish wifi links.

Happysurfer, while all the points you make are valid with a consumer AP, none of them should be required with an enterprise grade device. It is ridiculous for example to require rebooting of a device at frequent intervals (unless the device is running Windoze)

I have decided to postpone rebooting until tomorrow in the hope that the current state of The SOHO might give a clue to what happened. Hopefully someone from Peplink/Pepwave will respond to this issue even without an open ticket.

7

Hey

Download the diagnostics (Status>Device) for engineering before rebooting the SOHO if you open a ticket.

You probably didn’t get deauth’d maliciously - stuff just glitches out all the time. The problem is the net result is basically the same and unless you are running Kismet, a sys log server, wireshark or other security monitoring tools its hard to know what happened.

Reconnecting multiple difficult to access Wi-Fi cameras is a huge PITA compared to just punching in a new password on a spoofed iPhone so I apologize in advance that this happened to you. That’s why I strongly recommended hardwiring them - to avoid this headache - every time it happens.

I do not mean to be rude but there is a reason why those 2.4 Ghz wifi cameras are always “on sale” and cost $20USD each: it’s because they run 10 year old recycled firmware written by students for other products (usually toys) on ancient insecure Linux distros and get hacked all the time or crash all the time exposing your network and data. Even mid range cameras suffer from this. Google the camera model name and the terms “security issues” and see what you find. Most inexpensive foreign made DVR’s and Wi-Fi cameras are backdoored or easy to hack. There are whole websites dedicated to showing hacked Wi-Fi camera feeds. Are you running these cameras on the same VLAN as you do your banking on? If so stop - isolate them on their own VLAN. Sorry to be the bearer of bad news but “reconnecting” them might be the least of your worries. I suffer heavily from pronoia - so I try to avoid these products like the plague.

Sidenote:

When I got evil twinned and deauth’d I had no idea - an average home user has no way to know. The Wi-Fi would just stop working, or I had to reboot it, or that didn’t work and I had to hard reset the router and redo the set up. Momentary deauths are barely noticeable. I did that for years before I finally bought a SOHO.

Evil twin attacks are different. The client device would not reconnect nor would rebooting get it to reconnect. The attacks are a great way to drive someone crazy - I opted instead to install Kismet, up my game, get a SOHO and catch them. No more attacks - for now.

After a successful evil twin attack I had to rename the AP and make a new password - easy to do on the SOHO. Then enter that info on the affected client device. I had a bad neighbor deauth me and break into my network for years before they got caught. On most consumer grade routers its literally child’s play at this point. Guess your lucky and live in a good neighborhood?

Why did I make 16 SSIDs / VLANS? Because if I put all my devices on the same network - then I would have to redo all of them every time I got hacked. That’s when I figured out how to compartmentalize everything onto their own SSID / VLAN. Spoof one of my networks and it only affects one of my devices - not 20. Plus now they know I am watching so they stopped.

The SOHO is solid in my experience and has safe guarded my network and data repeatedly against attacks - something the consumer grade stuff completely failed to do. The SOHO MK3 is literally worth its weight in gold in my opinion. And on top of that 16 SSIDs / 16 VLANs = 16 routers for $200USD

I wouldn’t go with any other router in this neighborhood. Some people here actually turn off their Wi-Fi when they are not using it - I have never seen that before - that’s how bad this neighborhood is for hacking. No router can fully prevent these types of DOS attacks at this time - but no one has been able to break into the SOHO. As per the documentation and links I sent you, detecting them and capturing evidence is labour intensive / PITA.

Most enterprise grade equipment is 10 times more expensive than a SOHO and then you have to buy expensive intrusion detection software on top of that that dwarfs the cost of the router. But then again the average home user isn’t going to spend ~$5000USD on their networking equipment.

I got hacked, stalked and harassed for over 5 years before I bought a SOHO, (and multiple malicious hackers got caught eventually). Multiple. That’s why I chose the name happysurfer. Just a silly name right?

Thanks for protecting me guys! Thank-you routersecurity.org.

Thank-you Peplink.

1 Like
8

Great idea of downloading the diagnostics for further investigation - thanks.
The 2.4 GHz cameras are all running on Raspberry pi’s, mostly pi zeroW’s and are running the latest release of motioneye os. If it was just the cameras, it would be one thing but there are other devices like cell phones that got dumped.
I have installed Kismet and am now trying to figure out how to get it to do anything.
I do not run individual vlan’s because I need to be able to look at any of the camera feeds. A lot of them are different views of my 3D printers.build surface.
Regarding an evil twin attack - why does it require renaming the AP or a new password?

9

Not a security expert - so my best guess based on what I have seen is that the router (if it’s a good one) and client (if it’s a good one) correctly detect the attack and then “fail safe” by refusing to connect to anything forcing the user to redo their SSID / password.

Could you avoid having to redo the SSID and password? Maybe sometimes - but I was never able to on my affected apple client device. Plus if you’re getting hacked it’s probably a good time to change login credentials? My apple client devices would refuse to connect to it - I suspect the software in the client device blacklisted the SSID - or the router blacklisted the clients.

Maybe less sophisticated or less secure products may lack this level of security and attempt to reconnect - but the SOHO would probably blacklist them as well preventing reconnection.

Plus if you are getting deauth’d the attacker could be trying to crack the password = you should change it anyways. Hardcoded wifi credentials on devices like 3d printers and hard to access wifi cameras are a vulnerability because they are hard to change - so many users won’t - or they will choose easy to enter unsecure passwords.

I practiced using 30-40 digit upper lower case alpha numeric symbolic passwords. A would be attacker will most likely never crack it - especially if you do it monthly - and if they did - it would only be one network out of 16. Enjoy.

http://passwordstrengthcalculator.org/index.php

30 digit upper lower case alpha numeric symbolic password = 94^30 = 1.56x10^59

The Supercomputer Defeats The Password Within:
24,774,163,839,209,919,611,963,739,970,994,176 Years

40 digit upper lower case alpha numeric symbolic password = 94^40 = 8.41x10^78

“It is estimated that the there are between 10^78 to 10^82 atoms in the known, observable universe. In layman’s terms, that works out to between ten quadrillion vigintillion and one-hundred thousand quadrillion vigintillion atoms.”

The attacker certainly isn’t going to socially engineer it.

1 Like
10

Rebooted the SOHO and all devices on both wifi networks came back.
Looks like Kismet will require a LOT more effort to get anything out of it :frowning:

11

Lots of tutorials on youtube on Kismet.

Sounds like the common denominator for both routers were the 2.4Ghz clients you are using causing them to crash or lock up.

Try setting the clients and the router to 802.11abg (HT20) and avoid n if you can’t get a good signal?

Bad signal means constant reboots and slow connections. Google how you should orient the built in antenna on the Pi zero W to get a good signal. Or just try different positions and monitor the received signal strength in the client list. -65dbm should be good. What are they currently getting?

Maybe getting a USB 5Ghz 802.11n Wi-Fi adapter would do the trick?

1 Like
12

-65dbm should be good. What are they currently getting?

All but one are -60 db or better … the point I have been trying to make from the beginning though is that irrespective of signal strength, multipath issues or the phase of the moon, a quality AP should never ever require a reboot based on what the clients are doing. Yes, it will loose a connection … but requiring a reboot just reeks of bad code.

13

BTW, what also reeks of bad code is a client list that reports one client at IP 0.0.0.0 yet shows that client as connected with a signal strength of -57 dbm. In this case the client, a pi zero W, is unreachable but in other situations the client was fully operational yet showed an IP of 0.0.0.0.
I have no doubt the fact that the client is unreachable is a client issue in this case but the fact that it shows up as connected with good signal strength and an ip of 0.0.0.0 is clearly an issue with the SOHO firmware.

14

Interesting hypothesis. What test tools did you use? Can you share the data sets? How many TB of wireshark data did you analyze?

Yeah I know what you mean. Without looking at any code or test data I know I can’t come up with any conclusions either.

All I see is the common denominator was the Pi Zero W boards crashing multiple routers from multiple manufacturers. Maybe try some rPi help forums? Search terms like “pi zero w, wifi problems” might yield some clues?

Personally, I suspect the low gain antenna, budget oriented RF components, and unstable Wi-Fi client software on the rPi Zero W - but that’s just my hypothesis. I want to get some and try them out - I am missing out on all the fun!

Maybe connect a more expensive 802.11ac/n adapter to the rPi Zero W - it will probably cost more than the Pi Zero W for a reason! The Pi Zero W is just a starting point for experimenters - for $5 I literally wouldn’t expect much.

I certainly wouldn’t put a Wi-Fi enabled(?) $5 Pi Zero W on the same network I do my personal banking on.

I would never use it to control a 3D printer that could burn my house down if it malfunctions or off gas hydrogen cyanide if it overheats ABS filament. But those are just my personal preferences - I am not sure what your use case is exactly.

Connectivity problems with a fun little $5 board meant for children could be the least of your worries.

Anyhow I would try playing around with different Wi-Fi settings with them and have fun!

:partying_face:

Additional Info:

I suspect your 2.4Ghz network is a mixture of 802.11b/g/n devices that are all fighting the router due to the fact they all have different parameters - some are g / some are n.

Try configuring all the 2.4Ghz devices to use the same standard so they will play nicely together.

Try isolating them on separate VLANs - certainly get them off your main VLAN ASAP. If you need to look at video then make a firewall rule to allow that.

If any other users have had similar experiences with them I am sure they will PM you.

Eitherway it’s very frustrating - I am sorry to hear that. In the interim you could also opt to hardwire them with actual ethernet cable with the little dongle they came with?

Maybe upgrade to the Pi 3 or 4 as they are a lot better and keep the Pi Zero W’s to play around with?

1 Like
15

0.0.0.0 means there was a problem in assigning an IP address.

The last rPi user that had that problem had accidentally made a double reservation - you can’t do that.

He made a reserved IP address for it when plugged in with an ethernet cable when he was configuring or testing it I believe. Then he disconnected the ethernet cable and connected to it wirelessly and he assigned the same IP address to the wireless adapter. You can’t assign the same IP address to 2 different network adapters (ie wireless and ethernet) or else it confuses the router and it issues a 0.0.0.0

Flush the DHCP by rebooting (you did) and then look for the double entry in the assigned IP addresses under Network>Network Settings .

Click on the VLAN that device is connected to - if you only have an “untagged LAN” then just click that.

On that screen at the bottom shows “DHCP Reservation” clients. If you assigned the same address twice then delete the one you don’t want or delete both and it will just assign a new one which you can reserve again from the client list.

Maybe there is a double reservation in the “DHCP Reservation” list?

1 Like
16

An update:
I have now opened a support ticket with 5Gstores to get to the bottom of this issue. Today the peplink has dumped all 2.4GHz clients several times (3 times that I know of). It has recovered and gradually added all dumped clients after some undetermined amount of time.
I run the old Asus router in parallel to the Peplink SOHO and it does not loose any clients which to me means that it is not an interference issue and it’s also unlikely that someone is mad at me and is trying to disable my networks somehow.
I have tried to install Kismet in order to see if there is anything untoward happening but I suspect that my available wifi radios are not suitable for this job as I have not been able to get anything out of Kismet.
The 5 GHz network on the SOHO has been working fine so far but I will be transitioning all 2.4 GHz clients back to the Asus router as the SOHO is not usable for 2.4 GHz operation at this point.
All-in-all I am not a happy camper with this router !

17

Hello and Good Day

And you for SURE changed the admin password to the WEB UI

And firmware is the latest

19

Yes, password was changed, firmware is latest

20

I can only point to the fact that all 2.4 GHz devices that are serviced by my Asus router are doing just fine while the Peplink dumps all it’s clients.
I suppose I can reverse the channel assignments between the two devices to see if the issue is related to that.

21

Try moving the SOHO as close as possible to the 2.4Ghz clients and as far away as possible from all Bluetooth devices. Shoot for at least -45 dbm in the client list for the affected clients. Try to get the SOHO at least 60 feet away from the nearest Bluetooth device. Try to avoid using Bluetooth.

Also centrally locating the router (as opposed to placing it in a room) can greatly improve indoor coverage. Repositioning clients can have big impact as well. Similar to light, indoor RF can have shadows or deadspots - experiment with moving the router or clients to overcome this. Verify this in the client list signal improvement or degradation. A great example of this is if you take a portable AM radio and tune in to a local station - now try walking around the house and you will observe it doesn’t always get a good signal. Just moving the radio 12" back or forth in some areas can mean the difference between no signal at all and perfect reception.

The problem with Bluetooth is that it operates on the same frequency band as 2.4Ghz Wi-Fi. They can coexist but not peaceably. There will always be interference issues until you stop using either one or the other or locate them as far away from each other as possible - no matter what router you use.

Quote: “Bluetooth 2.0+ should play well with B and G but trying to use it with wide channel N will result in reduced performance. When I got a wireless keyboard I went 800Mhz for the longer range and the fact that it wouldn’t interfere at all.”

As I suggested before you could try setting the SOHO’s 2.4Ghz AP to 20Mhz channel width (802.11bg) to attempt to overcome this interference issue. You could also try “auto channel” selection so it will attempt to avoid the interference - whatever the source. You could also play around with channel selection manually.

You could also dedicate the ASUS to the 2.4Ghz clients if it seems to work better with them particularly with your setup. You could even make the ASUS and 3d printers a stand alone 2.4Ghz network and join it as needed. Don’t run both routers on 2.4Ghz if possible. Sorry if a lot of this is redundant - just pick and choose whatever works best.