I am a small MSP and have been getting requests recently for failover internet connections for my customers. I have used devices in the past utilizing different cell data connections tied into SonicWall Firewalls to achieve this with the success you’d expect.
Since having started my business I decided to get my customers a reliable solution and after some research and comparisons, I decided to five into a MAX-BR1-ENT-LTEA-R-T modem. The device was put into a bridge mode to send its WAN connection right to the SonicWall and it is up to the SonicWall to monitor the normal ISP connection through cable internet to decide if it needs to failover to the Pepwave for an alternative Internet source.
I thought this was set up and working beautifully after testing. The customer was happy and life moved on. This was back in October. However, recently there have been reports of network troubles and poor call quality over VoIP, dropped VPN connections, and you name it. My finger immediately went to the SonicWall and the investigation told me the CPU’s Core 0 is constantly spiking to 100% on the SonicWall.
Support said this is an issue they were seeing as a configuration problem and went through settings until they eventually caved and sent me an RMA device. The RMA arrived today and I decided to set it up on the Pepwave connection to minimize downtime for the client. The sad part is that the same behavior was seen after configuration and even with factory settings. This was a bummer. I went back to the old SonicWall and saw that the CPU behavior now was normal. This immediately tells me that the source of my problem is this Pepwave modem.
Has anyone seen anything like this or does anyone have any suggestions? Could I be setting up the Pepwave incorrectly and possibly using it to send the WAN IP to the SonicWall is a poor choice? I am just looking for any suggestions to help resolve this issue.
Do you mean you reinstalled the old sonicwall without a Peplink device inline and the CPU issue stopped happening? Can you recreate the CPU spike every time immediately / easily? If so, have you tried leaving the BR1 in NAT mode rather than bridge mode?
Which process was consuming the CPU resources on the sonicwall? Maybe that will point us in the right direction…
Fair enough on this. I have seen and read about them too. I did have some nice discussions with their engineers and team about this. In general they seem to resolved down to a configuration issue that they see and I agree with them. I have about 10 SonicWalls in the wild that I manage and this is the only one with this behavior.
-So what I meant was that the OLD SonicWall was never removed from the LAN. I left it in place to keep their network alive while I prepared the new RMA device. I have 2 separate SonicWalls that are the same model now. This allowed me to leverage the Pepwave to configure the device while keeping their LAN connected to the Internet via their main ISP connection.
-This behavior is easily replicated at will. If the Pepwave is connected to the SonicWall, Core 0 of its Dual Core CPU spikes immediately and bounces up and down but basically hovers in the high 90s. The original SonicWall did this behavior and so does the new RMA device. The original device is no longer connected to the Pepwave and CPU Core 0 is steady about about 2-4%.
-I have not tried leaving it in NAT mode. I will give that a try tomorrow. It just seems so strange to me that it would matter. I would imagine that for this setup bridge mode would be preferred but not if it fixes the issue of course.
-Unfortunately the logs did not clearly showcase what process was consuming the CPU. This is the real pain point with SonicWall is that their logs seem to suck. They weren’t even the ones who discovered this. I was able to isolate the issue while trying to showcase the problem to the customer this afternoon. I used his computer to show him the spiking CPU and my jaw hit the floor and I felt like an idiot when the CPU did not spike. Then the light bulb went off.
-Also I don’t think it would matter but the Pepwave is connected to Verizon cellular service.
Unless I’ve missed it the other thing you’ve not tried is an alternative device on the second WAN link of your Sonicwall…
It is all well saying “I removed the Peplink and the problem went away” but at that point did you also put some other deivce in place of the Peplink to feed the Sonicwall with a second uplink?
I would be more thinking along the lines of some process or combination of features that are being used in the Sonicwall are the problem here, the box providing the connectivity should not really factor into it - it is just providing an Ethernet connection to your firewall.
Is there any recurring pattern to the CPU spikes that would align with any health checks for the links that the Sonicwall is performing (or the Peplink for that matter) - cadence of these events and trying to correlate them with configuration elements that run against a set timer could be very helpful here but you may need some fairly high resolution monitoring of the CPUs to get there.
One thing I want to set straight is that the Pepwave was set to IP Passthrough, not bridge mode. Perhaps that is just semantics, but I wanted to be sure that I am accurate in case this helps anyone in the future.
The detail I think you might be overlooking is that the Pepwave recreates the same issue on another SonicWall with factory defaults. The RMA device SonicWall sends is an advanced replacement device so it gave me the benefit of having two devices to allow me flexibility to do some additional testing.
I did a little more research on the SonicWall and I ran a packet capture on the interface that the Pepwave was connected to. The results displayed that there was an extremely high volume of UTP traffic on ports 67 + 68. DHCP requests and responses coming to the SonicWall. This is the information that Core 0 directly would handle from the WAN.
Ultimately something does not play nice with SonicWall and the IP Passthrough with the Verizon wireless network. I don’t know if there is a best configuration practice that SonicWall would recommend for this type of connection but I wouldn’t think it should matter in this setup.
I am going to see if SonicWall wants to test with me while I have two devices. Perhaps it could be for their benefit as well as mine because it seems like they have a few cases like this that they are unable to solve.
As a workaround/solution I removed the IP Pass through on the Pepwave and created a DHCP reservation on the Pepwave for the SonicWall’s Interface. This seems to work. I will do a little more testing to make sure VPN connections have no issue connecting, but at the end of the day this is a alternate Internet gateway meant for rainy days. If my client is using this device I would assume they’ll be happy just to maintain some kind of access to the Internet.
If anyone wants to add some additional input to this topic, feel free. I would be interested in more speculation to this topic.
Sorry, I had missed that you tried it in againast a factory defauled Sonicwall - I thought you were taking the config from your first Sonicwall and then using a new unit with the same config to rule out bad hardware
In some regards it is, as all the Peplink should be doing is offerig up an IP via DHCP to the device behind it… but then you mention this:
How high is a high volume here - perhaps if you can measure and express this in terms of pps or requests per second?.
It does sound like that was cuasing this issue, in that there is something about the passthrough / bridge mode creating a problem for you, but whether that is as a result of how the Peplink behaves in this mode or is a combination of how the Peplink behaves and how VZW are offering up their IPs.
Out of interest in the packet captures how long are the leases being offered for the IP, could that explain a very rapid DHCP cycle?
Another test could be to put a laptop with wireshark on it straight into the Peplink in bridge mode, does the laptop get the same volume of DHCP traffic?
Perhaps repeat the test with a differnet SIM / cell network in use to see if it is something specific to one carrier or another?
Does your VZW service give you a static public IP on the cell side, if so you may just need to do some port forwarding or service passthrough config on the Peplink if you are referring to inbound VPN connections to the Sonicwall, you could also look at using a FusionHub hosted somewhere that would give you a static IP to do the same if not.
