I am currently experiencing an issue at one of my branch locations, where we are getting flooded with IPSec VPN connections. All the connections are refused, but causing the CPU to peg at 100% on our MAX Transit router and consequently drop our valid Site-to-Site IPSec VPN connection.
We’ve tried adding explicit firewall rules to block the traffic from the offending IP addresses, but it does not appear to have made any change, as nothing is getting logged in the Firewall logs that show the rules are even being tripped.
In looking at other forums, it appears the best approach is to try and black hole these requests and route them to a non-existent IP address. I’m struggling to find the right way to do that through the admin UI for my peplink.
Does anyone have any guidance / advice for ways to properly block this traffic and avoid spiking the CPU?
If nothing is being logged that would suggest the rule is not correctly configured as no traffic is hitting it, that said if you are being DDoS’d enabling logging on the rule to match that traffic is probably a bad idea.
You also say “offending IP addresses” - how many are you talking about? My experience of mitigating DDoS attacks normally involves null routing at our border routers and further upstream as it tends to be a huge range of source IPs not just a handful, hence the term “distributed”.
In Peplink I’d probably created a grouped network that contained the bad IPs and then an inbound firewall rule to drop any traffic sourced from those addresses, or if your legit site to site link only ever initiates form a known IP drop any incoming IPSEC apart from traffic sourced from that known address.
This may not really help you though, depending on the volume of connections the Peplink firewall and thus CPU still has to process that traffic.
I am not actually sure there is a way to add null routes in Peplink, at least not that I can think of without abusing outbound policy and that won’t really help you anyway - that kind of mitigation is normally done upstream of you and is in the case of DDoS mitigation is normally done by dropping traffic towards your WAN IP(s) not based on source.
Try fixing your firewall rules, but really as above you need to filter the traffic before it arrives at the Peplink. Do you have any kind of ISP managed CPE router that might be able to do that?
Thank you for the reply. It’s not a wide variety of IP addresses at the moment. There are 6 distinct IP addresses that are pushing requests every second, based on what I can see in the IPSec VPN Event log.
The result of which is that the Peplink gets slowly overwhelmed and subsequently pegs at 100% CPU and drops the expected traffic we want. Even after a reboot, the Peplink stabilizes a bit, and then handling the flood of connection requests slowly push it back up to max capacity.
Given the limited number of IP addresses, I created an inbound firewall rule to deny all traffic from those IP addresses. It hasn’t seemed to have any effect, as the Peplink still needs to process that traffic (as you reference). It’s possible that I configured the firewall rules incorrectly, but I’m not sure how, given I’m trying to block all inbound traffic from those IP addresses, it seems pretty straightforward. I’ve revised it to leverage the grouped network, as you describe, but maybe I’m missing something in the configuration:
I’ve tried talking with our ISP, to see if we can do anything upstream of our Router, but they’ve been of limited help. It’s been frustrating, so I appreciate you taking the time to review.
Having had a look too I think Paul is correct here, you may need to use the local services rule list to filter traffic.
I had a quick look and could not see an explicit option for IPSEC VPNs trying to terminate on the Peplink you could try the PepVPN option and see if that works.
You also cannot use grouped networks in these rules so would need to define individual IPs, otherwise as previously suggested work the other way around - if your incoming site to site vpn is always from a known IP permit that and drop everything else. Obviously not helpful if you have things connecting in from IPs that change frequently.
Not exactly surprising I suppose, as a last resort you could ask them to just allocate you a new WAN IP and see if the problem follows you around? Hardly ideal but could be a low tech solution, but may not be a permanent mitigation if people are finding your Peplink via some DNS record that you’d then update to your new IP…
Quite an odd one to be honest, 6 distinct addresses is really not a lot - you could try reaching out to the abuse contacts at who owns the IPs although depending on who that is you may get about as much help as you did from your ISP, and you would likely need to provide some reasonable level of evidence of the abuse (I’d have some packet captures on hand and log extracts as a bare minimum).
Thanks for the replies. I saw a post that referenced using the Local Service Firewall Rules, so I added them there as well (as individual IPs). Unfortunately, I didn’t see a significant change in behavior.
We’re going the route of changing our Public IP for now, and we’ll see if that addresses the issue or not. It is weird that it is a small number of IP addresses, as it’s just enough to make our VPNs unstable - but not enough to impact the office’s connectivity.
I’ve sent notices to the ISPs (AT&T, COX, Flexential, and Google) last Wednesday. Given it’s been a few days, I’m not optimistic they’re going to do much, but we’ll have to see. If anyone’s gone through the engagement on that, I’m curious how long it took to get some meaningful action.
