URGENT: huge data cost from a DOS attack on standby cellular IP


#1

Not sure if this should be a feature request or not. kind of a mix of some questions and a request

We just has an issue where someone did a large probe or DOS attach on the IP of a cellular interface which was in standby. Of course the pepwave simply dropped all the incoming unsolicited packets. The problem is that it totaled 27G of traffic in three days, at a cost of $270. There are several problems:

  • The pepwave does not appear to track/display this data. During this period it shows the “real usage” of a few MB from it maintaining the speedfusion tunnels over the cellular link for backup.
  • Since the interface was in standby we did not see any symptom such as a saturated link.
    My concern is that I have 150 static IPs on pepwaves. If someone were to attack them like this I would not know it until I received a bill for tens of thousands of Dollars.
    I cannot deal with it by using private IPs, as the cellular is used for the speedfusion traffic and to back up general Internet access from the locations.

So - my goal is to get one or more type of reporting/alerts in place so I can know an attack is happening and try to deal with it by shutting down the interface and changing IPs.

Several questions:

  1. Does the pepwave report just accepted traffic or all traffic including unsolicited traffic to all ports including those not open (i.e. show raw interface traffic). If not, can this be added?
  2. Can we get this type of raw interface traffic via SNMP? I would be willing to set up an SNMP server for this purpose. I would prefer to be alerted via incontrol, but if that is not possibe, SNMP may be the only solution

#2

Couldn’t you go with a dynamic IP Address instead of a static to prevent this? SpeedFusion would still work as long as your 150 locations are connecting to one location that does have a static IP. The dynamic IP would block all probing/attacks as the IP would not be exposed on the internet.

Regarding the problem you specifically had, when the embedded modem is in standby mode, the cellular modem isn’t going to register any inbound or outbound traffic, BUT as you discovered, the traffic still “makes it way” to the interface, it just ignore it. Of course, your cell provider (Verizon?) still has this traffic on its network, so they bill you.

I wonder if the cellular interface wasn’t in standby mode if this traffic would be “seen”?


#3

This is a between-a-rock-and-a-hard-place issue…I need static IPs because some POS systems need that. As I indicated, this would be easy if the speedfusion was the only traffic, but it is not. We are backing up the IP phones over the speedfusion links AND backing up critical internet access for the point of sale systems, for web orders to COME IN, and credit card processing to go out, and various other things like reports to function. So…some vendors systems reach out to the cloud and open a TCp connection. Those would work on dynamic IPs. Others still just hit an IP address at a given port. So we have port forwarding from the pepwave to the POS system firewall. Those really require static IPs.

I am considering several options. One, which would be REALLY annoying to manage is to have a combination of static, dynamic and private IPs. So based on which of the 100+ POS system vendors a customer has, we decide which kind of IP. That would limit the risk to the smaller percentage (maybe 20%) that really need them. But we still need some kind of real time alerts. So I will still be looking for the pepwave to track the interface traffic (as most routers do), and be able to report on that either via the incontrol API or via SNMP (bleh). Just trying to fix this before it is a disaster instead of after.

John


#4

John, I’ll be sending you information for how you can fix this issue without any headache. Look for my e-mail coming from kevin@3gstore.com You’ll be changing the standby state of the cellular connection to prevent inbound pings like this.


#5

Your carrier should be sympathetic. That was unsolicited traffic from their network. You have 0 control over what their network spams you with. If the data volume was that large, they should have been able to identify it as malicious and block it at the source.

It irks me that data is a commodity. Especially when there are no controls to limit it from the client side. When a client asks for data from the web, there is no way to specify the amount of data you are willing to accept. For example, what happens if google decided to change their homepage to include a couple of 1GB+ images - how does the client know that the one web request is going to end up being 2 or more GB of data? They don’t. I think cell services should only charge for outbound data since that is all anyone has any kind of control over. Not to mention most carriers charge for the same data twice. For example - FaceTime over LTE. Both customers are gonna pay for that call, even though there is only one data stream, two peeps are charged for it.

If they would stop trying to nickel and dime customers for data usage - imagine how much better cell networks would be. They sell a service - data should be free. You don’t go to a buffet and then get charged for each piece of food.


#6

Ah – I think I see it. Change standby state from default “remain connected” to “disconnected”. I just tried that on a test unit and surprisingly it seems to have no effect on recovery time when wan goes down. I expected a longer delay while the cellular connected.

John


#7

Ok – spoke too soon. With it set to cold standby it has to obtain IP address. Vpn goes down and comes up. disruptive


#8

I played with that and it REALLY has terrible effects on delay. The modem has to connect, obtain IP, then vpn comes up. It can be 60 to 120 seconds or more, vs 3 or 4 seconds and no drop of vpn. Basically, eliminates a great deal of the benefit. Could be OK for those customers who almost never go on backup, but we have people who go on and off backup every day, sometimes multiple times a day (for a minute or two). Cannot afford this kind of impact for them.

But it certainly is something I could do when a unit IS under attack. I just have to be able to identify that. I use the incontrol API to pull stats every 6 minutes now, so I know who is on backup etc. If the pepwave starts tracking and reporting interface bytes (or it can be pulled now via snmp), then that will work. Internally, it likely does have those traffic counters, just like any other NIC. Just need a way to see them.

John


#9

Does Verizon or who ever your carrier is have any reporting api that can show real time usage that you can alert on?


#10

Another thought, Can you leave the cellular enabled all the time with the new 7.0 beta firmware set it to the same priority as the wan, set outbound policies in priority order to control which data gets sent via the cellular, and then set a low data usage alert say 250mb?


#11

@jmpfas, I do agree with @jmjones. I strongly recommend reporting this to cellular provider to stop those attacks.


#12

Hello Jonh, @jmpfas
In Australia the cellular (mobile) carriers have to offer you the ability to receive a notification via email & SMS as your monthly data limit reaches the high percentages (the notification percentages and delay in the notification vary between carriers).
We all here have the same challenges, though if you can demonstrate to the carrier that it was a DoS attach through their network inbound to your IP, then they will sometimes waver the charges. Most carriers here in Australia have systems to detect and stop this on the cellular (mobile) networks before it becomes a customer issue even if you have a Public facing IP.

Recommendation:
Ask your provider to;
a) help with the charges
b) setup email usage alerts

Hope this helps,
Marcus :slight_smile:


#13

Verizon just told me that they misspoke – they ARE only billing for “traffic we accept”. So I am going to turn off ICMP.

In addition, they have a semi-firewalled product they just released. Not sure of details or cost increment yet, but it blocks unsolicited traffic. IF it also allows us to set ports to allow and is not too expensive, this would be a good solution.

John