Peplink Security Breach

Everything Else
1

TL:DR: Peplink engineering can enabled remote assistance on your device and access it without your consent or knowledge and did just that to me. You’d never know it, because remote assistance is not a condition triggered in the event logs.

On January 24, 2023, I submitted a ticket to Peplink engineering reporting a bug within 8.2.1 build 5018 and 8.3.0 RC4 in relation routing while OpenVPN, a license feature I paid for, is enabled. After speaking with Peplink engineering for a few days they requested I enabled remote assistance so they could access my MAX-BR1-PRO-5GH-T-PRM to reproduce the reported bug. I enabled remote assistance on January 26, 2023, around 1419CT and shared with Peplink engineering that I was ready for them to connect. After Peplink engineering had successfully reproduced the reported bug and notified me, I disabled remote assistance.

On January 26, 2023, around 1511ET I disabled remote assistance and asked Peplink engineering to reach out if they needed access again. Sometime between January 26, 2023, and January 30, 2023, Peplink engineering, without my knowledge or consent, accessed my device through IC2, enabled remote assistance and gained access to my MAX-BR1-PRO-5GH-T-PRM. It was only by pure luck I had found remote access was enabled when performing my daily system checks. Once I had found out this had happened, I immediately disabled remote assistance and reached out to Peplink engineering for answers.

Peplink engineering responded 22 hours later with a generic response and ignored my question of, “How did remote access get enabled.” I immediately replied to their response and asked the same question again. After my reply it took Peplink engineering 24 hours to respond just to state, “We have no idea how this was enabled.” I was done beating around the bush and directly accused Peplink engineering of enabling remote assistance and this was their response, 48 hours later.

PEPLINK’S RESPONSE
“Reading through your latest response, we have conducted an internal check to clarify the matter.
I must admit that we have made a mistake, as part of the troubleshooting process, the support personnel was anxious to find out the device condition and unintentionally turned ON the Remote Assistance via IC2, without prior consent from you. Only the Level 2 and above Support Personnel have this access privilege.

We have contacted that particular support personnel and all the support team members to highlight and stress that we shall never repeat the same mistake again in the future. As a standard practice, we should ask the customer/partner to enable Remote Assistance, then only we will access the device for investigation.
One more thing that I would like to share with you, there is an option to allow you, as the IC2 > Organization owner, to prevent Peplink from accessing your IC2, by enabling the “Block Peplink Support” option (as shown below).”
END OF PEPLINK’S RESPONSE

The feature being discussed above states, “Block Peplink Support – Prevent Peplink support from viewing this organization.” Nowhere in that statement does it say, “Prevent Peplink support from making changes.” What Peplink engineering did is unacceptable, and they need to be held accountable. Peplink devices are in use by individual users, commercial users, some with PCI DSS requirements, and Government agencies at all levels. To know that Peplink not only can enable remote assistance, but to enable it without the owner’s consent or knowledge is gravely concerning.

The ability to view customers unsanatized/unsalted configurations, passwords, certificates, etc. without their consent is unacceptable. A malicious employee has full access to the user’s network and could capture traffic if they enabled captures under “cgi-bin/MANGA/support.cgi”. Furthermore, they could offload those captures in real time to a remote host. I’ve been a Senior Network Engineer for 10+ years and I’ve worked with major vendors such as Palo Alto, Cisco, Juniper, Arista and Aruba and I cannot think of one time where an employee had the ability to circumvent a customer and/or their configurations and enable remote access.

I replied sharing my displeasure with Peplink engineering and requested a phone call to further discuss what is technically a security incident but have been 100% ignored. Peplink owes its customers an explanation as to why this was ever allowed in the first place and how/when they intend to do away with this “feature”. Searching online shows other customers have seen their remote access “randomly” enabled, too.

If you think you can block IC2 using the built-in firewall, think again. It appears the traffic for IC2 bypasses the firewall. Even with denying all inbound traffic and denying outbound traffic to ac1.peplink.com, ac2.peplink.com, ra.peplink.com and ra-geo.peplink.com, IC2 still works. I submitted a ticket to Peplink asking why IC2 was bypassing the firewall, but instead of answering the question they wanted to focus on why disabling it within the GUI wasn’t enough.

13 Likes
2

This is concerning, as we have seen so many breaches of MSPs lately to gain access to the customers managed by those providers, if PepLink was breached and the criminals were able to access our equipment, modify it as desired, this would be a huge security concern.

I’m surprised they have the ability to just turn on remote access from the Internet, as that is the first thing all security manuals tell you to do, is disable remote access on network connected routers.

I spend my work days researching breaches, vulnerabilities, etc. and a “That would never happen here” is simply not good enough. It WILL happen, it is just a matter of when the payoff is enough for the hackers to bother.

-Michele

5 Likes
3

My thoughts, too. It’ll happen again just like it has to others in the past that have posted here about remote access randomly being turned on.

4

I wonder if they have this capability if you use a privately hosted IC2 instance.

5

I’m not sure, but this “capability” shouldn’t even exist. There is no reason someone should be able to enable remote access to your device. I did try to contact Peplink multiple times, but they’d rather post about new rugged network switches. You can tell where their priorities fall.

6

Try contacting your distributor, they might be able to get you into contact with the proper department/person.

There are lots of people working at Peplink, so you can’t blame anyone posting questions about other products from other departments about where their priorities lie. I can understand you are upset, but that’s not something the forum can fix for you. Try going through the proper channels, we have good contact with our distributor in our country.

2 Likes
7

No, this has been known by Peplink for 7 days. They decided to go silent in the ticket I opened when I asked to discuss it further once they admitted to doing this. This is not a distributor problem so I do not want to waste their time. My intention of posting it in the forum is to finally get a response from someone in Peplink.

8

My experience is that the forum is not the fastest way to contact a vendor. This is not unique to Peplink. Whenever we would have an issue with a vendor and we couldn’t get in touch with them properly, we try through our distributors, as they normally have different contact channels.

Also, if this issue is something as important as you find it to be. It might be helpful to notify your distributor as well, as they might be of the same opinion as you and can help you achieve your goal faster.

That’s just my advice, do with it what you want.

1 Like
9

I do appreciate your response and your advice. In this case I do not have a distributor as I only have this device and it was purchased through Amazon.com.

10

I spoke with Travis and Keith from Peplink management by telephone. They both agreed that this is very concerning, were very apologetic and believe they both have a path forward. They stated Peplink will release a statement, but they did not say when. To be clear, the concern isn’t Peplink engineering utilizing remote assistance to access a device. The concern is Peplink engineering being able to enable remote access without a customers knowledge or consent thus allowing unauthorized access. Remote access is not a triggered event in the event log either.

3 Likes
12

You can disable the function that Peplink can access your inControl, it’s a setting set on the Organization Settings. This feature was already present to comply to certain regulations. You can disable this function when you need Support.

image.png
image.png1030×88 6.05 KB

No one can override this but an Administrator in your inControl2 organization.

1 Like
13

That feature will soon be enabled by default after my conversation with Travis and Keith yesterday.

4 Likes
14

Our team is actively working on a plan. We will respond with details in the next few days.

Thanks for your patience while we are working on it.

5 Likes
15

Kudos to Peplink for not killing this forum thread and for (hopefully) addressing the problem.
Thank you!

3 Likes
16

Had they killed this thread it would have only made things worse. Anyways, what’s done is done and I only hope Peplink can move forward in the positive direction we discussed and roll out their planned fix.

1 Like
17

Logging logging logging. The use of remote access by Peplink tech support should be noted in the Event Log. When it starts, when it ends and the source IP address.

As for blocking access to InControl2 via domain based firewall rules: are you sure those are all the domains? If so, maybe use DNS to assign them an invalid IP address.

Taking a step back, it is easy to see times when things are really fouled up that allowing Peplink into the router would be a godsend, and for cases like that, their use of hard coded IPs to avoid any DNS issues, could be a life saver. On the other hand, this thread.

Its much like truly secure end-to-end cloud based file storage. If you want the highest possible level of security, it means that losing your password loses all access to your files. Some people want/need that, others do not. The price for a get-out-of-jail card is that the cloud file storage company can read your files.

As long as it is made VERY clear that blocking Peplink tech support means you are on your own when things go bad, all good. Some customers will want this, others not. I see this as a documentation problem. That choice was not front and center and clearly communicated. One checkbox out of dozens is easily missed.

5 Likes
18

I agree 100% on your logging comment. I did open a ticket regarding not being able to block IC2 using the firewall. The work around provided by Peplink was to deny DNS resolution for ac1.peplink.com and ac2.peplink.com. That’s a good start, but if the IP addresses were to ever become hardcoded within their software, the firewall rule wouldn’t be matched anymore and traffic would be allowed. I shared that comment with Peplink support and they stated the IP addresses are not hardcoded and only use ac1.peplink.com and ac2.peplink.com

19

I can confirm that IC2 and remote assistance does not work even if enabled when blocking ac1.peplink.com, ac2.peplink.com and ra.peplink.com using outbound policies.

20

I am going to jump in on this one, as someone who has been involved in computer security since 1977 when I got in trouble for hacking ARPANET, worked on a bank wire transfer system moving a billion Dollars a day, and have written point of sale systems, AR/AP systems etc for large companies.

First - we all agree Peplink should not have accessed a device without consent.
BUT - reality check time: Any cloud managed device CAN be accessed remotely by staff at the company running the cloud management, unless there is a hard control to disallow it. If you disagree, you are a fool. And many non “cloud managed” devices have some form of remote access. That is why the “internet of things” is a hackers delight. Every web TV, IP camera, music player…all live connection to the cloud.

And, even if there IS a control, you are still trusting that the company will not have a backdoor to override it - just as you trust EVERY software and hardware (which is software) manufacturer to not have a back door sending them login credentials etc.
My point being that although most people on this thread are showing a proper amount of common sense in what they are requesting, one or two are asking for the impossible - to make it NOT POSSIBLE for the device to be accessed remotely. Only way to do that is not connect it to the Internet, which somewhat reduces the usefulness of a router. Sonicwall, mikrotik, cradlepoint - you name it, it is physically possible for them to write firmware to allow remote access. You just trust that it is not happening.

So, what is practical?

  • A clear statement that peplink will ever access a device or certain IC2 data without prior consent.
  • Logging of and real time reporting of access. i.e. email saying “tech xxx from IP yyy is accessing device zzz via RA”. Perhaps most recent RA event shows on dashboard, not just in log.
  • control in GUI to block RA being turned on remotely (did I get that correct, issue here is RA was “off” but was turned on, then accessed remotely?)
  • control in GUI plus perhaps finer control in IC2 to tell device to not display login creds in IC2. (may need those creds for IC2 to function at all, not sure of that)

Bottom line - Add a few more controls, add some logging, get clear statements of policy and just move on. At the end of the day, you just have to have faith.
We have about 2,000 peplink devices, some in place for six+ years now and have never had an actual security issue involving the peplink. And over the years they have responded well to enhancement requests involving security.

13 Likes
21

That’s true, but that’s also what backing up the configuration file is for. Worst case scenario, if I somehow mess up the configuration of my Peplink router and lock myself out, I can do a hard reset and then restore from my configuration backup file.

I want:

  1. a promise by Peplink that they will NEVER access my router without my express authorization,
  2. a promise by Peplink that any remote access is logged and an event notification is sent to my account e-mail address, and
  3. a setting configurable in the GUI that disables remote access by Peplink, that can’t be changed remotely, and that technologically prevents Peplink from remotely accessing my device (or changing that setting) without a change in the firmware. (And, of course, that setting should prevent Peplink from pushing out a firmware update too.)
3 Likes