Outbound Policy + Firewall + VLAN bug in firmware 8.5

I have an IOT device ( In.Touch 2 - Advanced Spa Control System | Gecko Alliance ) which worked fine in Firmware 8.4 but is now failing in 8.5.

The In.Touch 2 hardware has limited diagnostics available, but it does have a blue light which means “connected to the cloud server” and the iOS app will show whether your phone is connecting to the hardware device via a Local connection (e.g. via the same LAN subnet) or a “remote” connection using the cloud server.

After many trials & errors, I’ve narrowed it down to this set of issues:

  • In firmware 8.5 on a Balance One
  • With two VLANs (IOT VLAN, and untagged LAN)
  • If a Gecko In.Touch2 is hooked to an Ethernet Port set to Access for the IOT VLAN
  • and there is an Outbound Policy is set such that VLAN devices are set to Priority (with WAN2 in front of WAN1)
  • and there is an internal firewall rule with Source: VLAN network, Destination: Untagged LAN network, action = deny

Then the IOT device can not talk to its cloud server.

Additional info. I think the issue may be some subtle difference in firmware 8.4 and 8.5 in how it handles firewall rules?

In firmware 8.4 I can have a firewall rule which blocks all access from the Guest VLAN to the Untagged LAN:

But in firmware 8.5, this causes devices to stop working.

However, if I add another rule in firmware 8.5 which allows the VLAN devices to access the router itself (which is the .1 address):

Then the devices function again.

I wonder if in firmware 8.5, the first firewall rule is blocking something important (such as DNS?) and by adding the Allow rule, it’s working around that bug?

Hi soylentgreen, I appreciate your input and confirming that there is an issue (bug) in 8.5.0 with IOT devices on a VLAN no longer responding to commands. I don’t have any special Outbound Policy Firewall settings on this VLAN, but I do have these VLAN devices isolated from each other on the WIFI network APs. I have never had a problem with the previous Firmware versions using my WIFI configuration. I always keep my IOT devices isolated. I sure hope there are no data leaks in 8.5.0.
I hope someone is continuing to investigate as I don’t have the time right now to troubleshoot the problem.
Dave

1 Like

@Daking - thanks for sharing your situation.

I see another person @pbawesome may be having similar issues, though perhaps not identical.

I’m glad I noticed the problem right after I updated the firmware or I may have wasted a lot of time “going down the wrong rabbit hole”…
I’m sure others will discover the problem once they upgrade their firmware.
Hopefully it will be resolved in 8.5.1.
Thanks again,
Dave

@pbawesome , @Daking , @soylentgreen Has anyone raised a ticket with peplink support, I haven’t seen this issue, but perhaps I’m not replicating the problem exactly.

I just had a related bug with outbound policy on a VLAN. I needed a single device enforced to use a single connection, and I put it in the way I always have with the source as the single devices IP, but on 8.5, it would not work. I had to put the entire VLAN subnet as the source before it would work.

1 Like

source as IP and enforce rule or priority rule? Priority rule always seems to be more successful for me somehow.

Enforce rule. Used many times on many devices, never had an issue before. The source address was in a /29 subnet this time, that was the only difference than “normal”.

@Jonathan_Pitts I have not submitted a ticket. If you want to do some internal testing, here is my setup. Although my system does have WiFi, I can reproduce the issue using only ethernet devices, so I’m not mentioning my WiFi settings.

  • Balance One with Firmware 8.5.0
  • WANs: Two. One Static IP with 4 Additional IPs, one DHCP
  • LANs: untagged LAN 10.0.32.x , guest (IOT) LAN 10.0.64.x. Inter-VLAN routing is ON for both (but see the firewall rule below). Normal DHCP servers on both.
  • LAN: Bonjour Forwarding is enabled: Service Network: VLAN. Guest network: untagged lan.
  • Firewall Rules: to prevent the Guest VLAN from talking to the LAN, there is a firewall rule (see above). Protocol: Any: Source: IP Network 10.0.64.0 / 24. Dest: IP Network 10.0.32.0/24.
  • Outbound Policy: Source: IP Network 10.0.64.1/24. Destination: Any. Protocol: Any. Algorithm: Priority: WAN 2, WAN 1. Terminate session on connection recovery: Disabled.
  • Network/ Port Settings: Port 4 is set to Speed: Auto. Port Type: Access/ Guest VLAN.
  • IOT Device: Gecko In.Touch 2 plugged into port 4.
  • there is a DHCP reservation for my IOT device on the VLAN network

With this test setup, I find that in firmware 8.4.1, my IOT device can connect to its cloud server. With firmware 8.5.0, it fails. I believe this device is UDP only. I’m not sure what the failure is (e.g. DNS? UDP packets being blocked?)

Note: In 8.5.0, if I add another firewall rule which allows the VLAN to connect to the balance one’s IP address on the LAN, this seems to be a workaround (see images above).

Writing this down, I just realized that in some settings I’m using
10.0.64.1/24
and in others
10.0.64.0/24

Since a /24 network has a net mask of
255.255.255.0
I think the .0 vs. .1 difference shouldn’t matter, right?

I created a ticket 24091109 as I am having connectivity issues with my IOT devices that are on a VLAN. Each time I reboot there are some IOT devices that don’t work.
Inter-VLAN routing has always been turned off as I want the IOT devices isolated.
My configuration has never had issues until firmware 8.5.0.
Rolling back to 8.4.1 has resolved the problems.

2 Likes

Update us here with the results

Sorry, I meant to say that I am currently using 8.4.1 and that I have no problems after I rollied back the firmware.

Has this been addressed in Firmware 8.5.1 beta? Firmware 8.5.1 Beta 1 - #12 by rossh_pl_beta

If not, can we get an estimate of the fix date?

Any updates to ticket 24091109?

I did some more digging, using remote packet capture and Wireshark:
Link

What I’m seeing is a never ending stream of what appears to be DNS failures:

no. Time Source Destination Protocol Length Info
452 0.794268 10.0.64.104 10.0.64.1 DNS 82 Standard query 0x1234 A intouch2.geckoal.com
453 0.794275 10.0.64.104 10.0.64.1 DNS 82 Standard query 0x1234 A intouch2.geckoal.com
454 0.794441 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
455 0.794455 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
861 1.807191 10.0.64.104 10.0.64.1 DNS 82 Standard query 0x1234 A intouch2.geckoal.com
862 1.807198 10.0.64.104 10.0.64.1 DNS 82 Standard query 0x1234 A intouch2.geckoal.com
863 1.807404 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
864 1.807424 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)

10.0.64.1 is the Peplink’s IP address for my VLAN
10.0.64.104 is the IP address of my GeckoIntouch which is connected via ethernet.

Indeed, the Gecko has a Green light, which means "I can contact the hot tub, and contact the router, but I’m unable to contact the mothership at intouch2.geckoal.com )

I’m in over my head here, but from googling, this doesn’t make any sense.
Why would the Peplink be responding to a DNS query with an ICMP response (that is failing?)

Edit to add: I think what this means is the ICMP response is the error message in response to the DNS request, and basically saying that the DNS query has failed, because “Port unreachable”.

This is looking like a garden-variety DNS bug in the 8.5 firmware, isnt’ it?

(Isn’t there a famous saying: “It’s always DNS”)

With some more testing, I think this is, indeed, some sort of DNS bug.
On the Balance One, on my VLAN, I turned off the ‘Assign DNS Server automatically’ button, and instead put in the DNS IPs for my ISP:

I rebooted the IOT device, and immediately everything is working normally.

Here’s the packet capture when it starts working:

no. time source dest Protocol length Info
246336 405.138749 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
246337 405.138770 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
246704 406.153046 10.0.64.104 10.0.64.1 DNS 82 Standard query 0x1234 A intouch2.geckoal.com
246705 406.153053 10.0.64.104 10.0.64.1 DNS 82 Standard query 0x1234 A intouch2.geckoal.com
246706 406.153226 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
246707 406.153247 10.0.64.1 10.0.64.104 ICMP 110 Destination unreachable (Port unreachable)
256921 422.397930 10.0.64.104 209.xx.xx.xx DNS 82 Standard query 0x1234 A intouch2.geckoal.com
256922 422.397937 10.0.64.104 209.xx.xx.xx DNS 82 Standard query 0x1234 A intouch2.geckoal.com
256927 422.410666 209.xx.xx.xx 10.0.64.104 DNS 98 Standard query response 0x1234 A intouch2.geckoal.com A 23.101.153.137
256928 422.410690 209.xx.xx.xx 10.0.64.104 DNS 98 Standard query response 0x1234 A intouch2.geckoal.com A 23.101.153.137

Has this been addressed in Firmware 8.5.1 beta? Firmware 8.5.1 Beta 1 - #12 by rossh_pl_beta

No, not fixed. I did a long private ticket about it, and they flat out refuse to fix the issue. They could not even provide any justifications or valid use case for their errors.

1 Like