I am attempting to put LEDE project routers between my PEPLINK 380 and my transparent bridged modem. The reason for doing this is to control bufferbloat. In this position the bufferbloat protocol (luci-app-sqm) can be applied to each of the 4 WANs. The WANs are all DSL+ links at nominal 10 mbps supplied by CenturyLink.
I can do this in a configuration with the LEDE router connecting to the modem and supplying the PPOE connection. The LEDE router is then connected to the PEPLINK 380 via a static IP. DHCP is turned off on the LEDE router. DNS is cached in the PEPLINK. When I go to Status → Real-Time it shows activity download and upload on the WAN where the LEDE router is connected.
This works fine with one LEDE router connected to one of the WANs on the PEPLINK. Additional LEDE routers populating the remaining WANs make it more and more difficult to connect to web pages, the connection may never connect etc.
I have tried setting the IP interface on each LEDE router as a DNS server, but that does not mitigate the problem.
Maybe someone can suggest a solution before I fill out a ticket and see if PEPLINK folks can look at my router and come up with a solution.
Set up the Peplink to be your DNS proxy and use DNS service forwarding and force it to the local DNS and set your LAN DHCP to the Peplink IP. Let the Peplink be the controller of the LAN functions (DHCP and DNS) and WAN outbound policies and firewalls. Let the Lede bridges run in pure bridge mode with SQM applied.
Are you sure that the Lede routers are actually doing SQM in bridge mode?
I assume that the PPoE sessions are created on the Lede routers, or is the Peplink doing the authentication?
What are the IPs and Subnets for each of the Lede routers (both WAN and LAN sides)? What IPs are configured on the Peplink WAN/LAN/VLans (IPs and Subnets)? Have you been able to isolate which DSL link is causing issues? You can disable connections to see if the issue is limited to one link or not.
I imagine your configuration requires a bunch of static IPs and subnets between the PepLink and those Lede routers. Make sure that each link is in a different IP subnet from the rest. It is possible to limit a subnet to just three addresses. I believe it is a /30 network. Use 1 /30 network for each WAN to LAN jumper (from Peplink to Lede). Just make double sure that you have mutually exclusive Subnets defined everywhere.
I am rooting for you to make this work, but I bought a router that claims it will eliminate Lag entirely. I kind of knew they were full of shit, but it didn’t hurt to try. They have more features to break connections then to actually make them lag free. I think there are too many variables to completely eliminate buffer bloat. Have you been able to find any performance data as to the actual buffer counts on the Peplink? Are you going purely based on the internet speed test results?
I’m currently working on a hypothesis based on this train of thought:
I imagine your configuration requires a bunch of static IPs and subnets between the PepLink and those Lede routers. Make sure that each link is in a different IP subnet from the rest. It is possible to limit a subnet to just three addresses. I believe it is a /30 network. Use 1 /30 network for each WAN to LAN jumper (from Peplink to Lede). Just make double sure that you have mutually exclusive Subnets defined everywhere.
I have tested each router individually for bufferbloat control and they do a super job…get A or A+ on dslreports/speedtest.
And that does make a major difference in quality of internet connection.
I think the problem may be in the transparently bridged modems. They were all set to the same IP since, once in bridge modem they can’t be accessed…according to a few guides I found. But I discovered they can be accessed if the computer accessing them is on the same subnet. If they are 191.168.1.1 for example, then a computer set at 192.168.1.5
can access them. As you pointed out I am using a bunch of static IPs so I believe this is the main problem. I will reset my 4 modems and see if that helps…
I have tested each LEDE router individually and those tests show near complete bufferbloat control, A or A+ on dslreports/speedtest on BufferBloat. The qualitative feel of the internet connection is much improved by this treatment.
The worst thing about bufferbloat is it becomes greatest at precisely those times when you are most using the internet. So during intense game activity etc is precisely when it is the worst.
It may help get you access to the dsl modems or Lede routers without having to muck with IP settings all the time.
There is another page somewhere that talks about drop in mode and how to setup the /30 networks between bridge devices and the router. That may also help.
Yes I have looked at that page…but thanks for bringing it to my attention. All my modems, although they are in transparent bridges, now have different IPs than anything else in the system. But…not much if any change. The whole system does work but slows things down to a stand still. There must be a loop or something in the system to cause that amount of decrease. Testing with just one bufferbloat appliance on one lan doesn’t slow things down much. I have RIPv2 enabled but that doesn’t seem to do much.
I have thought of just using one WAN to find DNS…maybe looking for DNS on WANs makes things get too redundant.
I don’t think you would need any routing protocols. Everything should know about its neighbor through native network config, right?
Have you tried this setup (or similar)?
Peplink router settings
WAN1 192.168.1.1 255.255.255.0
WAN2 192.168.2.1 255.255.255.0
WAN3 192.168.3.1 255.255.255.0
WAN4 192.168.4.1 255.255.255.0
LAN 10.0.0.1 255.255.255.0
LEDE Bridge 1 (attached to WAN1 from Peplink)
LAN 192.168.1.2 255.255.255.0
WAN - PPOE assigned Static or DHCP (public/routeable)
LEDE Bridge 2 (attached to WAN2 from Peplink)
LAN 192.168.2.2 255.255.255.0
WAN - PPOE assigned Static or DHCP (public/routeable)
LEDE Bridge 3 (attached to WAN3 from Peplink)
LAN 192.168.3.2 255.255.255.0
WAN - PPOE assigned Static or DHCP (public/routeable)
LEDE Bridge 4 (attached to WAN4 from Peplink)
LAN 192.168.4.2 255.255.255.0
WAN - PPOE assigned Static or DHCP (public/routeable)
There should be as little as possible running on the LEDE bridges. They should be a pass through. All DNS and DHCP stuff should be done on the Peplink.
As I write this, I see that you are going to be double NAT on every link. I have had zero luck with IP Forwarding. Perhaps the LEDE bridges have an option to do a NAT mapping with wide open firewalls to make that a bit better. Technically, it is still double NAT, but one NAT is a One To One mapping which should eliminate any port (tcp/udp) contention for the first hop. In my head, that makes it better.
The only way I can think of to get around double NAT would be to acquire more publically routeable IPs. I believe you would want 4 for the WAN sides of the LEDE Bridges and then one for each WAN side of the Peplink. 8 in total. Do not go buy IP addresses - I am not certain it is required or even if it would work. I have had marginal success at best with multiple routers in a path. I feel a huge “AhhhHaa” moment in my future. There seems to be a gap between what I think I know and what actually works.
Just curious. How are you testing the Lede bridges independently? Are you excluding the Peplink entirely? I would suggest trying to leave the Peplink and the Lede bridges all connected and just disable connections inside the Peplink. Start with a minimal set of outbound rules (since you are going to be taking links down and up anyways). Start with one WAN and Bridge and see what it does. Add the second one by enabling WAN2. Use it with those two WANs for a bit. Then disable WAN2 and turn on WAN3. Once you make it through 2 WANs add a third and keep on testing until you either isolate the issue to a specific WAN or you figure out what networking NoNo you have been doing.
JMJones Yes I’ve done numbering similar to what you have. Additionally
there are seqential IPs on each LEDE “cake box”. I have thought that the
problem in my system is dns but could also be doubling up on nats. On that
thought, it might make for a more efficient connection in this set up to
let the “cake boxes” do the NAT and use the IP forwarding feature of the
PEPLINK, something I’ve not used yet:
The only way I was able to get IP Forwarding to work was to go through a VLan network.
Here was my setup
Balance30 192.168.19.0/24 LAN
Balance1 192.168.20.0/24 LAN
B30 was doing the stuff for both WANS. B1 was doing all the LAN. B1W1->B30L1. B1 was setup for IPForwarding. I would have thought that I could setup a VLan on B30 for the 192.168.20.0/24 network and just assign a 192.168.20.254 address to it. I then used the VLan tagging on the B1 Wan. I would have thought that this would have worked. It didn’t. I ended up having to use another VLan and static routes on the B30. VLan was created with 192.168.18.1/24 on the B30 and the WAN link on the B1 got an IP of 192.168.18.2. In this configuration, the B30 would not show any clients in the client list other than the VLan address created for the link (192.168.18.2). Kind of what you would expect when doing NAT.
Like I said, there must be something I am missing. One of the issues (I think) was that the Balance30 cannot do a custom trunk to allow tagged and untagged traffic. Since the outbound traffic was tagged on the WAN, I assume the traffic coming back would also be tagged. My devices are in the untagged LAN of the B1. So the tagged traffic died at the B1 on the way back since nothing on the LAN side used that VLanID. SO, I think I need to set up the trunk to allow both tagged and untagged because packets from LAN are untagged and packets from WAN are tagged.
Once the new firmware comes out (and hopefully the custom trunk works on the B30) - I will give it another shot. For now, I have gone back to a single router. Surprisingly, connecting the two WAN to WAN seemed to work the best, but NONE of the reporting features knew how to handle this type of setup. On the B30, the ONLY way to see if there was anything going on with the WAN links was the throughput numbers on the dashboard page. Active sessions wouldn’t show any connections. Client list was empty. I didn’t want to have to look at the LAN router to see what was going on with the WAN. I also hope that they change the way multicast and UPnP functions on the VLans. Currently, UPnP requests are only honored from a VLan if the source device uses a TTL higher than one. There is a hop from VLan IP to LAN IP of the router. TTL of one simply doesn’t it make it to the router gateway. Basically, all the features I want exist, but I cannot find the right combination to make it work perfectly.
I really hope Peplink includes the industry standard SQM features soon. I want low latency for gaming and VOIP, everything else can just do what it does.
