Need Active Queue Management for Bufferbloat (fq_codel)

Product Discussion Pepwave AP & Surf
1

Peplink, will you please respond to my 5/17/18 posting to the existing BufferBloat topic in Product Discussion. My 5/17/18 post is reproduced below. Please note that my posting below refers to several posting by others in the existing BufferBloat topic, so you may want to read that thread first. For example, others have tried to front end Peplink routers in a multi-WAN environment with other routers which have fq_codel upon which I comment. Thank you!

So what about it Peplink? Is it time for Peplink to step up to the new ways of handling QoS and Bufferbloat? Do you want to continue forever with the old way where an expert has to tune QoS and then it is never right for all situations? Or should Peplink adopt fq_codel which automates tuning dynamically to the type of tcp/ip connections which are currently active and dynamically to the amount of bandwidth your DSL or cable provider is providing at different times of the day?

The new way with Fq_codel takes care of automatically adjusting the speed of each tcp/ip stream to make sure your broadband connection(s) is not overloaded and incurring bufferbloat based on measuring RTT, round trip time, keeping it low for each connection. And it does it “fairly” (the F in fq_codel), dynamically handling whatever types of tcp/ip connections are currently active. How many different product discussions are currently pending in the Peplink forums such as “Single Youtube upload killing Internet traffic” or “Limit download throughput per WAN” that are non-issues with fq_codel?

To date, Peplink has given us ways to set the number of buffers, which as commented on earlier in this thread doesn’t really work, fixing one problem while introducing another. And as you will find on the web if you research, “Quality of Service (QoS) settings will help, but won’t solve bufferbloat completely. Why not? Any prioritization scheme works by pushing certain packets to the head of the queue, so they’re transmitted first. Packets farther back in the queue still must be sent eventually. New traffic that hasn’t been prioritized gets added to the end of the queue, and waits behind those previously queued packets. QoS settings don’t have any way to inform the big senders that they’re sending too fast/too much, so packets from those flows simply accumulate, increasing delay for all. Furthermore, you can spend a lot of time updating priorities, setting up new filters, and checking to see whether VoIP, gaming, ssh, netflix, torrent, etc. are “balanced”. (There is a whole cottage industry in updating WonderShaper rule sets. They all have terrible flaws, and they don’t help a lot.) Worst of all, these rules create a maintenance hassle. Each new rule has to be adjusted in the face of new kinds of traffic. And if the router changes, or speed changes, or there’s new traffic in the mix, then they need to be adjusted again.”

The $50 router mentioned in the previous post is most likely the Ubiquiti Edgerouter X which has fq_codel and works GREAT on reducing bufferbloat. Fq_codel, a proven Active Queue Management (AQM) algorithm, is available in an increasing number of commercial routers. For example, besides the Edgerouters, it is a component of Qualcomm’s “streamboost” QoS system. It is in Netgear’s “Dynamic Qos” feature”. The Zyxel Armor Z2 AC2600 MU-MIMO Wireless Router offers StreamBoost. And of course LEDE (now back to being OpenWrt), etc.

The new DOCSIS 3.1 standard includes DOSIS-PIE Active Queue Management for all DOCSIS 3.1 cable modems (too bad many cable providers don’t enable it).

Peplink, we are ready for you to step up to the plate. Yes, you have a lot of QoS and prioritization code that users have tuned their existing installations with, so you can’t just replace it with fq_codel. You can give fq_codel to us as an option to enable however. Please do so!

------------------------- Notes to Peplink users struggling with Bufferbloat ------------------------

I have provided a number of customers as well as friends and relatives with a lot of routers. Here is what I use and why.

Decide whether or not Bufferbloat will be a problem. If you have enough broadband bandwidth, you can get away without fq_codel. I.e., Peplink is great. That said, I always recommend a router with fq_codel for DSL situations.

For a simple home situation that will experience bufferbloat, the IQrouter is great. Don’t plan on doing anything sophisticated like VLAN’s though.

The Ubiquiti Edgerouters are truly excellent if you can get them configured and are willing to use AP’s for Wi-Fi. They are great for situations which fall neatly into configurations for which they have Wizards to create and you need to do little else, especially CLI commands as opposed to GUI configuration. For example, they have a wizard for WAN plus two LAN subnets. On the other hand, you probably don’t want to do multiple WAN’s with VLAN’s for example, unless you have lots of time to become an EdgeOs expert.

Ubiquiti has recently introduced the new Edgerouter ER-4 and Edgerouter ER-6P. These can do 1GB at $200 and $300 respectively as long as you don’t enable fq_codel which takes additional cpu power and reduces the throughput. The Peplink Balance One is limited to 600mb.

To the individuals who have tried to add an outboard fq_codel router between the modem and a multi-wan router, I’ve tried this configuration with both Ubiquiti and Netgear fq_codel. Netgear wasn’t reliable enough for me. Ubiquiti always did its job reliably, but unfortunately the multi-wan router I was using at the time (not Peplink) sometimes would declare a WAN link out of service and never bring it back into service, so I didn’t stick with that setup.

Wrapping this up, I like the IQrouter for its simplicity. I like the power and reliability of Edgerouters. I love the Peplink for its easy configuration, reliability, and support. That’s why I am hoping Peplink adds fq_codel support soon.

3 Likes
3

Hi, we understood your concern and looking at it.

1 Like
4

@Mark9, yea we understand your pain and actually this topic has been discussed within the team for a long time, may be I could share more information with you before we have some firmware for you to play with. From our internal test result quite some time ago, fq_codel alone is just not enough for a dynamic environment, like a cellular connection. That’s why we are still working on it. So far we already have some candidate to help improve the problem, but not finalized enough yet to build into the firmware, we will absolutely keep you posted once we have more information to share with you. :+1:t2:

2 Likes
5

Very glad to hear this is in the works! As you point out, dynamically adjusting to a changing environment is advantageous. IQrouter is making waves with their router doing this. These days, it isn’t just DSL bandwidth performance that sags at different times of day. So does cable. Fixed ingress and egress values for fq_codel doesn’t work very well in that environment.

Note to others: I can state my experiences upgrading to DOCSIS 3.0 cable modems with 24x8 or 32x8 channels is that it seems to be better statistically for maintaining low latency than older cable modems with lower channel counts. Of course new DOCSIS 3.1 modems are now available now too.

Thanks very much for the update!

2 Likes
6

Does Peplink have plans to implement Active Queue Management in 2019?

Steve wrote “From our internal test result quite some time ago, fq_codel alone is just not enough for a dynamic environment, like a cellular connection.” Would Cake, the successor to fq_codel and which is now quite stable, take care of the issues you ran into? And is the version of Linux which Peplink is currently based on 4.4 or greater which apparently makes Cake installation straightforward?

If AQM isn’t scheduled for 2019, is there any possibility Peplink would consider adding fq_codel in 2019? It works well in many situations, raising Speed test - how fast is your internet? | DSLReports, ISP Information results to a grade of “A” for cable, fiber and DSL. As it is not unusual for routers to support multiple AQM choices, Peplink can add a more sophisticated AQM or two in the future. Fq_codel is built into Linux so it should only take adding configuration of fq_codel into the Peplink GUI.

The following (4 year old) article has a good summary of what is better about Cake than fq_codel: Cake - Bufferbloat.net.

The benefits include (taken from other articles):

“A major enhancement in Cake over fq_codel is replacement of the plain hash function with an 8-way set-associative version. Plain hashes are susceptible to the ‘birthday problem’ in which the probability of hash collision reaches 50% when the table occupancy reaches the square root of the table size (32 flows for 1024 queues), assuming a high-quality hash; we have also found that the hash function fq_codel relies on is suboptimal.”

The “Nat” option “Instructs Cake to perform a NAT lookup before applying flow-isolation rules, to determine the true addresses and port numbers of the packet, to improve fairness between hosts ‘inside’ the NAT.”

“One of the good things cake does … is that it reduces superpackets (GRO offloads mainly) to their packets which does wonders (vs fq_codel) to hold latencies stable.”

Fall back to dropping a packet if necessary for congestion control if ECN not working.

1 Like
7

Thanks for the info @Mark9. In firmware 8.0.0 beta 3, if you go to support.cgi (Login to Web Admin, in the URL bar rename index.cgi to support.cgi then press Enter), you’ll see there is a new “Mitigate bufferbloat” option like this, at the bottom of the page:

image

Please try it out and let us know your result! :slight_smile: You can get the new beta firmware here:

2 Likes
8

This is very exciting! I must be doing something wrong however as I don’t see “Mitigate bufferbloat” on the support.cgi page. I am on 8.0.0b03 build 3594 (Balance 20) and see “Lowest Latency outbound algorithm threshold [click to configure]” at the bottom of the page.

Please advise. Thanks

9

Sorry @Mark9, this is not available yet on Balance 20 since this is currently an experimental feature, we are still working on it. Do you have other models to test?

1 Like
10

I also have the Surf Soho MK3 if that will do the trick. I’ve been considering a Balance One, but don’t need the speed yet, so haven’t had a good reason to pull the trigger.

If the Surf Soho doesn’t do the trick, what models do?

11

Yea Soho MK3 should do the trick! Looking forward to your result. :slight_smile:

1 Like
12

Either I didn’t properly enable Mitigate Bufferbloat, or it didn’t make any improvements as measured by Speed test - how fast is your internet? | DSLReports, ISP Information. My DSL line has a D bufferbloat grade both before and after the mitigation. My Comcast line has a B or C grade before and after the mitigation. I can improve my grade to an A with a Ubiquiti EdgeRouter with their implementation of fq_codel (which they call Smart Queue).

The testing was with 8.0.0b03 build 1397 on my Surf Soho Mk3. I enabled Mitigate Bufferbloat. Afterwards I went to the Dashboard for Apply. I even tried rebooting to see if it made a difference, but it didn’t. Revisiting Mitigate Bufferbloat showed that the setting was “Enable”. I even set my WAN Upload and Download Bandwidth to values 10% less than the ISP provisioned speeds in case those values were used by the mitigation. Did I miss something?

As I will be traveling starting tomorrow until March, I will have limited ability to do more testing. I can remotely run Speed test - how fast is your internet? | DSLReports, ISP Information on my main Comcast line. I won’t be able to reboot as the Peplink router as there is some type of bug where my Comcast WAN spins with a Connecting message forever. My process for fixing this is to unplug my Netgear CM600 modem from the Peplink, power off and on the CM600, and then reconnect to the Peplink (yes, I need to open a case with Peplink sometime, but I was hoping some fixes for similar situations afflicting other Peplink customers would fix my situation). My fallback DSL line is rather slow and has high latency, something I would rather not subject others in our house to (or me when I remotely connect into the home network).

1 Like
13

I’m delighted to see peplink making strides to fix their bufferbloat issues for their customers! With a single checkbox to enable it, though, I worry. For the ISP uplink, the sqm (HTB+fq_codel) code and the later sch_cake require that the underlying framing be detected correctly in order to get nearly 100% of the bandwidth in all circumstances (dsl in particular has all sorts of framing issues, and we have a special docsis mode in cake). The upload/download speeds from the ISP need to be accurately measured before setting

once those three steps are done right, dslreports will tend to report A or A+ scores for bufferbloat, and advanced techniques such as ecn enablement (lossless congestion control) will “just work”. The flent tool (the rrul test is popular) can look into behaviors in more detail.

It’s unknown to me what wifi chipset (above about 40Mbit the bloat generally moves to the wifi) is in the “MK3 version of the Surf SOHO”. Google wifi’s evaluation of their version of the debloating wifi code is here: http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf and has bloat vs rate/range plots at the end to die for.

I’d love to add this router to the bufferbloat.net supported list but more information is needed on how well whatever you’ve done works. Thx for paying attention, however, to addressing this problem, and I hope - if you’ve gone the extra mile to make configuration this easy - that one day you can just have it on by default!

5 Likes
14

Thanks @Mark9, is it possible for us to remotely check your device to get more details (Surf SOHO MK3 with 8.0b3 firmware)?

If possible, please enable Remote Assistance, and send us your Serial Number by creating a support ticket here:
https://contact.peplink.com/secure/create-support-ticket.html

1 Like
16

This is interesting @Dave_Taht. We for sure will look into it further and get back to you when we have some solid field test results come out. Really appreciate your hard works on this matter! :slight_smile:

1 Like
17

I just enabled the Mitigate Bufferbload feature using firmware 8 beta 3 on a Peplink Balance One. I ran a test using https://dslreports.net/speedtest and saw dramatic improvement in bufferbloat: from a “C” to an “A+” - roughly 200msec+ down to about 1msec.

Of note, the setting is a bit confusing: after you click the button to Enable it, you get a message that you have to Apply changes, but there’s no obvious button to click. Near the top of the support.cgi page is a link to the Dashboard, if you click that, you’ll get back to the main UI and can click Apply.

2 Likes
18

I’ve retried the same steps enabling Mitigate Bufferbloat that I did during my original testing with no change, steps which match what soylentgreen also did (successfully). My dslreports Bufferbloat score remains the same whether or not I am enabled or disabled (Surf Soho Mk3, Comcast, Netgear CM600 modem, testing through a LAN hardwired port). In this case, since I am running the test during the day, my grade was a B instead of a C or D at night. I observed that the Bufferbloat is occurring during the Upload phase of the dslreports speed test.

I went through several cycles of Mitigate Bufferbloat Enable and Disable, verifying the setting value was changing on the support.cgi page. I did not reboot and I am not in a position to do this as it will result in my Comcast connection hanging while I am traveling (as explained in a prior posting).

Remote Assistance is enabled. And I have created a ticket.

19

@Mark9

Thank you. Support team will followup with you via the created ticket

1 Like
20

Success! The support team found that I also needed to enable Advanced->QoS->Application->DSL/Cable Optimization. Once I adjusted my WAN download and upload values properly (around 10% below Comcast provisioned values), I was then able to get a Bufferbloat grade of A with Speed test - how fast is your internet? | DSLReports, ISP Information.

FYI, and as is expected with Bufferbloat algorithms, the extra cpu overhead reduces my download throughput on the Surf Soho to around 100Mbps +/- 5Mbps as measured on dslreports (the Surf Soho is normally rated for 120Mbps).

5 Likes
21

Good find - turns out I already had this setting enabled (on my Balance One, it’s under Network / QoS / Application / DSL/Cable Optimization)

2 Likes
22

Is this feature available on my Surf_SOHO device? I would really like to give it a try!

Thanks!

James…