Individual Bandwidth Limit & Latency

We have been playing with the individual bandwidth limits on users.
It works fine when doing a speed test. Example: we limit it to 5mb down and 5 mb up and it tests at those speeds.
The only issue is the latency goes to 3000 ms when it reaches that limit…

So if we cap people to 5mb and they start to download a file while working, they can’t do anything else because of buffer bloat…

How can we fix?