Community discussions

 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Severe Performance Drop RB3011

Fri Oct 12, 2018 6:58 am

I have a RB3011 in a location with Cable Internet. We just had the location upgraded from 300mbit download and 20mbit upload to 400mbit download and 20mbit upload. Only the download speed was upgraded.

Over recent weeks I have done periodic speed tests and hit about 120mbit over WiFi. In past times I’ve been able to hit the full 300mbit and actually beyond up to about 350mbit. For some reason speeds haven’t quite been as great but I attributed this to just WiFi issues.

Today when we got the speed upgraded, we did some tests and I had hard wired my laptop to the router and only got an average of about 215mbit, 250mbit maybe at best.

I did a test by going directly to the modem and got full speed, in fact, hitting up to 480mbit down. Quite a difference!

I’m doing some tests I disabled some firewall rules which improved some, but not great.

I’m doing further tests I disabled all my mangle rules, and what do you know, I was able to hit the full 400+mbit behind the 3011.

Simply enabling one mangle rule, any rule, reduced the speed test to an average of about 215mbit.

The 3011 is well capable of beyond 200mbit. What is making the 3011 only perform at this lower speed?

The mangle rules are simply marking packets for our queue tree and QoS.

Why is this having such a huge impact on our speed?

While performing a speed test I’ve checked CPU Usage and it’s hardly ever at 30%. So much more power available.

The 3011 is on the latest software and firmware currently available.

We have about 5 VLANs with DHCP servers for each. Some firewall rules to drop packets between certain VLANs.

Other simple firewall rules to allow IPSec/LT2P.

Some other NAT rules for our PBX and web server.

Nothing super complicated.

Any help is appreciated.
 
User avatar
vecernik87
Member Candidate
Member Candidate
Posts: 217
Joined: Fri Nov 10, 2017 8:19 am

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 9:04 am

what kind of speed test is it? If it is TCP based, how many paralel streams/connections? If only one, it might be the issue as such test is strongly affected by latency. Adding mangle rule is going to introduce slight delay as the packet must be processed in another block of code.

Just to make sure - do you have fasttrack rule set up?

If you share your /export hide-sensitive it might be easier to spot any possible issue straight away. Feel free to use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1. (credit for this sentence goes to sindy)
 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 3:15 pm

The test is SpeedTest.net. I’m not sure if they are UDP or TCP.

Regardless of this though, the latency doesn’t change when I enable/disable the mangle rules (we’re only talking about less than 10 of them in total).

Latency remains real well at about 9-14ms depending on the test.

Overall throughout changes big time.

With rules disabled I get all we are allowed to get at about 480mbit down.

With even just one mangle rule enabled a huge drop to about 215mbit.

Both times latency is excellent and seemingly unaffected.

I disabled fastrack in IP -> Settings last night when I was trying some stuff but it made no difference. I don’t have any fastrack rules either, though.

I’m not at my computer right now but I will see what I can do about piston the config.
 
sindy
Forum Guru
Forum Guru
Posts: 2406
Joined: Mon Dec 04, 2017 9:19 pm

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 5:54 pm

The test is SpeedTest.net. I’m not sure if they are UDP or TCP.
They are multi-stream TCP, testing the download direction first using four streams and then the upload one using another four streams.
With rules disabled I get all we are allowed to get at about 480mbit down.
This is due to one type of optimization - if there are no rules at all in the firewall, the firewall processing is skipped completely.
With even just one mangle rule enabled a huge drop to about 215mbit.

I disabled fastrack in IP -> Settings last night when I was trying some stuff but it made no difference. I don’t have any fastrack rules either, though.
Fasttracking is another kind of optimization, where you skip most of firewall processing in a controlled way for most mid-connection packets, so only the packets establishing the connection and every n-th mid-connection packet are handled by all stages of the firewall. Without fasttracking, the CPU may be insufficient to handle the traffic, depending on the RB model. The bad news is that fasttracking is incompatible with mangling (and IPsec policy matching) but the other way round - setting up a mangle rule does not disable fasttracking for all (which is good), you just get unexpected behaviour if you use both without taking additional measures. The correct way to make the two coexist is described here; if the /ip route rules with their limited number of match conditions are sufficient to cover your policy routing needs, you can use them instead of mangle rules and fasttracking will still work without any extra measures to take.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 8:46 pm

Thank you Sindy. I will take a look at that and do some testing.

I'm still not sure though why the performance is dropping so badly. CPU usage is not above 30% during that test. Even if the CPU was processing all those packets, there is plenty of processing power available. I would expect slow downs if the CPU usage was at 100%, right? Please correct me if my understanding of this is incorrect.

At the end of the day, I am trying to use the mangle to mark packets from specific groups of nodes within the network to prioritize and shape bandwidth.

My understanding to do this correctly is:

Add mangle rule to mark connections FROM IP (Upload traffic), passthrough = yes
Add mangle rule to mark packets matching connection above, passthrough = no

Add mangle rule to mark connections TO IP (Download traffic), passthrough = yes
Add mangle rule to mark packets matching connection above, passthrough = no

Then going to the queue tree, lets say we're starting from scratch, we add:

Global Upload queue
Global Download queue

Then we add:

Upload queue for marked packets from above (Of course setting any max limits etc) to the Global Upload queue
Download queue for marked packets from above (Of course setting any max limits etc) to the Global Download queue

Essentially this method works great. This way I can assign a priority accordingly as well as any bandwidth limitations.

In the same manner for our public WiFi which is on a separate VLAN, I mark connections from the public IP range, say 172.31.0.0/24, for example. Then mark their packets from that connection.

The queue is similar as described earlier but I now have the bandwidth throttle for the entire group to have a max and set it to a PCQ type of queue.

What I have described here is all I am trying to accomplish but unfortunately lose half our available bandwidth. So I'm trying to figure out why. There is nothing I can think of that should be affecting bandwidth this much, right? Or are the 3011's just not as good as I think they are?

I am considering a complete factory reset and starting over. It seems that something is just not right here. I appreciate your help in trying to dig in to this.
 
sindy
Forum Guru
Forum Guru
Posts: 2406
Joined: Mon Dec 04, 2017 9:19 pm

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 9:59 pm

Sadly, queueing is incompatible with fasttracking as well, for obvious reasons. So if you need to shape the traffic, pour in more horsepower, which means to replace the 3011 with something that can handle the requirements.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 10:17 pm

What would you suggest?

Going off of MikroTik's test results I figured the 3011 would be more than sufficient:

Their test in routing with 25 ip filter rules shows a result of 2,453.1 Mbps. Granted, this is not with queuing. But is queuing really this hard on the available horsepower?

Would Simple Queuing be less intensive? I don't like it though since I can't use address lists. But if thats the solution then I'd have to make it work.
 
User avatar
pcunite
Long time Member
Long time Member
Posts: 601
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: Severe Performance Drop RB3011

Fri Oct 12, 2018 10:34 pm

What would you suggest?

The new RB4011, only about $200.
 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 4:08 am

I've had my eye on the 4011, sadly it's not quite available just yet.
 
mducharme
Trainer
Trainer
Posts: 609
Joined: Tue Jul 19, 2016 6:45 pm

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 4:45 am

Going off of MikroTik's test results I figured the 3011 would be more than sufficient:

Their test in routing with 25 ip filter rules shows a result of 2,453.1 Mbps. Granted, this is not with queuing. But is queuing really this hard on the available horsepower?
For test results that approximate real world performance, look at 25 ip filter rules with 512 byte packet size (not 1518 byte packet size). With those test results, the 3011 is only capable of 836.0 Mbps with essentially a default config plus maybe a few more rules. Adding more rules etc are going to drop that rate from the 836 Mbps figure even lower.
 
Paternot
Member
Member
Posts: 422
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 5:41 am

Going off of MikroTik's test results I figured the 3011 would be more than sufficient:

Their test in routing with 25 ip filter rules shows a result of 2,453.1 Mbps. Granted, this is not with queuing. But is queuing really this hard on the available horsepower?
For test results that approximate real world performance, look at 25 ip filter rules with 512 byte packet size (not 1518 byte packet size). With those test results, the 3011 is only capable of 836.0 Mbps with essentially a default config plus maybe a few more rules. Adding more rules etc are going to drop that rate from the 836 Mbps figure even lower.
But his problem is getting only 120 Mbps - about one eight of the 512 byte/ 25 ip rules scenario. I know one can always use something CPU heavy, and get this result. But seems... excessive, in this case. Don't You think?
 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 6:18 am

That is exactly my point. Adding one simple mangle rule throws everything off.

I’ve had better results using an older 450G! Although I did use Simple Queues back then.

Something just isn’t right but I can’t put my finger on it. CPU usage never really goes above 30%.
 
ChildOTK
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 99
Joined: Sat Nov 25, 2006 7:40 am

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 6:19 am

I think i’m just going to have to do a factory reset and start over.
 
Kindis
Member Candidate
Member Candidate
Posts: 195
Joined: Tue Nov 01, 2011 6:54 pm

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 9:24 am

I have a 3011 and do not have this issue. Can you export your mangle rule? I can see if I can test it.
 
sindy
Forum Guru
Forum Guru
Posts: 2406
Joined: Mon Dec 04, 2017 9:19 pm

Re: Severe Performance Drop RB3011

Sat Oct 13, 2018 10:13 am

A single mangle rule may just activate some heavy processing which is not done without that rule, so yes, without seeing the actual configuration and pinpoint to that rule it is hard to guess.

Plus if it is as you describe, there is no guarantee that the 4011 will not fall into the same trap, except that it would probably perform better in absolute figures. It uses the same architecture. And so does e.g. hAP ac², so it would be easy to test e.g. for me if I had a fat enough pipe to the net.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.

Who is online

Users browsing this forum: No registered users and 59 guests