MikroTik

Posted: **Sat Dec 09, 2017 2:55 pm**

Using a CCR1009-7G-1C-1S+ in a large NAT scenario. Basically a big version of a home router with around 5,000 subscribers behind it. DHCP, DNS, etc all handled elsewhere, the CCR just does the NAT.

It has a /27 of public IPs on it and NAT rules to spread the subscribers between those to avoid running out of ports.

10G uplink to a CRS317 onto which are VLANs feeding the subscribers and the IP transit feed.

The CCR seems to max out at 2.1gbps. Flat line on the traffic graph at peak times.

There is plenty of capacity in the network presenting the subscribers to the router. It's all 10G in fact to the aggregation switches so no contention issues there. Also plenty of headroom in the upstream network. The CCR's CPU is at 45% and load is spread evenly over the cores so it's not something like the NAT engine only being single threaded.

Is there any reason why this might be happening? License issue perhaps? Or the CCR running out of capacity? (I would have assumed if it was the latter the CPU load would be close to 100%)

Anyone got any tips?

Chris

Posted: **Sat Dec 09, 2017 10:38 pm**

So the wan is sfp+ , how many LAN ports are you using for client traffic? If you are using 2 ports=2gbps

Posted: **Sun Dec 10, 2017 12:47 am**

So the wan is sfp+ , how many LAN ports are you using for client traffic? If you are using 2 ports=2gbps

No, it looks like both the WAN and the customers are on the SFP+ on different VLANs.

I have no idea what the limits are, I run CCR1009 only up to 1Gbps and without NAT... but with lots of other filter and mangle rules.
Of course he could get a CCR1036 instead.

Posted: **Mon Dec 11, 2017 11:24 am**

So the wan is sfp+ , how many LAN ports are you using for client traffic? If you are using 2 ports=2gbps

Everything is on the SFP+. It is done with two VLANs; clients in one and WAN in the other.

Of course he could get a CCR1036 instead.

Here's the thing: the CPU is nowhere near max'ed out on the 1009 so will a 1036 actually fix the problem? OK so I could bring the clients in on a separate SFP+ to the WAN but I am not sure that gains me anything.

Chris

Posted: **Mon Dec 11, 2017 11:41 am**

Some data collected last night. Note that a tweak in the client access network has allowed us to squeeze a bit more performance out of it (around 2.8gbps max) but there is definitely a 'flat top' to the graph which suggests to me that something is being pushed to it's limit somewhere in the network.

Posted: **Mon Dec 11, 2017 11:43 am**

I'm not sure either... I don't know what is the limiting factor, I never drive the CCR's to saturation because my internet links are too slow for that.
45% load is a figure I have never seen on a CCR I run... but with NAT my guess would be that the limit would be a mix between the amount of traffic and the number of new sessions per second, so there would not be a fixed capacity cap in Gbps.
Maybe there are others who have experience with a setup like that.

Edit: you made a post while I typed that and the capacity cap is not as flat as I guessed.

Posted: **Mon Dec 11, 2017 12:32 pm**

Mikrotik support: is there any way of seeing how many NAT sessions are open?

Chris

Posted: **Mon Dec 11, 2017 12:57 pm**

This forum is not a way to ask questions to MikroTik Support.
For that you need to mail support@mikrotik.com

Posted: **Mon Dec 11, 2017 1:18 pm**

Mikrotik support: is there any way of seeing how many NAT sessions are open?

Chris

/ip firewall connections print count-only where srcnat=yes

When you omit the "count-only" parameter, you will see all connections in a list.

-Chris

Posted: **Fri Dec 15, 2017 12:26 pm**

Code: Select all
/ip firewall connections print count-only where srcnat=yes
When you omit the "count-only" parameter, you will see all connections in a list.

Thanks for that, Chris

We replaced the 1009 with a 1036 2S+ yesterday afternoon. CPU load already massively reduced (as expected as it has 3x as many cores!). Hopefully find out tonight whether there is any improvement in maximum throughput.

graph_image.png

graph_image (1).png

Posted: **Fri Dec 15, 2017 4:35 pm**

Looks good!
Did you also separate the input and output over the 2 different SFP+ or is it still using a VLAN setup?

Posted: **Sun Dec 17, 2017 12:29 am**

SFP+ ports 1

Posted: **Wed Dec 20, 2017 9:00 pm**

Looks good!
Did you also separate the input and output over the 2 different SFP+ or is it still using a VLAN setup?

Yes, we now have WAN and LAN on separate SFP+. Not seeing any improvement so far unfortunately.

Posted: **Wed Dec 20, 2017 10:34 pm**

Attach output, please:
/tool profile cpu=all

Yours respectfully!

Posted: **Thu Dec 21, 2017 7:31 am**

You might be capping out one or two CPU cores per-VLAN. Maybe a good idea to check that your interface queues are set to multi-queue-ethernet-default?

Posted: **Thu Dec 21, 2017 12:50 pm**

You might be capping out one or two CPU cores per-VLAN. Maybe a good idea to check that your interface queues are set to multi-queue-ethernet-default?

I note on my CCR1009 is that VLAN's queue interface has "no-queue" and so in queues clients pppoe interface have "no-queue", must compare this evening if "multi-queue-ethernet-default" will make any difference to performance?

Posted: **Thu Dec 21, 2017 2:17 pm**

Those are not really interfaces like ethernet that have a hardware queue, an optional software queue, and an interrupt handler that moves packets between them.
VLAN and PPPoE are just putting on a header and moving data to the next layer (normally ethernet), and so they do not require a queue by themselves.

Posted: **Tue Dec 26, 2017 10:48 pm**

Do you maybe use bridge?
We have dramatic falling apart at 2Gbps on CCR1072 using RSTP. When we set everything to STP traffic started to run normaly. Problem was that hosts table was falling apart, losing records. Of course, no support response from MT...

Posted: **Fri Jul 05, 2019 5:11 am**

Hi @Bigcw, you have a solution for this problem? i have the same problem in a CCR1036 with Mikrotik SFP+ connected to our carrier... we have 3Gbps but at 2,2Gbps is limited and all disabled(firewall, mangles, etc etc)!

Posted: **Fri Jul 05, 2019 9:23 am**

I maybe don't understand something, but the CCR1009-7G-1C-1S+ only has ONE (1) SFP+ port.

Posted: **Fri Jul 05, 2019 10:44 am**

I maybe don't understand something, but the CCR1009-7G-1C-1S+ only has ONE (1) SFP+ port.

Halfway he changed to a CCR1036 to try to solve the problem...

Posted: **Thu Jul 11, 2019 7:17 pm**

I maybe don't understand something, but the CCR1009-7G-1C-1S+ only has ONE (1) SFP+ port.
Halfway he changed to a CCR1036 to try to solve the problem...

and? was solved? we have a CCR1036 doing the same problem...

Posted: **Fri Jul 12, 2019 9:52 pm**

and? was solved? we have a CCR1036 doing the same problem...

A had same problem doing bandwidth test on 1009. Solved by changing MTU to something over 8000

Posted: **Wed Sep 25, 2019 12:24 pm**

Same problem on CCR1072.

Posted: **Wed Sep 25, 2019 1:35 pm**

The internal "graphing" tool cannot display more than 2.1Gbps. But that does not mean the router cannot handle more.
You need to use an external graphing tool (with proper configuration to use 64-bit values or to poll more often) to be able to graph higher rates.
In your graph you can add the small peaks you see at the bottom above the limit line.

Posted: **Wed Sep 25, 2019 1:42 pm**

The internal "graphing" tool cannot display more than 2.1Gbps. But that does not mean the router cannot handle more.
You need to use an external graphing tool (with proper configuration to use 64-bit values or to poll more often) to be able to graph higher rates.
In your graph you can add the small peaks you see at the bottom above the limit line.

Thanks!
Perhaps you `re right

Posted: **Wed Sep 25, 2019 3:48 pm**

to estimate CPU load in devices with multiple CPU cores, do not only check average CPU usage, better check most loaded CPU core, its a better metric of actual usage and possible remaining resources.

MikroTik

CCR1009 maxes out at 2gbps?

CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?

Re: CCR1009 maxes out at 2gbps?