Page 1 of 1

CCR1009 maxes out at 2gbps?

Posted: Sat Dec 09, 2017 2:55 pm
by bigcw
Using a CCR1009-7G-1C-1S+ in a large NAT scenario. Basically a big version of a home router with around 5,000 subscribers behind it. DHCP, DNS, etc all handled elsewhere, the CCR just does the NAT.

It has a /27 of public IPs on it and NAT rules to spread the subscribers between those to avoid running out of ports.

10G uplink to a CRS317 onto which are VLANs feeding the subscribers and the IP transit feed.

The CCR seems to max out at 2.1gbps. Flat line on the traffic graph at peak times.

There is plenty of capacity in the network presenting the subscribers to the router. It's all 10G in fact to the aggregation switches so no contention issues there. Also plenty of headroom in the upstream network. The CCR's CPU is at 45% and load is spread evenly over the cores so it's not something like the NAT engine only being single threaded.

Is there any reason why this might be happening? License issue perhaps? Or the CCR running out of capacity? (I would have assumed if it was the latter the CPU load would be close to 100%)

Anyone got any tips?

Chris

Re: CCR1009 maxes out at 2gbps?

Posted: Sat Dec 09, 2017 10:38 pm
by InoX
So the wan is sfp+ , how many LAN ports are you using for client traffic? If you are using 2 ports=2gbps

Re: CCR1009 maxes out at 2gbps?

Posted: Sun Dec 10, 2017 12:47 am
by pe1chl
So the wan is sfp+ , how many LAN ports are you using for client traffic? If you are using 2 ports=2gbps
No, it looks like both the WAN and the customers are on the SFP+ on different VLANs.

I have no idea what the limits are, I run CCR1009 only up to 1Gbps and without NAT... but with lots of other filter and mangle rules.
Of course he could get a CCR1036 instead.

Re: CCR1009 maxes out at 2gbps?

Posted: Mon Dec 11, 2017 11:24 am
by bigcw
So the wan is sfp+ , how many LAN ports are you using for client traffic? If you are using 2 ports=2gbps
Everything is on the SFP+. It is done with two VLANs; clients in one and WAN in the other.
Of course he could get a CCR1036 instead.
Here's the thing: the CPU is nowhere near max'ed out on the 1009 so will a 1036 actually fix the problem? OK so I could bring the clients in on a separate SFP+ to the WAN but I am not sure that gains me anything.

Chris

Re: CCR1009 maxes out at 2gbps?

Posted: Mon Dec 11, 2017 11:41 am
by bigcw
Some data collected last night. Note that a tweak in the client access network has allowed us to squeeze a bit more performance out of it (around 2.8gbps max) but there is definitely a 'flat top' to the graph which suggests to me that something is being pushed to it's limit somewhere in the network.

Re: CCR1009 maxes out at 2gbps?

Posted: Mon Dec 11, 2017 11:43 am
by pe1chl
I'm not sure either... I don't know what is the limiting factor, I never drive the CCR's to saturation because my internet links are too slow for that.
45% load is a figure I have never seen on a CCR I run... but with NAT my guess would be that the limit would be a mix between the amount of traffic and the number of new sessions per second, so there would not be a fixed capacity cap in Gbps.
Maybe there are others who have experience with a setup like that.

Edit: you made a post while I typed that and the capacity cap is not as flat as I guessed.

Re: CCR1009 maxes out at 2gbps?

Posted: Mon Dec 11, 2017 12:32 pm
by bigcw
Mikrotik support: is there any way of seeing how many NAT sessions are open?

Chris

Re: CCR1009 maxes out at 2gbps?

Posted: Mon Dec 11, 2017 12:57 pm
by pe1chl
This forum is not a way to ask questions to MikroTik Support.
For that you need to mail support@mikrotik.com

Re: CCR1009 maxes out at 2gbps?

Posted: Mon Dec 11, 2017 1:18 pm
by cdiedrich
Mikrotik support: is there any way of seeing how many NAT sessions are open?

Chris
/ip firewall connections print count-only where srcnat=yes
When you omit the "count-only" parameter, you will see all connections in a list.

-Chris

Re: CCR1009 maxes out at 2gbps?

Posted: Fri Dec 15, 2017 12:26 pm
by bigcw
/ip firewall connections print count-only where srcnat=yes
When you omit the "count-only" parameter, you will see all connections in a list.
Thanks for that, Chris

We replaced the 1009 with a 1036 2S+ yesterday afternoon. CPU load already massively reduced (as expected as it has 3x as many cores!). Hopefully find out tonight whether there is any improvement in maximum throughput.
graph_image.png
graph_image (1).png

Re: CCR1009 maxes out at 2gbps?

Posted: Fri Dec 15, 2017 4:35 pm
by pe1chl
Looks good!
Did you also separate the input and output over the 2 different SFP+ or is it still using a VLAN setup?

Re: CCR1009 maxes out at 2gbps?

Posted: Sun Dec 17, 2017 12:29 am
by InoX
SFP+ ports 1

Re: CCR1009 maxes out at 2gbps?

Posted: Wed Dec 20, 2017 9:00 pm
by bigcw
Looks good!
Did you also separate the input and output over the 2 different SFP+ or is it still using a VLAN setup?
Yes, we now have WAN and LAN on separate SFP+. Not seeing any improvement so far unfortunately.

Re: CCR1009 maxes out at 2gbps?

Posted: Wed Dec 20, 2017 10:34 pm
by kujo
Attach output, please:
/tool profile cpu=all


Yours respectfully!

Re: CCR1009 maxes out at 2gbps?

Posted: Thu Dec 21, 2017 7:31 am
by aholmes
You might be capping out one or two CPU cores per-VLAN. Maybe a good idea to check that your interface queues are set to multi-queue-ethernet-default?

Re: CCR1009 maxes out at 2gbps?

Posted: Thu Dec 21, 2017 12:50 pm
by n21roadie
You might be capping out one or two CPU cores per-VLAN. Maybe a good idea to check that your interface queues are set to multi-queue-ethernet-default?
I note on my CCR1009 is that VLAN's queue interface has "no-queue" and so in queues clients pppoe interface have "no-queue", must compare this evening if "multi-queue-ethernet-default" will make any difference to performance?

Re: CCR1009 maxes out at 2gbps?

Posted: Thu Dec 21, 2017 2:17 pm
by pe1chl
Those are not really interfaces like ethernet that have a hardware queue, an optional software queue, and an interrupt handler that moves packets between them.
VLAN and PPPoE are just putting on a header and moving data to the next layer (normally ethernet), and so they do not require a queue by themselves.

Re: CCR1009 maxes out at 2gbps?

Posted: Tue Dec 26, 2017 10:48 pm
by pospanko
Do you maybe use bridge?
We have dramatic falling apart at 2Gbps on CCR1072 using RSTP. When we set everything to STP traffic started to run normaly. Problem was that hosts table was falling apart, losing records. Of course, no support response from MT...

Re: CCR1009 maxes out at 2gbps?

Posted: Fri Jul 05, 2019 5:11 am
by Alessio Garavano
Hi @Bigcw, you have a solution for this problem? i have the same problem in a CCR1036 with Mikrotik SFP+ connected to our carrier... we have 3Gbps but at 2,2Gbps is limited and all disabled(firewall, mangles, etc etc)!

Re: CCR1009 maxes out at 2gbps?

Posted: Fri Jul 05, 2019 9:23 am
by normis
I maybe don't understand something, but the CCR1009-7G-1C-1S+ only has ONE (1) SFP+ port.

Re: CCR1009 maxes out at 2gbps?

Posted: Fri Jul 05, 2019 10:44 am
by pe1chl
I maybe don't understand something, but the CCR1009-7G-1C-1S+ only has ONE (1) SFP+ port.
Halfway he changed to a CCR1036 to try to solve the problem...

Re: CCR1009 maxes out at 2gbps?

Posted: Thu Jul 11, 2019 7:17 pm
by Alessio Garavano
I maybe don't understand something, but the CCR1009-7G-1C-1S+ only has ONE (1) SFP+ port.
Halfway he changed to a CCR1036 to try to solve the problem...
and? was solved? we have a CCR1036 doing the same problem...

Re: CCR1009 maxes out at 2gbps?

Posted: Fri Jul 12, 2019 9:52 pm
by maxsaf
and? was solved? we have a CCR1036 doing the same problem...
A had same problem doing bandwidth test on 1009. Solved by changing MTU to something over 8000