Community discussions

 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

CCR1009-7G-1C Performance

Thu Jun 27, 2019 4:53 pm

I've got issues with a CCR1009 on our network edge which maxes out at around ~500mbps. CPU Load is usually around 5-10% so it doesn't appear to anywhere near maxed out.

On the router there's OSPF, PPPoE Server with ~10 clients, DHCP server and two hotspot servers on separate VLANs. Firewall is fairly simple with ~80 rules (inc hotspot) and 3 SRC-NAT rules.

Currently running 6.44.3 but older versions have been tried as well. Does anyone have any ideas what is limiting the throughput?
 
paulct
Member Candidate
Member Candidate
Posts: 286
Joined: Fri Jul 12, 2013 5:38 pm

Re: CCR1009-7G-1C Performance

Thu Jun 27, 2019 5:18 pm

How are you testing this limit?
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Thu Jun 27, 2019 5:54 pm

That's both real world usage with 1000+ users and Btest server.
 
paulct
Member Candidate
Member Candidate
Posts: 286
Joined: Fri Jul 12, 2013 5:38 pm

Re: CCR1009-7G-1C Performance

Thu Jun 27, 2019 6:30 pm

Rather test via IPerf, using a powerful PC on either side of the router. You might be maxing out the CPU using btest.
The CCR1009 should be able to push more than 500Mbps.
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Thu Jun 27, 2019 6:51 pm

I have previously done that with similar results. It is possible to push to about 600mbps with absolutely no other devices connected through the router and with a high stream count. I can guarantee though that the testing methods are not the issue. Something causes a bottleneck at about 500mbps. I can find other references on the forums as well but no real solutions. I've enabled fast track today but there's no measurable difference.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C Performance

Thu Jun 27, 2019 8:07 pm

CPU Load is usually around 5-10% so it doesn't appear to anywhere near maxed out.
Always make sure you check the CPU load in Tools->Profile and checking separately per CPU (selection "all").
The CPU load reported by default is the total for all CPUs so in this 9-core system when 1 core is 100% loaded and the other 8 are almost idle the reported load will be 11% but the system may still be CPU limited.
In that case, find what is saturating that single core and see if you can reconfigure the router to avoid that single-thread bottleneck.
(there are some techniques for that, depending on what it is)
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 10:31 am

Always make sure you check the CPU load in Tools->Profile and checking separately per CPU (selection "all").
That is what I am using. Screenshot attached. Currently about 200mbps going through this router.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 10:40 am

Ok so it is considerably more loaded at 200 Mbps than you said earlier!
At 200 Mbps it is like 28% loaded so it is not surprising you get problems at about 500.
But the good thing is that the load appears to be reasonably distributed across the cores.

Click twice on the Usage column to get the top usage and work from there.
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 10:52 am

The CPU load there is actually at the higher end of what I've seen. This screenshot is a lot closer to normal and overall is steady at 5%.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 11:47 am

But that was probably at a lighter load?
Also understand that when you run a connection-tracking firewall your load will be a lot higher when you have a lot of new connections.
It can easily be that the router passes 500 Mbps of continuous data with almost no load, but cannot handle 500 Mbps worth of new connections e.g. resulting from a port scan or DDoS.
On a network edge it is often not a good idea to have a connection-tracking firewall in the forward path. You may want to have it on input/output only.
It all depends on the use case.

Try to do the profile again when it has the loading similar to the previous screenshot and see what is the next load below "firewall", maybe it provides some hint what else could be loading it.
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 1:17 pm

Can this be improved with a config change on the router then?
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 3:51 pm

That is impossible to know because you did not include the config. However, you can research that yourself as well.
There are a couple of common mistakes that can take a lot of performance, you can find some MUM presentations about that in the MikroTik YouTube channel and also on the forum.
When you have analyzed a bit more where the CPU load is, you can focus on the parts of the config that need improvement.
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 4:06 pm

I've attached the firewall config if you want to have a look. Based on CPU usage this is a firewall issue despite it not maxing the CPU at 100%. I've spent a lot of time researching possible causes but not got anywhere. I can find references to similar limitations going back years but not really any resolution.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 5:27 pm

There are some points from those presentations that you can apply:
- do not use Masquerade for any other cases than src-nat on a dynamic address (e.g. typical home router)
instead use a src-nat with (one of your) external address as a to-address.
- it looks like you do not completely understand the chain=input and chain=forward difference.
there are many rules on chain=input that make no sense, fortunately many of them disabled. you can clean this up.
even for a dst-nat you don't put things in chain=input, but only in chain=forward.
- when you have one specific path where you want to block a lot of different things, but do not want to bother the
router to do those checks on all traffic, consider putting those rules in a separate chain. put a rule in chain=forward
which matches the input interface (or -list) and then jumps to your custom chain (action=jump jump-target=forward-incoming)
and then put your long list of rules in chain=forward-incoming. this can also make things a lot clearer.
however, in a connection tracking setup that makes less of a difference because you accept established,related immediately.
 
craigm
just joined
Topic Author
Posts: 11
Joined: Tue Dec 22, 2015 3:33 pm

Re: CCR1009-7G-1C Performance

Fri Jun 28, 2019 5:53 pm

Thanks for the advise so far.
- do not use Masquerade for any other cases than src-nat on a dynamic address (e.g. typical home router)
instead use a src-nat with (one of your) external address as a to-address.
The main IP range is using src-nat. The two hotspots which have very low usage are using masquarade. There's no particular reason so will change those.
- it looks like you do not completely understand the chain=input and chain=forward difference.
there are many rules on chain=input that make no sense, fortunately many of them disabled. you can clean this up.
even for a dst-nat you don't put things in chain=input, but only in chain=forward.
Are there any specific examples? Most of the rules are from Mikrotik recommendations. I assume the
- when you have one specific path where you want to block a lot of different things, but do not want to bother the
router to do those checks on all traffic, consider putting those rules in a separate chain. put a rule in chain=forward
which matches the input interface (or -list) and then jumps to your custom chain (action=jump jump-target=forward-incoming)
and then put your long list of rules in chain=forward-incoming. this can also make things a lot clearer.
however, in a connection tracking setup that makes less of a difference because you accept established,related immediately.
That's something I hadn't spotted. A lot of the config has been imported from various templates we use. It looks like at some point some of the rules that should be on their own chain have ended up on the input chain. Will get those changed.

Who is online

Users browsing this forum: No registered users and 11 guests