Community discussions

MikroTik App
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

CCR2004 : BGP Benchmarks

Tue May 12, 2020 3:34 am

Hello,

I just received my new CCR2004 and I couldn't wait to put it in the grill
For the record, i also tested few other devices :
- CCR1009 (7g-1c-1s+) v6.47b60
- CCR1016 (12s-1s+) v6.47b60
- RB4011 (igs-rm) v6.47b60

The CCR2004 runs its factory version 6.46.3 since there is no other package available to install.

Image

Test protocol :

Four BIRD (v4 only) daemons with 800k+ routes upstreamed from a CCR1072 located into AS57199
Four BGP sessions to each router, no IGP, nothing more than BGP !

I just waited for the VMs to converge, and i configured the 4 bgp peers in the same time and I used "/ip route count print" to monitor the amount of routes.

Here are the results :

BGP Insertion (4xFullviews, ~3,2M routes) :

1. RB4011 : 3m45s
2. CCR2004 : 5m38s
3. CCR1016 : 10m09s
4. CCR1009 : 10m45s

Impressive results for the RB4011, less for the CCR2004, maybe this is because of the 32bit version of RouterOS on 64bit hardware ?


For the next test, i just deconfigured the 4 peers and waited for the route count to return to almost 0.

BGP Removal (4xFullviews, ~3,2M routes) :

1. CCR1016 : 3m18s
2. CCR1009 : 3m25s
3. RB4011 : 8m25s
4. CCR2004 : 19m58s

And i have to say that I am a little bit disappointed. Why is the 2004 so bad ? Are some hacks or improvements added to tile version missing ?
i hope that future versions of RouterOS 6 (i'd like to be able to use my router before 2021 :D ) will fix this :(

Hugues
 
kos
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 11:48 am

Here you could find a link for 6.46.6:

viewtopic.php?f=2&t=161034

Also a long term version is available:

https://download.mikrotik.com/routeros/ ... 6.45.9.npk
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 12:51 pm

Yes, i am aware of it, i am the author of this topic. You can also see that this current topic is earlier than the answers in the other topic ;)
 
djdrastic
Member
Member
Posts: 367
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 2:07 pm

Disappointed by those 2004 numbers if I have to be honest.The removal times makes it a total no go in my edge routing cases pretty much.

Have you run some bandwidth/latency tests through the CCR2004 to compare it to the 1016's ?
Can it push 2.5+ Gbps of 1500 mtu TCP b/w per stream ?
Last edited by djdrastic on Tue May 12, 2020 2:39 pm, edited 1 time in total.
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 2:21 pm

I didn't run any bandwith tests for now, i trust the Mikrotik's Datasheet :)

A bit better with 6.46.6 :

BGP Insertion (4xFullviews, ~3,2M routes) :
CCR2004 : 1m50s

BGP Removal (4xFullviews, ~3,2M routes) :
CCR2004 : 7m30s

But still slower than a 1009 :(
Last edited by hugues on Tue May 12, 2020 3:01 pm, edited 1 time in total.
 
djdrastic
Member
Member
Posts: 367
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 2:57 pm

Those are much better numbers have to say.

Thanks sir for the update. Mikrotik Datasheet numbers I always look at with a very wary eye.
There tends to be a lot of asterisks and gotchas associated with it.

If it's possible could you do iperf UDP/TCP tests as well as posting CPU/Memory Resource utilization and latency when doing these tests ? I'd love to know how much CPU you're using when doing 10+ Gbps routing as sitting at 99% is effectively useless to me.

The community would really appreciate it getting actual results from the equipment.
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 3:03 pm

BGP Benchmarks are much better with a lower amount of routes :


1 Full (800k routes):
* Insertion : 47 seconds
* Removal : 7 seconds
2 Full (1,6M routes):
* Insertion : 1 minute 9 seconds
* Removal : 39 seconds
3 Full (2,4M routes) :
* Insertion : 1 minute 29 seconds
* Removal : 3 minutes
4 Full (3,2M routes) :
* Insertion : 1 minute 51 seconds
* Removal : 7 minutes 30 seconds


For the bandwidth tests, i don't have much stuff at home, i will try but i can't guarantee anything :)
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 5:09 pm

I simulated a big bgp flap (2 sessions reset + 1 added). It is better than a Tilera, but still miles above a decent router :

it took about 40 minutes to process all the routes to the CCR2004
it took about 1h23 to process all the routes to the CCR1016
It took about 2 minutes to process all the routes to a Cisco ASR9K (with rsp440-se) :P

To be clear : If you experience a big BGP flap, it is still better to reboot all your BGP Core rather than wait for a MikroTik to converge.
(PS : It could be a workaround to let the user flush the table and reset the sessions with a hidden command)
 
djdrastic
Member
Member
Posts: 367
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 6:26 pm

It could be that we'd have to wait for V7 to really see some big BGP performance increases .

We'd have to make a wager: What would get mass released first ? A vaccine for Coronavirus or ROS 7 ?
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 7:32 pm

I'm waiting on my CCR2004 to get here. Then i'm going to benchmark it with our iperf3 and BGP full table performance lab that we've used for MUM Presentations in the past.

We maxed out the CCR1072 when it first came out with 80Gbps of traffic, so we should be able to make the CCR1004 fall over ;-)
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Tue May 12, 2020 9:17 pm

You can reach 80gbps of unrealistic traffic with a 1072, but with real traffic, it begins to lost packets with a fraction of the max throughput. :?
 
Lonecrow
Member Candidate
Member Candidate
Posts: 136
Joined: Mon Dec 24, 2007 4:58 am

Re: CCR2004 : BGP Benchmarks

Wed May 13, 2020 4:21 pm

What realistically can you do before you notice issues? I have issues with the 1072's where they randomly reboot and the watchdog triggers. Maybe 1 time a week or less. No pattern to it and the supout shows nothing.
 
telcouk
newbie
Posts: 31
Joined: Mon Jan 08, 2018 3:50 pm

Re: CCR2004 : BGP Benchmarks

Thu May 14, 2020 12:18 pm

What realistically can you do before you notice issues? I have issues with the 1072's where they randomly reboot and the watchdog triggers. Maybe 1 time a week or less. No pattern to it and the supout shows nothing.
We have 3 1072's in a live environment with no reboot issues, albeit with low traffic levels.
However we have run 1-2Gb during testing and they have been fine.
 
paulct
Member
Member
Posts: 336
Joined: Fri Jul 12, 2013 5:38 pm

Re: CCR2004 : BGP Benchmarks

Thu May 14, 2020 5:53 pm

we have a couple 1072's doing BGP and around 4-5Gbps with no issues. Yes BGP convergence is slow (around 40 peers), but nothing wrong with the 1072.

We will upgrade to the future CCR's and/or CHR soon though as we want to take in a couple full feeds.
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Thu May 14, 2020 6:08 pm

Same here, around 30 CCRs 1072&1036 in production, around 5Gbps in peak, no reboot issues...
 
raffav
Member
Member
Posts: 345
Joined: Wed Oct 24, 2012 4:40 am

Re: CCR2004 : BGP Benchmarks

Thu Jun 04, 2020 8:14 pm

Same here, around 30 CCRs 1072&1036 in production, around 5Gbps in peak, no reboot issues...
is time to test using the new V7 beta 8 :)
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Thu Jun 04, 2020 8:23 pm

My 2004 is already in production, but i still have the CCR100X and the 4011 to lab the v7... For the moment, i'm trying to understand the new CLI...
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: CCR2004 : BGP Benchmarks

Fri Jun 05, 2020 4:15 pm

You can reach 80gbps of unrealistic traffic with a 1072, but with real traffic, it begins to lost packets with a fraction of the max throughput. :?
Have you tried with disabled connection tracking?
You will not have stateful firewall without connection tracking, but the performance is way better.

I had the same problems with CCR1036, especially during moderate DDoS attacks. By disabling connection tracking (or using the raw table for the forwarded traffic), the router now forwards 10gbit DDoS traffic without breaking a sweat.
One caveat: even without connection tracking, ROS becomes totally unresponsive during DDoS SYN Flood attacks (even at few Mbps/Kpps).
The only way I found to be able to handle SYN Flood attacks, was to disable Route Cache (which introduces other problems though...)
 
Abner
just joined
Posts: 14
Joined: Sat Dec 21, 2019 11:31 pm

Re: CCR2004 : BGP Benchmarks

Sat Jun 06, 2020 10:17 am

Here is CPU usage from my ccr2004 router.. 6.47 and ~1M routes

Image
 
telcouk
newbie
Posts: 31
Joined: Mon Jan 08, 2018 3:50 pm

Re: CCR2004 : BGP Benchmarks

Thu Jun 11, 2020 2:16 pm

The only way I found to be able to handle SYN Flood attacks, was to disable Route Cache (which introduces other problems though...)
We have route cache disabled on IPv4, what issues has it introduced that has caused you problems?
 
User avatar
sterling
Member Candidate
Member Candidate
Posts: 111
Joined: Tue Jan 18, 2011 8:55 am
Location: Utah
Contact:

Re: CCR2004 : BGP Benchmarks

Wed Jun 24, 2020 10:28 pm

We all know the CCR2004 is crippled until v7.

Can you/have you tried loading v7 beta on it and re-running the tests for BGP?

I would be very interested in those results.
 
hugues
just joined
Topic Author
Posts: 21
Joined: Wed Feb 28, 2018 11:59 am

Re: CCR2004 : BGP Benchmarks

Thu Jun 25, 2020 12:11 am

No, sorry, my 2004 is in production now :)
 
morf
Member Candidate
Member Candidate
Posts: 182
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR2004 : BGP Benchmarks

Thu Jul 16, 2020 2:27 pm

Hi, everybody. No one else wants to share their tests ?
 
sundalez
just joined
Posts: 11
Joined: Tue Dec 27, 2016 6:42 pm

Re: CCR2004 : BGP Benchmarks

Sun Jul 19, 2020 6:17 pm

Well, I have just been testing the ccr2004 and I must say I'm pretty disappointed.

I haven't even gotten to the more advanced stuff, as im already running into issues with very simple benchmarks.

I have 2 servers, one on each interface with a different vlan on each physical interface.
Then I run a iperf3 form host one to host 2.
running iperf3 with just a single thread gives a result of about 6 Gbits throughput.

I can see on the router it now have 100% load on one core.

With 4 threads on iperf3 performance goes up to about 9 Gbit. and 2 cores are almost maxed out.
'no matter what I do I can't seem to get the router to use more than 2 cores even when I thread the load. ??

and when I add just 1 firewall rule, the performance drops to about 5 Gbit no matter how many threads I throw at it.

this is all with mtu of 1500 and no firewall rules and connection tracking disabled. I have tested on 6.47.1 and also on the new 7 beta 8, same issues

also, if I physically connect the 2 servers through a 10gbit switch, I get 9,3 Gbit on single iperf3 stream, just to exclude server issues

Any ideas ??
 
kos
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Mon Oct 31, 2016 11:51 am

Re: CCR2004 : BGP Benchmarks

Mon Jul 20, 2020 9:41 am

Probably you have to generate far more sessions in order to see the other cores working.
 
sundalez
just joined
Posts: 11
Joined: Tue Dec 27, 2016 6:42 pm

Re: CCR2004 : BGP Benchmarks

Mon Jul 20, 2020 10:01 am

I did try with up to 50 threads, still same results :-(
 
morf
Member Candidate
Member Candidate
Posts: 182
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR2004 : BGP Benchmarks

Tue Jul 21, 2020 5:52 pm

I did try with up to 50 threads, still same results :-(
Released a new beta RouterOS 7. Try it on it, maybe something has changed?
 
Lonecrow
Member Candidate
Member Candidate
Posts: 136
Joined: Mon Dec 24, 2007 4:58 am

Re: CCR2004 : BGP Benchmarks

Thu Jul 23, 2020 7:38 pm

Some of us need to use nat for particulars so we need to keep connection tracking on. Even with a light amount of connection tracking it crashes weekly on our 1072's. Some of our 1072's with it disabled still randomly watchdog reboot. It all started a few versions back.
 
User avatar
sterling
Member Candidate
Member Candidate
Posts: 111
Joined: Tue Jan 18, 2011 8:55 am
Location: Utah
Contact:

Re: CCR2004 : BGP Benchmarks

Thu Jul 23, 2020 9:27 pm

Some of us need to use nat for particulars so we need to keep connection tracking on. Even with a light amount of connection tracking it crashes weekly on our 1072's. Some of our 1072's with it disabled still randomly watchdog reboot. It all started a few versions back.
Yeah, I've seen that as well on some versions of 6.x long term stable on our CCR 1072.
I had to offload most of the connection tracking stuff to a 1036 for now so it wouldn't keep locking up or watchdog rebooting.
I think it may have something to do with connection tracking overload, that is what MT told me when I sent them the support info.
Haven't tried v7 yet on anything, so this is really no surprise that v7 connection tracking is having issues as well.
Someone get back to me when MT has fixed that, because I'm not going to be their beta or even long term (beta) support person on live traffic/clients!
 
Lonecrow
Member Candidate
Member Candidate
Posts: 136
Joined: Mon Dec 24, 2007 4:58 am

Re: CCR2004 : BGP Benchmarks

Mon Jul 27, 2020 4:27 pm

Good idea about using a 1036, however in my environment that isn't possible. I need to keep some of the IP's private to pass through a traffic shaper before it gets to the nat 1072. Issue is we need more than 2x SFP+ ports that a 1036 would have.

How about Mikrotik just fixes the damn connection tracking issues on a $3k router that is supposed to be their flagship router?
 
User avatar
sterling
Member Candidate
Member Candidate
Posts: 111
Joined: Tue Jan 18, 2011 8:55 am
Location: Utah
Contact:

Re: CCR2004 : BGP Benchmarks

Mon Jul 27, 2020 8:56 pm

Yeah, I agree, it should just be fixed.

They wanted me to sit on it and watch while the problem/lockup/reboot happened and do a few things to diagnose.
But I wasn't going to be their tester in my production environment, my customers were already severely impacted by multiple failures and I was losing credibility fast.

I'm not even sure why the 1036 works in place of the 1072, it was just a hunch.

But you could use the 1036 and one of their SFP+ switches to VLAN to so you end up with more "ports" on the 1036.
I do that in a few places and it seems to work just fine, like handing off VPLS tunnels to a VLAN in the more powerful 1036/1072 and using the VLAN on a downstream switch to distribute etc.
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: CCR2004 : BGP Benchmarks

Tue Jul 28, 2020 1:44 am

Probably you have to generate far more sessions in order to see the other cores working.
we did some throughput tests with the CCR2004 - running 6.48b12. it actually performed quite well, we were able to reach peak ipv4 routing throughput of 32Gbps with just 8 streams. as for ipv6, the picture is a bit different...
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2095
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: CCR2004 : BGP Benchmarks

Tue Jul 28, 2020 3:01 am

Probably you have to generate far more sessions in order to see the other cores working.
we did some throughput tests with the CCR2004 - running 6.48b12. it actually performed quite well, we were able to reach peak ipv4 routing throughput of 32Gbps with just 8 streams. as for ipv6, the picture is a bit different...
If only there was IPv6 fastpath....
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: CCR2004 : BGP Benchmarks

Tue Jul 28, 2020 10:29 am

If only there was IPv6 fastpath....
well since there were no rules in firewall, it had the lightest load possible - basically packet forwarding along connected routes.
in terms of PPS it is pretty much understandable you will always get subpar results compared to IPv4. the header size is twice big, so it takes at best twice the time to parse it. but no worries on that, as most other vendors have their v6 performance halved compared to IPv4 forwarding rate. initial impressions from the short test are below.

with bridging on multiple ports (8) we got 950000 per port-pair per direction, so altogether 7.6Mpps.
on a single port pair we could get 3.2Mpps/direction = 6.4Mpps

IPv4 results were 10.4Mpps and 4.2Mpps respectively.
IPv6 numbers were far behind, such as 1.63Mpps and 1.3Mpps.

clearly IPv4 results leave everything else behind. surprisingly the cpu util was kind of balanced almost all the time.
we'll post the finished full test results on yt soon.
 
morf
Member Candidate
Member Candidate
Posts: 182
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR2004 : BGP Benchmarks

Tue Jul 28, 2020 8:45 pm

If only there was IPv6 fastpath....
well since there were no rules in firewall, it had the lightest load possible - basically packet forwarding along connected routes.
in terms of PPS it is pretty much understandable you will always get subpar results compared to IPv4. the header size is twice big, so it takes at best twice the time to parse it. but no worries on that, as most other vendors have their v6 performance halved compared to IPv4 forwarding rate. initial impressions from the short test are below.

with bridging on multiple ports (8) we got 950000 per port-pair per direction, so altogether 7.6Mpps.
on a single port pair we could get 3.2Mpps/direction = 6.4Mpps

IPv4 results were 10.4Mpps and 4.2Mpps respectively.
IPv6 numbers were far behind, such as 1.63Mpps and 1.3Mpps.

clearly IPv4 results leave everything else behind. surprisingly the cpu util was kind of balanced almost all the time.
we'll post the finished full test results on yt soon.
I'll look forward to it
 
User avatar
hero1c
just joined
Posts: 6
Joined: Sun Nov 19, 2017 5:23 pm

Re: CCR2004 : BGP Benchmarks

Thu Aug 06, 2020 3:30 am

Here is CPU usage from my ccr2004 router.. 6.47 and ~1M routes

Image
Hello,

What graphing software do you use?

Thanks, hero1c
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: CCR2004 : BGP Benchmarks

Thu Aug 06, 2020 12:05 pm

Looks like LibreNMS.
 
psannz
Member Candidate
Member Candidate
Posts: 127
Joined: Mon Nov 09, 2015 3:52 pm
Location: Renningen, Germany

Re: CCR2004 : BGP Benchmarks

Thu Aug 06, 2020 2:03 pm

What graphing software do you use?
The graph looks like it's rrdtool. Many monitoring systems use it. OpenNMS, LibreNMS, Cacti, Observium, ...
 
Abner
just joined
Posts: 14
Joined: Sat Dec 21, 2019 11:31 pm

Re: CCR2004 : BGP Benchmarks

Fri Aug 07, 2020 10:52 pm

Here is CPU usage from my ccr2004 router.. 6.47 and ~1M routes

Hello,

What graphing software do you use?



Thanks, hero1c

Observium
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: CCR2004 : BGP Benchmarks

Thu Aug 13, 2020 1:42 pm

BGP Insertion (4xFullviews, ~3,2M routes) :
1. RB4011 : 3m45s
2. CCR2004 : 5m38s
3. CCR1016 : 10m09s
4. CCR1009 : 10m45s

BGP Removal (4xFullviews, ~3,2M routes) :
1. CCR1016 : 3m18s
2. CCR1009 : 3m25s
3. RB4011 : 8m25s
4. CCR2004 : 19m58s


SUGGESTION:
These numbers also should be published on Test results pages for each model
https://mikrotik.com/product/ccr2004_1g ... estresults
 
soccrstar
just joined
Posts: 1
Joined: Sun Aug 23, 2020 6:55 pm

Re: CCR2004 : BGP Benchmarks

Sun Aug 23, 2020 6:57 pm

What program do you use to run these tests if you don't mind me asking.
 
chubbs596
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Fri Dec 06, 2013 6:07 pm

Re: CCR2004 : BGP Benchmarks

Fri Sep 04, 2020 10:50 am

The only way I found to be able to handle SYN Flood attacks, was to disable Route Cache (which introduces other problems though...)
We have route cache disabled on IPv4, what issues has it introduced that has caused you problems?
Im also interested in issues caused by disabling route cache and using BGP
 
andreipop
just joined
Posts: 3
Joined: Thu Jun 24, 2021 12:08 am

Re: CCR2004 : BGP Benchmarks

Thu Jun 24, 2021 12:13 am

BGP Benchmarks are much better with a lower amount of routes :


1 Full (800k routes):
* Insertion : 47 seconds
* Removal : 7 seconds
2 Full (1,6M routes):
* Insertion : 1 minute 9 seconds
* Removal : 39 seconds
3 Full (2,4M routes) :
* Insertion : 1 minute 29 seconds
* Removal : 3 minutes
4 Full (3,2M routes) :
* Insertion : 1 minute 51 seconds
* Removal : 7 minutes 30 seconds


For the bandwidth tests, i don't have much stuff at home, i will try but i can't guarantee anything :)
Hello.
I woud like to build a router from a PC, and would love to do some kind of test on it.

Can you please explain / send me to some resource about how this testing was done?
Thank you.

Andrei
 
markom
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Dec 17, 2009 10:42 pm

Re: CCR2004 : BGP Benchmarks

Mon Sep 13, 2021 10:22 am

Is there some updates on BGP performance with 7.1rc3 (Development) ?

Who is online

Users browsing this forum: No registered users and 15 guests