Community discussions

MikroTik App
 
jml
newbie
Topic Author
Posts: 39
Joined: Wed May 15, 2013 3:22 am

CCR IPSec performance

Fri Aug 08, 2014 6:20 am

Does anyone have stats on the CCRs for IPSec throughput?

Thanks.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26318
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: CCR IPSec performance

Fri Aug 08, 2014 9:21 am

max throughput 3.2Gbps with 34 tunnels (full duplex)

1.8Gbps with 16 tunnels (full duplex)
820Mbps with one GRE over IpSec tunnel (full duplex)

--CCR1009--

1.6Gbps with 8 tunnels (full duplex)
520Mbps with one GRE over IpSec tunnel (full duplex)



Tested with traffic-generator and 1470byte packets.
 
i4jordan
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Mon Sep 02, 2013 1:42 am

Re: CCR IPSec performance

Sat Aug 09, 2014 12:53 pm

Normis,

Thank you for the numbers. It helps a lot in designing VPN networks.

Do you also have some numbers on the 'older' RB1100AHx2 models?

I'd like to know speed of the ipsec tunnels and also the GRE+ipsec speed.

Other question, which is fatser: GRE+ipsec or IPIP+ipsec.

Thank you!
 
Petrovich
just joined
Posts: 6
Joined: Thu Feb 05, 2015 2:17 pm

Re: CCR IPSec performance

Mon Feb 09, 2015 2:49 pm

max throughput 3.2Gbps with 34 tunnels (full duplex)

520Mbps with one GRE over IpSec tunnel (full duplex)



Tested with traffic-generator and 1470byte packets.
Could you please provide your settings for this tunnel.
I did not manage to get even 200Mbit/s on CCR-1036
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR IPSec performance

Mon Feb 09, 2015 2:51 pm

What was your test procedure?
 
Petrovich
just joined
Posts: 6
Joined: Thu Feb 05, 2015 2:17 pm

Re: CCR IPSec performance

Fri Feb 13, 2015 4:42 pm

What was your test procedure?
That was two CCR-1036-8G-2S+ instances connected with 10Gbit/s link. GRE over IPSEC. Confuguration is almost default, you can find it here (just not to replear myselft).
http://forum.mikrotik.com/viewtopic.php ... 92#p467392

I had two laptops with gigabit ethernet adapter, each one was connected to the corresponding CCR. The test was to download 10Gbyte file from one laptop to another .
The first try was to download file over GRE without encyption. Speed was exactly 1Gbit/s.

Second step was to turn on ipsec with config provided. With aes-256 I had up to 80Mbit/s, with aes-128 without authentication I had up to 150Mbit/s
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 730
Joined: Tue Aug 25, 2009 12:01 am

Re: CCR IPSec performance

Sat Feb 14, 2015 2:26 am

Sounds damn close to what I got... And they kept telling me I was wrong.
 
ivan07
newbie
Posts: 26
Joined: Wed Mar 04, 2015 2:57 am

Re: CCR IPSec performance

Wed Mar 11, 2015 8:12 pm

Sounds damn close to what I got... And they kept telling me I was wrong.
What numbers are correct? :)
Maybe there is something wrong with your laptops?
I do not like the idea to download some file in such tests...
 
JanezFord
Member Candidate
Member Candidate
Posts: 269
Joined: Wed May 23, 2012 10:58 am

Re: CCR IPSec performance

Wed Mar 11, 2015 8:52 pm

I do not like the idea to download some file in such tests...
Why not? It's a real world test ... after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.

JF.
 
djdrastic
Member
Member
Posts: 367
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR IPSec performance

Wed Mar 11, 2015 9:01 pm

130-150 Meg is where I was maxing out as well on various Ros 6 versions with the CCRs.
All our private tunnels are still being terminated by our ancient (but fast) 1100AXH2's.
 
ivan07
newbie
Posts: 26
Joined: Wed Mar 04, 2015 2:57 am

Re: CCR IPSec performance

Wed Mar 11, 2015 10:31 pm

Why not? It's a real world test ...
Because first of all this is a testing i/o of your laptops and then network devices :)
after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.
IMHO if we are discussing the performance of the CCR series, we have to test these devices in the first place, rather than client devices.
 
ivan07
newbie
Posts: 26
Joined: Wed Mar 04, 2015 2:57 am

Re: CCR IPSec performance

Wed Mar 11, 2015 10:32 pm

All our private tunnels are still being terminated by our ancient (but fast) 1100AXH2's.
1100 is much faster than CCR in your case? Hmm...
 
djdrastic
Member
Member
Posts: 367
Joined: Wed Aug 01, 2012 2:14 pm

Re: CCR IPSec performance

Thu Mar 12, 2015 7:48 am

All our private tunnels are still being terminated by our ancient (but fast) 1100AXH2's.
1100 is much faster than CCR in your case? Hmm...
AXH2 will do 550 Megs 24/7/365
 
JanezFord
Member Candidate
Member Candidate
Posts: 269
Joined: Wed May 23, 2012 10:58 am

Re: CCR IPSec performance

Thu Mar 12, 2015 10:33 pm

Why not? It's a real world test ...
Because first of all this is a testing i/o of your laptops and then network devices :)
after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.
IMHO if we are discussing the performance of the CCR series, we have to test these devices in the first place, rather than client devices.
Read again very carefully the whole thread ... both laptops performed at 1Gbit/s over gre tunnel and when ipsec was enabled on the same setup throughput dropped down to 80Mbit/s... there is no i/o bottleneck on laptops here ... 1Gbit over two CCR routers without encryption, 80Mbit with encryption ...

JF.
 
ivan07
newbie
Posts: 26
Joined: Wed Mar 04, 2015 2:57 am

Re: CCR IPSec performance

Fri Mar 13, 2015 1:45 am

1Gbit over two CCR routers without encryption, 80Mbit with encryption ...
Alright, how normis got his very nice numbers with their CCRs?
I think it is hard for TileGX performs so slowly with encryption even with one core...
How many CCR cores were loaded in your tests with encryption enabled?

I hope normis will comment this strange situation.
 
Petrovich
just joined
Posts: 6
Joined: Thu Feb 05, 2015 2:17 pm

Re: CCR IPSec performance

Thu Mar 19, 2015 11:40 pm

Alright, how normis got his very nice numbers with their CCRs?
I think it is hard for TileGX performs so slowly with encryption even with one core...
How many CCR cores were loaded in your tests with encryption enabled?
In my case only one core was loaded. It is an issue.
I hope normis will comment this strange situation.
Everyone is waiting for his comments.
 
ivan07
newbie
Posts: 26
Joined: Wed Mar 04, 2015 2:57 am

Re: CCR IPSec performance

Tue Mar 24, 2015 4:48 am

In my case only one core was loaded. It is an issue.
I created ten 50mbit pptp encrypted clients in CCR, connected them all to remote pptp servers and CCR was perfoming very nice, where three to seven cores were loaded.
I know this is a just a fun test but as I may see CCR does perfom good in such ways.
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 730
Joined: Tue Aug 25, 2009 12:01 am

Re: CCR IPSec performance

Sat Apr 11, 2015 12:15 am

Sounds damn close to what I got... And they kept telling me I was wrong.
What numbers are correct? :)
Maybe there is something wrong with your laptops?
I do not like the idea to download some file in such tests...
GRE over IPSEC between 2 CCRs will perform very fast if you do the speed test from one router to another. IE: Bandwidth test or traffic generator running on router 1, going to router 2. Like 800mbit or so.

As soon as you start forwarding traffic out another interface, the performance falls flat on its face.. 80-90mbit MAX. Add MPLS/VPLS on top of that and you are down to about 4mbit.

The same tests (Except for MPLS) over straight IPSEC tunnel mode are back up to gigabit speeds.

The same test over GRE tunnel with no IPSEC are back up to gigabit speeds.

The combination of IPSEC, GRE, and forwarding to another interface makes the CCR squeal. Its a huge problem, but Mikrotik doesn't want to listen. They keep posting that it can do 800+ Mbit over IPSEC GRE when everyone else who tries it get the same numbers that I get. RB1100AHx2 outperforms the CCR by about 4-5x when it comes to GRE/IPSEC tunnels.
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 730
Joined: Tue Aug 25, 2009 12:01 am

Re: CCR IPSec performance

Sat Apr 11, 2015 12:19 am

Why not? It's a real world test ...
Because first of all this is a testing i/o of your laptops and then network devices :)
after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.
IMHO if we are discussing the performance of the CCR series, we have to test these devices in the first place, rather than client devices.
The problem is, if we test just the 2 routers involved in the tunnel, it works fine. If you take 2 CCRs, put an ethernet cable between then, setup a /30 ip on the ethernet interface and do IPSEC for GRE traffic between those 2 IPs, then setup a GRE tunnel with a /30 on each end and do the test from router 1 to router 2, it works great.

Add static routes on each side to a /24 on another interface and hook up a client device then performance goes dead.
 
Dilergore
just joined
Posts: 7
Joined: Wed Aug 05, 2015 12:05 pm

Re: CCR IPSec performance

Wed Aug 05, 2015 12:16 pm

Hi any news regarding this topic?

I'm planning to buy a CCR in the close future. I want to create a site to site VPN between my two flats. My old flat has 1000/100 Mbps internet where I have a pfSense as a router virtualized on Hyper-V (Core i5 - 16gigs of ram). In my new flat where I want to place the CCR I have 240/25 internet. I'm need high speed VPN between the two sites (mainly used for transferring big files around 25gigs from old flat to the new). As you can see currently 100Mbps VPN fulfills my need but I'd like to buy a device that can handle VPN traffic at least 500 Mbps (I believe my ISP in my old flat will increase the 100Mbps upload speed in the near future and in my new flat higher speed is available already).

Thanks!
Last edited by Dilergore on Wed Aug 05, 2015 4:40 pm, edited 1 time in total.
 
Dilergore
just joined
Posts: 7
Joined: Wed Aug 05, 2015 12:05 pm

Re: CCR IPSec performance

Wed Aug 05, 2015 12:33 pm

Hi any news regarding this topic?

I'm planning to buy a CCR in the close future. I want to create a site to site VPN between my two flats. My old flat has 1000/100 Mbps internet where I have a pfSense as a router virtualized on Hyper-V (Core i5 - 16gigs of ram). In my new flat where I want to place the CCR I have 240/25 internet. I'm need high speed VPN between the two sites (mainly used for transferring big files around 25gigs from old flat to the new). As you can see currently 100Mbps VPN fulfills my need but I'd like to buy a device that can handle VPN traffic at least 500 Mbps (I believe my ISP in my old flat will increase the 100Mbps upload speed in the near future and in my new flat higher speed is available already).

Thanks!
sorry, sent twice, still waiting for answer.
 
coylh
Member Candidate
Member Candidate
Posts: 159
Joined: Tue Jul 12, 2011 12:11 am

Re: CCR IPSec performance

Fri Aug 07, 2015 6:30 am

I wouldn't use CCR for more than 100Mb/s IPSEC VPN currently.
 
Dilergore
just joined
Posts: 7
Joined: Wed Aug 05, 2015 12:05 pm

Re: CCR IPSec performance

Fri Aug 07, 2015 8:06 am

I wouldn't use CCR for more than 100Mb/s IPSEC VPN currently.
ehh, sounds nice for a device that costs at least 500$....

Thanks for the info anyway

But if CCR is not the best choice here then I don't know what to buy... It would be good to have the SFP+ port and I need the high VPN speed too...
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Fri Aug 07, 2015 8:24 am

It could be 1100ahx2 with hardware encryption acceleration but it doesn't have sfp ports... I am afraid there is not better option for you.
 
Dilergore
just joined
Posts: 7
Joined: Wed Aug 05, 2015 12:05 pm

Re: CCR IPSec performance

Fri Aug 07, 2015 9:56 am

as I'm checking the forum now it is better to forget about the SFP+ port and wait for RB3011... (and to mention: it's much cheaper...)

Or to buy the newly announced RB850Gx2 with HW encryption. But as I read this device is not supporting fastpath... is it right?

...and what is the guarantee that these new devices will be able to handle (GRE - IPSEC - Route) the traffic at this high speed? I mean if you check this forum topic (posts by Normis) or the spec sheet CCR should be able to handle it but as I see it's not...
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR IPSec performance

Mon Aug 10, 2015 10:34 am

You should avoid fragmentation when running any type of tunnels. With latest ROS version we have reduced out-of-order packets to minimum improving TCP speed over ipsec significantly.

To get best performance, reduce MTU on GRE tunnel or run UDP with lower packet size. For TCP set up change-mss rules to avoid fragmentation.
 
i4jordan
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Mon Sep 02, 2013 1:42 am

Re: CCR IPSec performance

Mon Aug 10, 2015 1:04 pm

@MRZ

Can you give us some good examples with ipSec tunnel and ipSec over GRE/IPIP (transport) to get the optimal best performance?
I am dealing with this a lot and I see a lot of articles saying that MSS/packet size should be good to get optimal results, but I do not see any examples with the right Mangle and other rules.

Also since 6.20 or so there is a clamp-tcp-mss and a dont fragment option in the GRE and the IPIP tunnels. How is this function working in relation with ipsec.

Thanks a lot! It would make my day if I do get ipsec tunneling performing good.
We are using ipsec for tunnels which transport loads of data (backup/rdp etc.).
 
Dilergore
just joined
Posts: 7
Joined: Wed Aug 05, 2015 12:05 pm

Re: CCR IPSec performance

Mon Aug 10, 2015 7:16 pm

@MRZ

So this means that the CCR (1009) can perform with IPSEC - GRE +routing around 500Mbps? If the mentioned issues are no longer existing I should definitely buy a CCR.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR IPSec performance

Tue Aug 11, 2015 11:13 am

Yes it can handle a lot more than 500Mbps
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR IPSec performance

Tue Aug 11, 2015 4:42 pm

@MRZ
Also since 6.20 or so there is a clamp-tcp-mss and a dont fragment option in the GRE and the IPIP tunnels. How is this function working in relation with ipsec.
Clamp-tcp-mss adjusts mss value for new TCP connections based on current tunnel MTU.
If dont-fragment is set to inherit tunnel copies DF bit from encapsulated packet. This allows path MTU discovery to function and further detect and adjust correct tunnel MTU.

In scenario where tunnel runs over the links which MTU is limited somewhere in providers network (over ADSL lines with additional overhead and so on) dont-fragment and clamp-tcp-mss should be enabled. It is the most optimal setup to avoid fragmentation.

Note that path MTU discovery will not function properly if ICMP packets are dropped by any of the routers on the path.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 562
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CCR IPSec performance

Wed Sep 23, 2015 10:49 pm

Hello
on CCR 1009,with 6.30.4 what are the expeted performance for:

CCR 1009 central site. WAN: 1Gb uplink
"LAN" - 3 tunnels GRE (no encryption) to three remote sites at 10M, 30M, 300M.

In the central roueter I will do NAT for the remote sites.

Can I expect more than 500+mbps GRE traffic ? unencrypted?

Thank you
 
_saik0
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Aug 26, 2007 11:18 pm

Re: CCR IPSec performance

Sun Nov 22, 2015 1:12 pm

I'm planning on getting two CCR1036 for connecting two sites via VPN and need to have answers...

So in the end, did ANYONE succeed in creating a single IPSec/L2TP(or GRE) tunnel between two say CCR1036 and got 500Mbps+ between two clients from two routed networks behind those two CCRs ?

There's a million discussions about this and nothing conclusive - just claims from MT staff that CCRs should handle "a lot more" and yet nobody actually confirmed anything.

I suggest merging all topics with "CCR" and "IPsec" keywords so that we can finally have some definite answers.
Either the CCRs are a failure and MT doesn't want to admit that, or it actually took a lot of time to patch the ROS - but nobody confirmed that.
 
ATG
just joined
Posts: 24
Joined: Fri Feb 21, 2014 9:45 am

Re: CCR IPSec performance

Sun Nov 22, 2015 7:05 pm

Hello

I have a couple of CCR1009, each on their seperate location. Both with WAN 150/150Mbit Fiber. I have a IPSec with EoIP tunnel between the CCR's, running the latest 6.33.1 with latest firmware.

When I try to push rsync backup, routed between these unit, it maxes out on around 50Mbit over EoIP over a single TCP connection. The recieving CCR, has a low CPU % over all the cores. However, the transmitting CCR has approx 10% overall cpu usage, but one cpu core always are at 100%.

Based on this, still seems like the CCR1009 with ROS 6.33 struggels on loadbalancing between the cores, over IPSEC(EoIP) with single TCP(and maybe UDP) connections\transfers.

Regards
 
_saik0
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Aug 26, 2007 11:18 pm

Re: CCR IPSec performance

Sun Nov 22, 2015 8:57 pm

Thanks for the input!

Well yes, that pretty much answers my question and confirms my fears...

Seems i'd really be better of with two multi-core x86 servers/workstations :/
Yes it can handle a lot more than 500Mbps
Comments?
 
ATG
just joined
Posts: 24
Joined: Fri Feb 21, 2014 9:45 am

Re: CCR IPSec performance

Mon Nov 23, 2015 11:06 am

It may be that the CCR in my setup, would have preformed better with GRE instead of EoIP, have not tested.

Alternatily, if I had added serveral connections over the same tunnel, I also think througput would improve.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7041
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR IPSec performance

Mon Nov 23, 2015 12:14 pm

@ATG Make sure you have set everything mentioned in previous posts to avoid fragmentation. If you did then send a supout file to support, most likely there are other problems not related to ipsec, because 50mbps is too small bw especially in case with UDP.
 
_saik0
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Aug 26, 2007 11:18 pm

Re: CCR IPSec performance

Tue Jan 26, 2016 8:35 pm

So i've finally bought two of CCR1036 and am currently trialing them for GRE/IPSec VPN connectivity.

Using 6.34rc41 this is the result of running iperf in dualtest TCP mode.

PC1 ---- CCR1 --- [gre/ipsec_sha1_aes256cbc] --- CCR2 ---- PC2



Image

I'm releaved that the CCR is actually capable of providing advertised IPSEC performance.
Still there should be some improvement to stability.
There are rather big fluctuations in throughput during the test on a otherwise completely idle system.
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: CCR IPSec performance

Tue Jan 26, 2016 8:49 pm

How many TCP threads are you using in iperf and at what MTU size?

Here is a recap of our performance tests with IPSEC on CCRs

http://www.stubarea51.net/2015/10/16/10 ... ip-tunnel/
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2990
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CCR IPSec performance

Wed Jan 27, 2016 7:35 pm

How many TCP threads are you using in iperf and at what MTU size?
interesting question
 
_saik0
Member Candidate
Member Candidate
Posts: 129
Joined: Sun Aug 26, 2007 11:18 pm

Re: CCR IPSec performance

Wed Jan 27, 2016 8:39 pm

It was a single TCP connection per direction with TCP MSS clamping for the GRE tunnel, IPSec in transport mode.
So in the end the actual MTU for the tunnel is 1426B.

all devices were connected with a single 1Gbps link.
 
espacioint
just joined
Posts: 8
Joined: Thu Mar 16, 2017 11:12 pm
Location: Mijas
Contact:

Re: CCR IPSec performance

Mon Aug 26, 2019 1:22 pm

Hello, Im still investigating about this, Im trying to do a tunnel with gre and ipsec, and the performance goes down as soon I put ipsec in the tunnel.
Do u have any tip to sort this... is there any other experience since 2016 (that is the last post)
 
nbctcp
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Tue Sep 16, 2014 7:32 pm

Re: CCR IPSec performance

Thu Jun 23, 2022 3:50 pm

You said CCR1009
max ipsec tunnel throughput
3.2Gbps with 34 tunnels (full duplex)
1.8Gbps with 16 tunnels (full duplex)

?
1. How about CCR1036 maximum numbers of ipsec tunnels?
because it has 8GB RAM compare to just 2GB in CCR1009
1.8Gbps with 16 tunnels (full duplex)
820Mbps with one GRE over IpSec tunnel (full duplex)
2. do you have link on how to test with traffic generator
tq
max throughput 3.2Gbps with 34 tunnels (full duplex)



--CCR1009--

1.6Gbps with 8 tunnels (full duplex)
520Mbps with one GRE over IpSec tunnel (full duplex)



Tested with traffic-generator and 1470byte packets.

Who is online

Users browsing this forum: erlinden, sch and 29 guests