Page 1 of 1

CCR IPSec performance

Posted: Fri Aug 08, 2014 6:20 am
by jml
Does anyone have stats on the CCRs for IPSec throughput?

Thanks.

Re: CCR IPSec performance

Posted: Fri Aug 08, 2014 9:21 am
by normis
max throughput 3.2Gbps with 34 tunnels (full duplex)

1.8Gbps with 16 tunnels (full duplex)
820Mbps with one GRE over IpSec tunnel (full duplex)

--CCR1009--

1.6Gbps with 8 tunnels (full duplex)
520Mbps with one GRE over IpSec tunnel (full duplex)



Tested with traffic-generator and 1470byte packets.

Re: CCR IPSec performance

Posted: Sat Aug 09, 2014 12:53 pm
by i4jordan
Normis,

Thank you for the numbers. It helps a lot in designing VPN networks.

Do you also have some numbers on the 'older' RB1100AHx2 models?

I'd like to know speed of the ipsec tunnels and also the GRE+ipsec speed.

Other question, which is fatser: GRE+ipsec or IPIP+ipsec.

Thank you!

Re: CCR IPSec performance

Posted: Mon Feb 09, 2015 2:49 pm
by Petrovich
max throughput 3.2Gbps with 34 tunnels (full duplex)

520Mbps with one GRE over IpSec tunnel (full duplex)



Tested with traffic-generator and 1470byte packets.
Could you please provide your settings for this tunnel.
I did not manage to get even 200Mbit/s on CCR-1036

Re: CCR IPSec performance

Posted: Mon Feb 09, 2015 2:51 pm
by mrz
What was your test procedure?

Re: CCR IPSec performance

Posted: Fri Feb 13, 2015 4:42 pm
by Petrovich
What was your test procedure?
That was two CCR-1036-8G-2S+ instances connected with 10Gbit/s link. GRE over IPSEC. Confuguration is almost default, you can find it here (just not to replear myselft).
http://forum.mikrotik.com/viewtopic.php ... 92#p467392

I had two laptops with gigabit ethernet adapter, each one was connected to the corresponding CCR. The test was to download 10Gbyte file from one laptop to another .
The first try was to download file over GRE without encyption. Speed was exactly 1Gbit/s.

Second step was to turn on ipsec with config provided. With aes-256 I had up to 80Mbit/s, with aes-128 without authentication I had up to 150Mbit/s

Re: CCR IPSec performance

Posted: Sat Feb 14, 2015 2:26 am
by roadracer96
Sounds damn close to what I got... And they kept telling me I was wrong.

Re: CCR IPSec performance

Posted: Wed Mar 11, 2015 8:12 pm
by ivan07
Sounds damn close to what I got... And they kept telling me I was wrong.
What numbers are correct? :)
Maybe there is something wrong with your laptops?
I do not like the idea to download some file in such tests...

Re: CCR IPSec performance

Posted: Wed Mar 11, 2015 8:52 pm
by JanezFord
I do not like the idea to download some file in such tests...
Why not? It's a real world test ... after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.

JF.

Re: CCR IPSec performance

Posted: Wed Mar 11, 2015 9:01 pm
by djdrastic
130-150 Meg is where I was maxing out as well on various Ros 6 versions with the CCRs.
All our private tunnels are still being terminated by our ancient (but fast) 1100AXH2's.

Re: CCR IPSec performance

Posted: Wed Mar 11, 2015 10:31 pm
by ivan07
Why not? It's a real world test ...
Because first of all this is a testing i/o of your laptops and then network devices :)
after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.
IMHO if we are discussing the performance of the CCR series, we have to test these devices in the first place, rather than client devices.

Re: CCR IPSec performance

Posted: Wed Mar 11, 2015 10:32 pm
by ivan07
All our private tunnels are still being terminated by our ancient (but fast) 1100AXH2's.
1100 is much faster than CCR in your case? Hmm...

Re: CCR IPSec performance

Posted: Thu Mar 12, 2015 7:48 am
by djdrastic
All our private tunnels are still being terminated by our ancient (but fast) 1100AXH2's.
1100 is much faster than CCR in your case? Hmm...
AXH2 will do 550 Megs 24/7/365

Re: CCR IPSec performance

Posted: Thu Mar 12, 2015 10:33 pm
by JanezFord
Why not? It's a real world test ...
Because first of all this is a testing i/o of your laptops and then network devices :)
after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.
IMHO if we are discussing the performance of the CCR series, we have to test these devices in the first place, rather than client devices.
Read again very carefully the whole thread ... both laptops performed at 1Gbit/s over gre tunnel and when ipsec was enabled on the same setup throughput dropped down to 80Mbit/s... there is no i/o bottleneck on laptops here ... 1Gbit over two CCR routers without encryption, 80Mbit with encryption ...

JF.

Re: CCR IPSec performance

Posted: Fri Mar 13, 2015 1:45 am
by ivan07
1Gbit over two CCR routers without encryption, 80Mbit with encryption ...
Alright, how normis got his very nice numbers with their CCRs?
I think it is hard for TileGX performs so slowly with encryption even with one core...
How many CCR cores were loaded in your tests with encryption enabled?

I hope normis will comment this strange situation.

Re: CCR IPSec performance

Posted: Thu Mar 19, 2015 11:40 pm
by Petrovich
Alright, how normis got his very nice numbers with their CCRs?
I think it is hard for TileGX performs so slowly with encryption even with one core...
How many CCR cores were loaded in your tests with encryption enabled?
In my case only one core was loaded. It is an issue.
I hope normis will comment this strange situation.
Everyone is waiting for his comments.

Re: CCR IPSec performance

Posted: Tue Mar 24, 2015 4:48 am
by ivan07
In my case only one core was loaded. It is an issue.
I created ten 50mbit pptp encrypted clients in CCR, connected them all to remote pptp servers and CCR was perfoming very nice, where three to seven cores were loaded.
I know this is a just a fun test but as I may see CCR does perfom good in such ways.

Re: CCR IPSec performance

Posted: Sat Apr 11, 2015 12:15 am
by roadracer96
Sounds damn close to what I got... And they kept telling me I was wrong.
What numbers are correct? :)
Maybe there is something wrong with your laptops?
I do not like the idea to download some file in such tests...
GRE over IPSEC between 2 CCRs will perform very fast if you do the speed test from one router to another. IE: Bandwidth test or traffic generator running on router 1, going to router 2. Like 800mbit or so.

As soon as you start forwarding traffic out another interface, the performance falls flat on its face.. 80-90mbit MAX. Add MPLS/VPLS on top of that and you are down to about 4mbit.

The same tests (Except for MPLS) over straight IPSEC tunnel mode are back up to gigabit speeds.

The same test over GRE tunnel with no IPSEC are back up to gigabit speeds.

The combination of IPSEC, GRE, and forwarding to another interface makes the CCR squeal. Its a huge problem, but Mikrotik doesn't want to listen. They keep posting that it can do 800+ Mbit over IPSEC GRE when everyone else who tries it get the same numbers that I get. RB1100AHx2 outperforms the CCR by about 4-5x when it comes to GRE/IPSEC tunnels.

Re: CCR IPSec performance

Posted: Sat Apr 11, 2015 12:19 am
by roadracer96
Why not? It's a real world test ...
Because first of all this is a testing i/o of your laptops and then network devices :)
after testing with traffic generators you should always test with the type of traffic you will normally have to deal with as network admin... if your users use smb, ftp or nfs on regular basis you should test it exactly the way Petrovich did.
IMHO if we are discussing the performance of the CCR series, we have to test these devices in the first place, rather than client devices.
The problem is, if we test just the 2 routers involved in the tunnel, it works fine. If you take 2 CCRs, put an ethernet cable between then, setup a /30 ip on the ethernet interface and do IPSEC for GRE traffic between those 2 IPs, then setup a GRE tunnel with a /30 on each end and do the test from router 1 to router 2, it works great.

Add static routes on each side to a /24 on another interface and hook up a client device then performance goes dead.

Re: CCR IPSec performance

Posted: Wed Aug 05, 2015 12:16 pm
by Dilergore
Hi any news regarding this topic?

I'm planning to buy a CCR in the close future. I want to create a site to site VPN between my two flats. My old flat has 1000/100 Mbps internet where I have a pfSense as a router virtualized on Hyper-V (Core i5 - 16gigs of ram). In my new flat where I want to place the CCR I have 240/25 internet. I'm need high speed VPN between the two sites (mainly used for transferring big files around 25gigs from old flat to the new). As you can see currently 100Mbps VPN fulfills my need but I'd like to buy a device that can handle VPN traffic at least 500 Mbps (I believe my ISP in my old flat will increase the 100Mbps upload speed in the near future and in my new flat higher speed is available already).

Thanks!

Re: CCR IPSec performance

Posted: Wed Aug 05, 2015 12:33 pm
by Dilergore
Hi any news regarding this topic?

I'm planning to buy a CCR in the close future. I want to create a site to site VPN between my two flats. My old flat has 1000/100 Mbps internet where I have a pfSense as a router virtualized on Hyper-V (Core i5 - 16gigs of ram). In my new flat where I want to place the CCR I have 240/25 internet. I'm need high speed VPN between the two sites (mainly used for transferring big files around 25gigs from old flat to the new). As you can see currently 100Mbps VPN fulfills my need but I'd like to buy a device that can handle VPN traffic at least 500 Mbps (I believe my ISP in my old flat will increase the 100Mbps upload speed in the near future and in my new flat higher speed is available already).

Thanks!
sorry, sent twice, still waiting for answer.

Re: CCR IPSec performance

Posted: Fri Aug 07, 2015 6:30 am
by coylh
I wouldn't use CCR for more than 100Mb/s IPSEC VPN currently.

Re: CCR IPSec performance

Posted: Fri Aug 07, 2015 8:06 am
by Dilergore
I wouldn't use CCR for more than 100Mb/s IPSEC VPN currently.
ehh, sounds nice for a device that costs at least 500$....

Thanks for the info anyway

But if CCR is not the best choice here then I don't know what to buy... It would be good to have the SFP+ port and I need the high VPN speed too...

Posted: Fri Aug 07, 2015 8:24 am
by jarda
It could be 1100ahx2 with hardware encryption acceleration but it doesn't have sfp ports... I am afraid there is not better option for you.

Re: CCR IPSec performance

Posted: Fri Aug 07, 2015 9:56 am
by Dilergore
as I'm checking the forum now it is better to forget about the SFP+ port and wait for RB3011... (and to mention: it's much cheaper...)

Or to buy the newly announced RB850Gx2 with HW encryption. But as I read this device is not supporting fastpath... is it right?

...and what is the guarantee that these new devices will be able to handle (GRE - IPSEC - Route) the traffic at this high speed? I mean if you check this forum topic (posts by Normis) or the spec sheet CCR should be able to handle it but as I see it's not...

Re: CCR IPSec performance

Posted: Mon Aug 10, 2015 10:34 am
by mrz
You should avoid fragmentation when running any type of tunnels. With latest ROS version we have reduced out-of-order packets to minimum improving TCP speed over ipsec significantly.

To get best performance, reduce MTU on GRE tunnel or run UDP with lower packet size. For TCP set up change-mss rules to avoid fragmentation.

Re: CCR IPSec performance

Posted: Mon Aug 10, 2015 1:04 pm
by i4jordan
@MRZ

Can you give us some good examples with ipSec tunnel and ipSec over GRE/IPIP (transport) to get the optimal best performance?
I am dealing with this a lot and I see a lot of articles saying that MSS/packet size should be good to get optimal results, but I do not see any examples with the right Mangle and other rules.

Also since 6.20 or so there is a clamp-tcp-mss and a dont fragment option in the GRE and the IPIP tunnels. How is this function working in relation with ipsec.

Thanks a lot! It would make my day if I do get ipsec tunneling performing good.
We are using ipsec for tunnels which transport loads of data (backup/rdp etc.).

Re: CCR IPSec performance

Posted: Mon Aug 10, 2015 7:16 pm
by Dilergore
@MRZ

So this means that the CCR (1009) can perform with IPSEC - GRE +routing around 500Mbps? If the mentioned issues are no longer existing I should definitely buy a CCR.

Re: CCR IPSec performance

Posted: Tue Aug 11, 2015 11:13 am
by mrz
Yes it can handle a lot more than 500Mbps

Re: CCR IPSec performance

Posted: Tue Aug 11, 2015 4:42 pm
by mrz
@MRZ
Also since 6.20 or so there is a clamp-tcp-mss and a dont fragment option in the GRE and the IPIP tunnels. How is this function working in relation with ipsec.
Clamp-tcp-mss adjusts mss value for new TCP connections based on current tunnel MTU.
If dont-fragment is set to inherit tunnel copies DF bit from encapsulated packet. This allows path MTU discovery to function and further detect and adjust correct tunnel MTU.

In scenario where tunnel runs over the links which MTU is limited somewhere in providers network (over ADSL lines with additional overhead and so on) dont-fragment and clamp-tcp-mss should be enabled. It is the most optimal setup to avoid fragmentation.

Note that path MTU discovery will not function properly if ICMP packets are dropped by any of the routers on the path.

Re: CCR IPSec performance

Posted: Wed Sep 23, 2015 10:49 pm
by Maggiore81
Hello
on CCR 1009,with 6.30.4 what are the expeted performance for:

CCR 1009 central site. WAN: 1Gb uplink
"LAN" - 3 tunnels GRE (no encryption) to three remote sites at 10M, 30M, 300M.

In the central roueter I will do NAT for the remote sites.

Can I expect more than 500+mbps GRE traffic ? unencrypted?

Thank you

Re: CCR IPSec performance

Posted: Sun Nov 22, 2015 1:12 pm
by _saik0
I'm planning on getting two CCR1036 for connecting two sites via VPN and need to have answers...

So in the end, did ANYONE succeed in creating a single IPSec/L2TP(or GRE) tunnel between two say CCR1036 and got 500Mbps+ between two clients from two routed networks behind those two CCRs ?

There's a million discussions about this and nothing conclusive - just claims from MT staff that CCRs should handle "a lot more" and yet nobody actually confirmed anything.

I suggest merging all topics with "CCR" and "IPsec" keywords so that we can finally have some definite answers.
Either the CCRs are a failure and MT doesn't want to admit that, or it actually took a lot of time to patch the ROS - but nobody confirmed that.

Re: CCR IPSec performance

Posted: Sun Nov 22, 2015 7:05 pm
by ATG
Hello

I have a couple of CCR1009, each on their seperate location. Both with WAN 150/150Mbit Fiber. I have a IPSec with EoIP tunnel between the CCR's, running the latest 6.33.1 with latest firmware.

When I try to push rsync backup, routed between these unit, it maxes out on around 50Mbit over EoIP over a single TCP connection. The recieving CCR, has a low CPU % over all the cores. However, the transmitting CCR has approx 10% overall cpu usage, but one cpu core always are at 100%.

Based on this, still seems like the CCR1009 with ROS 6.33 struggels on loadbalancing between the cores, over IPSEC(EoIP) with single TCP(and maybe UDP) connections\transfers.

Regards

Re: CCR IPSec performance

Posted: Sun Nov 22, 2015 8:57 pm
by _saik0
Thanks for the input!

Well yes, that pretty much answers my question and confirms my fears...

Seems i'd really be better of with two multi-core x86 servers/workstations :/
Yes it can handle a lot more than 500Mbps
Comments?

Re: CCR IPSec performance

Posted: Mon Nov 23, 2015 11:06 am
by ATG
It may be that the CCR in my setup, would have preformed better with GRE instead of EoIP, have not tested.

Alternatily, if I had added serveral connections over the same tunnel, I also think througput would improve.

Re: CCR IPSec performance

Posted: Mon Nov 23, 2015 12:14 pm
by mrz
@ATG Make sure you have set everything mentioned in previous posts to avoid fragmentation. If you did then send a supout file to support, most likely there are other problems not related to ipsec, because 50mbps is too small bw especially in case with UDP.

Re: CCR IPSec performance

Posted: Tue Jan 26, 2016 8:35 pm
by _saik0
So i've finally bought two of CCR1036 and am currently trialing them for GRE/IPSec VPN connectivity.

Using 6.34rc41 this is the result of running iperf in dualtest TCP mode.

PC1 ---- CCR1 --- [gre/ipsec_sha1_aes256cbc] --- CCR2 ---- PC2



Image

I'm releaved that the CCR is actually capable of providing advertised IPSEC performance.
Still there should be some improvement to stability.
There are rather big fluctuations in throughput during the test on a otherwise completely idle system.

Re: CCR IPSec performance

Posted: Tue Jan 26, 2016 8:49 pm
by IPANetEngineer
How many TCP threads are you using in iperf and at what MTU size?

Here is a recap of our performance tests with IPSEC on CCRs

http://www.stubarea51.net/2015/10/16/10 ... ip-tunnel/

Re: CCR IPSec performance

Posted: Wed Jan 27, 2016 7:35 pm
by chechito
How many TCP threads are you using in iperf and at what MTU size?
interesting question

Re: CCR IPSec performance

Posted: Wed Jan 27, 2016 8:39 pm
by _saik0
It was a single TCP connection per direction with TCP MSS clamping for the GRE tunnel, IPSec in transport mode.
So in the end the actual MTU for the tunnel is 1426B.

all devices were connected with a single 1Gbps link.

Re: CCR IPSec performance

Posted: Mon Aug 26, 2019 1:22 pm
by espacioint
Hello, Im still investigating about this, Im trying to do a tunnel with gre and ipsec, and the performance goes down as soon I put ipsec in the tunnel.
Do u have any tip to sort this... is there any other experience since 2016 (that is the last post)