Hello all
We currently link two sites together with L2TPv3+IPSec using Cisco 2900 routers, but the IPSec performance is horrible (max 50mbps @100%CPU).
We want to replace with one of the above routers at either side. Obviously CCR is newer but is the IPSec hardware accelerated yet? Will either of the above be able to do say 150mbps of AES-128 of typical traffic?
Thanks
A
Both support HW accelerated aes-128-cbc → aes-256-cbc.
CCR also supports sha1 HW offload.
1100AHx2 will do about 400 MBit/s of IPSec at aes-128-cbc.
CCR will do much more - dependant on model.
CCR1036 can do up to 3.5GBit of IPsec encrypted traffic
Thanks. I assume that means CCR1009 can do at least 150mbit/sec IPSec. Will combining with EoIP tunneling cause much slowdown? I read in some other thread about IPSec encrypted GRE tunnels being slow - will the same issue affect me here?
Update: we bought 2 RB1100AHx2 and are running EoIP+IPSec between them across the WAN. Works great, getting over 400mbit/sec! 
Both the CCR1009 and RB1100AHx2 are great for IPsec.
I would spend a little more and get the CCR1009 for more features..
Well worth it..
-tp
Except higher processor speed, 4 ports less, only one switch group and no bypass switch, these “more” features being what?
Well, let’s see…
Dual/Redundant Power Supplies,
Multiple cores,
LCD Display,
Smart Card Slot,
Access to the MicrSD card in the back,
…Should be enough…eh?
Hmmm, let’s see, compared to the AHx2.
CCR1009-8G-1S: No redundant power supply, no LCD display, no smart card slot, 1 USB, half the memory, 1 SFP, 4 copper ports less: +75$
CCR1009-8G-1S-1S+: redundant power supply, LCD, same amount of RAM, 1 USB, 1 SFP, 1 SFP+, 4 copper ports less: +145$
So for the basic modell. except the speed (which is pretty much useless until you need more than 1Gb/sec speed), and the SFP, there is nothing to it.
I don’t want to sound naggy, but one has to choose the best solution for a given task. And in this case, the CCR is just a waste of money.
A RB100AHx2 would suffice, with an at least 75$ smaller price tag (that is 150$ for the pair).
I would dare to say that the upcoming RB850 would be the device of choice in this case.
Anyway, they have a couple of RB1100AHx2 now.
For a single connection, you won’t likely get 150mbit throughout with gre or eoip over IPSec on any ccr. S
Just an update here for anyone interested. After extensive tweaking and simplification of the config, we’ve managed to extract around 680mbit/sec out of the two RB1100AHx2 routers (including clocking them up to 1333MHz).
We bought 2xCCR1009s for another project, and decided to see how fast they would work for this particular requirement. We copied the config across, and the performance sucks. We were only able to get 25mbit/sec with the same config. It seems as soon as you combine EoIP or other tunneling with IPSec on the CCR series, the performance drops dramatically. Bandwidth test between the EoIP endpoints runs 600M+, and if you disable the encryption, EoIP runs at 900M+, but when you combine them, the performance drops to 25 
We tried fiddling with MTU, MSS clamping, changing ports etc, but could not fix the problem, so it looks like the 1100s will have to do for now.
Mikrotik, how about an updated PPC model with higher clock speed? 
I did a lot of testing this evening. I’m not getting extremely consistent results, but I was able to get around 500Mb/s on a CCR1036 to CCR1009 link. Both were using 6.27 with updated firmware. The only good results were with aes-256-cbc.
I notice that “in-state-sequence-errors” are very high though.
Here’s one side of the lab, if someone wants to reproduce. Note that there’s no NAT, firewalling, or connection tracking; YMMV. I was using iperf on windows in tcp mode with --window set to 16M and 8-16 parallel streams. Without ipsec, the same test produces about 850Mb/s.
# feb/24/2015 17:53:51 by RouterOS 6.27
#
#
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc
/port
set 0 name=serial0
set 1 name=serial1
/ip firewall connection tracking
set enabled=no
/ip address
add address=10.0.0.2/24 interface=ether1 network=10.0.0.0
add address=192.168.1.1/24 interface=ether5 network=192.168.1.0
/ip ipsec peer
add address=10.0.0.1/32 enc-algorithm=aes-256 nat-traversal=no secret=password
/ip ipsec policy
set 0 disabled=yes
add dst-address=192.168.0.0/24 sa-dst-address=10.0.0.1 sa-src-address=10.0.0.2 src-address=\
192.168.1.0/24 tunnel=yes
/ip route
add distance=1 gateway=10.0.0.1
/system clock
set time-zone-autodetect=no time-zone-name=America/Los_Angeles
/system ntp server
set enabled=yes
/system resource irq rps
set ether1 disabled=yes
set ether2 disabled=yes
set ether3 disabled=yes
set ether4 disabled=yes
/system routerboard settings
set cpu-frequency=1200MHz enter-setup-on=delete-key memory-frequency=1066DDR
If you have some time, would you be able to create an EoIP tunnel and encrypt the endpoints? I would be very keen to see if your performance drops 90%+ like it did for us.
Yes, I believe I’ve set this up. I haven’t tried this configuration before, so you’ll want to look at the config with some skepticism. Still, I see the SA byte counters moving, so it looks like the traffic is crossing both the EOIP tunnel and the IPSEC.
Here’s one side:
# feb/24/2015 20:34:11 by RouterOS 6.27
#
#
/interface bridge
add name=bridge1
/interface eoip
add !keepalive mac-address=02:02:96:29:2A:86 name=eoip-tunnel1 remote-address=192.168.0.1 tunnel-id=0
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=bridge1 interface=eoip-tunnel1
add bridge=bridge1 interface=ether5
/ip firewall connection tracking
set enabled=no
/ip address
add address=10.0.0.2/24 interface=ether1 network=10.0.0.0
add address=192.168.1.1/24 interface=ether5 network=192.168.1.0
/ip ipsec peer
add address=10.0.0.1/32 enc-algorithm=aes-256 exchange-mode=main-l2tp nat-traversal=no secret=password
/ip ipsec policy
set 0 disabled=yes
add dst-address=0.0.0.0/0 sa-dst-address=10.0.0.1 sa-src-address=10.0.0.2 src-address=0.0.0.0/0 tunnel=yes
/ip route
add distance=1 gateway=10.0.0.1
/system clock
set time-zone-autodetect=no time-zone-name=America/Los_Angeles
/system ntp server
set enabled=yes
/system resource irq rps
set ether1 disabled=yes
set ether2 disabled=yes
set ether3 disabled=yes
set ether4 disabled=yes
/system routerboard settings
set cpu-frequency=1200MHz enter-setup-on=delete-key memory-frequency=1066DDR
The results are not as good, but not terrible:
C:\iperf>iperf --client 172.16.5.55 --parallel 16 --window 16M
------------------------------------------------------------
Client connecting to 172.16.5.55, TCP port 5001
TCP window size: 16.0 MByte
------------------------------------------------------------
[ 18] local 172.16.5.44 port 51435 connected with 172.16.5.55 port 5001
[ 16] local 172.16.5.44 port 51433 connected with 172.16.5.55 port 5001
[ 15] local 172.16.5.44 port 51432 connected with 172.16.5.55 port 5001
[ 12] local 172.16.5.44 port 51429 connected with 172.16.5.55 port 5001
[ 11] local 172.16.5.44 port 51428 connected with 172.16.5.55 port 5001
[ 8] local 172.16.5.44 port 51425 connected with 172.16.5.55 port 5001
[ 7] local 172.16.5.44 port 51424 connected with 172.16.5.55 port 5001
[ 3] local 172.16.5.44 port 51420 connected with 172.16.5.55 port 5001
[ 17] local 172.16.5.44 port 51434 connected with 172.16.5.55 port 5001
[ 14] local 172.16.5.44 port 51431 connected with 172.16.5.55 port 5001
[ 13] local 172.16.5.44 port 51430 connected with 172.16.5.55 port 5001
[ 10] local 172.16.5.44 port 51427 connected with 172.16.5.55 port 5001
[ 9] local 172.16.5.44 port 51426 connected with 172.16.5.55 port 5001
[ 6] local 172.16.5.44 port 51423 connected with 172.16.5.55 port 5001
[ 5] local 172.16.5.44 port 51422 connected with 172.16.5.55 port 5001
[ 4] local 172.16.5.44 port 51421 connected with 172.16.5.55 port 5001
[ ID] Interval Transfer Bandwidth
[ 8] 0.0-10.0 sec 36.2 MBytes 30.4 Mbits/sec
[ 6] 0.0-10.0 sec 42.0 MBytes 35.2 Mbits/sec
[ 17] 0.0-10.0 sec 26.1 MBytes 21.8 Mbits/sec
[ 9] 0.0-10.0 sec 46.9 MBytes 39.2 Mbits/sec
[ 10] 0.0-10.0 sec 35.5 MBytes 29.6 Mbits/sec
[ 14] 0.0-10.1 sec 42.4 MBytes 35.3 Mbits/sec
[ 11] 0.0-10.1 sec 36.2 MBytes 30.2 Mbits/sec
[ 16] 0.0-10.1 sec 41.8 MBytes 34.7 Mbits/sec
[ 15] 0.0-10.1 sec 33.6 MBytes 27.9 Mbits/sec
[ 7] 0.0-10.1 sec 34.6 MBytes 28.7 Mbits/sec
[ 18] 0.0-10.1 sec 38.1 MBytes 31.5 Mbits/sec
[ 4] 0.0-10.1 sec 40.4 MBytes 33.4 Mbits/sec
[ 12] 0.0-10.2 sec 42.2 MBytes 34.9 Mbits/sec
[ 5] 0.0-10.2 sec 45.6 MBytes 37.6 Mbits/sec
[ 3] 0.0-10.2 sec 37.8 MBytes 30.9 Mbits/sec
[ 13] 0.0-10.7 sec 35.2 MBytes 27.8 Mbits/sec
[SUM] 0.0-10.7 sec 615 MBytes 484 Mbits/sec
I would like to try this across a couple 1100AHx2. I’ll post the results if I can cobble them together.
Interesting result. How much can you get on a single thread?
Are you intending to run eoip over l2tp+IPSec?
Our IPSec peer config looks like this:
/ip ipsec peer
add address=10.10.10.6/32 dpd-maximum-failures=10 enc-algorithm=aes-128 generate-policy=port-override lifetime=8h nat-traversal=no proposal-check=exact secret=password
eoip+ipsec single thread is about 135Mb/s. With this test tool, I consider it normal to be unable to saturate a link with a single tcp stream though, regardless of the tunnel configuration.
Ok, I hooked up a couple 1100AHx2 with more or less the same configuration as the CCR pair. I’m able to get around 800Mb/s EoIP + IPSEC. The downside of the better performance is that the dual CPU/irq is maxed out. I think running the routerboards at 90%+ is going to cause system stability problems, so I think you’re better off capping the link at something a bit lower (600Mb/s?).
While the CCR’s cpu appears to be off the hook for the lower tunnel performance, there’s something wrong with either the hw accelerator or the software implementation. I hope this gets fixed. For the time being, it looks like the 1100AHx2 is better for ipsec.
Guys, this is really surprising and baffling! I was sure that CCR with 9 cores and IPsec accelerator is indisputable faster then old RB1100AHx2 with accelerator and only 2 cores! This document http://www.tilera.com/files/drim__TILE-Gx8009_PB036-02_WEB_7663.pdf states it should deliver up to “10Gbps encryption throughput” :
TileGx.JPG
How is this possible that CCR is slower? Is MikroTik working on it’s optimization? Is this maximum IPsec performance?