CCR EOIP over IPSEC performance

RouterOS General
1

I have two 1009-8G-1S-1S+'s connected together using IPSEC (AES-128-CTR) and EOIP tunnels. I am only able to get 30Mb/s to 40Mb/s out of the tunnel. Latency across the tunnel is around 7 ms. One location is 100Mb fiber. Other location is a colo datacenter on 1Gb ethernet.

AES-128-CTR seems the be the best out of the bunch. 128-CBC is around 10Mb/s as well as 256 bit any flavor of AES. Blowfish and AES-128-CTR are pretty close at approx 35Mb/s.

I have tried so many different iterations here and am not able to max out even a single core on the Tile. I have read through other posts on this issue as well. Has anyone been able to resolve these issues? Is it a software issue? Hardware resources look fine. Any word from Mikrotik?

I am running 6.28 on both routers.

  • CKJ
2

Thank you for your post, CKJ. Although I am disappointed to not see any further activity on this thread, I wanted to at least thank you. Your post caused me to do a little looking into the different flavors of AES-128. Previously I had just been clicking the first one (CBC).

CBC’s performance was abysmal. I was banging my head on 2Mbit performance across a production tunnel and maxing out at 7.5Mbit on directly connected routers. Once I moved to CTR, the real-world test immediately jumped up to 30Mbit with two files being moved concurrently. I haven’t done any further testing yet, but that was a life-saver for me. It’s still not anywhere close to utilizing the 100Mbit line, but I haven’t tested yet with more concurrency than two files.

If you have found any solutions for increasing performance using the CCR series routers, please take a moment to update your thread.

3

How are you testing the throughput? I have 12 1009-8G-1S-1S+ in production and I can push the following:
Latency(RTT) Bandwidth (iperf -c -P8)
5ms 330Mbit
22ms 255Mbit
1ms 320Mbit

Note that on paper, the CCR can push 1Gbit+ over IPSec with hardware acceleration but good luck reconstructing that stream with TCP, it will be completely out of order. Some stacks/configs will handle it nicely but it isn’t a very nice flow. Testing this will look beautiful with UDP. Mikrotik loves to advertise this as a feature and benchmark but I fail to see anyone using this in the real world. The IPSec hardware acceleration is basically a useless feature until they can maintain the order on a per-flow basis.
TLDR: Don’t use CBC. It is a disaster:
See: http://forum.mikrotik.com/t/eoip-packet-reordering-with-ipsec-load-balancing-across-cores-per-packet-vs-per-flow/96935/2

I top out around 300Mbit for a single IPSec session. My throughput is still rather disappointing given the physical links are 1Gb+ but it works for my current workload.

My config rough config:

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 enc-algorithms=aes-128-ctr
/interface eoip
add allow-fast-path=no ipsec-secret=SECRET !keepalive mtu=1428 name=XXX remote-address=A.B.C.D tunnel-id=16
/ip firewall connection tracking
set enabled=no
4

I noticed that your config disables connection tracking. Did you happen to take note of the performance difference with and without conn-tracking enabled?

If it’s significant, I may need to look into adding a dedicated router for eoip tunnels – because I’m not sure that disabling connection-tracking would be feasible on my main gateway.

5

In my rough testing - it did show to be slower to have it enabled. IIRC around 20Mbit/s improvement with it disabled but I wouldn’t bet on this, it wasn’t in a clean lab environment. I use dedicated routers for my eoip tunnels, primarily so I can tune the setup and not have to worry about interactions with other configurations.

6

Thanks again. I think I’ll try loading a VM with RouterOS strictly for the eoip tunnels and run the rest of the connectivity and security on the main gateway.

7