I have two sites connected with IPsec in tunnel mode.
Site A: CCR1036 with 100/100 internet connection
Site B: RB3011 with 120/30 internet connection (LTE connection, so rtt is around 30ms)
The IPsec link is working, however there are some issues with the performance.
Using iperf3 on two PC’s, one at each site, if I use 30 parallel connections I can saturate the 100 mbit link, and in reverse the 30 mbit, so that is good. If however I use only a single iperf connection, the best i can achieve is around 20mbit, but 15 is more realistic. This is a big problem, because there is only a single user at the remote site using a single FTP and/or SMB connection, and it is too slow to work with.
Looking at the datasheet of the devices i can see that the single tunnel ipsec performance is around 60 and 40 mbit/s respectively for 64byte packets, which is the worst case. My question is, why I am not hitting at least 40mbit then?
So far what I tried: aes-128-cbc, aes-128-ctr, camellia-128 (but even with software encryption, with 30 parallel connections maxing out the link, i only get around 45% cpu usage on the rb3011, even less on the ccr). I have checked mtu settings and mss clamping, there is no fragmentation as far as i can tell, and pmtud is not blocked.
My guess would be the long RTT … it affects TCP throughput quite seriously. Depending on tunnel type it can already affect performance of tunnel itself. The effect on SMB and FTP will be there as well. Did you try with UDP test using iperf?
With an iperf BW limit of 20 mbit i get around the same values (15-20 mbit), but as soon as i push to 30 mbit limit, i get a ton of OUT OF ORDER packets reported in iperf, with 20-30% lost datagrams. Speed more consistent around 22-23 mbit.
If i push to 40 mbit limit the speed is more erratic, jumping between 15-27 mbit, but i get a lot more out of order packets, with a 45% lost datagram stat.
Going down to 10 mbit limit, almost no out of order packet, and lost datagram stats down to 1-2%. (with 10mbit consistent speed achieved)
Just for testing, pushing at 100mbit, yields erratic speeds between 9-35 mbit, many OOO packets, and 60-90% datagram loss.
CPU usage never went above 15% during the udp tests, aes-128-ctr was used.
EDIT: if i reverse the direction, and the rb3011 is sending, with 100mbit limit, I barely receive any out of order packets on the ccr side. speed consistent 16-19 mbit. (80%+ datagaram loss)
It looks as if encryption of individual packets of a single ESP stream was distributed among multiple CPU cores on the CCR, causing the packets to be sent in swapped order. Are you running an up-to-date RouterOS release on the CCR? If yes, it is worth a support ticket or replacing the CCR by another device.