Community discussions

MikroTik App
 
mcdouglas
just joined
Topic Author
Posts: 3
Joined: Fri Aug 14, 2020 4:31 pm

CCR1036/RB3011 IPsec throughput issue

Fri Nov 04, 2022 11:17 am

Hi!

I have two sites connected with IPsec in tunnel mode.

Site A: CCR1036 with 100/100 internet connection
Site B: RB3011 with 120/30 internet connection (LTE connection, so rtt is around 30ms)

The IPsec link is working, however there are some issues with the performance.

Using iperf3 on two PC's, one at each site, if I use 30 parallel connections I can saturate the 100 mbit link, and in reverse the 30 mbit, so that is good. If however I use only a single iperf connection, the best i can achieve is around 20mbit, but 15 is more realistic. This is a big problem, because there is only a single user at the remote site using a single FTP and/or SMB connection, and it is too slow to work with.

Looking at the datasheet of the devices i can see that the single tunnel ipsec performance is around 60 and 40 mbit/s respectively for 64byte packets, which is the worst case. My question is, why I am not hitting at least 40mbit then?

So far what I tried: aes-128-cbc, aes-128-ctr, camellia-128 (but even with software encryption, with 30 parallel connections maxing out the link, i only get around 45% cpu usage on the rb3011, even less on the ccr). I have checked mtu settings and mss clamping, there is no fragmentation as far as i can tell, and pmtud is not blocked.

Any recommendations? short of replacing the gear...
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11383
Joined: Thu Mar 03, 2016 10:23 pm

Re: CCR1036/RB3011 IPsec throughput issue

Fri Nov 04, 2022 12:39 pm

My guess would be the long RTT ... it affects TCP throughput quite seriously. Depending on tunnel type it can already affect performance of tunnel itself. The effect on SMB and FTP will be there as well. Did you try with UDP test using iperf?
 
mcdouglas
just joined
Topic Author
Posts: 3
Joined: Fri Aug 14, 2020 4:31 pm

Re: CCR1036/RB3011 IPsec throughput issue

Fri Nov 04, 2022 1:12 pm

UDP test is interesting.

With an iperf BW limit of 20 mbit i get around the same values (15-20 mbit), but as soon as i push to 30 mbit limit, i get a ton of OUT OF ORDER packets reported in iperf, with 20-30% lost datagrams. Speed more consistent around 22-23 mbit.

If i push to 40 mbit limit the speed is more erratic, jumping between 15-27 mbit, but i get a lot more out of order packets, with a 45% lost datagram stat.

Going down to 10 mbit limit, almost no out of order packet, and lost datagram stats down to 1-2%. (with 10mbit consistent speed achieved)

Just for testing, pushing at 100mbit, yields erratic speeds between 9-35 mbit, many OOO packets, and 60-90% datagram loss.

CPU usage never went above 15% during the udp tests, aes-128-ctr was used.

EDIT: if i reverse the direction, and the rb3011 is sending, with 100mbit limit, I barely receive any out of order packets on the ccr side. speed consistent 16-19 mbit. (80%+ datagaram loss)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: CCR1036/RB3011 IPsec throughput issue

Fri Nov 04, 2022 2:48 pm

It looks as if encryption of individual packets of a single ESP stream was distributed among multiple CPU cores on the CCR, causing the packets to be sent in swapped order. Are you running an up-to-date RouterOS release on the CCR? If yes, it is worth a support ticket or replacing the CCR by another device.
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: CCR1036/RB3011 IPsec throughput issue

Mon Nov 07, 2022 10:15 pm

mcdouglas
This is a known issue on MT hardware routers with ipsec tunnels and high-latency link: viewtopic.php?t=146665#p769858
You must use a CHR - this software router do not have this issue.

Who is online

Users browsing this forum: ItchyAnkle, JDF, RobertsN and 72 guests