GRE over IPSec tunnel - unusable on RB4011 above 7.15.3

RouterOS General
1

Hello,

Any RouterOS version above 7.15.3 (7.16.x and 7.17) sees critical performance issues. Affected are all TCP streams, most noticeable is performance drop while attempting SMB file operations. Wireskark sees multiple duplicated ACKs and retransmissions. Config (working on 7.15.3), GRE addressing not redacted.

/interface/gre/export
/interface gre
add allow-fast-path=no local-address=172.16.64.21 name=gre-mdggdn remote-address=172.16.64.20
add allow-fast-path=no local-address=172.16.64.23 name=gre-mdgwaw remote-address=172.16.64.22

/ip/ipsec/export
/ip ipsec peer
add address=xxxx/32 exchange-mode=ike2 name=node-a send-initial-contact=no
add address=xxxx/32 exchange-mode=ike2 name=node-b
/ip ipsec policy group
add name=common
/ip ipsec profile
set [ find default=yes ] dh-group=ecp521 dpd-interval=30s dpd-maximum-failures=3 enc-algorithm=aes-256 hash-algorithm=sha512
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha512 enc-algorithms=aes-256-cbc pfs-group=ecp521
/ip ipsec identity
add auth-method=digital-signature certificate=*redacted* generate-policy=port-strict match-by=\
    certificate peer=node-b policy-template-group=common remote-certificate=*redacted*
add auth-method=digital-signature certificate=*redacted* match-by=certificate peer=node-a \
    policy-template-group=common remote-certificate=*redacted*
/ip ipsec policy
add dst-address=172.16.64.20/32 level=unique peer=node-a protocol=gre src-address=172.16.64.21/32 tunnel=yes
add dst-address=172.16.64.22/32 level=unique peer=node-b protocol=gre src-address=172.16.64.23/32 tunnel=yes

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN protocol=!gre
2

When using any tunnel protocol, consider making MTU changes to take tunnel overhead into account for preventing packet fragmentation.

3

Quick calculations here:
1500B - (8B PPPoE) - (20B IPSec header) - (8B Nat-T) - (8B ESP header) - (16B ESP IV) - (24B GRE) - (34B ESP trailer) = 1382 bytes of actual MTU

both sides have this value, I even forced it manually to cross check - that didn’t help. Also, lowering it down to 1300 haven’t helped either.

4

TL;DR how does the CPU load (/tool/profile cpu=all) look under the traffic load?

There have been some changelog notes in last few versions regarding hardware AES acceleration specifically on the Alpine SoC family being broken and fixed and fixed again, etc.
RB4011 has an Alpine chip, so this might be somehow related - even though the changelogs were specifically highlighting AES-GCM which is not what you have in your config.

5

I ended with 1376 as working common number for IPSec GRE/IPIP over different providers across Poland.

6

@BartoszP - I don’t think this is really MTU issue. Autonegotiated MTU actually aligns with one I’ve calculated and lowering it does not help either. I may test it in direct connection with one of the peers to test against NAT-T or other potentially offending voodoo.

@wrkq - on 7.15.3 I see a spike during transfer up to 90% of single core utilization, more typical:

# 2025-01-26 22:02:06 by RouterOS 7.15.3
# software id = RNLB-4V15
#
Columns: NAME, CPU, USAGE
NAME          CPU  USAGE
console         0  0%   
firewall        0  0%   
networking      0  3.5% 
winbox          0  1%   
logging         0  0.5% 
management      0  2%   
encrypting      0  0%   
routing         0  0%   
profiling       0  1%   
bridging        0  0%   
unclassified    0  1.5% 
cpu0               9.5% 
console         1  0%   
firewall        1  0.5% 
networking      1  2%   
winbox          1  0%   
management      1  1.5% 
encrypting      1  0%   
routing         1  0%   
profiling       1  0.5% 
bridging        1  0%   
unclassified    1  0%   
cpu1               4.5% 
ethernet        2  0%   
console         2  0.5% 
firewall        2  0%   
networking      2  0%   
winbox          2  0%   
management      2  1%   
routing         2  0%   
profiling       2  0%   
bridging        2  0%   
unclassified    2  0%   
cpu2               1.5% 
ethernet        3  0.5% 
console         3  0%   
firewall        3  12%  
networking      3  40.5%
management      3  0%   
encrypting      3  0%   
routing         3  8.5% 
profiling       3  1%   
bridging        3  1%   
unclassified    3  2.5% 
cpu3               66%

With 7.17 I get this:

# 2025-01-26 22:08:13 by RouterOS 7.17
# software id = RNLB-4V15
#
Columns: NAME, CPU, USAGE
NAME        CPU  USAGE
ppp           0  0%   
networking    0  0%   
management    0  0%   
telnet        0  0%   
console       0  0%   
routing       0  0.5% 
wireless      0  0%   
cpu0             0.5% 
networking    1  0%   
management    1  0.5% 
console       1  0%   
wireless      1  0.5% 
firewall      1  0%   
kernel        1  0%   
led           1  0%   
cpu1             1%   
ppp           2  0%   
networking    2  0%   
management    2  1.5% 
winbox        2  0.5% 
console       2  1%   
crypto        2  0%   
routing       2  0%   
wireless      2  0.5% 
profiling     2  0%   
kernel        2  0.5% 
bridge2       2  1%   
cpu2             5%   
networking    3  0%   
management    3  2.5% 
winbox        3  0%   
console       3  0.5% 
wireless      3  2%   
firewall      3  0%   
bridging      3  0%   
cpu3             5%

but no transfer is actually seen.

7

@mwisniewski, how do iperf3 tests for UDP or TCP with different packet sizes impact throughput and CPU usage? Have you tried different algorithms for IPsec Hardware acceleration? Is there an RB4011 on both ends?

8

I had the same issue and worked through it with Mikrotik support. The issue is that they broke how GRE does segment reassembly then egresses out an interface with a lower MTU. Their work around which works for me is:

/ip firewall mangle add chain=prerouting action=clear-df in-interface=gre1

This removes the DF bit and allows the traffic to flow fragmented through the smaller interface where it's reassembled later in the chain.

They say they are working on a better fix, but not sure when that will come out.

9

Would you mind adding a little more context? I am nur sure I understand you descrption connectly:

Interface1 MTU 1500 >>>> GRE-Tunnel (MTU 1370) >>> Interface2 MTU 1450

Is the affected setup something like that? What is the best way to test if you are affected?

10

Mikrotik will know more, but from what I gather the GRE-Tunnel is reassembling segments and ends up with a frame larger than 1370 and drops it.

I experimented with clamping mss to 1200 and running a GRE interface at 2000 so that there was never a reason to fragment, but I couldn't make it work. Mikrotik support says they will have a fix in an upcoming release and recognize it as a bug, but for now clearing the DF bit make GRE tunnels work on firmware newer than 7.15.3.

11

Just curious, what speeds do you get before / after the proposed fix?

I run a very similar setup to yours and I struggle with the poor tunnel throughput. I found no direct relation to the ROS version I use. And when I say poor - it's really poor, like Windows Samba file transfer speeds at around 300-600KBps (jumping up to about 1.5MBps on good days - with no obvious reason). That's on a 1G Internet connection at both ends...

12

Before the update speeds were kilobits a second as it fought through the drops. Afterwards I get 3.3MB/sec but that's through various other tunnels so that number doesn't mean much other than to say, fast enough for VPN stuff I do.