IPsec tunnel, some specific packet size issue

RouterOS General
1

Hello,

I'm facing a very strange IPsec bug.

What I found is that using hardware-accelerated IPsec algorithms, some packets with a specific size are not properly received on the other side of a IPsec tunnel.
Generating in-state-protocol-errors IPsec errors.

Reproduced in 7.19.6, 7.20.8, 7.23.1 (did not test other versions).
On RB4011, RB5009, CCR2116 (did not test other devices).
So on AL21400, AL73400 and 88F7040 CPUs.

How to reproduce is below.

Take 2 devices, A (RB5009 / RB4011) and B (CCR2116).

Connect A and B through the same LAN, let's say :
A : 10.10.10.1/24
B : 10.10.10.2/24

Add a client LAN to A, let's say 192.168.20.0/24.

Add a loopback address to B, let's say 10.20.20.1/24.

Finally add a IPsec IKEv2 tunnel from A to B thanks to 10.10.10.0/24, from 192.168.20.0/24 to 10.20.20.0/24, typical configuration below (*).

OK, we have our test lab.

Then, from a client in LAN 192.168.20.0/24, do the following :
i=0; while [[ $i -le 2000 ]]; do ping -W 1 -c 1 -s $i 10.20.20.1 >/dev/null 2>1 || echo $i; i=$(($i+1));done

Do the test multiple times, modifying phase 2 encryption algorithms.
Choosing hardware-accelerated algorithms, such as aes-256 cbc, ctr or gcm, you will find that some ICMP packets are not properly received on the other tunnel side, generating in-state-protocol-errors IPsec errors.
There's no issue with software algorithms, such as chacha20.

Depending on the choosen algorithm, you will find that failing packet size is around 1390 / 1440 bytes, and that it’s OK below, and OK above ! Which makes things really strange.

I initially did these tests to be sure not to have MTU issue through my new IPsec tunnel, and finally found this really strange behaviour...

I opened a support ticket as it sounds like a bug, but in the meantime, did I perhaps miss something ?

Many thanks :+1:

(*) Typical IPsec configuration :

/ip ipsec policy group
add name=TUN_Test
/ip ipsec profile
add dh-group=ecp256 enc-algorithm=aes-256 hash-algorithm=sha512 lifetime=6h name=TUN_Test proposal-check=claim
/ip ipsec peer
add address=10.10.10.1/32 exchange-mode=ike2 local-address=10.10.10.2 name=TUN_Test passive=yes profile=TUN_Test
/ip ipsec proposal
add auth-algorithms=sha512 enc-algorithms=aes-256-ctr lifetime=6h name=TUN_Test pfs-group=ecp256
/ip ipsec identity
add comment=TUN_Test peer=TUN_Test policy-template-group=TUN_Test secret=goodmorning
/ip ipsec policy
add dst-address=192.168.20.0/24 peer=TUN_Test proposal=TUN_Test src-address=10.20.20.0/24 tunnel=yes

2

Hi! First I’d try to separate a real MTU/fragmentation issue from a possible hardware-accelerated issue, for example by testing with a lower client MTU/MSS, then with a higher L2/underlay MTU if possible, and comparing with IPsec hardware accelerated enabled vs disabled.

I’d also try to see which side matters most: swap A/B roles if possible, and test MikroTik-to-MikroTik versus another IPsec peer if you have one.

If the same packet sizes only fail with accelerated AES but not with a non-accelerated (ie software-only algorithm like Camellia, etc) then it might point away from a normal policy/proposal mistake and more toward an IPsec HW acceleration edge case. Check hw-aead under installed-SA to see if the SA is hardware-accelerated or not.

It would be helpful if you could provide the full IPsec config from both sides, plus the relevant interface MTU/L2MTU, bridge/VLAN config if used, and the installed-SA and statistics counters after a failing test.

3

Hi @Larsa, thank you very much for yout reply !

A and B use a very light configuration :

  • a bridge with VLAN filtering ;
  • IPsec configuration (as given above at the end of my previous post, same on both sides, symmetrically).

Everything works properly with a software IPsec algorithm.
Everything works properly with a hardware IPsec algorithm, and without any firewall rule.
But test fails with a hardware IPsec algorithm as soon there is a firewall rule in place, whatever it is (forward, srcnat, dsnat...), even if it does not concern IPsec flows, even if it does nothing at all (a rule targetting a down port...).

Whatever the receiving device is (RB4011, RB5009, CCR2116).

Seems then like there's a edge case between hardware-accelerated algorithms and firewall rules.

Here are how the counters look on the receiving side when issue occurs :

> /ip/ipsec/statistics/print
                  in-errors:   0
           in-buffer-errors:   0
           in-header-errors:   0
               in-no-states:   0
   in-state-protocol-errors: 199
       in-state-mode-errors:   0
   in-state-sequence-errors:   0
           in-state-expired:   0
        in-state-mismatches:   0
           in-state-invalid:   0
     in-template-mismatches:   0
             in-no-policies:   0
          in-policy-blocked:   0
           in-policy-errors:   0
                 out-errors:   0
          out-bundle-errors:   0
    out-bundle-check-errors:   0
              out-no-states:   0
  out-state-protocol-errors:   0
      out-state-mode-errors:   0
  out-state-sequence-errors:   0
          out-state-expired:   0
         out-policy-blocked:   0
            out-policy-dead:   0
          out-policy-errors:   0

I've sent supout files to MikroTik, I hope they will find something interesting.

4

That info actually changes the picture pretty much.

If it works with no firewall rules, but fails as soon as any rule exists, even one that should not touch the IPsec traffic, then I’d look more at the packet path changing. I’d start by explicitly ruling out FastTrack.

Btw, in ROS, auto means conntrack is not active until firewall rules exist, so adding even an unrelated rule may change the forwarding path. It would be helpful to see the complete config, including the firewall.

5

Yes, I did additional troubleshooting and found this additional behavior...

There's no FastTrack in place, as there's no firewall rule at all, but a dummy one, to trigger the issue.

Below is my full test lab configuration.
As you can see, it's very simple, for this PoC only.

So there's only one firewall rule.
Whatever rule you configure, even if it is useless, issue will trigger.
Without any rule, it works.

And you're right, without any rule, but with conntrack enforced, it fails (good catch !).
So it rather has to do with conntrack...

/export
# 2026-06-01 10:22:24 by RouterOS 7.23.1
# software id = UQJX-FUKT
#
# model = RB4011iGS+
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge vlan-filtering=yes
/interface vlan
add interface=bridge name=LAN_A vlan-id=20
add interface=bridge name=LAN_AB vlan-id=10
add interface=bridge name=LAN_MGMT vlan-id=2
/interface list
add name=WINBOX
/ip ipsec policy group
add name=TUN_Test
/ip ipsec profile
add dh-group=ecp256 enc-algorithm=aes-256 hash-algorithm=sha512 lifetime=6h name=TUN_Test proposal-check=claim
/ip ipsec peer
add address=10.10.10.2/32 exchange-mode=ike2 name=TUN_Test profile=TUN_Test
/ip ipsec proposal
add auth-algorithms=sha512 enc-algorithms=aes-256-ctr lifetime=6h name=TUN_Test pfs-group=ecp256
/ip pool
add name=LAN_A ranges=192.168.20.65-192.168.20.246
/ip dhcp-server
add address-pool=LAN_A interface=LAN_A lease-time=12h name=LAN_A
/interface bridge port
add bridge=bridge comment=LAN_MGMT frame-types=admit-only-untagged-and-priority-tagged interface=ether5 pvid=2
add bridge=bridge comment=LAN_A frame-types=admit-only-untagged-and-priority-tagged interface=ether6 pvid=20
add bridge=bridge comment=LAN_AB frame-types=admit-only-vlan-tagged interface=ether7
/ip neighbor discovery-settings
set discover-interface-list=WINBOX
/ip settings
set secure-redirects=no send-redirects=no tcp-syncookies=yes
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge comment=LAN_MGMT tagged=bridge vlan-ids=2
add bridge=bridge comment=LAN_A tagged=bridge vlan-ids=20
add bridge=bridge comment=LAN_AB tagged=bridge,ether7 vlan-ids=10
/interface list member
add interface=LAN_MGMT list=WINBOX
/ip address
add address=192.168.20.254/24 interface=LAN_A network=192.168.20.0
add address=10.10.10.1/24 interface=LAN_AB network=10.10.10.0
add address=10.30.30.1/24 interface=lo network=10.30.30.0
/ip dhcp-server network
add address=192.168.20.0/24 comment=LAN_A dns-server=192.168.20.254 gateway=192.168.20.254
/ip firewall nat
add action=accept chain=srcnat comment="accept ipsec" ipsec-policy=out,ipsec
/ip ipsec identity
add comment=TUN_Test peer=TUN_Test policy-template-group=TUN_Test secret=goodmorning
/ip ipsec policy
add dst-address=10.20.20.0/24 peer=TUN_Test proposal=TUN_Test src-address=192.168.20.0/24 tunnel=yes
add dst-address=192.168.30.0/24 peer=TUN_Test proposal=TUN_Test src-address=10.30.30.0/24 tunnel=yes
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=10.10.10.254 routing-table=main
/ip ssh
set host-key-size=4096 strong-crypto=yes
/system identity
set name=A
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=WINBOX
/tool mac-server ping
set enabled=no
6

Yeah, that confirms the conntrack angle, so it might actually be an edge case between conntrack and HW-accelerated IPsec. Let’s see what support says about it.

Meanwhile, you could still test removing/disabling the NAT rule, forcing conntrack on, flushing the SAs, and rerunning the test. If it still fails, that would pretty much rule out the firewall/NAT rule itself and point even more directly at conntrack + HW IPsec.

For example:

/ip firewall nat disable [find]
/ip firewall connection tracking set enabled=yes
/ip ipsec installed-sa flush

Then compare with conntrack disabled:

/ip firewall connection tracking set enabled=no
/ip ipsec installed-sa flush

And maybe also test the default/auto behavior again:

/ip firewall connection tracking set enabled=auto
/ip firewall nat enable [find]
/ip ipsec installed-sa flush
7

Thank you for following this @Larsa :+1:

So :

  • test 1 : failed
  • test 2 : passed
  • test 3 : failed

It clearly points to conntrack interfering with HW IPsec.
Rather scary thing !