hAP ax lite (L41G-2axD) - poor IPSec performance

[bbs2web]

Hi,

Have a 2025 hAP ax lite, running RouterOS 7.19.3, with a L2TP IPSec tunnel where AES hardware offloading is reported as being active. I’m however only able to achieve 10 Mbps throughput whilst one of the cores reaches 100% utilisation.

Test speed results don’t detail IPSec performance but otherwise shows good throughput in synthetic tests: https://mikrotik.com/product/hap_ax_lite#fndtn-testresults

Viewing security associations shows them making use of hardware offloading:

[admin@L41G-2axD] > /ip ipsec/installed-sa/print        
Flags: S - SEEN-TRAFFIC; H - HW-AEAD; E - ESP
Columns: SPI, STATE, SRC-ADDRESS, DST-ADDRESS, AUTH-ALGORITHM, ENC-ALGORITHM, ENC-KEY-SIZE
#     SPI        STATE   SRC-ADDRESS          DST-ADDRESS          AUTH-ALGORITHM  ENC-ALGORITHM  ENC-KEY-SIZE
0 SHE 0x26DA5E3  mature  51.49.23.89:4500     192.168.10.188:4500  sha1            aes-cbc                 256
1 SHE 0x41CFAE1  mature  192.168.10.188:4500  51.49.23.89:4500     sha1            aes-cbc                 256
2 SHE 0x7D03E1B  mature  51.49.22.39:4500     192.168.10.188:4500  sha1            aes-cbc                 128
3 SHE 0x168BFD3  mature  192.168.10.188:4500  51.49.22.39:4500     sha1            aes-cbc                 128

Any suggestions, we upgraded from a hAP ac due to performance on that also having been limited to 10 Mbps and CPUs maxing out, was expecting hardware offloading on the ARM architecture to provide considerably faster performance.

Regards
David Herselman

[lurker888]

Probably this:

The changelog for 7.20beta2 includes the following:
*) ipsec - fixed degraded IPsec performance for IPQ-6010 (introduced in v7.17);

Updating to version 7.20beta5 brings performance for aes128-cbc/sha256 up to expected.

Linky:

HAP AX2 IPSEC performance vs Wireguard

Well, well, well… I redid the test just for you, just this one time… and I got similar results to yours. Somehow aes128-cbc/sha1 or sha256 loads the cpu excessively. GCM modes give the results I expect and that I’ve outlined above. I don’t recall the details of my previous testing. It might not have involved cbc/hmac modes or this may be a regression. Oh, I did the testing on 7.19.3 as well. The “Hardware AEAD” flag is shown in all configured modes.

[bbs2web]

Thanks, the change log however refers to the IPQ-6010, whereas the hAP ax lite has a IPQ-5010 SoC.

The table showing which device CPUs support AES hardware offloading unfortunately doesn’t include the IPQ-5010:

IPsec - RouterOS - MikroTik Documentation

It however does include a search filter for the RouterBoard products (here) which does include the hAP ax lite. It however is a worry that this filter also includes the hAP ac as supporting AES hardware offloading when it definately doesn’t.

[lurker888]

They use the same offloading. BTW it was introduced in

What’s new in 7.10 (2023-Jun-15 08:17):
*) ipsec - added hardware acceleration support for IPQ-5010 (hAP ax lite);

Did you test the suggestion or did you just dismiss it?

[chechito]

i really hope you find a solution for this case
but
for better outcome i think for this kind of requirement nowadays at minimum a hAP-ax² is suggested

[bbs2web]

I’ve pre-downloaded 7.20b6 packages and requested a maintenance window during which to perform the upgrade. Should be able to do this tomorrow, waiting for someone to be available to swap the device out should there be any problems with the beta.

Will provide feedback and thank you to everyone that’s provided suggestions or information!

[bbs2web]

Hrm… So 7.20b6 does provide better performance but it’s far below the expectations we had.

L2TP IPSec tunnel with WiFi:

L2TP IPSec tunnel with ethernet:

Surprised to see the firewall and WiFi components using so much CPU time, especially at only 40 Mbps thoughput. Even when using ethernet, instead of WiFi, the speed is better but still only 40 Mbps…

[Amm0]

Your speeds in Ookla could be TCP fragmentation. It does not run a very long time, so TCP sometime does not have enough time to ramp up. I’d cross-check its results with something like https://speed.cloudflare.com or https://www.nperf.com/en. Those run the a longer test and show a bit more data.

But you may need some action=adjust-mss in your mangle rules to deal with the lower MTU. PMTU/TCP will adjust to the lower MTU. But need a few more detail before going down that road (i.e. your config).

Certainly better hardware might help. But I think the hAP ax Lite should do a little better than that but others with the actual unit might offer better advice on what’s expected on these (since I don’t have one).

[bbs2web]

We have working Path MTU discovery, reviewing packet captures does not show MTU fragmentation to be an issue.

Configuration summary:

• All ethernet ports are attached to bridge, using PVID=100 on ether1 to associate upstream traffic with upstream VLAN.
• Created vlan100 interface, attached to bridge, with DHCP to obtain IPT for router
• Created IPSec encrypted L2TP client and obtain default gateway in that, this interface is then associated with a VRF)
• WiFi is using RADIUS EAP to drop clients in to appropriate VLAN
• Ethernet ports 2-4 are setup for 802.1X to drop clients in to appropriate VLAN
• Corporate network is another VLAN interface on the bridge

Bridge:

Interface hierarchy:

ie: WiFi or ethetnet clients are generally associated with the bridge within VLAN 50 after being authed via RADIUS EAP on either WiFi or network ports.

PS: The vlan50_pfdhcp MACVLAN interface is so that we can run DHCP server on interface vlan50 and also relay DHCP requests to the DHCP snooping listener on PacketFence, so that it can automatically update the central firewall’s identity awareness (ie: 192.168.18.x is davidh).

This router’s L2TP client interface configuration:

/interface l2tp-client
add allow=mschap2 connect-to=51.49.23.89 disabled=no ipsec-secret=*************************************** name=vpn*** password=**************** use-ipsec=yes user=branch1
add allow=mschap2 connect-to=31.45.8.4 disabled=no ipsec-secret=*************************************** name=vpnstream-***** password=**************** use-ipsec=yes user=branch1

Head end MikroTik’s configuration (RO 6.49.18):

/interface l2tp-server server
  set authentication=mschap2 enabled=yes ipsec-secret=*************************************** max-mru=1379 max-mtu=1379 use-ipsec=required
/ppp profile
  set *FFFFFFFE only-one=yes
/ppp secret
  add local-address=100.127.252.37 name=branch1 password=**************** remote-address=100.127.252.38 service=l2tp

I’ll try raise a support query to see if MikroTik feels that this is expected performance of the dual core ARM platform.

[lurker888]

I can spare you the trouble. 60-80 Mbps is about the most you can expect in this setup. If you’re unwilling or unable to avoid fragmentation, what you’re seeing is approximately what’s expected.

[mkx]

@bbs2web

We have working Path MTU discovery, reviewing packet captures does not show MTU fragmentation to be an issue.

AFAIK L2 tunnels don’t behave in the way PMTUD would detect the lower MTU. Instead they fragment original payload packets, transport fragments over tunnel and re-assemble them on tunnel exit. So the fact that they can not transport whole payload packets is hidden from upper layers. However, packets still get fragmented and the overhead of fragmentation/defragmentation (both in terms of CPU power and memory needed on both tunnel ends and in terms of transit overhead/latency) is very much present.
So you better adjust the “L2 MTU” of tunnel interface to the tunnel max payload size … L3 MTU will adjust accordingly and then PMTUD will be able to do its job. And do what @Amm0 suggested (regarding setting adjust-mss).