I have done some testing with a commercial VPN provider using hAP ac2. The providers supports both WireGuard and IKEv2 IPSec, and I was curious to see how IPSec hardware acceleration comes into play. I used a computer with a static IP that was routed through VPN, using routing rules for WireGuard and srcnat for IPSec (to make it match the policy). Tested one scenario at a time, having disabled the other configuration.
The outcome was not what I would expect. Both options saturated my 120Mbit WAN link, so that is not a concern. With IPSec I saw one CPU core on the router hitting 70-80%, occasionally 90%. Profiling showed most CPU time on networking category. Other cores were much less used, the 4th core being hardly used at all. Overall utilization was 35-40%.
In the installed SA window I saw EH - HW-AEAD, which indicates the encryption was hardware accelerated.
With WireGuard overall CPU utilization was higher as expected, 50-60%, but it utilized all cores much better. Top core was 60-70% - lower than with IPSec. Top two cores were almost equally loaded. Profiling showed unclassified as the top category; wireguard category only a few percent. Because I don’t have to disable fasttrack with WireGuard, I watched that firewall rule graph as well. There was some fasttracking, around 3-8Mbps, some of it was likely other traffic on the network. But not enough to make a big impact on CPU in this particular test.
I have a few dozen firewall rules, queue trees enabled on interface HTB, and fasttracked enabled. For IPSec connection I had to add a rule to bypass fasttrack for the device routed through VPN. In both scenarios masquerade NAT is not fasttracked because after encapsulation the packet is originating from the router itself, hence output chain that is never fasttracked. So I don’t think fasttrack plays a role here.
The bottom line is hardware offloaded IPSec didn’t really have any advantage over CPU-based WireGuard encryption for this use case. In fact, it would hit the single-core ceiling sooner than WG. I have always thought I had IPSec acceleration in my back pocket as an extra hardware resource. But it turns out in real world the acceleration isn’t that useful. Of course, if I needed IPSec for other reasons, I’m sure without acceleration it would be much worse. I also realize not all devices have 4-core ARM CPU like hAP ac2 that can be leveraged by WireGuard so nicely. Something like hEX would very likely favor IPSec since it has higher CPU clock (better single core processing) and less cores (worse for WireGuard), although not sure if the two architectures can be compared directly.
Still, I fail to understand how it is possible that hardware accelerated IPSec was using more CPU. Yes, overall CPU was much less, but I’m not sure how much of it could be used for higher WAN bandwidth, for example, if it comes down to a single core being overloaded. The packet flow isn’t that much different between the two options:
WireGuard:
Device -> routing rule to WG -> srcnat -> WG encryption and encapsulation -> routing to WAN out -> masquerade NAT
IPSec:
Device -> routing to WAN out -> srcnat -> IPSec policy match -> IPSec HW encryption and encapsulation -> routing to WAN out -> masquerade NAT
If the above is accurate, then it probably means IPSec policy match processing is quite heavy. I also suspect only encryption is done in hardware, but encapsulation is still done in CPU. So that’s double encapsulation for IPSec (as opposed to single with WG) - ESP first and then NAT-T. Perhaps the MTU was not optimal for IPSec, so if there was more packet fragmentation, that would add to CPU utilization. Although I didn’t have to make anything special for MTU in either cases. I don’t think fasttrack plays any role here since in both cases the packets are effectively destined to the router itself, and then originate from the router itself after processing.
Am I missing something? I’m also curious to hear your experience with IPSec hardware acceleration on MikroTik devices and how it affects CPU.
