Hello, I have a problem with IPSEC performance.
I have the RB4011 and hAP ac2 connected directly via an ethernet cable.
4011 is the gateway for ac2, with ac2 I perform a bandwith test to the server and the local traffic exchange node. It then gets almost 1Gb/s and all cores in ac2 are maximally used. However, when I encrypt communication 4011 <-> ac2 I get only 200Mb/s where on the product page ac2 is written about 400Mb/s interestingly during tests with IPSEC the processor in ac2 is used at only 45% (one core 100% two 40% and the last 0%). Both devices are ROS version 6.45.1.

In the attachment I am sending a screen with tunnel parameters.

Please help, Thank you in advance.

Note that published results are strictly synthetic and achieved with only plain IPsec tunnel configured on the router. For example, connection tracking can significantly reduce the encrypted throughput. Also if you are using L2TP, it creates additional overhead thus bringing the encrypted throughput even lower.

Note that published results are strictly synthetic and achieved with only plain IPsec tunnel configured on the router. For example, connection tracking can significantly reduce the encrypted throughput. Also if you are using L2TP, it creates additional overhead thus bringing the encrypted throughput even lower.

The tunnel I made is a regular IPSEC (without L2TP, IP-IP, etc). The behavior is all the more strange because the IPSEC short circuit has incomplete use so theoretically there are resources to handle more traffic.

the IPSEC short circuit has incomplete use so theoretically there are resources to handle more traffic.

Wojtek, can you please reword this sentence (or say it in Polish and/or use more words, whatever)? I don't understand what "incomplete use" means here.

Other than that, have you used packet size 1400 bytes to test the throughput as the product page shows? The thing is that the IPsec adds some headers and sha authentication signature to the contents of the payload packet, so if the payload packet doesn't fit into a single transport packet due to this added margin, it is silently fragmented into two transport packets (and silently reassembled at the receiving end), and there is no way how to convey the actual MTU value to the sender as there is no interface in the path to which the MTU could be associated.

So to get the optimum, you need test packets which exactly fit - if they are larger, you get double the number of transport packets, if they are smaller, the pps stays unchanged but bps decreases.

By "incomplete use" I meant that the processor with IPSEC enabled was not fully used.
Honestly, I would not look for problems with MTU. More with h2 ac2 performance.

In the attachment I am sending screen of devices between which I am doing the test.

I am wondering about IRQ called qca_crypto. As if IPSEC was supported by software and not hardware.
On the left you can see 4011 (both devices work on ARM processors) and there is no such IRQ.

In the attachment I am sending screen of devices between which I am doing the test.

Looks like you're testing single core performance of a hAP ac2 by single threaded b-test here.

In the attachment I am sending screen of devices between which I am doing the test.

Looks like you're testing single core performance of a hAP ac2 by single threaded b-test here.

Ok, but see results with IPSEC off - the traffic spreads to all cores.
With IPSEC enabled one core is maximally saturated and the rest have a power reserve.

Honestly, I would not look for problems with MTU. More with h2 ac2 performance.

I cannot imagine Mikrotik intentionally publishing inflated test results, I can imagine them (like any other vendor) to publish results obtained under ideal conditions (which @Emils has suggested). So I've suggested my real experience. It's not "problems with MTU", it's how fragmentation affects network performance in general. Note that the issue is so significant that IPv6 has changed the rules how fragmentation may be handled.

In the attachment I am sending screen of devices between which I am doing the test.

(147.700.000 Mbits/s / 12.170 packets/s) / 8 bits/byte = 1517 bytes/packet, which means that the fragmentation is not the issue here - if it was, the result would be around 800 bytes/packet.

So @McSee's suggestion is the next one to deal with, as in the btest manual you can see a Note:
Bandwidth Test uses a lot of resources. If you want to test real throughput of a router, you should run bandwidth test through the tested router, not from or to it.

The IPsec performance degrades when it has to rearrange received packets, which is why the sending direction of any given SA is forcifully handled by a single core, and also all handling of any given packet tends to be done on a single core for performance reasons. So chances are high that btest sends the packets using the same core which encrypts them, maxing that core out.

I am wondering about IRQ called qca_crypto. As if IPSEC was supported by software and not hardware.
On the left you can see 4011 (both devices work on ARM processors) and there is no such IRQ.

That's a good point, however I'd rather expect it to indicate the reverse - a "crypto" interrupt suggests that the CPU is notified about "encryption finished" event by means of that interrupt when the hardware has finished the encryption.

I am absolutely not saying that the results given by MikroTik are distorted.
However, even after you have applied the steps you used, the speed is still around 230Mbps.
Performs iperf3 from a computer on the local network. On hAP ac2, connecion tracking is off.
ehter2 - LAN
ether5 - WAN

My config:

When using multiple IPSEC connections side by side the traffic tends hurdles still together on one core.

ehter2 - LAN
ether5 - WAN

I've used this information to double-check on your screenshot that the number of bytes and packets is similar on the plaintext (ether2) and encrypted (ether5) side, so I'm officially out of ideas.

However, even after you have applied the steps you used, the speed is still around 230Mbps.

These are pretty good numbers for IPsec single client / TCP, I've seen similar performance on RB750Gr3, which is pretty close to hAP ac2 in IPsec perf, in my quick tests.

I tried something else. Currently using an SFP for the WAN and that has in the RB760iGS (same processor as the RB750GR3). I connect it up differently replacing the SFP up-link with a ethernet port connected to a media converter. Now the usage of the processor is bit better and the firewall/network load is also more shifted to the other cores. I now read speeds over 240Mbit/s and before it was around 180Mbit/s.
Blockdiagram of the RB760iGS where can see that inserting a SFP split the bandwith and it would been better if it would be eth1 or SFP and not both usable at the same time.


Update:
Tested it with only NordVPN IKEv2 connection and I get with them 180/200Mbit/s (down/up).

I cannot imagine Mikrotik intentionally publishing inflated test results

Be sure they do it.
They use a few tricks to mislead:
1) they use UDP instead of TCP, despite the fact that ALL file transfer protocols (FTP, HTTP, SCP, SFTP, SMB) use TCP.
VPN tunnels on Mikrotik hardware routers shows good speed with UDP, but has very poor speed with TCP - you can find many topics on the forum about these issues, for example viewtopic.php?t=172641.
2) they use two streams (instead of single stream) for single tunnel, which doubles the total Mbps (one stream is limited by the speed of one core).

For real life results they should publish the results of TCP single stream! (and preferably using an iperf, not a Traffic Generator)
viewtopic.php?t=179779
viewtopic.php?t=97880

I use multiple streams to obtain maximum results.With Wireguard available this not needed anymore and I get great performance out of the box (RB4011).

An UDP tunnel is the best way to encapsulate traffic, any traffic. Inside the tunnel TCP can do it's work as normal.

Non tunneled encrypted traffic has advantages but is that that much better? You have then packet able to take different paths to the destination and have larger MTU.

Different paths is comparable to running multiple tunnels.But it depends if you a a server or a client.

for ipsec i'm using hEX S on both sites.

@WojtusW5 given me an idea ,im not sure whether is possible or not.

So hEX S it self has 4 CPU Count.
Im wondering it is possible to resevated 2 CPU Count for 19 crypto.
Even if it is possible, should i expect any improvement?

(147.700.000 Mbits/s / 12.170 packets/s) / 8 bits/byte = 1517 bytes/packet, which means that the fragmentation is not the issue here - if it was, the result would be around 800 bytes/packet.

sorry @sindy quick one
from where 8 bits/byte comming from?

from where 8 bits/byte comming from?

Someone's decision in late 1960s I guess, as the number of bits had to be a "round" power of 2 and 4 was already too few by that time? https://en.wikipedia.org/wiki/8-bit_computing

If you have in mind that the calculation is not precise because on top of the actual contents, some extra bits are spent on Ethernet preamble and CRC, you are right, but that's a negligible error. The fact that the minimum size of Ethernet payload is 60 bytes is more important I guess, but it was a rough calculation.