Community discussions

 
WojtusW5
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 64
Joined: Mon Oct 02, 2017 1:25 pm

IPSEC performance problem

Thu Jul 18, 2019 11:15 am

Hello, I have a problem with IPSEC performance.
I have the RB4011 and hAP ac2 connected directly via an ethernet cable.
4011 is the gateway for ac2, with ac2 I perform a bandwith test to the server and the local traffic exchange node. It then gets almost 1Gb/s and all cores in ac2 are maximally used. However, when I encrypt communication 4011 <-> ac2 I get only 200Mb/s where on the product page ac2 is written about 400Mb/s interestingly during tests with IPSEC the processor in ac2 is used at only 45% (one core 100% two 40% and the last 0%). Both devices are ROS version 6.45.1.

In the attachment I am sending a screen with tunnel parameters.

Please help, Thank you in advance.
You do not have the required permissions to view the files attached to this post.
I invite you to visit my blog
https://mikrotikon.pl/
 
User avatar
emils
MikroTik Support
MikroTik Support
Posts: 505
Joined: Thu Dec 11, 2014 8:53 am

Re: IPSEC performance problem

Thu Jul 18, 2019 11:37 am

Note that published results are strictly synthetic and achieved with only plain IPsec tunnel configured on the router. For example, connection tracking can significantly reduce the encrypted throughput. Also if you are using L2TP, it creates additional overhead thus bringing the encrypted throughput even lower.
 
WojtusW5
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 64
Joined: Mon Oct 02, 2017 1:25 pm

Re: IPSEC performance problem

Thu Jul 18, 2019 11:51 am

Note that published results are strictly synthetic and achieved with only plain IPsec tunnel configured on the router. For example, connection tracking can significantly reduce the encrypted throughput. Also if you are using L2TP, it creates additional overhead thus bringing the encrypted throughput even lower.
The tunnel I made is a regular IPSEC (without L2TP, IP-IP, etc). The behavior is all the more strange because the IPSEC short circuit has incomplete use so theoretically there are resources to handle more traffic.
I invite you to visit my blog
https://mikrotikon.pl/
 
sindy
Forum Guru
Forum Guru
Posts: 3926
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPSEC performance problem

Thu Jul 18, 2019 1:16 pm

the IPSEC short circuit has incomplete use so theoretically there are resources to handle more traffic.
Wojtek, can you please reword this sentence (or say it in Polish and/or use more words, whatever)? I don't understand what "incomplete use" means here.

Other than that, have you used packet size 1400 bytes to test the throughput as the product page shows? The thing is that the IPsec adds some headers and sha authentication signature to the contents of the payload packet, so if the payload packet doesn't fit into a single transport packet due to this added margin, it is silently fragmented into two transport packets (and silently reassembled at the receiving end), and there is no way how to convey the actual MTU value to the sender as there is no interface in the path to which the MTU could be associated.

So to get the optimum, you need test packets which exactly fit - if they are larger, you get double the number of transport packets, if they are smaller, the pps stays unchanged but bps decreases.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
WojtusW5
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 64
Joined: Mon Oct 02, 2017 1:25 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 12:02 am

By "incomplete use" I meant that the processor with IPSEC enabled was not fully used.
Honestly, I would not look for problems with MTU. More with h2 ac2 performance.

In the attachment I am sending screen of devices between which I am doing the test.

I am wondering about IRQ called qca_crypto. As if IPSEC was supported by software and not hardware.
On the left you can see 4011 (both devices work on ARM processors) and there is no such IRQ.
You do not have the required permissions to view the files attached to this post.
I invite you to visit my blog
https://mikrotikon.pl/
 
McSee
Frequent Visitor
Frequent Visitor
Posts: 67
Joined: Tue Feb 26, 2019 12:49 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 12:46 am

In the attachment I am sending screen of devices between which I am doing the test.
Looks like you're testing single core performance of a hAP ac2 by single threaded b-test here.
 
WojtusW5
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 64
Joined: Mon Oct 02, 2017 1:25 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 10:19 am

In the attachment I am sending screen of devices between which I am doing the test.
Looks like you're testing single core performance of a hAP ac2 by single threaded b-test here.
Ok, but see results with IPSEC off - the traffic spreads to all cores.
With IPSEC enabled one core is maximally saturated and the rest have a power reserve.
You do not have the required permissions to view the files attached to this post.
I invite you to visit my blog
https://mikrotikon.pl/
 
sindy
Forum Guru
Forum Guru
Posts: 3926
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 10:33 am

Honestly, I would not look for problems with MTU. More with h2 ac2 performance.
I cannot imagine Mikrotik intentionally publishing inflated test results, I can imagine them (like any other vendor) to publish results obtained under ideal conditions (which @Emils has suggested). So I've suggested my real experience. It's not "problems with MTU", it's how fragmentation affects network performance in general. Note that the issue is so significant that IPv6 has changed the rules how fragmentation may be handled.

In the attachment I am sending screen of devices between which I am doing the test.
(147.700.000 Mbits/s / 12.170 packets/s) / 8 bits/byte = 1517 bytes/packet, which means that the fragmentation is not the issue here - if it was, the result would be around 800 bytes/packet.

So @McSee's suggestion is the next one to deal with, as in the btest manual you can see a Note:
Bandwidth Test uses a lot of resources. If you want to test real throughput of a router, you should run bandwidth test through the tested router, not from or to it.

The IPsec performance degrades when it has to rearrange received packets, which is why the sending direction of any given SA is forcifully handled by a single core, and also all handling of any given packet tends to be done on a single core for performance reasons. So chances are high that btest sends the packets using the same core which encrypts them, maxing that core out.

I am wondering about IRQ called qca_crypto. As if IPSEC was supported by software and not hardware.
On the left you can see 4011 (both devices work on ARM processors) and there is no such IRQ.
That's a good point, however I'd rather expect it to indicate the reverse - a "crypto" interrupt suggests that the CPU is notified about "encryption finished" event by means of that interrupt when the hardware has finished the encryption.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
WojtusW5
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 64
Joined: Mon Oct 02, 2017 1:25 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 11:49 am

I am absolutely not saying that the results given by MikroTik are distorted.
However, even after you have applied the steps you used, the speed is still around 230Mbps.
Performs iperf3 from a computer on the local network. On hAP ac2, connecion tracking is off.
ehter2 - LAN
ether5 - WAN

My config:
/ip ipsec peer
add address=192.168.3.1/32 name=peer_test
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha256
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp2048
/ip pool
add name=dhcp_pool0 ranges=192.168.230.2-192.168.230.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=ether2 lease-time=1h name=dhcp1
/ip firewall connection tracking
set enabled=no
/ip address
add address=192.168.230.1/24 interface=ether2 network=192.168.230.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether5 use-peer-dns=no use-peer-ntp=no
/ip dhcp-server network
add address=192.168.230.0/24 dns-server=8.8.8.8 gateway=192.168.230.1
/ip ipsec identity
add generate-policy=port-strict peer=peer_test secret=EDF#4tG@%4tgf
/ip ipsec policy
add dst-address=178.216.40.125/32 peer=peer_test sa-dst-address=192.168.3.1 sa-src-address=0.0.0.0 src-address=192.168.230.0/24 tunnel=yes
/ip route
add distance=1 gateway=192.168.3.1
You do not have the required permissions to view the files attached to this post.
I invite you to visit my blog
https://mikrotikon.pl/
 
msatter
Forum Guru
Forum Guru
Posts: 1285
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: IPSEC performance problem

Fri Jul 19, 2019 12:22 pm

When using multiple IPSEC connections side by side the traffic tends hurdles still together on one core.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
sindy
Forum Guru
Forum Guru
Posts: 3926
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 12:43 pm

ehter2 - LAN
ether5 - WAN
I've used this information to double-check on your screenshot that the number of bytes and packets is similar on the plaintext (ether2) and encrypted (ether5) side, so I'm officially out of ideas.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
McSee
Frequent Visitor
Frequent Visitor
Posts: 67
Joined: Tue Feb 26, 2019 12:49 pm

Re: IPSEC performance problem

Fri Jul 19, 2019 1:50 pm

However, even after you have applied the steps you used, the speed is still around 230Mbps.
These are pretty good numbers for IPsec single client / TCP, I've seen similar performance on RB750Gr3, which is pretty close to hAP ac2 in IPsec perf, in my quick tests.
You do not have the required permissions to view the files attached to this post.
 
msatter
Forum Guru
Forum Guru
Posts: 1285
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: IPSEC performance problem

Fri Jul 19, 2019 5:18 pm

I tried something else. Currently using an SFP for the WAN and that has in the RB760iGS (same processor as the RB750GR3). I connect it up differently replacing the SFP up-link with a ethernet port connected to a media converter. Now the usage of the processor is bit better and the firewall/network load is also more shifted to the other cores. I now read speeds over 240Mbit/s and before it was around 180Mbit/s.
IKEVETH.JPG
Blockdiagram of the RB760iGS where can see that inserting a SFP split the bandwith and it would been better if it would be eth1 or SFP and not both usable at the same time.
Image

Update:
Tested it with only NordVPN IKEv2 connection and I get with them 180/200Mbit/s (down/up).
You do not have the required permissions to view the files attached to this post.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)

Who is online

Users browsing this forum: No registered users and 123 guests