CHR <-> RB750Gr3 via GRE over IPSEC Performance issue

RouterOS Virtualization
1

Hi All,

So I have a virtualized CHR running in a datacentre which has a GRE Tunnel running over IPSEC to my home router, which is a hEX v3 (RB750Gr3).

For some reason when using an IPSEC tunnel, I only seem to be able to achieve around 20Mbps with a bandwidth test from the CHR to RB750Gr3. However if I perform the bandwidth test directly from the CHR to the public IP of the RB750Gr3 (Directly over the internet instead of IPSEC tunnel), I am able to achieve maximum throughput.

The resources allocated to the CHR are:

CPU: 4 Cores of a D-1531 Xeon processor,
Memory: 1GB,
Network: VirtIO Adapters

When running a bandwidth test over the IPSEC tunnel, the CPU of the CHR sits at 25% with one core maxed out at 100%. The CPU of the hEX v3 sits happily around 10%.

I’m using the following IPSEC settings:

Auth Algorithm: sha1
Encr. Algorithm: aes-256-cbc
PFS group: modp1024


Does anyone have any idea how I could improve performance over IPSEC? I realize the hEX has hardware acceleration, but shouldn’t I be achieving more than a measly 20Mbps over an IPSEC tunnel between these routers?

2

Do you have a server with a single CPU core capable of more than 2.2GHz? Your post clearly stated the performance is capping out 1 core of the CHR. That is your limitation.

http://forum.mikrotik.com/t/ipsec-hardware-acceleration-on-chr/109905/1

Might be an interesting read, first things first. What version is your CHR? Make sure it is at least up to 6.39 and ensure your hardware and hypervisor is allowing the AES extensions through to the CHR VM.

3

I’m using 6.40.1 and I’ve ensured my hardware and hypervisor (Proxmox) allowed AES extensions.

The processor I’m using is the D-1531 with a base clock of 2.20GHz and 2.70GHz turbo.

I’m still stumped; I’ve tried emulating a KVM64 CPU type as well as the ‘host’ CPU type. Neither makes any difference to the throughput.

4

Try aes-128-cbc or even no encryption at all (AH protocol).
It is not useful to assign 4 processors to CHR - it will not use them for parallel processing for tasks like this.

5

Thanks for the suggestion.

I’ve just tried using AH Protocol and it hasn’t made any difference I’m afraid. I’m stumpted.

Any more ideas?

6

Try setting the CPU to host or host-passthrough to be 100% certain AES extensions are getting through and enabled.

Disregard: Just read you tried host mode already.

7

It must be another problem then. I am using AH protocol between a 2011 and CCR and I can saturate the link.
(which I cannot do using ESP because of the slow CPU in the 2011)
With your 750Gr3 you have accellerated AES and you could do ESP without problem, but apparently for you the CHR side is the bottleneck.
Put a 750Gr3 there too :slight_smile:

8

I’m running CHR on Intel Haswell, without TSX, to support high availability failover to Intel Xeon CPU E5-2640v3. I’ve confirmed AES pass through by booting the CHR guest using CentOS 7 recovery environment.

Confirming ‘aes’ instruction availability:
grep -m1 -o aes /proc/cpuinfo

We obtain the following benchmarks in the VM:
openssl speed -evp aes-128-cbc

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes                     
aes-128-cbc     511101.85k   547731.20k   555776.60k   560752.67k   558724.44k

openssl speed -evp aes-256-cbc

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes                     
aes-256-cbc     371521.68k   394245.03k   401446.36k   399955.91k   402183.22k

Directly on slowest hardware:
openssl speed -evp aes-128-cbc

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes                     
aes-128-cbc     541865.83k   585278.50k   595671.30k   602248.53k   603339.43k

openssl speed -evp aes-256-cbc

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes                     
aes-256-cbc     398849.01k   423887.98k   430012.84k   431852.89k   432622.25k

That equates to 5.2 Gbps, when using AES 128 bit CBC encoding within the virtual guest. I don’t see L2TP IPSec in CHR reporting ‘Hardware AEAD’ when reviewing the installed SAs either…

9

What is in /tool profile ?
Do you happen to run btest from the CHR itself?

10

Sorry, I haven’t really had time to look at this lately.

Today I’ve completely rebuilt the router with the 6.41 build but the same issue occurs.

Profile:

BTest to itself (CHR):

Any ideas?

11

A thought: it does not appear you accounting for various MTU overheads anywhere, so it may be an issue with path maximum transmission unit (MTU) discovery (PMTUD), or TCP maximum segmentation size (TCP MSS). Things to remember:

  • Traditional Ethernet MTU is 1500, so most things use this by default
  • TCP MSS = MTU -40
  • IPSec has an 8 bit overhead: MTU -8
  • If you or your ISP is using PPPoE: MTU -8
  • PMTUD is generally unreliable.