I'm testing out CHR as a VPN Concentrator for up to 500 concurrent connections with maybe 50 actively using it at any one time. Up until now i've only used actual mikrotik hardware, so its possible I don't know how to optimize the setup but I'm hitting hard caps and want to know if anyone has any experience with this.
I have two machines for the test:
Machine 1 (M1):
AMD Ryzen 5900x with HT off
x570 Board
32gb Ram
Mellanox Connectx-3 sfp+ card
ESXi
VMXNET 3
1 CHR P-10 Instance with 11 cores & 4gb ram
"Public" IP: 192.168.100.10
"Private" IP: 192.168.110.1
no nat/firewall rules
Machine 2 (M2):
AMD Ryzen 2950x with HT off
x399 Board
32gb Ram
Mellanox Connectx-3 sfp+ card
ESXi
VMXNET 3
3 CHR P-10 Instances
"Public" IP: 192.168.100.20
"Private" IP: 192.168.120.1
no nat/firewall rules
Both are connected via DAC to a CRS309.
Running a bandwidth test from the public address of M2 (1 instance) to M1 (1 instance) I get a stable 9.5+Gbps in both tcp tests and udp tests.
Running a bandwidth test from the private address of M2 (1 instance) to M1 (1 instance) through an ike2 ipsec tunnel with sha1 AES-GCM 128 I get a stable ~2.5gbps tcp.
Running a bandwidth test from the private address of M2 (3 instance) to M1 (1 instance) through an ike2 ipsec tunnel with sha1 AES-GCM 128 I get wildly fluctuating ~6gbps tcp, with about 50% CPU utilization.
Running a bandwidth test from the private address of M2 (1 instance) to M1 (1 instance) through a wireguard tunnel I get a stable ~2.5gbps tcp.
Running a bandwidth test from the private address of M2 (3 instance) to M1 (1 instance) through a wireguard tunnel I get wildly fluctuating ~5gbps tcp with about 25% CPU utilization.
I can't seem to get full line speed, and I don't know what the bottleneck is.