MikroTik

Hi,

I just bought two hEX Routers (750G r3, 6.41.2) and set up an IPsec tunnel between them. Both are running in Bridge Mode behind another DSL Router. I only want to use them as a VPN Gateway. Problem is that I only get about 16 MBit/s throughput. Internet connection should allow 40 MBit/s. I disabled the firewall by deleting all rules. How can I improve the IPsec tunnel speed?

My configuration: Thank you,
Niclas

Hi,

I just bought two hEX Routers (750G r3, 6.41.2) and set up an IPsec tunnel between them. Both are running in Bridge Mode behind another DSL Router. I only want to use them as a VPN Gateway. Problem is that I only get about 16 MBit/s throughput. Internet connection should allow 40 MBit/s. I disabled the firewall by deleting all rules. How can I improve the IPsec tunnel speed?

My configuration:

Code: Select all

/ip ipsec peer print
 0     ;;; vpn10
       address=88.77.66.44/32 auth-method=pre-shared-key
       secret="xxx" generate-policy=no
       policy-template-group=default exchange-mode=ike2
       send-initial-contact=yes hash-algorithm=sha256 enc-algorithm=aes-256
       dh-group=modp4096 lifetime=1d dpd-interval=2m

/ip ipsec policy print
 1  A  ;;; vpn10
       src-address=192.168.20.0/24 src-port=any dst-address=192.168.10.0/24
       dst-port=any protocol=all action=encrypt level=require
       ipsec-protocols=esp tunnel=yes sa-src-address=0.0.0.0
       sa-dst-address=88.77.66.44 proposal=secure-proposal ph2-count=1
       
/ip ipsec proposal print
1    name="secure-proposal" auth-algorithms=sha256 enc-algorithms=aes-256-cbc
      lifetime=30m pfs-group=modp4096

Thank you,
Niclas

If the data sent through VPN is not sensitive data, maybe use PPTP VPN, much lighter and faster

The data is more or less sensitive, but I want this strong encryption. Shouldn't the hEX routers be much more powerful at IPsec? Two cores of the CPU are just at about 8%.

Yes, I would think you should get ~50Mb.

Do you get 16Mb both ways? Not maybe limited by one sides upload speed?

Site 1 has 50/10 MBit/s (Down/up) and site 2 has 100/50 MBit/s. For testing I was at site 1 and downloaded (https) a file from site 2. So there is a bandwidth limit of 50 MBit/s but I only got 16 MBit/s through the IPsec tunnel.

Some ideas: Would it be better to use IPv6 between the sites? Should I play with MTU?

Site 1 has 50/10 MBit/s (Down/up) and site 2 has 100/50 MBit/s.

If those are the correct numbers I am surprised you get 16 MBit/s, since upload speed on site 1 is 10 MBit/s...

Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive

If those are the correct numbers I am surprised you get 16 MBit/s, since upload speed on site 1 is 10 MBit/s...

I thought I would get 50 MBit/s bandwidth limit in one direction and 10 MBit/s in the other direction. Can you tell me why I am wrong? I think you have misunderstood my setup.

Edit:

Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

I will test it.

Site 1 has 50/10 MBit/s (Down/up) and site 2 has 100/50 MBit/s. For testing I was at site 1 and downloaded (https) a file from site 2. So there is a bandwidth limit of 50 MBit/s but I only got 16 MBit/s through the IPsec tunnel.

Some ideas: Would it be better to use IPv6 between the sites? Should I play with MTU?

Not sure how much knowledge / experience you have, but I would do some packet sniffing to see what is happening on the network

Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive

I downgraded both routers to 6.40.5, but the bridge option still exists and speed hadn't changed. Is this what you expected? I thought pre-Bridge means that this option is not available.

Not sure how much knowledge / experience you have, but I would do some packet sniffing to see what is happening on the network

Which packets shall I sniff and where? How can I see if somethings wrong?

Mikrotik has a packet sniffer in Tools menu, sniff the bridge in both directions for IP of your PC while you doing a download.Save the sniffed packet file

You can then view the details with open source product called Wireshark, but you will need in depth knowledge of how protocols, i.e. IP, TCP, UDP, etc work.

Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive

I downgraded both routers to 6.40.5, but the bridge option still exists and speed hadn't changed. Is this what you expected? I thought pre-Bridge means that this option is not available.

These are bridges between the ports on the Mikrotik itself and before 6.41 Master-Slave mode was used:

viewtopic.php?f=21&t=128915

Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive

I downgraded both routers to 6.40.5, but the bridge option still exists and speed hadn't changed. Is this what you expected? I thought pre-Bridge means that this option is not available.

These are bridges between the ports on the Mikrotik itself and before 6.41 Master-Slave mode was used:

viewtopic.php?f=21&t=128915

I suspect this will make no difference, reason being is that bride goes via cpu, master / slave config did not. but when routing is involved, it has to go via cpu, so will make no difference here

I looked at the sniffed traffic while downloading a file over https:

1. There are many many retransmissions (TCP Fast Retransmissions, TCP Retransmissions and TCP Dup ACK)
2. ESP packet size is 1506
3. IP packet size (my download) 1436

I never saw IPsec traffic and I don't know what I have to expect, but these restransmissions seem to be problematic.

Hi,
today I had time to build a small test network like this: I tested some different configurations but end up by only 3 MByte/s through the IPsec tunnel (SMB data transfer).

Then I tried another network config: Now I got about 11 MByte/s through the IPsec tunnel (SMB data transfer). I think this is the maximum network speed as the switch is only able to handle 100MBit/s.

So the problem seems to be the NAT at the FritzBox. Does someone why these boxes are slowing me down?

EDIT: Just tested the second network config with a gigabit switch and got 28 MByte/s. Really impressive.

100Mbits/s equates to approx. 12,5 MBytes/s, with overhead, etc 11 MBytes/s is about correct.

Move the PPPoE / DSL authentication, etc off the FritzBox to the Mikrotik and configure the FritzBox as a "modem" only

Sorry for this late reply. Had no time to test with VDSL modems. Now my setup looks like this:
The Mikrotik Routers are connected via a IPSec GRE Tunnel. If I ping from one Local Network to the other I have a Latency of about 40ms.
Now the problem: I only get 2,3 MByte/s via SMB, but the full speed of about 4,3 MByte/s via FTP. Is there anything I can do to improve the SMB speed? Some Clients need SMB so there is no option to use FTP only. I already set the SMB Version to 3.1.

Tunnel Configuration:
Actual MTU: 1406
DSCP: inherit
Dont fragment: no
Clamp TCP MSS: yes
Allow fast path: no

Nothing more to do on the router if FTP maxes out your connection.

SMB is a chatty protocol, latency is a killer. You'll have to look more into SMB to see if it can be tuned for better throughput on high latency networks.

@nicku How your firewall looks like? use RAW instead of NAT.

@nichky: my Firewall settings are the defaults: Is there anything I can do better?

@Van9018: I tried playing with the MTU and I set DisableBandwidthThrottling on the windows client, which at least stabilizes the speed to 20 Mbps. With FTP I get 39 Mbps. Do you have any suggestions for me on how tu tune that SMB performance?

So you playing with tunnel mode and you didn't set up NAT or RAW rule to accept your LAN and LAN from other side. Can you ping other side?

I changed my initial vpn configuration to a GRE Tunnel (Interface) with IPSec secret and set up a IP Route, like I wrote some posts ago. And yes ping and SMB is working. Just too slow.

Can you quickly do only tunnel without IPSec and do test. And how you checking your result by B-test, watchout B-test is CPU extensive.

Without IPSec I get slightly better performance, but it is just a plus of about 1 Mbps at peak. For testing I copy a file from a SMB share of a QNAP NAS to my computer.

Concerning WAN limits: I have a download of 50 MBit/s and on the other site I have a upload of 40 MBit/s. So the FTP speed I measured of 39 MBit/s is the bandwidth limit.

I have axact the same situation.
Any suggestions for the solution? The sniiffer collects a lot of TCP Retransmission.
Thanks in advanced

I have axact the same situation.
Any suggestions for the solution? The sniiffer collects a lot of TCP Retransmission.
Thanks in advanced

Do you see retransmissions on VPN interface or on underlying interface?

I just had a chance to configure two RB750Gr3 with ipsec. No firewall at all on both routers, uplinks allow 100mbps between routers. IPSec uses AES128 for encryption algorithm and SHA1 for authentication algorithm. ~10Mbps ipsec traffic, CPU gets ~38%. One of the CPU reaches to 60% among 4 CPUs. Estimating IPsec max throughput will be not more than 30Mbps in this setup. That is quite slow performance considering 880MHz x 2 CPUs and plus hardware offloading enabled. Even RB450G (single core, no hardware offload on ipsec) can reach to similar performance. On top of slow throughput, RB750Gr3 adds notable delays on IPSec traffic even in case of very tiny traffic usage. In some cases, I see ICMP loss on ipsec packets while there is no packet loss between WAN addresses of two routers.
Quite upset. Seems like this is the capability of RB750gr3.

Forgot to mention that brigde settings enable between ether2-ether5. But this should be offloaded to switch chip. So I don`t think this affects to overall performance that much.

Same here, after upgrade from version 6.42.7

So the question remains, is this a software bug or a hardware limitation?

Sorry. My mistake. No problems with IPsec over L2TP. It was ISP problem in my case

Hello!

First post, so my greetings to All!

I have a noob question about hardware encryption in hEX S;
My VPN Provider (ExpressVPN) supports AES-256-CBC, and I see that hEX S support this kind of hardware encryption.

But the problem might be in RSA certificate identification by the hashing algorithm SHA-512 when hEX S supports only SHA-256.

My question is will it be anyhow hardware encrypted if I select in IPSEC settings SHA512 and if someone has those combination (hEX S and ExpressVPN), which VPN performance could I expect and is it hardware encryption possible with ExpressVPN?

Is there someone solved ipsec issue of hEX? I have no change to test it with latest RouterOS.

I'm still seeing similar performance limitations on 6.47.1 with SMB over an L2TP IPsec tunnel via Windows clients, so the tunnel is SHA-1 & AES-128 which are fully supported by the Hex's Mediatek offloading / fasttrack. Disabling fasttrack from the firewall rules has no effect on the performance levels, so this is not a fasttrack bug.

With these Mikrotiks, web traffic will scale up to 71-75Mb over the tunnel (90Mb fiber), if I cfg the tunnel to allow use of the remote gateway and test via Speedtest.net. SMB bottlenecks at 10-11 Mb when tested with Totusoft's Lan Speed Test. I did some prior testing with a lower end RB951G and it scaled up to around 15Mb, which has no fasttrack involved as it is a simpler Atheros platform with no IPsec offloading support. So this issue does appear to be more of a problem on the Mediatek vs Atheros platform.

This is definitely strange behavior, as the Ubiquiti Edgerouter X's I am upgrading from bottleneck both protocols at the same ~32Mb due to their broken (for years) L2TP IPsec offloading. The Edgerouter Lite (Cavium, with functional L2TP IPSec offloading) scales up to around 40Mb for web traffic and bottlenecks at a similar 30-35Mb for SMB using identical VPN cfgs on 600/100Mb connections from the same cable ISP.

All of my testing is to SMB test targets on up to date Synology NAS units, and with optimized MTU's of 1400 specified in the Ubiquiti & Mikrotik L2TP server cfgs.

I've also confirmed identical WAN MTU's of 1500 for the different ISP's involved, 600/100 on the Edgerouter X & Lite being cable and the 90/90 being fiber.

In case it wasn't clear in those two prior posts, all interfaces involved have optimized / confirmed non-fragmenting MTUs for both the L2TP VPN tunnels and WAN interfaces.