Page 1 of 1

hEX - IPsec Tunnel slow

Posted: Fri Feb 09, 2018 9:01 pm
by nicku
Hi,

I just bought two hEX Routers (750G r3, 6.41.2) and set up an IPsec tunnel between them. Both are running in Bridge Mode behind another DSL Router. I only want to use them as a VPN Gateway. Problem is that I only get about 16 MBit/s throughput. Internet connection should allow 40 MBit/s. I disabled the firewall by deleting all rules. How can I improve the IPsec tunnel speed?

My configuration:
/ip ipsec peer print
 0     ;;; vpn10
       address=88.77.66.44/32 auth-method=pre-shared-key
       secret="xxx" generate-policy=no
       policy-template-group=default exchange-mode=ike2
       send-initial-contact=yes hash-algorithm=sha256 enc-algorithm=aes-256
       dh-group=modp4096 lifetime=1d dpd-interval=2m

/ip ipsec policy print
 1  A  ;;; vpn10
       src-address=192.168.20.0/24 src-port=any dst-address=192.168.10.0/24
       dst-port=any protocol=all action=encrypt level=require
       ipsec-protocols=esp tunnel=yes sa-src-address=0.0.0.0
       sa-dst-address=88.77.66.44 proposal=secure-proposal ph2-count=1
       
/ip ipsec proposal print
1    name="secure-proposal" auth-algorithms=sha256 enc-algorithms=aes-256-cbc
      lifetime=30m pfs-group=modp4096
Thank you,
Niclas

Re: hEX - IPsec Tunnel slow

Posted: Sat Feb 10, 2018 7:49 am
by CZFan
Hi,

I just bought two hEX Routers (750G r3, 6.41.2) and set up an IPsec tunnel between them. Both are running in Bridge Mode behind another DSL Router. I only want to use them as a VPN Gateway. Problem is that I only get about 16 MBit/s throughput. Internet connection should allow 40 MBit/s. I disabled the firewall by deleting all rules. How can I improve the IPsec tunnel speed?

My configuration:
/ip ipsec peer print
 0     ;;; vpn10
       address=88.77.66.44/32 auth-method=pre-shared-key
       secret="xxx" generate-policy=no
       policy-template-group=default exchange-mode=ike2
       send-initial-contact=yes hash-algorithm=sha256 enc-algorithm=aes-256
       dh-group=modp4096 lifetime=1d dpd-interval=2m

/ip ipsec policy print
 1  A  ;;; vpn10
       src-address=192.168.20.0/24 src-port=any dst-address=192.168.10.0/24
       dst-port=any protocol=all action=encrypt level=require
       ipsec-protocols=esp tunnel=yes sa-src-address=0.0.0.0
       sa-dst-address=88.77.66.44 proposal=secure-proposal ph2-count=1
       
/ip ipsec proposal print
1    name="secure-proposal" auth-algorithms=sha256 enc-algorithms=aes-256-cbc
      lifetime=30m pfs-group=modp4096
Thank you,
Niclas
If the data sent through VPN is not sensitive data, maybe use PPTP VPN, much lighter and faster

Re: hEX - IPsec Tunnel slow

Posted: Sat Feb 10, 2018 11:58 am
by nicku
The data is more or less sensitive, but I want this strong encryption. Shouldn't the hEX routers be much more powerful at IPsec? Two cores of the CPU are just at about 8%.

Re: hEX - IPsec Tunnel slow

Posted: Sat Feb 10, 2018 12:29 pm
by CZFan
Yes, I would think you should get ~50Mb.

Do you get 16Mb both ways? Not maybe limited by one sides upload speed?

Re: hEX - IPsec Tunnel slow

Posted: Sat Feb 10, 2018 1:15 pm
by nicku
Site 1 has 50/10 MBit/s (Down/up) and site 2 has 100/50 MBit/s. For testing I was at site 1 and downloaded (https) a file from site 2. So there is a bandwidth limit of 50 MBit/s but I only got 16 MBit/s through the IPsec tunnel.

Some ideas: Would it be better to use IPv6 between the sites? Should I play with MTU?

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 10:30 am
by andlil
Site 1 has 50/10 MBit/s (Down/up) and site 2 has 100/50 MBit/s.
If those are the correct numbers I am surprised you get 16 MBit/s, since upload speed on site 1 is 10 MBit/s...

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 11:45 am
by msatter
Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 11:50 am
by nicku
If those are the correct numbers I am surprised you get 16 MBit/s, since upload speed on site 1 is 10 MBit/s...
I thought I would get 50 MBit/s bandwidth limit in one direction and 10 MBit/s in the other direction. Can you tell me why I am wrong? I think you have misunderstood my setup.

Edit:
Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.
I will test it.

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 12:38 pm
by CZFan
Site 1 has 50/10 MBit/s (Down/up) and site 2 has 100/50 MBit/s. For testing I was at site 1 and downloaded (https) a file from site 2. So there is a bandwidth limit of 50 MBit/s but I only got 16 MBit/s through the IPsec tunnel.

Some ideas: Would it be better to use IPv6 between the sites? Should I play with MTU?
Not sure how much knowledge / experience you have, but I would do some packet sniffing to see what is happening on the network

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 12:48 pm
by nicku
Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive
I downgraded both routers to 6.40.5, but the bridge option still exists and speed hadn't changed. Is this what you expected? I thought pre-Bridge means that this option is not available.

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 1:22 pm
by nicku
Not sure how much knowledge / experience you have, but I would do some packet sniffing to see what is happening on the network
Which packets shall I sniff and where? How can I see if somethings wrong?

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 3:30 pm
by CZFan
Mikrotik has a packet sniffer in Tools menu, sniff the bridge in both directions for IP of your PC while you doing a download.Save the sniffed packet file

You can then view the details with open source product called Wireshark, but you will need in depth knowledge of how protocols, i.e. IP, TCP, UDP, etc work.

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 3:59 pm
by msatter
Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive
I downgraded both routers to 6.40.5, but the bridge option still exists and speed hadn't changed. Is this what you expected? I thought pre-Bridge means that this option is not available.
These are bridges between the ports on the Mikrotik itself and before 6.41 Master-Slave mode was used:

viewtopic.php?f=21&t=128915

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 4:10 pm
by CZFan
Try it with a pre-Bridge firmware and that is 6.40.5. I had the strangest things happening with the 750Gr3 on the Bridged firmware version and RC.

https://mikrotik.com/download/archive
I downgraded both routers to 6.40.5, but the bridge option still exists and speed hadn't changed. Is this what you expected? I thought pre-Bridge means that this option is not available.
These are bridges between the ports on the Mikrotik itself and before 6.41 Master-Slave mode was used:

viewtopic.php?f=21&t=128915
I suspect this will make no difference, reason being is that bride goes via cpu, master / slave config did not. but when routing is involved, it has to go via cpu, so will make no difference here

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 11, 2018 7:42 pm
by nicku
I looked at the sniffed traffic while downloading a file over https:
  1. There are many many retransmissions (TCP Fast Retransmissions, TCP Retransmissions and TCP Dup ACK)
  2. ESP packet size is 1506
  3. IP packet size (my download) 1436
I never saw IPsec traffic and I don't know what I have to expect, but these restransmissions seem to be problematic.

Re: hEX - IPsec Tunnel slow

Posted: Sat Feb 17, 2018 7:49 pm
by nicku
Hi,
today I had time to build a small test network like this:
Laptop -- hEX -- FritzBox (Ports forwarded) -- Switch (my "test internet") -- FritzBox -- hEX -- Laptop
I tested some different configurations but end up by only 3 MByte/s through the IPsec tunnel (SMB data transfer).

Then I tried another network config:
Laptop -- hEX -- Switch (my "test internet") -- hEX -- Laptop
Now I got about 11 MByte/s through the IPsec tunnel (SMB data transfer). I think this is the maximum network speed as the switch is only able to handle 100MBit/s.

So the problem seems to be the NAT at the FritzBox. Does someone why these boxes are slowing me down?

EDIT: Just tested the second network config with a gigabit switch and got 28 MByte/s. Really impressive.

Re: hEX - IPsec Tunnel slow

Posted: Sun Feb 18, 2018 10:52 am
by CZFan
100Mbits/s equates to approx. 12,5 MBytes/s, with overhead, etc 11 MBytes/s is about correct.

Move the PPPoE / DSL authentication, etc off the FritzBox to the Mikrotik and configure the FritzBox as a "modem" only

Re: hEX - IPsec Tunnel slow

Posted: Tue Jun 12, 2018 5:34 pm
by nicku
Sorry for this late reply. Had no time to test with VDSL modems. Now my setup looks like this:
Local Network 1 -- Mikrotik  -- Modem (100/40 Down/Up) --  Internet -- Modem (50/10 Down/Up) -- Mikrotik -- Local Network 2
The Mikrotik Routers are connected via a IPSec GRE Tunnel. If I ping from one Local Network to the other I have a Latency of about 40ms.
Now the problem: I only get 2,3 MByte/s via SMB, but the full speed of about 4,3 MByte/s via FTP. Is there anything I can do to improve the SMB speed? Some Clients need SMB so there is no option to use FTP only. I already set the SMB Version to 3.1.

Tunnel Configuration:
Actual MTU: 1406
DSCP: inherit
Dont fragment: no
Clamp TCP MSS: yes
Allow fast path: no

Re: hEX - IPsec Tunnel slow

Posted: Wed Jun 13, 2018 4:08 am
by Van9018
Nothing more to do on the router if FTP maxes out your connection.

SMB is a chatty protocol, latency is a killer. You'll have to look more into SMB to see if it can be tuned for better throughput on high latency networks.

Re: hEX - IPsec Tunnel slow

Posted: Wed Jun 13, 2018 7:43 am
by nichky
@nicku How your firewall looks like? use RAW instead of NAT.

Re: hEX - IPsec Tunnel slow

Posted: Fri Jun 15, 2018 12:06 am
by nicku
@nichky: my Firewall settings are the defaults:
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="Admin page" dst-port=80 protocol=tcp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
Is there anything I can do better?

@Van9018: I tried playing with the MTU and I set DisableBandwidthThrottling on the windows client, which at least stabilizes the speed to 20 Mbps. With FTP I get 39 Mbps. Do you have any suggestions for me on how tu tune that SMB performance?

Re: hEX - IPsec Tunnel slow

Posted: Fri Jun 15, 2018 1:05 am
by nichky
So you playing with tunnel mode and you didn't set up NAT or RAW rule to accept your LAN and LAN from other side. Can you ping other side?

Re: hEX - IPsec Tunnel slow

Posted: Fri Jun 15, 2018 1:11 am
by nicku
I changed my initial vpn configuration to a GRE Tunnel (Interface) with IPSec secret and set up a IP Route, like I wrote some posts ago. And yes ping and SMB is working. Just too slow.

Re: hEX - IPsec Tunnel slow

Posted: Fri Jun 15, 2018 1:20 am
by nichky
Can you quickly do only tunnel without IPSec and do test. And how you checking your result by B-test, watchout B-test is CPU extensive.

Re: hEX - IPsec Tunnel slow

Posted: Fri Jun 15, 2018 1:38 am
by nicku
Without IPSec I get slightly better performance, but it is just a plus of about 1 Mbps at peak. For testing I copy a file from a SMB share of a QNAP NAS to my computer.

Concerning WAN limits: I have a download of 50 MBit/s and on the other site I have a upload of 40 MBit/s. So the FTP speed I measured of 39 MBit/s is the bandwidth limit.

Re: hEX - IPsec Tunnel slow

Posted: Sun Nov 25, 2018 12:07 pm
by Illuru
I have axact the same situation.
Any suggestions for the solution? The sniiffer collects a lot of TCP Retransmission.
Thanks in advanced

Re: hEX - IPsec Tunnel slow

Posted: Sun Nov 25, 2018 2:50 pm
by mkx
I have axact the same situation.
Any suggestions for the solution? The sniiffer collects a lot of TCP Retransmission.
Thanks in advanced
Do you see retransmissions on VPN interface or on underlying interface?

Re: hEX - IPsec Tunnel slow

Posted: Mon Mar 25, 2019 4:40 am
by otgooneo
I just had a chance to configure two RB750Gr3 with ipsec. No firewall at all on both routers, uplinks allow 100mbps between routers. IPSec uses AES128 for encryption algorithm and SHA1 for authentication algorithm. ~10Mbps ipsec traffic, CPU gets ~38%. One of the CPU reaches to 60% among 4 CPUs. Estimating IPsec max throughput will be not more than 30Mbps in this setup. That is quite slow performance considering 880MHz x 2 CPUs and plus hardware offloading enabled. Even RB450G (single core, no hardware offload on ipsec) can reach to similar performance. On top of slow throughput, RB750Gr3 adds notable delays on IPSec traffic even in case of very tiny traffic usage. In some cases, I see ICMP loss on ipsec packets while there is no packet loss between WAN addresses of two routers.
Quite upset. Seems like this is the capability of RB750gr3.

Forgot to mention that brigde settings enable between ether2-ether5. But this should be offloaded to switch chip. So I don`t think this affects to overall performance that much.

Re: hEX - IPsec Tunnel slow

Posted: Tue Mar 26, 2019 11:35 pm
by voxmaster
Same here, after upgrade from version 6.42.7

Re: hEX - IPsec Tunnel slow

Posted: Wed Mar 27, 2019 12:30 am
by anav
So the question remains, is this a software bug or a hardware limitation?

Re: hEX - IPsec Tunnel slow

Posted: Wed Mar 27, 2019 9:21 am
by voxmaster
Sorry. My mistake. No problems with IPsec over L2TP. It was ISP problem in my case :roll: