RB750Gr3 l2tp/ipsec unbearably slow

RouterOS General
1

At home, I have a CCR1009 on an 1000/50 mbps (down/up) connection. L2TP/IPsec through my VPN provider gets me 200+ mbps down, and 10-20 mbps up. The upload could be faster, but it’s usable.

At my 2nd house, I have an RB750Gr3 on a 60/5 mbps connection. The same L2TP/IPsec through the same VPN provider gets me ~20 mbps down and ~10 kbps up..
Here are some iperf3 results without the tunnel enabled:

chiem@rpi:~ $ iperf3 -c remote 
Connecting to host remote, port 5201
[  4] local 192.168.125.2 port 47800 connected to #.#.#.# port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  28.5 KBytes   233 Kbits/sec    0   14.3 KBytes       
[  4]   1.00-2.00   sec  28.5 KBytes   234 Kbits/sec    0   15.7 KBytes       
[  4]   2.00-3.00   sec  42.8 KBytes   350 Kbits/sec    0   20.0 KBytes       
[  4]   3.00-4.00   sec  32.8 KBytes   269 Kbits/sec    0   27.1 KBytes       
[  4]   4.00-5.00   sec   154 KBytes  1.26 Mbits/sec    0   49.9 KBytes       
[  4]   5.00-6.00   sec   228 KBytes  1.87 Mbits/sec    0   78.4 KBytes       
[  4]   6.00-7.00   sec   261 KBytes  2.14 Mbits/sec    0    121 KBytes       
[  4]   7.00-8.00   sec   329 KBytes  2.70 Mbits/sec    0    182 KBytes       
[  4]   8.00-9.00   sec   425 KBytes  3.48 Mbits/sec    0    247 KBytes       
[  4]   9.00-10.00  sec   800 KBytes  6.56 Mbits/sec    0    374 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.28 MBytes  1.91 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.85 MBytes  1.55 Mbits/sec                  receiver

iperf Done.
chiem@rpi:~ $ iperf3 -c remote -R
Connecting to host remote, port 5201
Reverse mode, remote host remote is sending
[  4] local 192.168.125.2 port 47804 connected to #.#.#.# port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   587 KBytes  4.81 Mbits/sec                  
[  4]   1.00-2.00   sec  1.88 MBytes  15.7 Mbits/sec                  
[  4]   2.00-3.00   sec  4.56 MBytes  38.3 Mbits/sec                  
[  4]   3.00-4.00   sec  2.67 MBytes  22.4 Mbits/sec                  
[  4]   4.00-5.00   sec  3.57 MBytes  29.9 Mbits/sec                  
[  4]   5.00-6.00   sec  3.48 MBytes  29.2 Mbits/sec                  
[  4]   6.00-7.00   sec  3.63 MBytes  30.4 Mbits/sec                  
[  4]   7.00-8.00   sec  2.02 MBytes  16.9 Mbits/sec                  
[  4]   8.00-9.00   sec  2.52 MBytes  21.1 Mbits/sec                  
[  4]   9.00-10.00  sec  1.65 MBytes  13.8 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  33.6 MBytes  28.2 Mbits/sec  648             sender
[  4]   0.00-10.00  sec  28.8 MBytes  24.2 Mbits/sec                  receiver

iperf Done.

And now with the tunnel enabled:

chiem@rpi:~ $ iperf3 -c remote 
Connecting to host remote, port 5201
[  4] local 192.168.125.2 port 47808 connected to #.#.#.# port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  26.8 KBytes   219 Kbits/sec    0   13.4 KBytes       
[  4]   1.00-2.00   sec  1.34 KBytes  11.0 Kbits/sec    1   2.68 KBytes       
[  4]   2.00-3.00   sec  1.34 KBytes  11.0 Kbits/sec    3   2.68 KBytes       
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   2.68 KBytes       
[  4]   4.00-5.00   sec  1.34 KBytes  11.0 Kbits/sec    3   2.68 KBytes       
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   2.68 KBytes       
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   2.68 KBytes       
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   2.68 KBytes       
[  4]   8.00-9.00   sec  1.34 KBytes  11.0 Kbits/sec    3   2.68 KBytes       
[  4]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   2.68 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  32.1 KBytes  26.3 Kbits/sec   10             sender
[  4]   0.00-10.00  sec  5.35 KBytes  4.38 Kbits/sec                  receiver

iperf Done.
chiem@rpi:~ $ iperf3 -c remote -R
Connecting to host remote, port 5201
Reverse mode, remote host remote is sending
[  4] local 192.168.125.2 port 47812 connected to #.#.#.# port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   409 KBytes  3.35 Mbits/sec                  
[  4]   1.00-2.00   sec  1.60 MBytes  13.4 Mbits/sec                  
[  4]   2.00-3.00   sec  2.81 MBytes  23.6 Mbits/sec                  
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  4]   4.00-5.00   sec  4.08 MBytes  34.2 Mbits/sec                  
[  4]   5.00-6.00   sec  2.36 MBytes  19.8 Mbits/sec                  
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  4]   7.00-8.00   sec  2.97 MBytes  24.9 Mbits/sec                  
[  4]   8.00-9.00   sec  1.50 MBytes  12.5 Mbits/sec                  
[  4]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  24.9 MBytes  20.9 Mbits/sec   25             sender
[  4]   0.00-10.00  sec  17.2 MBytes  14.4 Mbits/sec                  receiver

iperf Done.

Except for the L2TP/IPsec, the RB750Gr3 is running pretty much an out of the box quick set config for NATing. I’ve tried disabling the IPsec for an unencrypted L2TP, but the results are the same.
I’ve also tried PPTP, and the results are still the same. Those are the only connections my VPN provider supports, other than OpenVPN (which I can’t use until the fabled v7 is released). This would seem like a problem with the VPN provider, except that it works fine on my CCR1009. Any idea why the tunneled upload is so damn slow on the RB750Gr3 ?

2

Hi

Could it be packet size / MTU and/or fragmentation related?
Please try with smaller transmission unit.

3

I’ve tried all the way down to 600.

4

bump

5

Bump to this posting and check if fasttracking is set on the ipsec.

http://forum.mikrotik.com/t/radius-server-not-working-in-2-8-11/127/1

6

with/without fastpath, the RB750Gr3 hardware should be able to do way more than this.
Check for full/half duplex mismatch settings and errors like collisions on 100Mb/s Ethernet interfaces

7

Did you resolve the problem with the 750Gr3?

8

I was the one that posted the linked comment. Fasttrack bypass definitely helped me, though I’m still seeing CPU pegged and absolutely never get any higher than 100Mbps. While this is pretty good, this is on a 350Mbps connection, and a similar connection from a laptop will get up to 320Mbps over the L2TP/IPsec connection. After double checking that I’m using hardware supported encryption (AES-128-CBC/SHA-1), I’m guessing it just doesn’t get any faster than that.

Ultimately though L2TP/IPsec using a third party provider isn’t particularly secure, as the preshared key is public knowledge (at least in my case). So I’m just going with a direct connection over OpenVPN.

TL;DR Make sure your bypassing fasttrack and fastpath, and don’t expect anywhere higher than 100Mbps at an absolute best, usually It’ll stick around 50-60 for me.

9

You need the pre-shares public key (plus username and password) to encode the negotiating for the one time key for your private session.

I come close to 100MB and look also at the position of your filtering/RAW lines to squeeze the last bits out of the connection.

10

If those were issues, they should have affected the transfer rate with and without l2tp/ipsec, but that was not the case.

11

Sorry about the delay, YES! Disabling:

/ip firewall add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related disabled=yes

.. fixed the issue. Is it possible to enable fasttrack for non-IPsec traffic? I tried:

/ip firewall add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related protocol=!ipsec-esp

But that didn’t work.

12

To enable fasttrack for traffic which is not going to be matched by the IPsec policy, you have to do the following:

/ip firewall filter print

Look for the number of your disabled fasttrack rule, use it instead of N in the following commands.

Now create an exception from the fasttrack rule:

/ip firewall filter add chain=forward src-address=s.s.s.s/smask dst-address=d.d.d.d/dmask connection-state=related,established action=accept place-before=N

s.s.s.s/smask and d.d.d.d/dmask are the same like in your IPsec policy.

This way, packets which need to be matched by the policy will be accepted before the fasttrack rule could match them.

Then, re-enable the original fasttrack rule previously disabled:

/ip firewall filter enable N+1

.

Your fasttrack rule with protocol=!ipsec-esp matches on packets which have not been encapsulated into IPsec yet, so it effectively replaces the original one you have disabled.

13

Actually, I forgot that the problem isn’t with fasttrack and IPsec, since it’s slow with or without it. The problem is with fasttrack and tunnels. Is there a way to disable fasttrack just for tunneled traffic?

Edit:
It looks like I can just filter it by interface:

/ip firewall add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related in-interface=!l2tp-out out-interface=!l2tp-out

14

OK, it was too late down here while I was reading your previous post. Actually, fasttracking makes processing of forwarded packets faster as fasttracked packets bypass some firewall handling, but the downside of it is they also bypass IPsec policy matching. But the latter part is not relevant to your case - you are not using policy-based “direct” IPsec where policy matching of forwarded packets is necessary, you use L2TP over IPsec so forwarded packets are routed to the L2TP virtual interface and their IPsec-encapsulated versions are not forwarded but sent locally so they are handled by a different firewall filter chain (“output” rather than “forward”).

So I actually cannot understand how disabling fasttracking could have speeded up your L2TP/IPsec processing. Can you compare the CPU load with and without fasttracking the L2TP traffic?

15

My understanding is that routing-marks are used to route though the l2tp interface. Routes with routing-marks are bypassed with fast-track as well.

16

I can’t really compare cpu load with and without fasttracking on the l2tp tunnel since with fasttracking, I get about 10 kbps upload through the tunnel. Without fasttracking on untunneled traffic, I get about 5% average cpu on 40 mbps throughput; 1-2% average cpu with fasttrack on the same throughput.

17

Yes, I’m using:

/ip firewall mangle add action=mark-routing chain=prerouting comment=VPN dst-address=!192.168.0.0/16 new-routing-mark=vpn passthrough=yes src-address=192.168.125.0/24

Along with /ip route rules matching that routing mark to send only traffic to specific networks through the l2tp tunnel. They appeared to be hitting the fasttrack rule/counter until I filtered out the l2tp interface though.

18

Well, my question was whether the 10 kbps upload means 100 % CPU on something (networking?) or the whether the data flow is slow but for some other reason than a fully loaded CPU.

But you’ve made me really curious, I’ll give it a try in the lab.

19

This could be the key in terms that not all packets of fasttracked connections are actually fasttracked. So if the fact that a packet belongs to an established connection does not substitue the routing mark, only those packet retransmissions which escape fasttracking for any reason would make it to the destination, so that would explain the drastic speed drop.

This is not easy to prove by simply setting the exception rule to check presence of routing mark because only packets sent to the tunnel bear the routing mark while packets coming from the tunnel can make the whole connection fasttracked as well, but it should be visible on packet sniffing - if this is the reason, then with fasttracking enabled there should be tons of TCP retransmissions on the LAN interface.

20

Whether a connection is fast-tracked or not will be visible in the connection tracking list. If mangling is needed, even for one direction, FT may not be used, as FT is a connection level marking.
An option/solution would be to mark connection for FT only on new connection event in ConnTrack. Once established, don’t mark them for FT-ing.