Hello. We are facing some problems trying to manage good L2 connectivity across WAN link. From one side we have CCR1009, from another x86 mikrotik kvm based VM. Fw version 6.42.12. We set up EOIP + IPsec. Both ends has static ip address, connection speed is 1Gbit/sec, latency 60ms. I know this is really bad for such network desing, but it is very bad for now. Services that relies on TCP, demonstrate very poor performance, SMB file copy - 20-30Mbit/sec, rsync, nfs, scp about the same. I know about tcp window size, and many people dealing with high latency links this way, but iperf -w does not help and shows the same result, as i mentioned above. I thought it can be MTU issue, but settings are pretty common, 1500 on lan-bridge and auto MTU on EOIP interface (1308 calculated by PMTUD). So, any help would be appreciated

Look into L2TP BCP Bridging, it works better
https://wiki.mikrotik.com/wiki/Manual:B ... _bridging)

Do you require bridging (L2 interconnect)? If not try ipsec in tunnel mode (over UDP).

With regards to throughput, verify with iperf using UDP, and then TCP. I also think it's mtu related.

Do you require bridging (L2 interconnect)? If not try ipsec in tunnel mode (over UDP).

With regards to throughput, verify with iperf using UDP, and then TCP. I also think it's mtu related.

Thank you for reply. Yes we do require L2 connectivity. I can completely disable ipsec for this tunnel, but results are the same.

I can confirm same issue with TCP sessions over EOIP links with or without IPSEC

No solution or workaround have been found so far

Without eoip, on the same latency, do you get better results?
I can't imagine how could you get any reasonable speed on tcp with 60ms latency. That delay is just killing it.

As long as there is no packetloss TCP will scale up the bandwidth even on high latency links.

But every tiny bit of packetloss on high latency links will kill throughput.

Do you have any update about the subject?
I have the latest routeros with 2 ccr1009's and a gigabit connection in between

UDP 980 Mbps
TCP gives only 380 Mbps.

this is a known problem with mikrotik routers on high latency links.
this has been discussed many times on the forum.
you must contact support.

L2TP is generally no solution for anything on mikrotik as it is not stable. See my corresponding question from 15th Feb. L2TP on IPSec is very slow with single TCP streams. And I have not found any solution for both. Has anyone?
This is not dependant on high latency. L2TP has a stability problem per se.

@internetolog My test results are consistent with yours. I've contacted support, but no solution yet. Do not understand why TCP is so slow compared to UDP. Based upon posted data sheets, possible that IPSec tunnel with hardware acceleration is limited to around 500Mbps even on large CCRs. Cross referencing threads viewtopic.php?t=172641

@vikinggeek,
I solved my problem with multiple l2tp connections without security. I divided traffic into 4 and now I can handle more than 1G.
I hope this helps.

L2TP is generally no solution for anything on mikrotik as it is not stable. See my corresponding question from 15th Feb. L2TP on IPSec is very slow with single TCP streams. And I have not found any solution for both. Has anyone?
This is not dependant on high latency. L2TP has a stability problem per se.

Solution, solution is very simple - just use Multilink Protocol on ppp connection = set mrru to 1600 on both sites and disable tcp mss on the ppp profile too !

@internetolog and @JohnTRIVOLTA:

I've never had any problem with the stability of the L2TP connection (except for 6.48, but that's another story, 6.48.1 took care of it). My challenge is to get a single stream IPSec perform adequately. We need high file transfer performance on a single link. Looking at a PPP MultiLink configuration, it seems to be built to aggregate over two or more devices. I have only one SPF fiber interface or ONT to work with. It is possible that CCRs cannot deliver this type of performance? You both seem to suggest that aggregation will solve the problem. I understand aggregation will increase total capacity, but are we still not limited to the ~300Mpbs single stream?

@internetolog and @JohnTRIVOLTA:

I've never had any problem with the stability of the L2TP connection (except for 6.48, but that's another story, 6.48.1 took care of it). My challenge is to get a single stream IPSec perform adequately. We need high file transfer performance on a single link. Looking at a PPP MultiLink configuration, it seems to be built to aggregate over two or more devices. I have only one SPF fiber interface or ONT to work with. It is possible that CCRs cannot deliver this type of performance? You both seem to suggest that aggregation will solve the problem. I understand aggregation will increase total capacity, but are we still not limited to the ~300Mpbs single stream?

Not only physical links, mlp can transport the full size of packet instead tcp mss clamping on L4 ! Finally you must use AES 128 CBC for use hardware encryption !

mlp can transport the full size of packet instead tcp mss clamping on L4

Can you explain why hidden fragmentation of 1500-byte TCP packets (i.e. 2 PPP packets per each payload one) should provide a better TCP throughput than transmission of 1462-byte TCP packets using one PPP packet per each payload one?

I understand that MLPPP puts away the headache associated to PMTUD failures by allowing 1500-byte packets to get through without IP fragmentation at payload level, but I never thought of it as a method to improve throughput (on a single link, that is).

mlp can transport the full size of packet instead tcp mss clamping on L4

Can you explain why hidden fragmentation of 1500-byte TCP packets (i.e. 2 PPP packets per each payload one) should provide a better TCP throughput than transmission of 1462 byte TCP packets using one PPP packet per each payload one?
I understand that MLPPP puts away the headache associated to PMTUD failures by allowing 1500-byte packets to get through without IP fragmentation at payload level, but I never thought of it as a method to improve throughput (on a single link, that is).

The CPU load is less at L3 fragmentation vs L4 segmentation, this also reflects on latency ... this is my experience . Let's not forget the quality of the internet between the two locations !

Looking at a PPP MultiLink configuration, it seems to be built to aggregate over two or more devices.

Not only. It splits the payload packet into multiple transport ones, and it can use the same link to transport both/all of them, providing a hidden fragmentation of the payload, hence payload protocols lacking fragmentation capability can be transported without packet/frame size limitations.

Unless something has changed recently, RouterOS only supports actual multi-link capability as a PPPoE client; in all other cases (PPPoE server, L2TP, ...), it supports just a single-link, but with MLPPP if set accordingly (mrru set to some value).

And as you say, from the outside, and hence also from the IPsec perspective, an L2TP session using MLPPP is still a single UDP stream.