Hi.

I have several offices and i want to jojn them using IPIPtunnel witch IPsec so i bought several MT CCR1036 and CCR1009 because they have hardware IPsec acceleration very good throughput (based on ruouterbord.com). But when I setup everything I was surprised by very low speed when i try copy file between two offices (both offices have symmetric 500/500 fiber), I got file transfer around 4-10MB/s that is about 40-80 Mbps. So I create lab environment with very simple configuration, without any firewall configuration. Only IP address, basic static routing, and IPIP tunnel with IPsec encryption on default settings.
Client1->CCR1009<-----IPIP+IPsec------>CCR1009<-Client2
And result was this same (windows file copy, FTP, HTTP... ) max transfer was about 40-80Mbps, so It look oddly for that powerful hardware.
So I create series of tests using Iperf as reliable testing software running on client computers, So mikrotik not have to waste resources to generate traffic.
After several test, turned out that mikrotik has no problem with resources but with number of connections.
When i try send something through ipip+ipsec tunnel using only one connection i got 40-80Mps but when i try send something but using 20 simultaneous connection I got 800 Mbps, what is already a good result (That transfer is in one direction from client1->client2)

I attach several files witch diagram of my testing lab, files witch configuration, and table of Test Results.

Maybe someone knows how to solve this problem and improve transfer on single connection?
I known that results on ruterboard.com was for UDP traffic and looks very nice but in real word most users use TCP, and in real word when user try copy file from one office to another or try download file from company website through IPsec tunnel and got only 50Mbps file transfer It does not look good, especially after buying almost the most powerful device available from manufacturer.

the rest of attachments

I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818

Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.

I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818

Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.

Oh man you were spot on. Hardware encryption is broken in my case all the way from 6.23.1 up to 6.35.1 on CCR1016 and is pretty much useless one threaded. This means every file copy through tge tunnel.
Thanks for mentioning this.

this test confirm the issue

http://forum.mikrotik.com/viewtopic.php ... 95#p503869

1.7 Gig IPSEC encrypted (1500 MTU) - Minimum of 25 to get max throughput.

something like 68mbps per tcp connection

maybe using smb3.0 can improve the situation using multiple connections per transfer

https://blogs.technet.microsoft.com/jos ... d-smb-3-0/

Hello
i see that you use Windows for the iperf test - can you redo the test with a linux distro? I did same setup and have very big difference if i use Windows or Linux distro.

I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818

Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.

Oh man you were spot on. Hardware encryption is broken in my case all the way from 6.23.1 up to 6.35.1 on CCR1016 and is pretty much useless one threaded. This means every file copy through tge tunnel.
Thanks for mentioning this.

In talking to Mikrotik, it sounds like the issue with hardware right now is having packets from the same connection processed by different cores causes the encapsulated packets to be sent out of order. This results in the device on the remote end of the tunnel getting duplicate acks, out-of-order packets, etc (tcp assumes packets are lost, tries to compensate, but that takes additional time and bandwidth to be consumed in the process). All that provides poor throughput on a single stream. Of course, that is worse for certain connections (higher latency) and services like smb that aren't built to perform well on error prone connections (especially older versions). Many times you can't control what type of traffic and services will be used on a connection, so it is best to just fix the issue on the device doing the encryption that is introducing these issues. I think one solution they might be considering is locking a connection to a single core. The good news is that the Tile chipset should be able to do several hundred mbps of hardware encryption on a single core, which is better than the non-offloaded, software encryption.

I'm doing multiple software tunnels with load balancing as a bandaid fix until they can release the fix for the hardware encryption driver. This gives me ~150Mbps per stream multiplied by number of tunnels, which is better than I can get with the current poor quality hardware encryption.

I asked for an update on my ticket with Mikrotik and they say they are working on a fix, which is great news. Hopefully that won't take too long.

Hey Alex,

It looks like I am fighting the same annoying battle: http://forum.mikrotik.com/viewtopic.php?f=2&t=106960
I also have a ticket open with MT as of April 7th (#2016040766000158) and I continue to get vague "we are working on it" with no timeline. Out of curiosity, when did you notify them of the issue for your ticket?

Yes, this is likely the same issue. I have been going back and forth with them since December of last year. Wasn't until March they admitted to/understood the issue. It sounded like they had some ideas on how to fix it when I talked to them in April. My last update was May 20 when they said, "We are working on the fix." 

My last update was June 10th:

We are working on the problem. It will be in one of the upcoming releases. You will definitely see it in changelog.

I fear we could be waiting a very long time.

It sounds like you have gone with aes-256-ctr to force it over to software as a workaround? Was this your final stable config? I have been running it with the hardware acceleration and it has been decent for TCP but we now have some UDP flows that are getting destroyed from the reordering.

Yeah that's what I'm using for now. Software is much higher quality, but unfortunately comes with a performance bottleneck. I'm with you; hopefully they get this worked out soon so we get high quality and throughput (meaning higher goodput). 

My last update was June 10th:

We are working on the problem. It will be in one of the upcoming releases. You will definitely see it in changelog.

I fear we could be waiting a very long time.

CCR is manufactured already for 4 years.
4 years, Carl!
You think they did not have time to work on the problem?