Community discussions

MikroTik App
 
dev246
just joined
Topic Author
Posts: 16
Joined: Thu Jun 26, 2014 4:24 pm

Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Fri Apr 01, 2016 2:09 pm

Hi.

I have several offices and i want to jojn them using IPIPtunnel witch IPsec so i bought several MT CCR1036 and CCR1009 because they have hardware IPsec acceleration very good throughput (based on ruouterbord.com). But when I setup everything I was surprised by very low speed when i try copy file between two offices (both offices have symmetric 500/500 fiber), I got file transfer around 4-10MB/s that is about 40-80 Mbps. So I create lab environment with very simple configuration, without any firewall configuration. Only IP address, basic static routing, and IPIP tunnel with IPsec encryption on default settings.
Client1->CCR1009<-----IPIP+IPsec------>CCR1009<-Client2
And result was this same (windows file copy, FTP, HTTP... ) max transfer was about 40-80Mbps, so It look oddly for that powerful hardware.
So I create series of tests using Iperf as reliable testing software running on client computers, So mikrotik not have to waste resources to generate traffic.
After several test, turned out that mikrotik has no problem with resources but with number of connections.
When i try send something through ipip+ipsec tunnel using only one connection i got 40-80Mps but when i try send something but using 20 simultaneous connection I got 800 Mbps, what is already a good result (That transfer is in one direction from client1->client2)

I attach several files witch diagram of my testing lab, files witch configuration, and table of Test Results.

Maybe someone knows how to solve this problem and improve transfer on single connection?
I known that results on ruterboard.com was for UDP traffic and looks very nice but in real word most users use TCP, and in real word when user try copy file from one office to another or try download file from company website through IPsec tunnel and got only 50Mbps file transfer It does not look good, especially after buying almost the most powerful device available from manufacturer.
You do not have the required permissions to view the files attached to this post.
 
dev246
just joined
Topic Author
Posts: 16
Joined: Thu Jun 26, 2014 4:24 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Fri Apr 01, 2016 2:17 pm

the rest of attachments
You do not have the required permissions to view the files attached to this post.
 
alexjhart
Member Candidate
Member Candidate
Posts: 196
Joined: Thu Jan 20, 2011 8:03 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Thu Apr 28, 2016 6:44 pm

I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818

Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.
-----
Alex Hart

The Brothers WISP
 
mortar8
just joined
Posts: 21
Joined: Mon Sep 16, 2013 1:41 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Fri Apr 29, 2016 2:01 pm

I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818

Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.
Oh man you were spot on. Hardware encryption is broken in my case all the way from 6.23.1 up to 6.35.1 on CCR1016 and is pretty much useless one threaded. This means every file copy through tge tunnel.
Thanks for mentioning this.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 1772
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Mon May 02, 2016 6:21 am

this test confirm the issue

http://forum.mikrotik.com/viewtopic.php ... 95#p503869
1.7 Gig IPSEC encrypted (1500 MTU) - Minimum of 25 to get max throughput.
something like 68mbps per tcp connection

maybe using smb3.0 can improve the situation using multiple connections per transfer

https://blogs.technet.microsoft.com/jos ... d-smb-3-0/
 
Engitech
Trainer
Trainer
Posts: 65
Joined: Mon Feb 13, 2012 1:59 pm
Location: Geneva - Switzerland
Contact:

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Sat May 07, 2016 8:59 am

Hello
i see that you use Windows for the iperf test - can you redo the test with a linux distro? I did same setup and have very big difference if i use Windows or Linux distro.
 
alexjhart
Member Candidate
Member Candidate
Posts: 196
Joined: Thu Jan 20, 2011 8:03 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Fri May 20, 2016 7:07 pm

I wish I had a more detailed post to direct you to, but check out this for now:
http://forum.mikrotik.com/viewtopic.php ... 50#p534818

Likely what you are seeing is an issue with the hardware encryption driver that causes problems with packet delivery. With TCP, this often results in poor single thread performance. You can verify by switching to software encryption.
Oh man you were spot on. Hardware encryption is broken in my case all the way from 6.23.1 up to 6.35.1 on CCR1016 and is pretty much useless one threaded. This means every file copy through tge tunnel.
Thanks for mentioning this.
In talking to Mikrotik, it sounds like the issue with hardware right now is having packets from the same connection processed by different cores causes the encapsulated packets to be sent out of order. This results in the device on the remote end of the tunnel getting duplicate acks, out-of-order packets, etc (tcp assumes packets are lost, tries to compensate, but that takes additional time and bandwidth to be consumed in the process). All that provides poor throughput on a single stream. Of course, that is worse for certain connections (higher latency) and services like smb that aren't built to perform well on error prone connections (especially older versions). Many times you can't control what type of traffic and services will be used on a connection, so it is best to just fix the issue on the device doing the encryption that is introducing these issues. I think one solution they might be considering is locking a connection to a single core. The good news is that the Tile chipset should be able to do several hundred mbps of hardware encryption on a single core, which is better than the non-offloaded, software encryption.

I'm doing multiple software tunnels with load balancing as a bandaid fix until they can release the fix for the hardware encryption driver. This gives me ~150Mbps per stream multiplied by number of tunnels, which is better than I can get with the current poor quality hardware encryption.

I asked for an update on my ticket with Mikrotik and they say they are working on a fix, which is great news. Hopefully that won't take too long.
-----
Alex Hart

The Brothers WISP
 
nathan1
Member Candidate
Member Candidate
Posts: 159
Joined: Sat Jan 16, 2016 7:05 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Mon Jun 13, 2016 7:09 am

Hey Alex,

It looks like I am fighting the same annoying battle: http://forum.mikrotik.com/viewtopic.php?f=2&t=106960
I also have a ticket open with MT as of April 7th (#2016040766000158) and I continue to get vague "we are working on it" with no timeline. Out of curiosity, when did you notify them of the issue for your ticket?
 
alexjhart
Member Candidate
Member Candidate
Posts: 196
Joined: Thu Jan 20, 2011 8:03 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Mon Jun 13, 2016 7:23 am

Yes, this is likely the same issue. I have been going back and forth with them since December of last year. Wasn't until March they admitted to/understood the issue. It sounded like they had some ideas on how to fix it when I talked to them in April. My last update was May 20 when they said, "We are working on the fix." 
-----
Alex Hart

The Brothers WISP
 
nathan1
Member Candidate
Member Candidate
Posts: 159
Joined: Sat Jan 16, 2016 7:05 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Mon Jun 13, 2016 7:29 am

My last update was June 10th:
We are working on the problem. It will be in one of the upcoming releases. You will definitely see it in changelog.
I fear we could be waiting a very long time.

It sounds like you have gone with aes-256-ctr to force it over to software as a workaround? Was this your final stable config? I have been running it with the hardware acceleration and it has been decent for TCP but we now have some UDP flows that are getting destroyed from the reordering.
 
alexjhart
Member Candidate
Member Candidate
Posts: 196
Joined: Thu Jan 20, 2011 8:03 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Mon Jun 13, 2016 7:33 am

Yeah that's what I'm using for now. Software is much higher quality, but unfortunately comes with a performance bottleneck. I'm with you; hopefully they get this worked out soon so we get high quality and throughput (meaning higher goodput). 
-----
Alex Hart

The Brothers WISP
 
mikruser
Member
Member
Posts: 462
Joined: Wed Jan 16, 2013 6:28 pm

Re: Very low TCP transfer speed on IPIP+IPsec on CCR1009 and CCR1036

Thu Jun 16, 2016 11:58 am

My last update was June 10th:
We are working on the problem. It will be in one of the upcoming releases. You will definitely see it in changelog.
I fear we could be waiting a very long time.
CCR is manufactured already for 4 years.
4 years, Carl!
You think they did not have time to work on the problem?
do not ask me why it is necessary.

Who is online

Users browsing this forum: aesmith, Baidu [Spider], Bing [Bot], TimurA and 110 guests