Community discussions

 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Sat Sep 24, 2016 2:39 am

Fixed. See below and recent posts.
Update: 3/12/2017: 6.39rc51 is shown to have fixed the re-ordering issue. Testing is still being done to confirm performance. Please test if you can. This has been confirmed by fixed by two users. Bandwidth @ 1ms is around 700Mbit (1Gbit link) single stream TCP on MTU 1400.
Update: 2/14/2017: Still not fixed. Latest update from Alex Hart: http://forum.mikrotik.com/viewtopic.php ... 02#p583602
Update: 1/18/2017: Still not fixed. alexjhart and I have both checked our tickets with Mikrotik and received the following response:
We will continue to work on this problem when ike2 main features will be finished.
Update: 10/18/2016: Mikrotik has just released another hardware accelerated platform (MIPS based) that does NOT have a re-ordering problem: http://forum.mikrotik.com/viewtopic.php ... 59#p563380

This post *ONLY* impacts the Tile based platforms. The CCR platform is still suitable if your traffic is low enough (~<250Mbit), if you disable hardware acceleration: http://forum.mikrotik.com/viewtopic.php ... 01#p558911

-----

Since Mikrotik has been rather silent about an actual ETA about fixing this, I will create a thread to update as versions are released to notify everyone if it is fixed or not.

The IPSec hardware acceleration is useless in general, packets are randomly reordered creating disastrous effects on TCP and UDP streams. Until this is fixed, you should only use non-hardware accelerated implementations (CTR).

References:
http://forum.mikrotik.com/viewtopic.php?t=106960
http://forum.mikrotik.com/viewtopic.php ... 48#p538801
http://forum.mikrotik.com/viewtopic.php?t=84465
http://forum.mikrotik.com/viewtopic.php?t=98526
http://forum.mikrotik.com/viewtopic.php?t=106857

6.37 (NOT fixed):
# date; ping -c 10 -l 10 10.50.85.140
Fri Sep 23 19:34:11 EDT 2016
PING 10.50.85.140 (10.50.85.140) 56(84) bytes of data.
64 bytes from 10.50.85.140: icmp_seq=3 ttl=62 time=0.742 ms
64 bytes from 10.50.85.140: icmp_seq=4 ttl=62 time=0.755 ms
64 bytes from 10.50.85.140: icmp_seq=6 ttl=62 time=0.749 ms
64 bytes from 10.50.85.140: icmp_seq=8 ttl=62 time=0.742 ms
64 bytes from 10.50.85.140: icmp_seq=9 ttl=62 time=0.750 ms
64 bytes from 10.50.85.140: icmp_seq=1 ttl=62 time=0.801 ms
64 bytes from 10.50.85.140: icmp_seq=7 ttl=62 time=0.763 ms
64 bytes from 10.50.85.140: icmp_seq=10 ttl=62 time=0.753 ms
64 bytes from 10.50.85.140: icmp_seq=2 ttl=62 time=0.788 ms
64 bytes from 10.50.85.140: icmp_seq=5 ttl=62 time=0.777 ms

--- 10.50.85.140 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.742/0.762/0.801/0.019 ms, pipe 10
Last edited by nathan1 on Wed Mar 15, 2017 3:27 am, edited 11 times in total.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Sep 26, 2016 5:58 pm

I asked for another update on 8/30/16 via email to my ticket on this issue and was basically told to stop asking for updates and just watch the changelog.
-----
Alex Hart

The Brothers WISP
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Sep 26, 2016 9:28 pm

I asked for another update on 8/30/16 via email to my ticket on this issue and was basically told to stop asking for updates and just watch the changelog.
Yep, this is basically what I got the last time I tried to get information. Rather disappointing.
 
Ascendo
Frequent Visitor
Frequent Visitor
Posts: 67
Joined: Sun Sep 09, 2012 12:06 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Wed Sep 28, 2016 11:19 am

I wonder if this is related to the issue we are having with the RB1100AHx2: after some time (1-2days) encryption drops from 500+mbps to around 20mbps. Our solution is to disable/enable the IPSec encrypted EoIP tunnel with the scheduler.

We gave up with support on this as our workaround only results in 1-2 dropped pings every 12 hours and it's only replication traffic being affected.
 
mikruser
Member
Member
Posts: 377
Joined: Wed Jan 16, 2013 6:28 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Fri Sep 30, 2016 3:15 pm

In my opinion it will never be fixed.
This is a problem in hardware (tilera).
do not ask me why it is necessary.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Fri Sep 30, 2016 3:31 pm

Fixing it requires software handling to separate the traffic into streams, like with fair queuing.
A hash is computed from each packet header (at least source and destination IP, additionally maybe port numbers or other traffic keys).
Encryption is queued to the hardware in such a way that either a single stream is queued to a single processor, or the requests
are assigned sequence numbers and the resulting output is queued and resequenced before being sent on the network.

It is likely to cause a drop in performance, especially in benchmark tests (where the "test traffic" may be identified as a single stream
and thus processed serially rather than being distributed over the processors). So probably there will be whiners who observe a low
performance in their tests, while the performance is OK in normal use.

It will not be a simple fix, I guess... both technically and politically.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Fri Sep 30, 2016 6:08 pm

In my opinion it will never be fixed.
This is a problem in hardware (tilera).
Mikrotik has already agreed to fix it and have discussed possible solutions with me. I don't think this is an if, but when. Unfortunately, they won't commit to a timeline, so I'm not sure when it will be.
-----
Alex Hart

The Brothers WISP
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 03, 2016 11:55 pm

Even if RouterOS somehow forced all of the packets for a single IPSec session (not per inner flow) to hit a single core so they remain ordered then the performance would still be better than the software encryption workaround, at least in many use cases. I'd even be happy to designate a single core for all of my IPSec sessions.

Can anyone from Mikrotik chime in on why hardware IPSec acceleration is even advertised? I can't think of a single real world network that can run it without strange problems. It seems like it should be stricken from the feature set entirely right now.

Are any end users actually using it and getting expected behaviors?
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 11:10 am

I can't think of a single real world network that can run it without strange problems.
I think it is mostly a Windows problem. I run IPsec sessions with that behaviour between Linux systems and between Windows and Linux
with the data mainly flowing from Windows to Linux, and they do not appear to be affected too much. Apparently the TCP implementation
in Linux is much better than in Windows.
Are any end users actually using it and getting expected behaviors?
There is something else: I use L2TP/IPsec. When I use it between CCR and RB951G I get the nonsequenced behaviour
when running your test, but when I have a connection btween CCR and RB2011 it behaves nicely. I have configured compression
at the L2TP (PPP) level on all boxes, but it appears it is only successfully negotiated between RB2011 and CCR, and not
between RB951G and CCR, with the same configuration. It is not clear to me why that is. A difference is that my RB951G boxes
are behind NAT (so IPsec with NAT-T) and my RB2011 is not. I don't think that should affect the PPP layer, though.
Anyway, when compression is active at the PPP layer it resequences the packets much like I described before as a requirement
at the IPsec layer when multicore hardware encryption is done.
 
mikruser
Member
Member
Posts: 377
Joined: Wed Jan 16, 2013 6:28 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 6:41 pm

In my opinion it will never be fixed.
This is a problem in hardware (tilera).
Mikrotik has already agreed to fix it and have discussed possible solutions with me. I don't think this is an if, but when. Unfortunately, they won't commit to a timeline, so I'm not sure when it will be.
Look at the date of the message http://forum.mikrotik.com/viewtopic.php?t=84465
Its "Apr 28, 2014". Mikrotik team can not fix it within a few years. I no longer believe in their promises.
do not ask me why it is necessary.
 
mikruser
Member
Member
Posts: 377
Joined: Wed Jan 16, 2013 6:28 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 6:48 pm

Even if RouterOS somehow forced all of the packets for a single IPSec session (not per inner flow) to hit a single core so they remain ordered then the performance would still be better than the software encryption workaround, at least in many use cases. I'd even be happy to designate a single core for all of my IPSec sessions.

Can anyone from Mikrotik chime in on why hardware IPSec acceleration is even advertised? I can't think of a single real world network that can run it without strange problems. It seems like it should be stricken from the feature set entirely right now.

Are any end users actually using it and getting expected behaviors?
I monitored CPU usage on CCR1009 when copying a file via L2TP/IPsec tunnel (100Mbit WAN):
ccr_ipsec.png
very strange - why CCR uses 3 cores with very low usage???
copying speed very slow ~30...35 Mbit/s
You do not have the required permissions to view the files attached to this post.
do not ask me why it is necessary.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 6:58 pm

Even if RouterOS somehow forced all of the packets for a single IPSec session (not per inner flow) to hit a single core so they remain ordered then the performance would still be better than the software encryption workaround, at least in many use cases. I'd even be happy to designate a single core for all of my IPSec sessions.

Can anyone from Mikrotik chime in on why hardware IPSec acceleration is even advertised? I can't think of a single real world network that can run it without strange problems. It seems like it should be stricken from the feature set entirely right now.

Are any end users actually using it and getting expected behaviors?
I monitored CPU usage on CCR1009 when copying a file via L2TP/IPsec tunnel:
ccr_ipsec.png
very strange - why CCR uses 3 cores with very low usage???
Is your L2TP/IPSec tunnel is using hardware accelerated crypto? If so, then it makes sense. The Mikrotik is load balancing on a per packet basis, which effectively will distribute the load randomly across some set of cores. You would see a behavior that looks like this - this is exactly what creates the reordering.
 
mikruser
Member
Member
Posts: 377
Joined: Wed Jan 16, 2013 6:28 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 7:19 pm

>>Is your L2TP/IPSec tunnel is using hardware accelerated crypto?
Yes

>>The Mikrotik is load balancing on a per packet basis, which effectively will distribute the load randomly across some set of cores.
Ok, how can i change this to per-connection basis? One core should be enough to handle 100Mbit/s
do not ask me why it is necessary.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 7:33 pm

>>Is your L2TP/IPSec tunnel is using hardware accelerated crypto?
Yes

>>The Mikrotik is load balancing on a per packet basis, which effectively will distribute the load randomly across some set of cores.
Ok, how can i change this to per-connection basis? One core should be enough to handle 100Mbit/s
You can't. This is entirely the problem we are talking about here. Switch your settings to non-hardware accelerated crypto (like aes-128-ctr), a CCR1009 can handle over 200Mbit with software encryption.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 9:00 pm

I have given Mikrotik evidence that router-to-router-ipsec-with-hardware-encryption traffic is received out-of-order, therefore affecting the encapsulated traffic, and they have confirmed this is a problem. It does affect throughput. How much depends on latency, protocol, etc. SMB with Windows is one that is greatly impacted but the traffic being out of order. They know what the problem is and how they are going to fix. Mikrotik staff have confirmed with me multiple times in person and via email. Unfortunately, they just haven't been able to commit to a timeline on getting the new driver pushed out. The new driver will fix the quality issue by having packets arrive at the far router in order. I believe they will do this by forcing the ipsec connection to use a single core, rather than sending packets to different cores and have them processed at different times (out of order). My work around, which people like nathan1 also use, is to force software encryption. While that means less overall throughput with the tunnel, the user's application is actually able to push a lot more traffic because of better quality tunneling (for example 150Mbps vs 5-30Mbps using the same test and only changing hardware vs software ipsec encryption driver on routers). I would rather have users get a better overall experience than just get a good reading on speedtest.net. Who cares about speedtest.net if they have slow data transfers on applications they actually use.
-----
Alex Hart

The Brothers WISP
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 9:12 pm

SMB with Windows is one that is greatly impacted but the traffic being out of order.
I wonder why you reported the problem to MikroTik and expect them to fix it, instead of to Microsoft who are the actual owner of the problem?
After all, IP specifies an unsequenced unreliable datagram delivery, and routers are free to re-order packets.
For example, you could have 10 connections between two locations and routers could distribute the traffic over those 10 connections
without obligation to resequence them. It is the task of the end system to do that, and apparently Microsoft is failing at that job.

See https://en.wikipedia.org/wiki/Internet_Protocol under the Reliability header, heck even Microsoft writes this on
Technet: https://technet.microsoft.com/nl-nl/lib ... s.10).aspx under Network Interface layer.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 04, 2016 10:19 pm

SMB with Windows is one that is greatly impacted but the traffic being out of order.
I wonder why you reported the problem to MikroTik and expect them to fix it, instead of to Microsoft who are the actual owner of the problem?
After all, IP specifies an unsequenced unreliable datagram delivery, and routers are free to re-order packets.
For example, you could have 10 connections between two locations and routers could distribute the traffic over those 10 connections
without obligation to resequence them. It is the task of the end system to do that, and apparently Microsoft is failing at that job.

See https://en.wikipedia.org/wiki/Internet_Protocol under the Reliability header, heck even Microsoft writes this on
Technet: https://technet.microsoft.com/nl-nl/lib ... s.10).aspx under Network Interface layer.
Wait a minute here, packets are not guaranteed to be in order, but in general, they are and just about every method of bonding/load sharing does hashing in order to maintain this. Doing per packet load balancing is an absolute disaster for UDP flows. While the general TCP settings for Linux as defaulted and tuned by major distributions is favorable to some amount of re-ordering, it is not impervious to be impacted. You will see more re-transmissions on a single TCP stream over a single EoIP/IPSec session between two physically connected CCR1009s that are using hardware encryption vs. one that isn't. This is without a doubt a Mikrotik problem. I agree that it feels like they will never fix it but I'm hoping we are wrong about that.

Per packet load balancing is often a requested feature on platforms - for very specific applications. Mikrotik is the first vendor that I have ever seen to ship it as "default" and with no way to even disable it. Normally you would expect an L2 or an L3+L4 hash and something like per packet would be very explicitly configured.

http://www.cisco.com/c/en/us/td/docs/io ... /pplb.html
https://www.juniper.net/documentation/e ... rview.html
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 714
Joined: Tue Aug 25, 2009 12:01 am

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Wed Oct 05, 2016 2:59 am

The problem with misordered packets is that the receiving side has to wait for everything to come in so it can reorder them and feed them to the application. This results in the connection slowing down and scaling the window size smaller in order to guarantee delivery. You could tweak the settings in Linux to make it fast but that isn't real world.

Between waiting for bgp and IPv6 problems to be resolved and IPSec promises to be made good on (after being told repeatedly that "it works fine"), we've pulled most of the mikrotiks out of our environment. Sure. It cost us more money but the hardware we put in worked out of box without having to beg.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Wed Oct 05, 2016 3:37 am

The problem with misordered packets is that the receiving side has to wait for everything to come in so it can reorder them and feed them to the application. This results in the connection slowing down and scaling the window size smaller in order to guarantee delivery. You could tweak the settings in Linux to make it fast but that isn't real world.

Between waiting for bgp and IPv6 problems to be resolved and IPSec promises to be made good on (after being told repeatedly that "it works fine"), we've pulled most of the mikrotiks out of our environment. Sure. It cost us more money but the hardware we put in worked out of box without having to beg.
What hardware are you replacing with for IPSec?
-----
Alex Hart

The Brothers WISP
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Wed Oct 05, 2016 7:29 pm

The problem with misordered packets is that the receiving side has to wait for everything to come in so it can reorder them and feed them to the application. This results in the connection slowing down and scaling the window size smaller in order to guarantee delivery. You could tweak the settings in Linux to make it fast but that isn't real world.

Between waiting for bgp and IPv6 problems to be resolved and IPSec promises to be made good on (after being told repeatedly that "it works fine"), we've pulled most of the mikrotiks out of our environment. Sure. It cost us more money but the hardware we put in worked out of box without having to beg.
What hardware are you replacing with for IPSec?
I'm curious as well. I've considered Vyos on a SuperMicro SC515 or similar to replace my CCRs but so far I haven't pulled the trigger.
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 714
Joined: Tue Aug 25, 2009 12:01 am

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Sat Oct 08, 2016 2:20 am

Brocade MLXs. Having 400gbit of IPSec throughout and 2.4million hardware route scale is just a happy side effect. The reality is. We collapsed a lot of devices into 1 per building (9 total MLXs). The internet facing units have the newer 20 port 10gig IPSec enabled line cards.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Sat Oct 08, 2016 3:32 am

Brocade MLXs. Having 400gbit of IPSec throughout and 2.4million hardware route scale is just a happy side effect. The reality is. We collapsed a lot of devices into 1 per building (9 total MLXs). The internet facing units have the newer 20 port 10gig IPSec enabled line cards.
Very nice looking units but you definitely need a bit more physical space. I'm hoping to find a 1U solution to possibly the replace my CCRs at some point.
 
roadracer96
Forum Veteran
Forum Veteran
Posts: 714
Joined: Tue Aug 25, 2009 12:01 am

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Sun Oct 09, 2016 5:10 pm

Maybe that look at all he brocade 7450s. They have an available IPSec module. Now. Keep in mind this isn't really useful for road warrior. It's only designed for router to router links.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Sun Oct 09, 2016 5:39 pm

I only use the CCRs for site to site IPSec so the 7450 looks like a solid alternative. Thanks for the pointer.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:01 pm

So the hex r3 was just released and supposedly supports hardware encryption on a MIPS chip. Has anyone found out if this platform maintains order? It would be amusing if they do which would mean these $60 units outperform our $500+ units.

http://www.mikrotik.com/download/share/hexr3.pdf
http://forum.mikrotik.com/viewtopic.php?t=113068
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:06 pm

So the hex r3 was just released and supposedly supports hardware encryption on a MIPS chip. Has anyone found out if this platform maintains order? It would be amusing if they do which would mean these $60 units outperform our $500+ units.

http://www.mikrotik.com/download/share/hexr3.pdf
http://forum.mikrotik.com/viewtopic.php?t=113068
The 3011 hardware features encryption in hardware too, but their driver doesn't support that device yet. I'd be surprised if the hex r3 already had software support for the hardware encryption.
-----
Alex Hart

The Brothers WISP
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5910
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:16 pm

Yes, it does. It was already announced that hex 3 has HW support.
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:17 pm

Yes, it does. It was already announced that hex 3 has HW support.
mrz, can you speak to the problem we are all fighting here? Does the hex3 maintain packet order per-flow?
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:19 pm

Yes, it does. It was already announced that hex 3 has HW support.
Hey, Mikrotik finally joined the thread.

Does routeros have support for the hardware support the hex 3 offers, or is it like the 3011 that doesn't yet?
-----
Alex Hart

The Brothers WISP
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:21 pm

Yes, it does. It was already announced that hex 3 has HW support.
Hey, Mikrotik finally joined the thread.

Does routeros have support for the hardware support the hex 3 offers, or is it like the 3011 that doesn't yet?
Hey Alex, did you take a look at http://www.mikrotik.com/download/share/hexr3.pdf ? It seems to suggest (and I think mrz just confirmed) end to end support for HW accel.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:33 pm

Yes, it does. It was already announced that hex 3 has HW support.
mrz, can you speak to the problem we are all fighting here? Does the hex3 maintain packet order per-flow?
I would love to know this too.
-----
Alex Hart

The Brothers WISP
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5910
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:44 pm

Hex v3 is not affected by reordering problem you see on CCRs.
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 6:56 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
64 bytes from 192.168.50.1: seq=2 ttl=63 time=15.640 ms
64 bytes from 192.168.50.1: seq=3 ttl=63 time=15.546 ms
64 bytes from 192.168.50.1: seq=4 ttl=63 time=15.652 ms
64 bytes from 192.168.50.1: seq=5 ttl=63 time=15.066 ms
64 bytes from 192.168.50.1: seq=6 ttl=63 time=14.897 ms
64 bytes from 192.168.50.1: seq=7 ttl=63 time=15.763 ms
64 bytes from 192.168.50.1: seq=8 ttl=63 time=15.866 ms
64 bytes from 192.168.50.1: seq=9 ttl=63 time=15.708 ms
64 bytes from 192.168.50.1: seq=10 ttl=63 time=15.107 ms
64 bytes from 192.168.50.1: seq=11 ttl=63 time=15.716 ms
64 bytes from 192.168.50.1: seq=12 ttl=63 time=14.844 ms
64 bytes from 192.168.50.1: seq=13 ttl=63 time=15.183 ms
64 bytes from 192.168.50.1: seq=14 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=15 ttl=63 time=16.155 ms
64 bytes from 192.168.50.1: seq=16 ttl=63 time=15.043 ms
64 bytes from 192.168.50.1: seq=17 ttl=63 time=15.379 ms
64 bytes from 192.168.50.1: seq=18 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=19 ttl=63 time=15.172 ms
64 bytes from 192.168.50.1: seq=20 ttl=63 time=15.759 ms

It just hap lite and 10Mbps ADSL on the other side, so can't say anything about performance or anything, but so far it seems to be working ok.
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 7:00 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
64 bytes from 192.168.50.1: seq=2 ttl=63 time=15.640 ms
64 bytes from 192.168.50.1: seq=3 ttl=63 time=15.546 ms
64 bytes from 192.168.50.1: seq=4 ttl=63 time=15.652 ms
64 bytes from 192.168.50.1: seq=5 ttl=63 time=15.066 ms
64 bytes from 192.168.50.1: seq=6 ttl=63 time=14.897 ms
64 bytes from 192.168.50.1: seq=7 ttl=63 time=15.763 ms
64 bytes from 192.168.50.1: seq=8 ttl=63 time=15.866 ms
64 bytes from 192.168.50.1: seq=9 ttl=63 time=15.708 ms
64 bytes from 192.168.50.1: seq=10 ttl=63 time=15.107 ms
64 bytes from 192.168.50.1: seq=11 ttl=63 time=15.716 ms
64 bytes from 192.168.50.1: seq=12 ttl=63 time=14.844 ms
64 bytes from 192.168.50.1: seq=13 ttl=63 time=15.183 ms
64 bytes from 192.168.50.1: seq=14 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=15 ttl=63 time=16.155 ms
64 bytes from 192.168.50.1: seq=16 ttl=63 time=15.043 ms
64 bytes from 192.168.50.1: seq=17 ttl=63 time=15.379 ms
64 bytes from 192.168.50.1: seq=18 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=19 ttl=63 time=15.172 ms
64 bytes from 192.168.50.1: seq=20 ttl=63 time=15.759 ms

It just hap lite and 10Mbps ADSL on the other side, so can't say anything about performance or anything, but so far it seems to be working ok.

hap lite isn't the new hex 3 (mmips). Your test is showing software performance, not hardware encryption, which this thread talks about.
-----
Alex Hart

The Brothers WISP
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 7:11 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
64 bytes from 192.168.50.1: seq=2 ttl=63 time=15.640 ms
64 bytes from 192.168.50.1: seq=3 ttl=63 time=15.546 ms
64 bytes from 192.168.50.1: seq=4 ttl=63 time=15.652 ms
64 bytes from 192.168.50.1: seq=5 ttl=63 time=15.066 ms
64 bytes from 192.168.50.1: seq=6 ttl=63 time=14.897 ms
64 bytes from 192.168.50.1: seq=7 ttl=63 time=15.763 ms
64 bytes from 192.168.50.1: seq=8 ttl=63 time=15.866 ms
64 bytes from 192.168.50.1: seq=9 ttl=63 time=15.708 ms
64 bytes from 192.168.50.1: seq=10 ttl=63 time=15.107 ms
64 bytes from 192.168.50.1: seq=11 ttl=63 time=15.716 ms
64 bytes from 192.168.50.1: seq=12 ttl=63 time=14.844 ms
64 bytes from 192.168.50.1: seq=13 ttl=63 time=15.183 ms
64 bytes from 192.168.50.1: seq=14 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=15 ttl=63 time=16.155 ms
64 bytes from 192.168.50.1: seq=16 ttl=63 time=15.043 ms
64 bytes from 192.168.50.1: seq=17 ttl=63 time=15.379 ms
64 bytes from 192.168.50.1: seq=18 ttl=63 time=15.769 ms
64 bytes from 192.168.50.1: seq=19 ttl=63 time=15.172 ms
64 bytes from 192.168.50.1: seq=20 ttl=63 time=15.759 ms

It just hap lite and 10Mbps ADSL on the other side, so can't say anything about performance or anything, but so far it seems to be working ok.

hap lite isn't the new hex 3 (mmips). Your test is showing software performance, not hardware encryption, which this thread talks about.
There is hap lite in the other side of the tunnel, but I have new hex3 on my side and because the encryption method is aes-128-cbc, I presume it is using hw encryption..
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 7:39 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
..
It is important to do the ping with -l 10 to preload a burst. The default timing of ping will not allow you to see a re-order unless your termination devices are heavily loaded. Note the sequence numbers in my first post vs. yours - yours looks good, mine are re-ordered.
 
Fraction
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jan 16, 2013 9:42 pm
Location: Helsinki, Finland

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Oct 17, 2016 10:25 pm

Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)
Not 100% sure what nathan1 meant by adding ping output to first post, but this ping is going aes-128-cbc eoip tunnel (hex3 - hap lite):
64 bytes from 192.168.50.1: seq=0 ttl=63 time=16.875 ms
64 bytes from 192.168.50.1: seq=1 ttl=63 time=15.062 ms
..
It is important to do the ping with -l 10 to preload a burst. The default timing of ping will not allow you to see a re-order unless your termination devices are heavily loaded. Note the sequence numbers in my first post vs. yours - yours looks good, mine are re-ordered.
Yes, I noticed the difference, thats why I posted it. :) mrz said that everything should be ok with this, and it looked better than yours. :)
But ok, I "forgot" to use preload so here is the new one, different destination address, but still using the same tunnel:

sudo ping -c 10 -l 10 192.168.110.1
PING 192.168.110.1 (192.168.110.1) 56(84) bytes of data.
64 bytes from 192.168.110.1: icmp_seq=1 ttl=62 time=15.9 ms
64 bytes from 192.168.110.1: icmp_seq=2 ttl=62 time=17.9 ms
64 bytes from 192.168.110.1: icmp_seq=3 ttl=62 time=19.7 ms
64 bytes from 192.168.110.1: icmp_seq=4 ttl=62 time=22.0 ms
64 bytes from 192.168.110.1: icmp_seq=5 ttl=62 time=23.9 ms
64 bytes from 192.168.110.1: icmp_seq=6 ttl=62 time=26.4 ms
64 bytes from 192.168.110.1: icmp_seq=7 ttl=62 time=28.5 ms
64 bytes from 192.168.110.1: icmp_seq=8 ttl=62 time=30.5 ms
64 bytes from 192.168.110.1: icmp_seq=9 ttl=62 time=32.6 ms
64 bytes from 192.168.110.1: icmp_seq=10 ttl=62 time=34.7 ms

--- 192.168.110.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.929/25.261/34.718/6.060 ms, pipe 10
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 18, 2016 4:46 am

Yes, I noticed the difference, thats why I posted it. :) mrz said that everything should be ok with this, and it looked better than yours. :)
But ok, I "forgot" to use preload so here is the new one, different destination address, but still using the same tunnel:

sudo ping -c 10 -l 10 192.168.110.1
PING 192.168.110.1 (192.168.110.1) 56(84) bytes of data.
64 bytes from 192.168.110.1: icmp_seq=1 ttl=62 time=15.9 ms
64 bytes from 192.168.110.1: icmp_seq=2 ttl=62 time=17.9 ms
64 bytes from 192.168.110.1: icmp_seq=3 ttl=62 time=19.7 ms
64 bytes from 192.168.110.1: icmp_seq=4 ttl=62 time=22.0 ms
64 bytes from 192.168.110.1: icmp_seq=5 ttl=62 time=23.9 ms
64 bytes from 192.168.110.1: icmp_seq=6 ttl=62 time=26.4 ms
64 bytes from 192.168.110.1: icmp_seq=7 ttl=62 time=28.5 ms
64 bytes from 192.168.110.1: icmp_seq=8 ttl=62 time=30.5 ms
64 bytes from 192.168.110.1: icmp_seq=9 ttl=62 time=32.6 ms
64 bytes from 192.168.110.1: icmp_seq=10 ttl=62 time=34.7 ms

--- 192.168.110.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.929/25.261/34.718/6.060 ms, pipe 10
Yep, looks good. Now I just need to replace my 24 CCR1009-8G-1S-1S+ with these, a bargain at $1440 for all of them. The form factor isn't quite right for me and I need SFP+ so that isn't quite a reality. I'm glad to see this piece of hardware is being advertised with accurate and useful hardware acceleration though.

Mikrotik, can we be next? Disable all but one of my Tile cores for IPSec, to force serialization. I will be better off than I am right now, I'd take this as a real stop-gap fix. Really :)
 
Jeroen1000
Member Candidate
Member Candidate
Posts: 195
Joined: Fri Feb 18, 2011 2:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 18, 2016 12:19 pm

Does anyone know whether this occurs with regular TCP/UDP streams too (so without HW encryption)? Secondly, is SSTP working ok or is that HW accelerated too?
Bit of a shocker this thread:-)
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 18, 2016 2:43 pm

Does anyone know whether this occurs with regular TCP/UDP streams too (so without HW encryption)? Secondly, is SSTP working ok or is that HW accelerated too?
Bit of a shocker this thread:-)
Yes, it impacts all traffic. Note that you can use IPSec ciphers/hashes that will not use hardware encryption - allowing the platform to still work but slower (MUCH slower). This only affects Tile as far as we know (CCR*).
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Oct 18, 2016 6:06 pm

Does anyone know whether this occurs with regular TCP/UDP streams too (so without HW encryption)? Secondly, is SSTP working ok or is that HW accelerated too?
Bit of a shocker this thread:-)
Yes, it impacts all traffic. Note that you can use IPSec ciphers/hashes that will not use hardware encryption - allowing the platform to still work but slower (MUCH slower). This only affects Tile as far as we know (CCR*).
To clarify, this affects all types of traffic that is being encrypted by hardware offloaded encryption on tile chip. So if you are using ccr without encryption, that traffic isn't affected, (for example, http tcp without encryption is fine). That's why Nathan1 and I are forcing aes-ctr, which isn't supported by hardware acceleration and therefore software driven and not affected.

As for sstp, I don't believe it uses hardware acceleration, but could be wrong. If it doesn't, it won't be affected. I found one mum presentation that said it wasn't. I recall that tilera supports aes-cbc for hardware encryption. If you pass traffic and watch the cpu, you'll know pretty quick of it is using hardware acceleration (cpu utilization isn't affected nearly as much).

Hope that helps
-----
Alex Hart

The Brothers WISP
 
Jeroen1000
Member Candidate
Member Candidate
Posts: 195
Joined: Fri Feb 18, 2011 2:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Wed Oct 19, 2016 11:09 am

Thanks Alex and Nathan. Since I don't need more than 100 megabit CTR is ok. It's considered safe so there is no security trade off at least. Thanks for maintaining this thread pushing for a fix!
 
sallen
just joined
Posts: 11
Joined: Tue Feb 25, 2014 12:57 am

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Thu Oct 20, 2016 11:17 pm

Can someone tell me if this reasoning is correct? Even though the new hex r3 can support HW accelerated encryption without these reordering issues, it seems that there is no way to use one of those successfully on a site-to-site VPN with a CCR one one endpoint. As I understand it, the only way to force software encryption on the CCR is to use CTR mode, and the only way for hex r3 to use HW encryption is to use CBC mode. So therefore you are stuck, and there is no way to get decent speeds (for Windows SMB TCP traffic) between the two in a site-to-site IPSEC configuration.

I understand that Mikrotik says the ordering problem is being fixed (but when? ROS v7?). But can we temporarily get an option in the v6.x version to disable HW acceleration on the CCR platform, so that we can do software CBC on the CCR and hardware CBC on the hex 3r?
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Thu Oct 20, 2016 11:28 pm

Can someone tell me if this reasoning is correct? Even though the new hex r3 can support HW accelerated encryption without these reordering issues, it seems that there is no way to use one of those successfully on a site-to-site VPN with a CCR one one endpoint. As I understand it, the only way to force software encryption on the CCR is to use CTR mode, and the only way for hex r3 to use HW encryption is to use CBC mode. So therefore you are stuck, and there is no way to get decent speeds (for Windows SMB TCP traffic) between the two in a site-to-site IPSEC configuration.
...
Wow, if that is true, you have yourself in an even tighter jam than Alex and I are in. Maybe Fraction can chime in, I believe he was running this exact setup that you describe but he was peaking at 100Mbit.

Edit...thinking about this some more. You are definitely right if that is true, thats a big problem. Fraction, have you tried pushing ~100Mbit with that hex setup setup? According to to the hex r3 PDF (http://www.mikrotik.com/download/share/hexr3.pdf) , it seems like you are only going to manage around 40Mbit. Do you have some alternate config that seems to be working well?
 
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 107
Joined: Sat Jan 16, 2016 7:05 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Nov 07, 2016 8:39 pm

Have you picked up on something that may offer us relief or just another reference point?
Do you feel like they are trolling us? Their newsletter specifically states:
http://download2.mikrotik.com/news/news_73.pdf
To get the best performance two simple rules must be taken into account:
• avoid packet fragmentation;
• there must be no packet reordering.
And then one of their flagship platforms (CCR) that they have advertised as capable of high performance IPSec does exactly that (tons of reordering).
 
alexjhart
Member Candidate
Member Candidate
Posts: 191
Joined: Thu Jan 20, 2011 8:03 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Mon Nov 07, 2016 10:33 pm

Have you picked up on something that may offer us relief or just another reference point?
Just another reference point. I felt the same way when I saw the notes in the newsletter. I am somewhat comforted that they understand the important of the CCR IPsec driver issue.
-----
Alex Hart

The Brothers WISP
 
Pengu1n
just joined
Posts: 5
Joined: Tue Jan 31, 2012 12:24 am

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Tue Nov 22, 2016 5:40 pm

Got 1 hEX r3 for backup link. Connected it to RB2011 with IPSEC (aes128cbc-sha1-dh2). Many packet losses. Firmware 6.37.1. Today upgraded to 6.37.2. Same result.
Changing proposal to aes128ctr solves problem but IPSEC performance decreases: CPU load about 25-30% with 25mbps.
No such problem with RB1100AHx2 v6.31. hEX is going to table to wait for new firmware (and new bugs).
Thank you Mikrotik.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5910
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Fri Nov 25, 2016 1:40 pm

try to switch from sha1 to md5.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5545
Joined: Mon Jun 08, 2015 12:09 pm

Re: Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

Fri Nov 25, 2016 2:49 pm

I understand that Mikrotik says the ordering problem is being fixed (but when? ROS v7?). But can we temporarily get an option in the v6.x version to disable HW acceleration on the CCR platform, so that we can do software CBC on the CCR and hardware CBC on the hex 3r?
I vote for that as well! Preferably settable at the policy level (so you can do hardware accel for some and don't do it for others).
I now have several VPN (L2TP/Ipsec) clients that are Linux-based printers that have no issue at all with reordering, and one
GRE//IPsec VPN to a hEX r3 on a LAN with Windows workstations where the reordering is a problem. I would like to disable
hardware accel on that statically defined policy but keep it on the other ones. However, disable hardware accel for all AES-CBC
on the CCR would be a first step.

Who is online

Users browsing this forum: No registered users and 8 guests