Is re-ordering fixed yet with IPSec and hardware acceleration? (Updating thread)

mikruser · Thu Dec 01, 2016 12:03 pm

I understand that Mikrotik says the ordering problem is being fixed (but when? ROS v7?). But can we temporarily get an option in the v6.x version to disable HW acceleration on the CCR platform, so that we can do software CBC on the CCR and hardware CBC on the hex 3r?
I vote for that as well!

it has already been proposed: http://forum.mikrotik.com/viewtopic.php?f=1&t=113911

mikruser · Thu Dec 01, 2016 12:06 pm

Why reordering issue occurs with hardware multicore, but not occurs with software multicore?

th0massin0 · Thu Dec 01, 2016 3:59 pm

It may be sticked to architecture (of CPU).

alexjhart · Thu Dec 01, 2016 9:56 pm

My latest update from support on this (from yesterday) is:
"We are working on the ipsec problem right now."

I'm not sure what that means for timeline, but it does show they are giving attention to this issue that I brought up with them about 1 year ago now.

nathan1 · Fri Dec 02, 2016 6:09 am

My latest update from support on this (from yesterday) is:
"We are working on the ipsec problem right now."

I'm not sure what that means for timeline, but it does show they are giving attention to this issue that I brought up with them about 1 year ago now.

Have you been poking them via email or did they reach out to you as an update? Crossing my fingers...

alexjhart · Wed Dec 14, 2016 6:35 pm

Just poking via ticket email.

pchott · Tue Jan 03, 2017 11:05 am

Just poking via ticket email.

Any news on this poking??

alexjhart · Tue Jan 03, 2017 10:59 pm

Just poking via ticket email.
Any news on this poking??

I just replied to my ticket again for another update. I haven't heard from them in over a month. Maybe 6.39 will be the lucky release?

alexjhart · Wed Jan 04, 2017 8:37 pm

Just poking via ticket email.
Any news on this poking??
I just replied to my ticket again for another update. I haven't heard from them in over a month. Maybe 6.39 will be the lucky release?

Support said:

We are still working on ike2, right after this we will continue to work on reordering problem.

If there last update was true, it seems like they must have taken time off to work on ike2 instead for a bit.

nathan1 · Thu Jan 19, 2017 3:13 am

My ticket on this as of 1/18/2017:

We will continue to work on this problem when ike2 main features will be finished.

The wait goes on.

alexjhart · Thu Jan 19, 2017 3:35 am

My ticket on this as of 1/18/2017:
We will continue to work on this problem when ike2 main features will be finished.
The wait goes on.

Thanks for joining in the struggle to get this fixed. Don't forget to update your starting post in the thread.

alexjhart · Tue Feb 14, 2017 7:02 pm

My last update says they're still working on ike2, but they will try to fix this in "one of the next versions". Unfortunately, the fix will be for lower core routers (CCR1009 and CCR1016) first. "Because solution for these routers are almost ready. For others, not yet." Hopefully support for the more expensive 1036 and 1072 will follow shortly after. We'll see. I'm not sure what the technical explanation is for not being able to do all at once.

Ascendo · Fri Feb 17, 2017 8:32 am

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.

nathan1 · Fri Feb 17, 2017 5:20 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.

I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?

Ascendo · Fri Feb 17, 2017 6:34 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.
I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?

We get around 750mbit/sec with EoIP + IPSEC. Would get a bit more if we had used the correct ports according to the block diagram. We did contact support - same story, no timeframe for a fix but 1009/1016 will be sorted first.

nathan1 · Fri Feb 17, 2017 6:55 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.
I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?
We get around 750mbit/sec with EoIP + IPSEC. Would get a bit more if we had used the correct ports according to the block diagram. We did contact support - same story, no timeframe for a fix but 1009/1016 will be sorted first.

750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.

pe1chl · Fri Feb 17, 2017 7:12 pm

750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.

Do you have issues in day-to-day operation or only when running benchmarks on Windows machines?
I ask because we use IPsec on CCR1009 without problem. But the traffic is from/to many different machines, not a single-connection-TCP-benchmark.
But even that works fine when between Linux systems.

nathan1 · Fri Feb 17, 2017 7:21 pm

750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.
Do you have issues in day-to-day operation or only when running benchmarks on Windows machines?
I ask because we use IPsec on CCR1009 without problem. But the traffic is from/to many different machines, not a single-connection-TCP-benchmark.
But even that works fine when between Linux systems.

We only have Linux machines but we have a lot of non-TCP flows, which the issue wreaks havoc on. Linux does cope well with the TCP re-ordering but it creates a very unstable flow, bandwidth and window sizes fluctuate significantly. At this point I have just entirely disabled the hardware acceleration and the network is very stable but throughput is lower than the physical capability. The biggest show stopper was with OSPF running over the EoIP sessions, it caused constant flapping and made for complete instability of the routing core and in turn the network.

alexjhart · Fri Feb 17, 2017 7:33 pm

We've been struggling to get GCM (i.e. software only encryption) working between our CCR1009s and Cisco ASAs. As such I'm in a real pickle having recommended Mikrotik.

We'll take any kind of workaround right now (e.g. run CCR1009 as single core router) until there is a proper fix.

For now, RB1100AHx2 is still the best router Mikrotik makes. Going down the Tilera path has been a mistake.
I've tried to suggest the single core workaround without success. Try contacting support, a little more demand certainly can't hurt. What kind of throughput are you able to get out of IPSec on a RB1100AHx2?
We get around 750mbit/sec with EoIP + IPSEC. Would get a bit more if we had used the correct ports according to the block diagram. We did contact support - same story, no timeframe for a fix but 1009/1016 will be sorted first.
750Mbit/sec EoIP + IPSec is pretty nice. I'm stuck with 14 of the 1009s doing 250Mbit at best. If I had known about the re-ordering issue before I deployed these 1009s, I'd be on the AHx2.
I think we are now going on over a year without resolution now and the CCR platform continues to be advertised as a high performance IPSec platform.

That's absolutely the case here too. We went with the tilera chip because of the promised hardware encryption throughput, but don't get it because of this driver problem. Unfortunately, we spent even more on 1036 and 1072 units. Insult to injury, they'll be the last to be fixed. I started reporting the issue to mikrotik well over 1 year ago. In the case of 10Gbps interfaces, we don't even have the option to replace with the AHx2. So I'm stuck waiting for them to get their act together or going with a different vendor.

StubArea51 · Wed Feb 22, 2017 10:47 pm

Glad to see this issue is getting attention still. Been waiting to see a fix on it ever since Alex brought it up at the 2016 US MUM.

Ascendo · Thu Feb 23, 2017 8:34 pm

Glad to see this issue is getting attention still. Been waiting to see a fix on it ever since Alex brought it up at the 2016 US MUM.

An actual update from Mikrotik staff would be great...

Fri Mar 10, 2017 3:20 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

nathan1 · Fri Mar 10, 2017 3:35 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

Thanks for the update! Testing soon...

nathan1 · Fri Mar 10, 2017 4:57 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

mrz, is EoIP in this version obeying the default proposals? I've attempted switching to hardware accelerated proposals and then recreating the EoIP but it doesn't seem quite right. Any suggestions on what configuration I should be using to test this fixed hardware acceleration?

 0  * name="default" auth-algorithms=sha256 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=modp1024
 
  0  D  ;;; XXX
       address=169.254.17.11/32 local-address=169.254.17.12 auth-method=pre-shared-key 
       secret="XXXXX" generate-policy=no policy-template-group=default exchange-mode=main 
       send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128,3des 
       dh-group=modp1024 lifetime=1d dpd-interval=2m dpd-maximum-failures=5

alexjhart · Fri Mar 10, 2017 5:47 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;

This is fantastic news. Does this work with all models, or as suggested by support earlier will it just be a subset at this time? Any details on what the specific behavior is now vs before?

Fri Mar 10, 2017 7:12 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;
mrz, is EoIP in this version obeying the default proposals? I've attempted switching to hardware accelerated proposals and then recreating the EoIP but it doesn't seem quite right. Any suggestions on what configuration I should be using to test this fixed hardware acceleration?
Code: Select all
 0  * name="default" auth-algorithms=sha256 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=modp1024
 
  0  D  ;;; XXX
       address=169.254.17.11/32 local-address=169.254.17.12 auth-method=pre-shared-key 
       secret="XXXXX" generate-policy=no policy-template-group=default exchange-mode=main 
       send-initial-contact=yes nat-traversal=yes proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128,3des 
       dh-group=modp1024 lifetime=1d dpd-interval=2m dpd-maximum-failures=5 

Peer proposal is phase1 and has nothing to do with hardware acceleration. Phase2 proposal can be selected in "/ip ipsec proposal" menu, this is hardware acceleration related.

Fri Mar 10, 2017 7:12 pm

What's new in 6.39rc51 (2017-Mar-10 12:50):

!) tile - fixed IPsec hardware acceleration out-of-order packet problem, significantly improved performance;
This is fantastic news. Does this work with all models, or as suggested by support earlier will it just be a subset at this time? Any details on what the specific behavior is now vs before?

Fix is for all CCR models.

nathan1 · Fri Mar 10, 2017 7:36 pm

Peer proposal is phase1 and has nothing to do with hardware acceleration. Phase2 proposal can be selected in "/ip ipsec proposal" menu, this is hardware acceleration related.

Phase 2 was not completing for me with sha256/aes-256-cbc, just seemed to stall. What proposal do you recommend testing with?

nathan1 · Sat Mar 11, 2017 12:41 am

So I believe that I have the rc working with sha1/aes-128-cbc but performance is not what I'd expect. Re-ordering does appear to be fixed but performance isn't anywhere near line rate (1Gbit in this case).

mrz, can you elaborate on what we should be seeing for performance on a single IPSec session with a UDP flow (or many TCP connections)? It feels as if the performance is nearly the same as a software aes ctr.

Has anyone else been able to test?

Ascendo · Sat Mar 11, 2017 9:55 am

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...

nathan1 · Sat Mar 11, 2017 12:44 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...

Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.

Ascendo · Sat Mar 11, 2017 2:03 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...
Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.

We currently get ~2Mbit file copies over a 85ms link, so I'll take 300 any day! You mentioned EoIP - are you seeing more than one core being maxed out during the UDP test?

nathan1 · Sat Mar 11, 2017 3:24 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...
Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.
We currently get ~2Mbit file copies over a 85ms link, so I'll take 300 any day! You mentioned EoIP - are you seeing more than one core being maxed out during the UDP test?

You can do 300 with the software encryption without the rc fix. What settings are you using? I use EoIP and a single core does max out with software but not hardware.

Ascendo · Sat Mar 11, 2017 3:37 pm

Nathan1, what sort of speeds are you getting per flow, and how high is the latency between the devices? At least there is some progress...
Latency is .7ms RTT and I was seeing around 300Mbit, very similar to the ctr. Blasting UDP peaks around 350Mbit, in clear I can push 1Gbit. I was only able to briefly test so hopefully someone can confirm or deny what I am seeing so I know which direction to go.
We currently get ~2Mbit file copies over a 85ms link, so I'll take 300 any day! You mentioned EoIP - are you seeing more than one core being maxed out during the UDP test?
You can do 300 with the software encryption without the rc fix. What settings are you using? I use EoIP and a single core does max out with software but not hardware.

We were unable to get GCM working with the particular Cisco ASA version we have on the remote end, so we've been stuck using hardware encryption (aes-128-cbc). Most of our use case is high latency low bandwidth so not sure we can add much to the discussion right now.

We do run some RB1100AHx2 with a 1Gb link between our primary and DR site (EoIP) which we were hoping to replace with CCRs at some point. I'll try replacing them with some CCR1009s next week (with the RC code) to see if they work any better.

nathan1 · Sat Mar 11, 2017 3:48 pm

We were unable to get GCM working with the particular Cisco ASA version we have on the remote end, so we've been stuck using hardware encryption (aes-128-cbc). Most of our use case is high latency low bandwidth so not sure we can add much to the discussion right now.

We do run some RB1100AHx2 with a 1Gb link between our primary and DR site (EoIP) which we were hoping to replace with CCRs at some point. I'll try replacing them with some CCR1009s next week (with the RC code) to see if they work any better.

I see. I think in your situation, since it is a compatibility issue, you will see huge improvements from the rc.

nathan1 · Sun Mar 12, 2017 3:37 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.

Ascendo · Sun Mar 12, 2017 4:50 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.

That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.

nathan1 · Sun Mar 12, 2017 5:13 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.
That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.

I would but I can't do it easily - I don't have any windows machines in my infrastructure.

alexjhart · Sun Mar 12, 2017 5:59 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.
That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.
I would but I can't do it easily - I don't have any windows machines in my infrastructure.

When I get a chance, I'll test and report back.

Ascendo · Sun Mar 12, 2017 6:14 pm

So I did some additional testing, I can actually push around 800Mbit with a UDP flow and CPU usage is pretty good. However, when I run a TCP benchmark, CPU usage is very high (networking/cpu under profile) which I believe is constraining the TCP throughput now. I think I may have an issue with fragmentation/DF/MSS that I am looking into.

Edit: I have confirmed that there is some issue with fragmentation in my setup during the hardware IPSec test. It is not clear what is going on but if I force the TCP MSS on iperf, I can confirm heavily increased throughput nearing the UDP test. It seems the increased CPU usage is due to the CCR being forced to fragment/reconstruct the TCP packets.

TLDR: This fix looks like what we have all been waiting for.
That's interesting. Any chance of doing a single-threaded windows copy (explorer/robobocopy) test? Very interested to see if it beats our ~450mbps result on the RB1100AHx2.
I would but I can't do it easily - I don't have any windows machines in my infrastructure.

Fair enough

I'll pick up some CCR1009s during the week and try replacing the existing RB1100AHx2s and see how it goes.

Now the only question is when the fix will be added to the bugfix/current builds. Hopefully the RC is stable enough for production.

Ascendo · Mon Mar 13, 2017 7:48 am

We have deployed this to a couple of small, low bandwidth remote sites (CCR1009) and are happy to report that the performance is now exactly what it should be. Will hopefully do some testing this week with more throughput but so far so good.

@MT - thanks for getting this sorted. Hopefully it'll be put into current/bugfix version soon.

theprojectgroup · Mon Mar 13, 2017 10:00 pm

Looking good here on CCR1016-12G, tested before and after the update

Site2Site IPIP Tunnel Spain (fibre 300mbits ISP: consumer) <-----------> Germany (fibre 100mbits ISP: m-net corp) with latency:60ms

SMB2 traffic:

speed.png

nathan1 · Wed Mar 15, 2017 1:07 am

Also confirming that this is looking very good. I have deployed it across my global network, backed up by a set of CCRs using the old software setup, in the event that the rc fails. Seeing peak performance around 800Mbit single TCP stream on my fastest connection (1Gbit @ .7ms). Currently using sha1/aes-128-cbc. I've had a few strange issues with 256 not wanting to go to phase 2 every time, not sure why. I did drop my MTU to compensate for the extra overhead vs. ctr. I am currently running at 1400 on the L2 VLANs.

Wed Mar 15, 2017 9:03 am

Itcould be great if anyone with 72core router could verify if it work,too.

alexjhart · Wed Mar 15, 2017 9:32 am

Itcould be great if anyone with 72core router could verify if it work,too.

I should be able to do that soon and will get back to you.

anuser · Sun Mar 19, 2017 9:51 pm

We have deployed this to a couple of small, low bandwidth remote sites (CCR1009) and are happy to report that the performance is now exactly what it should be. Will hopefully do some testing this week with more throughput but so far so good..

@Ascendo: What throughput do you get with IPSec on CCR1009?

@All: Does anyone use CCR1009 with EoIP and IPSec enabled with latest software? What throughput do you get in this situation?

Ascendo · Mon Mar 20, 2017 7:00 am

We have deployed this to a couple of small, low bandwidth remote sites (CCR1009) and are happy to report that the performance is now exactly what it should be. Will hopefully do some testing this week with more throughput but so far so good..
@Ascendo: What throughput do you get with IPSec on CCR1009?

@All: Does anyone use CCR1009 with EoIP and IPSec enabled with latest software? What throughput do you get in this situation?

The site we've deployed to only has 10mbps and it now maxes out the link when doing file copies to the remote DC (currently 69ms latency). Previously we'd be lucky to get 2mbps.

We have not tried replacing the pair of RB1100AHx2s which are running EoIP+IPSEC yet (1ms 1Gbps link).

Mon Mar 20, 2017 10:29 am

@anuser: if it is any use to you we got 2.4Gbps on CCR1009 with multiple streams (UDP 1400byte packets)

pchott · Mon Mar 20, 2017 10:42 am

@mrz: With UDP there is usual not such great error as at TCP.

I think the relevant speed test ist TCP. On my test i can get over 2.5Gb (before anythis changes) over UDP but under 300Mb over TCP. Since now is all in production I was not able to test this update-

Mon Mar 20, 2017 10:44 am

~2Gbps TCP 1360MSS

pchott · Mon Mar 20, 2017 10:48 am

~2Gbps TCP 1360MSS

Great to hear that. Thank you for info. I am looking forward that i can test this on CCR1036 that I use.

Mon Mar 20, 2017 10:51 am

~2Gbps TCP 1360MSS

Impressive!

Thank you for the baseline mrz

mikruser · Mon Mar 20, 2017 10:52 am

When we can expect this fix in "bugfix" branch?

anuser · Mon Mar 20, 2017 12:06 pm

~2Gbps TCP 1360MSS

Great! Is this EoIP including IPSec? Or IPSec only? Which parameter did you use?

Mon Mar 20, 2017 12:30 pm

It is pure ipsec.
EoIP over ipsec should be a bit slower because of overhead and ability to send smaller packets over the tunnel.

And note that these numbers cannot be achieved on single tunnel.

Siona · Mon Mar 20, 2017 1:21 pm

What's your single tunnel performance?

Wysłane z mojego ONEPLUS A3003 przy użyciu Tapatalka

Mon Mar 20, 2017 1:25 pm

1tun 650/700Mbps total 1.3Gbps (UDP), did not test with TCP.

nathan1 · Wed Mar 22, 2017 2:05 am

FYI - I am running single tunnel EoIP and can achieve single flow 700Mbit TCP - MTU 1400.

/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 disabled=no enc-algorithms=aes-128-cbc lifetime=30m name=default pfs-group=modp1024

normalcy · Wed Mar 22, 2017 4:05 am

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....

nathan1 · Wed Mar 22, 2017 12:36 pm

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....

theprojectgroup tested SMB: viewtopic.php?f=1&t=112545&start=50#p588615
The re-ordering is definitely fixed so as long as your MTU is good, you should see the performance that you expected to see back when we all put CCRs in place.

alexjhart · Mon Mar 27, 2017 8:11 pm

Itcould be great if anyone with 72core router could verify if it work,too.

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....

I'm using GRE with IPSEC. Also SMB traffic with Windows clients. I was able to get the RC loaded on a 1072 and a few 1036 CCRs this weekend. Initial testing shows that I can do at least the amount of throughput on hardware as software encryption was doing (CBC vs CTR), so that is a big step in the right direction. I have yet to do a capture and check how the packet order is looking. I'll try to do that soon. I'm not seeing anywhere near the throughput in an encrypted tunnel as am I in the same tunnel with encryption turned off. I don't know if that is directly related to the CCR1072 (since Mikrotik specifically asked about verifying it).

So I'm not ecstatic yet about the results, but I see that you guys are getting much more so maybe I have some more tweaking to do.

nathan1 · Fri Mar 31, 2017 10:16 pm

Itcould be great if anyone with 72core router could verify if it work,too.

Has anyone tried GRE over IPSec transport? Or SMB traffic on windows clients? That's been what we have had issues with.

Looking forward to this fix filtering down - I'll be sorely tempted to use current rather than waiting for bugfix....
I'm using GRE with IPSEC. Also SMB traffic with Windows clients. I was able to get the RC loaded on a 1072 and a few 1036 CCRs this weekend. Initial testing shows that I can do at least the amount of throughput on hardware as software encryption was doing (CBC vs CTR), so that is a big step in the right direction. I have yet to do a capture and check how the packet order is looking. I'll try to do that soon. I'm not seeing anywhere near the throughput in an encrypted tunnel as am I in the same tunnel with encryption turned off. I don't know if that is directly related to the CCR1072 (since Mikrotik specifically asked about verifying it).

So I'm not ecstatic yet about the results, but I see that you guys are getting much more so maybe I have some more tweaking to do.

These are the results that I got before I sorted out my MTU. The fragmentation will fry it. Have you confirmed that you aren't getting fragmentation over the tunnel?

normalcy · Sat Apr 01, 2017 12:48 pm

Thanks for the feedback nathan1 and alexjhart (and for you persistence chasing this!!).

My testing CCR1016 has been borrowed by a colleague but I look forward to trying this out on it when I get it back. We have multiple CCR1036 acting as VPN concentrators (never got the chance to swap them out after discovering this issue) that will benefit from this.

sbeauchamp · Mon Apr 03, 2017 10:46 pm

Should I expect this to be fixed on CCR1009-8G-1S-1S+? I haven't had a chance to test yet, but I ask because i notice this model isn't listed for sale anymore.

pe1chl · Mon Apr 03, 2017 11:41 pm

Should I expect this to be fixed on CCR1009-8G-1S-1S+? I haven't had a chance to test yet, but I ask because i notice this model isn't listed for sale anymore.

I think there is no reason to fear... the new model is mostly the same except it does not have the switch chip and it has more directly connected links to the processor instead.
The software running on those models is exactly the same. That is one advantage of MikroTik over many other manufacturers: common software over the entire product range, and thus less reason to terminate software support on out of sale models.

alexjhart · Tue Apr 04, 2017 2:05 am

These are the results that I got before I sorted out my MTU. The fragmentation will fry it. Have you confirmed that you aren't getting fragmentation over the tunnel?

So far as I can tell, this is not the case. However, if you made changes and got the performance expected, perhaps I am missing something. Any advice?

nathan1 · Tue Apr 04, 2017 2:27 am

These are the results that I got before I sorted out my MTU. The fragmentation will fry it. Have you confirmed that you aren't getting fragmentation over the tunnel?
So far as I can tell, this is not the case. However, if you made changes and got the performance expected, perhaps I am missing something. Any advice?

Run a ping with DF at the max size you THINK you can move across the tunnel and then run the packet sniffer on the encapsulating interface and the source interface, see if if you see one big packet followed by a smaller one. I also used ping with do-not-fragment=yes to ping mikrotik-mikrotik to determine the actual MTU. In my case, with EoIP and the change of encryption, my MTU was slightly decreased vs. what I had on CTR software.

Edit: Another way that I proved I was having MTU problems was to force the MTU down between two hosts that talk over the tunnel. With Linux, I was able to do this with iproute2 but I'm not sure how you do this with Windows? Perhaps use a UDP benchmark where you can set the datagram length and start low around 1200 just to get a baseline.

sbeauchamp · Tue Apr 04, 2017 2:48 pm

Heres what im seeing with the CSR1009-8g-1s-1s+

IPIP tunnel with IPSEc, holds ~200mbps
turning on QoS takes a pretty big hit down to ~70mbps
turning off connection tracking seems to give a 30-60mbps bump depending on the situation
no QoS and no connection tracking reaches about 300mbps.
upload seems odd, about 35mbps is where it tops out regardless of what features are turned on. (this may be on my end, not sure at the moment)

I'm using simple queue to prioritize certain types of traffic such as voice. Is there a better way or any more optimal settings to increase my performance from what I see above? Im using mangle rules to identify and mark the packets then use this queue config below:

0 name="LOU-HUB" target=lsv,lsv parent=none packet-marks="" priority=8/8 queue=default/default limit-at=0/0 max-limit=0/0
burst-limit=0/0 burst-threshold=0/0 burst-time=1s/0s bucket-size=0.1/0.1 total-queue=default

1 name="BGP-LOU1" parent=LOU-HUB packet-marks=BGP priority=1/1 queue=default/default limit-at=10k/10k max-limit=500k/500k
burst-limit=500k/500k burst-threshold=500k/500k burst-time=1s/1s bucket-size=0.1/0.1

2 name="BFD-LOU1" parent=LOU-HUB packet-marks=BFD priority=1/1 queue=default/default limit-at=10k/10k max-limit=10k/10k
burst-limit=10k/10k burst-threshold=10k/10k burst-time=1s/1s bucket-size=0.1/0.1

3 name="VOIPUDSP-LOU1" parent=LOU-HUB packet-marks=VOIP priority=2/2 queue=default-small/default-small limit-at=384k/384k
max-limit=512k/512k burst-limit=512k/512k burst-threshold=512k/512k burst-time=1s/1s bucket-size=0.1/0.1

4 name="SIGNALING-LOU1" parent=LOU-HUB packet-marks=SIP,SKINNY priority=3/3 queue=default/default limit-at=50k/50k
max-limit=500k/500k burst-limit=500k/500k burst-threshold=500k/500k burst-time=1s/1s bucket-size=0.1/0.1

5 name="OTHER-LOU1" parent=LOU-HUB packet-marks=OTHER priority=8/8 queue=default/default limit-at=50k/50k max-limit=500M/500M
burst-limit=500M/500M burst-threshold=500M/500M burst-time=1s/1s bucket-size=0.1/0.1 total-queue=default

the target in number 0 is my IPIP interface. For some reason it adds it in there twice. If i try to just have it once, the rule becomes invalid. It seems to work like this though.

Tue Apr 04, 2017 5:59 pm

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.

sbeauchamp · Thu Apr 06, 2017 2:59 pm

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.

That makes sense. This issue does indeed seem fixed to me. Im unfamiliar with how the releases go here, will this fix get pushed down to the bugfix line 6.37.x at some point?

I should expect the bigger CCR models with higher core counts to achieve higher total bandwidth (than the 1009) then with IPSEC+Queues as it does spread the streams among the cores correct?

royalpublishing · Mon Apr 10, 2017 5:55 pm

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.
That makes sense. This issue does indeed seem fixed to me. Im unfamiliar with how the releases go here, will this fix get pushed down to the bugfix line 6.37.x at some point?

I should expect the bigger CCR models with higher core counts to achieve higher total bandwidth (than the 1009) then with IPSEC+Queues as it does spread the streams among the cores correct?

I wouldn't get your hopes up too much about the higher core CCR's, I see the exact same thing you are seeing throughput wise. I am on 6.39rc58 on all of my devices (CCR1036 core and CCR1016 remote sites), but unfortunately even after trying all of the different hardware encryption combinations (SHA1/SHA256 and AES-128/192/256 CBC) I still cannot obtain an upload speed greater than around 42Mbps, but more realistically it sits about the mid 30's. I've been dealing with this problem since June of 2013 and with these updates I finally thought to myself "oh thank god they've finally got the problem fixed and I'll actually be able to use the full potential of my fiber connections", but that is not the case. Even with the re-ordering problem "fixed", I have seen zero difference whatsoever in upload throughput. In fact, I still get higher upload throughput using the software encryption. Even if I completely disable my queue trees, I see no changes in throughput. I do understand that having some queues, firewall/mangle rules, and connection tracking enabled will degrade performance, but I'm not sure that is even the cause of my issues here. I can still sit here and watch the state sequence errors increase while I try to upload a large file. It shouldn't be a fragmentation issue either as I've got mangle rules set up on both ends for MSS of 1360. I desperately would like to find a fix for this.

Jocksor · Mon Apr 10, 2017 7:12 pm

In my case 6.39rc58 improved bandwidth from ~70Mbps to ~250Mbps using EoIP + IPsec + Firewall w/o queues. Download\upload are the same.

It is like in any other configuration, adding simple queues or software queues on interface will significantly reduce performance. Stream is classified before encryption and all actions with packets from that stream are done on the same core.
That makes sense. This issue does indeed seem fixed to me. Im unfamiliar with how the releases go here, will this fix get pushed down to the bugfix line 6.37.x at some point?

I should expect the bigger CCR models with higher core counts to achieve higher total bandwidth (than the 1009) then with IPSEC+Queues as it does spread the streams among the cores correct?
I wouldn't get your hopes up too much about the higher core CCR's, I see the exact same thing you are seeing throughput wise. I am on 6.39rc58 on all of my devices (CCR1036 core and CCR1016 remote sites), but unfortunately even after trying all of the different hardware encryption combinations (SHA1/SHA256 and AES-128/192/256 CBC) I still cannot obtain an upload speed greater than around 42Mbps, but more realistically it sits about the mid 30's. I've been dealing with this problem since June of 2013 and with these updates I finally thought to myself "oh thank god they've finally got the problem fixed and I'll actually be able to use the full potential of my fiber connections", but that is not the case. Even with the re-ordering problem "fixed", I have seen zero difference whatsoever in upload throughput. In fact, I still get higher upload throughput using the software encryption. Even if I completely disable my queue trees, I see no changes in throughput. I do understand that having some queues, firewall/mangle rules, and connection tracking enabled will degrade performance, but I'm not sure that is even the cause of my issues here. I can still sit here and watch the state sequence errors increase while I try to upload a large file. It shouldn't be a fragmentation issue either as I've got mangle rules set up on both ends for MSS. I desperately would like to find a fix for this.

royalpublishing · Mon Apr 10, 2017 7:37 pm

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?

sbeauchamp · Mon Apr 10, 2017 8:48 pm

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?

I tried some of those 1100 optimizations on the CCR, but in my case it didn't really make a difference.

on the 1100 and on CHR x86 systems turning RPS off and assigning 1 CPU core per interface, and setting everything else to a core unused by interfaces seemed to greatly improve performance in my case. RPS actually seems to work correctly on the CCR, at least under crypto and with the latest RC software.

By the way on your CCRs, what are you seeing for your download speeds now?

royalpublishing · Mon Apr 10, 2017 9:56 pm

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?
I tried some of those 1100 optimizations on the CCR, but in my case it didn't really make a difference.

on the 1100 and on CHR x86 systems turning RPS off and assigning 1 CPU core per interface, and setting everything else to a core unused by interfaces seemed to greatly improve performance in my case. RPS actually seems to work correctly on the CCR, at least under crypto and with the latest RC software.

By the way on your CCRs, what are you seeing for your download speeds now?

Unfortunately, I can really only do a speed test in one direction at the moment until all of my other fiber circuits get turned up. Currently my main site has a 100/200Mbps fiber connection and one of the remote sites has a 40/100Mbps fiber connection, so I'm fairly limited in the amount of testing I can do. Interestingly enough, I can run a tcp bandwidth test from the main site to the branch and can get like 75Mbps, but an SMB upload from a Windows 8 machine only gets the 35Mbps like I was saying. I'm going to do some more testing with various types of uploads like from a linux box, ftp, robocopy, etc to see if maybe that's where I'm hitting the wall.

nathan1 · Wed Apr 12, 2017 4:58 am

@Jocksor, unfortunately I'm not using EoIP. I'm just using the basic IPSec in tunnel mode. On the IPSec wiki page, there are some optimizations for the RB1100AHx2 to set the irq's, would this help on the CCR?
I tried some of those 1100 optimizations on the CCR, but in my case it didn't really make a difference.

on the 1100 and on CHR x86 systems turning RPS off and assigning 1 CPU core per interface, and setting everything else to a core unused by interfaces seemed to greatly improve performance in my case. RPS actually seems to work correctly on the CCR, at least under crypto and with the latest RC software.

By the way on your CCRs, what are you seeing for your download speeds now?
Unfortunately, I can really only do a speed test in one direction at the moment until all of my other fiber circuits get turned up. Currently my main site has a 100/200Mbps fiber connection and one of the remote sites has a 40/100Mbps fiber connection, so I'm fairly limited in the amount of testing I can do. Interestingly enough, I can run a tcp bandwidth test from the main site to the branch and can get like 75Mbps, but an SMB upload from a Windows 8 machine only gets the 35Mbps like I was saying. I'm going to do some more testing with various types of uploads like from a linux box, ftp, robocopy, etc to see if maybe that's where I'm hitting the wall.

What is the latency between these sites? Can your saturate the connection using the same testing method, without the IPSec? The numbers you mention (100/200Mbps) I had previously achieved no problem with the software encryption. I can easily exceed them with the ordering fix w/hardware encryption.

mikruser · Tue Jun 27, 2017 5:33 pm

I repeat my question:
When we can expect this fix in "bugfix" branch?

Tue Jun 27, 2017 5:43 pm

Probably when v6.39 will become a bugfix.

mikruser · Mon Oct 02, 2017 5:38 pm

But when 6.39 become a bugfix???

very strange situation:

6.41.x = RC
6.40.x = Current
6.39.x = WTF??? Where??? When???
6.38.x = Bugfix

Mon Oct 02, 2017 7:38 pm

6.39.x = WTF??? Where??? When???

When it is stable enough, I believe.

normalcy · Wed Oct 18, 2017 3:55 am

I've just seen 6.39.3 come into the bugfix train.

Can anyone confirm it includes the fix? Unless I'm blind I didn't see it mentioned in the system -> packages brief release notes.

Out of the office for another week before I can try it on a cloud core.

Wed Oct 18, 2017 11:07 am

Unless I'm blind I didn't see it mentioned in the system -> packages brief release notes.

It was mentioned in the changelog for the original 6.39 release, so should be fixed in the latest 6.39 release as well.

miro · Fri Oct 20, 2017 11:54 am

Hi guys.
Do you have to change MSS to 1360, to achieve max performance or you still have reordering issue, if you don't change MSS... ?

Fri Oct 20, 2017 1:18 pm

The original problem was not related to MSS and/or packet fragmentation. The usual stream of non-fragmented TCP packets was also affected.

nathan1 · Sun Nov 05, 2017 2:38 am

Hi guys.
Do you have to change MSS to 1360, to achieve max performance or you still have reordering issue, if you don't change MSS... ?

Yes, I have my MTU dropped to 1400 on my EoIP interfaces. Last I checked, if packets need to be fragmented, it will fallback to software. This might look like a re-ordering issue in terms of performance but it is a different problem.

clamp-tcp-mss=yes mtu=1400 dont-fragment=no
using sha1/aes-128 cbc/modp1024

Who is online