GRE over IPSEC, CCR, VERY SLOW

roadracer96 · Mon Apr 28, 2014 4:12 pm

simple GRE tunnel over IPSEC transport mode, AES-CBC (Tried other algs as well), between CCR 1036 and RB1100AHx2 maxes at about 50Mbit aggregate throughput. 25/25 tx/rx, 5/45 tx/rx, etc.

Same tunnel disabling the IPSEC policy nets over 500mbit aggregate throughput.

Running the same setup between to RB1100AHx2s and I get about 500mbit aggregate throughput with encryption enabled.

Obviously some problem with CCR.

If I just do IPSEC tunnel mode across the CCR, performance seems good, unfortunately, I need to use routing protocols.

ROS 6.12, also tried 6.11 with same results (6.11 actually seemed slower?)

i4jordan · Mon Apr 28, 2014 4:49 pm

i can second this. GRE (or IPIP) + ipsec seems very slow between CCR and 1100AHx2.
I also tried almost all AES variants. but the performance seems to be limited to around 50Mbps.

roadracer96 · Mon Apr 28, 2014 5:46 pm

I emailed support before posting this to open a case.

Do you already have a case open?

i4jordan · Tue Apr 29, 2014 12:10 am

No, I do not have a case open for this.

But next coming weeks I will spend some time testing all variant of ipsec connections.
Strange thing indeed is that ipsec tunnels seems much faster than ipsec over IPIP or GRE tunnels.
We have one ipsec tunnel from a SonicWall NSA3500 series to our CCR1036, this one tops at 17-18MB/s (around 180Mbps) which is fair because our CCR is connected at 500Mbps and the SonicWall at 200Mbps. This ipsec is AES-256 at Phase 2.
Both connections are fiber connected to the same ISP. So in best case they both are connected to the same switch.

I will be back with a table of all kinds of tested ipsec speeds.

roadracer96 · Tue Apr 29, 2014 2:21 pm

Same here.. If I use tunnel mode instead of transport for subnet to subnet communication, its as fast as you would expect it to be. Its only when tunneling GRE or IPIP that it slows. It sounds like a operating system quirk when the 2 are combined. Unfortunately, I need to run GRE tunnels for routing protocols, multicast, and VPLS.

Ive tried several variations. All 3 forms of AES - GCM, CTR, CBC, 3des, blowfish, sha and md5 hashing. Even setup L2TP/IPSEC and got the same results. If I replace the CCR with an RB1100AHx2, speed is nearly 10x faster on the same config.

i4jordan · Tue Apr 29, 2014 3:00 pm

Did you try 2x CCR for the GRE+ipsec setup?

I am planning to buy a second CCR just to test this.
I do not have 2x RB1100AHx2 to test such a setup.

But I indeed do suspect a not optimized piece of code in the CCR firmware.

roadracer96 · Tue Apr 29, 2014 3:22 pm

I have 2 CCRs, but haven't tested it that way. They are a redundant pair... 2x CCR will end up serving 5x sites each with 2x RB1100AHx2. I suppose I could create a tunnel between them to test, but without hearing from support, its kind of a moot point.

roadracer96 · Mon May 05, 2014 9:04 pm

1 week, multiple emails to support, no response.. I know they know about it.. After searching the forums, I can see it is a problem with their code on the Tilera CPU... Alpha quality at best.

i4jordan · Thu May 15, 2014 11:51 pm

Today a few CCR and RB1100AHx2 models arrived. I am going to use those to test all kinds of VPN tunnels and will report the measured speeds.

I will test:
- IPIP + ipsec (transport)
- GRE + ipsec (transport)
- EOIP + ipsec (transport)
- ipsec tunnel

Mainly with AES encryption because this should be hardware supported.

I will test at least the following combinations.

CCR1036 <-> CCR1036
CCR1009 <-> CCR1009
CCR1036 <-> CCR1009
CCR1xxx <-> RB1100AHx2
CCR1xxx <-> SonicWall SRA
RB1100Ahx2 <-> SonicWall SRA

Have some patience, I will keep you updated next coming days.

roadracer96 · Fri May 16, 2014 1:43 am

They have already responded. That's the way it is. The tcp connection, gre tunnel, forwarding, and ipsec are all processed on one core. Apparently 1 core of a rb1100ahx2 is faster than 1 core of a tilera.

erkel · Mon Aug 25, 2014 3:35 pm

Bump, any prospect of the CCR being usable in this kind of configuration?, or do I have downgrade to a RB1100AHx2

erkel · Wed Aug 27, 2014 5:07 pm

I have decided to simply pull the mikrotik hardware out and run a couple of x86 box with router OS on them.

Problem solved.

Bump, any prospect of the CCR being usable in this kind of configuration?, or do I have downgrade to a RB1100AHx2

i4jordan · Fri Oct 10, 2014 2:53 pm

I think there is a huge problem with working with GRE tunnels with ipsec on teh CCR series.
Also with IPIP + ipsec.

We have a RB1100AHx2 connected via a GRE tunnel to a CCR1036.
The RB1100AHx2 is connected with cable 120/10Mbit and the CCR with fiber 500/500 Mbit.

If we disable ipsec on that tunnel I do get the expected max speed;

RB1100 -> CCR is stable at 1,4MB/sec data
CCR->RB1100 is stable at 12MB/sec data.

If I enable ipsec on that tunnel I do get the same max speed from RB1100 -> CCR.
But CCR->RB1100 maxs at around (500kB/sec data).

Can anybody help or explain this strange problems.

wa4zlw · Fri Oct 10, 2014 7:44 pm

Hi there! This is not good.

What I need to know if there are any specs on the various hardware platforms for performance especially throughput on VPNs. That is something I need access to. THen we can compare the specs to real-world.

Mikrotik needs to be more forthcoming all the way around as well as swat bugs not adding new features on all hardware platforms.

Thanks leon

Mon Oct 13, 2014 12:46 pm

In my test setup between two CCRs, gre over ipsec had no problems fowarding 500Mbps with 1450 byte packets.

ATG · Thu Oct 23, 2014 1:58 pm

Hello,

I just did a test now EoIP(encrypted IPSec 3des), both sites running running CCR-1009-8G-1S-1S+ v6.20, 3.19 firmware.

Seems that only one CPU core handles the traffic(100% on one core, 0% on the rest), top speed over the EoIP tunnel maxes out on approx 43Mbps...

Will this get sorted out?

Or will GRE work better(Higher speed)?

roadracer96 · Tue Oct 28, 2014 5:02 pm

In my test setup between two CCRs, gre over ipsec had no problems fowarding 500Mbps with 1450 byte packets.

Its pretty obvious that your perfect conditions test case doesn't reflect real world performance.

EnigmAX · Mon Nov 03, 2014 11:44 pm

In my test setup between two CCRs, gre over ipsec had no problems fowarding 500Mbps with 1450 byte packets.
Its pretty obvious that your perfect conditions test case doesn't reflect real world performance.

I landed on this page, because I'm having *exactly* the issue described in this topic.

@mrz: can you be so kind to share the relevant parts of your config, to see how your tunnel was configured?
This really looks like a serious issue and I think it would be good to look into this...

Tue Nov 04, 2014 11:04 am

The problem is only with TCP traffic because of fragmentation, out-of-order packets and TCP retransmits.

Stateless protocols (for example UDP) does not have such problems.

We will improve crypto driver for TCP connections in the future.

SystemErrorMessage · Tue Nov 04, 2014 10:06 pm

The problem with the CCR performance you are experiencing is because you are using IPSEC and GRE tunnel with only 1 tunnel to 1 other host. From the design of routerOS on this it will not use more than 1 or 2 cores for this.

You can boost the performance by using multiple tunnels to the same host and setting up multiple load balancing. For example if you tried mikrotik btest server you will find that it wont use more than 1 core but if you run multiple sessions of it you can get 100% CPU usage with many many sessions from the same computer and using 64 byte packets.

Arjen · Wed Nov 05, 2014 2:53 pm

The problem is only with TCP traffic because of fragmentation, out-of-order packets and TCP retransmits.

Stateless protocols (for example UDP) does not have such problems.

We will improve crypto driver for TCP connections in the future.

This really sounds so unbelievable to me. It's like a car manufacturer explaining their customers that this specific car only tops out at 15km/h on tarmac roads and they're working on the issue and fix it "somewhere" in the future. But offroad the car is fine. Like people use their car to go to work "offroad" each day.

To be honest, everyone buying these CCR's are using it for TCP. This bottleneck is really critical and should be fixed asap. I cannot digest that I just bought a CCR for a pretty steep price, not being able to handle TCP traffic properly on some ipsec tunnel.

I guess, your credibility is at stake here.

i4jordan · Wed Nov 05, 2014 4:07 pm

@mrz

I am very surprised with you explanation.
We bought several CCR models to be able to handle ipsec VPN tunnels at high speed. Also because of the hardware AES support.
We build VPN networks for our customers. That's our job.

Now you are telling us that the ipsec speed problems (which are mentioned a lot on this forum) will be fixed in the future???
Please make fast ipsec tunnel and transport handling top priority.

We are going to consider other brands for VPN if Mikrotik is not going to improve.

SystemErrorMessage · Wed Nov 05, 2014 6:49 pm

It is very difficult to find other vendors selling TILE. For now you should try my advice on having multiple tunnels. CPUs like TILE and GPUs are something new to mikrotik so there are bound to be difficulties making full use of it.

Wed Nov 05, 2014 6:53 pm

If this really just a fragmentation-related issue, wouldn't explicit MSS clamping solve the issue? Just wondering.
I personally don't have access to any of the CCRs, unfortunately, otherwise I would have tested it myself.

SystemErrorMessage · Wed Nov 05, 2014 7:12 pm

Its ok, these OEM users dont seem to want to listen to people who use mikrotik hardware in their hacking activities or even experienced and expert users in ways to overcome or get around the problem. With that attitude I'm just glad im not their customers for the services they offer.

Edit: on the history of the VLIW architecture there have been problems making full use of it as evident on the AMD 5xxx and 6xxx series GPUs.

roadracer96 · Thu Nov 06, 2014 7:45 pm

It is very difficult to find other vendors selling TILE. For now you should try my advice on having multiple tunnels. CPUs like TILE and GPUs are something new to mikrotik so there are bound to be difficulties making full use of it.

Yeah. Because I really want to go from 10 tunnels to 40 just to get the throughput I need. Talk about network diagram nightmares.

Arjen · Sat Nov 08, 2014 1:21 pm

If this really just a fragmentation-related issue, wouldn't explicit MSS clamping solve the issue? Just wondering.
I personally don't have access to any of the CCRs, unfortunately, otherwise I would have tested it myself.

It's not "just" fragmentation. I'm using MSS clamping, throughput is still pretty poor.
I do believe it has to do something with the offloading, because I don't see any CPU spikes on the Tilera CPU.

i4jordan · Thu Dec 11, 2014 3:21 pm

It looks like Mikrotik has finally improved VPN performance in CCR models.

Here is the changelog for 6.24rc2
----
What's new in 6.24rc2 (2014-Dec-10 11:04):

*) fixed problem where some of ethernet cards do not work on x86;
*) improved CCR ethernet driver (less dropped packets);
*) improved queue tree parent=global performance (especially on SMP systems and CCRs);
*) eoip/eoipv6/gre/gre6/ipip/ipipv6/6to4 tunnels have improved per core balancing on CCRs;
*) fixed tx for 6to4 tunnels with unspecified dst address;
*) fixed vrrp - could sometimes not work properly because of advertising bad set of ip addresses;
----

alexjhart · Thu Dec 17, 2015 1:05 am

My testing started at 6.33.3. While others may be reporting performance improvements from earlier versions, I find the hardware encryption quality with IPSEC on CCR1036 dissatisfying. Testing with everything identical except setting and unsetting ipsec-secret in GRE config (enabling and disabling encryption) shows night and day difference in connection quality (TCP retransmission, out of order packets, packet loss, etc). Unfortunately, many applications are sensitive to this. For example, my SMB tests dropped by about 10x (went from 250Mbps/450Mbps down/up to 25/45Mbps). While software encryption improves the latter SMB numbers by about 4x (not seeing the same retransmissions, out of order, loss, etc as before), that also means that I no longer benefit from parallel streams (now limited by single CPUs software encryption abilities). That tells me the single core in the hardware encryption isn't the limitation (since it still benefits from parallel streams), rather there are issues with quality as already mentioned. It has been over 1 year since that 6.24rc2 release. I hope that doesn't mean work to fix this has ceased.

I also noticed in-state-sequence-errors under /ip ipsec statistics increasing when quality is poor.

mikruser · Fri Dec 18, 2015 10:46 am

This only "GRE Tunnel + IPsec" problems? Or "IP Tunnel + IPsec" also have problems?

Zorro · Fri Dec 18, 2015 8:36 pm

its consist 3 bottlenecking things:
1. "general" ROS multi-core scalability. whcih is slowly, but improved from versions to versions.
2. lack of truly-scalable cihper modes.
3. scalabe chipers itself.

for example Serpent(https://en.wikipedia.org/wiki/Serpent_%28cipher%29) in EAX mode (https://en.wikipedia.org/wiki/EAX_mode ) was Wastly more scalable in multi and Many -core environment than Rinjael/AES in CBS, XTS and GCM modes (EAX had bit more overhead as 2-p things, than say CCM, but it SCALABLE !!! and benefits from both better hardware and software fine-tuning, while more ancent things - are NOT)

mikruser · Sat Dec 19, 2015 12:39 am

AES-GCM should be enough
https://en.wikipedia.org/wiki/Galois/Counter_Mode

Zorro · Sun Dec 20, 2015 8:22 pm

AES-GCM should be enough
https://en.wikipedia.org/wiki/Galois/Counter_Mode

for "multi-core"(ie with strea/core-count equal to 4x and below) - maybe.
but for many-core environment, including most of tilera chips for example - its simply both waste lot of resources and DO NOT scale well.
there isn't any "should be" in engineering or science.
there only "can do" and relevant "know how" of Good engineers and "can't do that, Dave" excuses of Bad ones.
and thats basically all. among not-technical aspects of it.
and in my opinion OCB, CWC, EAX support essential in both performance, scalability, securty as GCM support for similar reasons are(compared to CCM and more ancient modes)
btw there also was very funny and interesting GCM fork aswell, named SGCM https://en.wikipedia.org/wiki/Sophie_Ge ... unter_Mode which may be handy in networking for obvious reasons/advantages over generic/original GCM and Considerably boost its securty too.
but so far i think CWC is Most interesting candidate to start such work with(more clear benefits, less legal obstacles, documented src-code, etc).

artemlight · Sat Dec 26, 2015 12:03 am

+1 for massive packet reordering in simpe IPSEC+GRE case.

Bug can be easily reproduced on 2 CCRs connected by typical 5..15Mb WAN.
iperf3 in UDP mode reporting out of order packets even on manual bandwidth regulation (10-30% of actual link bandwidth)

[root@gw-sev.bigcar.local ~]# iperf3 -c 192.168.2.2 -b 1M  -u -V
iperf 3.0.11
Linux gw-sev.bigcar.local 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Time: Fri, 25 Dec 2015 21:51:28 GMT
Connecting to host 192.168.2.2, port 5201
      Cookie: gw-sev.bigcar.local.1451080288.34838
[  4] local 10.1.0.7 port 45391 connected to 192.168.2.2 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   112 KBytes   917 Kbits/sec  14  
[  4]   1.00-2.00   sec   120 KBytes   983 Kbits/sec  15  
[  4]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec  16  
[  4]   3.00-4.00   sec   120 KBytes   983 Kbits/sec  15  
[  4]   4.00-5.00   sec   120 KBytes   983 Kbits/sec  15  
[  4]   5.00-6.00   sec   128 KBytes  1.05 Mbits/sec  16  
[  4]   6.00-7.00   sec   120 KBytes   983 Kbits/sec  15  
[  4]   7.00-8.00   sec   120 KBytes   983 Kbits/sec  15  
[  4]   8.00-9.00   sec   120 KBytes   983 Kbits/sec  15  
[  4]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec  16  
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.19 MBytes   996 Kbits/sec  0.833 ms  42/152 (28%)  
[  4] Sent 152 datagrams
CPU Utilization: local/sender 0.4% (0.0%u/0.4%s), remote/receiver 0.0% (0.0%u/0.0%s)

(iperf client counts reordered packets as lost)

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.1.0.7, port 46784
[  5] local 192.168.2.2 port 5201 connected to 10.1.0.7 port 45391
iperf3: OUT OF ORDER - incoming packet = 3 and received packet = 4 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 6 and received packet = 7 AND SP = 5
....
iperf3: OUT OF ORDER - incoming packet = 148 and received packet = 149 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 151 and received packet = 152 AND SP = 5
[  5]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec  0.833 ms  5/16 (31%)  
[  5]  10.00-10.09  sec  0.00 Bytes  0.00 bits/sec  0.833 ms  0/0 (-nan%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-10.09  sec  1.19 MBytes   987 Kbits/sec  0.833 ms  42/152 (28%)  
[SUM]  0.0-10.1 sec  42 datagrams received out-of-order
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

And this is only 1Mbit between 2 CCRs.

IPSec is working stable - UDP flooding over IPSEC always utilise all bandwidth. Statistics is almost clean:

[artem@mow01.r.severscan.ru] > /ip ipsec statistics print 
                  in-errors: 0
           in-buffer-errors: 0
           in-header-errors: 0
               in-no-states: 60
   in-state-protocol-errors: 416
       in-state-mode-errors: 0
   in-state-sequence-errors: 0
           in-state-expired: 0
        in-state-mismatches: 0
           in-state-invalid: 49
     in-template-mismatches: 0
             in-no-policies: 0
          in-policy-blocked: 0
           in-policy-errors: 0
                 out-errors: 0
          out-bundle-errors: 0
    out-bundle-check-errors: 0
              out-no-states: 594195
  out-state-protocol-errors: 4242
      out-state-mode-errors: 0
  out-state-sequence-errors: 0
          out-state-expired: 4242
         out-policy-blocked: 0
            out-policy-dead: 0
          out-policy-errors: 0

All routerboards are flashed with latest firmware

[artem@noz01.r.severscan.ru] > /system package print 
Flags: X - disabled 
 #   NAME                     VERSION                     SCHEDULED              
 0   routeros-tile            6.33                                               
 1   system                   6.33                                               
 2 X wireless-cm2             6.33                                               
 3 X ipv6                     6.33                                               
 4   wireless-fp              6.33                                               
 5   hotspot                  6.33                                               
 6   dhcp                     6.33                                               
 7   mpls                     6.33                                               
 8   routing                  6.33                                               
 9   ppp                      6.33                                               
10   security                 6.33                                               
11   advanced-tools           6.33  

[artem@noz01.r.severscan.ru] > /system routerboard print 
       routerboard: yes
             model: CCR1009-8G-1S
     serial-number: *************
     firmware-type: tilegx
  current-firmware: 3.27
  upgrade-firmware: 3.27

Mon Dec 28, 2015 10:37 am

@artemlight Run the same setup between directly connected CCRs and see if you still have the same problem

artemlight · Tue Dec 29, 2015 10:23 am

Sorry, but I do not have another CCR to connect it directly.

artemlight · Sat Jan 09, 2016 8:24 pm

Solved for me by switching to AES-GCM algo.

ucs75 · Sun Sep 11, 2016 4:55 am

Could you please post a sample config with GCM? There's clearly more to it than just changing the encryption algorithm in the proposal. I wasn't able to achieve a tunnel after updating the proposal. What else needs to be set?!

Mon Sep 12, 2016 11:23 am

You just need to remove auth algorithm and set enc algorithm to gcm.

mikruser · Fri Mar 22, 2019 12:08 pm

GRE+IPsec still slow:
viewtopic.php?f=2&t=146665

Who is online