poor ipsec performance on ccr with aws vpn

RouterOS General
1

Few days ago we set up a vpn connection to aws vpc.
On our site we use ccr1009-8g-1s-1s+ an followed the example of jimr
http://forum.mikrotik.com/t/amazon-aws-vpn-a-working-configuration-example-and-bug/79770/1 without using nat.

After we script around the mikrotik policy issue the connection seems to be useable for a while.
Yesterday we started testing for throughput on this link and the result was really poor.
We got only 80mbps on this link.

Whats wrong here?

Config look like this:

/interface ethernet
set [ find default-name=ether8 ] name=ether8_uplink

/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc lifetime=8h

/routing bgp instance
add as=65000 client-to-client-reflection=no name=vgw1 redistribute-static=yes router-id=169.254.237.18
add as=65000 client-to-client-reflection=no name=vgw2 redistribute-static=yes router-id=169.254.237.22

/ip address
add address=xx.xx.xx.xx/30 interface=ether8_uplink network=xx.xx.xx.xx
add address=169.254.237.18/30 interface=ether8_uplink network=169.254.237.16
add address=169.254.237.22/30 interface=ether8_uplink network=169.254.237.20

/ip ipsec peer
add address=yy.yy.yy.yy/32 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 lifetime=1h nat-traversal=no secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
add address=yy.yy.yy.xx/32 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 lifetime=1h nat-traversal=no secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

/ip ipsec policy
add dst-address=169.254.237.18/32 sa-dst-address=yy.yy.yy.yy sa-src-address=xx.xx.xx.xx src-address=169.254.237.17/32 tunnel=yes
add dst-address=169.254.237.17/32 sa-dst-address=yy.yy.yy.yy sa-src-address=xx.xx.xx.xx src-address=169.254.237.18/32 tunnel=yes
add dst-address=169.254.237.22/32 sa-dst-address=yy.yy.yy.xx sa-src-address=xx.xx.xx.xx src-address=169.254.237.21/32 tunnel=yes
add dst-address=169.254.237.21/32 sa-dst-address=yy.yy.yy.xx sa-src-address=xx.xx.xx.xx src-address=169.254.237.22/32 tunnel=yes
add dst-address=172.31.0.0/16 sa-dst-address=yy.yy.yy.xx sa-src-address=xx.xx.xx.xx src-address=192.168.88.0/24 tunnel=yes

/routing bgp network
add network=192.168.88.0/24

/routing bgp peer
add hold-time=10s in-filter=vgw1 instance=vgw1 name=vpc1 out-filter=vgw1 remote-address=169.254.237.17 remote-as=7224 ttl=default update-source=169.254.237.18
add hold-time=10s in-filter=vgw2 instance=vgw2 name=vpc2 out-filter=vgw2 remote-address=169.254.237.21 remote-as=7224 ttl=default update-source=169.254.237.22

/routing filter
add chain=vgw1 set-distance=20
add chain=vgw2 set-distance=10

With another vendor appliance we got higher throughput results but less stability.

2

RouterOS and firmware versions?

Have you run Tools > Profile for CPU and resource usage when doing the tests?

Are there any queues or mangle in that router?

3

Highest core usage rate is 5% while running tests.

Config on ccr is very basic to prevent side-effects and simplify debugging.

/system resource print
             uptime: 3d3h21m52s
            version: 6.33.1 (stable)
         build-time: Nov/17/2015 09:55:23
        free-memory: 1747.9MiB
       total-memory: 1956.2MiB
                cpu: tilegx
          cpu-count: 9
      cpu-frequency: 1000MHz
           cpu-load: 1%
     free-hdd-space: 79.0MiB
    total-hdd-space: 128.0MiB
  architecture-name: tile
         board-name: CCR1009-8G-1S-1S+
           platform: MikroTik




/system routerboard print
                     routerboard: yes
             model: CCR1009-8G-1S-1S+
     serial-number: 
     firmware-type: tilegx
  current-firmware: 3.27
  upgrade-firmware: 3.27




[dne@vpn-gate-1] > /queue export
# nov/27/2015 11:11:50 by RouterOS 6.33.1
# software id = NH0A-MNRS
#
[dne@vpn-gate-1] > /ip firewall mangle export
# nov/27/2015 11:11:58 by RouterOS 6.33.1
# software id = NH0A-MNRS
#
4

What kind of traffic? What packet size?

5

We used tcp traffic generated by netio and iperf.

dne@xxx:~/netio/bin$ ./linux-x86_64 -s

NETIO - Network Throughput Benchmark, Version 1.31
(C) 1997-2010 Kai Uwe Rommel

UDP server listening.
TCP server listening.
TCP connection established ...
Receiving from client, packet size  1k ...  9116.44 KByte/s
Sending to client, packet size  1k ...  7694.83 KByte/s
Receiving from client, packet size  2k ...  9058.97 KByte/s
Sending to client, packet size  2k ...  7827.14 KByte/s
Receiving from client, packet size  4k ...  8145.50 KByte/s
Sending to client, packet size  4k ...  7817.76 KByte/s
Receiving from client, packet size  8k ...  8898.51 KByte/s
Sending to client, packet size  8k ...  7899.10 KByte/s
Receiving from client, packet size 16k ...  8588.56 KByte/s
Sending to client, packet size 16k ...  7376.13 KByte/s
Receiving from client, packet size 32k ...  8714.80 KByte/s
Sending to client, packet size 32k ...  7822.97 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size 64 ...  8076.24 KByte/s
Sending to client, packet size 64 ...  7912.76 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size 128 ...  9060.91 KByte/s
Sending to client, packet size 128 ...  7719.30 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size 256 ...  8312.16 KByte/s
Sending to client, packet size 256 ...  5824.54 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size 512 ...  8834.41 KByte/s
Sending to client, packet size 512 ...  7961.55 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size  1k ...  8579.08 KByte/s
Sending to client, packet size  1k ...  7761.30 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size 1460 ...  8563.63 KByte/s
Sending to client, packet size 1460 ...  8163.70 KByte/s
Done.
TCP server listening.
TCP connection established ...
Receiving from client, packet size 1500 ...  8807.63 KByte/s
Sending to client, packet size 1500 ...  7358.85 KByte/s
Done.




dne@xxx:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.88.111 port 5001 connected with 172.31.22.165 port 39198
------------------------------------------------------------
Client connecting to 172.31.22.165, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  6] local 192.168.88.111 port 43887 connected with 172.31.22.165 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  40.1 MBytes  33.5 Mbits/sec
[  4]  0.0-10.1 sec  44.6 MBytes  37.2 Mbits/sec
[  5] local 192.168.88.111 port 5001 connected with 172.31.22.165 port 39199
[  5]  0.0-10.1 sec  86.4 MBytes  71.8 Mbits/sec

edit:
Instance (m4.10xlarge) at aws site is connected via 10gbit to VPC and on local via 1gbit.
Uplink for ccr on datacenter site is 1gbit dedicated for this test.

6

No matter what packet size you use kbps are the same, it looks like there are some traffic shaping.

7

There is no shaping on uplinks to our ISP.

8

We were able to get 1.7 Gbps of IPSEC on a 1500 byte MTU between two CCR1036 routers in our lab. Sounds like you may have other factors in the transport or the AWS endpoint that may be limiting you.

http://www.stubarea51.net/2015/10/16/10-gbps-of-layer-2-throughput-is-possible-using-mikrotiks-eoip-tunnel/

9

In cooperation with our ISP we try a few other upstreams with no improvement.

Throughput on CCR is only 60% compared to a Watchguard xtm330 with very limited ipsec capabilities.

@IPANetEngineer:
I think AWS is definitely not using CCR for their VPN service.
So it´s not really comparable to your setup.

10

Can you provide details on how you are testing? While I am able to saturate the link with testing traffic, performance is greatly impacted by the connection quality (details here). This is also witnessed by iperf3 test traffic, but is easier to overcome with multiple parallel streams (and acceptance of loss). Wondering if you were only testing TCP and disregarded retransmissions and loss.