+1 for massive packet reordering in simpe IPSEC+GRE case.
Bug can be easily reproduced on 2 CCRs connected by typical 5..15Mb WAN.
iperf3 in UDP mode reporting out of order packets even on manual bandwidth regulation (10-30% of actual link bandwidth)
[root@gw-sev.bigcar.local ~]# iperf3 -c 192.168.2.2 -b 1M -u -V
iperf 3.0.11
Linux gw-sev.bigcar.local 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Time: Fri, 25 Dec 2015 21:51:28 GMT
Connecting to host 192.168.2.2, port 5201
Cookie: gw-sev.bigcar.local.1451080288.34838
[ 4] local 10.1.0.7 port 45391 connected to 192.168.2.2 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 112 KBytes 917 Kbits/sec 14
[ 4] 1.00-2.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 3.00-4.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 4.00-5.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 6.00-7.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 7.00-8.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 8.00-9.00 sec 120 KBytes 983 Kbits/sec 15
[ 4] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 16
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 1.19 MBytes 996 Kbits/sec 0.833 ms 42/152 (28%)
[ 4] Sent 152 datagrams
CPU Utilization: local/sender 0.4% (0.0%u/0.4%s), remote/receiver 0.0% (0.0%u/0.0%s)
(iperf client counts reordered packets as lost)
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.1.0.7, port 46784
[ 5] local 192.168.2.2 port 5201 connected to 10.1.0.7 port 45391
iperf3: OUT OF ORDER - incoming packet = 3 and received packet = 4 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 6 and received packet = 7 AND SP = 5
....
iperf3: OUT OF ORDER - incoming packet = 148 and received packet = 149 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 151 and received packet = 152 AND SP = 5
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 0.833 ms 5/16 (31%)
[ 5] 10.00-10.09 sec 0.00 Bytes 0.00 bits/sec 0.833 ms 0/0 (-nan%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-10.09 sec 1.19 MBytes 987 Kbits/sec 0.833 ms 42/152 (28%)
[SUM] 0.0-10.1 sec 42 datagrams received out-of-order
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
And this is only 1Mbit between 2 CCRs.
IPSec is working stable - UDP flooding over IPSEC always utilise all bandwidth. Statistics is almost clean:
[artem@mow01.r.severscan.ru] > /ip ipsec statistics print
in-errors: 0
in-buffer-errors: 0
in-header-errors: 0
in-no-states: 60
in-state-protocol-errors: 416
in-state-mode-errors: 0
in-state-sequence-errors: 0
in-state-expired: 0
in-state-mismatches: 0
in-state-invalid: 49
in-template-mismatches: 0
in-no-policies: 0
in-policy-blocked: 0
in-policy-errors: 0
out-errors: 0
out-bundle-errors: 0
out-bundle-check-errors: 0
out-no-states: 594195
out-state-protocol-errors: 4242
out-state-mode-errors: 0
out-state-sequence-errors: 0
out-state-expired: 4242
out-policy-blocked: 0
out-policy-dead: 0
out-policy-errors: 0
All routerboards are flashed with latest firmware
[artem@noz01.r.severscan.ru] > /system package print
Flags: X - disabled
# NAME VERSION SCHEDULED
0 routeros-tile 6.33
1 system 6.33
2 X wireless-cm2 6.33
3 X ipv6 6.33
4 wireless-fp 6.33
5 hotspot 6.33
6 dhcp 6.33
7 mpls 6.33
8 routing 6.33
9 ppp 6.33
10 security 6.33
11 advanced-tools 6.33
[artem@noz01.r.severscan.ru] > /system routerboard print
routerboard: yes
model: CCR1009-8G-1S
serial-number: *************
firmware-type: tilegx
current-firmware: 3.27
upgrade-firmware: 3.27