Community discussions

MikroTik App
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Sat Jan 16, 2016 7:05 pm

EoIP packet reordering with IPSec - load balancing across cores per packet vs per flow

Wed Apr 06, 2016 4:27 am

Note: Post edited to confirm that it only seems to happen with EoIP and IPSec. It doesn't happen without the IPSec secret enabled.
Hi,

On a CCR1009-8G-1S-1S+ running 6.34.2: I have recently discovered a significant amount of packet reordering when using EoIP with an ipsec secret. I believe packets are being distributed across the cores for crypto per packet (vs per flow) with UDP and ICMP and possibly TCP (unconfirmed). I believe this is what creates the packet reordering when there is a bursty flow.

Has anyone else run into this?

Is there any way to change the hashing to have these packets land on a single core to prevent the reordering?

This can be demonstrated by creating an EoIP connection between a clean path and then running a preloaded ping:
# ping -l 10 192.168.55.2 
PING 192.168.55.2 (192.168.55.2) 56(84) bytes of data.
64 bytes from 192.168.55.2: icmp_seq=4 ttl=64 time=23.9 ms
64 bytes from 192.168.55.2: icmp_seq=5 ttl=64 time=23.9 ms
64 bytes from 192.168.55.2: icmp_seq=7 ttl=64 time=23.9 ms
64 bytes from 192.168.55.2: icmp_seq=8 ttl=64 time=24.0 ms
64 bytes from 192.168.55.2: icmp_seq=3 ttl=64 time=24.0 ms
64 bytes from 192.168.55.2: icmp_seq=6 ttl=64 time=24.0 ms
64 bytes from 192.168.55.2: icmp_seq=1 ttl=64 time=24.0 ms
64 bytes from 192.168.55.2: icmp_seq=10 ttl=64 time=24.0 ms
64 bytes from 192.168.55.2: icmp_seq=2 ttl=64 time=24.0 ms
64 bytes from 192.168.55.2: icmp_seq=9 ttl=64 time=24.0 ms

--- 192.168.55.2 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 23.928/24.032/24.082/0.146 ms, pipe 10
21:23:18.234793 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 2, length 64
21:23:18.234809 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 2, length 64
21:23:18.234834 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 4, length 64
21:23:18.234841 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 4, length 64
21:23:18.234845 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 6, length 64
21:23:18.234848 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 6, length 64
21:23:18.234858 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 5, length 64
21:23:18.234862 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 5, length 64
21:23:18.234865 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 7, length 64
21:23:18.234868 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 7, length 64
21:23:18.234870 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 8, length 64
21:23:18.234874 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 8, length 64
21:23:18.234888 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 3, length 64
21:23:18.234893 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 3, length 64
21:23:18.234895 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 1, length 64
21:23:18.234899 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 1, length 64
21:23:18.234902 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 10, length 64
21:23:18.234906 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 10, length 64
21:23:18.234908 IP 192.168.55.1 > 192.168.55.2: ICMP echo request, id 44608, seq 9, length 64
21:23:18.234912 IP 192.168.55.2 > 192.168.55.1: ICMP echo reply, id 44608, seq 9, length 64
 
nathan1
Member Candidate
Member Candidate
Topic Author
Posts: 159
Joined: Sat Jan 16, 2016 7:05 pm

Re: EoIP packet reordering with IPSec - load balancing across cores per packet vs per flow

Fri Apr 08, 2016 3:02 pm

For anyone that might be running into this problem, MikroTik has confirmed the issue and I have been told: "We are trying to fix ipsec right now. It could be ready for next version or one after that." (TIcket #2016040766000158).
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 1885
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Sun Apr 10, 2016 12:36 pm

Nice one.

I'm glad you posted about this as I was just about to implement it.
 
alexjhart
Member Candidate
Member Candidate
Posts: 196
Joined: Thu Jan 20, 2011 8:03 pm

Re: EoIP packet reordering with IPSec - load balancing across cores per packet vs per flow

Mon Jun 13, 2016 7:27 am

-----
Alex Hart

The Brothers WISP

Who is online

Users browsing this forum: alltechhobies, Baidu [Spider], Google [Bot] and 81 guests