Hi,
On our CCR2004-1G-12S+2XS (RouterOS 7.24.4, also tested on 7.22.2), every packet processed by the CPU (VXLAN, routing, or bridging with use-ip-firewall=yes) can leave the router out of order. It breaks our UDP game servers (RakNet treats reordered packets as lost, which causes desync).
Test: one UDP flow with sequence numbers, bursts of 20 x 1100 bytes every 20 ms (~9 Mbit/s). No loss, but out-of-order packets:
- VXLAN encapsulated by the CCR2004: 18-25 %
- VXLAN decapsulated by the CCR2004: 9-12 %
- plain routing through the CCR2004: 13 %
- bridge SFP+ to SFP+ with use-ip-firewall=yes: 9.6 %
- bridge hardware-switched (use-ip-firewall=no): 0.5 %
- same VXLAN config on a CCR1009 and on a CHR: 0 %
Root cause found with /tool sniffer (the "cpu" field of each packet): the single flow is split 50/50 between cpu1 and cpu2 (1969 / 1951 packets), alternating packet by packet:
0:cpu2 1:cpu1 2:cpu1 3:cpu2 4:cpu1 5:cpu2 7:cpu1 6:cpu2 ...
The sender is 100 % in order, the CCR2004 egress is already 18.7 % out of order. It looks like switch1 (Marvell 98PX1012) distributes one flow over both CPU uplinks (al-eth-switch0-0 / 0-1) and the two cores process it in parallel.
Already tried without effect: RPS on the port, single pfifo queue, vxlan hw=no, checksum=yes, VTEP outside VRF, traffic-flow off, other ingress port (ether1), other underlay (GRE), RouterOS 7.22.2 / 7.24.2 / 7.24.4. The al-eth-switch IRQs are read-only.
Has anyone seen this on the CCR2004-1G-12S+2XS? Is there any way to keep a flow on one core (flow-aware hashing to the CPU)?
Support ticket: SUP-XXXXX
Thanks!