I have CCR2116-12G-4S+. It is connected to a switch via single SFP+ port. I am trying to get Fasttrack connections offloaded to HW. I do not want full L3HW routing, just Fasttrack connections. I have a few VLANs and I see that Fasttrack works, but it does not get offloaded. I’m getting speeds around 3.4 Gbps when copying a large file from SSD to SSD from one VLAN to another (using 1 TCP connection). Just to test it out, I tried setting l3-hw-offloading=yes on the SFP+ port. This enabled full L3HW routing and same test showed speeds approaching 10 Gbps. In both cases l3-hw-offloading=yes was enabled for the switch. I have three bridges, but only one with physical port. To verify that Fasttrack connections are not offloaded I used L3HW monitor which always has fasttrack-ipv4-conns=0. Here is the relevant parts of the configuration (for the case when I try to offload only Fasttrack connections):
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge vlan-filtering=yes
/interface vlan
add interface=bridge name=vlan-lan vlan-id=101
add interface=bridge name=vlan-dmz vlan-id=203
# there are more VLANs configured, but I was testing with these two
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1
# bridge is automatically added as tagged member when creating vlan interfaces
/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus1 vlan-ids=101
add bridge=bridge tagged=sfp-sfpplus1 vlan-ids=203
/ip address
add address=10.1.1.1/24 interface=vlan-lan network=10.1.1.0
add address=10.2.3.1/24 interface=vlan-dmz network=10.2.3.0
/ip firewall filter
add action=fasttrack-connection chain=forward comment=\
"fasttrack established and related" connection-state=established,related \
hw-offload=yes
add action=accept chain=forward comment="forward established and related" \
connection-state=established,related
# there are more rules, but I see fastback counter increasing, indicating that it works
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
/interface ethernet switch
set 0 l3-hw-offloading=yes
# Not sure if following matters...
/interface bridge
add name=bridge-mdns protocol-mode=none
/interface macvlan
add interface=vlan-iot mode=private name=macvlan-iot
add interface=vlan-lan mode=private name=macvlan-lan
/interface bridge port
add bridge=bridge-mdns interface=macvlan-lan
add bridge=bridge-mdns interface=macvlan-iot
/interface bridge
add name=bridge-containers
/interface veth
add address=... dhcp=no gateway=... gateway6=... name=veth-dns
/interface bridge port
add bridge=bridge-containers interface=veth-dns
# I also have one extra VRF
/interface list
add name=allowed-vpn-out
/interface list member
add interface=vlan-vpn list=allowed-vpn-out
add interface=vpn-guest list=allowed-vpn-out
/ip vrf
add interfaces=allowed-vpn-out name=vrf-vpn
I’m not sure what am I doing wrong here. Is it even supposed to work? Since 7.2 bridges with enabled vlan filtering can be offloaded, but does it work for Fasttrack? Or for full L3HW routing only?
Read the official docs, especially the first section.
What you want is to turn on l3hw for the switch but turn off l3hw for the ports. This enables flow (fasttrack) offloading without doing full hw routing.
Well... take care of the usual suspects. Reboot (l3hw is sometimes sticky). Verify version. Verify that fasttrack is actually working without hw offload.
To be honest, the 3.4Gbps is really strange. Your speeds should be well beyond that even without either offload or fasttrack. I'd look into your testing methodology.
Thanks for the hints. I’ve tried rebooting - that did not help. I thought that maybe there are some bottlenecks when processing a single TCP connection? CPU load is < 10% during the test (with one core being at 100%, and a few more cores having some load).
The test I’m doing is pretty simple - copy a file via Samba from Linux machine to Windows machine (SSDs on both machines). Same test runs at ~10 Gbps when full L3HW routing is enabled with no CPU load at all. This suggests that test is capable of reaching this speed. I’m copying the same file every time, using the same machines.
I have checked the list of connections - the connection has F flag, which means it is Fasttracked. I can also observe fasttrack firewall rule counters go up during the file transfer.
EDIT: I have noticed that vlan 101 has MACVLAN interface. To eliminate the possibility of this affecting things, I have just tried with Windows machine in a different VLAN (201), which does not have MACVLAN interface. The results were even sadder - 2.8 Gbps with one core at 100%, other cores almost 0%. I guess it makes sense that it uses single core to process single TCP connection.
I agree that the Gbps number might be off. I can try running iperf3 once the Fasttrack offloading is resolved. My main problem here is that Fasttrack connections are not offloaded at all, not a single one. L3HW monitor always shows fasttrack-ipv4-conns: 0.
I’ve disabled RSTP on bridge, rebooted the router and ran the test. Same result, not a Fasttrack single connection is offloaded.
At this point I would really try different versions. Some things may have changed. Then contact support.
Again, are you sure that fasttrack is working? Just that the connection is marked F and the associated rule counter is incremented doesn't mean that the connection is fasttracked. This would be in line with your measured results. Are the actual fasttrack counters increasing?
I seem to recall a bug around this setup in older firmwares.
I think you might be onto something here… I’ve disabled Fasttrack firewall rule and observed the same speed. This time connection did not have the F flag. After that I have re-enabled the rule, repeated the test and still got the same speed, even though connection now had the F flag again. It would indeed seem that Fasttrack is not working at all, even though connections are marked as fast tracked.
I will try to run this with older versions of RouterOS.
One of the possible reasons for this behavior (connections are marked as fasttracked with the F flag, but the "dummy" dynamic rules added by fasttrack at the top of the firewall tables count nothing, and there is no speed improvement) is that somehow FastPath is no longer possible on the bridge. When the fasttracked connection is between the ports of the same bridge (not the usual case where the WAN port is outside of the bridge), fasttrack is only effective if bridge FastPath could be used on that bridge (if there was no firewall). It's the bolded condition below:
Requirements
FastTrack is active if the following conditions are met:
no mesh, metarouter interface configuration;
sniffer, torch, and traffic generator are not running;
"/tool mac-scan" is not actively used;
"/tool ip-scan" is not actively used;
FastPath and Route cache are enabled under IP/Settings (route cache condition does not apply to RouterOS v7 or newer);
bridge FastPath is enabled if a connection is going over the bridge interface;
When I experienced this "fasttrack ineffective" issue, it was due to DHCP Snooping being enabled for example.
Check if these conditions can be satisfied:
A Bridge handler is used if the following conditions are met:
there are no bridge Calea, filter, NAT rules;
use-ip-firewall is disabled;
no mesh, MetaRouter interface configuration;
sniffer, torch, and traffic generator are not running;
bridge vlan-filtering is disabled (condition is removed since RouterOS 7.2 version);
bridge dhcp-snooping is disabled.
FastPath on the vlan-filtering bridge does NOT support priority-tagged packets (packets with VLAN header but VLAN ID = 0). Those packets are redirected via a slow path.
The case for fasttrack being used for inter-vlan is special in the Mikrotik world. It has several gotchas and was broken and fixed several times. That's why I'm repeating for maybe the fifth time to please disclose the version.
I've seen cases where recreating the bridge was required for it to work. I've even had to completely nuke the config, after which it worked flawlessly.
In my case everything is on one physical port. I cannot do without VLANs. I understand that I cannot have FastPath with firewall. I’m only hoping for Fasttrack.
Initially I was using RouterOS v7.20.4. I have tried downgrading to v17.20, v7.19, v7.15.3 and v7.10. In all cases Fasttrack did not seem to work (even though F flag was present). For versions v7.15.3 and v7.10 speed was halved.
I have just realised that when downgrading I was not downgrading the RouterBOARD firmware, only the package versions. I guess I should try with downgraded firmware?
Yes, as I wrote it's normal that FastPath is not active when you have firewall rules. However, fasttrack is only effective if the conditions that allow FastPath to work are present (if you no longer have the firewall).
And that you have only one port is exactly the situation that everything is within the same bridge (same bridge but many VLANs).
Many people also had bridge that lost some of the FastPath conditions, but they still saw fasttrack "working" because they have the WAN ports outside of the bridge for example. In that case forwarding between LAN/VLAN (on bridge) and WAN (out of bridge) is still benefits from fasttrack.
But if the WAN was also on bridge (in a separate VLAN) which make LAN-WAN traffic also within the same bridge (is effectively inter-VLAN traffic) then they will see that fasttrack is not effective if the condition for FastPath cannot be satisfied.
The first quote which goes over the FastTrack conditions seems to be met by my configuration. I’m not sure I understand the last point though:
bridge FastPath is enabled if a connection is going over the bridge interface;
I have two VLAN interfaces with bridge as their parent interface, so it would seem that my connection is going over the bridge interface.
The second quote with FastPath requirements is not met, but that should be fine, right? As I understand it, I do not need to meet all FastPath conditions in order for FastTrack to work.
Would it be sufficient to wipe the configuration and then use the exported script to restore it? This should avoid any possible binary artefacts, right? If not, what would you suggest to try?
That, after the connection has been marked with fasttrack, it will be sent using FastPath. If the packets only travel within the same bridge (can be between different VLANs of the same bridge), then it will use the bridge FastPath handler. So fasttrack is effective in that situation only if the bridge FastPath handler can work in the same conditions.
In the post above I quoted the conditions for the bridge FastPath handler to be usable.
Thanks for the detailed reply. I’ll try to digest it… Is the following correct?
When one interface is in the bridge and another interface is not - FastTrack can work even if not all FastPath conditions are met (this is often the case for LAN/WAN interfaces).
When both interfaces are in the same bridge FastTrack can only work if all FastPath conditions are met (this is often the case for different LAN VLANs).
If my understanding is correct - this means that inter-VLAN FastTrack within a single bridge is impossible because one of the FastPath conditions, disabled vlan filtering on the bridge, cannot be met. Do you agree with this conclusion?
P.S. For older devices we used to have bridge per VLAN, but nowadays Mikrotik suggests having one bridge with vlan filtering enabled. Moreover, having multiple bridges might not work with L3HW offload anyway. So the old way of configuring VLANs might not help.
No because if you read the docs above, this condition (bridge VLAN filtering not enabled) has been removed since 7.2.
When Bridge VLAN Filtering is active FastPath is now supported, but with the warning at the bottom
FastPath on the vlan-filtering bridge does NOT support priority-tagged packets (packets with VLAN header but VLAN ID = 0). Those packets are redirected via a slow path.
