Hello, I am trying to enable Layer 3 hardware offloading, or at least FastTrack hardware offloading, on a CCR2216. Even trying just to offload FastTrack connections the NAT usage is 0 and the counter fasttrack ipv4 connections is 0. Below I attach my configuration (I tried to sanitize it). Thank you in advance for any help 
I can provide more details, if necessary. (I hope the format is ok, this is my first time here).
Device
Model: CCR2216-1G-12XS-2XQ
RouterOS: 7.23.1
Switch chip: Marvell Prestera 98DX8525
Network layout
The router has:
-
one 802.3ad bonding interface toward upstream A;
-
one tagged VLAN on a QSFP28 port toward upstream B/IX;
-
several routed VLANs toward downstream routers;
-
two physical downstream ports in a bridge;
-
two EoIP interfaces in the same bridge;
-
IPv4 BGP routes from multiple upstreams;
-
approximately 97,000 IPv4 routes;
-
firewall, connection tracking and several SRC-NAT, DST-NAT and NETMAP rules.
The desired behaviour is:
-
all new connections must initially pass through the CPU firewall and NAT;
-
established FastTrack connections should be hardware offloaded;
-
internal routed traffic may be fully L3 hardware offloaded where possible;
-
upstream traffic must not bypass firewall or NAT.
Current symptoms
The global switch setting is enabled:
/interface ethernet switch
set 0 l3-hw-offloading=yes
All physical switch ports currently have:
l3-hw-offloading=no
The FastTrack firewall rule is enabled, but normally:
fasttrack-ipv4-conns: 0
nat-usage: 0
No useful hardware-offloaded route or connection counters are observed.
On some occasions FastTrack IPv4 connections unexpectedly appear. During one connectivity failure toward a downstream router, the advanced L3HW monitor showed a positivenat-usage value. After connectivity recovered, all FastTrack and NAT hardware counters returned to zero.
Enabling L3HW on selected ports did not produce an improvement.
Sanitized relevant configuration
Bridge
/interface bridge
add name=Routers port-cost-mode=short
add name=loopback port-cost-mode=short protocol-mode=none
The Routers bridge currently uses the default:
vlan-filtering=no
There is no /interface bridge vlan table.
Physical interfaces
/interface ethernet
set [find default-name=qsfp28-1-1] \
comment="UPSTREAM-B-IX" \
auto-negotiation=no \
speed=100G-baseSR4-LR4
set [find default-name=sfp28-1] comment="UPSTREAM-A-LACP-1"
set [find default-name=sfp28-2] comment="UPSTREAM-A-LACP-2"
set [find default-name=sfp28-3] comment="DOWNSTREAM-VPN"
set [find default-name=sfp28-5] comment="DOWNSTREAM-1"
set [find default-name=sfp28-7] comment="DOWNSTREAM-2"
set [find default-name=sfp28-6] disabled=yes
set [find default-name=sfp28-8] disabled=yes
Bonding
/interface bonding
add name=bonding1 \
mode=802.3ad \
slaves=sfp28-1,sfp28-2 \
transmit-hash-policy=layer-3-and-4
The bonding interface has a routed point-to-point public subnet and an eBGP session.
VLAN interfaces
/interface vlan
add interface=qsfp28-1-1 \
name=vlan4-UPSTREAM-B \
vlan-id=4
add interface=Routers \
name=vlan254-MANAGEMENT \
vlan-id=254
add interface=Routers \
name=vlan255-MANAGEMENT \
vlan-id=255
add interface=Routers \
name=vlan4000-DOWNSTREAM-1 \
vlan-id=4000 \
mtu=1472
add interface=Routers \
name=vlan4002-DOWNSTREAM-2 \
vlan-id=4002 \
mtu=1472
add interface=sfp28-3 \
name=vlan4005-DOWNSTREAM-VPN \
vlan-id=4005
The upstream VLAN is created directly on the QSFP28 physical interface. The VPN VLAN is also created directly on a physical interface.
Switch and per-port L3HW configuration
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set [find] l3-hw-offloading=no
Expanded current state:
/interface ethernet switch port
set qsfp28-1-1 l3-hw-offloading=no
set qsfp28-1-2 l3-hw-offloading=no
set qsfp28-1-3 l3-hw-offloading=no
set qsfp28-1-4 l3-hw-offloading=no
set qsfp28-2-1 l3-hw-offloading=no
set qsfp28-2-2 l3-hw-offloading=no
set qsfp28-2-3 l3-hw-offloading=no
set qsfp28-2-4 l3-hw-offloading=no
set sfp28-1 l3-hw-offloading=no
set sfp28-2 l3-hw-offloading=no
set sfp28-3 l3-hw-offloading=no
set sfp28-4 l3-hw-offloading=no
set sfp28-5 l3-hw-offloading=no
set sfp28-6 l3-hw-offloading=no
set sfp28-7 l3-hw-offloading=no
set sfp28-8 l3-hw-offloading=no
set sfp28-9 l3-hw-offloading=no
set sfp28-10 l3-hw-offloading=no
set sfp28-11 l3-hw-offloading=no
set sfp28-12 l3-hw-offloading=no
Bridge ports
/interface bridge port
add bridge=Routers interface=sfp28-5
add bridge=Routers interface=sfp28-7
add bridge=Routers \
interface=eoip-site-1
add bridge=Routers \
interface=eoip-site-2
add bridge=Routers \
interface=sfp28-6 \
disabled=yes
add bridge=Routers \
interface=sfp28-8 \
disabled=yes
The two EoIP interfaces are software tunnel interfaces.
Bridge filter
/interface bridge filter
add chain=forward \
action=drop \
in-interface-list=downstream-ports \
out-interface-list=downstream-ports
This rule prevents Layer 2 forwarding directly between downstream router ports.
Sanitized IP addressing
/ip address
add address=198.51.100.2/30 interface=bonding1
add address=192.0.2.91/24 interface=vlan4-UPSTREAM-B
add address=10.0.0.1/30 interface=vlan4000-DOWNSTREAM-1
add address=10.0.0.9/30 interface=vlan4002-DOWNSTREAM-2
add address=10.0.0.21/30 interface=vlan4005-DOWNSTREAM-VPN
add address=10.10.0.1/21 interface=vlan254-MANAGEMENT
add address=10.10.8.1/22 interface=vlan255-MANAGEMENT
add address=203.0.113.1/24 interface=loopback
IP settings
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set max-neighbor-entries=8192
Connection tracking
/ip firewall connection tracking
set tcp-established-timeout=45m \
udp-timeout=10s
FastTrack and firewall
The FastTrack rule is the first rule in the forward chain:
/ip firewall filter
add chain=forward \
action=fasttrack-connection \
connection-state=established,related \
hw-offload=yes
add chain=forward \
action=accept \
connection-state=established,related
add chain=forward \
action=drop \
src-address-list=blocked
add chain=forward \
action=drop \
dst-address-list=blocked
Input traffic is handled by ordinary stateful firewall rules.
Mangle
/ip firewall mangle
add chain=forward \
action=log \
connection-state=new \
src-address=10.0.0.0/8
add chain=postrouting \
action=change-mss \
protocol=tcp \
tcp-flags=syn \
new-mss=1432 \
src-address-list=adjust-mtu
Only new connections are logged by the first rule.
NAT
The actual configuration contains many NAT rules. These are representative examples:
/ip firewall nat
add chain=srcnat \
action=accept \
dst-address-list=no-nat-dst
add chain=srcnat \
action=netmap \
src-address=10.20.0.0/18 \
to-addresses=203.0.113.10
add chain=srcnat \
action=src-nat \
src-address=10.30.0.0/16 \
out-interface-list=upstreams \
to-addresses=203.0.113.20
add chain=srcnat \
action=masquerade \
src-address=10.40.0.0/16 \
out-interface-list=upstreams
add chain=dstnat \
action=dst-nat \
dst-address=203.0.113.30 \
protocol=tcp \
dst-port=443 \
to-addresses=10.50.0.10
The complete configuration contains:
-
many one-to-one NETMAP rules;
-
SRC-NAT;
-
masquerade;
-
DST-NAT;
-
ICMP destination NAT;
-
protocol-specific destination NAT.
Static routes
Static routes are not suppressed from hardware offloading:
/ip route
add dst-address=0.0.0.0/0 \
gateway=198.51.100.1 \
distance=30 \
suppress-hw-offload=no
add dst-address=203.0.113.0/24 \
blackhole \
suppress-hw-offload=no
add dst-address=10.60.0.0/16 \
gateway=10.0.0.22 \
suppress-hw-offload=no
Dynamic routing
The router has:
-
eBGP over
bonding1; -
eBGP over
vlan4-UPSTREAM-B; -
several BGP sessions toward downstream routers;
-
approximately 103,000 active IPv4 routes;
-
all routes in the main routing table;
-
no routing filter intentionally setting
suppress-hw-offload=yes.