CCR2216 L3HW acceptable configuration combinations

Jakica · Tue Mar 25, 2025 1:43 pm

Hi,

Just a few days ago, we bought the CCR2216 and I am trying some configurations, but some things are not really clear to me.
We are trying to configure the router for full L3HW for some inter-VLAN routing and routing for our public IP pools (servers, special clients etc.) out to BGP peers. Besides that, we would like to have some VLANs that have some private IPs, and need to get NATed and then fasttracked to also use the L3HW

According to the documentation on the link:
https://help.mikrotik.com/docs/spaces/R ... onExamples
There are 2 lines in this documentation that say:

Traffic from/to the WAN port gets processed by the CPU/Firewall first. Then Fasttrack connections get offloaded to the hardware (Hardware-Accelerated L4 Stateful Firewall). NAT applies both on CPU- and HW-processed packets.

Packets get routed by the hardware only if both source and destination ports have l3-hw-offloading=yes. If at least one of them has l3-hw-offloading=no, packets will go through the CPU/Firewall while offloading only the Fasttrack connections.

So intuitively instead of switching off the L3HW on the WAN port (sfp28-1), i tried switching it off just on that specific LAN port (sfp28-8 - that carries that vlans with private IPs). Based on the text, the packets that go from/to that LAN port should go through the CPU, but that doesn't seem to be the case, as it doesn't work.

So then i played with the configuration a little bit more, and it seems the only configuration that works for the abovementioned use case is that the port that will be used for L3HW offloaded fasttracked NAT (sfp28-11), NEEDS to be removed completely from the bridge, and NEEDS to have its l3hw setting disabled in the switch->port menu.
The LAN port that is used for basic L3 inter-VLAN routing (sfp28-6) works as intended.

Here is the complete test config:

# 2025-03-25 12:31:58 by RouterOS 7.18.2
# model = CCR2216-1G-12XS-2XQ
/interface bridge
add name=mainBridge protocol-mode=mstp region-name=main vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp28-1 ] comment="WAN port"
set [ find default-name=sfp28-6 ] comment="inter-VLAN routing - works"
set [ find default-name=sfp28-8 ] comment=\
    "is in swtich- fasttrack does NOT work"
set [ find default-name=sfp28-11 ] comment=\
    "is NOT in switch - fasttrack works"
/interface vlan
add interface=mainBridge name=vlan240WAN vlan-id=240
add interface=mainBridge name="vlan1000LAN-intraVLAN routing" vlan-id=1000
add interface=mainBridge name="vlan2000-doesn't work fasttrack" vlan-id=2000

#ports sfp28-8 and sfp28-11 have disabled L3HW
/interface ethernet switch port
set 15 l3-hw-offloading=no
set 18 l3-hw-offloading=no
/ip pool
add name=dhcp_pool0 ranges=192.168.213.2-192.168.213.254
add name=dhcp_pool1 ranges=192.168.216.2-192.168.216.254
add name=dhcp_pool2 ranges=192.168.200.2-192.168.200.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=sfp28-11 name=dhcp1
add address-pool=dhcp_pool1 interface="vlan1000LAN-intraVLAN routing" name=\
    dhcp2
add address-pool=dhcp_pool2 interface="vlan2000-doesn't work fasttrack" name=\
    dhcp3
/interface bridge port
add bridge=mainBridge interface=qsfp28-1-1
add bridge=mainBridge interface=qsfp28-2-1
add bridge=mainBridge interface=sfp28-1 pvid=240
add bridge=mainBridge interface=sfp28-2
add bridge=mainBridge interface=sfp28-3
add bridge=mainBridge interface=sfp28-4
add bridge=mainBridge interface=sfp28-5
add bridge=mainBridge interface=sfp28-6 pvid=1000
add bridge=mainBridge interface=sfp28-7
add bridge=mainBridge interface=sfp28-8 pvid=2000
add bridge=mainBridge interface=sfp28-9
add bridge=mainBridge interface=sfp28-10
add bridge=mainBridge disabled=yes interface=sfp28-11
add bridge=mainBridge interface=sfp28-12
/interface ethernet switch l3hw-settings
set autorestart=yes
/interface bridge vlan
add bridge=mainBridge tagged=mainBridge untagged=sfp28-1 vlan-ids=240
add bridge=mainBridge tagged=mainBridge untagged=sfp28-6 vlan-ids=1000
add bridge=mainBridge tagged=mainBridge untagged=sfp28-8 vlan-ids=2000
/interface ethernet switch
set 0 l3-hw-offloading=yes
/ip address
add address=10.0.240.85/24 interface=vlan240WAN network=10.0.240.0
add address=192.168.213.1/24 interface=sfp28-11 network=192.168.213.0
add address=192.168.216.1/24 interface="vlan1000LAN-intraVLAN routing" \
    network=192.168.216.0
add address=192.168.200.1/24 interface="vlan2000-doesn't work fasttrack" \
    network=192.168.200.0
/ip dhcp-server network
add address=192.168.200.0/24 dns-server=1.1.1.1,8.8.8.8 gateway=192.168.200.1
add address=192.168.213.0/24 dns-server=1.1.1.1,8.8.8.8 gateway=192.168.213.1
add address=192.168.216.0/24 dns-server=1.1.1.1,8.8.8.8 gateway=192.168.216.1
/ip dns
set servers=1.1.1.1,8.8.8.8
/ip firewall filter
add action=fasttrack-connection chain=forward hw-offload=yes out-interface=\
    vlan240WAN src-address=192.168.213.0/24
add action=fasttrack-connection chain=forward hw-offload=yes out-interface=\
    vlan240WAN src-address=192.168.200.0/24
/ip firewall nat
add action=src-nat chain=srcnat src-address=192.168.213.0/24 to-addresses=\
    10.0.240.85
add action=src-nat chain=srcnat src-address=192.168.200.0/24 to-addresses=\
    10.0.240.85
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.0.240.1 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10

Is this all there is to it? Or am I missing something important here, and there is some other way to do this configuration?

EDIT:
It seems that it is possible to remove WAN port (sfp28-1) from the bridge (and set the IP on the interface itself, not the VLAN), and the basic L3HW offloaded routing still works, and fasttracked connections that go out of WAN port still work. It seems the more important setting is l3hw checkbox on the port iself.

elcano89 · Tue Mar 25, 2025 8:04 pm

All ports need to be on the single l3hw bridge with vlan filtering, but those that do firewall functions need to be set to l3hw=disabled, as example item 8 (sfp28-1) here is doing firewalling:

/interface/ethernet/switch/port> pri
Flags: R - RUNNING
Columns: NAME, SWITCH, L3-HW-OFFLOADING, STORM-RATE
 #   NAME         SWITCH   L3-HW-OFFLOADING  STORM-RATE
 0   qsfp28-1-1   switch1  yes                      100
 1   qsfp28-1-2   switch1  yes                      100
 2   qsfp28-1-3   switch1  yes                      100
 3   qsfp28-1-4   switch1  yes                      100
 4   qsfp28-2-1   switch1  yes                      100
 5   qsfp28-2-2   switch1  yes                      100
 6   qsfp28-2-3   switch1  yes                      100
 7   qsfp28-2-4   switch1  yes                      100
 8 R sfp28-1      switch1  no                       100
 9   sfp28-2      switch1  yes                      100
10   sfp28-3      switch1  yes                      100
11   sfp28-4      switch1  yes                      100
12   sfp28-5      switch1  yes                      100
13   sfp28-6      switch1  yes                      100
14   sfp28-7      switch1  yes                      100
15 R sfp28-8      switch1  yes                      100
16 R sfp28-9      switch1  yes                      100
17 R sfp28-10     switch1  yes                      100
18 R sfp28-11     switch1  yes                      100
19 R sfp28-12     switch1  yes                      100
20 R switch1-cpu  switch1                           100

Jakica · Wed Mar 26, 2025 1:05 pm

All ports need to be on the single l3hw bridge with vlan filtering, but those that do firewall functions need to be set to l3hw=disabled...

I tried this, as you can see in my configuration, the port sfp28-8 has the l3-hw-offloading setting disabled, but if i don't remove it from the bridge, the firewall and NAT doesn't work. I need to remove it completely from the bridge and disable L3HW on it to work.
It is basically the reverse of the example in the documentation where they do this with the WAN port.

If I leave all ports in the bridge, the only configuration that works is if i disable L3HW on both sfp28-1 (BGP WAN) and sfp28-8 (LAN port that needs firewalling), but then I don't have l3hw on the basic routing between other ports and WAN port, because I have some public subnets i want to attach to other vlans and have them route on the hardware.

I need to test this a little bit more, because i have a gut feeling if i use tagged vlans on these ports that i throw out of the bridge, I won't have any offloading then.

sirbryan · Wed Mar 26, 2025 3:05 pm

If I leave all ports in the bridge, the only configuration that works is if i disable L3HW on both sfp28-1 (BGP WAN) and sfp28-8 (LAN port that needs firewalling), but then I don't have l3hw on the basic routing between other ports and WAN port, because I have some public subnets i want to attach to other vlans and have them route on the hardware.

I need to test this a little bit more, because i have a gut feeling if i use tagged vlans on these ports that i throw out of the bridge, I won't have any offloading then.

Yes, it gets tricky if you're trying to use a port to bridge VLANs through while also using a firewall or NAT for any VLAN (or untagged traffic) on the same physical interface. I eventually gave up with hardware-assisted firewall/NAT for that very reason.

Jakica · Wed Mar 26, 2025 3:23 pm

I didn't give up yet.

Just before 5 minutes i tried with switch ACL rules. In below examples it didn't matter if i enabled or disabled L3HW on the lan port.

In the 1st rule i redirected to CPU all traffic with source address of these private addresses (192.168.200.0/24) with fastrrack hw offload enabled. It seems this makes the download direction (from WAN to LAN) get hw offloaded through fasttrack, because the packet goes from the sfp28-8 port, it gets redirected to the CPU, then the CPU applies NAT, and the connection then goes to the fasttrack rule, which makes it hardware offloaded.
The problem is upload traffic, because it always hits the rule, and even though the process is the same and fasttrack creates the rule and it works, the CPU gets blasted with all those upload packets for nothing (If i make a speedtest with 1gb NIC in my PC, i see 2gb/s on the VLAN 240 when testing upload)

This is of course better performance wise, but I don't like the configuration at all, who knows what is happening in the hardware...

So back to square one...

Jakica · Wed Mar 26, 2025 4:40 pm

Okay, so after more testing, the closest I got to the desired configuration is by throwing out the sfp28-8 port (the LAN port that needs NAT) out of the bridge, and disabling the L3HW on it.
Now all other ports have basic L3HW, and sfp28-8 is fasttrack offloaded. Basically the reverse of the documentation example on the CRS317.

But the only problem is, it only works on the port itself. If i add a VLAN to sfp28-8 and try to use l3hw fasttrack offloaded tagged traffic, it doesn't work, all traffic goes through the CPU.

This is really stupid and I am starting to think this is a bug. The port should go through CPU if i disable L3HW on it while it is in bridge, thus enabling the NAT and firewall on it, and it should get NAT-ed even if the destination is in some of the vlans that are present on ports that have L3HW checked.
This basically limits us with one vlan per one physical port for fasttrack offload...

I had this setup for a while:

CCR1072(current production router) 10.0.240.1/24 sfp-sfpplus6 <-------> 10.0.240.85/24 sfp28-1 CCR2216 192.168.200.1/24 sfp28-8 <------------> 192.168.200.254/24 PC

The PC is on that sfp28-8 port that should get NATed and fasttracked. CCR2216 has static 0.0.0.0/0 route to 10.0.240.1. (As in config)
If i torch sfp-sfpplus6 port on the CCR1072, and then ping while L3HW is turned off for port sfp28-8 i get some interesting results:

Ping from PC to 192.168.200.1 (CCR2216): This works as expected, the CCR2216 replies
Ping from PC to 10.0.240.1 (CCR1072): This works for some reason as it should, the CCR1072 sees the 10.0.240.85 as source IP, i get a response.
Ping from PC to 1.1.1.1: This doesn't work, the CCR1072 sees private IP (192.168.200.254) as the source IP, and can't respond as it doesn't have the route.

How come if i ping the gateway, CCR2216 applies NAT, and if i ping anything else that uses default route, it doesn't work?

EDIT: I created a bug report, reference is #SUP-183864

N0rm0L · Tue Apr 01, 2025 11:30 am

Hello, no news on the problem??

CCR2216 L3HW acceptable configuration combinations

CCR2216 L3HW acceptable configuration combinations

Re: CCR2216 L3HW acceptable configuration combinations

Re: CCR2216 L3HW acceptable configuration combinations

Re: CCR2216 L3HW acceptable configuration combinations

Re: CCR2216 L3HW acceptable configuration combinations

Re: CCR2216 L3HW acceptable configuration combinations

Re: CCR2216 L3HW acceptable configuration combinations