CCR2116 L3HW Offloading Question

RouterOS General
1

I am replacing a CCR2004 with a CCR2116. Since the 2116 has L3HW offloading ability I am wanting to see if I can take advantage of that. The CCR2004 right now is a tower router for one of our ISP tower sites connecting around 250 customers.

My CCR2004 config is not made with L3HW offloading in mind so I have to modify the config. Doing a lab setup I am testing a simplified setup and have L3HW working but I am not sure if it is working by the book as my configuration might be non-standard and I want advice on if the way I have it setup is wrong and will lead to issues when scaled up and put in the field.

My lab setup is R1→CCR2116→Managed switch→R2.

R1 has VLAN interface on Ether9 of 1003. 10.10.97.61/30 is the IP on VLAN1003 interface.

CCR2116 ROS 7.21.2

  • L3 HW is enabled on CCR2116 switch ports
  • L3 HW disabled on CCR2116 switch
  • Bridge with ports Ether 12 and Ether 1
  • Bridge vlan filtering enabled
  • Ether 12 PVID - 1003
  • Ether 1 PVID - 22
  • Bridge has Vlan 1003 with bridge and ether 12 tagged
  • Bridge has vlan 22 with bridge and ether 1 tagged
  • Created Vlan interface Vlan1003 on bridge and enabled L3 HW Offloading
  • Put IP address 10.10.97.62/30 on Vlan1003
  • Created vlan interface Vlan22 on bridge and enabled L3 HW Offloading
  • Created default route of 0.0.0.0/0 w/gateway of 10.10.97.61
  • Put IP address 10.20.103.1/26 on Vlan22
  • Put IP address of 172.16.2.61/30 on Ether1 for switch management
  • Created IP pool and then DHCP server on Vlan22 interface
  • Managed Switch is connected to Ether1

Managed switch

  • Management IP is 172.16.2.62/30
  • Port 5 is Trunk port for vlan 22 and vlan 1
  • Port 1 is Access port for vlan 22

R2

  • WAN port set to DHCP to get IP address from CCR2116
  • WAN port connected to Port 1 (Access port) of Managed switch.

I have the above working at the IP level. R2 gets an IP address in the 10.20.103.x IP pool. R2 can ping and reach R1. R2 can do a Btest to R1. R1 can ping and access CCR2116. R1, R2 and CCR2116 can access management IP of managed switch.

Doing a Btest from R2 to R1 and pushing 1G TCP with L3 HW offload disabled on the CCR2116 switch CPU usage per core is 0-8% (usually only around 4 cores active) and Fastpath counters under IP→Settings are incrementing meaning the low CPU usage is due to fastpath.

If I enabled L3 HW offload CPU usage on all cores go to 0% with 1 core between 0-2% (I assume for non-routing processes). So this config seems to work right.

Now I disabled L3 offload added some FW rules to the system.
Input chain

  • Accept established/related
  • Accept port 80, 8291
  • Drop invalid
  • Drop all

Now when I did this fastpath stopped incrementing and CPU usage went to around 16-20% for 4 of the 16 cores. As expected as the CPU is now processing all the traffic because of those FW rules. FW Conn tracker shows 20 TCP connection between R2 and R1 for the Btest with the rate values per connection showing traffic flowing.

I then turned on L3 offload and all core CPU went back to 0% but fast path counters are not incrementing. However, the Input FW rules still function as expected and their counter increase appropriately. L3 HW Monitor shows 2 IPv4 hosts and only 1 IPv4 HW route. It shows 23 CPU routes. 0 Fasttrack anything.

IP→Setting shows 0 Fastpath and 0 Fasttrack traffic. FW Conn tracker shows a single TCP connection between R2 and R1 but Rate is 0bps.

Is that expected to work that way? Will this cause issues when there is normal customer traffic?

Another scenario:

I turn back off L3 offload and then add FW Forward rules along with the previous Input FW rules.

  • Fasttrack established/related
  • Accept established/related

CPU usage with fasttrack is lower than without fasttrack and fasttrack counters increment in IP-Settings.

Now if I keep the Forward FW rules and then enable L3 Offload CPU usage goes back to 0% on all cores. Forward FW rules counters stop increasing. Also under IP→settings Fastpath and Fasttrack counters do not increase but Fasttrack shows as enabled. FW Conn tracker only shows a single TCP connection from R2 to R2 with a rate of 0bps.

Under L3 Monitor Fasttrack section shows all 0 as well.

In this setup with L3 Offload enabled it seems the Forward rules do nothing and can be removed but the Input FW rules still work.

Is this a valid setup? I ask because the L3 HW offload help page many times makes statements that unexpected things can happen if you do not have it configured correctly.

2

Two things:

Do not put IP addresses directly on interfaces that are part of the bridge. Always put them on the bridge itself or VLAN interfaces associated with the bridge. For out-of-band management, use one of the Ethernet ports that’s not a part of the bridge.

HW offload for Firewall/NAT operations requires the upstream port (WAN) to not have its port enabled in the L3HW offload port settings. Only the inside (LAN) ports are to be enabled. The reason is the CPU has to see the packets so it knows what to accept/block and what to track. By offloading the WAN port, all the traffic stays in the ASIC(s) and bypasses the CPU, and you get the results you’re seeing.

If it’s a trunk port with multiple VLANs, such as a WAN VLAN and other internal VLANs, then the HW offload acts funny (or not at all). At least, that’s been my experience.

What should work is something like Ether2 as upstream port, assign an IP directly to the port, and disable L3HW offload in the switch just for that port, and leave it on for everything else.

3

I will try to post the current config/or screen shots once I am back in the office tomorrow.

I am not doing any NAT on the router. I really only need the FW for the Input chain and not the forward chain.

I do not do any Inter VLAN routing for the LAN as these are internet customers and their traffic should all be routed out the WAN port. No customer-to-customer direct traffic.

I currently have my lab setup working with OSPF between the CCR2116 and 2 other routers and L3 HW seems to be working correctly per CPU usage, but your reply makes me think I may run into issue in production with my current config?

The WAN port is part of the filtered bridge with the WAN IP on a VLAN interface attached to the bridge. The LAN ports are also on the bridge with IPs attached to other VLAN inferfaces attached to the bridge.

All Switch ports currently have L3 HW enabled and the Vlan interfaces also have L3 HW enabled. L3 HW is also enabled on the switch.

Input rules on the FW appear to be working as expected.

So you are saying I should instead put VLAN 1003 on the WAN port (ether 12), remove the WAN port (ether 12) from the bridge and also disable L3 HW on that port? Should I also disable L3 HW on the VLAN 1003 interface that would be attached to Ether 12 (WAN)?

In that setup would I then be relying on fasttrack to handle the L3 accelleration for traffic entering/existing the WAN? If so I know my current tracked connection count on my existing CCR2004 would exceed the 4.5k fasttrack limit of the CCR2116.

So basically I want as much L3 offloaded as possible while protecting the input chain with FW rules, working OSPF, and no oddities or acting funny (as you put it).

4

Here is my current test config.

/interface bridge
add name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] name=sfpplus1_BREMC
/interface vlan
add interface=bridge name=vlan22_APN vlan-id=22
add interface=bridge name=vlan23_APE vlan-id=23
add interface=bridge l3-hw-offloading=no name=vlan99 vlan-id=99
add interface=bridge name=vlan1003_BREMC vlan-id=1003
/interface list
add name=LAN
add name=WAN
/ip pool
add name=Advance_AP1 ranges=10.20.103.2-10.20.103.62
add name=dhcp_pool2 ranges=10.20.103.2-10.20.103.62
/ip dhcp-server
add add-arp=yes address-pool=dhcp_pool2 interface=vlan22_APN lease-time=3d \
    name=Advance_APN
/routing id
add disabled=no name=ospf-v2 select-dynamic-id=only-loopback select-from-vrf=\
    main
/routing ospf instance
add disabled=no name=v2inst router-id=ospf-v2
/routing ospf area
add area-id=0.0.0.2 default-cost=1 disabled=no instance=v2inst name=area2 \
    nssa-translator=candidate type=nssa
/interface bridge port
add bridge=bridge interface=sfpplus1_BREMC pvid=1003
add bridge=bridge interface=ether1
/interface bridge vlan
add bridge=bridge tagged=sfpplus1_BREMC,bridge vlan-ids=1003
add bridge=bridge tagged=ether1,bridge vlan-ids=22
add bridge=bridge tagged=ether1,bridge vlan-ids=23
add bridge=bridge untagged=bridge,ether1 vlan-ids=1
add bridge=bridge tagged=bridge,ether1 vlan-ids=99
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface list member
add interface=vlan22_APN list=LAN
add interface=sfpplus1_BREMC list=WAN
add interface=vlan1003_BREMC list=WAN
add interface=bridge list=LAN
add interface=vlan23_APE list=LAN
add interface=vlan99 list=LAN
/ip address
add address=192.168.88.1/24 comment="local management" interface=ether13 \
    network=192.168.88.0
add address=10.0.0.3 comment=Loopback interface=lo network=10.0.0.3
add address=10.10.97.62/30 comment="WAN Port - BREMC" interface=\
    vlan1003_BREMC network=10.10.97.60
add address=10.20.103.1/26 comment="APN Public" interface=vlan22_APN \
    network=10.20.103.0
add address=172.16.21.33/29 comment="Advance North mgmnt" interface=\
    vlan22_APN network=172.16.21.32
add address=172.16.2.61/30 comment="Main Switch Mgmnt" interface=vlan99 \
    network=172.16.2.60
add address=10.10.97.50/30 comment="WAN Port - backup" interface=ether5 \
    network=10.10.97.48
add address=10.20.103.65/26 comment="APE Public" interface=vlan23_APE \
    network=10.20.103.64
add address=172.16.2.148/30 comment="Advance 57NE mgmnt" interface=vlan23_APE \
    network=172.16.2.148
/ip dhcp-server network
add address=10.20.103.0/26 dns-server=66.103.104.10,66.103.104.11 gateway=\
    10.20.103.1
add address=192.168.88.0/24 dns-none=yes gateway=192.168.88.1
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=drop chain=input connection-state=invalid
add action=accept chain=input protocol=icmp
add action=drop chain=input dst-port=22,80,2000,8291 protocol=tcp \
    src-address=10.10.97.2
add action=accept chain=input dst-port=22,80,2000,8291 protocol=tcp
add action=accept chain=input protocol=ospf
add action=drop chain=input
add action=fasttrack-connection chain=forward connection-state=\
    established,related
add action=fasttrack-connection chain=forward connection-state=\
    established,related hw-offload=no
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward connection-state=invalid
/ip route
add disabled=no distance=120 dst-address=0.0.0.0/0 gateway=10.10.97.49 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/routing ospf interface-template
add area=area2 comment=loopback disabled=no networks=10.0.0.3/32 passive
add area=area2 disabled=no interfaces=LAN passive
add area=area2 comment="to Core" cost=10 disabled=no networks=10.10.97.60/30 \
    priority=0
add area=area2 comment="to Park" cost=15 disabled=no networks=10.10.97.48/30 \
    type=ptp

Here is my routing table

#
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, o - OSPF; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, ROUTING-TABLE, DISTANCE
#      DST-ADDRESS       GATEWAY                      ROUTING-TABLE  DISTANCE
0   s  0.0.0.0/0         10.10.97.49                 main                120
  DAoH 0.0.0.0/0         10.10.97.61%vlan1003_BREMC  main                110
  DAoH 10.0.0.1/32       10.10.97.61%vlan1003_BREMC  main                110
  DAoH 10.0.0.2/32       10.10.97.49%ether5          main                110
  DAc  10.0.0.3/32       lo                           main                  0
  DAoH 10.10.97.32/30   10.10.97.61%vlan1003_BREMC  main                110
  DAcH 10.10.97.48/30   ether5                       main                  0
  DAcH 10.10.97.60/30   vlan1003_BREMC               main                  0
  DAcH 10.20.103.0/26   vlan22_APN                   main                  0
  DAcH 10.20.103.64/26  vlan23_APE                   main                  0
  DAc  172.16.2.60/30    vlan99                       main                  0
  DAcH 172.16.2.148/30   vlan23_APE                   main                  0
  DAcH 172.16.21.32/29   vlan22_APN                   main                  0
  DAc  192.168.88.0/24   ether13                      main                  0

Here are the L3 Monitor stats. What I do not understand is that is says I have 37 routes but the routing table only contains 14 routes.

 ipv4-routes-total:   37
           ipv4-routes-hw:    2
          ipv4-routes-cpu:   34
  ipv4-shortest-hw-prefix:    0
               ipv4-hosts:    5
         route-queue-size:    0
         route-queue-rate:    0
       route-process-rate:    0
              nexthop-cap: 8192
            nexthop-usage:   86
    vxlan-mtu-packet-drop:    0
     fasttrack-ipv4-conns:    0
     fasttrack-queue-size:    0
     fasttrack-queue-rate:    0
   fasttrack-process-rate:    0
   fasttrack-hw-min-speed:    0
   fasttrack-hw-offloaded:    0
    fasttrack-hw-unloaded:    0
                  lpm-cap: 6720
                lpm-usage:   18
             lpm-bank-cap:  336
           lpm-bank-usage:    4
                              0
                              0
                              0
                              4
                              0
                              0
                              0
                              5
                              0
                              0
                              0
                              5
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                  pbr-cap: 4608
                pbr-usage:    0
             pbr-lpm-bank:    1
                nat-usage:    0

As mentioned, L3 HW offload is working. Input FW rules are also working.

5

I then tried your suggestion and pulled both the main WAN and backup WAN port out of the bridge and turned off L3 HW on their ports. I saw no improvement in performance with L3 HW enabled or disabled. Also even though I have a fasttrack forward FW rule with L3 Offload enabled none of the connections were being offloaded to HW.

Here is the alternate config

/interface bridge
add name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] name=sfpplus1_BREMC
/interface vlan
add interface=bridge name=vlan22_APN vlan-id=22
add interface=bridge name=vlan23_APE vlan-id=23
add interface=bridge l3-hw-offloading=no name=vlan99 vlan-id=99
add interface=sfpplus1_BREMC l3-hw-offloading=no name=vlan1003_BREMC vlan-id=\
    1003
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 8 l3-hw-offloading=no
/interface list
add name=LAN
add name=WAN
/ip pool
add name=Advance_AP1 ranges=10.20.103.2-10.20.103.62
add name=dhcp_pool2 ranges=10.20.103.2-10.20.103.62
/ip dhcp-server
add add-arp=yes address-pool=dhcp_pool2 interface=vlan22_APN lease-time=3d \
    name=Advance_APN
/routing id
add disabled=no name=ospf-v2 select-dynamic-id=only-loopback select-from-vrf=\
    main
/routing ospf instance
add disabled=no name=v2inst router-id=ospf-v2
/routing ospf area
add area-id=0.0.0.2 default-cost=1 disabled=no instance=v2inst name=area2 \
    nssa-translator=candidate type=nssa
/interface bridge port
add bridge=bridge interface=ether1
/interface bridge vlan
add bridge=bridge tagged=ether1,bridge vlan-ids=22
add bridge=bridge tagged=ether1,bridge vlan-ids=23
add bridge=bridge untagged=bridge,ether1 vlan-ids=1
add bridge=bridge tagged=bridge,ether1 vlan-ids=99
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface list member
add interface=vlan22_APN list=LAN
add interface=sfpplus1_BREMC list=WAN
add interface=vlan1003_BREMC list=WAN
add interface=bridge list=LAN
add interface=vlan23_APE list=LAN
add interface=vlan99 list=LAN
/ip address
add address=192.168.88.1/24 comment="local management" interface=ether13 \
    network=192.168.88.0
add address=10.0.0.3 comment=Loopback interface=lo network=10.0.0.3
add address=10.10.97.62/30 comment="WAN Port - BREMC" interface=\
    vlan1003_BREMC network=10.10.97.60
add address=10.20.103.1/26 comment="APN Public" interface=vlan22_APN \
    network=10.20.103.0
add address=172.16.21.33/29 comment="Advance North mgmnt" interface=\
    vlan22_APN network=172.16.21.32
add address=172.16.2.61/30 comment="Main Switch Mgmnt" interface=vlan99 \
    network=172.16.2.60
add address=10.10.97.50/30 comment="WAN Port - backup" interface=ether5 \
    network=10.10.97.48
add address=10.20.103.65/26 comment="APE Public" interface=vlan23_APE \
    network=10.20.103.64
add address=172.16.2.148/30 comment="Advance 57NE mgmnt" interface=vlan23_APE \
    network=172.16.2.148
/ip dhcp-server network
add address=10.20.103.0/26 dns-server=66.103.104.10,66.103.104.11 gateway=\
    10.20.103.1
add address=192.168.88.0/24 dns-none=yes gateway=192.168.88.1
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=drop chain=input connection-state=invalid
add action=accept chain=input protocol=icmp
add action=drop chain=input dst-port=22,80,2000,8291 protocol=tcp \
    src-address=10.10.97.2
add action=accept chain=input dst-port=22,80,2000,8291 protocol=tcp
add action=accept chain=input protocol=ospf
add action=drop chain=input
add action=fasttrack-connection chain=forward connection-state=\
    established,related
add action=fasttrack-connection chain=forward connection-state=\
    established,related hw-offload=no
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward connection-state=invalid
/ip route
add disabled=no distance=120 dst-address=0.0.0.0/0 gateway=10.10.97.49 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/routing ospf interface-template
add area=area2 comment=loopback disabled=no networks=10.0.0.3/32 passive
add area=area2 disabled=no interfaces=LAN passive
add area=area2 comment="to Core" cost=10 disabled=no networks=10.10.97.60/30 \
    priority=0
add area=area2 comment="to Park" cost=15 disabled=no networks=10.10.97.48/30 \
    type=ptp

Here are the L3 stats and Routing table

   ipv4-routes-total:   39
           ipv4-routes-hw:    0
          ipv4-routes-cpu:   38
  ipv4-shortest-hw-prefix:    0
               ipv4-hosts:    1
         route-queue-size:    0
         route-queue-rate:    0
       route-process-rate:    0
              nexthop-cap: 8192
            nexthop-usage:   84
    vxlan-mtu-packet-drop:    0
     fasttrack-ipv4-conns:    0
     fasttrack-queue-size:    0
     fasttrack-queue-rate:    8
   fasttrack-process-rate:    8
   fasttrack-hw-min-speed:    0
   fasttrack-hw-offloaded:    0
    fasttrack-hw-unloaded:    0
                  lpm-cap: 6720
                lpm-usage:    2
             lpm-bank-cap:  336
           lpm-bank-usage:    0
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                              2
                              0
                              0
                              0
                              0
                              0
                              0
                              0
                  pbr-cap: 4608
                pbr-usage:    0
             pbr-lpm-bank:    1
                nat-usage:    0
# 2026-01-30 12:12:56 by RouterOS 7.21.2
# software id = YQGJ-55VZ
#
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, o - OSPF; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, ROUTING-TABLE, DISTANCE
#      DST-ADDRESS       GATEWAY                      ROUTING-TABLE  DISTANCE
0   s  0.0.0.0/0         10.10.97.49                 main                120
  DAo  0.0.0.0/0         10.10.97.61%vlan1003_BREMC  main                110
  DAo  10.0.0.1/32       10.10.97.61%vlan1003_BREMC  main                110
  DAo  10.0.0.2/32       10.10.97.49%ether5          main                110
  DAc  10.0.0.3/32       lo                           main                  0
  DAo  10.10.97.32/30   10.10.97.61%vlan1003_BREMC  main                110
  DAc  10.10.97.48/30   ether5                       main                  0
  DAc  10.10.97.60/30   vlan1003_BREMC               main                  0
  DAcH 10.20.103.0/26   vlan22_APN                   main                  0
  DAcH 10.20.103.64/26  vlan23_APE                   main                  0
  DAc  172.16.2.60/30    vlan99                       main                  0
  DAcH 172.16.2.148/30   vlan23_APE                   main                  0
  DAcH 172.16.21.32/29   vlan22_APN                   main                  0
  DAc  192.168.88.0/24   ether13                      main                  0
6

With L3HW offload enabled, traffic bound for the CPU, such as DHCP, OSPF, BGP, management, and anything else on the input chain, will continue to work as expected and is properly governed by firewall rules. Counters also increment accordingly.

FW rules on the forward chain are ignored due to L3HW offload unless you disable HW offload on the physical port(s) where the firewalled traffic originates from. The CPU has to see the traffic to be able to limit or block it.

FW/NAT HW offload works by the CPU tracking the packets, then putting the connection tracking information into the ASIC. Future packets associated with that flow are sent to the switching hardware. In this way, you get HW-assisted offload on the forward chain. It’s a middle-ground between no hardware assistance at all and L3HW offload bypassing all rules entirely.

Also, with L3HW offload enabled, only counters on physical ports will be accurate. Traffic on VLAN and bridge interfaces will only register if the CPU sees it. This includes traffic to routes that aren’t offloaded, either due to route rules or because the routes don’t fit in the ASIC.

(EDIT: In 7.20 and later, it appears the counters are properly updated on VLAN interfaces even with HW offload enabled.)

7

So you are saying then my config should work with no funny business as long as I do not need any forward chain FW rules or NAT?

I do not think I need any Forward FW rules if L3 Offload is working. I only use it on non-L3 Offload capable routers to accept/fasttrack established/related and drop invalid. I also usually have a single SNMP block rule for forward, but I think I can handle that with an ACL rule redirecting matches to the CPU where it will then process the FW rule.

Not really sure why I cannot get anything to Fasttrack HW offload when I remove the WAN interfaces from the Bridge/disabled HW offload on the switch port. Should I not be removing the WAN interface from the bridge?

Also, still not sure why the L3 Monitor stats shows so many routes but very few of them in HW yet traffic seems to be working as expected?

8

I was able to get fasttrack HW offload connections to show up in the FW Conn table. I had to add the WAN port back to the bridge. Even then it only seemed to work consistently with TCP traffic, but UDP traffic sometimes would offload and other times would not.

Offload was 100% in my previous setup with the WAN port having L3 offload enabled.

So as before, beyond lack of forward FW rules was there any downside to having the WAN ports L3 enabled?

9

Correct. At least, the initial explanation of your config looks fine, except where you assign an IP to Ether 1, and also have Ether 1 added to the bridge with the PVID of 22, with an IP on Vlan22. That’s where I said you don’t want an IP assigned directly to Ether1 if it’s a part of the bridge, too.

I usually leave the accept and fasttrack rules on the forward chain enabled as well. If HW offload does get disabled, at least the CPU isn’t hit hard taking care of traffic while I’m sorting out an issue.

Fasttrack HW offload is the middle ground, as it were, between CPU-managed ports and HW-offloaded ports. You want your hardware-offload settings enabled everywhere else (the bridge, the routing, etc.) that you can so the CPU knows to push routes and connection tracking to the ASIC. The only place you leave a HW-offload setting off is on the switch port configuration.

10

Thanks. I did correct the IP address on Ether 1 issue by creating a Vlan99 management vlan for the switch and then putting the switch management subnet on that VLan.

Thanks for the help. The L3 HW offload stuff is a little confusing compared to how I learned to do things before on the older Mtik routers so I was not sure.