L3HW Offload Weird Issues

RouterOS General
1

Hello everyone,

All routers and switches are running 7.13.5.

I’ve poured over all the documentation, so I apologize if I’m just dense and didn’t see the answer. I’ve gone over these two pages with a fine tooth comb:
http://forum.mikrotik.com/t/how-does-l3hw-actually-works/155752/1
https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offloading#L3HardwareOffloading-Introduction

After reading those, I determined that as of now, I need my IP firewall, so I opted for the “FastTrack Connection HW Offloading”

So I ran these commands:
/interface/ethernet/switch/port set [find] l3-hw-offloading=no
/interface/ethernet/switch set 0 l3-hw-offloading=yes

I have this as the first firewall rule:
/ip firewall filter
add action=fasttrack-connection chain=forward comment=“FastTrack for established and related connections” connection-state=established,related hw-offload=yes

However I’m experiencing some really weird issues.
Example A:
I have two CCR2216’s connected with the 100Gbps interfaces. They usually push 5-10Gbps. On Saturday, randomly, with no changes made they started pushing 85Gbps TX and RX. All I did was log in and turn l3-hw off and back on and the traffic went back down to 5-10Gbps. Then I logged into the router this morning and it’s back to 85Gbps… I turn l3-hw off and back on and it goes back to 3-4 Gbps…

Example B:
I have a CRS328 that’s behind a CCR2116. The CCR2116 is configured the same way. When I try to winbox into the CRS328 switch it’s extremely slow, hardly loads, or updates. I turn off l3-hw offload and boom, the switch loads, updates, refreshes and life is good. What’s weird I have another setup on the network that’s identical, and it doesn’t have this issue. With l3-hw enabled the switch loads fast and is responsive.

Example C:
I have two CCR2216’s connected together with the SFP28 ports (So they are negotiating at 25Gbps. As soon as I enable l3-hw offload, they immediately start pushing 20-25Gbps between them, when they usually pass 3-4Gbps. I turn it off, 3-4Gbps. I turn it on, back to 20-25Gbps.

Any ideas? Do I have something configured incorrectly?

2

Any ideas? Do I have something configured incorrectly?

How can we know?
If we reverse the situation, if you were a forum user… do you have information to analyze that situation?

Where is the conf?

3

Sorry, I wasn’t sure if what I posted was enough. Makes me a bit nervous putting our configs out there. Here’s the config of the two CCR2216’s that are connected together on their qsfp28 interfaces.

/interface bridge
add name=facilities port-cost-mode=short
add name=loopback port-cost-mode=short
/interface ethernet
set [ find default-name=qsfp28-2-1 ] name=qsfp28-2-1-lewis
set [ find default-name=sfp28-1 ] name=sfp28-1-rtr
set [ find default-name=sfp28-2 ] name=sfp28-2-sea-a
set [ find default-name=sfp28-3 ] name=sfp28-3-sea-b
set [ find default-name=sfp28-4 ] name=sfp28-4-lewis-b
set [ find default-name=sfp28-12 ] name=sfp28-12-rtr-servers
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface list
add name=discovery
/port
set 0 name=serial0
/routing ospf instance
add disabled=no name=default-v2 out-filter-chain=ospf-out redistribute=connected,static,vpn,dhcp,modem router-id=172.31.251.173
/routing ospf area
add disabled=no instance=default-v2 name=backbone-v2
/snmp community
set [ find default=yes ] name=airbridge
/system logging action
set 1 disk-lines-per-file=10000
set 3 remote=xx.xxx.32.25 src-address=172.31.251.173
/ip neighbor discovery-settings
set discover-interface-list=discovery
/interface list member
add interface=ether1 list=discovery
add interface=qsfp28-1-1 list=discovery
add interface=qsfp28-1-2 list=discovery
add interface=qsfp28-1-3 list=discovery
add interface=qsfp28-1-4 list=discovery
add interface=qsfp28-2-1-lewis list=discovery
add interface=qsfp28-2-2 list=discovery
add interface=qsfp28-2-3 list=discovery
add interface=qsfp28-2-4 list=discovery
add interface=sfp28-1-rtr list=discovery
add interface=sfp28-2-sea-a list=discovery
add interface=sfp28-3-sea-b list=discovery
add interface=sfp28-4-lewis-b list=discovery
add interface=sfp28-5 list=discovery
add interface=sfp28-6 list=discovery
add interface=sfp28-7 list=discovery
add interface=sfp28-8 list=discovery
add interface=sfp28-9 list=discovery
add interface=sfp28-10 list=discovery
add interface=sfp28-11 list=discovery
add interface=sfp28-12-rtr-servers list=discovery
add interface=facilities list=discovery
add interface=loopback list=discovery
/ip address
add address=172.31.251.173 interface=loopback network=172.31.251.173
add address=xx.xxx.40.244 interface=sfp28-1-rtr network=xx.xxx.40.245
add address=xx.xxx.40.234 interface=qsfp28-2-1-lewis network=xx.xxx.40.235
add address=xx.xxx.32.253 interface=sfp28-2-sea-a network=xx.xxx.32.252
add address=xx.xxx.40.247 interface=sfp28-3-sea-b network=xx.xxx.40.246
add address=xx.xxx.32.74 interface=sfp28-12-rtr-servers network=xx.xxx.32.75
add address=xx.xxx.32.50 interface=sfp28-4-lewis-b network=xx.xxx.32.51
/ip dns
set servers=xx.xxx.32.2,xx.xxx.32.3
/ip firewall address-list
add address=xx.xxx.xxx.0/24 comment=VPN list=Allowed_SSH
add address=xx.xxx.xxx.0/24 comment=VPN list=Allowed_FTP
add address=xx.xxx.xxx.0/24 comment=VPN list=Allowed_Winbox
add address=xx.xxx.xxx.0/24 comment=VPN list=Allowed_SNMP
add address=xx.xxx.xxx.0/24 comment=VPN list=Management
add address=xx.xxx.35.2 comment=VPN list=Allowed_SSH
add address=xx.xxx.35.2 comment=VPN list=Allowed_FTP
add address=xx.xxx.35.2 comment=VPN list=Allowed_Winbox
add address=xx.xxx.35.2 comment=VPN list=Allowed_SNMP
add address=xx.xxx.35.2 comment=VPN list=Management
add address=xx.xxx.35.10 comment=VPN list=Allowed_SSH
add address=xx.xxx.35.10 comment=VPN list=Allowed_FTP
add address=xx.xxx.35.10 comment=VPN list=Allowed_Winbox
add address=xx.xxx.35.10 comment=VPN list=Allowed_SNMP
add address=xx.xxx.35.10 comment=VPN list=Management
add address=xx.xxx.32.0/27 comment=Servers list=Management
add address=xx.xxx.32.0/27 comment=Servers list=Allowed_SNMP
add address=xx.xxx.32.0/24 comment=dude list=Connectors
add address=xx.xxx.40.0/24 comment=dude list=Connectors
add address=xx.xxx.35.206 comment=lewiston list=Allowed_SSH
add address=xx.xxx.35.206 comment=lewiston list=Allowed_FTP
add address=xx.xxx.35.206 comment=lewiston list=Allowed_Winbox
add address=xx.xxx.35.206 comment=lewiston list=Allowed_SNMP
add address=xx.xxx.35.206 comment=lewiston list=Management
/ip firewall filter
add action=fasttrack-connection chain=forward comment="FastTrack for established and related connections" connection-state=established,related hw-offload=yes
add action=accept chain=input comment="Allow established and related connections" connection-state=established,related
add action=accept chain=input comment="Allow ICMP for troubleshooting" protocol=icmp
add action=accept chain=input comment="Allow UDP 33434-33534 for UDP traceroute" dst-port=33434-33534 protocol=udp
add action=accept chain=input comment="Allow OSPF from Connectors" protocol=ospf src-address-list=Connectors
add action=accept chain=input comment="Allow BFD Control from Connectors" dst-port=3784 protocol=udp src-address-list=Connectors
add action=accept chain=input comment="Allow SNMP from Allowed_SNMP" dst-port=161 protocol=udp src-address-list=Allowed_SNMP
add action=accept chain=input comment="Allow Winbox from Allowed_Winbox" dst-port=8291 protocol=tcp src-address-list=Allowed_Winbox
add action=accept chain=input comment="Allow SSH from Allowed_SSH" dst-port=22 protocol=tcp src-address-list=Allowed_SSH
add action=drop chain=input comment="Drop all other INPUT"
add action=drop chain=forward comment="Drop access to management network unless on Management list" dst-address=172.16.0.0/12 src-address-list=!Management
/ip route
add blackhole comment="Customer Network" disabled=no distance=254 dst-address=100.67.84.0/22
add blackhole comment="Management Network" disabled=no distance=254 dst-address=172.18.184.0/22
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/routing bfd configuration
add address-list=Connectors disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5 vrf=main
/routing filter rule
add chain=ospf-out comment="Ubiquiti Configuration" disabled=no rule="if (dst == 192.168.1.0/24) { accept; }"
add chain=ospf-out comment="All Public IP's" disabled=no rule="if (dst in xx.xxx.32.0/22 && dst-len in 0-32) { accept; }"
add chain=ospf-out comment="All Public IP's" disabled=no rule="if (dst in xx.xxx.40.0/22 && dst-len in 0-32) { accept; }"
add chain=ospf-out comment="All Public IP's" disabled=no rule="if (dst in xx.xx.112.0/22 && dst-len in 0-32) { accept; }"
add chain=ospf-out comment="Management Network" disabled=no rule="if (dst == 172.18.184.0/22) { accept; }"
add chain=ospf-out comment="Drop everything not explicitly allowed" disabled=no rule="reject;"
/routing ospf interface-template
add area=backbone-v2 disabled=no interfaces=loopback networks=172.31.251.173 priority=1
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxx cost=2 disabled=no interfaces=sfp28-1-rtr networks=xx.xxx.40.244/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxx cost=2 disabled=no interfaces=qsfp28-2-1-lewis networks=xx.xxx.40.234/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxx cost=11 disabled=no interfaces=sfp28-4-lewis-b networks=xx.xxx.32.50/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxx cost=2 disabled=no interfaces=sfp28-2-sea-a networks=xx.xxx.32.252/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxx cost=2 disabled=no interfaces=sfp28-3-sea-b networks=xx.xxx.40.246/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxx cost=2 disabled=no interfaces=sfp28-12-rtr-servers networks=xx.xxx.32.74/31 priority=1 type=ptp use-bfd=yes
/snmp
set contact="AirBridge Broadband" enabled=yes location=clark
/system clock
set time-zone-name=America/Los_Angeles
/system identity
set name=rtr-edge.clark
/system logging
set 0 action=disk
set 1 action=disk
set 2 action=disk
set 3 action=disk
add action=remote topics=critical
add action=remote topics=error
add action=remote topics=info
add action=remote topics=warning
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=xx.xxx.32.6
add address=xx.xxx.32.7
/system routerboard settings
set auto-upgrade=yes enter-setup-on=delete-key
/tool mac-server
set allowed-interface-list=discovery
/tool mac-server mac-winbox
set allowed-interface-list=discovery
/tool mac-server ping
set enabled=no
4 
/interface bridge
add name=facilities port-cost-mode=short
add name=loopback port-cost-mode=short
/interface ethernet
set [ find default-name=qsfp28-2-1 ] name=qsfp28-2-1-clark
set [ find default-name=sfp28-1 ] name=sfp28-1-clark-b
set [ find default-name=sfp28-6 ] name=sfp28-6-rr
set [ find default-name=sfp28-8 ] name=sfp28-8-lewis
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface list
add name=discovery
/port
set 0 name=serial0
/routing ospf instance
add disabled=no name=default-v2 out-filter-chain=ospf-out redistribute=connected,static,vpn,dhcp,modem router-id=172.31.251.171
/routing ospf area
add disabled=no instance=default-v2 name=backbone-v2
/snmp community
set [ find default=yes ] name=airbridge
/system logging action
set 1 disk-lines-per-file=10000
set 3 remote=xx.xxx.32.25 src-address=172.31.251.171
/interface bridge port
add bridge=facilities interface=ether1
/ip neighbor discovery-settings
set discover-interface-list=discovery
/interface list member
add interface=ether1 list=discovery
add interface=qsfp28-1-1 list=discovery
add interface=qsfp28-1-2 list=discovery
add interface=qsfp28-1-3 list=discovery
add interface=qsfp28-1-4 list=discovery
add interface=qsfp28-2-1-clark list=discovery
add interface=qsfp28-2-2 list=discovery
add interface=qsfp28-2-3 list=discovery
add interface=qsfp28-2-4 list=discovery
add interface=sfp28-1-clark-b list=discovery
add interface=sfp28-2 list=discovery
add interface=sfp28-3 list=discovery
add interface=sfp28-4 list=discovery
add interface=sfp28-5 list=discovery
add interface=sfp28-6-rr list=discovery
add interface=sfp28-7 list=discovery
add interface=sfp28-8-lewis list=discovery
add interface=sfp28-9 list=discovery
add interface=sfp28-10 list=discovery
add interface=sfp28-11 list=discovery
add interface=sfp28-12 list=discovery
add interface=facilities list=discovery
add interface=loopback list=discovery
/ip address
add address=172.31.251.171 interface=loopback network=172.31.251.171
add address=xx.xxx.40.235 interface=qsfp28-2-1-clark network=xx.xxx.40.234
add address=172.18.168.1/28 interface=facilities network=172.18.168.0
add address=xx.xxx.40.238 interface=sfp28-8-lewis network=xx.xxx.40.239
add address=xx.xxx.32.51 interface=sfp28-1-clark-b network=xx.xxx.32.50
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.1
/ip dns
set servers=xx.xxx.32.2,xx.xxx.32.3
/ip firewall address-list
add address=xx.xxx.254.0/24 comment=VPN list=Allowed_SSH
add address=xx.xxx.254.0/24 comment=VPN list=Allowed_FTP
add address=xx.xxx.254.0/24 comment=VPN list=Allowed_Winbox
add address=xx.xxx.254.0/24 comment=VPN list=Allowed_SNMP
add address=xx.xxx.254.0/24 comment=VPN list=Management
add address=xx.xxx.35.2 comment=VPN list=Allowed_SSH
add address=xx.xxx.35.2 comment=VPN list=Allowed_FTP
add address=xx.xxx.35.2 comment=VPN list=Allowed_Winbox
add address=xx.xxx.35.2 comment=VPN list=Allowed_SNMP
add address=xx.xxx.35.2 comment=VPN list=Management
add address=xx.xxx.35.10 comment=VPN list=Allowed_SSH
add address=xx.xxx.35.10 comment=VPN list=Allowed_FTP
add address=xx.xxx.35.10 comment=VPN list=Allowed_Winbox
add address=xx.xxx.35.10 comment=VPN list=Allowed_SNMP
add address=xx.xxx.35.10 comment=VPN list=Management
add address=xx.xxx.32.0/27 comment=Servers list=Management
add address=xx.xxx.32.0/27 comment=Servers list=Allowed_SNMP
add address=xx.xxx.32.0/24 comment=dude list=Connectors
add address=xx.xxx.40.0/24 comment=dude list=Connectors
add address=xx.xxx.35.206 comment=Lewiston list=Allowed_Winbox
add address=xx.xxx.35.206 comment=VPN list=Management
add address=xx.xxx.35.206 comment=VPN list=Allowed_SSH
add address=xx.xxx.35.206 comment=lewiston list=Allowed_FTP
add address=xx.xxx.35.206 comment=lewiston list=Allowed_SNMP
/ip firewall filter
add action=fasttrack-connection chain=forward comment="FastTrack for established and related connections" connection-state=established,related hw-offload=yes
add action=accept chain=input comment="Allow established and related connections" connection-state=established,related
add action=accept chain=input comment="Allow ICMP for troubleshooting" protocol=icmp
add action=accept chain=input comment="Allow UDP 33434-33534 for UDP traceroute" dst-port=33434-33534 protocol=udp
add action=accept chain=input comment="Allow OSPF from Connectors" protocol=ospf src-address-list=Connectors
add action=accept chain=input comment="Allow BFD Control from Connectors" dst-port=3784 protocol=udp src-address-list=Connectors
add action=accept chain=input comment="Allow SNMP from Allowed_SNMP" dst-port=161 protocol=udp src-address-list=Allowed_SNMP
add action=accept chain=input comment="Allow Winbox from Allowed_Winbox" dst-port=8291 protocol=tcp src-address-list=Allowed_Winbox
add action=accept chain=input comment="Allow SSH from Allowed_SSH" dst-port=22 protocol=tcp src-address-list=Allowed_SSH
add action=drop chain=input comment="Drop all other INPUT"
add action=drop chain=forward comment="Drop access to management network unless on Management list" dst-address=172.16.0.0/12 src-address-list=!Management
/ip route
add blackhole comment="Customer Network" disabled=no distance=254 dst-address=100.67.84.0/22
add blackhole comment="Management Network" disabled=no distance=254 dst-address=172.18.168.0/22
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/routing bfd configuration
add address-list=Connectors disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5 vrf=main
/routing filter rule
add chain=ospf-out comment="Ubiquiti Configuration" disabled=no rule="if (dst == 192.168.1.0/24) { accept; }"
add chain=ospf-out comment="All Public IP's" disabled=no rule="if (dst in xx.xxx.32.0/22 && dst-len in 0-32) { accept; }"
add chain=ospf-out comment="All Public IP's" disabled=no rule="if (dst in xx.xxx.40.0/22 && dst-len in 0-32) { accept; }"
add chain=ospf-out comment="All Public IP's" disabled=no rule="if (dst in xx.xx.112.0/22 && dst-len in 0-32) { accept; }"
add chain=ospf-out comment="Management Network" disabled=no rule="if (dst == 172.18.168.0/22) { accept; }"
add chain=ospf-out comment=Loopbacks disabled=no rule="if (dst == 172.31.251.0/21) { accept; }"
add chain=ospf-out comment="Drop everything not explicitly allowed" disabled=no rule="reject;"
/routing ospf interface-template
add area=backbone-v2 disabled=no interfaces=loopback networks=172.31.251.171 priority=1
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxxx cost=2 disabled=no interfaces=qsfp28-2-1-clark networks=xx.xxx.40.234/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxxx cost=11 disabled=no interfaces=sfp28-1-clark-b networks=xx.xxx.32.50/31 priority=1 type=ptp use-bfd=yes
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxxxxxxx cost=2 disabled=no interfaces=sfp28-8-lewis networks=xx.xxx.40.238/31 priority=1 type=ptp use-bfd=yes
/snmp
set contact="AirBridge Broadband" enabled=yes location=lewis
/system clock
set time-zone-name=America/Los_Angeles
/system identity
set name=rtr-edge.lewis
/system logging
set 0 action=disk
set 1 action=disk
set 2 action=disk
set 3 action=disk
add action=remote topics=critical
add action=remote topics=error
add action=remote topics=info
add action=remote topics=warning
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=xx.xxx.32.6
add address=xx.xxx.32.7
/system routerboard settings
set auto-upgrade=yes enter-setup-on=delete-key
/tool mac-server
set allowed-interface-list=discovery
/tool mac-server mac-winbox
set allowed-interface-list=discovery
/tool mac-server ping
set enabled=no
5

L3HW offload is hit and miss on the 2116’s. I have it disabled for now on all of mine.

There are known issues with ECMP (multiple identical-cost routes) and L3HW offload. They fixed some of it (for plain routing) a release or two ago, but I found new issues with it when NAT is enabled on a 2116.

With 7.11 and 7.12, it’s working on a dozen CRS300 series switches acting as routers.

I haven’t combed through your configs, but it sounds to me like one or more of the routes is getting “stuck” on the ASIC and it’s looping all the traffic back to itself. I’ve seen situations where OSPF and BGP are announcing one set of routes, but the wrong ones are loaded into the switch chip. Or other situations where the routes have been withdrawn by the originating router, but other routers deeper in the network are still showing them as active and it takes a reboot to clear them out (or update them to the current path).