Community discussions

MikroTik App
 
willr
just joined
Topic Author
Posts: 6
Joined: Mon Nov 22, 2021 1:27 pm

L3HW not working properly

Mon Nov 22, 2021 3:13 pm

Hi there,

Have a CRS317-1G-16S+. A few weeks ago I tested L3HW and was getting ~10Gbps in each direction using iperf3. There are no firewall rules in place, and the device exists for inter-VLAN routing via the bridge.

A few weeks later I'm now testing again and finding that throughput is way lower - more like 700-1000 Mbps. It seems to be hitting the CRS CPU too, so I think L3HW is really not working right now.

The only thing that has been changed in that time is VLAN membership of various ports (i.e. /interface/vlan). BUT on the L3HW documentation page, it does say "It is recommended to turn off L3HW offloading during L2 configuration.". Unfortunately I'm pretty certain we didn't do that before changing the VLAN membership!

Therefore I realise we haven't followed instructions correctly, so maybe this has caused the drop in throughput?

What I would like to know is - how do I remedy the problems that might have been caused by not following the guidance? Can I just do something simple like reboot the CRS device (my preference!)? Or do I need to reconfigure the ports (/interface/vlan) from scratch? Or (worse) do I need to clear the config and start again from a clean device??

Switch is currently running v7.1.4rc4.

Thank you in advance!

Will
 
willr
just joined
Topic Author
Posts: 6
Joined: Mon Nov 22, 2021 1:27 pm

Re: L3HW not working properly

Mon Nov 22, 2021 3:25 pm

Here's my config by the way (removed serial number and MAC).

The routing I'm doing is from VLAN 4 (10.9.4.0/24) to VLAN 6 (10.9.6.0/24).
# feb/23/1970 19:07:02 by RouterOS 7.1rc4
# software id = TSFV-NY6N
#
# model = CRS317-1G-16S+
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf mtu=9000 name=\
    bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus1 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus2 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus3 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus4 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus5 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus6 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus7 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus8 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus9 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus10 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus11 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus12 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus13 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus14 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus15 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus16 ] l2mtu=9022 mtu=9000
/interface vlan
add interface=bridge name=vlan2 vlan-id=2
add interface=bridge name=vlan4 vlan-id=4
add interface=bridge name=vlan6 vlan-id=6
add interface=bridge name=vlan99 vlan-id=99
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus4 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus5 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus6 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus7 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus14 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface bridge vlan
add bridge=bridge tagged="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus13,sfp-sfpplus1\
    4,sfp-sfpplus15,sfp-sfpplus16" vlan-ids=3
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=5
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=7
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=8
add bridge=bridge tagged=sfp-sfpplus3,sfp-sfpplus13,sfp-sfpplus14 vlan-ids=50
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=2
add bridge=bridge tagged=bridge,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=6
add bridge=bridge tagged=\
    bridge,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=4
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 vlan-ids=99
add bridge=bridge tagged=sfp-sfpplus4,sfp-sfpplus13,sfp-sfpplus14 vlan-ids=51
add bridge=bridge tagged="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus13,sfp-sfpplus1\
    4,sfp-sfpplus15,sfp-sfpplus16" vlan-ids=10
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
add address=10.9.4.254/24 interface=vlan4 network=10.9.4.0
add address=10.9.6.254/24 interface=vlan6 network=10.9.6.0
add address=10.9.99.1/24 interface=vlan99 network=10.9.99.0
add address=10.9.2.250/24 interface=vlan2 network=10.9.2.0
/ip dhcp-relay
add dhcp-server=10.9.6.11 disabled=no interface=vlan4 name=relay1
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.9.99.254 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/system routerboard settings
set boot-os=router-os
 
willr
just joined
Topic Author
Posts: 6
Joined: Mon Nov 22, 2021 1:27 pm

Re: L3HW not working properly

Tue Nov 23, 2021 12:46 pm

Well, I didn't change the config, but I rebooted the switch and upgraded to rc6, and L3HW is working again now.

Now I'm just really looking forward to L3HW working with both conntrack offload and bridge at the same time! I understand it's in development...
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: L3HW not working properly

Wed Nov 24, 2021 10:26 am

Hi, and sorry for a late reply,

I'm glad that the problem got resolved by upgrading to v7.1rc6. Rc4 had an issue with MTU offloading so that most likely was the case.

Regarding:
It is recommended to turn off L3HW offloading during L2 configuration.
Configuring L2 while L3HW is enabled does not cause any permanent effects - rebooting should fix all the problems. The recommendation is given due to various edge cases where L2/L3/SW/HW configurations can be out-of-sync with each other. Currently, we don't have time to address all those minor issues due to the focus on stabilizing v7.1, then Fasttrack offloading on VLAN-filtered bridge and L3HW IPv6. Turning off L3HW while configuring L2 is a temporary safety switch.
 
willr
just joined
Topic Author
Posts: 6
Joined: Mon Nov 22, 2021 1:27 pm

Re: L3HW not working properly

Mon Dec 13, 2021 3:50 pm

Sadly this problem has come back again with rc6 :( CPU is maxed out with routing, and L3 isn't being used. Looks like it's a bit of a time-bomb - things works fine for a while and then suddenly L3HW breaks!

I will update to v7.1 release tonight, but this time I've noticed the log is full of the following -

13:46:06 route,warning L3HW: FDB host 10.9.4.67 offload FAILED (-14)
13:46:09 route,warning L3HW: FDB host 10.9.4.50 offload FAILED (-14)
13:46:14 route,warning L3HW: FDB host 10.9.4.26 offload FAILED (-14)
13:46:18 route,warning L3HW: FDB host 10.9.4.67 offload FAILED (-14)
13:46:23 route,warning L3HW: FDB host 10.9.4.69 offload FAILED (-14)
13:46:33 route,warning L3HW: FDB host 10.9.4.67 offload FAILED (-14)
13:46:53 route,warning L3HW: FDB host 10.9.4.68 offload FAILED (-14)
13:47:03 route,warning L3HW: FDB host 10.9.4.68 offload FAILED (-14)
13:47:15 route,warning L3HW: FDB host 10.9.4.68 offload FAILED (-14)

I guess this was the same last time but I didn't check properly!

Config is the same as I pasted above - nothing has changed at all.

Any help/advice appreciated!
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: L3HW not working properly

Mon Dec 13, 2021 4:55 pm

"-14" is an internal error code, meaning that the L3HW driver is turning off or restarting. Were you rebooting the router or setting "l3-hw-offloading=no" when those log messages appeared?
 
willr
just joined
Topic Author
Posts: 6
Joined: Mon Nov 22, 2021 1:27 pm

Re: L3HW not working properly

Mon Dec 13, 2021 5:19 pm

No changes, that just happens with it running. The config is the same since the firmware upgrade to rc6 on 22nd Nov.

That part of log above was just an excerpt - the log is entirely full of those messages from the last 3 hours (except for a few showing when I logged in).

The reason I logged in was to investigate the low inter-VLAN throughput - in other words the problem was already there before I even logged in.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: L3HW not working properly

Wed Dec 15, 2021 9:50 am

If a host offload fails, then the traffic to that host (IP) goes via CPU, and, therefore, causes a performance drop. Are those hosts in the log real, or are you running a network test by sending packets to random destinations in the subnet? Hosts can be offloaded only after resolving ARP (IP-MAC). Hence, "fake hosts" cannot be offloaded and always go through the CPU.

Also, when the problem occurs, please check the result of the following command:
/in/bridge/host print count-only
 
joshedmonds
just joined
Posts: 1
Joined: Wed Dec 22, 2021 12:02 am

Re: L3HW not working properly

Wed Dec 22, 2021 12:07 am

I have the same problem running 7.1 stable on a CRS309. L3 hardware offloading will just stop randomly and requires a power cycle and/or toggling the l3-hw-offloading setting to get it going again.
Last edited by joshedmonds on Wed Dec 22, 2021 12:09 am, edited 1 time in total.
 
willr
just joined
Topic Author
Posts: 6
Joined: Mon Nov 22, 2021 1:27 pm

Re: L3HW not working properly

Mon Jan 24, 2022 11:58 am

If a host offload fails, then the traffic to that host (IP) goes via CPU, and, therefore, causes a performance drop. Are those hosts in the log real, or are you running a network test by sending packets to random destinations in the subnet? Hosts can be offloaded only after resolving ARP (IP-MAC). Hence, "fake hosts" cannot be offloaded and always go through the CPU.

Also, when the problem occurs, please check the result of the following command:
/in/bridge/host print count-only
Sorry for the delay in replying.

Current state - I updated to 7.1 stable on 13th Dec.

To answer your question - the hosts are indeed real - mostly desktop PCs with real MAC addresses (and a couple of server NICs for Hyper-V). I should point out that this is a pretty busy office network, so if there's a rare case that causes L3HW to fail, it might be happening more often for me because of the high traffic (compared to a SOHO environment).

The issue reoccurred today but I'm afraid I forgot to run the command you requested. I'll do that next time. This time I tried toggling L3HW off and then on, and that resolved it per joshedmonds message. Nice not to have to schedule a switch reboot!

I should point out that this time unfortunately the switch didn't log the 'offload failed (-14)' messages. The only clue was that the CPU was mostly at 100% and inter-VLAN throughput was being reported as low.
 
dvdhngs
just joined
Posts: 6
Joined: Sun Feb 19, 2023 2:09 pm

Re: L3HW not working properly

Mon Feb 27, 2023 10:23 pm

Hi... old topic, but its almost my case, the difference its i'm on 7.8 stable...

CRS326-24G-2S+

did you solved your problem?

i'm with 10 vlans (interface/vlan)
600 devices on /in/bridge/host print count-only
(+/- 20 nvr and 400 ip cameras)
and everything using cpu (800mbps on bridge and 98% cpu)

any tip?
thanks in advance
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: L3HW not working properly

Tue Feb 28, 2023 6:21 am

It may be working on previous versions <7.6 but any version > 7.6 it is not working. Please check this thread and add your situation or send a support ticket as others have identified this as not working as well.

viewtopic.php?t=193770
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: L3HW not working properly

Tue Feb 28, 2023 7:29 pm

Hi... old topic, but its almost my case, the difference its i'm on 7.8 stable...

CRS326-24G-2S+

i'm with 10 vlans (interface/vlan)
600 devices on /in/bridge/host print count-only
and everything using cpu (800mbps on bridge and 98% cpu)
Basic L3HW offload is working on 7.4.1, 7.6, and 7.7 on CRS310, 317, and CCR2116.
Configure all your Layer 2 stuff first:
1) Configure port L2MTU's, configure bonds (LACP)
2) Build your bridge, add ports to bridge
3) Create VLANs (assigned to bridge), change VLAN MTUs (if not 1500), assign IP's to VLANs, enable VLANs on bridge ports
4) Enable bridge VLAN filtering (unless all VLANs are always available to all ports)

Configure OSPF, BGP, static routes, etc. as normal.

Check that Layer 2 and basic routing on Layer 3 are working. Most CRS300's can handle up to a gigabit of traffic in the CPU.

Once you've confirmed everything is working, enable L3HW offload on the switch (/interface/ethernet/switch) and on the switch ports (/interface/ethernet/switch/ports). In the case of my CRS300's, I enable it on all ports. I don't want the CPU handling anything it doesn't need to.

If you ever make changes to the switch config (new ports, new VLAN's with new IP's), you may need to disable L3HW offload on the switch interface, and then enable it again. This causes the OS to push the config down to the ASIC. Sometimes I find I have to disable it on all the ports as well as the switch, then re-enable it on everything again.

Some have noticed certain routes stop working and have scheduled the switch to disable/enable every day or so. I have the CRS317 scheduled to reload every two hours. The CRS310's don't seem to have the same problem, but then they handle fewer network changes and route less traffic.

Also, there is currently a bug if you use ECMP (equal cost multi-path routing). When the routes fail and come back, it doesn't push any of them to the ASIC. The workaround is to either manually disable/enable L3HW offload, or to change the cost of one of the routes, essentially breaking the load sharing benefits of ECMP. They have reproduced the problem and a fix is slated for a future release.

Who is online

Users browsing this forum: No registered users and 20 guests