Community discussions

MikroTik App
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

After enabling hardware offloading, console and mgmt stops responding  [SOLVED]

Thu Dec 30, 2021 8:22 pm

Hey all

I'm having a weird issue where when I enable L3 hardware offload the console port AND management on my CRS309 stop responding. Oddly, I can still ping my vlan interfaces or point to point IPs. I just can't manage at all. I've blown away the config and started from scratch and it happens again after I'm done applying my config. For the record, I'm following the Mikrotik documentation where it says you should only enable l3 hardware offloading AFTER you are done making L2 changes. Any ideas? I would really like to have this enabled since I will have a lot of inter-VLAN traffic.

EDIT: Noting I did a reboot and as soon as it gets to the Login prompt the console freezes.

EDIT #2: I started from scratch, with just the default base config and enabling l3 hw offload looks fine. So its a problem where if I have a specific config enabled (something on the attached config I pasted below) and then re-enabling l3 hw offload causes the CRS to lock up... I'm confused how you can make changes in the future without interruption because the documentation states you need to disable l3 hw offload, apply your l2 changes, and then re-enable.

EDIT #3: Ok, did some more testing by re-applying the config piece by piece. It freezes after applying the interface vlan configurations. So specifically this part here:
/interface vlan
add interface=bridge name=GuestWiFi vlan-id=81
add interface=bridge name=HomeWiFi vlan-id=51
add interface=bridge name=HomeWired vlan-id=50
add interface=bridge name=IOTDevices vlan-id=80
add interface=bridge name=NetworkMgmt vlan-id=15
add interface=bridge name=Server vlan-id=52
add interface=bridge name=Storage vlan-id=53
add interface=bridge name=iDRACMgmt vlan-id=16
Once I've done all that, I set l3-hw-offload to yes and it took, and then I went ahead to disable it again to add another vlan and the console froze.

EDIT #4: Ok, did some more testing. I added the interface vlans one by one to see if there was any difference by doing this every time.
/interface ethernet switch set 0 l3-hw-offloading=no
/interface vlan add interface=bridge name=GuestWiFi vlan-id=81
/interface ethernet switch set 0 l3-hw-offloading=yes
I did that for each vlan. It was working all the way up until I added the 8th and final vlan configuration and as soon as I enabled hw offloading, the switch immediately crashed. Seems like at some point 8 vlans is too much for this CRS309???

EDIT #5: So that was the issue and I saw the 7.1.1 has a fix for the issue where hw offloading for 7 or more vlans has issues. I'll test that version out... Leaving this here for posterity.
 
glow
newbie
Posts: 29
Joined: Sun Dec 05, 2021 1:56 am

Re: After enabling hardware offloading, console and mgmt stops responding

Fri Dec 31, 2021 5:18 am

Please let everyone know if 7.1.1 does resolve this. I don't have nearly as many VLANs, but I'd still like to know!
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: After enabling hardware offloading, console and mgmt stops responding

Sun Jan 02, 2022 6:27 pm

So far so good on that particular issue.
 
User avatar
galvesribeiro
newbie
Posts: 38
Joined: Mon Apr 12, 2021 4:34 am

Re: After enabling hardware offloading, console and mgmt stops responding

Fri Jul 15, 2022 3:25 pm

I have the same issue. I'm using 7.3.1 (latest stable as of now) and I had the management port with no link after enabled the HW offload. My bridge has 8 VLANs as well.

I've disabled the HW offload, rebooted the router, but still, no link on the management port (eth1).

I'm using a CCR2216-1G-12XS-2XQ. Any idea why that port would not work?

Thanks!
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: After enabling hardware offloading, console and mgmt stops responding

Mon Jul 18, 2022 5:38 pm

I have the same issue. I'm using 7.3.1 (latest stable as of now) and I had the management port with no link after enabled the HW offload. My bridge has 8 VLANs as well.

I've disabled the HW offload, rebooted the router, but still, no link on the management port (eth1).

I'm using a CCR2216-1G-12XS-2XQ. Any idea why that port would not work?

Thanks!
Hi,

Please share your config so we can take a look:
/interface export
/ip export
 
User avatar
galvesribeiro
newbie
Posts: 38
Joined: Mon Apr 12, 2021 4:34 am

Re: After enabling hardware offloading, console and mgmt stops responding

Mon Aug 08, 2022 1:42 pm

Sorry, I've missed your reply @raimondsp. Here is the output (masked public IPs)
/interface export hide-sensitive 
# aug/08/2022 07:33:36 by RouterOS 7.4
# model = CCR2216-1G-12XS-2XQ
/interface bridge
add admin-mac=02:4A:B4:F6:2D:A4 auto-mac=no ingress-filtering=no name=bridge-LAN vlan-filtering=yes
add name=bridge-Loopback
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1584 name=ether-Management
set [ find default-name=qsfp28-1-1 ] auto-negotiation=no fec-mode=off name=qsfp28-1-1-Spine-1 speed=25Gbps
set [ find default-name=qsfp28-1-2 ] auto-negotiation=no fec-mode=off name=qsfp28-1-2-Spine-1 speed=25Gbps
set [ find default-name=qsfp28-1-3 ] auto-negotiation=no fec-mode=off name=qsfp28-1-3-Spine-2 speed=25Gbps
set [ find default-name=qsfp28-1-4 ] auto-negotiation=no fec-mode=off name=qsfp28-1-4-Spine-2 speed=25Gbps
set [ find default-name=qsfp28-2-1 ] auto-negotiation=no fec-mode=off name=qsfp28-2-1-Spine-1 speed=25Gbps
set [ find default-name=qsfp28-2-2 ] auto-negotiation=no fec-mode=off name=qsfp28-2-2-Spine-1 speed=25Gbps
set [ find default-name=qsfp28-2-3 ] auto-negotiation=no fec-mode=off name=qsfp28-2-3-Spine-2 speed=25Gbps
set [ find default-name=qsfp28-2-4 ] auto-negotiation=no fec-mode=off name=qsfp28-2-4-Spine-2 speed=25Gbps
set [ find default-name=sfp28-1 ] auto-negotiation=no name=sfp28-1-HostFiber speed=10Gbps
set [ find default-name=sfp28-2 ] auto-negotiation=no name=sfp28-2-HostFiber speed=10Gbps
set [ find default-name=sfp28-3 ] auto-negotiation=no name=sfp28-3-Azure-1 speed=25Gbps
set [ find default-name=sfp28-4 ] auto-negotiation=no name=sfp28-4-Azure-2 speed=25Gbps
set [ find default-name=sfp28-5 ] auto-negotiation=no name=sfp28-5-Azure-1 speed=25Gbps
set [ find default-name=sfp28-6 ] auto-negotiation=no name=sfp28-6-Azure-2 speed=25Gbps
set [ find default-name=sfp28-7 ] auto-negotiation=no name=sfp28-7-MacPro speed=10Gbps
set [ find default-name=sfp28-8 ] auto-negotiation=no name=sfp28-8-MacPro speed=10Gbps
set [ find default-name=sfp28-9 ] auto-negotiation=no name=sfp28-9-iDRAC speed=10Gbps
set [ find default-name=sfp28-10 ] auto-negotiation=no name=sfp28-10-iDRAC speed=10Gbps
set [ find default-name=sfp28-11 ] auto-negotiation=no name=sfp28-11-Guto speed=10Gbps
set [ find default-name=sfp28-12 ] auto-negotiation=no name=sfp28-12-Vivo
/interface pppoe-client
add disabled=no interface=sfp28-12-Vivo name=pppoe-Vivo profile=profile-Vivo user=cliente@cliente
/interface vlan
add interface=bridge-LAN name=vlan20-Guest vlan-id=20
add interface=bridge-LAN name=vlan30-Doors vlan-id=30
add interface=bridge-LAN name=vlan31-IoT vlan-id=31
add interface=bridge-LAN name=vlan40-Staff vlan-id=40
add interface=bridge-LAN name=vlan50-UniFi vlan-id=50
add interface=bridge-LAN name=vlan80-iDRAC vlan-id=80
add interface=bridge-LAN name=vlan90-Management vlan-id=90
add interface=bridge-LAN name=vlan100-Servers vlan-id=100
/interface bonding
add mode=802.3ad name=bond-HostFiber slaves=sfp28-1-HostFiber,sfp28-2-HostFiber transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=bond-MacPro slaves=sfp28-7-MacPro,sfp28-8-MacPro transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=bond-Spine-1 slaves=qsfp28-1-1-Spine-1,qsfp28-1-2-Spine-1,qsfp28-2-1-Spine-1,qsfp28-2-2-Spine-1 \
    transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=bond-Spine-2 slaves=qsfp28-1-3-Spine-2,qsfp28-1-4-Spine-2,qsfp28-2-3-Spine-2,qsfp28-2-4-Spine-2 \
    transmit-hash-policy=layer-2-and-3
add mode=802.3ad name=bond-iDRAC slaves=sfp28-10-iDRAC,sfp28-9-iDRAC transmit-hash-policy=layer-2-and-3
/interface vlan
add interface=bond-HostFiber name=vlan-HostFiber-AWS-993 vlan-id=993
add interface=bond-HostFiber name=vlan-HostFiber-Azure-1-995 vlan-id=995
add interface=bond-HostFiber name=vlan-HostFiber-Azure-2-1011 vlan-id=1011
add interface=bond-HostFiber name=vlan-HostFiber-IP-2836 vlan-id=2836
/interface ethernet switch
set 0 l3-hw-offloading=yes name=switch-chip
/interface ethernet switch port
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface list
add name=WAN
add name=FastTrack
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge-LAN interface=bond-Spine-1
add bridge=bridge-LAN interface=bond-Spine-2
add bridge=bridge-LAN ingress-filtering=no interface=bond-MacPro pvid=100
add bridge=bridge-LAN interface=bond-iDRAC
add bridge=bridge-LAN interface=sfp28-11-Guto
add bridge=bridge-LAN interface=sfp28-3-Azure-1 pvid=100
add bridge=bridge-LAN interface=sfp28-4-Azure-2 pvid=100
add bridge=bridge-LAN interface=sfp28-5-Azure-1 pvid=100
add bridge=bridge-LAN interface=sfp28-6-Azure-2 pvid=100
/interface bridge vlan
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN vlan-ids=20
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN vlan-ids=30
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN vlan-ids=31
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN vlan-ids=40
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN vlan-ids=50
add bridge=bridge-LAN tagged=bridge-LAN,bond-iDRAC vlan-ids=80
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN vlan-ids=90
add bridge=bridge-LAN tagged=bond-Spine-1,bond-Spine-2,bridge-LAN untagged=\
    sfp28-3-Azure-1,sfp28-4-Azure-2,sfp28-5-Azure-1,sfp28-6-Azure-2,bond-MacPro vlan-ids=100
add bridge=bridge-LAN untagged=bridge-LAN,sfp28-11-Guto vlan-ids=1
/interface list member
add interface=pppoe-Vivo list=WAN
add interface=vlan-HostFiber-IP-2836 list=WAN
/interface ovpn-server server
set auth=sha1,md5

/ip export hide-sensitive 
# aug/08/2022 07:37:00 by RouterOS 7.4
# software id = 7KAG-BQDH
# model = CCR2216-1G-12XS-2XQ
/ip dhcp-server option
add code=119 name=domain-search value=0x0b7069706173747564696f7303636f6d00
/ip dhcp-server option sets
add name=dhcp-options options=domain-search
/ip ipsec policy group
add name=group-Staff-VPN
/ip ipsec profile
add dh-group=modp2048,modp1536,modp1024 dpd-interval=disable-dpd enc-algorithm=aes-256,aes-192,aes-128 hash-algorithm=sha256 name=profile-Staff-VPN
/ip ipsec peer
add exchange-mode=ike2 name=peer-Staff-VPN-HostFiber passive=yes profile=profile-Staff-VPN send-initial-contact=no
/ip ipsec proposal
add auth-algorithms=sha512,sha256,sha1 enc-algorithms=aes-256-cbc,aes-256-ctr,aes-256-gcm,aes-192-ctr,aes-192-gcm,aes-128-cbc,aes-128-ctr,aes-128-gcm lifetime=8h name=proposal-Staff-VPN \
    pfs-group=none
/ip pool
add name=pool-Staff-VPN ranges=10.41.0.2-10.41.0.254
add name=pool-Doors ranges=10.30.0.54-10.30.0.254
add name=pool-Guest ranges=10.20.0.54-10.20.0.254
add name=pool-IoT ranges=10.31.0.54-10.31.0.254
add name=pool-LAN ranges=10.10.0.54-10.10.0.254
add name=pool-Staff ranges=10.40.0.54-10.40.255.254
add name=pool-UniFi ranges=10.50.0.150-10.50.0.254
/ip dhcp-server
add add-arp=yes address-pool=pool-Doors dhcp-option-set=dhcp-options interface=vlan30-Doors lease-time=1d name=dhcp-Doors
add add-arp=yes address-pool=pool-Guest interface=vlan20-Guest name=dhcp-Guest
add add-arp=yes address-pool=pool-IoT dhcp-option-set=dhcp-options interface=vlan31-IoT lease-time=1d name=dhcp-IoT
add add-arp=yes address-pool=pool-LAN dhcp-option-set=dhcp-options interface=bridge-LAN lease-time=1d name=dhcp-LAN
add add-arp=yes address-pool=pool-Staff dhcp-option-set=dhcp-options interface=vlan40-Staff lease-time=1d name=dhcp-Staff
add add-arp=yes address-pool=pool-UniFi dhcp-option-set=dhcp-options interface=vlan50-UniFi lease-time=1d name=dhcp-UniFi
/ip ipsec mode-config
add address-pool=pool-Staff-VPN address-prefix-length=32 name=modeconfig-Staff-VPN split-dns="pipastudios.com,adf.azure.com,blob.core.windows.net,database.windows.net,web.core.windows.ne\
    t,privatelink.web.core.windows.net,privatelink.blob.core.windows.net,privatelink.database.windows.net" split-include=10.0.0.0/8 static-dns=10.100.90.10,10.200.1.10 system-dns=no
/ip address
add address=10.41.0.1/24 interface=bridge-Loopback network=10.41.0.0
add address=169.254.248.26/30 comment="AWS DirectConnect" interface=vlan-HostFiber-AWS-993 network=169.254.248.24
add address=192.168.100.129/30 comment="Azure Express Route" interface=vlan-HostFiber-Azure-1-995 network=192.168.100.128
add address=192.168.100.133/30 interface=vlan-HostFiber-Azure-2-1011 network=192.168.100.132
add address=1.2.3.165/26 comment="HostFiber IP" disabled=yes interface=vlan-HostFiber-IP-2836 network=1.2.3.128
add address=1.2.3.166/26 disabled=yes interface=vlan-HostFiber-IP-2836 network=1.2.3.128
add address=1.2.3.171/26 disabled=yes interface=vlan-HostFiber-IP-2836 network=1.2.3.128
add address=1.2.3.177/26 interface=vlan-HostFiber-IP-2836 network=1.2.3.128
add address=10.10.0.1/24 interface=bridge-LAN network=10.10.0.0
add address=10.20.0.1/24 interface=vlan20-Guest network=10.20.0.0
add address=10.30.0.1/24 interface=vlan30-Doors network=10.30.0.0
add address=10.31.0.1/24 interface=vlan31-IoT network=10.31.0.0
add address=10.40.0.1/16 interface=vlan40-Staff network=10.40.0.0
add address=10.50.0.1/24 interface=vlan50-UniFi network=10.50.0.0
add address=10.100.80.1/24 interface=vlan80-iDRAC network=10.100.80.0
add address=10.100.90.1/24 interface=vlan90-Management network=10.100.90.0
add address=10.100.100.1/24 interface=vlan100-Servers network=10.100.100.0
add address=10.100.110.1/24 interface=vlan100-Servers network=10.100.110.0
/ip dhcp-server lease
add address=10.40.244.192 client-id=1:0:d7:6d:59:f7:a4 mac-address=00:D7:6D:59:F7:A4 server=dhcp-Staff
/ip dhcp-server network
add address=10.10.0.0/24 dhcp-option-set=dhcp-options dns-server=10.100.90.10,10.200.1.10 domain=pipastudios.com gateway=10.10.0.1
add address=10.20.0.0/24 dns-server=10.100.90.10,8.8.8.8,1.1.1.1 gateway=10.20.0.1
add address=10.30.0.0/24 dhcp-option-set=dhcp-options dns-server=10.100.90.10,10.200.1.10 domain=pipastudios.com gateway=10.30.0.1
add address=10.31.0.0/24 dhcp-option-set=dhcp-options dns-server=10.100.90.10,10.200.1.10 domain=pipastudios.com gateway=10.31.0.1
add address=10.40.0.0/16 dhcp-option-set=dhcp-options dns-server=10.100.90.10,10.200.1.10 domain=pipastudios.com gateway=10.40.0.1
add address=10.50.0.0/24 dhcp-option-set=dhcp-options dns-server=10.100.90.10,10.200.1.10 domain=pipastudios.com gateway=10.50.0.1
/ip dns
set servers=8.8.8.8,1.1.1.1
/ip firewall address-list
add address=10.10.0.0/24 list=bgp-Networks
add address=10.41.0.0/24 list=bgp-Networks
add address=10.100.90.0/24 list=bgp-Networks
add address=10.100.100.0/24 list=bgp-Networks
add address=10.40.0.0/16 list=bgp-Networks
add address=10.100.110.0/24 list=bgp-Networks
add address=10.100.80.0/24 list=bgp-Networks
add address=1.2.3.177 list=vpn-IPs
add address=4.5.6.77 list=vpn-IPs
add address=10.40.0.3 list=printers
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip firewall filter
add action=passthrough chain=forward comment=IN in-interface-list=WAN
add action=passthrough chain=forward comment=OUT out-interface-list=WAN
add action=passthrough chain=forward comment="IN / OUT"
add action=drop chain=forward comment="Drop dumb" src-address=10.40.244.192
add action=drop chain=input src-address=10.40.244.192
add action=drop chain=output src-address=10.40.244.192
add action=fasttrack-connection chain=forward comment=FastTrack connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
add action=drop chain=input comment="Drop DNS Requests" dst-port=53 in-interface-list=WAN protocol=tcp
add action=drop chain=input dst-port=53 in-interface-list=WAN protocol=udp
add action=drop chain=forward comment="Drop Guest" disabled=yes dst-address-list=printers src-address=10.20.0.0/24
add action=accept chain=input comment=Winbox dst-port=8223 in-interface=all-vlan protocol=tcp
add action=accept chain=input dst-port=8223 in-interface-list=WAN protocol=tcp
add action=accept chain=input comment="Established / Related" connection-state=established,related
add action=accept chain=input comment=IPSec dst-address-list=vpn-IPs dst-port=4500,500 protocol=udp
add action=accept chain=input dst-address-list=vpn-IPs protocol=ipsec-esp
add action=drop chain=input comment="Drop invalid" connection-state=invalid
add action=drop chain=input comment="-- DROP ALL --" in-interface-list=WAN
/ip firewall nat
add action=src-nat chain=srcnat disabled=yes out-interface=vlan-HostFiber-IP-2836 to-addresses=1.2.3.165
add action=src-nat chain=srcnat disabled=yes out-interface=pppoe-Vivo to-addresses=4.5.6.77
add action=dst-nat chain=dstnat comment="iPerf3 Internet Test" disabled=yes dst-port=959 protocol=tcp to-addresses=10.10.0.240
add action=accept chain=srcnat ipsec-policy=out,ipsec
add action=masquerade chain=srcnat out-interface-list=WAN
/ip ipsec identity
add auth-method=eap-radius certificate="Pipa VPN,GoDaddy,GoDaddy_1,GoDaddy_2" generate-policy=port-strict mode-config=modeconfig-Staff-VPN peer=peer-Staff-VPN-HostFiber \
    policy-template-group=group-Staff-VPN
/ip ipsec policy
add dst-address=10.0.0.0/8 group=group-Staff-VPN proposal=proposal-Staff-VPN src-address=0.0.0.0/0 template=yes
/ip route
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=1.1.1.1 routing-table=main scope=30 suppress-hw-offload=no target-scope=11
add comment="HostFiber monitor" disabled=no distance=1 dst-address=8.8.8.8/32 gateway=1.2.3.129 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=8.8.8.8 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=11
add comment="Vivo monitor" dst-address=1.1.1.1 gateway=4.5.6.49 scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh port=8222
set api disabled=yes
set winbox port=8223
set api-ssl disabled=yes

Last edited by BartoszP on Wed Apr 19, 2023 10:02 pm, edited 1 time in total.
Reason: Use proper tags: quote to quote, code for code - keep forum tidy
 
expo
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Tue Jan 27, 2009 7:57 am

Re: After enabling hardware offloading, console and mgmt stops responding

Fri Apr 07, 2023 12:01 am

Having similar behavior on the 2216 locking up and watchdog is enabled. Not even console will respond. Still investigating config.
 
webtelza
just joined
Posts: 10
Joined: Fri Feb 24, 2023 5:21 pm

Re: After enabling hardware offloading, console and mgmt stops responding

Wed May 10, 2023 7:15 pm

Hello

Did you find a solution to the router locking up?

I experienced this last week when L3HW offload was enabled. After a few minutes I was disconnected from router and it was totally unresponsive. No access via console cable or ethernet cable. We had to reboot the router and restore last backup (disabling L3HW offloading did not seem to make a difference as the router hanged a second time).

ROS in use is ROS 7.7
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: After enabling hardware offloading, console and mgmt stops responding

Thu May 11, 2023 9:51 am

Hi,

We have fixed multiple l3hw-related issues in RouterOS v7.10. I suggest giving it a try (in a lab - please do not use betas in production!)

Who is online

Users browsing this forum: tiernano and 25 guests