L3HW not working properly

willr · Mon Nov 22, 2021 3:13 pm

Hi there,

Have a CRS317-1G-16S+. A few weeks ago I tested L3HW and was getting ~10Gbps in each direction using iperf3. There are no firewall rules in place, and the device exists for inter-VLAN routing via the bridge.

A few weeks later I'm now testing again and finding that throughput is way lower - more like 700-1000 Mbps. It seems to be hitting the CRS CPU too, so I think L3HW is really not working right now.

The only thing that has been changed in that time is VLAN membership of various ports (i.e. /interface/vlan). BUT on the L3HW documentation page, it does say "It is recommended to turn off L3HW offloading during L2 configuration.". Unfortunately I'm pretty certain we didn't do that before changing the VLAN membership!

Therefore I realise we haven't followed instructions correctly, so maybe this has caused the drop in throughput?

What I would like to know is - how do I remedy the problems that might have been caused by not following the guidance? Can I just do something simple like reboot the CRS device (my preference!)? Or do I need to reconfigure the ports (/interface/vlan) from scratch? Or (worse) do I need to clear the config and start again from a clean device??

Switch is currently running v7.1.4rc4.

Thank you in advance!

Will

willr · Mon Nov 22, 2021 3:25 pm

Here's my config by the way (removed serial number and MAC).

The routing I'm doing is from VLAN 4 (10.9.4.0/24) to VLAN 6 (10.9.6.0/24).

# feb/23/1970 19:07:02 by RouterOS 7.1rc4
# software id = TSFV-NY6N
#
# model = CRS317-1G-16S+
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf mtu=9000 name=\
    bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus1 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus2 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus3 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus4 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus5 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus6 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus7 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus8 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus9 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus10 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus11 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus12 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus13 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus14 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus15 ] l2mtu=9022 mtu=9000
set [ find default-name=sfp-sfpplus16 ] l2mtu=9022 mtu=9000
/interface vlan
add interface=bridge name=vlan2 vlan-id=2
add interface=bridge name=vlan4 vlan-id=4
add interface=bridge name=vlan6 vlan-id=6
add interface=bridge name=vlan99 vlan-id=99
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus4 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus5 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus6 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus7 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus14 pvid=6
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface bridge vlan
add bridge=bridge tagged="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus13,sfp-sfpplus1\
    4,sfp-sfpplus15,sfp-sfpplus16" vlan-ids=3
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=5
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=7
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=8
add bridge=bridge tagged=sfp-sfpplus3,sfp-sfpplus13,sfp-sfpplus14 vlan-ids=50
add bridge=bridge tagged=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=2
add bridge=bridge tagged=bridge,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=6
add bridge=bridge tagged=\
    bridge,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16 vlan-ids=4
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 vlan-ids=99
add bridge=bridge tagged=sfp-sfpplus4,sfp-sfpplus13,sfp-sfpplus14 vlan-ids=51
add bridge=bridge tagged="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus13,sfp-sfpplus1\
    4,sfp-sfpplus15,sfp-sfpplus16" vlan-ids=10
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
add address=10.9.4.254/24 interface=vlan4 network=10.9.4.0
add address=10.9.6.254/24 interface=vlan6 network=10.9.6.0
add address=10.9.99.1/24 interface=vlan99 network=10.9.99.0
add address=10.9.2.250/24 interface=vlan2 network=10.9.2.0
/ip dhcp-relay
add dhcp-server=10.9.6.11 disabled=no interface=vlan4 name=relay1
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.9.99.254 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/system routerboard settings
set boot-os=router-os

willr · Tue Nov 23, 2021 12:46 pm

Well, I didn't change the config, but I rebooted the switch and upgraded to rc6, and L3HW is working again now.

Now I'm just really looking forward to L3HW working with both conntrack offload and bridge at the same time! I understand it's in development...

Wed Nov 24, 2021 10:26 am

Hi, and sorry for a late reply,

I'm glad that the problem got resolved by upgrading to v7.1rc6. Rc4 had an issue with MTU offloading so that most likely was the case.

Regarding:

It is recommended to turn off L3HW offloading during L2 configuration.

Configuring L2 while L3HW is enabled does not cause any permanent effects - rebooting should fix all the problems. The recommendation is given due to various edge cases where L2/L3/SW/HW configurations can be out-of-sync with each other. Currently, we don't have time to address all those minor issues due to the focus on stabilizing v7.1, then Fasttrack offloading on VLAN-filtered bridge and L3HW IPv6. Turning off L3HW while configuring L2 is a temporary safety switch.

willr · Mon Dec 13, 2021 3:50 pm

Sadly this problem has come back again with rc6

CPU is maxed out with routing, and L3 isn't being used. Looks like it's a bit of a time-bomb - things works fine for a while and then suddenly L3HW breaks!

I will update to v7.1 release tonight, but this time I've noticed the log is full of the following -

13:46:06 route,warning L3HW: FDB host 10.9.4.67 offload FAILED (-14)
13:46:09 route,warning L3HW: FDB host 10.9.4.50 offload FAILED (-14)
13:46:14 route,warning L3HW: FDB host 10.9.4.26 offload FAILED (-14)
13:46:18 route,warning L3HW: FDB host 10.9.4.67 offload FAILED (-14)
13:46:23 route,warning L3HW: FDB host 10.9.4.69 offload FAILED (-14)
13:46:33 route,warning L3HW: FDB host 10.9.4.67 offload FAILED (-14)
13:46:53 route,warning L3HW: FDB host 10.9.4.68 offload FAILED (-14)
13:47:03 route,warning L3HW: FDB host 10.9.4.68 offload FAILED (-14)
13:47:15 route,warning L3HW: FDB host 10.9.4.68 offload FAILED (-14)

I guess this was the same last time but I didn't check properly!

Config is the same as I pasted above - nothing has changed at all.

Any help/advice appreciated!

Mon Dec 13, 2021 4:55 pm

"-14" is an internal error code, meaning that the L3HW driver is turning off or restarting. Were you rebooting the router or setting "l3-hw-offloading=no" when those log messages appeared?

willr · Mon Dec 13, 2021 5:19 pm

No changes, that just happens with it running. The config is the same since the firmware upgrade to rc6 on 22nd Nov.

That part of log above was just an excerpt - the log is entirely full of those messages from the last 3 hours (except for a few showing when I logged in).

The reason I logged in was to investigate the low inter-VLAN throughput - in other words the problem was already there before I even logged in.

Wed Dec 15, 2021 9:50 am

If a host offload fails, then the traffic to that host (IP) goes via CPU, and, therefore, causes a performance drop. Are those hosts in the log real, or are you running a network test by sending packets to random destinations in the subnet? Hosts can be offloaded only after resolving ARP (IP-MAC). Hence, "fake hosts" cannot be offloaded and always go through the CPU.

Also, when the problem occurs, please check the result of the following command:

/in/bridge/host print count-only

joshedmonds · Wed Dec 22, 2021 12:07 am

I have the same problem running 7.1 stable on a CRS309. L3 hardware offloading will just stop randomly and requires a power cycle and/or toggling the l3-hw-offloading setting to get it going again.

willr · Mon Jan 24, 2022 11:58 am

If a host offload fails, then the traffic to that host (IP) goes via CPU, and, therefore, causes a performance drop. Are those hosts in the log real, or are you running a network test by sending packets to random destinations in the subnet? Hosts can be offloaded only after resolving ARP (IP-MAC). Hence, "fake hosts" cannot be offloaded and always go through the CPU.

Also, when the problem occurs, please check the result of the following command:
Code: Select all
/in/bridge/host print count-only

Sorry for the delay in replying.

Current state - I updated to 7.1 stable on 13th Dec.

To answer your question - the hosts are indeed real - mostly desktop PCs with real MAC addresses (and a couple of server NICs for Hyper-V). I should point out that this is a pretty busy office network, so if there's a rare case that causes L3HW to fail, it might be happening more often for me because of the high traffic (compared to a SOHO environment).

The issue reoccurred today but I'm afraid I forgot to run the command you requested. I'll do that next time. This time I tried toggling L3HW off and then on, and that resolved it per joshedmonds message. Nice not to have to schedule a switch reboot!

I should point out that this time unfortunately the switch didn't log the 'offload failed (-14)' messages. The only clue was that the CPU was mostly at 100% and inter-VLAN throughput was being reported as low.

dvdhngs · Mon Feb 27, 2023 10:23 pm

Hi... old topic, but its almost my case, the difference its i'm on 7.8 stable...

CRS326-24G-2S+

did you solved your problem?

i'm with 10 vlans (interface/vlan)
600 devices on /in/bridge/host print count-only
(+/- 20 nvr and 400 ip cameras)
and everything using cpu (800mbps on bridge and 98% cpu)

any tip?
thanks in advance

blacksnow · Tue Feb 28, 2023 6:21 am

It may be working on previous versions <7.6 but any version > 7.6 it is not working. Please check this thread and add your situation or send a support ticket as others have identified this as not working as well.

viewtopic.php?t=193770

sirbryan · Tue Feb 28, 2023 7:29 pm

Hi... old topic, but its almost my case, the difference its i'm on 7.8 stable...

CRS326-24G-2S+

i'm with 10 vlans (interface/vlan)
600 devices on /in/bridge/host print count-only
and everything using cpu (800mbps on bridge and 98% cpu)

Basic L3HW offload is working on 7.4.1, 7.6, and 7.7 on CRS310, 317, and CCR2116.
Configure all your Layer 2 stuff first:
1) Configure port L2MTU's, configure bonds (LACP)
2) Build your bridge, add ports to bridge
3) Create VLANs (assigned to bridge), change VLAN MTUs (if not 1500), assign IP's to VLANs, enable VLANs on bridge ports
4) Enable bridge VLAN filtering (unless all VLANs are always available to all ports)

Configure OSPF, BGP, static routes, etc. as normal.

Check that Layer 2 and basic routing on Layer 3 are working. Most CRS300's can handle up to a gigabit of traffic in the CPU.

Once you've confirmed everything is working, enable L3HW offload on the switch (/interface/ethernet/switch) and on the switch ports (/interface/ethernet/switch/ports). In the case of my CRS300's, I enable it on all ports. I don't want the CPU handling anything it doesn't need to.

If you ever make changes to the switch config (new ports, new VLAN's with new IP's), you may need to disable L3HW offload on the switch interface, and then enable it again. This causes the OS to push the config down to the ASIC. Sometimes I find I have to disable it on all the ports as well as the switch, then re-enable it on everything again.

Some have noticed certain routes stop working and have scheduled the switch to disable/enable every day or so. I have the CRS317 scheduled to reload every two hours. The CRS310's don't seem to have the same problem, but then they handle fewer network changes and route less traffic.

Also, there is currently a bug if you use ECMP (equal cost multi-path routing). When the routes fail and come back, it doesn't push any of them to the ASIC. The workaround is to either manually disable/enable L3HW offload, or to change the cost of one of the routes, essentially breaking the load sharing benefits of ECMP. They have reproduced the problem and a fix is slated for a future release.

misaka818 · Fri Sep 06, 2024 9:20 am

Same issue on CRS317-1G-16S+ with l3hw, using Inter-VLAN.
RouterOS 7.15.3
After rebooting the switch, things become normal but several days or even hours later, L3HW broke again.
Considering switch to Cisco or Juniper

 14:00:15 route,warning L3HW: FDB host 10.10.60.16 offload FAILED (-14)
 14:00:15 route,warning L3HW: FDB host 10.10.100.65 offload FAILED (-14)
 14:00:15 route,warning L3HW: FDB host 10.10.60.20 offload FAILED (-14)
 14:00:19 route,warning L3HW: FDB host 10.10.60.15 offload FAILED (-14)
 14:00:30 route,warning L3HW: FDB host 10.10.60.194 offload FAILED (-14)
 14:00:33 route,warning L3HW: FDB host 10.10.60.14 offload FAILED (-14)
 14:00:36 route,warning L3HW: FDB host 10.10.60.197 offload FAILED (-14)
 14:00:44 route,warning L3HW: FDB host 10.10.60.11 offload FAILED (-14)
 14:01:20 route,warning L3HW: FDB host 10.10.50.50 offload FAILED (-14)
 14:01:46 route,warning L3HW: FDB host 10.10.50.5 offload FAILED (-14)

Here is my config

# 2024-09-06 14:12:19 by RouterOS 7.15.3
# software id = NZIX-36BC
#
# model = CRS317-1G-16S+
# serial number = D7EC0F2B1FEC
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf dhcp-snooping=yes \
    name=main-bridge port-cost-mode=short vlan-filtering=yes
add add-dhcp-option82=yes dhcp-snooping=yes name=mgmt-bridge port-cost-mode=\
    short
/interface ethernet
set [ find default-name=ether1 ] comment=MGMT-Interface
set [ find default-name=sfp-sfpplus1 ] comment=NAS l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus2 ] comment=NAS l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus3 ] comment="Uplink RB5009" l2mtu=9100 \
    mtu=9000
set [ find default-name=sfp-sfpplus4 ] l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus5 ] l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus6 ] l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus7 ] l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus8 ] l2mtu=1600
set [ find default-name=sfp-sfpplus9 ] comment=Inspur-DvSwitch l2mtu=9100 \
    mtu=9000
set [ find default-name=sfp-sfpplus10 ] comment=Inspur-DvSwitch l2mtu=9100 \
    mtu=9000
set [ find default-name=sfp-sfpplus11 ] comment=Dell-DvSwitch l2mtu=9100 mtu=\
    9000
set [ find default-name=sfp-sfpplus12 ] comment=Dell-DvSwitch l2mtu=9100 mtu=\
    9000
set [ find default-name=sfp-sfpplus13 ] comment="Second Floor" l2mtu=9100 \
    mtu=9000
set [ find default-name=sfp-sfpplus14 ] comment="Second Floor" l2mtu=9100 \
    mtu=9000
set [ find default-name=sfp-sfpplus15 ] comment=Downlink l2mtu=9100 mtu=9000
set [ find default-name=sfp-sfpplus16 ] comment=Downlink l2mtu=9100 mtu=9000
/interface vlan
add interface=main-bridge name=vlan10_main vlan-id=10
add interface=main-bridge name=vlan20_home vlan-id=20
add interface=main-bridge name=vlan30_public vlan-id=30
add interface=main-bridge name=vlan50_server vlan-id=50
add interface=main-bridge name=vlan60_IoT vlan-id=60
add interface=main-bridge name=vlan80_kube vlan-id=80
add interface=main-bridge name=vlan90-kube-frr vlan-id=90
add interface=main-bridge name=vlan100_mgmt vlan-id=100
add interface=main-bridge name=vlan110-ac-ap vlan-id=110
add interface=main-bridge name=vlan200_mgmt_highspeed vlan-id=200
add interface=main-bridge name=vlan4000-lan-only vlan-id=4000
/interface bonding
add comment=Inspur mode=802.3ad mtu=9000 name=ESXi-Inspur slaves=\
    sfp-sfpplus9,sfp-sfpplus10
add comment=NAS mode=802.3ad mtu=9000 name=NAS_Bond slaves=\
    sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
add comment="ZTE Switch" mode=802.3ad mtu=9000 name=ZXR10_Bond slaves=\
    sfp-sfpplus15,sfp-sfpplus16
/interface ethernet switch
set 0 l3-hw-offloading=yes name=swtich1
/interface ethernet switch port
set 0 storm-rate=1
set 1 storm-rate=1
set 2 storm-rate=1
set 3 storm-rate=1
set 4 storm-rate=1
set 5 storm-rate=1
set 6 storm-rate=1
set 7 storm-rate=1
set 8 storm-rate=1
set 9 storm-rate=1
set 10 storm-rate=1
set 11 storm-rate=1
set 12 storm-rate=1
set 13 storm-rate=1
set 14 storm-rate=1
set 15 storm-rate=1
set 16 storm-rate=1
set 17 storm-rate=1
/interface list
add name=WAN
add name=bypass-if
add exclude=bypass-if include=all name=LAN
add name=MGMT
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server option
add code=60 name=tp-link-ap value="'TP-LINK'"
add code=138 name=tp-link-ac value="'10.10.100.100'"
/ip dhcp-server option sets
add name=tp-link-wireless options=tp-link-ap,tp-link-ac
/ip pool
add name=vlan10_pool ranges=192.168.10.30-192.168.10.240
add name=vlan20_pool ranges=192.168.20.20-192.168.20.240
add name=vlan30_pool ranges=192.168.30.20-192.168.30.240
add name=vlan50_pool ranges=10.10.50.50-10.10.50.100
add name=ovpn_pool ranges=10.8.10.50-10.8.10.150
add name=vlan100_pool ranges=10.10.100.150-10.10.100.210
add name=vlan60_pool ranges=10.10.60.10-10.10.60.200
add name=vlan15_pool ranges=10.10.15.100-10.10.15.150
add name=vlan111_pool ranges=192.168.111.15-192.168.111.200
add name=vlan4000-pool ranges=192.168.250.10-192.168.250.250
add name=vlan110-pool ranges=10.10.110.5-10.10.110.250
add name=dhcp_pool12 ranges=10.88.10.2-10.88.10.254
/ip dhcp-server
add address-pool=vlan10_pool interface=vlan10_main lease-time=12h name=\
    dhcp_vlan10
add address-pool=vlan20_pool interface=vlan20_home lease-time=12h name=\
    dhcp_vlan20
add address-pool=vlan30_pool interface=vlan30_public lease-time=1h name=\
    dhcp_vlan30
add address-pool=vlan50_pool interface=vlan50_server name=dhcp_vlan50
add address-pool=vlan100_pool interface=vlan100_mgmt name=dhcp_vlan100
add address-pool=vlan60_pool interface=vlan60_IoT lease-time=5214w2d name=\
    dhcp_vlan60
add address-pool=vlan4000-pool interface=vlan4000-lan-only name=dhcp_vlan4000
add address-pool=vlan110-pool interface=vlan110-ac-ap name=dhcp-vlan110
add address-pool=dhcp_pool12 interface=mgmt-bridge name=dhcp_mgmt
/ip smb users
set [ find default=yes ] disabled=yes
/ipv6 pool
add name=ipv6-ula-pool prefix=fdfd:2000::/24 prefix-length=64
/port
set 0 name=serial0
/routing bgp template
set default disabled=yes routing-table=main
add as=65501 disabled=no hold-time=3s keepalive-time=1s multihop=yes name=\
    k8s-default routing-table=main
add as=65510 disabled=no hold-time=3s keepalive-time=1s multihop=yes name=\
    calico routing-table=main
/routing ospf instance
add disabled=yes name=bird router-id=10.10.15.1 routing-table=main
/routing ospf area
add disabled=yes instance=bird name=bird
/interface bridge port
add bridge=mgmt-bridge comment=defconf interface=ether1 internal-path-cost=10 \
    path-cost=10
add bridge=main-bridge comment=defconf disabled=yes frame-types=\
    admit-only-untagged-and-priority-tagged interface=sfp-sfpplus1 \
    internal-path-cost=10 path-cost=10 pvid=50
add bridge=main-bridge comment=defconf disabled=yes interface=sfp-sfpplus2 \
    internal-path-cost=10 path-cost=10 pvid=50
add bridge=main-bridge comment=defconf interface=sfp-sfpplus3 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf interface=sfp-sfpplus4 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged interface=sfp-sfpplus5 \
    internal-path-cost=10 path-cost=10 pvid=10
add bridge=main-bridge comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged interface=sfp-sfpplus6 \
    internal-path-cost=10 path-cost=10 pvid=10
add bridge=main-bridge comment=defconf interface=sfp-sfpplus7 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf interface=sfp-sfpplus8 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf disabled=yes interface=sfp-sfpplus9 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf disabled=yes interface=sfp-sfpplus10 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf interface=sfp-sfpplus11 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf interface=sfp-sfpplus12 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged interface=sfp-sfpplus13 \
    internal-path-cost=10 path-cost=10 pvid=10
add bridge=main-bridge comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged interface=sfp-sfpplus14 \
    internal-path-cost=10 path-cost=10 pvid=10
add bridge=main-bridge comment=defconf disabled=yes interface=sfp-sfpplus15 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge comment=defconf disabled=yes interface=sfp-sfpplus16 \
    internal-path-cost=10 path-cost=10
add bridge=main-bridge interface=ZXR10_Bond internal-path-cost=10 path-cost=\
    10
add bridge=main-bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=NAS_Bond internal-path-cost=10 path-cost=10 pvid=50
add bridge=main-bridge interface=ESXi-Inspur internal-path-cost=10 path-cost=\
    10
/interface bridge port-controller
set bridge=main-bridge switch=swtich1
/interface bridge settings
set use-ip-firewall=yes use-ip-firewall-for-vlan=yes
/interface ethernet switch l3hw-settings
set ipv6-hw=yes
/ip firewall connection tracking
set udp-timeout=10s
/ipv6 settings
set accept-router-advertisements=no
/interface bridge vlan
add bridge=main-bridge tagged=main-bridge,ZXR10_Bond,ESXi-Inspur vlan-ids=10
add bridge=main-bridge tagged=ZXR10_Bond,ESXi-Inspur,*1F,main-bridge \
    untagged=ether1 vlan-ids=20
# sfp-sfpplus9,sfp-sfpplus10 not a bridge port
add bridge=main-bridge tagged=\
    ZXR10_Bond,main-bridge,ESXi-Inspur,sfp-sfpplus9,sfp-sfpplus10 untagged=\
    NAS_Bond vlan-ids=50
add bridge=main-bridge tagged=main-bridge,ZXR10_Bond,ESXi-Inspur untagged=\
    ether1 vlan-ids=30
# sfp-sfpplus9,sfp-sfpplus10 not a bridge port
add bridge=main-bridge tagged=\
    main-bridge,ZXR10_Bond,ESXi-Inspur,sfp-sfpplus9,sfp-sfpplus10 vlan-ids=\
    100
# sfp-sfpplus9,sfp-sfpplus10 not a bridge port
add bridge=main-bridge tagged=\
    main-bridge,ZXR10_Bond,ESXi-Inspur,sfp-sfpplus9,sfp-sfpplus10 untagged=\
    ether1 vlan-ids=60
# sfp-sfpplus9,sfp-sfpplus10 not a bridge port
add bridge=main-bridge comment=Bypass tagged=\
    ZXR10_Bond,sfp-sfpplus3,ESXi-Inspur,sfp-sfpplus9,sfp-sfpplus10 vlan-ids=\
    15
# sfp-sfpplus9,sfp-sfpplus10 not a bridge port
add bridge=main-bridge tagged=\
    main-bridge,ZXR10_Bond,ESXi-Inspur,sfp-sfpplus9,sfp-sfpplus10 untagged=\
    ether1 vlan-ids=80
add bridge=main-bridge tagged=ZXR10_Bond,ESXi-Inspur,main-bridge untagged=\
    sfp-sfpplus3 vlan-ids=4000
add bridge=main-bridge tagged=ZXR10_Bond,ESXi-Inspur,*1F,main-bridge \
    untagged=ether1 vlan-ids=110
# sfp-sfpplus9,sfp-sfpplus10 not a bridge port
add bridge=main-bridge comment=VPN tagged=\
    ESXi-Inspur,sfp-sfpplus3,ZXR10_Bond,sfp-sfpplus9,sfp-sfpplus10 vlan-ids=\
    150
add bridge=main-bridge tagged=main-bridge,ZXR10_Bond,ESXi-Inspur untagged=\
    ether1 vlan-ids=90
add bridge=main-bridge tagged=main-bridge,ZXR10_Bond,ESXi-Inspur vlan-ids=200
/interface dot1x server
add disabled=yes guest-vlan-id=30 interface=sfp-sfpplus6 reauth-timeout=1s \
    reject-vlan-id=30 server-fail-vlan-id=30
add comment="Second Floor" disabled=yes guest-vlan-id=30 interface=\
    sfp-sfpplus13 server-fail-vlan-id=30
add disabled=yes interface=MGMT
add disabled=yes interface=sfp-sfpplus8
add comment="Second Floor" disabled=yes guest-vlan-id=30 interface=\
    sfp-sfpplus14 server-fail-vlan-id=30
/interface ethernet switch rule
add comment="Allow DNS" dst-port=53 ports="sfp-sfpplus3,ether1,sfp-sfpplus1,sf\
    p-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus\
    8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp\
    -sfpplus14,sfp-sfpplus15,sfp-sfpplus16" switch=swtich1
add comment="Allow mDNS" dst-port=5353 ports="sfp-sfpplus3,ether1,sfp-sfpplus1\
    ,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpp\
    lus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpplus13,\
    sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16" switch=swtich1
add comment="Allow 80 Port - 2" dst-address=10.70.100.10/32 dst-port=80 \
    ports="sfp-sfpplus3,ether1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpp\
    lus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp\
    -sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfp\
    plus16" switch=swtich1
add comment="Allow 443 Port - 2" dst-address=10.70.100.10/32 dst-port=443 \
    ports="sfp-sfpplus3,ether1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpp\
    lus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp\
    -sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfp\
    plus16" switch=swtich1
add comment="Allow 443 Port" dst-address=10.60.100.10/32 dst-port=443 ports="s\
    fp-sfpplus3,ether1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp\
    -sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus\
    11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16" \
    switch=swtich1
add comment="Allow 80 Port" dst-address=10.60.100.10/32 dst-port=80 ports="sfp\
    -sfpplus3,ether1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-s\
    fpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11\
    ,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16" \
    switch=swtich1
add comment=Block-not-K8S-Subnet dst-address=10.233.0.0/16 new-dst-ports="" \
    ports="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus5,sf\
    p-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplu\
    s11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16,\
    swtich1-cpu" src-address=192.168.0.0/16 switch=swtich1
add comment=Block-not-K8S-Subnet-IPv6-1 dst-address6=fd88::/48 mac-protocol=\
    ipv6 new-dst-ports="" ports="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sf\
    pplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sf\
    p-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sf\
    pplus15,sfp-sfpplus16,swtich1-cpu" switch=swtich1
add comment=Block-not-K8S-Subnet-IPv6-2 dst-address6=fd89::/108 mac-protocol=\
    ipv6 new-dst-ports="" ports="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sf\
    pplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sf\
    p-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sf\
    pplus15,sfp-sfpplus16,swtich1-cpu" switch=swtich1
add comment="Allow K8S-Subnet-Test" disabled=yes dst-address=10.233.0.0/16 \
    ports="sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus1\
    5,sfp-sfpplus16" switch=swtich1 vlan-id=50
add comment="Allow K8S-Subnet" disabled=yes dst-address=10.233.0.0/16 ports="s\
    fp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplu\
    s6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-\
    sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16,swtich1-\
    cpu" src-address=10.10.80.0/24 switch=swtich1
add comment="Allow K8S-Subnet-IPv6-1" disabled=yes dst-address6=fd88::/48 \
    mac-protocol=ipv6 ports="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplu\
    s4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sf\
    pplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplu\
    s15,sfp-sfpplus16,swtich1-cpu" src-address6=fdfd:2000:10:80::/64 switch=\
    swtich1 vlan-id=80
add comment="Allow K8S-Subnet-IPv6-2" disabled=yes dst-address6=fd89::/108 \
    mac-protocol=ipv6 ports="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplu\
    s4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus9,sfp-sf\
    pplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpplus13,sfp-sfpplus14,sfp-sfpplu\
    s15,sfp-sfpplus16,swtich1-cpu" src-address6=fdfd:2000:10:80::/64 switch=\
    swtich1
add comment="Allow DNS" disabled=yes dst-port=53 ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 switch=swtich1
add comment="Allow mDNS" disabled=yes dst-port=5353 ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 switch=swtich1
add comment="Allow Server 80" disabled=yes dst-address=10.60.100.0/24 \
    dst-port=80 ports=sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 \
    switch=swtich1
add comment="Allow Server 443" disabled=yes dst-address=10.60.100.0/24 \
    dst-port=443 ports=sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 \
    switch=swtich1
add comment="Allow Camera" disabled=yes dst-address=10.10.60.0/24 ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 src-address=\
    10.10.50.65/32 switch=swtich1
add comment="Allow Camera" disabled=yes dst-address=10.10.60.0/24 ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 src-address=\
    10.10.60.10/32 switch=swtich1
add comment="Allow Camera" disabled=yes dst-address=10.10.60.0/24 ports="sfp-s\
    fpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus13,sfp-sfpplu\
    s14" src-address=192.168.10.0/24 switch=swtich1
add comment="Block Camera" disabled=yes dst-address=10.10.60.196/32 \
    new-dst-ports="" ports="sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfppl\
    us10,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus13,sfp\
    -sfpplus14" switch=swtich1
add comment="AP Portal" disabled=yes dst-address=10.10.100.100/32 dst-port=\
    8080 ports=sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 switch=\
    swtich1
add comment="Test Rate Download" disabled=yes dst-address=192.168.10.0/24 \
    ports="sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus5\
    ,swtich1-cpu,sfp-sfpplus1,sfp-sfpplus2" rate=300.0Mbps switch=swtich1
add comment="Test Rate Upload" disabled=yes ports="sfp-sfpplus15,sfp-sfpplus16\
    ,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus5,swtich1-cpu,sfp-sfpplus1,sfp-sfpp\
    lus2" rate=100.0Mbps src-address=192.168.10.0/24 switch=swtich1
add comment="Block Server Subnet" disabled=yes dst-address=10.0.0.0/8 \
    new-dst-ports="" ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 src-address=\
    192.168.16.0/20 switch=swtich1
add comment="Block KubeNet Subnet" disabled=yes dst-address=10.10.80.0/24 \
    new-dst-ports="" ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 src-address=\
    192.168.16.0/20 switch=swtich1
add comment="Block Management" disabled=yes dst-address=10.10.100.0/24 \
    new-dst-ports="" ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10 src-address=\
    192.168.16.0/20 switch=swtich1
add comment="Allow Vlan 4000 Access Server" disabled=yes dst-address=\
    10.60.0.0/16 ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10,swtich1-cpu,ether1 \
    src-address=192.168.250.0/24 switch=swtich1
add comment="Allow Vlan 4000 Access Server 2" disabled=yes dst-address=\
    192.168.10.0/24 ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10,swtich1-cpu,ether1 \
    src-address=192.168.250.0/24 switch=swtich1
add comment="Block Vlan 4000" disabled=yes ports=\
    sfp-sfpplus15,sfp-sfpplus16,sfp-sfpplus9,sfp-sfpplus10,swtich1-cpu,ether1 \
    redirect-to-cpu=yes switch=swtich1 vlan-id=4000
add comment="VLAN 10 - Allow All" disabled=yes ports=\
    sfp-sfpplus15,sfp-sfpplus16 switch=swtich1 vlan-id=10
add comment="VLAN 50 - Allow All" disabled=yes ports=\
    sfp-sfpplus15,sfp-sfpplus16 switch=swtich1 vlan-id=50
add comment="Block Internal LAN" disabled=yes dst-address=10.0.0.0/8 \
    new-dst-ports="" ports=sfp-sfpplus15,sfp-sfpplus16 src-address=\
    192.168.0.0/16 switch=swtich1
add comment="Forbidden Router Admin" disabled=yes dst-address=172.16.10.1/32 \
    dst-port=80 new-dst-ports="" ports=sfp-sfpplus15,sfp-sfpplus16 \
    src-address=192.168.0.0/16 switch=swtich1
add comment="Forbidden Router SSH" disabled=yes dst-address=172.16.10.1/32 \
    dst-port=22 new-dst-ports="" ports=sfp-sfpplus15,sfp-sfpplus16 \
    src-address=192.168.0.0/16 switch=swtich1
add comment="Block Mikrotik Admin Page" disabled=yes dst-address=\
    192.168.20.1/32 dst-port=80 new-dst-ports="" ports=\
    sfp-sfpplus15,sfp-sfpplus16 src-address=192.168.0.0/16 switch=swtich1
add comment="VLAN 30 - Block Mikrotik Admin Page" disabled=yes dst-address=\
    192.168.30.1/32 dst-port=80 new-dst-ports="" ports=\
    sfp-sfpplus15,sfp-sfpplus16 src-address=192.168.0.0/16 switch=swtich1 \
    vlan-id=30
add comment="Forbidden Internet" disabled=yes dst-address=172.16.10.1/32 \
    new-dst-ports="" ports=sfp-sfpplus15,sfp-sfpplus16 switch=swtich1
add disabled=yes dst-port=67 mac-protocol=ip ports="sfp-sfpplus3,ether1,sfp-sf\
    pplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sf\
    p-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpp\
    lus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16" protocol=udp \
    redirect-to-cpu=yes src-port=68 switch=swtich1
add disabled=yes dst-port=68 mac-protocol=ip ports="sfp-sfpplus3,ether1,sfp-sf\
    pplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sf\
    p-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpp\
    lus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16" protocol=udp \
    redirect-to-cpu=yes src-port=67 switch=swtich1
add disabled=yes dst-port=67 mac-protocol=ip ports="sfp-sfpplus3,ether1,sfp-sf\
    pplus1,sfp-sfpplus2,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sf\
    p-sfpplus8,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp-sfpp\
    lus13,sfp-sfpplus14,sfp-sfpplus15,sfp-sfpplus16" protocol=udp \
    redirect-to-cpu=yes src-port=67 switch=swtich1
/interface list member
add interface=ether1 list=WAN
add interface=*25 list=bypass-if
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
add interface=sfp-sfpplus9 list=LAN
add interface=sfp-sfpplus10 list=LAN
add interface=sfp-sfpplus11 list=LAN
add interface=sfp-sfpplus12 list=LAN
add interface=sfp-sfpplus13 list=WAN
add interface=sfp-sfpplus14 list=WAN
add interface=sfp-sfpplus15 list=LAN
add interface=sfp-sfpplus16 list=LAN
/interface ovpn-server server
set auth=md5,sha256,sha512 certificate=server cipher="blowfish128,aes128-cbc,a\
    es192-cbc,aes256-cbc,aes128-gcm,aes192-gcm,aes256-gcm" default-profile=\
    OVPN_Profile enabled=yes port=32999 require-client-certificate=yes
/ip address
add address=172.16.10.5/24 comment=defconf interface=main-bridge network=\
    172.16.10.0
add address=192.168.10.1/24 interface=vlan10_main network=192.168.10.0
add address=192.168.20.1/24 interface=vlan20_home network=192.168.20.0
add address=10.10.50.1/24 interface=vlan50_server network=10.10.50.0
add address=192.168.30.1/24 interface=vlan30_public network=192.168.30.0
add address=10.10.100.1/24 interface=vlan100_mgmt network=10.10.100.0
add address=10.10.60.1/24 interface=vlan60_IoT network=10.10.60.0
add address=10.10.80.1/24 interface=vlan80_kube network=10.10.80.0
add address=192.168.250.1/24 interface=vlan4000-lan-only network=\
    192.168.250.0
add address=10.10.110.1/24 interface=vlan110-ac-ap network=10.10.110.0
add address=10.10.90.1/24 interface=vlan90-kube-frr network=10.10.90.0
add address=10.88.10.1/24 interface=mgmt-bridge network=10.88.10.0
add address=10.10.200.1/24 interface=vlan200_mgmt_highspeed network=\
    10.10.200.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add disabled=yes interface=main-bridge
/ip dhcp-server config
set radius-password=same-as-user
/ip dhcp-server lease
/ip dhcp-server network
add address=10.10.50.0/24 dns-server=172.16.10.1  gateway=\
    10.10.50.1 netmask=24 ntp-server=172.16.10.1
add address=10.10.60.0/24 dns-server=172.16.10.1  gateway=\
    10.10.60.1 netmask=24 ntp-server=172.16.10.1
add address=10.10.100.0/24 dns-server=172.16.10.1  \
    gateway=10.10.100.1 ntp-server=172.16.10.1
add address=10.10.110.0/24 dhcp-option-set=tp-link-wireless dns-server=\
    172.16.10.1  gateway=10.10.110.1 ntp-server=\
    172.16.10.1
add address=10.88.10.0/24 dns-server=172.16.10.1 gateway=10.88.10.1
add address=192.168.10.0/24 dns-server=172.16.10.1  \
    gateway=192.168.10.1 ntp-server=172.16.10.1
add address=192.168.20.0/24 dns-server=172.16.10.1  \
    gateway=192.168.20.1 ntp-server=172.16.10.1
add address=192.168.30.0/24 dns-server=10.10.150.50  \
    gateway=192.168.30.1 ntp-server=172.16.10.1
add address=192.168.111.0/24 dns-server=172.16.10.1  \
    gateway=192.168.111.1 ntp-server=172.16.10.1
add address=192.168.250.0/24 dns-server=172.16.10.1 gateway=192.168.250.1
/ip dns
set allow-remote-requests=yes cache-max-ttl=0s cache-size=4096KiB \
    max-concurrent-queries=1000 max-concurrent-tcp-sessions=200 \
    max-udp-packet-size=8192 query-server-timeout=10s query-total-timeout=20s \
    servers=172.16.10.1,fdfd:1000:10:150::50
/ip firewall address-list
add address=192.168.10.0/24 list=vlan10
add address=192.168.20.0/24 list=vlan20
add address=192.168.30.0/24 list=vlan30
add address=0.0.0.0/8 comment="defconf: RFC6890" list=no_forward_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=no_forward_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=no_forward_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=no_forward_ipv4
add address=192.168.233.233 list=BOT
add address=192.168.0.0/16 list=private
add address=10.0.0.0/8 list=private
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=\
    established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related \
    disabled=yes
add action=accept chain=forward disabled=yes in-interface=*1A
add action=accept chain=input disabled=yes in-interface=*1A
add action=accept chain=input comment="OVPN Pass" disabled=yes dst-port=32999 \
    protocol=tcp
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=yes dst-address-list=\
    !private in-interface=*25 new-routing-mark=*400 passthrough=yes
add action=mark-routing chain=prerouting disabled=yes dst-address=\
    10.10.15.0/24 new-routing-mark=*2 passthrough=yes src-address=0.0.0.0
add action=mark-routing chain=prerouting disabled=yes dst-address=0.0.0.0 \
    new-routing-mark=main passthrough=yes routing-mark=*2 src-address=\
    10.10.15.0/24
add action=mark-routing chain=prerouting disabled=yes dst-address-list=\
    !private log=yes new-routing-mark=*400 passthrough=yes src-address=\
    192.168.111.0/24
/ip firewall nat
add action=masquerade chain=srcnat disabled=yes out-interface-list=WAN
add action=masquerade chain=srcnat disabled=yes
/ip route
add comment="DO NOT DELETE" disabled=no distance=250 dst-address=0.0.0.0/0 \
    gateway=172.16.10.1 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
/ipv6 route
add blackhole comment="defconf: RFC6890 - loopback address" disabled=yes \
    distance=1 dst-address=::1/128 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - unspecified address" disabled=yes \
    distance=1 dst-address=::/128 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - IPv4-IPv6 translate" disabled=yes \
    distance=1 dst-address=64:ff9b::/96 gateway="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - IPv4-mapped address" disabled=yes \
    distance=1 dst-address=/96 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - discard-only address block" \
    disabled=yes distance=1 dst-address=100::/64 gateway="" routing-table=\
    main scope=30 suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - IETF protocol assignments" \
    disabled=yes distance=1 dst-address=2001::/23 gateway="" routing-table=\
    main scope=30 suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - TEREDO" disabled=yes distance=1 \
    dst-address=2001::/32 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - benchmarking" disabled=yes \
    distance=1 dst-address=2001:2::/48 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - documentation" disabled=yes \
    distance=1 dst-address=2001:db8::/32 gateway="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - ORCHID" disabled=yes distance=1 \
    dst-address=2001:10::/28 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - 6to4" disabled=yes distance=1 \
    dst-address=2002::/16 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - unique-local" disabled=yes \
    distance=1 dst-address=fc00::/7 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add blackhole comment="defconf: RFC6890 - linked-scoped unicast" disabled=yes \
    distance=1 dst-address=fe80::/10 gateway="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=yes distance=50 dst-address=fdfd:6343:2e79::/48 gateway=\
    main-bridge routing-table=main scope=30 target-scope=10
add disabled=no distance=1 dst-address=fdfd:1000::/24 gateway=\
    fdfd::1%main-bridge routing-table=main scope=30 target-scope=10
/ip smb shares
set [ find default=yes ] directory=/flash/pub
/ip traffic-flow
set enabled=yes
/ipv6 address
add address=::1 from-pool=ipv6-pool interface=vlan10_main
add address=::1 from-pool=ipv6-pool interface=vlan100_mgmt
add address=::1 from-pool=ipv6-pool interface=vlan50_server
add address=::1 disabled=yes from-pool=ipv6-pool interface=vlan80_kube
add address=fdfd::2/16 advertise=no interface=main-bridge
add address=fdfd:2000:10:80::1 advertise=no interface=vlan80_kube
add address=fdfd:2000:10:50::1 advertise=no interface=vlan50_server
add address=fdfd:2000:10:100::1 advertise=no interface=vlan100_mgmt
add address=fdfd:2000:10:110::1 advertise=no interface=vlan110-ac-ap
add address=::1 disabled=yes from-pool=ipv6-pool interface=vlan60_IoT
add address=::1 from-pool=ipv6-pool interface=vlan200_mgmt_highspeed
/ipv6 dhcp-client
add add-default-route=yes default-route-distance=250 interface=main-bridge \
    pool-name=ipv6-pool request=prefix use-peer-dns=no
/ipv6 firewall address-list
add address=fdac::2/128 comment="lanconf: local DNS server" list=\
    local_dns_ipv6
add address=fdac::3/128 comment="lanconf: local DNS server" list=\
    local_dns_ipv6
add address=fe80::/10 comment="defconf: RFC6890 Linked-Scoped Unicast" list=\
    no_forward_ipv6
add address=ff00::/8 comment="defconf: multicast" list=no_forward_ipv6
add address=::1/128 comment="defconf: RFC6890 lo" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: RFC6890 IPv4 mapped" list=\
    bad_ipv6
add address=2001::/23 comment="defconf: RFC6890" list=bad_ipv6
add address=2001:db8::/32 comment="defconf: RFC6890 documentation" list=\
    bad_ipv6
add address=2001:10::/28 comment="defconf: RFC6890 orchid" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: RFC6890 Discard-only" list=\
    not_global_ipv6
add address=2001::/32 comment="defconf: RFC6890 TEREDO" list=not_global_ipv6
add address=2001:2::/48 comment="defconf: RFC6890 Benchmark" list=\
    not_global_ipv6
add address=fc00::/7 comment="defconf: RFC6890 Unique-Local" list=\
    not_global_ipv6
add address=::/128 comment="defconf: unspecified" list=bad_dst_ipv6
add address=::/128 comment="defconf: unspecified" list=bad_src_ipv6
add address=ff00::/8 comment="defconf: multicast" list=bad_src_ipv6
add address=::/128 comment="ddosconf: DDoS" list=ddos_targets_ipv6
add address=::/128 comment="ddosconf: DDoS" list=ddos_attackers_ipv6
/ipv6 firewall filter
add action=accept chain=forward disabled=yes
add action=accept chain=input disabled=yes
add action=accept chain=output disabled=yes
add action=accept chain=input comment="defconf: accept ICMPv6 after RAW" \
    protocol=icmpv6
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation" dst-port=546 log=yes \
    log-prefix="[ipv6-pd]" protocol=udp src-address=fe80::/10
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment="defconf: drop bad forward IPs" \
    src-address-list=no_forward_ipv6
add action=drop chain=forward comment="defconf: drop bad forward IPs" \
    dst-address-list=no_forward_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6 after RAW" \
    protocol=icmpv6
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=jump chain=forward comment="ddosconf: DDoS" connection-state=new \
    jump-target=detect-ddos
add action=return chain=detect-ddos comment="ddosconf: DDoS SYN-ACK Flood" \
    dst-limit=50,50,src-and-dst-addresses/10s log=yes log-prefix=\
    "[syn-ack-flood]" protocol=tcp tcp-flags=syn,ack
add action=return chain=detect-ddos comment="ddosconf: DDoS" dst-limit=\
    200,200,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddos_targets_ipv6 \
    address-list-timeout=10m chain=detect-ddos comment="ddosconf: DDoS"
add action=add-src-to-address-list address-list=ddos_attackers_ipv6 \
    address-list-timeout=10m chain=detect-ddos comment="ddosconf: DDoS" log=\
    yes log-prefix="[ddos-ipv6]"
/ipv6 firewall mangle
add action=change-mss chain=forward comment="defconf: fix IPv6 mss for WAN" \
    new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
/ipv6 firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade IPv6" \
    disabled=yes out-interface-list=WAN
add action=accept chain=dstnat comment=\
    "lanconf: accept local DNS server's query (UDP)" dst-port=53 \
    in-interface-list=LAN protocol=udp src-address-list=local_dns_ipv6
add action=accept chain=dstnat comment=\
    "lanconf: accept local DNS server's query (TCP)" dst-port=53 \
    in-interface-list=LAN protocol=tcp src-address-list=local_dns_ipv6
add action=redirect chain=dstnat comment="lanconf: redirect DNS query (UDP)" \
    dst-port=53 in-interface-list=LAN protocol=udp to-ports=53
add action=redirect chain=dstnat comment="lanconf: redirect DNS query (TCP)" \
    dst-port=53 in-interface-list=LAN protocol=tcp to-ports=53
/ipv6 firewall raw
add action=accept chain=prerouting comment=\
    "defconf: enable for transparent firewall" disabled=yes
add action=drop chain=prerouting comment="ddosconf: DDoS" dst-address-list=\
    ddos_targets_ipv6 src-address-list=ddos_attackers_ipv6
add action=drop chain=prerouting comment=\
    "defconf: drop IPv6 extension headers types 0,43" headers=\
    hop,route:contains
add action=accept chain=prerouting comment="defconf: RFC4291, section 2.7.1" \
    dst-address=ff02::1:ff00:0/104 icmp-options=135 protocol=icmpv6 \
    src-address=::/128
add action=drop chain=prerouting comment="defconf: drop bogon IPs" \
    src-address-list=bad_ipv6
add action=drop chain=prerouting comment="defconf: drop bogon IPs" \
    dst-address-list=bad_ipv6
add action=drop chain=prerouting comment=\
    "defconf: drop packets with bad SRC ipv6" src-address-list=bad_src_ipv6
add action=drop chain=prerouting comment=\
    "defconf: drop packets with bad DST ipv6" dst-address-list=bad_dst_ipv6
add action=drop chain=prerouting comment="defconf: drop non global from WAN" \
    in-interface-list=WAN src-address-list=not_global_ipv6
add action=drop chain=prerouting comment="defconf: drop UDP port 0" log=yes \
    log-prefix="[udp-port-0]" port=0 protocol=udp
add action=jump chain=prerouting comment="defconf: jump to ICMPv6 chain" \
    jump-target=icmp6 protocol=icmpv6
add action=jump chain=prerouting comment="defconf: jump to TCP chain" \
    jump-target=bad-tcp protocol=tcp
add action=accept chain=prerouting comment=\
    "defconf: accept local multicast scope" dst-address=ff02::/16
add action=drop chain=prerouting comment=\
    "defconf: drop other multicast destinations" dst-address=ff00::/8
add action=accept chain=prerouting comment=\
    "defconf: accept everything else from LAN" in-interface-list=LAN
add action=accept chain=prerouting comment=\
    "defconf: accept everything else from WAN" in-interface-list=WAN
add action=drop chain=prerouting comment="defconf: drop the rest"
add action=drop chain=bad-tcp comment="defconf: TCP flag filter" protocol=tcp \
    tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=bad-tcp comment="defconf: drop flags fin,syn" protocol=\
    tcp tcp-flags=fin,syn
add action=drop chain=bad-tcp comment="defconf: drop flags fin,rst" protocol=\
    tcp tcp-flags=fin,rst
add action=drop chain=bad-tcp comment="defconf: drop flags fin,!ack" \
    protocol=tcp tcp-flags=fin,!ack
add action=drop chain=bad-tcp comment="defconf: drop flags fin,urg" protocol=\
    tcp tcp-flags=fin,urg
add action=drop chain=bad-tcp comment="defconf: drop flags syn,rst" protocol=\
    tcp tcp-flags=syn,rst
add action=drop chain=bad-tcp comment="defconf: drop flags rst,urg" protocol=\
    tcp tcp-flags=rst,urg
add action=drop chain=bad-tcp comment="defconf: drop TCP port 0" log=yes \
    log-prefix="[tcp-port-0]" port=0 protocol=tcp
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 drop ll if hop-limit!=255" dst-address=fe80::/10 \
    hop-limit=not-equal:255 protocol=icmpv6
add action=accept chain=icmp6 comment="defconf: rfc4890 DST unreachable" \
    icmp-options=1:0-255 protocol=icmpv6
add action=accept chain=icmp6 comment="defconf: rfc4890 packet too big" \
    icmp-options=2:0-255 protocol=icmpv6
add action=accept chain=icmp6 comment="defconf: rfc4890 time exceeded" \
    icmp-options=3:0-1 protocol=icmpv6
add action=accept chain=icmp6 comment="defconf: rfc4890 parameter problem" \
    icmp-options=4:0-2 protocol=icmpv6
add action=accept chain=icmp6 comment="defconf: rfc4890 echo request" \
    icmp-options=128:0-255 protocol=icmpv6
add action=accept chain=icmp6 comment="defconf: rfc4890 echo response" \
    icmp-options=129:0-255 protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast listener query only LAN" icmp-options=\
    130:0-255 in-interface-list=LAN protocol=icmpv6 src-address=fe80::/10
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast listener query report only LAN" icmp-options=\
    131:0-255 in-interface-list=LAN protocol=icmpv6 src-address=fe80::/10
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast listener query done only LAN" icmp-options=\
    132:0-255 in-interface-list=LAN protocol=icmpv6 src-address=fe80::/10
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 router solic only LAN" hop-limit=equal:255 \
    icmp-options=133:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 router advert only LAN" hop-limit=equal:255 \
    icmp-options=134:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 neighbor solic only LAN" hop-limit=equal:255 \
    icmp-options=135:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 neighbor advert only LAN" hop-limit=equal:255 \
    icmp-options=136:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 inverse ND solic only LAN" hop-limit=equal:255 \
    icmp-options=141:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 inverse ND advert only LAN" hop-limit=equal:255 \
    icmp-options=142:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast listener report v2 only LAN" icmp-options=\
    143:0-255 in-interface-list=LAN protocol=icmpv6 src-address=fe80::/10
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 certificate path solicitation only LAN" hop-limit=\
    equal:255 icmp-options=148:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 certificate path advertisement only LAN" hop-limit=\
    equal:255 icmp-options=149:0-255 in-interface-list=LAN protocol=icmpv6
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast router advertisement only LAN" hop-limit=\
    equal:1 icmp-options=151:0-255 in-interface-list=LAN protocol=icmpv6 \
    src-address=fe80::/10
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast router solicitation only LAN" hop-limit=\
    equal:1 icmp-options=152:0-255 in-interface-list=LAN protocol=icmpv6 \
    src-address=fe80::/10
add action=accept chain=icmp6 comment=\
    "defconf: rfc4890 multicast router termination only LAN" hop-limit=\
    equal:1 icmp-options=153:0-255 in-interface-list=LAN protocol=icmpv6 \
    src-address=fe80::/10
add action=drop chain=icmp6 comment="defconf: drop all other ICMPv6" \
    protocol=icmpv6
/ipv6 nd
set [ find default=yes ] disabled=yes hop-limit=64 other-configuration=yes \
    ra-interval=1m-2m
add disabled=yes hop-limit=64 interface=main-bridge ra-interval=1m-2m
add dns=fdfd::1 hop-limit=64 interface=vlan10_main ra-interval=5m-15m \
    ra-lifetime=45m
add dns=fdfd::1 hop-limit=64 interface=vlan50_server ra-interval=5m-15m \
    ra-lifetime=45m
add dns=fdfd::1 hop-limit=64 interface=vlan110-ac-ap ra-interval=5m-15m \
    ra-lifetime=45m
add dns=fdfd::1 hop-limit=64 interface=vlan100_mgmt ra-interval=5m-15m \
    ra-lifetime=45m
add disabled=yes dns=fdfd::1 hop-limit=64 interface=vlan80_kube ra-interval=\
    5m-15m ra-lifetime=45m
add dns=fdfd::1 hop-limit=64 interface=vlan200_mgmt_highspeed ra-interval=\
    5m-15m ra-lifetime=45m
add dns=fdfd::1 hop-limit=64 interface=vlan60_IoT ra-interval=5m-15m \
    ra-lifetime=45m
/ipv6 nd prefix default
set preferred-lifetime=5m valid-lifetime=15m
/ppp profile
add bridge=*22 change-tcp-mss=yes dns-server=10.10.10.1 local-address=\
    10.8.10.1 name=OVPN_Profile remote-address=ovpn_pool use-encryption=yes \
    use-ipv6=no
/ppp secret
add name=misaka profile=OVPN_Profile service=ovpn
/radius
add address=10.10.50.12 require-message-auth=no service=\
    ppp,login,hotspot,wireless,dhcp,ipsec,dot1x
add accounting-backup=yes address=10.10.50.11 require-message-auth=no \
    service=ppp,login,hotspot,wireless,dhcp,ipsec,dot1x
/radius incoming
set accept=yes
/routing bgp connection
add address-families=ip,ipv6 as=65501 disabled=yes hold-time=3s \
    keepalive-time=1s local.role=ebgp multihop=yes name=kubernetes \
    remote.address=10.10.80.0/24 .as=65500 router-id=10.10.80.1 \
    routing-table=main templates=k8s-default
add address-families=ip,ipv6 as=65502 disabled=yes local.role=ebgp multihop=\
    yes name=k8s-frr remote.address=10.10.90.0/24 .as=65500 router-id=\
    10.10.90.1 routing-table=main templates=k8s-default
add address-families=ip as=65510 disabled=no hold-time=3s keepalive-time=1s \
    local.role=ibgp multihop=yes name=calico remote.address=10.10.80.0/24 \
    .as=65510 router-id=10.10.80.1 routing-table=main templates=calico
add address-families=ipv6 as=65510 disabled=no hold-time=3s keepalive-time=1s \
    local.role=ibgp multihop=yes name=calico-ipv6 remote.address=\
    fdfd:2000:10:80::/64 .as=65510 router-id=10.10.80.1 routing-table=main \
    templates=calico
/routing filter rule
add chain=ospf-input disabled=yes rule="set suppress-hw-offload yes; accept"
add chain=bgp-backup disabled=yes rule=\
    "if (gw in 10.10.80.32/32) {set distance +10; accept}"
add chain=bgp-ecmp disabled=yes rule=\
    "if (bgp-as-path 65500) {set scope-target 10; accept}"
/routing id
add disabled=yes id=192.168.20.1 name=id-1 select-dynamic-id=only-vrf \
    select-from-vrf=*2
add disabled=yes id=10.10.15.1 name=id-2 select-dynamic-id=only-vrf \
    select-from-vrf=*3
/routing igmp-proxy
set quick-leave=yes
/routing ospf interface-template
add area=bird disabled=yes interfaces=sfp-sfpplus1
/snmp
set enabled=yes trap-version=3
/system clock
set time-zone-name=Asia/Hong_Kong
/system identity
set name=Mikrotik-CRS317
/system logging
set 0 disabled=yes
add disabled=yes topics=dhcp
add disabled=yes topics=debug
add topics=radius
/system note
set note="\r\
    \n/interface/ethernet/switch set 0 l3-hw-offloading=yes\r\
    \n/interface/ethernet/switch/port set [find] l3-hw-offloading=yes\r\
    \n\r\
    \n\r\
    \n/interface/ethernet/switch set 0 l3-hw-offloading=no\r\
    \n/interface/ethernet/switch/port set [find] l3-hw-offloading=no" \
    show-at-login=no
/system ntp client
set enabled=yes
/system ntp server
set manycast=yes multicast=yes
/system ntp client servers
add address=172.16.10.1
/system routerboard settings
set auto-upgrade=yes boot-device=nand-only boot-os=router-os
/tool bandwidth-server
set authenticate=no enabled=no
/tool netwatch
add disabled=yes down-script=use-default-dns host=10.10.15.150 http-codes="" \
    interval=5s test-script="" type=simple up-script=use-sgw-dns
add disabled=yes down-script=ADDC2-Domain-DNS host=10.10.50.10 http-codes="" \
    interval=5s test-script="" type=simple up-script=ADDC1-Domain-DNS
/tool romon
set enabled=yes
/tool sniffer
set filter-interface=NAS_Bond streaming-server=10.10.10.66
/user aaa
set default-group=full use-radius=yes

Fri Sep 06, 2024 1:19 pm

Same issue on CRS317-1G-16S+ with l3hw, using Inter-VLAN.
RouterOS 7.15.3
After rebooting the switch, things become normal but several days or even hours later, L3HW broke again.
Considering switch to Cisco or Juniper

 14:00:15 route,warning L3HW: FDB host 10.10.60.16 offload FAILED (-14)
 14:00:15 route,warning L3HW: FDB host 10.10.100.65 offload FAILED (-14)
 14:00:15 route,warning L3HW: FDB host 10.10.60.20 offload FAILED (-14)
 14:00:19 route,warning L3HW: FDB host 10.10.60.15 offload FAILED (-14)
 14:00:30 route,warning L3HW: FDB host 10.10.60.194 offload FAILED (-14)
 14:00:33 route,warning L3HW: FDB host 10.10.60.14 offload FAILED (-14)
 14:00:36 route,warning L3HW: FDB host 10.10.60.197 offload FAILED (-14)
 14:00:44 route,warning L3HW: FDB host 10.10.60.11 offload FAILED (-14)
 14:01:20 route,warning L3HW: FDB host 10.10.50.50 offload FAILED (-14)
 14:01:46 route,warning L3HW: FDB host 10.10.50.5 offload FAILED (-14)

How many routes do you have ?

misaka818 · Sat Sep 07, 2024 1:55 pm

Same issue on CRS317-1G-16S+ with l3hw, using Inter-VLAN.
RouterOS 7.15.3
After rebooting the switch, things become normal but several days or even hours later, L3HW broke again.
Considering switch to Cisco or Juniper

 14:00:15 route,warning L3HW: FDB host 10.10.60.16 offload FAILED (-14)
 14:00:15 route,warning L3HW: FDB host 10.10.100.65 offload FAILED (-14)
 14:00:15 route,warning L3HW: FDB host 10.10.60.20 offload FAILED (-14)
 14:00:19 route,warning L3HW: FDB host 10.10.60.15 offload FAILED (-14)
 14:00:30 route,warning L3HW: FDB host 10.10.60.194 offload FAILED (-14)
 14:00:33 route,warning L3HW: FDB host 10.10.60.14 offload FAILED (-14)
 14:00:36 route,warning L3HW: FDB host 10.10.60.197 offload FAILED (-14)
 14:00:44 route,warning L3HW: FDB host 10.10.60.11 offload FAILED (-14)
 14:01:20 route,warning L3HW: FDB host 10.10.50.50 offload FAILED (-14)
 14:01:46 route,warning L3HW: FDB host 10.10.50.5 offload FAILED (-14)

How many routes do you have ?

Not too much

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, b - BGP; H - HW-OFFLOA>
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#      DST-ADDRESS       GATEWAY                 DISTANCE
;;; DO NOT DELETE
0  AsH 0.0.0.0/0         172.16.10.1                  250
  DAcH 10.10.50.0/24     vlan50_server                  0
  DAcH 10.10.60.0/24     vlan60_IoT                     0
  DAcH 10.10.80.0/24     vlan80_kube                    0
  DAcH 10.10.90.0/24     vlan90-kube-frr                0
  DAcH 10.10.100.0/24    vlan100_mgmt                   0
  DAcH 10.10.110.0/24    vlan110-ac-ap                  0
  DAcH 10.10.200.0/24    vlan200_mgmt_highspeed         0
  DAbH 10.60.100.0/24    10.10.80.31                  200
  D bH 10.60.100.0/24    10.10.80.201                 200
  D bH 10.60.100.0/24    10.10.80.33                  200
  D bH 10.60.100.0/24    10.10.80.32                  200
  DAbH 10.60.100.10/32   10.10.80.31                  200
  D bH 10.60.100.10/32   10.10.80.201                 200
  D bH 10.60.100.10/32   10.10.80.33                  200
  D bH 10.60.100.10/32   10.10.80.32                  200
  DAbH 10.70.100.0/24    10.10.80.31                  200
  D bH 10.70.100.0/24    10.10.80.201                 200

But the most strange thing is that IPv6 not affected, only the IPv4 failed to offload.

Sun Sep 08, 2024 9:19 am

Odd, my CRS326 is working perfectly with a higher number of IPv4 and IPv6 routes, both received via BGP.

I am about to migrate to a CRS317, so hopefully I don't experience the same problem you are having.

misaka818 · Wed Sep 11, 2024 10:26 am

May found the solution, just disable the "Use IP Firewall For VLAN" in the Bridge Settings, several days gone and it seems no error again.

I disable IP Firewall but leave the Use IP Firewall For VLAN checked before. I though "Use IP Firewall For VLAN" should be a part of "Use IP Firewall" because when disable "Use IP Firewall", "Use IP Firewall For VLAN" can not be set so I leave it alone. So I tried to enable "Use IP Firewall" to disable "Use IP Firewall For VLAN" and disable "Use IP Firewall" again to see what will happen. And finally it works! Maybe bug or something else?

Hope it can help someone who face such a problem.

mkx · Mon Sep 16, 2024 3:08 pm

Maybe bug or something else?

I seem to remember a discussion about this exact problem a while ago (could be many moths ago) and @Normis acknowledged the bug. I'm pretty sure it was supposed to be fixed since then, but I've no idea in which version this was fixed (if at all). So if the problem happens on a recent v7, then this means that the bug re-occurred.

Mon Sep 16, 2024 3:56 pm

I seem to remember a discussion about this exact problem a while ago (could be many moths ago) and @Normis acknowledged the bug. I'm pretty sure it was supposed to be fixed since then, but I've no idea in which version this was fixed (if at all). So if the problem happens on a recent v7, then this means that the bug re-occurred.

Correct.
Was with 7.11 chain and fixed with 7.11rc3. Timeframe: mid 2023.

(and how do I know ? Because I had the problem too on AX Lite and tested several things before I could consider it solved. And then it's easy to find back the communications).

L3HW not working properly

L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Re: L3HW not working properly

Who is online