Trying to use VLANs & L3 HW offload

RouterOS General
1

I’m trying to do something that should be pretty simple, but I can’t get it to work.

What I want to do is use my CRS309 as my internet router (using L3 offload). My provider uses VLAN 300 on the uplink, so I need that on my uplink port (SFP-SFPPLUS8). The rest of the ports need no VLANs/can be access ports in a single VLAN. And I need L3 offload as I have an 8G internet link. With current config (see below) I can’t reach the internal IP address on interface VLAN1 and Winbox doesn’t see it. If I put the internal IP address on the bridge itself it works, but if I understand correctly, L3 offload won’t work in that case. What am I missing? The firewall rules are messy, I know.


# 2024-01-12 12:24:18 by RouterOS 7.13
# software id = 3TW5-3585
#
# model = CRS309-1G-8S+
# serial number = HDC085CV1JS
/interface bridge
add admin-mac=18:FD:74:EB:AB:5D auto-mac=no comment=defconf name=bridge \
    port-cost-mode=short protocol-mode=none
/interface vlan
add interface=bridge name=vlan1 vlan-id=1
add interface=bridge name=vlan300 vlan-id=300
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
/interface list
add name=LAN
add name=WAN
add name=MGMT
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=pool1 ranges=192.168.192.2-192.168.192.200
/ip dhcp-server
add address-pool=pool1 disabled=yes interface=bridge lease-time=1d name=dhcp1
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether1 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus1 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus2 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus3 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus4 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus5 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus6 \
    internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus7 \
    internal-path-cost=10 path-cost=10
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp-sfpplus8 \
    pvid=300
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus8 vlan-ids=300
add bridge=bridge tagged=bridge untagged="sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplu\
    s3,sfp-sfpplus4,sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,ether1" vlan-ids=1
/interface list member
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=WAN
add interface=ether1 list=MGMT
/interface ovpn-server server
set auth=sha1,md5
/ip address
add address=192.168.192.249/24 interface=vlan1 network=192.168.192.0
/ip dhcp-client
add interface=vlan300 use-peer-dns=no
/ip dhcp-server network
add address=192.168.192.0/24 dns-server=8.8.8.8 gateway=192.168.192.249
/ip dns
set servers=192.168.192.1
/ip firewall address-list
add address=192.168.192.2-192.168.192.254 list=allowed_to_router
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=\
    not_in_internet
/ip firewall filter
add action=accept chain=input src-address-list=allowed_to_router
add action=accept chain=input protocol=icmp
add action=drop chain=input
add action=fasttrack-connection chain=forward comment=FastTrack \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="Established, Related" \
    connection-state=established,related
add action=drop chain=forward comment="Drop invalid" connection-state=invalid \
    log=yes log-prefix=invalid
add action=drop chain=forward comment=\
    "Drop tries to reach not public addresses from LAN" dst-address-list=\
    not_in_internet in-interface=bridge log=yes log-prefix=!public_from_LAN \
    out-interface=!bridge
add action=jump chain=forward comment="jump to ICMP filters" jump-target=icmp \
    protocol=icmp
add action=drop chain=forward comment=\
    "Drop incoming from internet which is not public IP" in-interface=vlan300 \
    log=yes log-prefix=!public src-address-list=not_in_internet
add action=drop chain=forward comment=\
    "Drop packets from LAN that do not have LAN IP" in-interface=bridge log=\
    yes log-prefix=LAN_!LAN src-address=!192.168.192.0/24
add action=accept chain=icmp comment="echo reply" icmp-options=0:0 protocol=\
    icmp
add action=accept chain=icmp comment="net unreachable" icmp-options=3:0 \
    protocol=icmp
add action=accept chain=icmp comment="host unreachable" icmp-options=3:1 \
    protocol=icmp
add action=accept chain=icmp comment=\
    "host unreachable fragmentation required" icmp-options=3:4 protocol=icmp
add action=accept chain=icmp comment="allow echo request" icmp-options=8:0 \
    protocol=icmp
add action=accept chain=icmp comment="allow time exceed" icmp-options=11:0 \
    protocol=icmp
add action=accept chain=icmp comment="allow parameter bad" icmp-options=12:0 \
    protocol=icmp
add action=drop chain=icmp comment="deny all other types"
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.192.1
/routing bfd configuration
add disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5
/system identity
set name=RouterOS
/system note
set show-at-login=no
/system routerboard settings
set boot-os=router-os
2

Why did you buy a switch for the purpose of using it as a router? The throughput will not be very high on the routing side…

https://help.mikrotik.com/docs/display/ROS/CRS3xx%2C+CRS5xx%2C+CCR2116%2C+CCR2216+switch+chip+features
https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offloading
https://www.youtube.com/watch?v=YLtGQAQ8iS0

3

With L3 offload it will. The CRS309 is the cheapest switch that will do full L3 HW offload (with FW/NAT handling). That’s why I got it to play with; for now my CCR1036 does a good job :smiley:

4

Bump. Anyone?

5

I use my CRS309 as internet router. I noticed that you don’t have pvid parameter for your VLAN 1 in your /interface/bridge/port, and from my experience that parameter is pretty much required for the untagged ports, even you have the tagged/untagged configuration under /interface/bridge/vlan…

6

L3HW offloading relies on L2HW offloading and that ports are members of the same bridge where you enable bridge vlan filtering, which I don’t see from your config.
This piece is missing from your config:
/interface bridge
add name=bridge vlan-filtering=yes
Without it enabled, settings in /interface bridge vlan won’t get active, but only defined.

You can see an example from MikroTik help page: https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offloading#L3HardwareOffloading-ConfigurationExamples
Also, in general, is not advised to use vlan id=1 for your vlan, but a different value (even though you can make it work as well as any other vlan id).

7

Ah-hah! That might well be it. I wasn’t aware that you actively needed to activate VLAN-filtering to make this work, as I am not really interested in using any VLANs other than the VLAN 300 on my provider port. I’ll give that a try when I get home this evening.

8

You also need to remember that whenever you do some change to L2 (settings in VLAN, bridge etc) you also have to sync up with L3HW.
So before any change to L2, disable L3HW globally for the switch, then apply your changes and finally re-enable L3HW for the switch.
This is also indicated in that document and failing to do that, you might have issues with routed traffic that didn’t get in sync with the L2 part.

9

Yep, that was it! When I enabled VLAN filtering it logged ‘Hardware offloading enabled for ports …’ and Winbox saw the IP address on the VLAN 1 interface.

But then I made the mistake of upgrading to 7.13.2 from the 7.13 it was running at the time… Result: no lights. Not even power. Power cycling it brought it back, or at least the LEDs worked again and the management port showed traffic, but it wasn’t visible in Winbox. I tried resetting to default and even netinstall, but netinstall doesn’t see it either. Although Wireshark shows incoming BOOTP requests from it and Netinstall answering them, but the CRS not doing anything with the reply. :frowning:

10

Beside the current issue that you have now, hopefully you can recover the unit somehow.
On the L3HW notes, despite what I suggested you with ports being part of bridge and vlan members, today I have seen something which goes against this.
I have 2 ports used as WANs which are not part of bridge neither are vlan as they are directly connected to WAN devices.
I have tried to enable L3HW on these ports and they did work.
I was able to see such traffic is no longer visible in connection tracking and it’s offloaded.
So it seems that it’s also possible to have L3HW even if the port is not specifically part of the bridge and defined with its VLAN.
I just wanted to share this as you were also working on this topic too.
But I’m not sure whether it’s working now as a consequence of some bug (I’m using latest 7.13.2) and so it might not work like this once fixed.
Anyway I will stick with the way it’s documented, but it’s worth mentioning just in case.

11

I also found out what went wrong with the software upgrade. In short: doing an upgrade with the switch powered by PoE is Not A Good Idea. When upgrading, the ports flap, including the management/PoE in port. You can guess the result, and the switch doesn’t like it. At all. Luckily, a netinstall with ‘keep config’ brought it back (the earlier problem with netinstall was user error).

12

Would be great to hear your results using CRS309 with 8Gbps ISP together with L3HW nat.
I had tried before with 1Gbps (DHCP) provider and worked fine, until I got a new provider which required PPPoe which killed this switch within 10 minutes of usage.

13

I can’t speak for the 309 as a NAT router, but as L3HW offloaded router, I’ve got a pair doing it right now as my core aggregation routers, talking to CRS310’s and NetPower 16’s out in the field.

I have two CCR2116’s doing CGNAT, and with L3HW offload, NAT load drops to near zero. Unfortunately, it breaks ECMP when I do that (sends traffic to only one of the two CRS309’s, even though all the routes behind them are equidistant), so I have it turned back off for now.