Router on a Stick with Inter-VLAN L3 Routing on Switch CRS504-4XQ-IN

RouterOS General
1

Hello Everyone,

I have an upstream OPNsense router that has two VLANs defined (with NAT, DHCP, and DNS on each) and assigned to a single trunk interface which is connected to my CRS504-4XQ-IN’s qsfp28-1-1 interface. The VLANs are called VLAN_BACKUP_80 (10.80.0.1/24) and VLAN_WORKLOAD_60 (10.60.0.1/24). The router’s trunk interface connects at only 1Gb, whereas the corresponding switch’s two VLAN tagged ports are 25Gb.

Here is a diagram to provide more context.
inter-vlan-routing.png
I have followed the L3 Hardware Offloading guide and when I create IP addresses that collide with the upstream router’s default VLAN gateway (10.60.0.1), I get approximately the full 25Gb throughput using iperf. However, I then am not able to reach the public internet because the router’s gateway is shadowed by the switch’s gateway (because it is using the same IP address) and hence NAT and the outside world are inaccessible.

I currently have the Mikrotik VLAN IP address (gateway) set to 10.60.0.5, but my understanding is that the client computers are being given the router’s default gateway of 10.60.0.1 by DHCP on the upstream router and that these clients will just skip over the Mikrotik gateway and directly connect to 10.60.0.1 because that address is within the same subnet.

In a different attempt, I have set the upstream router’s VLAN DHCP default gateway to 10.60.0.5 (the Mikrotik VLAN gateway) so that clients do use the Mikrotik L3 VLAN gateway. This also gives me approximately 25Gb between VLANs, but I am not sure how to specify a static route that sends all non-VLAN traffic to the router’s default gateway of 10.60.0.1.

Here is my current config:

# 2024-09-23 16:58:49 by RouterOS 7.12.1
# software id = T4J6-105L
#
# model = CRS504-4XQ
# serial number = XXX
/interface bridge
add name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=qsfp28-1-1 ] auto-negotiation=no speed=10G-baseCR
/interface vlan
add interface=bridge name=VLAN_BACKUP_80 vlan-id=80
add interface=bridge name=VLAN_WORKLOAD_60 vlan-id=60
/interface ethernet switch
set 0 l3-hw-offloading=yes name=switch-fast
/interface bridge port
add bridge=bridge interface=qsfp28-1-1
add bridge=bridge interface=qsfp28-4-2 pvid=60
add bridge=bridge interface=qsfp28-4-4 pvid=80
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge tagged=bridge,qsfp28-1-1 untagged=qsfp28-4-2 vlan-ids=60
add bridge=bridge tagged=bridge,qsfp28-1-1 untagged=qsfp28-4-4 vlan-ids=80
/ip address
add address=10.80.0.5/24 interface=VLAN_BACKUP_80 network=10.80.0.0
add address=10.60.0.5/24 interface=VLAN_WORKLOAD_60 network=10.60.0.0
/ip dhcp-client
add interface=ether1

My greatest hope is that someone can point me to a guide that explains the best way to define these two different gateways and corresponding routes to ensure that inter-VLAN traffic only uses the Mikrotik L3 VLAN gateway and all non-VLAN (public) traffic uses the upstream router’s gateway.

This is my first inter-VLAN adventure, so I appreciate any help that you can throw my way.

Thank you,
Mark

2

Why do you create addresses that are the same as the DHCP servers’ if I understand your lexicon correctly? Since you’re running DHCP, why don’t let the servers assign such:

/ip dhcp-client
add interface=VLAN_BACKUP_80
add interface=VLAN_WORKLOAD_60

And why should VLAN and non-VLAN traffic (if there is such apart from the untagged one) use different gateways when only tagged frames are forwarded between the two devices? The gateway should be the router. If there is no internet connectivity, then the problem is on the router and not on the switch