Route wireguard peers through vxlan

Hi everyone,

I’m struggling to make vxlan work with wireguard peers. I’ve setup a wireguard connection between two mikrotik routers through a WAN interface and spanned a vxlan interface over it. As a client of local network, I am able to access devices on the opposite end of tunnel in the said vxlan without any problems. However, when I connect as a peer through the wireguard tunnel to one of the routers and try to do the same, the traffic is indeed forwarded to the opposite router but is unable to return. On one side, I can see the connection hanging in syn sent, on the other end it’s in syn received. I can make this work with additional route back to the originating router using some other interface but i would expect the traffic to return the same way it came in. What am I missing?

Contextually similar to this thread …
http://forum.mikrotik.com/t/unable-to-get-basic-vxlan-tunnel-to-work-over-wireguard/178426/1

That’s exactly what does not happen automatically. The basic routing only takes into account the destination address. If you want the route to be chosen according to any additional criteria, like the source address, protocol, source and/or destination port, in-interface, or even the in-interface of a previously received packet that has triggered the sending of the current one, you have to use “policy routing”, i.e. use mangle rules and/or routing rules that match on those additional criteria and assign a name of a dedicated routing table to be used for the packet. Reading this post, starting from the last paragraph, should give you the idea.

The problem in the other thread is a bit different, I think.

This is my network:

Client A — (LAN) —> Router A — (VXLAN over wireguard over WAN) —> Router B — (LAN) —> Client B

This is working fine, however I wanted this to behave the same way when i tunnel into Router A from WAN:

Client A — (wireguard over WAN) —> Router A — (VXLAN over wireguard over WAN) —> Router B — (LAN) —> Client B

But this gets me to the point that I’ve described in my post.


So… I’ve read through the topic and tried to setup the mangling and policy routing but I’m still missing something because the return traffic is routed using the catchall route to the WAN interface where its getting lost obviously.

I’ve done the following:

/routing table
add disabled=no fib name=A

/ip firewall mangle
add action=jump chain=prerouting connection-state=new jump-target=mark-connections
add action=accept chain=prerouting in-interface-list=WAN
add action=mark-routing chain=prerouting connection-mark=A new-routing-mark=A passthrough=no
add action=mark-connection chain=mark-connections new-connection-mark=A passthrough=yes src-address=xxx.xxx.xxx.xxx/24

/routing rule
add action=lookup disabled=no dst-address=xxx.xxx.xxx.xxx/24 routing-mark=A table=A

Can anyone help? Here is the full config:


# RouterA
/interface bridge
add admin-mac=REDACTED auto-mac=no dhcp-snooping=yes fast-forward=no \
    frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1598 name=ether1-wan
/interface wireguard
add listen-port=REDACTED mtu=1550 name=wireguard89-home
add listen-port=REDACTED mtu=1550 name=wireguard189-home
/interface vlan
add interface=bridge1 mtu=1492 name=vlan35-wan vlan-id=35
add interface=bridge1 name=vlan88-home vlan-id=88
add interface=bridge1 name=vlan99-admin vlan-id=99
/interface list
add name=WAN
add name=LAN
add name=ADMIN
add name=HOME
/ip pool
add name=pool99-admin ranges=192.168.99.2-192.168.99.30
add name=pool88-home ranges=192.168.88.2-192.168.88.30
/ip dhcp-server
add add-arp=yes address-pool=pool99-admin interface=vlan99-admin lease-time=\
    1d name=server99-admin
add add-arp=yes address-pool=pool88-home interface=vlan88-home lease-time=1d \
    name=server88-home
/interface vxlan
add mac-address=REDACTED name=vxlan1-admin port=REDACTED vni=1 vrf=main \
    vteps-ip-version=ipv4
add mac-address=REDACTED name=vxlan2-home port=REDACTED vni=2 vrf=main \
    vteps-ip-version=ipv4
/ppp profile
add change-tcp-mss=yes interface-list=WAN name=profile1 use-encryption=yes \
    use-ipv6=no
/interface pppoe-client
add add-default-route=yes disabled=no interface=vlan35-wan max-mru=1492 \
    max-mtu=1492 name=pppoe35-wan profile=profile1 service-name=REDACTED \
    user=REDACTED
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1-wan
add bridge=bridge1 edge=yes frame-types=\
    admit-only-untagged-and-priority-tagged interface=vxlan1-admin pvid=99
add bridge=bridge1 edge=yes frame-types=\
    admit-only-untagged-and-priority-tagged interface=vxlan2-home pvid=88
/interface bridge settings
set allow-fast-path=no use-ip-firewall=yes use-ip-firewall-for-vlan=yes
/ip neighbor discovery-settings
set discover-interface-list=ADMIN
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 vlan-ids=99
add bridge=bridge1 tagged=bridge1 vlan-ids=88
add bridge=bridge1 tagged=bridge1,ether1-wan vlan-ids=35
/interface list member
add interface=vlan99-admin list=LAN
add interface=vlan99-admin list=ADMIN
add interface=vxlan1-admin list=LAN
add interface=vxlan1-admin list=ADMIN
add interface=vlan88-home list=LAN
add interface=vlan88-home list=HOME
add interface=vxlan2-home list=LAN
add interface=vxlan2-home list=HOME
add interface=wireguard89-home list=LAN
add interface=wireguard89-home list=HOME
add interface=wireguard189-home list=LAN
add interface=wireguard189-home list=HOME
add interface=lo list=LAN
/interface vxlan vteps
add interface=vxlan1-admin remote-ip=192.168.89.2
add interface=vxlan2-home remote-ip=192.168.89.2
/interface wireguard peers
add allowed-address=192.168.89.2/32 endpoint-address=REDACTED \
    endpoint-port=REDACTED interface=wireguard89-home name=peer1-RouterB \
    persistent-keepalive=35s public-key="REDACTED"
add allowed-address=192.168.189.2/32 interface=wireguard189-home name=\
    peer2-ClientA public-key="REDACTED"
/ip address
add address=192.168.99.1/24 interface=vlan99-admin network=192.168.99.0
add address=192.168.88.1/24 interface=vlan88-home network=192.168.88.0
add address=192.168.89.1/24 interface=wireguard89-home network=192.168.89.0
add address=192.168.189.1/24 interface=wireguard189-home network=\
    192.168.189.0
/ip dhcp-server lease
add address=192.168.88.2 comment="ClientA" mac-address=REDACTED server=\
    server88-home
/ip dhcp-server network
add address=192.168.99.0/24 dns-server=192.168.99.1 gateway=192.168.99.1
add address=192.168.88.0/24 dns-server=192.168.88.1 gateway=192.168.88.1
/ip firewall address-list
add address=192.168.99.0/24 list=admin-clients
add address=192.168.88.0/24 list=home-clients
add address=192.168.89.0/24 list=home-clients
add address=192.168.189.0/24 list=home-clients
/ip firewall filter
add action=accept chain=input comment="accept established and related" \
    connection-state=established,related
add action=accept chain=input comment="accept wireguard from WAN" \
    dst-port=REDACTED in-interface-list=WAN protocol=udp
add action=accept chain=input comment="accept traffic from LAN" \
    in-interface-list=LAN
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=drop chain=input comment="drop everything else"
add action=fasttrack-connection chain=forward comment=\
    "fasttrack established and related" connection-state=established,related \
    hw-offload=yes
add action=accept chain=forward comment="accept established and related" \
    connection-state=established,related
add action=accept chain=forward comment="accept traffic inside HOME" \
    in-interface-list=HOME out-interface-list=HOME
add action=accept chain=forward comment="accept traffic from LAN to WAN" \
    in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment=\
    "accept mikrotik neighbour discovery broadcast" \
    dst-address-type=broadcast dst-port=5678 protocol=udp
add action=accept chain=forward comment="accept traffic from ADMIN clients" \
    src-address-list=admin-clients
add action=accept chain=forward comment="accept traffic between HOME clients" \
    dst-address-list=home-clients src-address-list=home-clients
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop everything else"
/ip firewall nat
add action=masquerade chain=srcnat comment="masquerade WAN connections" \
    out-interface-list=WAN
/ip firewall service-port
set sip disabled=yes
/ip service
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system identity
set name=RouterA
/tool mac-server
set allowed-interface-list=ADMIN
/tool mac-server mac-winbox
set allowed-interface-list=ADMIN
/tool romon
set enabled=yes
/tool romon port
set [ find default=yes ] forbid=yes
add disabled=no interface=vlan99-admin
add disabled=no interface=vxlan1-admin


###########################


# RouterB
/interface bridge
add admin-mac=REDACTED auto-mac=no dhcp-snooping=yes fast-forward=no \
    frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1598 name=ether1-wan
set [ find default-name=ether2 ] name=ether2-ClientB
/interface wireguard
add listen-port=REDACTED mtu=1550 name=wireguard89-home
/interface vlan
add interface=bridge1 mtu=1492 name=vlan35-wan vlan-id=35
add interface=bridge1 name=vlan88-home vlan-id=88
add interface=bridge1 name=vlan99-admin vlan-id=99
/interface list
add name=WAN
add name=LAN
add name=ADMIN
add name=HOME
/ip pool
add name=pool99-admin ranges=192.168.99.34-192.168.99.62
add name=pool88-home ranges=192.168.88.34-192.168.88.62
/ip dhcp-server
add add-arp=yes address-pool=pool99-admin interface=vlan99-admin lease-time=\
    1d name=server99-admin
add add-arp=yes address-pool=pool88-home interface=vlan88-home lease-time=1d \
    name=server88-home
/interface vxlan
add mac-address=REDACTED name=vxlan1-admin port=REDACTED vni=1 vrf=main \
    vteps-ip-version=ipv4
add mac-address=REDACTED name=vxlan2-home port=REDACTED vni=2 vrf=main \
    vteps-ip-version=ipv4
/ppp profile
add change-tcp-mss=yes interface-list=WAN name=profile1 use-encryption=yes \
    use-ipv6=no
/interface pppoe-client
add add-default-route=yes disabled=no interface=vlan35-wan max-mru=1492 \
    max-mtu=1492 name=pppoe35-wan profile=profile1 service-name=REDACTED \
    user=REDACTED
/routing table
add disabled=no fib name=A
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1-wan
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2-ClientB pvid=88
add bridge=bridge1 edge=yes frame-types=\
    admit-only-untagged-and-priority-tagged interface=vxlan1-admin pvid=99
add bridge=bridge1 edge=yes frame-types=\
    admit-only-untagged-and-priority-tagged interface=vxlan2-home pvid=88
/interface bridge settings
set allow-fast-path=no use-ip-firewall=yes use-ip-firewall-for-vlan=yes
/ip neighbor discovery-settings
set discover-interface-list=ADMIN
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 vlan-ids=99
add bridge=bridge1 tagged=bridge1 vlan-ids=88
add bridge=bridge1 tagged=bridge1,ether1-wan vlan-ids=35
/interface list member
add interface=vlan99-admin list=LAN
add interface=vlan99-admin list=ADMIN
add interface=vxlan1-admin list=LAN
add interface=vxlan1-admin list=ADMIN
add interface=vlan88-home list=LAN
add interface=vlan88-home list=HOME
add interface=vxlan2-home list=LAN
add interface=vxlan2-home list=HOME
add interface=wireguard89-home list=LAN
add interface=wireguard89-home list=HOME
add interface=lo list=LAN
/interface vxlan vteps
add interface=vxlan1-admin remote-ip=192.168.89.1
add interface=vxlan2-home remote-ip=192.168.89.1
/interface wireguard peers
add allowed-address=192.168.89.1/32 endpoint-address=REDACTED \
    endpoint-port=REDACTED interface=wireguard89-home name=peer1-RouterA \
    persistent-keepalive=35s public-key="REDACTED"
/ip address
add address=192.168.99.33/24 interface=vlan99-admin network=192.168.99.0
add address=192.168.88.33/24 interface=vlan88-home network=192.168.88.0
add address=192.168.89.2/24 interface=wireguard89-home network=192.168.89.0
/ip dhcp-server lease
add address=192.168.88.34 comment="ClientB" mac-address=REDACTED server=\
    server88-home
/ip dhcp-server network
add address=192.168.99.0/24 dns-server=192.168.99.33 gateway=192.168.99.33
add address=192.168.88.0/24 dns-server=192.168.88.33 gateway=192.168.88.33
/ip firewall address-list
add address=192.168.99.0/24 list=admin-clients
add address=192.168.88.0/24 list=home-clients
add address=192.168.89.0/24 list=home-clients
add address=192.168.189.0/24 list=home-clients
/ip firewall filter
add action=accept chain=input comment="accept established and related" \
    connection-state=established,related
add action=accept chain=input comment="accept wireguard from WAN" \
    dst-port=REDACTED in-interface-list=WAN protocol=udp
add action=accept chain=input comment="accept traffic from LAN" \
    in-interface-list=LAN
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=drop chain=input comment="drop everything else"
add action=fasttrack-connection chain=forward comment=\
    "fasttrack established and related" connection-state=established,related \
    hw-offload=yes
add action=accept chain=forward comment="accept established and related" \
    connection-state=established,related
add action=accept chain=forward comment="accept traffic inside HOME" \
    in-interface-list=HOME out-interface-list=HOME
add action=accept chain=forward comment="accept traffic from LAN to WAN" \
    in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment=\
    "accept mikrotik neighbour discovery broadcast" \
    dst-address-type=broadcast dst-port=5678 protocol=udp
add action=accept chain=forward comment="accept traffic from ADMIN clients" \
    src-address-list=admin-clients
add action=accept chain=forward comment="accept traffic between HOME clients" \
    dst-address-list=home-clients src-address-list=home-clients
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop everything else"
/ip firewall mangle
add action=jump chain=prerouting connection-state=new \
    jump-target=mark-connections
add action=accept chain=prerouting in-interface-list=WAN
add action=mark-routing chain=prerouting connection-mark=A new-routing-mark=A \
    passthrough=no
add action=mark-connection chain=mark-connections new-connection-mark=A \
    passthrough=yes src-address=192.168.189.0/24
/ip firewall nat
add action=masquerade chain=srcnat comment="masquerade WAN connections" \
    out-interface-list=WAN
/ip firewall service-port
set sip disabled=yes
/ip service
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/routing rule
add action=lookup-only-in-table routing-mark=A table=A
/system identity
set name=RouterB
/tool mac-server
set allowed-interface-list=ADMIN
/tool mac-server mac-winbox
set allowed-interface-list=ADMIN
/tool romon
set enabled=yes
/tool romon port
set [ find default=yes ] forbid=yes
add disabled=no interface=vlan99-admin
add disabled=no interface=vxlan1-admin

ClientA at 192.168.189.2 can’t reach ClientB at 192.168.88.34 while ClientA at 192.168.88.2 can.

I’ve reviewed the post I’ve linked and it indeed does not deal with anything but the mangle rules, assuming that the reader already understands the rest of the concept of policy routing, sorry for that.

So the part you are missing is that it is not enough to tell the routing to use a particular table (using mangle rules and/or routing rules), but you also have to populate that routing table with some routes, otherwise the routing falls back to the main table (unless you prevent that from happening by using action=lookup-only-in-table in the routing rule, causing the packet to be dropped if no route for it is available in the indicated table). So on router B, add the following route:
routing-table=A dst-address=0.0.0.0/0 gateway=wireguard89-home

On the other hand, in IPv4, routing mark X assigned using a mangle rule is translated to routing table name X automatically, and a routing rule is only necessary if you want to override this automatic translation. In another words, you can disable or removethe routing rule and the outcome will be the same.

With the route configured like this, the return traffic goes back through the wireguard89-home interface but it completely skips the vxlan interface.
As I’ve observed, the packet flow looks like this:

RouterA: → wireguard89-home
RouterA: ← vlan88-home
RouterA: ← bridge1
RouterA: ← vxlan2-home
RouterB: → vxlan2-home
RouterB: ← ether2-ClientB
RouterB: → ether2-ClientB
RouterB: → bridge1
RouterB: → vlan88-home
RouterB: ← wireguard89-home
RouterA: → wireguard89-home
RouterA: ← wireguard89-home

I would expect it to be a bit different, Is this possible to make the packet go the same way back in this configuration? Through the vxlan interface it came from, like:

RouterA: → wireguard89-home
RouterA: ← vlan88-home
RouterA: ← bridge1
RouterA: ← vxlan2-home
RouterB: → vxlan2-home
RouterB: ← ether2-ClientB
RouterB: → ether2-ClientB
RouterB: → bridge1
RouterB: → vlan88-home
RouterB: ← vxlan2-home
RouterA: → vxlan2-home
RouterA: → bridge1
RouterA: → vlan88-home
RouterA: ← wireguard89-home

Since VxLAN is normally used for L2 tunneling, I did not study your config too deeply at first and assumed that the issue was with routing of the VxLAN transport packets (the UDP ones carrying the payload L2 frames inside), sorry for this lack of concentration.

However, when looking at it with more focus now, I cannot correlate your configs with the result of your sniffing - the sniffer shows that a frame has arrived to Router B via vxlan2-home and then went out via ether2. But ether2 is not a member port of any bridge, nor is any IP address attached to it, so it is not clear to me how a packet that came in via vxlan2-home could appear there. What am I missing, or what is missing in your configuration?

Sorry for the confusion… you are right, indeed the ether2-ClientB was missing in the provided configuration of RouterB… I must have redacted too much :confused:
I have reviewed and fixed the configuration in my previous post. Would you mind looking at it once again?

OK, so indeed ether2-clientB is an access port to VLAN 88 on bridge1, same like vxlan2-home. But in that case, the topic actually changes from “why is the packet for (presumably) 192.168.189.x that came in via ether2-ClientB routed using a ‘wrong’ route” to “why is the packet that came in via ether2-ClientB routed by Router B at all”. The thing is that at Router A, you have attached 192.168.88.1/24 to the access interface to VLAN 88 on bridge1, and indicate that address as a gateway to DHCP clients, whilst at router B, you have attached 192.168.88.33/24 to the access interface to VLAN 88 on bridge1, and you also indicate that address as a gateway to DHCP clients. So external devices connected to that VLAN get DHCP offers from both DHCP servers, and the local offer typically wins because it arrives a tad faster than the remote one. This is bad enough alone. But worse, the Client B device gets 192.168.88.33 as the default gateway, so instead of sending the packet for 192.168.189.x via the VxLAN tunnel to Router A (192.168.88.1) for getting routed, it sends it to router B (192.168.88.33). And, logically, router B uses the regular routing to deliver it.

The whole suggestion regarding connection-mark assignment based on packet received from Router A via the VxLAN and routing-mark assignment to “responses” to that packet based on the connection-mark value was based on the wrong assumption that what needs to be handled are the VxLAN transport packets.

What you actually need to address is that packets for 192.168.189.x that come in via ether2-ClientB would not be routed by RouterB at all.

There are multiple ways to achieve that:

  • the cleanest one would be to disable the DHCP server for VLAN 88 on router B at all, but I guess there is some idea behind having the DHCP servers on both sites, indicating different gateways.
  • another possibility would be to remove the action=mark-connection rule completely, change the action=mark-routing rule so that it would assign the new-routing-mark A based on in-interface=vlan88-home dst-address=192.168.189.0/24 rather than on connection-mark=A, and set the gateway of the only route in routing table A to 192.168.88.1. With this, router B would route the first packet from Client B for 192.168.189.x, send the packet to the MAC address of bridge1 of outer A, notice that the in-interface and out-interface is the same, and based on that fact, it would send an ICMP message to Client B saying “hey, there is a better gateway towards that destination than me, 192.168.88.1, use that one for subsequent packets to that destination”. But depending on what IP stack Client B uses, this might or might not succeed.
  • yet another possibility would be to let the DHCP server use Option 121 (for Linux-based clients) or Option 249 (for Windows clients) to push a route towards 192.168.189.0/24 via 192.168.88.1 to the DHCP clients at Router B, but some devices ask neither for Option 121 nor for Option 249 so the previous way would have to remain as a fallback scenario
  • you could also use bridge nat rules to redirect packets coming to the MAC address of bridge1 at Router B to the MAC address of bridge1 at router A if their destination address was 192.168.189.0/24, but that would only work if they were not VLAN-tagged when entering the bridge, which is not the case (the bridge nat rules handle the frames after they get tagged on ingress, and bridge firewall rules cannot match on IP address fo tagged frames)

In addition to all the above, the concept of using a separate VxLAN for each of the two VLANs (88 and 99) on the same bridge is also incorrect, as the Rapid Spanning Tree Protocol doesn’t care about VLAN IDs and the two VxLANs are just two links between the same bridges for it, hence it only allows one of them to be used at a time to prevent L2 loops. /interface/bridge/port/monitor [find] will show you more - at least on one of the routers, one of the VxLAN ports of bridge1 must be showing forwarding: no. You could switch to Multiple Spanning Tree Protocol that allows different priorities for each group of VLANs, but it seems like an overkill to me for this case, so instead, I would use just a single VxLAN and make it a trunk port of bridge1 at both ends, so both VLAN 88 and 99 would pass through it.

First of all, thanks for the detailed explanation, think I get the idea now.


As for the cross DHCP offers, I have enabled DHCP Snooping on the bridge and made the VXLAN port untrusted, is there still a problem?


Well yes, there are multiple independent sites with private WAN connections and own clients that should be able to communicate with each other as they would locally within the same site.
There cannot be a single shared DHCP server as each site should not relay on any other site for their own functioning.


I’ve decided to try this one but had to change the in-interface=vlan88-home to src-address=192.168.88.32/27 and now the return traffic can reach my device.
But, it bothers me that is must be done manually for every site and every VLAN. Is it worth setting up OSPF to distribute these routes?


Unfortunately, the solution must reliably working for all clients, even mobile, and not require additional configuration on their side.


Everything will be VLAN-tagged so this one would probably not work.


Thanks for the suggestion, I haven’t thought about it this way. I’ve switched to a single VXLAN and now it’s working fine.

I’m not sure the treatment of DHCP packets on a VLAN-enabled bridge is any more useful than matching by IP address in bridge filter rules, you have to try - the manual is silent about that.


The way you put it, an “anycast gateway address” seems to me the best approach to fit your needs. In short, devices on all sites would be told to use the same IP address as a default gateway, yet on each site that address would be translated to a MAC address accessible only on that site. This can be achieved using multiple ways, but to me, the most manageable one (because the settings are identical on all sites) seems to be to add a VRRP interface to the VLAN interface where the actual address of each router for that subnet is attached, and attach the anycast gateway address (like 192.168.88.1) to it. But to serve the purpose, you have to use rules in chain input of ip firewall filter to drop received VRRP protocol messages from other sites on each router, so that all the VRRP interfaces would stay in master mode, and on the bridge filter, drop frames with dst-mac-address=00:00:5e:00:00:00/FF:FF:FF:00:00:00 in forward. So the clients in that subnet will send an ARP request for the gateway IP address, get the VRRP MAC address in response, and send frames carrying the IP packets that need to be routed to that MAC address; since the bridge forward drops frames for that MAC address, only the local router will receive them.

This approach addresses routing to the internet; inter-VLAN routing will end up non-symmetrical because if two VLANs are bridged among multiple sites, the packet from site A subnet X to site B subnet Y will be routed from subnet X to subnet Y by router A but the response packet from site B subnet Y to subnet X will be routed by router B. So the stateful firewall on each router will only see one direction of the connection and behave accordingly (block some connections, use different timeouts for others).

How much of an issue the above is in your use case? Do you need inter-VLAN routing at all?


Not sure whether OSPF could take care of this at all, but as a minimum, all the endpoint devices would have to support OSPF, as the ICMP redirection is used to tell the endpoint devices to use another gateway. Normally, only routers use OSPF to inform each other about destinations they can handle.


This topic is actually a bit of a gray zone - the RFC regarding VXLANs suggests that VXLANs should drop VLAN tagged frames (whereas Mikrotik currently lets them get through), but it also doesn’t anticipate any handling for STP BPDUs because it uses its own mechanisms to prevent L2 loops. So it assumes VXLANs to completely replace VLANs and the L3 redundancy mechanisms used to route the VXLAN transport packets to prevent occurrence of loops in the L2 payload domain rather than STP. So on each datacenter host, each VXLAN is presented as a whole VLAN-agnostic bridge rather than just a single port of a bridge, i.e. the virtual network interfaces of the virtual machines are linked directly to the VXLAN rather than to an intermediate bridge. And to cause an L2 loop, a VM would have to internally bridge two network interfaces connected to the same VXLAN, which is far less likely than plugging a patchcord to a wrong port in the physical world.

So a “future-proof” (as in “change of mind on Mikrotik side - proof”) way of mimicking this on Mikrotik would actually be to go back to one VXLAN per VLAN approach but force the VXLAN ports of the local bridge to be edge ones and let the bridge filter drop BPDUs (dst-mac-address=01:80:C2:00:00:00/FF:FF:FF:FF:FF:FF) received via these ports.

Just an idea: ever thought of using EoIP instead of VXLAN?

I have EoIP running over wg-tunnels and I‘m very happy with this! Easy and stable.

Ralf.