Community discussions

MikroTik App
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Multicast/OSPF over ZeroTier

Sun Apr 20, 2025 3:06 pm

I'm trying to setup OSPF over a ZeroTier L2 domain (so mode broadcast) among two Mikrotik routers and an OPNsense box.

Apparently "Allow Ethernet bridging" should be enabled on ZeroTier central for the Mikrotik routers otherwise Multicast Hello packets are not received and adjacency can not be established.

From my understanding this setting disable a filter that prevent traffic from mac addresses unrelated to the ZeroTier node to enter the ZeroTier network.

It should have nothing to do with preventing multicast traffic going toward the ZeroTier node from the ZeroTier network or viceversa.

Indeed on OPNsense is absolutely not needed.

Since allowing bridging pose an additional load on the node, I would like to understand if and how I can disable without disabling multicast traffic.
Last edited by unlikely on Tue Apr 22, 2025 6:06 pm, edited 1 time in total.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Sun Apr 20, 2025 5:10 pm

You may need to change the "Flow Rules" for the ZT network on my.zerotier.com (see ZeroTier docs: https://docs.zerotier.com/rules/#rule-d ... n-language generally or examples here https://www.zerotier.com/blog/using-flo ... -services/ etc.). By default, the flow rules allow only UDP/TCP, but that won't cover OSPF.

If lab/test, you may want to modify you flow rules to just "accept;", instead of the "drop if not" likely in your rules. You can allow OSPF protocol specifically in ZT flow rules, but I'd recommend just test with the "accept;" initially to confirm it's flow rules why OSPF isn't working.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 21, 2025 10:22 pm

Thanks for reply.
Default flow rules allow for ip4, ip6 and arp, so ospf should be included.
By the way I also tried to put just accept and still doens't work on Mikrotik.
But it works on OPNsense with default or permissives flow rules and without bridging enabled.
Seems a Mikrotik thing.
Last edited by unlikely on Tue Apr 22, 2025 5:58 pm, edited 1 time in total.
 
xrlls
Member Candidate
Member Candidate
Posts: 112
Joined: Sun Jan 13, 2019 4:43 pm
Location: Copenhagen, DK

Re: Multicast/OSPF over ZeroTier

Mon Apr 21, 2025 10:45 pm

For what it’s worth, I have OSPF running between a few ‘tiks without any issues. My flow rules are default, and I have not enabled bridging, so I would believe that it should just work.

The only issue I had was relating to firewall, where I needed to allow OSPF (or any other traffic for that matter) on the Zerotier interface.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Mon Apr 21, 2025 11:16 pm

Seems a Mikrotik thing.
Perhaps.

Someone else had similar issues with OSPF broadcast mode and ZeroTier:
viewtopic.php?p=1118612#p1118520

There too, I thought it was flow rules, but OP was using Mikrotik controller which has NO flow rules. And I forgot the it the different IP protocol (89) that OSPF uses, not ethertype — so default flow rules should allow OSPF broadcast.

If you sure OSPF is generally working... it might worth a ticket to Mikrotik at help.mikrotik.com. Include a supout.rif if you do file a ticket.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Tue Apr 22, 2025 6:02 pm

For what it’s worth, I have OSPF running between a few ‘tiks without any issues. My flow rules are default, and I have not enabled bridging, so I would believe that it should just work.

The only issue I had was relating to firewall, where I needed to allow OSPF (or any other traffic for that matter) on the Zerotier interface.
With broadcast/multicast?
Without static neighbors?

I think my firewall it's ok because with bridging enabled, OSPF is working.

Could also be a specific ROS version or hardware type.

My test are for a CCR2004 and a RB5009, both with ROS 18.2, both have the issue.
Last edited by unlikely on Tue Apr 22, 2025 6:05 pm, edited 1 time in total.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Tue Apr 22, 2025 6:05 pm

Seems a Mikrotik thing.
There too, I thought it was flow rules, but OP was using Mikrotik controller which has NO flow rules. And I forgot the it the different IP protocol (89) that OSPF uses, not ethertype — so default flow rules should allow OSPF broadcast.

If you sure OSPF is generally working... it might worth a ticket to Mikrotik at help.mikrotik.com. Include a supout.rif if you do file a ticket.
I can confirm that OSPF over ZeroTier is generally working when bridging is enabled for Mikrotik routers nodes.
But as soon I remove the bridging it stops working, and restart as soon as I set again bridging.
I think I'll report to Mikrotik.
 
xrlls
Member Candidate
Member Candidate
Posts: 112
Joined: Sun Jan 13, 2019 4:43 pm
Location: Copenhagen, DK

Re: Multicast/OSPF over ZeroTier

Tue Apr 22, 2025 6:40 pm

What I have is this:
 /zerotier/interface> /zerotier/interface/print detail 
Flags: D - dynamic, X - disabled; R - running 
 0  R name="zerotier1" mac-address=3E:65:02:6A:A1:37 mtu=2800 arp-timeout=auto network="xxxxxxxxxxxxxx" 
      instance=zt1 allow-managed=yes allow-global=no allow-default=no bridge=no dhcp=no 
      network-name="Area 0.0.0.0 (Backbone)" status="OK" type="PRIVATE"
- no bridging enabled
 /zerotier/interface> /routing/ospf/interface-template/print where interfaces=zerotier1 
Flags: X - disabled, I - inactive 
 0   area=backbone_v2 interfaces=zerotier1 instance-id=0 type=broadcast retransmit-interval=5s transmit-delay=1s 
     hello-interval=10s dead-interval=40s priority=128 cost=1000 

 3   area=backbone_v3 interfaces=zerotier1 instance-id=0 type=broadcast retransmit-interval=5s transmit-delay=1s 
     hello-interval=10s dead-interval=40s priority=128 cost=1000 
- using broadcast
 /routing/ospf/neighbor/print 
Flags: V - virtual; D - dynamic 
 0  D instance=v3inst area=backbone_v3 address=fe80::3c6b:edff:fe56:c8d1%zerotier1 priority=128 
      router-id=192.168.32.1 dr=192.168.32.1 bdr=192.168.16.1 state="Full" state-changes=6 adjacency=19h26m1s 
      timeout=34s 

 1  D instance=v3inst area=backbone_v3 address=fe80::3c2d:d6ff:fe01:7419%zerotier1 priority=128 
      router-id=192.168.48.1 dr=192.168.48.1 bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s 

 4  D instance=v2inst area=backbone_v2 address=172.18.0.186 priority=128 router-id=192.168.48.1 dr=172.18.0.186 
      bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s 

 5  D instance=v2inst area=backbone_v2 address=172.18.0.205 priority=128 router-id=192.168.32.1 dr=172.18.0.205 
      bdr=172.18.0.139 state="Full" state-changes=5 adjacency=19h26m1s timeout=34s 

- neighbors found (output slightly redacted to remove some irrelevant Wireguard and direct connections).

This particular device is running ROS7.19beta8, and the neighbors are running 7.18.2. But it has been running forever on many different versions of ROS.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Tue Apr 22, 2025 9:25 pm

What I have is this:
Thanks.
I can't see anything special or substantially different from my setup.
When I turn off bridging in ZeroTier Central for my Mikrotik Node, I can't see anymore OSPF traffic logged by my very first Raw Prerouting firewall rule.
OSPF log in Mikrotik only show "send-hello"
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Tue Apr 22, 2025 11:41 pm

Just curious, but why even bother using OSPF on a Layer 2 network? Even using OSPF with IP seems kind of weird since all internal routing is already built into ZeroTier, right? Maybe I’m missing something here...
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1767
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: Multicast/OSPF over ZeroTier

Wed Apr 23, 2025 10:33 am

Just curious, but why even bother using OSPF on a Layer 2 network? Even using OSPF with IP seems kind of weird since all internal routing is already built into ZeroTier, right? Maybe I’m missing something here...

There are a few use cases for it.

1) ZeroTier's controller under the free version can only hold 2 or 3 routes (paid version is 128 routes I believe) so if you need more complex routing on an existing ZeroTier network, you might have redistribute between the zerotier routing table and other routers.

2) ZeroTier allows you to create multiple networks. Not all of them have to be designed as remote access for users. It's not uncommon to build one network for remote users and another for DCI or DCI mgmt. I've created several in this fashion.

3) If you don't want to manage multiple VPN types. ZeroTier may not always be the "fastest" VPN type, but it's one of the easiest. If you already have a gateway VPN router that's running ZT and want to add networks or functionality to reach other ZT routers, it's much easier administratively to keep building ZT networks and mix it with dynamic routing as opposed to managing ZT, Wireguard, gre/ipsec, etc, etc.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Wed Apr 23, 2025 11:58 am

ZeroTier doesn’t have visible “routes”, just virtual networks and devices/routers associated to a virtual network. Since last year, new free accounts get 10 devices/routers total, while older accounts had 25 (still valid). Extra licenses are very cheap.

When someone asks about OSPF or other routing protocols on top of ZeroTier, it’s usually a misunderstanding. Sure, there might be special cases, but people planning for those are most likely already familiar with SD-WAN solutions like ZeroTier and don’t need to ask about it here (unless it's about ROS v7 quirks).
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1767
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: Multicast/OSPF over ZeroTier

Thu Apr 24, 2025 11:12 am

ZeroTier doesn’t have visible “routes”, just virtual networks and devices/routers associated to a virtual network. Since last year, new free accounts get 10 devices/routers total, while older accounts had 25 (still valid). Extra licenses are very cheap.

That's not completely accurate. ZeroTier uses the routing table just like anything else. The ZeroTier networks you create inject connected and dynamic static routes into the OS of the device the ZT software runs on top of. The controller maintains a routing table that is pushed to every endpoint. It doesn't interact with dynamic routing protocols at the controller level, but you can absolutely redistribute ZT VPN routes into OSPF, IS-IS, BGP, etc. This is very common when using ZT for OOB mgmt with data center networks.

The device count you're mentioning is separate from the controller route count.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Thu Apr 24, 2025 6:21 pm

There is absolutely no distinction in the type of devices used by ZeroTier. It can be ten smartphones or ten routers connecting the same number of subnets. What matters is the number of activated devices listed in the management console. Also, there's no such thing as a "controller route count" exposed or checked by the ZeroTier controller.

You have virtual networks, and each one simply tracks its active endpoints. Each virtual network can be configured as a Layer 2 virtual switch, a Layer 3 virtual router, or a mix of both, that might be managed by the ZeroTier Rules Engine.

ZeroTier does of course utilize standard IP routing on each endpoint device, which is kinda basic networking. What I was referring to is the internal ZeroTier protocol and its peer-to-peer routing mechanisms, which are abstracted away from the user. There is no way to configure those internal routes manually, and they're not exposed like in traditional routes.

The ZeroTier controller injects connected routes into the OS routing table on the endpoints, but this happens behind the scenes when you connect to a virtual SD-WAN network. And while it is true that you can redistribute the virtual subnet, like any other subnet, into OSPF, BGP or similar protocols, that is typically only relevant in more complex SDN setups such as data centers or hybrid SD-WAN environments.

For most users, especially those asking about adding OSPF/BGP on top of ZeroTier, it is simply overkill. ZeroTier already handles the routing needed for full connectivity within its core virtual network. People familiar with SD-WAN generally know this, which is why those questions usually indicate a misunderstanding of what ZeroTier already provides.
Last edited by Larsa on Thu Apr 24, 2025 6:46 pm, edited 3 times in total.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Thu Apr 24, 2025 6:40 pm

Larsa makes good points. Personally I'd use ZT routes if possible, since it just so simple.

I've assumed OP already had OSPF infra, perhaps with non-ZT things, and there OSPF over ZT would seem reasonable. But if you're using OSPF for route distribution ONLY for ZeroTier, that would seem silly.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Thu Apr 24, 2025 7:02 pm

The OP said "OSPF over a ZeroTier L2 domain (so mode broadcast)," which sounds a bit contradictory. OSPF is basically a L3 routing helper, so I don't get how Layer 2 fits into the picture after reading the thread. I can understand running OSPF on one of the nodes to expose the ZT network, but I’d really like to hear more about what use case the OP has in mind for "ZT over L2".
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Fri Apr 25, 2025 9:02 am

I've assumed OP already had OSPF infra, perhaps with non-ZT things, and there OSPF over ZT would seem reasonable. But if you're using OSPF for route distribution ONLY for ZeroTier, that would seem silly.
My idea was to use ZT as a secondary way for distributing routes and route packets among routers when better ways fails or are impossible.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Fri Apr 25, 2025 9:08 am

The OP said "OSPF over a ZeroTier L2 domain (so mode broadcast)," which sounds a bit contradictory.
I mean some of the routers partecipating in OSPF are connected all to the same ZeroTier network / L2 domain, so brodcast mode should be possible.
But every routers also belongs to other networks.
 
xrlls
Member Candidate
Member Candidate
Posts: 112
Joined: Sun Jan 13, 2019 4:43 pm
Location: Copenhagen, DK

Re: Multicast/OSPF over ZeroTier

Sun Apr 27, 2025 11:35 am

Returning to the OP originally problem, I previously stated that it works for me… and it does… on some of my routers.

I currently have 3 MikroTik routers connected to the same ZeroTier network, and OSPF is working on two of them over Zerotier. On the third, a RB5009, running 7.18.2, I only see outgoing multicast traffic, no incoming traffic when running the packet sniffer. I suspected that this might be a firewall issue, but unicast traffic is passing with no issues. I have coincidentally had some issues running RoMon over Zerotier. RoMon over Zerotier is working/not working, on the same devices as OSPF.

I suspect that there may be a general issue with multicast on some configurations, while I cannot spot the difference between my working and non working device, which is a combination of RB5009 on 7.18.2 (not working, no incoming multicast on Zerotier interface)
L009 and 7.19beta8 (working)
hAP ax^2 on 7.18.2 (working)

It is not super important for me to get this working, as Zerotier is primarily serving as redundancy for Wireguard tunnels, so I won’t be puting more effort into debugging it for now.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 2:43 pm

Returning to the OP originally problem, I previously stated that it works for me… and it does… on some of my routers.

I currently have 3 MikroTik routers connected to the same ZeroTier network, and OSPF is working on two of them over Zerotier. On the third, a RB5009, running 7.18.2, I only see outgoing multicast traffic, no incoming traffic when running the packet sniffer. I suspected that this might be a firewall issue, but unicast traffic is passing with no issues. I have coincidentally had some issues running RoMon over Zerotier. RoMon over Zerotier is working/not working, on the same devices as OSPF.

I suspect that there may be a general issue with multicast on some configurations, while I cannot spot the difference between my working and non working device, which is a combination of RB5009 on 7.18.2 (not working, no incoming multicast on Zerotier interface)
L009 and 7.19beta8 (working)
hAP ax^2 on 7.18.2 (working)

It is not super important for me to get this working, as Zerotier is primarily serving as redundancy for Wireguard tunnels, so I won’t be puting more effort into debugging it for now.
would you mind just trying allowing bridging on zerotier central for RB5009 to see if you start receiving multicast/OSPF packets?
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 6:34 pm

I mean some of the routers partecipating in OSPF are connected all to the same ZeroTier network / L2 domain, so brodcast mode should be possible.
But every routers also belongs to other networks.

Thanks for the clarification. I get that OSPF can work over the shared L2 domain, but that still leaves me wondering why you would want to do it this way.

If all routers are already L2-connected through ZeroTier, it feels a bit like connecting a big switch to all your subnets instead of properly routing between them. I mean, wouldn't it be simpler and cleaner to just run OSPF over the existing routed networks instead of bridging everything?

Maybe there is a specific reason for the setup that I have missed?
 
xrlls
Member Candidate
Member Candidate
Posts: 112
Joined: Sun Jan 13, 2019 4:43 pm
Location: Copenhagen, DK

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 8:18 pm

@unlikely
I tried enabling bridging for the Zerotier endpoint (Rb5009, 7.18.2) not working and … heureka… it started working! I then disabled bridging again, and the multicast traffic stopped arriving on the RB5009. Enabling bridging on the non-working endpoint also fixed my other issue with RoMon, which also relies on multicast for discovery. There is definitely something odd at play here, where some devices do not allow incoming multicast on Zerotier unless bridging is enabled. Mind you, I have 3 routers on this Zerotier network, and only one of them requires bridging enabled, the two others just works without it.

@larsa
I think there are many reasons to run OSPF on an L2 segment. For me they are:
  • Failover. Zerotier is my backup solution for site to site. If the primary connection is between sites are lost, the routes are reconfigured, independently of what kind of backup connections are used. I run a hub and spoke topology, and if the hub fails, the remaining sites reconfigures to connect over Zerotier.
  • Easy deployment of new routes. Make a change on a single router, and routes are automatically deployed to the other routers.
  • Easy configuration of OSPF. For Wireguard is is necessary to configure static neighbors. While for Zerotier (and L2 in general) it just works, as the neighbors are automatically discovered through multicast.


Ultimately I think it is more a question of easing the route deployment than the type of interconnect, whether your interconnect is L2 or L3, you will need to deploy routes to each individual router in order to have end to end reachability, unless, everything is just bridged in a single L2 segment covering all sites. While it would be possible to go all in on bridging, that is not how I am using it.

I might ask the other way, what would your recommendation instead?
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 8:38 pm

I might ask the other way, what would your recommendation instead?
You can use ZT to push any route. ZT does not care if the destination is within ZT's IP range — ZT is agnostic on gateway so you can often use ZT for just route distribution. And RouterOS will happy add whatever it gets from ZT directly to the [main] routing table.

If you already using OSPF, you can still duplicate the routes. If you do this you'd want to pay attention to the default-route-distance= in ZT settings as on RouterOS, since imagine you'd want OSPF one to be a lower distance.

Also, while I have not tested it, I'd hope the new "dynamic-in" routing filter would work for ZT routes... so, in theory, you could use a route filter in dynamic-in chain to adjust anything, similar to BGP. See example with /ip/dhcp-client inject dynamic routes: viewtopic.php?t=215181&hilit=dynamic+in#p1130346
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 10:30 pm

Easy configuration of OSPF. For Wireguard is is necessary to configure static neighbors. While for Zerotier (and L2 in general) it just works, as the neighbors are automatically discovered through multicast.
For me ZeroTier is kind of a backup.
I have some ptp wireguard link, they don't need static neighbors configuration.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 10:33 pm

@xrlls, I think I get your reasoning, but I also think you are unnecessarily complicating things.

If you're already using ZeroTier, then running OSPF on top of it is simply redundant. ZeroTier can fully handle dynamic route distribution and failover without needing a separate routing protocol like OSPF. Also, keep in mind that OSPF is a Layer 3 protocol designed for IP routing between networks, not for being used inside a Layer 2 domain.

Building a shared L2 segment across sites (even partially) is risky and goes against good network design principles. You risk broadcast storms, unnecessary coupling between networks, and also problems with scalability and troubleshooting. IP routing is much safer, cleaner and more predictable, especially when you're spanning different sites.

So I'd suggest using ZeroTier to push routes dynamically between your routers and keeping the networks cleanly separated using IP (L3). Use WireGuard tunnels only as backup management access if needed, in case ZeroTier or the primary path fails.

Bottom line, try avoiding mixing Layer 2 (a switch network) and Layer 3 (IP), which will greatly simplify control and troubleshooting.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 10:43 pm


I get that OSPF can work over the shared L2 domain, but that still leaves me wondering why you would want to do it this way.

If all routers are already L2-connected through ZeroTier, it feels a bit like connecting a big switch to all your subnets instead of properly routing between them. I mean, wouldn't it be simpler and cleaner to just run OSPF over the existing routed networks instead of bridging everything?

Maybe there is a specific reason for the setup that I have missed?
The routers are in different sites.
The smarter routers are all connected to a hub by Wireguard spokes and belong to the same ZeroTier network together with the hub.
There are also few non-OSPF routers connected by Wireguard to the hub and few direct wireguard links between most important sites.
ZeroTier is kind of a backup for wireguard because it's slower than wireguard but expected to works when under some circumstances wireguard when could fail. I also prefer avoid manually managing routes in ZeroTier central and rely only on ZeroTier.
Last edited by unlikely on Mon Apr 28, 2025 10:57 pm, edited 1 time in total.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 10:46 pm

@unlikely, I get what you are saying about using ZeroTier as backup and your WireGuard links not needing static neighbor setup.

BUT still, the same idea I mentioned to @xrlls applies here. If you already have ZeroTier in the mix, running OSPF on top is just extra work without any benefit, quite the contrary as I explained to @xrlls. Using Layer 2 across sites might cause broadcast storms, messy network coupling and make everything harder to scale and troubleshoot.

Sticking to clean Layer 3 routing is way safer and easier. If you go L2, you are on your own.

If you still do not get it, good luck!
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:01 pm

There are also few non-OSPF routers connected by Wireguard to the hub and few direct wireguard links between most important sites.
ZeroTier is kind of a backup for wireguard. ZeroTier is slower with our slow connections. And I don't want to rely on routes manually defined in ZeroTier network.
Fair enough, I get if you're using OSPF that should be definitive. You just ask, and do think using ZT for distributing something like WireGuard or other routes is likely simpler if one did not already know OSPF, thus I mention it. With point being the traffic doesn't have to use [slower] ZT gateway, even if ZT injected route. So just different approach...

I'm flummox at what the issue here is, since OSPF broadcast should "just work" (without bridging IMO). Perhaps check /tool/sniffer to see if OSPF multicast is even hitting the zerotier interface (and ideally on far-end, to see if got there) – that confirm if ZT config issue, OR, bug in ZT+OSPF on RouterOS.

Also, you don't have use broadcast with OSPF, which avoid the mutlicast issue with ZT and potential complexity @Larsa mentions. While I'm more sanguine than @Larsa of OSPF over ZT as a use case...that be predicated on familiarity and existing usage of OSPF.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:06 pm

@unlikely, I get what you are saying about using ZeroTier as backup and your WireGuard links not needing static neighbor setup.

BUT still, the same idea I mentioned to @xrlls applies here. If you already have ZeroTier in the mix, running OSPF on top is just extra work without any benefit, quite the contrary as I explained to @xrlls. Using Layer 2 across sites might cause broadcast storms, messy network coupling and make everything harder to scale and troubleshoot.

Sticking to clean Layer 3 routing is way safer and easier. If you go L2, you are on your own.

If you still do not get it, good luck!
I dont' get what you recommend, because if I have to avoid L2 coupling, I think I need to avoid ZeroTier completely, so I should stay with L3 wireguard tunnels and nothing else?

Then when my sites fail over LTE ISP2 with CGNAT my wireguard link fails too. Or when I need to do some maintenance on wireguard links.

What could be a better approach?
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:13 pm

You seem to like your current topology. And if you have CGNAT, ZT is likely best. Whether you can limit ZT to just those sites, IDK....

The only other approach is abuse BackToHome (BTH) - that does deal with CGNAT and is just WireGuard under the covers. i.e. if a site had a fixed public IP, and LTE back... BTH be same as WG when "primary WAN", but if failover BTH "proxy" WG via LTE CGNAT. The issue is @normis repeats it's a "home" feature (as implied by name), so whether that's be a "better" approach IDK.

What I'm saying is I'm not sure whether there is a bug or just OSPF/ZT config error. I'd either collect some sniffer or re-create small example of it not working in lab/CHRs/etc. If repo-able in small setup, that seem like a bug to MikroTik. ZeroTier should be transparent to OSPF – now OSPF then be connected everywhere, so that have be considered in your OSPF design. This simple example of problem be critical here...
Last edited by Amm0 on Mon Apr 28, 2025 11:17 pm, edited 1 time in total.
 
xrlls
Member Candidate
Member Candidate
Posts: 112
Joined: Sun Jan 13, 2019 4:43 pm
Location: Copenhagen, DK

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:15 pm

My point is that I think OSPF is less work. While I understand that routes can be pushed through Zerotier, relying on it, would require me to maintain both the Zerotier routing configuration and another route distribution method for the non-Zerotier connections. So I would have two maintain the routes in at least two different realms as my networks consists of Zerotier, Wireguard, and direct cabled connections.

In terms of security I want to underline that I am not bridging multiple L2 segments, i.e the Zerotier network is not member of any bridge on any routers. It’s just yet another interface with yet another IP address. Not unlike a wireguard interface with multiple peers, except not having to configure the OSPF static neighbors manually. The payload carrying traffic traveling over the Zerotier has passed L3 routing in order to get there. The only non-routed traffic passing my Zerotier network is OSPF and RoMon.

In terms of L2, Zerotier is just designed, as being L2, and then IP can be put on top, which I am doing.

If I gave the impression that I was bridging everything as one big happy L2 network, that is not what I am doing, and I agree that in such a configuration OSPF does not make sense.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:15 pm

@unlikely: There is no way to give you a real answer because you have not shared a complete network topology or explained your actual goals.

Without that, any suggestion would be pointless guessing.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:19 pm

If I gave the impression that I was bridging everything as one big happy L2 network, that is not what I am doing, and I agree that in such a configuration OSPF does not make sense.
I think OP isn't actually bridging zerotier on RouterOS bridge – although be to confirm... OP is just checking the "Allow Bridging" option on the ZT controller, AFAIK...
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:24 pm

@xrlls, like I told @unlikely, there’s no way to give you a real answer because you haven’t shared a full network topology or explained what you’re actually trying to do. From what I can tell, it sounds like you’re a bit unclear on basic L2/L3 networking. Maybe @Amm0 can help you out here.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4965
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:26 pm

Maybe @Amm0 can help you out here.
@Amm0 already explained to look at sniffers, or lab a smaller example. But ZeroTier "L2" should be transparent to "L3" [multicast] OSPF.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:28 pm

Fair enough, I get if you're using OSPF that should be definitive. You just ask, and do think using ZT for distributing something like WireGuard or other routes is likely simpler if one did not already know OSPF, thus I mention it. With point being the traffic doesn't have to use [slower] ZT gateway, even if ZT injected route. So just different approach...

I'm flummox at what the issue here is, since OSPF broadcast should "just work" (without bridging IMO). Perhaps check /tool/sniffer to see if OSPF multicast is even hitting the zerotier interface (and ideally on far-end, to see if got there) – that confirm if ZT config issue, OR, bug in ZT+OSPF on RouterOS.

Also, you don't have use broadcast with OSPF, which avoid the mutlicast issue with ZT and potential complexity @Larsa mentions. While I'm more sanguine than @Larsa of OSPF over ZT as a use case...that be predicated on familiarity and existing usage of OSPF.
Until a few days ago, I was using the same combined WireGuard and ZeroTier approach to manage several sites, with static routes of varying distances configured on each router.
Honestly, I had never considered the option of using ZeroTier solely to distribute routes to routers.
Now OSPF still needs some tweaking, but appear to be working properly, and I don't think I'll go back to a manual approach, especially since I've added a few more sites to the setup.
Moreover, the idea of manually managing routes across multiple routers or within the ZeroTier controller no longer appeals to me.
That said, I'm open to experimenting with other solutions that could achieve the same results in a more efficient way.
I understand that OSPF can be configured to avoid using broadcast, but broadcast is precisely what makes OSPF simple and resilient in my opinion.
To return to the main topic, I am about to publish MikroTik's response on the matter.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:32 pm

@Amm0; Yeah, exactly. It might be something as simple as a firewall blocking the LSA packets. If they sniff the packets and turn on OSPF LSA logging, they should be able to figure out pretty quickly what's actually going on.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:34 pm

I think OP isn't actually bridging zerotier on RouterOS bridge – although be to confirm... OP is just checking the "Allow Bridging" option on the ZT controller, AFAIK...
Correct.
ZeroTier interface is not part of any bridge.
So the answer from Mikrotik is more puzzling.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:35 pm

@xrlls, you should probably open your own thread since it gets complicated when mixing two different use cases in the same thread.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:40 pm

Just a tip, @unlikely: make a full network topology diagram, do a complete export, start packet sniffing, turn on OSPF logging, and if you still think there’s a bug somewhere, send everything over to support. But I still don't get why you're insisting on using L2 instead of just doing it the normal way with L3 IP and OSPF PTP links.
Last edited by Larsa on Mon Apr 28, 2025 11:47 pm, edited 1 time in total.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:44 pm

I don't believe I need a sniffer; the MikroTik logs should be sufficient. Below is part of what I wrote to MikroTik support:
Please note that OSPF generally functions correctly in my setup when bridging is enabled on two MikroTik nodes (CCR2004 running ROS 7.18.2 and RB5009 running ROS 7.18.2 or 7.19beta8) on ZeroTier Central.
When bridging is disabled, OSPF adjacency is maintained for a short period. However, if I disable and re-enable the OSPF instance while bridging is still disabled, neighbors are typically not discovered. The OSPF logs on each node display only “sending hello.” To restore OSPF functionality, I have to re-enable bridging.
When I mention "enabling bridging", I am specifically referring to checking "Allow Bridging" in ZeroTier Central. The ZeroTier interface on the MikroTik devices is never part of any bridge.

This is the response I received from MikroTik:
"Allow Ethernet Bridging" enables the ZeroTier node to bridge Layer 2 multicast traffic (i.e., Ethernet frames with multicast destination MAC addresses).
This includes protocols like OSPF Hello packets, which are necessary in this scenario.
And here is my reply:
I don’t need to bridge multicast traffic or any traffic.
What I need is for my MikroTik device to receive multicast traffic from the ZeroTier network.
What do you think?
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Mon Apr 28, 2025 11:53 pm

Support gave you a proper answer. You can run any L2 traffic you want over Zerotier, but you need to know how to manage it using ZT flow rules together with correct ROS routing and firewall rules. And yes, it works as expected.

Have you checked your firewall and the logs for OSPF LSA messages? Have you sniffed ingress and egress traffic for OSPF Hello packets? Are you sure your ZT flow rules are correct?

Again, I still don't get why you're making it so complicated though. Why insist on using L2 with all these issues instead of just doing it the normal way with IP routing and OSPF PTP links???

If you want further help from me, please provide complete network topology.
Last edited by Larsa on Tue Apr 29, 2025 12:07 am, edited 1 time in total.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Tue Apr 29, 2025 12:05 am

Support gave you an adequate answer. You can run whatever L2 traffic you want over Zerotier, but you need to know how to manage it with flow rules. And yes, it works as expected.
I don't think I can agree.
To me it doens't seems a correct L2 behavior.
It is not as if my routers were connected to the same physical switch.
Flow rules are not relevant because the default should allow all IPv4 IPv6 ARP and OSPF is IP.
L009 and 7.19beta8 and hAP ax^2 on 7.18.2 are working without "allow bridging".
OPNsense works without "allow bridging".
My pc with wireshark can receive the ospf packet from the ZT network without "allow bridging".
There still could be some misconfiguration somewhere in Mikrotik, but I cannot guess where and I don't think it's the correct behavior.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 2000
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multicast/OSPF over ZeroTier

Tue Apr 29, 2025 12:09 am

Well, since you keep repeating yourself over and over again, don't listen to @Amm0's or my advice, and don't even bother to answer my questions, you're on your own. Good luck!
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Tue Apr 29, 2025 12:25 am

Well, since you're not listening to my advice or bothering to answer my questions, you're on your own. Good luck!
I’m open to all advice and suggestions and truly appreciate everyone’s contributions.
However, I still want to address or at least better understand my initial issue.
You’ve stated that this behavior is correct, and I acknowledge that, but I would also like to hear from others.
You’ve also mentioned that I didn’t respond to your questions, but I hadn’t seen that question earlier—it seems you edited and added it afterward.
 
unlikely
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Feb 21, 2020 1:16 pm

Re: Multicast/OSPF over ZeroTier

Tue Apr 29, 2025 10:07 pm

The only other approach is abuse BackToHome (BTH) - that does deal with CGNAT and is just WireGuard under the covers. i.e. if a site had a fixed public IP, and LTE back... BTH be same as WG when "primary WAN", but if failover BTH "proxy" WG via LTE CGNAT. The issue is @normis repeats it's a "home" feature (as implied by name), so whether that's be a "better" approach IDK.
Interesting feature to know (BTH) I wasn't aware of.