Multicast/OSPF over ZeroTier

I’m trying to setup OSPF over a ZeroTier L2 domain (so mode broadcast) among two Mikrotik routers and an OPNsense box.

Apparently “Allow Ethernet bridging” should be enabled on ZeroTier central for the Mikrotik routers otherwise Multicast Hello packets are not received and adjacency can not be established.

From my understanding this setting disable a filter that prevent traffic from mac addresses unrelated to the ZeroTier node to enter the ZeroTier network.

It should have nothing to do with preventing multicast traffic going toward the ZeroTier node from the ZeroTier network or viceversa.

Indeed on OPNsense is absolutely not needed.

Since allowing bridging pose an additional load on the node, I would like to understand if and how I can disable without disabling multicast traffic.

You may need to change the “Flow Rules” for the ZT network on my.zerotier.com (see ZeroTier docs: https://docs.zerotier.com/rules/#rule-definition-language generally or examples here https://www.zerotier.com/blog/using-flow-rules-to-direct-users-to-services/ etc.). By default, the flow rules allow only UDP/TCP, but that won’t cover OSPF.

If lab/test, you may want to modify you flow rules to just “accept;”, instead of the “drop if not” likely in your rules. You can allow OSPF protocol specifically in ZT flow rules, but I’d recommend just test with the “accept;” initially to confirm it’s flow rules why OSPF isn’t working.

Thanks for reply.
Default flow rules allow for ip4, ip6 and arp, so ospf should be included.
By the way I also tried to put just accept and still doens’t work on Mikrotik.
But it works on OPNsense with default or permissives flow rules and without bridging enabled.
Seems a Mikrotik thing.

For what it’s worth, I have OSPF running between a few ‘tiks without any issues. My flow rules are default, and I have not enabled bridging, so I would believe that it should just work.

The only issue I had was relating to firewall, where I needed to allow OSPF (or any other traffic for that matter) on the Zerotier interface.

Perhaps.

Someone else had similar issues with OSPF broadcast mode and ZeroTier:
http://forum.mikrotik.com/t/question-on-massive-site-to-site-vpn-implementation/181097/14

There too, I thought it was flow rules, but OP was using Mikrotik controller which has NO flow rules. And I forgot the it the different IP protocol (89) that OSPF uses, not ethertype — so default flow rules should allow OSPF broadcast.

If you sure OSPF is generally working… it might worth a ticket to Mikrotik at help.mikrotik.com. Include a supout.rif if you do file a ticket.

With broadcast/multicast?
Without static neighbors?

I think my firewall it’s ok because with bridging enabled, OSPF is working.

Could also be a specific ROS version or hardware type.

My test are for a CCR2004 and a RB5009, both with ROS 18.2, both have the issue.

I can confirm that OSPF over ZeroTier is generally working when bridging is enabled for Mikrotik routers nodes.
But as soon I remove the bridging it stops working, and restart as soon as I set again bridging.
I think I’ll report to Mikrotik.

What I have is this:

 /zerotier/interface> /zerotier/interface/print detail 
Flags: D - dynamic, X - disabled; R - running 
 0  R name="zerotier1" mac-address=3E:65:02:6A:A1:37 mtu=2800 arp-timeout=auto network="xxxxxxxxxxxxxx" 
      instance=zt1 allow-managed=yes allow-global=no allow-default=no bridge=no dhcp=no 
      network-name="Area 0.0.0.0 (Backbone)" status="OK" type="PRIVATE"
  • no bridging enabled
 /zerotier/interface> /routing/ospf/interface-template/print where interfaces=zerotier1 
Flags: X - disabled, I - inactive 
 0   area=backbone_v2 interfaces=zerotier1 instance-id=0 type=broadcast retransmit-interval=5s transmit-delay=1s 
     hello-interval=10s dead-interval=40s priority=128 cost=1000 

 3   area=backbone_v3 interfaces=zerotier1 instance-id=0 type=broadcast retransmit-interval=5s transmit-delay=1s 
     hello-interval=10s dead-interval=40s priority=128 cost=1000
  • using broadcast
 /routing/ospf/neighbor/print 
Flags: V - virtual; D - dynamic 
 0  D instance=v3inst area=backbone_v3 address=fe80::3c6b:edff:fe56:c8d1%zerotier1 priority=128 
      router-id=192.168.32.1 dr=192.168.32.1 bdr=192.168.16.1 state="Full" state-changes=6 adjacency=19h26m1s 
      timeout=34s 

 1  D instance=v3inst area=backbone_v3 address=fe80::3c2d:d6ff:fe01:7419%zerotier1 priority=128 
      router-id=192.168.48.1 dr=192.168.48.1 bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s 

 4  D instance=v2inst area=backbone_v2 address=172.18.0.186 priority=128 router-id=192.168.48.1 dr=172.18.0.186 
      bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s 

 5  D instance=v2inst area=backbone_v2 address=172.18.0.205 priority=128 router-id=192.168.32.1 dr=172.18.0.205 
      bdr=172.18.0.139 state="Full" state-changes=5 adjacency=19h26m1s timeout=34s
  • neighbors found (output slightly redacted to remove some irrelevant Wireguard and direct connections).

This particular device is running ROS7.19beta8, and the neighbors are running 7.18.2. But it has been running forever on many different versions of ROS.

Thanks.
I can’t see anything special or substantially different from my setup.
When I turn off bridging in ZeroTier Central for my Mikrotik Node, I can’t see anymore OSPF traffic logged by my very first Raw Prerouting firewall rule.
OSPF log in Mikrotik only show “send-hello”

Just curious, but why even bother using OSPF on a Layer 2 network? Even using OSPF with IP seems kind of weird since all internal routing is already built into ZeroTier, right? Maybe I’m missing something here…

There are a few use cases for it.

  1. ZeroTier’s controller under the free version can only hold 2 or 3 routes (paid version is 128 routes I believe) so if you need more complex routing on an existing ZeroTier network, you might have redistribute between the zerotier routing table and other routers.

  2. ZeroTier allows you to create multiple networks. Not all of them have to be designed as remote access for users. It’s not uncommon to build one network for remote users and another for DCI or DCI mgmt. I’ve created several in this fashion.

  3. If you don’t want to manage multiple VPN types. ZeroTier may not always be the “fastest” VPN type, but it’s one of the easiest. If you already have a gateway VPN router that’s running ZT and want to add networks or functionality to reach other ZT routers, it’s much easier administratively to keep building ZT networks and mix it with dynamic routing as opposed to managing ZT, Wireguard, gre/ipsec, etc, etc.

ZeroTier doesn’t have visible “routes”, just virtual networks and devices/routers associated to a virtual network. Since last year, new free accounts get 10 devices/routers total, while older accounts had 25 (still valid). Extra licenses are very cheap.

When someone asks about OSPF or other routing protocols on top of ZeroTier, it’s usually a misunderstanding. Sure, there might be special cases, but people planning for those are most likely already familiar with SD-WAN solutions like ZeroTier and don’t need to ask about it here (unless it’s about ROS v7 quirks).

That’s not completely accurate. ZeroTier uses the routing table just like anything else. The ZeroTier networks you create inject connected and dynamic static routes into the OS of the device the ZT software runs on top of. The controller maintains a routing table that is pushed to every endpoint. It doesn’t interact with dynamic routing protocols at the controller level, but you can absolutely redistribute ZT VPN routes into OSPF, IS-IS, BGP, etc. This is very common when using ZT for OOB mgmt with data center networks.

The device count you’re mentioning is separate from the controller route count.

There is absolutely no distinction in the type of devices used by ZeroTier. It can be ten smartphones or ten routers connecting the same number of subnets. What matters is the number of activated devices listed in the management console. Also, there’s no such thing as a “controller route count” exposed or checked by the ZeroTier controller.

You have virtual networks, and each one simply tracks its active endpoints. Each virtual network can be configured as a Layer 2 virtual switch, a Layer 3 virtual router, or a mix of both, that might be managed by the ZeroTier Rules Engine.

ZeroTier does of course utilize standard IP routing on each endpoint device, which is kinda basic networking. What I was referring to is the internal ZeroTier protocol and its peer-to-peer routing mechanisms, which are abstracted away from the user. There is no way to configure those internal routes manually, and they’re not exposed like in traditional routes.

The ZeroTier controller injects connected routes into the OS routing table on the endpoints, but this happens behind the scenes when you connect to a virtual SD-WAN network. And while it is true that you can redistribute the virtual subnet, like any other subnet, into OSPF, BGP or similar protocols, that is typically only relevant in more complex SDN setups such as data centers or hybrid SD-WAN environments.

For most users, especially those asking about adding OSPF/BGP on top of ZeroTier, it is simply overkill. ZeroTier already handles the routing needed for full connectivity within its core virtual network. People familiar with SD-WAN generally know this, which is why those questions usually indicate a misunderstanding of what ZeroTier already provides.

Larsa makes good points. Personally I’d use ZT routes if possible, since it just so simple.

I’ve assumed OP already had OSPF infra, perhaps with non-ZT things, and there OSPF over ZT would seem reasonable. But if you’re using OSPF for route distribution ONLY for ZeroTier, that would seem silly.

The OP said “OSPF over a ZeroTier L2 domain (so mode broadcast),” which sounds a bit contradictory. OSPF is basically a L3 routing helper, so I don’t get how Layer 2 fits into the picture after reading the thread. I can understand running OSPF on one of the nodes to expose the ZT network, but I’d really like to hear more about what use case the OP has in mind for “ZT over L2”.

My idea was to use ZT as a secondary way for distributing routes and route packets among routers when better ways fails or are impossible.

I mean some of the routers partecipating in OSPF are connected all to the same ZeroTier network / L2 domain, so brodcast mode should be possible.
But every routers also belongs to other networks.

Returning to the OP originally problem, I previously stated that it works for me… and it does… on some of my routers.

I currently have 3 MikroTik routers connected to the same ZeroTier network, and OSPF is working on two of them over Zerotier. On the third, a RB5009, running 7.18.2, I only see outgoing multicast traffic, no incoming traffic when running the packet sniffer. I suspected that this might be a firewall issue, but unicast traffic is passing with no issues. I have coincidentally had some issues running RoMon over Zerotier. RoMon over Zerotier is working/not working, on the same devices as OSPF.

I suspect that there may be a general issue with multicast on some configurations, while I cannot spot the difference between my working and non working device, which is a combination of RB5009 on 7.18.2 (not working, no incoming multicast on Zerotier interface)
L009 and 7.19beta8 (working)
hAP ax^2 on 7.18.2 (working)

It is not super important for me to get this working, as Zerotier is primarily serving as redundancy for Wireguard tunnels, so I won’t be puting more effort into debugging it for now.

would you mind just trying allowing bridging on zerotier central for RB5009 to see if you start receiving multicast/OSPF packets?