I’m trying to setup OSPF over a ZeroTier L2 domain (so mode broadcast) among two Mikrotik routers and an OPNsense box.
Apparently “Allow Ethernet bridging” should be enabled on ZeroTier central for the Mikrotik routers otherwise Multicast Hello packets are not received and adjacency can not be established.
From my understanding this setting disable a filter that prevent traffic from mac addresses unrelated to the ZeroTier node to enter the ZeroTier network.
It should have nothing to do with preventing multicast traffic going toward the ZeroTier node from the ZeroTier network or viceversa.
Indeed on OPNsense is absolutely not needed.
Since allowing bridging pose an additional load on the node, I would like to understand if and how I can disable without disabling multicast traffic.
If lab/test, you may want to modify you flow rules to just “accept;”, instead of the “drop if not” likely in your rules. You can allow OSPF protocol specifically in ZT flow rules, but I’d recommend just test with the “accept;” initially to confirm it’s flow rules why OSPF isn’t working.
Thanks for reply.
Default flow rules allow for ip4, ip6 and arp, so ospf should be included.
By the way I also tried to put just accept and still doens’t work on Mikrotik.
But it works on OPNsense with default or permissives flow rules and without bridging enabled.
Seems a Mikrotik thing.
For what it’s worth, I have OSPF running between a few ‘tiks without any issues. My flow rules are default, and I have not enabled bridging, so I would believe that it should just work.
The only issue I had was relating to firewall, where I needed to allow OSPF (or any other traffic for that matter) on the Zerotier interface.
There too, I thought it was flow rules, but OP was using Mikrotik controller which has NO flow rules. And I forgot the it the different IP protocol (89) that OSPF uses, not ethertype — so default flow rules should allow OSPF broadcast.
If you sure OSPF is generally working… it might worth a ticket to Mikrotik at help.mikrotik.com. Include a supout.rif if you do file a ticket.
I can confirm that OSPF over ZeroTier is generally working when bridging is enabled for Mikrotik routers nodes.
But as soon I remove the bridging it stops working, and restart as soon as I set again bridging.
I think I’ll report to Mikrotik.
/routing/ospf/neighbor/print
Flags: V - virtual; D - dynamic
0 D instance=v3inst area=backbone_v3 address=fe80::3c6b:edff:fe56:c8d1%zerotier1 priority=128
router-id=192.168.32.1 dr=192.168.32.1 bdr=192.168.16.1 state="Full" state-changes=6 adjacency=19h26m1s
timeout=34s
1 D instance=v3inst area=backbone_v3 address=fe80::3c2d:d6ff:fe01:7419%zerotier1 priority=128
router-id=192.168.48.1 dr=192.168.48.1 bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s
4 D instance=v2inst area=backbone_v2 address=172.18.0.186 priority=128 router-id=192.168.48.1 dr=172.18.0.186
bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s
5 D instance=v2inst area=backbone_v2 address=172.18.0.205 priority=128 router-id=192.168.32.1 dr=172.18.0.205
bdr=172.18.0.139 state="Full" state-changes=5 adjacency=19h26m1s timeout=34s
neighbors found (output slightly redacted to remove some irrelevant Wireguard and direct connections).
This particular device is running ROS7.19beta8, and the neighbors are running 7.18.2. But it has been running forever on many different versions of ROS.
Thanks.
I can’t see anything special or substantially different from my setup.
When I turn off bridging in ZeroTier Central for my Mikrotik Node, I can’t see anymore OSPF traffic logged by my very first Raw Prerouting firewall rule.
OSPF log in Mikrotik only show “send-hello”
Just curious, but why even bother using OSPF on a Layer 2 network? Even using OSPF with IP seems kind of weird since all internal routing is already built into ZeroTier, right? Maybe I’m missing something here…
ZeroTier’s controller under the free version can only hold 2 or 3 routes (paid version is 128 routes I believe) so if you need more complex routing on an existing ZeroTier network, you might have redistribute between the zerotier routing table and other routers.
ZeroTier allows you to create multiple networks. Not all of them have to be designed as remote access for users. It’s not uncommon to build one network for remote users and another for DCI or DCI mgmt. I’ve created several in this fashion.
If you don’t want to manage multiple VPN types. ZeroTier may not always be the “fastest” VPN type, but it’s one of the easiest. If you already have a gateway VPN router that’s running ZT and want to add networks or functionality to reach other ZT routers, it’s much easier administratively to keep building ZT networks and mix it with dynamic routing as opposed to managing ZT, Wireguard, gre/ipsec, etc, etc.
ZeroTier doesn’t have visible “routes”, just virtual networks and devices/routers associated to a virtual network. Since last year, new free accounts get 10 devices/routers total, while older accounts had 25 (still valid). Extra licenses are very cheap.
When someone asks about OSPF or other routing protocols on top of ZeroTier, it’s usually a misunderstanding. Sure, there might be special cases, but people planning for those are most likely already familiar with SD-WAN solutions like ZeroTier and don’t need to ask about it here (unless it’s about ROS v7 quirks).
That’s not completely accurate. ZeroTier uses the routing table just like anything else. The ZeroTier networks you create inject connected and dynamic static routes into the OS of the device the ZT software runs on top of. The controller maintains a routing table that is pushed to every endpoint. It doesn’t interact with dynamic routing protocols at the controller level, but you can absolutely redistribute ZT VPN routes into OSPF, IS-IS, BGP, etc. This is very common when using ZT for OOB mgmt with data center networks.
The device count you’re mentioning is separate from the controller route count.
There is absolutely no distinction in the type of devices used by ZeroTier. It can be ten smartphones or ten routers connecting the same number of subnets. What matters is the number of activated devices listed in the management console. Also, there’s no such thing as a “controller route count” exposed or checked by the ZeroTier controller.
You have virtual networks, and each one simply tracks its active endpoints. Each virtual network can be configured as a Layer 2 virtual switch, a Layer 3 virtual router, or a mix of both, that might be managed by the ZeroTier Rules Engine.
ZeroTier does of course utilize standard IP routing on each endpoint device, which is kinda basic networking. What I was referring to is the internal ZeroTier protocol and its peer-to-peer routing mechanisms, which are abstracted away from the user. There is no way to configure those internal routes manually, and they’re not exposed like in traditional routes.
The ZeroTier controller injects connected routes into the OS routing table on the endpoints, but this happens behind the scenes when you connect to a virtual SD-WAN network. And while it is true that you can redistribute the virtual subnet, like any other subnet, into OSPF, BGP or similar protocols, that is typically only relevant in more complex SDN setups such as data centers or hybrid SD-WAN environments.
For most users, especially those asking about adding OSPF/BGP on top of ZeroTier, it is simply overkill. ZeroTier already handles the routing needed for full connectivity within its core virtual network. People familiar with SD-WAN generally know this, which is why those questions usually indicate a misunderstanding of what ZeroTier already provides.
Larsa makes good points. Personally I’d use ZT routes if possible, since it just so simple.
I’ve assumed OP already had OSPF infra, perhaps with non-ZT things, and there OSPF over ZT would seem reasonable. But if you’re using OSPF for route distribution ONLY for ZeroTier, that would seem silly.
The OP said “OSPF over a ZeroTier L2 domain (so mode broadcast),” which sounds a bit contradictory. OSPF is basically a L3 routing helper, so I don’t get how Layer 2 fits into the picture after reading the thread. I can understand running OSPF on one of the nodes to expose the ZT network, but I’d really like to hear more about what use case the OP has in mind for “ZT over L2”.
I mean some of the routers partecipating in OSPF are connected all to the same ZeroTier network / L2 domain, so brodcast mode should be possible.
But every routers also belongs to other networks.
Returning to the OP originally problem, I previously stated that it works for me… and it does… on some of my routers.
I currently have 3 MikroTik routers connected to the same ZeroTier network, and OSPF is working on two of them over Zerotier. On the third, a RB5009, running 7.18.2, I only see outgoing multicast traffic, no incoming traffic when running the packet sniffer. I suspected that this might be a firewall issue, but unicast traffic is passing with no issues. I have coincidentally had some issues running RoMon over Zerotier. RoMon over Zerotier is working/not working, on the same devices as OSPF.
I suspect that there may be a general issue with multicast on some configurations, while I cannot spot the difference between my working and non working device, which is a combination of RB5009 on 7.18.2 (not working, no incoming multicast on Zerotier interface)
L009 and 7.19beta8 (working)
hAP ax^2 on 7.18.2 (working)
It is not super important for me to get this working, as Zerotier is primarily serving as redundancy for Wireguard tunnels, so I won’t be puting more effort into debugging it for now.