Perhaps.Seems a Mikrotik thing.
With broadcast/multicast?For what it’s worth, I have OSPF running between a few ‘tiks without any issues. My flow rules are default, and I have not enabled bridging, so I would believe that it should just work.
The only issue I had was relating to firewall, where I needed to allow OSPF (or any other traffic for that matter) on the Zerotier interface.
I can confirm that OSPF over ZeroTier is generally working when bridging is enabled for Mikrotik routers nodes.There too, I thought it was flow rules, but OP was using Mikrotik controller which has NO flow rules. And I forgot the it the different IP protocol (89) that OSPF uses, not ethertype — so default flow rules should allow OSPF broadcast.Seems a Mikrotik thing.
If you sure OSPF is generally working... it might worth a ticket to Mikrotik at help.mikrotik.com. Include a supout.rif if you do file a ticket.
/zerotier/interface> /zerotier/interface/print detail
Flags: D - dynamic, X - disabled; R - running
0 R name="zerotier1" mac-address=3E:65:02:6A:A1:37 mtu=2800 arp-timeout=auto network="xxxxxxxxxxxxxx"
instance=zt1 allow-managed=yes allow-global=no allow-default=no bridge=no dhcp=no
network-name="Area 0.0.0.0 (Backbone)" status="OK" type="PRIVATE"
/zerotier/interface> /routing/ospf/interface-template/print where interfaces=zerotier1
Flags: X - disabled, I - inactive
0 area=backbone_v2 interfaces=zerotier1 instance-id=0 type=broadcast retransmit-interval=5s transmit-delay=1s
hello-interval=10s dead-interval=40s priority=128 cost=1000
3 area=backbone_v3 interfaces=zerotier1 instance-id=0 type=broadcast retransmit-interval=5s transmit-delay=1s
hello-interval=10s dead-interval=40s priority=128 cost=1000
/routing/ospf/neighbor/print
Flags: V - virtual; D - dynamic
0 D instance=v3inst area=backbone_v3 address=fe80::3c6b:edff:fe56:c8d1%zerotier1 priority=128
router-id=192.168.32.1 dr=192.168.32.1 bdr=192.168.16.1 state="Full" state-changes=6 adjacency=19h26m1s
timeout=34s
1 D instance=v3inst area=backbone_v3 address=fe80::3c2d:d6ff:fe01:7419%zerotier1 priority=128
router-id=192.168.48.1 dr=192.168.48.1 bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s
4 D instance=v2inst area=backbone_v2 address=172.18.0.186 priority=128 router-id=192.168.48.1 dr=172.18.0.186
bdr=0.0.0.0 state="Init" state-changes=1 timeout=34s
5 D instance=v2inst area=backbone_v2 address=172.18.0.205 priority=128 router-id=192.168.32.1 dr=172.18.0.205
bdr=172.18.0.139 state="Full" state-changes=5 adjacency=19h26m1s timeout=34s
Thanks.What I have is this:
Just curious, but why even bother using OSPF on a Layer 2 network? Even using OSPF with IP seems kind of weird since all internal routing is already built into ZeroTier, right? Maybe I’m missing something here...
ZeroTier doesn’t have visible “routes”, just virtual networks and devices/routers associated to a virtual network. Since last year, new free accounts get 10 devices/routers total, while older accounts had 25 (still valid). Extra licenses are very cheap.
My idea was to use ZT as a secondary way for distributing routes and route packets among routers when better ways fails or are impossible.I've assumed OP already had OSPF infra, perhaps with non-ZT things, and there OSPF over ZT would seem reasonable. But if you're using OSPF for route distribution ONLY for ZeroTier, that would seem silly.
I mean some of the routers partecipating in OSPF are connected all to the same ZeroTier network / L2 domain, so brodcast mode should be possible.The OP said "OSPF over a ZeroTier L2 domain (so mode broadcast)," which sounds a bit contradictory.
would you mind just trying allowing bridging on zerotier central for RB5009 to see if you start receiving multicast/OSPF packets?Returning to the OP originally problem, I previously stated that it works for me… and it does… on some of my routers.
I currently have 3 MikroTik routers connected to the same ZeroTier network, and OSPF is working on two of them over Zerotier. On the third, a RB5009, running 7.18.2, I only see outgoing multicast traffic, no incoming traffic when running the packet sniffer. I suspected that this might be a firewall issue, but unicast traffic is passing with no issues. I have coincidentally had some issues running RoMon over Zerotier. RoMon over Zerotier is working/not working, on the same devices as OSPF.
I suspect that there may be a general issue with multicast on some configurations, while I cannot spot the difference between my working and non working device, which is a combination of RB5009 on 7.18.2 (not working, no incoming multicast on Zerotier interface)
L009 and 7.19beta8 (working)
hAP ax^2 on 7.18.2 (working)
It is not super important for me to get this working, as Zerotier is primarily serving as redundancy for Wireguard tunnels, so I won’t be puting more effort into debugging it for now.
I mean some of the routers partecipating in OSPF are connected all to the same ZeroTier network / L2 domain, so brodcast mode should be possible.
But every routers also belongs to other networks.
You can use ZT to push any route. ZT does not care if the destination is within ZT's IP range — ZT is agnostic on gateway so you can often use ZT for just route distribution. And RouterOS will happy add whatever it gets from ZT directly to the [main] routing table.I might ask the other way, what would your recommendation instead?
For me ZeroTier is kind of a backup.Easy configuration of OSPF. For Wireguard is is necessary to configure static neighbors. While for Zerotier (and L2 in general) it just works, as the neighbors are automatically discovered through multicast.
The routers are in different sites.
I get that OSPF can work over the shared L2 domain, but that still leaves me wondering why you would want to do it this way.
If all routers are already L2-connected through ZeroTier, it feels a bit like connecting a big switch to all your subnets instead of properly routing between them. I mean, wouldn't it be simpler and cleaner to just run OSPF over the existing routed networks instead of bridging everything?
Maybe there is a specific reason for the setup that I have missed?
Fair enough, I get if you're using OSPF that should be definitive. You just ask, and do think using ZT for distributing something like WireGuard or other routes is likely simpler if one did not already know OSPF, thus I mention it. With point being the traffic doesn't have to use [slower] ZT gateway, even if ZT injected route. So just different approach...There are also few non-OSPF routers connected by Wireguard to the hub and few direct wireguard links between most important sites.
ZeroTier is kind of a backup for wireguard. ZeroTier is slower with our slow connections. And I don't want to rely on routes manually defined in ZeroTier network.
I dont' get what you recommend, because if I have to avoid L2 coupling, I think I need to avoid ZeroTier completely, so I should stay with L3 wireguard tunnels and nothing else?@unlikely, I get what you are saying about using ZeroTier as backup and your WireGuard links not needing static neighbor setup.
BUT still, the same idea I mentioned to @xrlls applies here. If you already have ZeroTier in the mix, running OSPF on top is just extra work without any benefit, quite the contrary as I explained to @xrlls. Using Layer 2 across sites might cause broadcast storms, messy network coupling and make everything harder to scale and troubleshoot.
Sticking to clean Layer 3 routing is way safer and easier. If you go L2, you are on your own.
If you still do not get it, good luck!
I think OP isn't actually bridging zerotier on RouterOS bridge – although be to confirm... OP is just checking the "Allow Bridging" option on the ZT controller, AFAIK...If I gave the impression that I was bridging everything as one big happy L2 network, that is not what I am doing, and I agree that in such a configuration OSPF does not make sense.
@Amm0 already explained to look at sniffers, or lab a smaller example. But ZeroTier "L2" should be transparent to "L3" [multicast] OSPF.Maybe @Amm0 can help you out here.
Until a few days ago, I was using the same combined WireGuard and ZeroTier approach to manage several sites, with static routes of varying distances configured on each router.Fair enough, I get if you're using OSPF that should be definitive. You just ask, and do think using ZT for distributing something like WireGuard or other routes is likely simpler if one did not already know OSPF, thus I mention it. With point being the traffic doesn't have to use [slower] ZT gateway, even if ZT injected route. So just different approach...
I'm flummox at what the issue here is, since OSPF broadcast should "just work" (without bridging IMO). Perhaps check /tool/sniffer to see if OSPF multicast is even hitting the zerotier interface (and ideally on far-end, to see if got there) – that confirm if ZT config issue, OR, bug in ZT+OSPF on RouterOS.
Also, you don't have use broadcast with OSPF, which avoid the mutlicast issue with ZT and potential complexity @Larsa mentions. While I'm more sanguine than @Larsa of OSPF over ZT as a use case...that be predicated on familiarity and existing usage of OSPF.
Correct.I think OP isn't actually bridging zerotier on RouterOS bridge – although be to confirm... OP is just checking the "Allow Bridging" option on the ZT controller, AFAIK...
When I mention "enabling bridging", I am specifically referring to checking "Allow Bridging" in ZeroTier Central. The ZeroTier interface on the MikroTik devices is never part of any bridge.Please note that OSPF generally functions correctly in my setup when bridging is enabled on two MikroTik nodes (CCR2004 running ROS 7.18.2 and RB5009 running ROS 7.18.2 or 7.19beta8) on ZeroTier Central.
When bridging is disabled, OSPF adjacency is maintained for a short period. However, if I disable and re-enable the OSPF instance while bridging is still disabled, neighbors are typically not discovered. The OSPF logs on each node display only “sending hello.” To restore OSPF functionality, I have to re-enable bridging.
And here is my reply:"Allow Ethernet Bridging" enables the ZeroTier node to bridge Layer 2 multicast traffic (i.e., Ethernet frames with multicast destination MAC addresses).
This includes protocols like OSPF Hello packets, which are necessary in this scenario.
What do you think?I don’t need to bridge multicast traffic or any traffic.
What I need is for my MikroTik device to receive multicast traffic from the ZeroTier network.
I don't think I can agree.Support gave you an adequate answer. You can run whatever L2 traffic you want over Zerotier, but you need to know how to manage it with flow rules. And yes, it works as expected.
I’m open to all advice and suggestions and truly appreciate everyone’s contributions.Well, since you're not listening to my advice or bothering to answer my questions, you're on your own. Good luck!
Interesting feature to know (BTH) I wasn't aware of.The only other approach is abuse BackToHome (BTH) - that does deal with CGNAT and is just WireGuard under the covers. i.e. if a site had a fixed public IP, and LTE back... BTH be same as WG when "primary WAN", but if failover BTH "proxy" WG via LTE CGNAT. The issue is @normis repeats it's a "home" feature (as implied by name), so whether that's be a "better" approach IDK.