I noticed something strange when setting up a D53G-5HacD2HnD&EG06-A to use Zerotier over LTE and Wi-Fi. It apears that ZeroTier can use inactive default routes in the routing table to re-estrablish its tunnel.
Here is what I did. I simulated a case where the Wi-Fi station connection has an internet failure upsteam by blocking traffic on the upstream router and clearing the connections in the firewall. This does not triger an ARP gateway check or a interface failure on the D53G. The routing table remains in a state where the Wi-Fi path default route remains active with an admin distance of 1 and the LTE default route remains inactive with an admin distance of two. In this case I see the ZeroTier tunnel somehow figure out that the default route via Wi-Fi is not working and starts working via the LTE connection automaticaly. So the ZeroTier tunnel is working but pings to the internet are not due to the default route still being the Wi-Fi path.
This looks like ZeroTier is able to somehow use the inactive default route via LTE while the rest of RouterOS is stuck on the broken Wi-Fi path. This is great for my use case but I did not see any documentation about this capability. It is acting like an SD-WAN router that autodetects internet failure and uses alternative internet paths automaticaly. It even switches back to using the active default route via Wi-Fi after internet is restored on that path.
Some additional details on this setup. This is going to be a remote access device that can be mailed to clients to allow remote access via Serial console and ethernet. I am building it to work over LTE but also be able to connect to onsite wifi to save on LTE data. Being able to auto detect ways to the internet is critical so that I don’t lose access to the device. It is doing exactly what I want but I dont fully trust it without seeing some documentation on how it is doing this.
The docs aren't clear what path selection logic is ZeroTier is using. While ZeroTier protocol does support many customization, including "multipath" support...RouterOS does not let you control this. See https://docs.zerotier.com/multipath/ what's not available to control, but the supposition is that it use balance-aware.
At a higher level, ZeroTier follows its own logic based on available interface to the ZeroTier instance – not following routing table – since it's looking for all possible peers. Since Wi-Fi appear to be a "bad" path (although theoretically if some phone was connected to same ZeroTier network it might even try using the LTE on phone to create a path - but I'm not sure about the exact logic, but that's kinda the idea of ZeroTier since it always want to find some path).
But if you want to avoid it using Wi-Fi, you can configure the interfaces on the ZeroTier instance (default: zt1) to not use "all". You can either pick the LTE interface, or use an /interface/list like WAN to force it only trying only real routes.
Oh thats interesting. Its not using the RouterOS routing table. I guess that makes sense as its running in its own container.
I wonder how ZeroTier obtains the next hop IP it should use for each interface the instance is configured to run on?
For my setup just the next hop as the LTE interfaces is enough for the default route as LTE conections are like point to point links and you can just send everything to the interface.
But for Wi-Fi it is not point to point as its a broadcast domain so you need to know what the IP of the default gateway is.
Is Zerotier viewing default routes in the RouterOS table to learn the default gateway to use on each interface?
If you have a setup with multiple paths to the internet is it worth adding all of the default routes to the routing table but with higher admin distances so that ZeroTier will see and use those extra paths?
My ZeroTier setup seems to prefer the path to the internet that RouterOS has as the active default route. When I restore internet to the RouterOS active default route path ZeroTier eventualy moves traffic back to match the RouterOS routing table. Is ZeroTier using some type of priority routing based off of what the active default route in the RouterOS routing table is?
Maybe all of this is pending an update to support multipath like you mentioned. But it would be nice to figure out what the current routing logic is.
I should have been a bit clear, ZT is using the information in the routing table but ZT tunnels do not following distance/scope, and instead uses it's own logic. Some of this is described by ZeroTier's docs on VL1 tunnels. For example, it prefers IPv6 to IPv4 when creating tunnels, among others.
The ZT docs explain a bit about the protocol. TL;DR is that is prefers direct connection between between peers, and since that is often possible with IPv6 and/or hole punching so it's why your primary internet is used over wireless since that be indirect connection thus less preferred. There are bunch of ZT options - not exposed by RouterOS - so control is pretty limited to the interface selection on the ZT instance in RouterOS.
The "Peers" view in ZeroTier that will show you some stats and the current connection (including if it's "active"). So you can view that see at least what potential paths exist there. But control them is not readily possible, outside restricting what interface it will look at.
100%. Also why its hard to be definitive on what's happening in your specific case. But the principle is that ZT will try VERY hard to keep some path up, now, whether that's the most optimal one is not easy to guarantee currently.
MikroTik has not done much recently to enhance its support. Like exposing the underlying ZeroTer config file mentioned in the ZT docs that offer a many possible schemes and customization. Or, for example, they offer a "low bandwidth mode" that be helpful for OOB management (my use case) since ZT tunneling is pretty "chatty" to ensure a direct path (at expense of using more bandwidth just to detect paths).
But on RouterOS, it's basically what interfaces do you want it to use & it will figure out the rest. In other words, it's more using the list of interfaces, than list of IP routes (although it can certainly read the routes internally).