We’ve recently done a deployment that involves ZeroTier, and a dual ISP set-up. The two ISP’s at the customer are configured for fail-over (both default routes for both ISP’s are configured, both with check-gateway, one with a lower weight than the other).
Currently ZeroTier connects, as expected over the active/primary ISP. However customer wants ZeroTier to use the back-up ISP.
We’ve tried the interface option under the instance menu, by default this is set to “ALL”. When we configure this for the interface connected to the back-up ISP ZeroTier is unable to connect. Only when I disconnect the primary ISP will it work over the back-up ISP. If I configure the ZeroTier instance interface to the interface of the primary ISP (assuming both ISP’s are active) the connection also remains online.
In other words, ZeroTier will only use the default route to connect to the internet.
Is there a way I can force ZeroTier to work over the secondary/back-up ISP?
Your suggestions sounds logical, I expected that’d do it. However it appears not to do much, although I do see some ZT traffic through the secondary ISP, which wasn’t the case previously.
I’ve made some adaptations:
I’ve changed DST/9993 to SRC/9993 for the mangle rules
I hadded a second mangle rule as prerouting, I see more traffic going through that one compared to the output rule
I challange I see is that although UDP/9993 is default most peers use other random UDP ports.
The interface is the inner tunnel traffic (e.g. “zerotier1” - the ZeroTier network/“LAN”) - so that not where routing/NAT is going wrong here. It’s the “outer” vt1 instance tunnels (e.g. VL1 tunnels to “peers”) the OP want going out a specific WAN & these (“zt1” etc) are NOT selectable as in “interface” in firewall. Now… using port to identify isn’t great either since there is NO guarantee ZT will always use port 9993 (although it the first it tries, but not only method)…
In theory, the default NAT WAN rule should work if the secondary WAN was selected at the ZT instance. So that’s kinda weird here. Also, ZT will use IPv6 FIRST… so if that’s enabled that’s what’s going to be used BEFORE IPv4 things.
Perhaps the config help figure this out. I think you may need to routing tables/rules, in addition to basic failover just using distance= on default routes.
as a last resort though it’s not the real solution he/she could install zerotier in the other device using his/her 2nd ISP as a next-hop then use ROMON if the purpose of this zerotier is just for management purposes no more fiddling with mangle
For my understanding, if the instance on the MT router is listening om UDP/9993, this is before NAT or after NAT? Hence if I use a mangle rule to mark all packets/connections originating from this UDP/9993, this should work? (although I believe I’ve already observed that it doesn’t)
