We have a unique situation at one of our sites where we would like to use Zerotier. We receive a /30 from the ISP and a /29 to use for publics. The /30 is where the default gateway lives but is not routable to the internet. We have to use one of the /29 addresses in order to route to the internet. Zerotier, from what we can tell, does not support a source address and will try to use the /30 for a connection. This causes Zerotier to not be able to connect. Anyone have a good idea on how to get around this?
Strange my impression was that zerotier was transparent to WAN shenanigans.
@Wolfraider: We use 2 public ip addresses at a branch office without any problems. Do you use per-packet load balancing?
nope
# feb/10/2023 09:04:36 by RouterOS 7.6
#
# model = RB4011iGS+
/interface bridge
add name=Loopback protocol-mode=none
add auto-mac=no comment=defconf name=bridge protocol-mode=none
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip dhcp-server option
add code=43 name=unifi value=0xXXXXXXXXXXX
/ip pool
add name=dhcp_pool1 ranges=172.16.1.1-172.16.255.254
/ip dhcp-server
add address-pool=dhcp_pool1 interface=bridge lease-time=1d10m name=dhcp1
/zerotier
set zt1 comment="ZeroTier Central controller - https://my.zerotier.com/" identity="XXXXXXXXXXXXXXXXXXXXXXX" name=zt1 port=9993
/zerotier interface
add allow-default=no allow-global=no allow-managed=no disabled=no instance=zt1 name=zerotier1 network=XXXXXXXXXXX
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge comment=defconf ingress-filtering=no interface=ether6
add bridge=bridge comment=defconf ingress-filtering=no interface=ether7
add bridge=bridge comment=defconf ingress-filtering=no interface=ether8
add bridge=bridge comment=defconf ingress-filtering=no interface=ether9
add bridge=bridge comment=defconf ingress-filtering=no interface=ether10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp-sfpplus1
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=172.16.0.1/16 comment=defconf interface=bridge network=172.16.0.0
add address=X.X.0.122/30 interface=ether1 network=X.X.0.120 [b]#This address range is not routable to the internet[/b]
add address=X.X.105.41/29 interface=Loopback network=X.X.105.40
/ip dhcp-server network
add address=172.16.0.0/16 dhcp-option=unifi dns-server=1.1.1.1,8.8.8.8 gateway=172.16.0.1
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=src-nat chain=srcnat src-address=172.16.0.0/16 to-addresses=X.X.105.41
/ip firewall service-port
set h323 disabled=yes
set sip disabled=yes
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=X.X.0.121 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
By default, ZeroTier uses “all” interface in its discovery of paths. Perhaps ZT’s ZL1 protocol is confused by a public address that isn’t routable, but dunno.
But you should be able to fix this setting /zerotier set interfaces=Loopback - so it will only use the Loopback bridge interface, not ether1, to get out.
I have to finish a few things today but I might have some spare time tomorrow to give it try (no promise)
Tried that with no luck
You do know that if a flying mouse doesnt keep promises, the feathers start to fall out…
ZeroTier network isn’t allow through the firewall (e.g. the results of the tunnels going out the WAN)
Perhaps, this would do the trick:.
/interface list member add comment=defconf interface=zerotier1 list=LAN
I grabbed a spare 4011 router that I had laying around and setup a /30 PTP network using a private IP address and assigned a public IP address to the loopback. No firewall rules. I can ping 8.8.8.8 using the src address of the public IP address I assigned to the router. Zerotier would not connect, even when changing the interface to the loopback.
Moved the test router to have a routable public /30 PTP address and Zerotier worked. Looks like it doesnt like non routable PTP addresses.
It’s a strange case: non-routable public IPs..
Still ZT just have just not used the route if it could not get to a ZT root/etc server & found the default route the router was using (assuming the router could “:ping 1.1.1.1” etc.). Perhaps firewall blocked too here.
So I would have said as a NAT rule for the tunnel to get a “source address”, but you can’t identify the ZT tunnels in the firewall.
But the underlying “issue” is that Mikrotik doesn’t treat the zt1 instance as an interface – only the zerotrier1 network is an interface. To the firewall connection tracking, it’s unnamed local traffic that can’t be identified in the firewall to do anything specific to the ZT tunnels. I ran into similar thing in troubleshoot/understand where the tunnel/ZT1 traffic was going, so I wrote a script that tries that uses /zerotier/peers to figure out where ZT tunnels were going, see http://forum.mikrotik.com/t/zerotier-on-mikrotik-a-rosetta-stone-v7-1-1/155978/18 but it’s not perfect at doing it (e.g. I should fix some bugs in that script one day).
@Wolfraider, did you manage to get it to work? Otherwise, I’m happy to give it a try.
However before we continue, I’d like to have a fully working export minus sensitive information like zt-networkid and public addresses. The previous export had some gaps and I’m too lazy and can’t bear to reverse engineer the network topology.