Reach LAN from Zerotier with own controller

Hello,

Can not reach any of my LAN addresses from ZeroTier network, I’m using own controller on MT.

I had a very basic Zerotier configuration, following docs here
https://help.mikrotik.com/docs/display/ROS/ZeroTier
and using Managed Routes on my.zerotier.com I was able to reach my LAN addresses from ZT and other way round.

At some point decided to switch from global https://my.zerotier.com to controller located on MT. Now I can reach ZT addresses from LAN but can not reach LAN from ZT.

My config:

> print detail
Flags: X - disabled, R - online; F - tcp-fallback
 0 R  ;;; ZeroTier Central controller - https://my.zerotier.com/
      name="zt1" disabled=no port=9993
      identity="914..."
      interfaces=all route-distance=1
      identity.public="914..."
      state="running"
> /zerotier/controller/print detail
Flags: X - disabled, I - inactive
 0 I disabled=no instance=zt1 name="ZTCP1" network="914..." private=yes broadcast=yes mtu=2800 multicast-limit=32 ip-range=12.12.12.10-12.12.12.255 ip6-rfc4193=no ip6-6plane=no routes=192.168.88.0@12.12.12.12,12.12.12.0

> /zerotier/interface/print detail
Flags: D - dynamic, X - disabled; R - running
 0    name="zti1" mac-address=8E:63:00:00:00:00 mtu=2800 arp-timeout=auto network="914..." instance=zt1 allow-managed=yes allow-global=no allow-default=no
 
> /ip/firewall/filter/print
Flags: X - disabled, I - invalid; D - dynamic
 0    chain=forward action=accept in-interface=zti1

 1    chain=input action=accept in-interface=zti1

 2  D ;;; special dummy rule to show fasttrack counters
      chain=forward action=passthrough

 3    ;;; defconf: accept established,related,untracked
      chain=input action=accept connection-state=established,related,untracked

 4    ;;; defconf: drop invalid
      chain=input action=drop connection-state=invalid

 5    ;;; defconf: accept ICMP
      chain=input action=accept protocol=icmp

 6    ;;; defconf: accept to local loopback (for CAPsMAN)
      chain=input action=accept dst-address=127.0.0.1

 7    ;;; defconf: drop all not coming from LAN
      chain=input action=drop in-interface-list=!LAN

 8    ;;; defconf: accept in ipsec policy
      chain=forward action=accept ipsec-policy=in,ipsec

 9    ;;; defconf: accept out ipsec policy
      chain=forward action=accept ipsec-policy=out,ipsec

10    ;;; defconf: fasttrack
      chain=forward action=fasttrack-connection hw-offload=yes connection-state=established,related

11    ;;; defconf: accept established,related, untracked
      chain=forward action=accept connection-state=established,related,untracked

12    ;;; defconf: drop invalid
      chain=forward action=drop connection-state=invalid

13    ;;; defconf: drop all from WAN not DSTNATed
      chain=forward action=drop connection-state=new connection-nat-state=!dstnat in-interface-list=WAN
      
> /ip/firewall/nat/print detail
Flags: X - disabled, I - invalid; D - dynamic
 0    ;;; defconf: masquerade
      chain=srcnat action=masquerade out-interface-list=WAN log=no log-prefix="" ipsec-policy=out,none

> /ip/route/print detail
Flags: D - dynamic; X - disabled, I - inactive, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, i - is-is, d - dhcp, v - vpn, m - modem, y - bgp-mpls-vpn; H - hw-offloaded; + - ecmp
   DAd   dst-address=0.0.0.0/0 routing-table=main gateway=10.0.1.254 immediate-gw=10.0.1.254%ether1 distance=1 scope=30 target-scope=10 vrf-interface=ether1 suppress-hw-offload=no

   DAc   dst-address=10.0.1.0/24 routing-table=main gateway=ether1 immediate-gw=ether1 distance=0 scope=10 suppress-hw-offload=no local-address=10.0.1.31%ether1

   DAc   dst-address=12.12.12.0/24 routing-table=main gateway=zti1 immediate-gw=zti1 distance=0 scope=10 suppress-hw-offload=no local-address=12.12.12.12%zti1

   DAc   dst-address=192.168.88.0/24 routing-table=main gateway=bridge immediate-gw=bridge distance=0 scope=10 suppress-hw-offload=no local-address=192.168.88.1%bridge
   
> /ip/address/print detail
Flags: X - disabled, I - invalid, D - dynamic
 0   ;;; defconf
     address=192.168.88.1/24 network=192.168.88.0 interface=bridge actual-interface=bridge

 1 D address=10.0.1.31/24 network=10.0.1.0 interface=ether1 actual-interface=ether1

 2   address=12.12.12.12/24 network=12.12.12.0 interface=zti1 actual-interface=zti1

Thanks for Help!

The ZeroTier interface works just like an Ethernet interface. You need to allow and set up proper routing between your LAN and the ZT subnet, or alternatively use src-nat from ZT to the LAN.

I assume you’ve already added your LAN in ZeroTier Central > Networks > Advanced Settings > Managed Routes.

Well, the /zerotier/controller is marked as “I” INVALID, so that a first level problem here. You have

 /zerotier/controller/print detail
Flags: X - disabled, I - inactive
 0 I disabled=no instance=zt1 name="ZTCP1" network="914..." private=yes broadcast=yes mtu=2800 multicast-limit=32 ip-range=12.12.12.10-12.12.12.255 ip6-rfc4193=no ip6-6plane=no routes=192.168.88.0@12.12.12.12,12.12.12.0

If it’s invalid, the controller is not running. Specifically I think the routes need use the CIDR subnet for (LAN) route to be:
routes=192.168.88.0@12.12.12.12,12.12.12.0**/24**
…but not 100% that your only issue since using the controller on Mikrotik does make this more complex.

I did not as I’m using own controller as per snippet above. I have added LAN in my own controller though.

my bad, I was changing config last minute for cleaner printouts it was and is VALID

/zerotier/controller/print detail
Flags: X - disabled, I - inactive
0 disabled=no instance=zt1 name=“ZTCP1” network=“914…” private=yes broadcast=yes mtu=2800 multicast-limit=32 ip-range=12.12.12.10-12.12.12.255 ip6-rfc4193=no ip6-6plane=no routes=192.168.88.0@12.12.12.12,12.12.12.0

Also applied your suggestion

/zerotier/controller/print detail
Flags: X - disabled, I - inactive
0 disabled=no instance=zt1 name=“ZTCP1” network=“914…” private=yes broadcast=yes mtu=2800 multicast-limit=32 ip-range=12.12.12.10-12.12.12.255 ip6-rfc4193=no ip6-6plane=no routes=192.168.88.0@12.12.12.12,12.12.12.0/24

…but no effect on behavior.

Can you spot any traffic from the ZT interface to your LAN using the ROS packet sniffer?

Btw, what does the zerotier-cli peer status say?

This shows up when I ping one of the LAN addressed from ZT


tool/sniffer/packet> print detail
 0 time=33.871 num=1 direction=tx src-mac=8E:63: dst-mac=33:33:00: interface=zti1 src-address=fe80:: (discovery) dst-address=ff02::1:5678 (discovery) protocol=ipv6 ip-protocol=udp size=225 cpu=3 ip-packet-size=171 ttl=1

 1 time=33.871 num=2 direction=tx src-mac=8E:63: dst-mac=FF:FF:FF: interface=zti1 src-address=12.12.12.12:5678 (discovery) dst-address=255.255.255.255:5678 (discovery) protocol=ip ip-protocol=udp size=205 cpu=3 ip-packet-size=191 ip-header-size=20 dscp=0 identification=0
   fragment-offset=0 ttl=64

 2 time=33.871 num=3 direction=tx src-mac=8E:63: dst-mac=01:00:0C: interface=zti1 protocol=802.2 size=123 cpu=3

 3 time=33.871 num=4 direction=tx src-mac=8E:63:1A: dst-mac=01:80:C2: interface=zti1 protocol=lldp size=174 cpu=3



Btw, what does the zerotier-cli peer status say?


zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
62f865ae71 -      PLANET    -1 RELAY
778cde7190 -      PLANET   156 RELAY 24186    84819    103.195.103.66/9993
914.. 1.10.3 LEAF       9 RELAY 3920     6092     192.168.88.1/9993
cafe04eba9 -      PLANET   102 RELAY 3920     110219   84.17.53.155/9993
cafe9efeb9 -      PLANET    -1 RELAY
NOTE: Currently tunneling through a TCP relay. Ensure that UDP is not blocked.

Alright, dual stack (hmm…). It might be a routing issue since there aren’t any replay packets and I don’t see any ICMP packets coming in. As a temporary workaround for IPv4, try a source NAT approach using the command below. Just replace ZZZZZ with your ZT subnet and XXXXX with the name of your LAN interface, like ether1 or bridge, etc.:

‘/ip firewall nat add chain=srcnat src-address=ZZ.ZZ.ZZ.ZZ/24 out-interface=XXXXX action=masquerade’

Run a packet sniffer again, and this time let it ping every second for 10 seconds or more while recording. Btw, what ZT subnet and other settings are you using? Global, etc.. You don’t need to censor the subnets for your LAN or ZT if they’re private, it’ll make it easier to follow the trace.

No change in behavior after applying your suggestion.

I posted my settings up top, do you need something in particular?

I have now disabled this rule (just to be sure), and set routes this way

routes=192.168.88.0**/24**@12.12.12.12,12.12.12.0/24


> /zerotier/controller/print detail
Flags: X - disabled, I - inactive
 0   disabled=no instance=zt1 name="ZTCP1" network="914..." private=yes broadcast=yes mtu=2800 multicast-limit=32 ip-range=12.12.12.10-12.12.12.255 ip6-rfc4193=no ip6-6plane=no routes=192.168.88.0/24@12.12.12.12,12.12.12.0/24

and now I’m getting zt->LAN connectivity as expected.

Thanks all for the pointers.