Question on using the Internal Zerotier Controller

RouterOS Beginner Basics
1

Hey all,

I’m experimenting with Zerotier and have been following the steps listed in the documentation at: https://help.mikrotik.com/docs/spaces/ROS/pages/83755083/ZeroTier#ZeroTier-Controller

I’ve successfully created a Zerotier connection using the Zerotier Central portal. I’ve also successfully created a connection using the controller example in the documentation. But I want to take it a step further.

First of all, in the documentation, they set up the network like this:

controller/add name=ZT-private instance=zt1 ip-range=172.27.27.10-172.27.27.20 private=yes routes=172.27.27.0/24

That’s fine but the problem is, while I get access to anything in the 172.27.27.0/24 network, packets are not routed to my LAN. My LAN is at 192.168.0.0/23. On top of that, I have a second address space on VLANs using the 10.0.0.0/8 network.

The documentation states:

routes (IP@GW; Default: ) Push routes in the following format:
Routes ::= Route[,Routes]
Route ::= Dst[@Gw]

But it seems like if I try anything other than the example format of just the Zerotier address space, I get no IP address assigned. Something is broken. I’ve tried things like this:

controller/add name=ZT-Test instance=zt1 ip-range=172.27.10.30-172.27.10.40 private=yes routes=17.27.10.0/24,192.168.0.0/23,10.0.0.0/8

or

controller/add name=ZT-Test instance=zt1 ip-range=172.27.10.10-172.17.10.20 private=yes routes=172.27.10.0/24,0.0.0.0/0@172.27.10.11

or

controller/add name=ZT-Test instance=zt1 ip-range=172.27.10.10-172.17.10.20 private=yes routes=172.27.10.0/24,192.168.0.0/23@172.27.10.11,10.0.0.0/8@172.27.10.11

In all of these cases, I fail to get an IP addresses handed out…Obviously I am doing something incorrect.

Where am I going wrong?

Thanks!

Jon

2

Well, to begin with, the documentation for the controller is a masterpiece of vagueness, to say the least. :wink: Unfortunately, the people who wrote it forgot to include an example of how to add a route to a gateway. The only cryptic and inconsistent explanation you get is:
routes (IP@GW; Default: ) Push routes in the following format:
Routes ::= Route[,Routes]
Route ::= Dst[@Gw]
Let’s say 172.27.27.11 is the ZeroTier address of the node that acts as the gateway to your LAN 192.168.0.0/23. Then you should write it exactly the way you did earlier ie ‘routes = 192.168.0.0/23@172.27.27.11, …’ Also, make sure your firewall allows forwarding between the related subnets.

Are the clients able to connect to the ZT network, and how does the routing table look like?

3

When I set the config up like that, I don’t get an IP Address assigned to the router’s ZT client. Nothing gets assigned. I assumed I was doing something wrong.

As far as forwarding between subnets - not sure. I’ve got multiple subnets on the router already. I don’t have an issue with those or when using a Wireguard VPN either. Is there another firewall config that is needed for ZT? I added the one shown in the ZT documentation.

4

The guys at Mikrotik are various levels of user friendliness :slight_smile:. The given example is one such. Actually it is exact and not in the least vague. The given syntax in given in the so-called Backus-Naur form. (https://en.wikipedia.org/wiki/Backus–Naur_form)

I’ve used the controller and it works reliably and exactly as it should.

You probably have to go over the tutorial step-by-step. For example: is the client showing up? Is it’s access restricted (a client has to be allowed into a private network) Is an address assigned to it on the controller side?

5

I have followed the tutorial exactly. When I enter the steps exactly like they say and with the route example they use (and I’ve tried it with multiple different IP subnets) things work fine and the instance of the router is given an IP address. It’s when I try to add the extra routes that I get zero IP addresses handed out. I’m more than happy to post a transcript of my terminal session…

6

Here’s an example of one of my attempts at this. Excuse some of the typos in the terminal session! In this case, I tried to add a global 0.0.0.0/0 route to the 172.27.10.11 address.

[admin@MikroTik] /zerotier> controller/add name=ZT-NA9D instance=zt1 ip-range=172.27.10.10-172.17.10.20 private=yes routes=172.27.10.0/24,0.0.0.0/0@172.27.10.11 
[admin@MikroTik] /zerotier> controller/print
Flags: I - INACTIVE
Columns: INSTANCE, NAME, PRIVATE
#   INSTANCE  NAME     PRIVATE
0 I zt1       ZT-NA9D  yes    
[admin@MikroTik] /zerotier> controller/print
Flags: I - INACTIVE
Columns: INSTANCE, NAME, PRIVATE
#   INSTANCE  NAME     PRIVATE
0 I zt1       ZT-NA9D  yes    
[admin@MikroTik] /zerotier> controller/print
Columns: INSTANCE, NAME, NETWORK, PRIVATE
# INSTANCE  NAME     NETWORK           PRIVATE
0 zt1       ZT-NA9D  5fb30d356dc2cd47  yes    
[admin@MikroTik] /zerotier> interface/add network=5fb30d356dc2cd47 name=NA9DNET instance=zt1
[admin@MikroTik] /zerotier> print interval=1
Flags: R - ONLINE; F - TCP-FALLBACK
Columns: NAME, PORT
#    NAME  PORT
;;; ZeroTier Central controller - https://my.zerotier.com/
0 RF zt1   9993

[admin@MikroTik] /zerotier> interface/print interval=1
Columns: NAME, MAC-ADDRESS, NETWORK, STATUS
# NAME     MAC-ADDRESS        NETWORK           STATUS       
0 NA9DNET  46:92:71:60:00:60  5fb30d356dc2cd47  ACCESS_DENIED

[admin@MikroTik] /zerotier> controller/member/print
Columns: NETWORK, ZT-ADDRESS
#  NETWORK  ZT-ADDRESS
0  ZT-NA9D  5fb30d356d
[admin@MikroTik] /zerotier> controller/member/set 0 authroized=yes
expected end of command (line 1 column 25)
[admin@MikroTik] /zerotier> controller/member/print               
Columns: NETWORK, ZT-ADDRESS
#  NETWORK  ZT-ADDRESS
0  ZT-NA9D  5fb30d356d
[admin@MikroTik] /zerotier> controller/member/set 0 authorized=yes
[admin@MikroTik] /zerotier> /ip/address/print where interface~"Zero" 

[admin@MikroTik] /zerotier> /ip/address/print where interface~"Zero"

[admin@MikroTik] /zerotier> /ip/address/print where interface~"Zero"
7

I demand that you smash one of our fingers with a claw hammer mob-style.

There’s your answer: ACCESS_DENIED clients, are, well, denied access. You have to (on the controller configuration) print out the members, and set the client to be allowed to join.

8

No. I did. That’s one of the steps. When you first add the network you are denied. But then you add it:

Columns: NAME, MAC-ADDRESS, NETWORK, STATUS
# NAME     MAC-ADDRESS        NETWORK           STATUS       
0 NA9DNET  46:92:71:60:00:60  5fb30d356dc2cd47  ACCESS_DENIED

So yes. Here is it access denied. Now you go and list the members under the controller and then authorize the correct one..


[admin@MikroTik] /zerotier> controller/member/print               
Columns: NETWORK, ZT-ADDRESS
#  NETWORK  ZT-ADDRESS
0  ZT-NA9D  5fb30d356d
[admin@MikroTik] /zerotier> controller/member/set 0 authorized=yes

There. Now it is authorized. So I should get an IP address, but I don’t…


[admin@MikroTik] /zerotier> /ip/address/print where interface~"Zero"

I know this was going to trip you up. :wink:

9

After authorization, you should see an ip address being assigned in the controller/member area. I don’t know how frequently the client tries to reconnect; maybe you should try disabling/enabling the zt interface.

EDIT:
And in case you’re adding this member as a gw of a route, you really would want to manually assign an ip address to it, similar to where you did authorized=yes.

Again, the documentation is at a minimum, but what is provided there is actually correct.

10

Well, that "client’ is the router itself. My point is that when the extra routes are added, an IP address is never assigned. I’m happy to try it all over again for the tenth time. The IP address is assigned rapidly when using the example setup. So I don’t think that it is taking a while.

11

LOL, so true. But an short example wouldn’t hurt… And the do check the BNF in docs exactly – include the [@Gw] – but what happens without a route destination is pretty unclear…

And they do seem to validate it, exactly, so you need a valid ip-prefix/ip6-prefix. But if you don’t include the @ route part, that is accepted although unclear what happens.

What I love is they do resolve a classFULL shortcut as prefix, just like everywhere else in CLI, but boy some typo in an classless IP most folks won’t expect it:

/zerotier/controller
# as array
set [find] routes=("2.0/24@10.1.1.1","17.0/8@10.1.1.1")
# or as string
set [find] routes="2.0/8@10.1.1.1,17.0/8@10.1.1.1"
# both forms work - so routes US military and Apple IPs to a ZT member at 10.1.1.1
# & resolve the 2.0 into 2.0.0.0, so that consistent at least (although still questionable in my book)
:put [get [find] routes]
2.0.0.0/8@10.1.1.1;17.0.0.0/8@10.1.1.1
12

Oh- so you can come back after the fact and add the routes? That was what I wasn’t sure how to do.

So you would use

zerotier/controller/set [find] routes = "10.0.0.0/8@172.27.10.2"

Like that?

I’m a little confused by the “[find]” in the command. What is that for?

13

For me everything works just fine.

Commands:

/zerotier/controller/set 0 private=yes ip-range=172.30.30.100-172.30.30.100 routes=172.30.30.0/24,0.0.0.0/0@172.30.30.1
/zerotier/controller/member/set 0 authorized=yes ip-address=172.30.30.1

Afterwards:

> /zerotier/controller/member/print
Flags: A - AUTHORIZED
Columns: NETWORK, ZT-ADDRESS, IP-ADDRESS
#    NETWORK  ZT-ADDRESS  IP-ADDRESS 
0 A  tst      23221c3302  172.30.30.1

> /ip/address/print
Flags: I - INVALID; D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
#    ADDRESS            NETWORK        INTERFACE            
[...]
8  D 172.30.30.1/24     172.30.30.0    zerotier1

> /ip/route/print
Flags: D - DYNAMIC; X - DISABLED, I - INACTIVE, A - ACTIVE; c - CONNECT, s - STATIC, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS       GATEWAY                DISTANCE
[...]
  DAc 172.30.30.0/24    zerotier1                     0
[...]

I didn’t set “allow default” in the zerotier interface config, that’s why the default route is not added.

This all seems quite fine to me.

14

Hmm.

OK. Interesting…I’ll try it again…

Could it be that since you are specifying a specific gateway IP address in the ZeroTier subnet that the router doesn’t do a DHCP assignment but instead is expecting a fixed IP?

15

Wait a minute…

Looking at your entries:

/zerotier/controller/set 0 private=yes ip-range=172.30.30.100-172.30.30.100 routes=172.30.30.0/24,0.0.0.0/0@172.30.30.1
/zerotier/controller/member/set 0 authorized=yes ip-address=172.30.30.1

You set the IP address of the router OUTSIDE of the IP address range specified for the ZT network. Is that allowed?

16

DHCP in never used in ZT. It is explicitly filtered in all ZT networks. (It’s part of the “source” distribution for all zt clients. The desginers thought that it would be a security threat when joining networks run by people you don’t really trust. Many people use ZT to for example run Minecraft servers for ad-hoc groups of people.)

All addresses are assigned by the controller. If unspecified, one from the ip-range is assigned.

EDIT:
Yes, you can give out any address you want. The ip-range is for automatic assignment.

17

HEY! That worked!!! WOOT!

Lurker888 thank you once again. And thanks to the others here as well. Looks like the key is that if you specify additional routes with a gateway address, you MUST assign that gateway address to whatever member is functioning as the gateway - it takes a static assignment. It makes sense too. The docs just had no example about that so it wasn’t clear…

18

Glad it works. :slight_smile:

But your explanation is off. (Using the same config.) If I delete the member, rejoin, and don’t give it an explicit IP address assignment, I get:

 > /zerotier/controller/member/print
Flags: A - AUTHORIZED
Columns: NETWORK, ZT-ADDRESS, IP-ADDRESS
#    NETWORK  ZT-ADDRESS  IP-ADDRESS   
0 A  tst      23221c3302  172.30.30.100

/ip/address/print
Flags: I - INVALID; D - DYNAMIC
Columns: ADDRESS, NETWORK, INTERFACE
#    ADDRESS            NETWORK        INTERFACE                           
[...]
8  D 172.30.30.100/24   172.30.30.0    zerotier1

/ip/route/print
Flags: D - DYNAMIC; X - DISABLED, I - INACTIVE, A - ACTIVE; c - CONNECT, s - STATIC, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS       GATEWAY                DISTANCE
[...]
  DAc 172.30.30.0/24    zerotier1                     0
[...]

So it’s assigned the first address in the automatically assigned range.

Of course with this setup, the default route - if it was accepted - couldn’t be resolved. That’s why it’s natural to give a fixed IP to something used as a gateway… But it still works as it should.

Something else was off. We may never know exactly.

19

OK. Strange. I’ll try creating another network and seeing if I can duplicate my problem…

20

Just an unrelated note: If you have a routable WAN address (even if dynamic), you should open port 9993/udp for the ZeroTier service. This enables other clients on the network (even if they are behind NAT) to make a direct connection and not have to use relays. You should especially open this port if your device functions as a controller for a network.