I have the zt1 instance assigned to a loopback interface as the WAN interface IP is not a publicly routable address, the lo0 IP is. I’ve also tested an L2TP connection from that loopback and it works fine when the loopback is specified as the src-address.
I see the outbound connections on UDP port 9993 but I’m not seeing anything coming back in.. Is there an issue with ZeroTier and binding to a specific interface? I’m going to throw this in a LAB and see what the hell is going on or if I can reproduce.
The Datacenter we connect to claims nothing is being filtered, only unique thing with them is their handoff with us is unnumbered (/32) and not publicly routable but the /27 they’ve provided to us is public and routable. No NAT, no dynamic routing just a static default route, very basic setup. It all worked fine on the bench, only change was the WAN IP.. All I can think is there is something wrong with the interface binding and ZeroTier.
The datacenter handoff is a SM fiber from their Cisco to our RB5009 (ROS7.8) with a /32 (unnumbered) assigned.
/zerotier
set zt1 comment="ZeroTier Central controller - https://my.zerotier.com/" \
identity="my_random_identity" interfaces=lo0 name=\
zt1 port=9993
/zerotier interface
add allow-default=no allow-global=no allow-managed=no disable-running-check=no \
disabled=no instance=zt1 name=zerotier1 network=my_network_id
/zerotier> peer/pr
Columns: INSTANCE, ZT-ADDRESS, ROLE
# INSTANCE ZT-ADDRESS ROLE
0 zt1 62f865ae71 PLANET
1 zt1 778cde7190 PLANET
2 zt1 cafe04eba9 PLANET
3 zt1 cafe9efeb9 PLANET
/zerotier/interface> pr
Columns: NAME, MAC-ADDRESS, NETWORK, STATUS
# NAME MAC-ADDRESS NETWORK STATUS
0 zerotier1 06:16:AA:73:08:B7 **** REQUESTING_CONFIGURATION
Normally, you don’t need to specify any interface at all, unless you want to force the traffic in some way. But if you force zt to a specific interface that doesn’t have internet access or is filtered, it won’t work for obvious reasons. Btw, is the node approved by ZeroTIer Central?
The interface set on the zerotier instance (zt1) controls where it will look for paths. I suspect the problem is the lo0 isn’t getting NAT’ed when zerotier tries to communicate the my.zerotier.com controller.
Any reason you’re using a loopback interface, instead of a WAN interface? Otherwise you’ll need a your own controller available via the lo0.
Because the DC providing our WAN connection decided to make the /31 subnet between their router and ours inaccesabile from the outside world. I have a feeling this is causing the issue… unless I can bind the managment traffic and dns lookups to the LAN IP addresses this is going to be an uphill battle… that or I’ll have to do a whole lot of mangle / nat fuckery to get it to work.
Zerotier isn’t forming the adjacencies via the peer to peer network.. There are no paths shown in the /zerotier/peer/pr tables. I’d suspect it’s also not resolving DNS for my.zerotier.com because I can’t speficy the src-address.
It was part of the troubleshooting process.. was trying to use an IP not in use by a customer device.
But i just proved my theory.. SRC-NAT’d the traffic from the WAN IP to the loopback and zerotier connected right up… Now I’m going to see if I can get a routable subnet on the WAN side so I don’t have to screw with this goofy config.
I think it all related to the WAN IP not being internet accessible. If I could have binded the service to specific IP I don’t think this would have been an issue.
if your router is the default gateway and is working, configure zt to use any interface. Make sure outbound traffic is not filtered.
If you’re behind nat/cg-nat, expose (is dst-nat) the zt instance port to avoid possible forwarding though external servers. You can check the relay status using the “zerotier-cli peers” command on any node that has zerotier installed.
It was a NAT issue. WAN IP is not reachable via internet, had to src-nat it to one of the LAN ip’s that are publicly routed to get it to connect… All good!
I have the zt1 instance assigned to a loopback interface as the WAN interface IP is not a publicly routable address,
actually has been resolved by your self
see the outbound connections on UDP port 9993 but I’m not seeing anything coming back in.
it was rejected by your own router.
then… you have made it worked
It was a NAT issue. WAN IP is not reachable via internet, had to src-nat it to one of the LAN ip’s that are publicly routed to get it to connect… All good!
probably, for many of us, handed over a private non internet routable ip was weird. isp not necessarily doing nat for you. it is resource intensive for their router.
but, from the service provider perspective, that is how they reserve those limited ipv4.
no need to worry, those /27 for your lo0 - their bgp router knows how to reach those ip via that private wan ip interface. they injected a static route for your /27 on their wan side connected to your router.
just do on your side, some dst-nat to the lo0 public ip.
