SDWAN using Zerotier [SOLVED]

mafiosa · Sat Jan 18, 2020 9:35 pm

Please implement some sort of SDWAN stuff. Zerotier can be a good start. Edge routers and openwrt have support for Zerotier!

StubArea51 · Sun Jan 19, 2020 11:27 pm

I would also love to see ZeroTier implemented.

We use it for remote access to DCs as well as an encrypted transport path between DCs. It's one of my favorite SDWAN implementations.

andrewe02000 · Wed Jan 22, 2020 9:32 pm

I would also get a lot of use from this. Please Implement.

syadnom · Sat Feb 15, 2020 1:14 am

+1 for zerotier.

edgerouters do NOT have support for zerotier. Updates kill zerotier installed by hand. Need first-class support. I bet the zerotier guys would help with creating a mikrotik package for user install also.

rogierb · Sun Mar 01, 2020 10:48 pm

Here a definit +1 for ZeroTier support as well!!

Michaelcrapse · Tue Mar 10, 2020 9:11 pm

Zerotier is definitely a wanted feature

metricmoose · Thu Mar 12, 2020 3:44 am

Zerotier would be a great extra. Right now there's no good way to do a mesh VPN on Mikrotik and it's the reason why we had to deploy Cisco for part of our network.

next111 · Tue Mar 17, 2020 2:21 pm

+1
For zerotier

N8jar · Thu Apr 23, 2020 2:19 pm

+1 ZeroTier Integration

linkwave · Mon Apr 27, 2020 12:59 pm

+1 for ZeroTier too

nicob · Mon Apr 27, 2020 2:44 pm

I would prefer Tailscale (wireguard SDWAN) over ZeroTier

mozerd · Mon Apr 27, 2020 4:17 pm

I would prefer Tailscale (wireguard SDWAN) over ZeroTier

100% correct

and 100% faster ..... KISS

https://tailscale.com/

syadnom · Mon Apr 27, 2020 4:32 pm

100% zero chance whatsoever you'll see a commercial wrapper (tailscale) on wireguard on mikrotik.

MikesellT · Thu May 28, 2020 7:44 pm

Yes, please add support for Zerotier. I love Mikrotiks, but they are seriously lacking in some kind of SDWAN solution. Zerotier would be a very cheap and easy way to set this up. The folks at Mikrotik haven't seemed very interested in implementing something to address SDWAN. Mikrotik routers are powerful, and adding Zerotier support would make them even more amazing.

jpostel000 · Wed Jun 10, 2020 6:20 am

zerotier+mikrotik would be marvellous

next111 · Sun Jun 14, 2020 10:04 pm

+1 zerotier

chaigeo · Mon Sep 07, 2020 9:19 pm

+1 zerotier

Gnubyte · Tue Sep 08, 2020 1:04 am

I like Mikrotik.
I dont like SDWAN.

FiiMitch · Tue Sep 08, 2020 2:25 am

I like Mikrotik.
I dont like SDWAN.

You're on the right forum.

And the wrong thread.

+1 for zerotier

mducharme · Wed Sep 09, 2020 5:58 am

+1 for ZeroTier, if possible

troffasky · Wed Sep 23, 2020 6:44 pm

Yes, please add support for Zerotier. I love Mikrotiks, but they are seriously lacking in some kind of SDWAN solution. Zerotier would be a very cheap and easy way to set this up.

How do you know it would be cheap? I don't see how they could comply with this license if they included it in ROS:

https://github.com/zerotier/ZeroTierOne ... ICENSE.txt

so they would need to pursue an "alternative commercial license", which would probably end up being paid for per-router [which I don't have a problem with].

Think7 · Tue Sep 29, 2020 4:02 pm

+11111!!!!!!

Please please please please please please please add native support for ZeroTier. Very happy to pay for it.

Think7 · Fri Nov 13, 2020 3:09 pm

mmlea · Fri Nov 13, 2020 3:21 pm

+1 please implement it

gjjennings · Tue Feb 09, 2021 8:56 am

another +1 for zerotier, could we donate money to encourage its development?

krafg · Fri Feb 19, 2021 9:25 pm

+1 for ZeroTier, it would be interesting. For now I connect to a 24/7 power on Windows machine to get access.

Regards.

rass121 · Thu Mar 25, 2021 2:07 am

+1
Has this been implement or an equivalent SDwan solution for mikrotik?

krafg · Sun Mar 28, 2021 5:31 pm

Not yet, sadly. The best of ZeroTier is that works including on mobile connections! Is really perfect. The bad news is that they on last year said that they are not sure if ZeroTier would works on all MikroTik devices. I not know why, but is not too much encouraging.

We hope that some day it can be implemented, so, OpenWRT and OPNsense are ready and also DD-WRT I believe.

I tried all VPN's that MikroTik offers but nothing works with my mobile connection.

Regards.

mgiammarco · Tue Apr 20, 2021 6:55 pm

I like also very much the idea to see Zerotier in Mikrotik. But I ask: what is the showstopper? Why Mikrotik does not reply to us? I have asked in Zerotier forum and I have read that Zerotier developers have a great interest in putting Zerotier in Mikrotik and they want to do themselves but they need API from Mikrotik. It seems to me that the best thing is that Mikrotik calls Zerotier or viceversa and they do an agreement.
Otherwise we are only dreaming.

aglabs · Thu Apr 22, 2021 5:52 am

I like also very much the idea to see Zerotier in Mikrotik. But I ask: what is the showstopper? Why Mikrotik does not reply to us? I have asked in Zerotier forum and I have read that Zerotier developers have a great interest in putting Zerotier in Mikrotik and they want to do themselves but they need API from Mikrotik. It seems to me that the best thing is that Mikrotik calls Zerotier or viceversa and they do an agreement.
Otherwise we are only dreaming.

The api exists today. All of my wireguard config on mikrotik is done via the rest api introduced in v7. There shouldn't be any reason zerotier couldn't leverage it.

parham · Thu Apr 22, 2021 7:54 pm

Definitely Yes for ZeroTier. please added to V7

bradknowles · Tue May 25, 2021 9:53 am

I would also be happy for a ZeroTier or Tailscale implementation.

Count me in!

warwickchapman · Tue Jun 22, 2021 7:13 pm

Danidak · Thu Jun 24, 2021 8:35 pm

More +1 for ZeroTier.
Every time I have to add an OpenWRT biside to RouterOS (it starts to get me bored).
More than once time I wondered if I could do without the Routerboard....
Before you start losing customers, please support SDWAN aka ZeroTier feature,
...again please...

kiler129 · Sun Jul 04, 2021 11:32 pm

I don't want to crush your dreams guys but I don't think ZT will ever be a thing on MT due to licensing ZT uses now. They used to use GPLv3 but since they changed their licensing model to.... freemium(?):

ZeroTier’s software kit is licensed under the ZeroTier BSL, which allows source code access and free use for all with the exception of hosting a network controller for commercial purposes or embedding the ZeroTier source code in a commercial application. You can self-host ZeroTier controllers and nodes for free if you use it for non-commercial purposes. Please contact us to learn more.

via https://www.zerotier.com/pricing/

Also, maybe I'm not up to the speed but what problem ZT solves which WG+OSPF doesn't?

mducharme · Mon Jul 05, 2021 12:33 am

Also, maybe I'm not up to the speed but what problem ZT solves which WG+OSPF doesn't?

Zerotier builds a full mesh and uses the lowest latency path between any two nodes. If there is any loss (indicating congestion) it shifts that traffic to a backup path automatically. You can build a full mesh with wireguard and OSPF but OSPF does not adjust routing cost based on latency between nodes and will not automatically redirect traffic the moment it detects congestion.

syadnom · Mon Jul 05, 2021 1:33 am

Also, maybe I'm not up to the speed but what problem ZT solves which WG+OSPF doesn't?
Zerotier builds a full mesh and uses the lowest latency path between any two nodes. If there is any loss (indicating congestion) it shifts that traffic to a backup path automatically. You can build a full mesh with wireguard and OSPF but OSPF does not adjust routing cost based on latency between nodes and will not automatically redirect traffic the moment it detects congestion.

This statement as-is is false. Zerotier operates as a star topology and then attempts to build p2p adjacencies and prioritises those direct paths when possible, otherwise it falls back to the relayed ie star topology. It doesn't account for latency at all, simply treats the shortest path between zt nodes as optimal. It does connection monitoring via keep alives just like OSPF does. If you enable multipath on the client you do get some connection quality measurements to determine which uplink to use but that's only if you have multiple WAN links for zerotier to use and those tests are only between that zt node and the remote node to see which of the two interfaces look better. Otherwise, it does nothing except check if the direct path is up. Zerotier doesn't route around slow or lossy nodes until they fail and then it falls back to the star topology ie 'relayed'.

A wireguard+ospf or wireguard+ebgp type setup will work and feel very much like zerotier, advantage to zerotier for simplicity and ability to punch through most NAT but wireguard is a lighter tunnel so for performance, slight advantage to wg.

A very direct comparison to make here is to look at tailscale, which is wireguard+their take on cisco dmvpn type ad-hoc mesh topology. That's a 1:1 comparison in end user experience between wireguard+extra kit vs zerotier.

The only mature meshing tech available to shove into routeros7 that actually takes interface quality properly into consideration is batman-adv which could easily be added to routeros because it's all open source and already in the kernel. That can operate over any interface that looks like ethernet, so p2p wireguard links into a batman-adv interface WOULD take connection latency, packet loss, jitter, and so on into account. batman-adv doesn't 'fail' a link just reduces it's preference down to near-nothing so if it's the last operational link it will still operate.

mducharme · Mon Jul 05, 2021 2:27 am

Thanks for the clarification/correction. I set up Zerotier once, and read about the multipath support but must have misunderstood.

caspat · Tue Aug 17, 2021 1:39 am

+1 for Zerotier!

mada3k · Tue Aug 17, 2021 6:15 pm

Would it not make more sense to implement DMVPN?

syadnom · Tue Aug 17, 2021 6:29 pm

Would it not make more sense to implement DMVPN?

DMVPN is more a suite of tools requiring adding a number of components to routeros, plus DMVPN really struggles with NAT. zerotier is a single daemon and all of the smarts are external and it finds ways to get through NAT much better. Much simpler to implement.

njalmeister · Tue Aug 24, 2021 5:31 pm

+1 for ZeroTier

oreggin · Mon Aug 30, 2021 1:05 pm

Hi!

I working in ISP sector and we operating with low-mid budget so we can't buy high-end SD-WAN solutions yet for 10X-20X the price. So we need to find the optimal solution at all times for the following:

using various underlaying network, PPPoE, DOCSIS, metro ethernet, DF etc. with optional IPSec encryption where it is possible

the underlay circuits has random global addresses and some of it has (CG)NAT/FW in the path which may blocks GRE and/or ESP protos

providing VPNV4, VPNV6, L2VPN over underlaying network

At this moment we testing it with MTik MPLS over L2TP/PPPoE. In the test setup there are 1-4 geographically redundant LNS and clients in HUB & Spoke topology. As MTik PPP stack can handle MPLS we don't need any other magic just pure MPLS. Moreover we can use MLPPP so we can handle jumbo frames in L2VPN if we would so.
In the near past I tested it with an RB4011 and I get 700Mbps with an L2VPN over MPLS over L2TP (MLPPP). I measure it with single UDP stream generated with iperf3. One CPU core was 100% on the RB4011. Maybe CCR5009 would can 1Gbps. We are before a multilink test where L2TP client has multiple link to more LNSes and lets see how ECMP and Multi-core PPP do its job.

oreggin

syadnom · Mon Aug 30, 2021 5:31 pm

Hi!

I working in ISP sector and we operating with low-mid budget so we can't buy high-end SD-WAN solutions yet for 10X-20X the price. So we need to find the optimal solution at all times for the following:
using various underlaying network, PPPoE, DOCSIS, metro ethernet, DF etc. with optional IPSec encryption where it is possible

the underlay circuits has random global addresses and some of it has (CG)NAT/FW in the path which may blocks GRE and/or ESP protos

providing VPNV4, VPNV6, L2VPN over underlaying network
At this moment we testing it with MTik MPLS over L2TP/PPPoE. In the test setup there are 1-4 geographically redundant LNS and clients in HUB & Spoke topology. As MTik PPP stack can handle MPLS we don't need any other magic just pure MPLS. Moreover we can use MLPPP so we can handle jumbo frames in L2VPN if we would so.
In the near past I tested it with an RB4011 and I get 700Mbps with an L2VPN over MPLS over L2TP (MLPPP). I measure it with single UDP stream generated with iperf3. One CPU core was 100% on the RB4011. Maybe CCR5009 would can 1Gbps. We are before a multilink test where L2TP client has multiple link to more LNSes and lets see how ECMP and Multi-core PPP do its job.

oreggin

This whole setup is really nothing like zerotier. you have tunnels to concentrators so it's a multi-hub & spoke model. All data must flow through a hub.

Zerotier is a peer to peer model so effectively creates a full mesh with vastly better performance and reduced costs because the 4 hub sites don't even exist, no 'big' internet pipes to buy etc. It requires a single line of configuration to bring a site on-net and has strong access controls in the controller with policy routing etc. Vastly simpler, easier to configure and admin, and with essentially zero moving parts. Also punches through NAT in ways that L2TP tunnels just don't so even makes deployment simpler.

Right now I just hang a raspberry pi 4 off a port and run zerotier there, binding zt to a VLAN. Works exceptionally well, but it's an additional piece of hardware I'd rather not have.

oreggin · Mon Aug 30, 2021 7:21 pm

Yeah, ZT build up tunnels between Spokes, IF Spokes can talk to each other, but 1: this not alway possible, 2: we don't really need horizontal traffic engineering because of a lot of reason. However if we need horizontal traffic (L2VPN for example) then it goes through regional aggregation, and never on core, as on ZT initial. I mentioned only a 4 LNS testbed with 4 L2TP tunnel to test ECMP-SMP capabilities of ROS, it is not a real scenario. There is no big fat pipes (except on cores), and I didn't mentioned them, I don't know where did you get fat pipes.

Also punches through NAT in ways that L2TP tunnels just don't

I don't understand this. How can an UDP based, VXLAN-like (modified?) tunnel goes through better over NAT than also UDP based L2TPv2 or UDP based IPSec? Maybe you overmistified the capabilities of ZT?
So, ZT is not made for us

Summarize, it seems to me ZT is not an ISP level SD-WAN solution, but it is good SD-WAN-like solution for whom has a lot of horizontal traffic, and less vertical, and the limitation is not a point of pain.

syadnom · Mon Aug 30, 2021 7:48 pm

I'm not sure what you are arguing in the zerotier feature request...

zerotier uses various techniques to link up through NAT. Unlike L2TP which can only do a basic 1:1 UDP NAT session, zerotier will try everything from upnp, to opening multiple ports through multiple UDP sessions to make ports available for peers. UDP is great for simplicity and speed, but it's a 1:1 port to host mapping so L2TP ends up being a host-to-server only model as it only opens a single port per session. Zerotier opens more ports and dynamically maps them to remote peers that should have data move between them. The way UDP works allows that peer to send data at the opened port and this generally works unless the firewall is aggressive. This is something L2TP/ipsec cannot do, there is no mechanism. As a fallback it can relay.

From an end user perspective it works like a DMVPN by dynamically peering when needed, though under the hood it's nothing like that. with L2TP tunnels you have to build many tunnels and you still don't have any model to handle p2p connectivity except for building a tunnel between the endpoints. It's not always possible for spokes to link up, but zerotier is FAR better at this than L2TP because zerotier opens UDP sessions with whatever firewall/NAT it's going through and then tells the remote peer that port.

It handles MTU, fragmentation, encryption, and session establishment and is quite happy operating in a more hub-and-spoke use case if you like and you can even enforce that on a MAC or zt id basis. Or you can keep it full mesh if you like. You can allow your datacenters to communicate out to everyone, everyone to communicate to the datacenters, but no one communicate with each other... if you like. or only allow snmp traffic. It's ultimate flexability. Heck, you can even use zerotier as transit and run BGP or OSPF across it if you really want.

In your use case where you mostly just need data from sites to datacenters and not much p2p, zerotier happily does this as well with superior NAT traversal. And zerotier can go directly on the servers eliminating the need for additional VPN concentrator hardware or upgrading of a router to handle ipsec encryption on your L2TP tunnels.

Zerotier is also multi-interface, so you can do dual-wan and zerotier will load balance / failover on those links. any number of links.

Also, from a security perspective, zerotier routing/firewall rules are applied on both sides. So if you allow mobile devices access to port 3389 on the server, that rule is applied on the mobile device AND the server and you can do it at a mac, zerotier id, or ip level.

My point is that zerotier works better for your more hub-n-spoke network needs than L2TP/ipsec does, and it works for full mesh better.

If we can get zerotier into routeros, and we already have wireguard coming along in v7, l2tp will become a relic.

parham · Tue Aug 31, 2021 6:45 pm

MT just released ZeroTier to Arm base MT.

viewtopic.php?f=1&t=178063

mafiosa · Wed Sep 01, 2021 7:25 am

This feature is added to v7.1 RC2 onwards.

mafiosa · Wed Sep 01, 2021 7:25 am

This feature is added to v7.1 RC2 onwards.

ghostinthenet · Thu Sep 09, 2021 9:09 pm

Very cool that it's been added for the ARM architecture, but I'm wondering if there are any plans to go beyond this. Support on x86/CHR would be ideal for the overall management of routers in the field.

slvfibergarrett · Fri Mar 18, 2022 9:01 am

Has anyone been able to get multi-wan working with ZT on ROS? I have 3x WAN links with appropriate routing tables and policy routes for each interface. My other ZT boxes see several paths, but it never seems to aggregate and pulling the active WAN link down causes about five seconds of packet loss. With ZT on generic Linux you can define custom policies for WAN bonding in local.conf and it works decently well. There doesn't seem to be any way to edit local.conf for ZT on ROS.

Amm0 · Fri Mar 18, 2022 4:01 pm

Has anyone been able to get multi-wan working with ZT on ROS? I have 3x WAN links with appropriate routing tables and policy routes for each interface. My other ZT boxes see several paths, but it never seems to aggregate and pulling the active WAN link down causes about five seconds of packet loss. With ZT on generic Linux you can define custom policies for WAN bonding in local.conf and it works decently well. There doesn't seem to be any way to edit local.conf for ZT on ROS.

AFAIK you cannot bond ("Multipath") with Mikrotik's current support. I personally been waiting for ZT multipath, very curious how well it bond multiple LTE interfaces.

There is some hope, in the /zerotier/peers there is an attribute "bonded" (and flag "B - bonded"), so maybe it's coming at some point... Right now, all peers show "bonded=no" however. RouterOS doesn't directly expose the the local.conf & you can't add the needed bonding configuration. Hopefully it just some ROS UI/CLI away around the local.conf that just missing at this point, since clearly part of their ZeroTier implementation seems aware of the possibility of multipath/bonding:

[user@device] > /zerotier/peer/print detail 
Flags: B - bonded 
 0   instance=zt1 zt-address="62f865ae71" bonded=no latency=188ms role="PLANET" 
     path=active,preferred,2001:xxxx:xxxx:2::2/9993,recvd:6s916ms,active,
     50.7.252.138/9993,recvd:16s734ms,sent:12s157ms

Larsa · Fri Mar 18, 2022 5:39 pm

AFAIK you cannot bond ("Multipath") with Mikrotik's current support. I personally been waiting for ZT multipath, very curious how well it bond multiple LTE interfaces.

Multipath has been a part of zt since v1.6 (2020-11-24) but was actually announced already in v1.4 (2019-07-29) although that version was more of a beta. The latest update was in v1.8.5 (2022-02-22).

I wasn't able to find anything mentioned in the release notes why they capped multipath/bonding. Any idea of the reason behind the restrictions?

Amm0 · Fri Mar 18, 2022 6:02 pm

AFAIK you cannot bond ("Multipath") with Mikrotik's current support. I personally been waiting for ZT multipath, very curious how well it bond multiple LTE interfaces.

Multipath has been a part of zt since v1.6 (2020-11-24) but was actually announced already in v1.4 (2019-07-29) although that version was more of a beta. The latest update was in v1.8.5 (2022-02-22).

I wasn't able to find anything mentioned in the release notes why they capped multipath/bonding. Any idea of the reason behind the restrictions?

Hard to know, maybe Mikrotik can answer...

The Mikrotik ZT "instance" allows you pick the interface (or I guess a few interfaces) it could use. But there is no RouterOS version of ZeroTier's "defaultBondingPolicy" that be set to the bonding mode.

Larsa · Fri Mar 18, 2022 7:01 pm

The Mikrotik ZT "instance" allows you pick the interface (or I guess a few interfaces) it could use. But there is no RouterOS version of ZeroTier's "defaultBondingPolicy" that be set to the bonding mode.

It seems that some of the zt multipath/bonding policies still have problems and will be fixed in the next release (unclear which ones though). Mikrotik could easily have isolated the faulting bonding policy/policies but I guess they prefer to wait for everything to be fixed before they open access to the entire bonding functionality. One possible reason for the limitation is perhaps that the problems had a major impact on ros