DMVPN is more a suite of tools requiring adding a number of components to routeros, plus DMVPN really struggles with NAT. zerotier is a single daemon and all of the smarts are external and it finds ways to get through NAT much better. Much simpler to implement.
+1 for ZeroTier 
Hi!
I working in ISP sector and we operating with low-mid budget so we can’t buy high-end SD-WAN solutions yet for 10X-20X the price. So we need to find the optimal solution at all times for the following:
- using various underlaying network, PPPoE, DOCSIS, metro ethernet, DF etc. with optional IPSec encryption where it is possible
- the underlay circuits has random global addresses and some of it has (CG)NAT/FW in the path which may blocks GRE and/or ESP protos
- providing VPNV4, VPNV6, L2VPN over underlaying network
At this moment we testing it with MTik MPLS over L2TP/PPPoE. In the test setup there are 1-4 geographically redundant LNS and clients in HUB & Spoke topology. As MTik PPP stack can handle MPLS we don’t need any other magic just pure MPLS. Moreover we can use MLPPP so we can handle jumbo frames in L2VPN if we would so.
In the near past I tested it with an RB4011 and I get 700Mbps with an L2VPN over MPLS over L2TP (MLPPP). I measure it with single UDP stream generated with iperf3. One CPU core was 100% on the RB4011. Maybe CCR5009 would can 1Gbps. We are before a multilink test where L2TP client has multiple link to more LNSes and lets see how ECMP and Multi-core PPP do its job.
oreggin
This whole setup is really nothing like zerotier. you have tunnels to concentrators so it’s a multi-hub & spoke model. All data must flow through a hub.
Zerotier is a peer to peer model so effectively creates a full mesh with vastly better performance and reduced costs because the 4 hub sites don’t even exist, no ‘big’ internet pipes to buy etc. It requires a single line of configuration to bring a site on-net and has strong access controls in the controller with policy routing etc. Vastly simpler, easier to configure and admin, and with essentially zero moving parts. Also punches through NAT in ways that L2TP tunnels just don’t so even makes deployment simpler.
Right now I just hang a raspberry pi 4 off a port and run zerotier there, binding zt to a VLAN. Works exceptionally well, but it’s an additional piece of hardware I’d rather not have.
Yeah, ZT build up tunnels between Spokes, IF Spokes can talk to each other, but 1: this not alway possible, 2: we don’t really need horizontal traffic engineering because of a lot of reason. However if we need horizontal traffic (L2VPN for example) then it goes through regional aggregation, and never on core, as on ZT initial. I mentioned only a 4 LNS testbed with 4 L2TP tunnel to test ECMP-SMP capabilities of ROS, it is not a real scenario. There is no big fat pipes (except on cores), and I didn’t mentioned them, I don’t know where did you get fat pipes.
Also punches through NAT in ways that L2TP tunnels just don’t
I don’t understand this. How can an UDP based, VXLAN-like (modified?) tunnel goes through better over NAT than also UDP based L2TPv2 or UDP based IPSec? Maybe you overmistified the capabilities of ZT?
So, ZT is not made for us 
Summarize, it seems to me ZT is not an ISP level SD-WAN solution, but it is good SD-WAN-like solution for whom has a lot of horizontal traffic, and less vertical, and the limitation is not a point of pain.
I’m not sure what you are arguing in the zerotier feature request…
zerotier uses various techniques to link up through NAT. Unlike L2TP which can only do a basic 1:1 UDP NAT session, zerotier will try everything from upnp, to opening multiple ports through multiple UDP sessions to make ports available for peers. UDP is great for simplicity and speed, but it’s a 1:1 port to host mapping so L2TP ends up being a host-to-server only model as it only opens a single port per session. Zerotier opens more ports and dynamically maps them to remote peers that should have data move between them. The way UDP works allows that peer to send data at the opened port and this generally works unless the firewall is aggressive. This is something L2TP/ipsec cannot do, there is no mechanism. As a fallback it can relay.
From an end user perspective it works like a DMVPN by dynamically peering when needed, though under the hood it’s nothing like that. with L2TP tunnels you have to build many tunnels and you still don’t have any model to handle p2p connectivity except for building a tunnel between the endpoints. It’s not always possible for spokes to link up, but zerotier is FAR better at this than L2TP because zerotier opens UDP sessions with whatever firewall/NAT it’s going through and then tells the remote peer that port.
It handles MTU, fragmentation, encryption, and session establishment and is quite happy operating in a more hub-and-spoke use case if you like and you can even enforce that on a MAC or zt id basis. Or you can keep it full mesh if you like. You can allow your datacenters to communicate out to everyone, everyone to communicate to the datacenters, but no one communicate with each other… if you like. or only allow snmp traffic. It’s ultimate flexability. Heck, you can even use zerotier as transit and run BGP or OSPF across it if you really want.
In your use case where you mostly just need data from sites to datacenters and not much p2p, zerotier happily does this as well with superior NAT traversal. And zerotier can go directly on the servers eliminating the need for additional VPN concentrator hardware or upgrading of a router to handle ipsec encryption on your L2TP tunnels.
Zerotier is also multi-interface, so you can do dual-wan and zerotier will load balance / failover on those links. any number of links.
Also, from a security perspective, zerotier routing/firewall rules are applied on both sides. So if you allow mobile devices access to port 3389 on the server, that rule is applied on the mobile device AND the server and you can do it at a mac, zerotier id, or ip level.
My point is that zerotier works better for your more hub-n-spoke network needs than L2TP/ipsec does, and it works for full mesh better.
If we can get zerotier into routeros, and we already have wireguard coming along in v7, l2tp will become a relic.
MT just released ZeroTier to Arm base MT.
http://forum.mikrotik.com/t/zerotier-added-to-routeros-v7-1rc2/151475/1
This feature is added to v7.1 RC2 onwards.
This feature is added to v7.1 RC2 onwards.
Very cool that it’s been added for the ARM architecture, but I’m wondering if there are any plans to go beyond this. Support on x86/CHR would be ideal for the overall management of routers in the field.
Has anyone been able to get multi-wan working with ZT on ROS? I have 3x WAN links with appropriate routing tables and policy routes for each interface. My other ZT boxes see several paths, but it never seems to aggregate and pulling the active WAN link down causes about five seconds of packet loss. With ZT on generic Linux you can define custom policies for WAN bonding in local.conf and it works decently well. There doesn’t seem to be any way to edit local.conf for ZT on ROS.
AFAIK you cannot bond (“Multipath”) with Mikrotik’s current support. I personally been waiting for ZT multipath, very curious how well it bond multiple LTE interfaces.
There is some hope, in the /zerotier/peers there is an attribute “bonded” (and flag “B - bonded”), so maybe it’s coming at some point… Right now, all peers show “bonded=no” however. RouterOS doesn’t directly expose the the local.conf & you can’t add the needed bonding configuration. Hopefully it just some ROS UI/CLI away around the local.conf that just missing at this point, since clearly part of their ZeroTier implementation seems aware of the possibility of multipath/bonding:
[user@device] > /zerotier/peer/print detail
Flags: B - bonded
0 instance=zt1 zt-address="62f865ae71" bonded=no latency=188ms role="PLANET"
path=active,preferred,2001:xxxx:xxxx:2::2/9993,recvd:6s916ms,active,
50.7.252.138/9993,recvd:16s734ms,sent:12s157ms
Multipath has been a part of zt since v1.6 (2020-11-24) but was actually announced already in v1.4 (2019-07-29) although that version was more of a beta. The latest update was in v1.8.5 (2022-02-22).
I wasn’t able to find anything mentioned in the release notes why they capped multipath/bonding. Any idea of the reason behind the restrictions?
Hard to know, maybe Mikrotik can answer…
The Mikrotik ZT “instance” allows you pick the interface (or I guess a few interfaces) it could use. But there is no RouterOS version of ZeroTier’s “defaultBondingPolicy” that be set to the bonding mode.
It seems that some of the zt multipath/bonding policies still have problems and will be fixed in the next release (unclear which ones though). Mikrotik could easily have isolated the faulting bonding policy/policies but I guess they prefer to wait for everything to be fixed before they open access to the entire bonding functionality. One possible reason for the limitation is perhaps that the problems had a major impact on ros