Hi.
we are planning to setup hub and spoke network using Mikrotik and ZeroTier.
As a start there will be 500-2000 spokes. And in the next following years will growing up, total it will have up to 30K spokes.
For the beginning we are planning to use a pair of CCR2004 as hub (active & standby).
As i understand, ZeroTier is a full mesh SDWAN, we are worried about the resources once all peers are connected to the hub. Hence we need to have plan to scale it.
Anyone have experience using the similar Mikrotik hardware as ZeroTier hub ? how many spokes can be served by a single Mikrotik (CCR) without having any performance issue ?
thanks
ZeroTier is Wireguard wrapped up in a nice management package. How much processing power you need boils down to how many clients you plan to have connected to it, and how much traffic (packets per second more than bandwidth) you’re going to push.
If all you’re connecting is a bunch of smart devices sending telemetry data, you’ll probably be OK.
If it’s a bunch of people connecting their devices (or entire homes), you’re going to run out of CPU pretty quickly. 2116’s are going to have much more CPU, but even then you’ll probably need to bump up to beefy CPU’s on x86 or CHR, or spread the load out across several 2116’s. Also, your throughput could at times be limited by ZeroTier’s relaying mechanisms if it has a hard time connecting endpoints to each other due to multiple layers of NAT, etc.
ZeroTier is a full-mesh SD-WAN that automatically utilizes point-to-point connections when at least one of the parties has a public IP address or can perform NAT hole punching. Otherwise, it falls back on public ZeroTier relay servers. This is the same approach used by Tailscale and similar services.
However, in terms of throughput, performance, and configuration options, the ZeroTier package on Mikrotik RouterOS has not seen significant updates or improvements for quite some time. As a result, it struggles with performance due to the lack of hardware acceleration and features compared to running the more current ZeroTier version 1.14 on fully supported platforms like Linux or Windows.
Bottom line: MikroTik devices aren’t suitable as ZeroTier VPN concentrators for business or performance-critical applications.
It’s better to run ZeroTier natively on a more robust platform, like Linux or Windows, which offers better performance and configuration options. Starting in a hosted environment for a POC is probably your best bet.
Although have not specifically tried a CCR2004…but most of performance comes from the switch chip, not the CPU. And CPU is needed for ZT, or even WG AFAIK. As @larsa importantly notes RouterOS is both older version and lacks the full range of configuration options, that likely limit you. RouterOS does support ZT acting as a controller, which might help for a “less-meshed” scenerio - but again the CPU power still be a problem.
Further ZeroTier is not available for X86/CHR that get you more CPU.
I think some VM platform on X86 using “real” Linux with ZT on one instance, and CHR another instance to deal routing/BGP/etc/etc. might be a better approach at this scale.
it will be small traffic, not data intensive pull/ push from the spoke side.
Our plan initially at the spoke will use small ARM Mikrotik with ZT joining single network - /24 (to limit the peers).
while on the hub at start will use CCR & ZT configured with multiple networks, and it will spread across with multiple CCR once it growing.
well noted @Amm0 & @Larsa,
I guess now our option is either using X86 if still keep using ZeroTier or other “more mature” SDWAN solution.
The issue isn’t with ZeroTier itself but rather the MikroTik implementation which is flawed due to using an older version (v1.10.3) with various bugs and lacking the ability to configure standard ZeroTier features such as custom root servers, multi-path, trusted-path, allow DNS, etc.
ZeroTier can handle millions of nodes per network (and you can have unlimited networks) thanks to its 40-bit network ID system which allows for up to 1 trillion (2^40) possible network IDs. The real limitation usually comes down to the environment it’s running in like whether there might be a cluster of exit nodes to handle various services. We’ve been involved in deploying a project that now has over 200,000 connected devices on a single ZeroTier network.
If you’re only planning to run telemetry (aka IoT) data along with ZeroTier’s default settings, it might work. But even small amounts of data per connected device can lead to significant bursts with 50,000 nodes. My advice is to seriously consider running a proof of concept to identify any potential bottlenecks regardless of which SD-WAN solution and runtime environment you choose.