I have 2 identical hEX refresh routers online, one is hanging off a 4G router’s LAN, the other is direct public-facing connected by pppOe on a fibre broadband connection.
each one is registered with a zerotier IP (I can see them in zt, and they both are on the same 172.24.0.0/16 subnet.
I can ping each one’s own ZT managed IP from itself, but I cannot ping across to the “other” one’s ZT managed IP.
it must be a firewall problem because I have a note that I was able to do it last week when I set up ZT from a totally blank hEX’s.
please could someone glance at my firewall settings to see why the ping won’t “traverse” the ZT switch
Why don't you use any interface lists, especially LAN and WAN ?
You have to allow zerotier interface on forward chain in your firewall.
Usually this is done with interface lists (and then the easiest is to directly add zerotier to LAN). I have an interface list VPN which contains wireguard and zerotier and work from there.
But since you selected another approach in your firewall, you have to do it otherwise.
Easiest is to simply add an accept rule on forward chain for zerotier and put that before any drop rule in that same chain.
However, cleanest way (and easier to maintain) is to use interface lists.
Personal suggestion:
for yourself, it might be easier to order the firewall rules block by block.
First input, then forward, then output (if any).
Makes it also easier to see what happens where.
Performance wise you should put the most used rules more at the top (while still observing proper sequence of rules) but for most this will be a non-issue.
had a brief go at doing this but couldnt work out how in 10seconds and didnt think it was that important for a small project but I clearly dont understand the benefits of lists. my “other” hEX which was built up from a defconfig DOES have lists of course.
I visualised the zt service as a switch that I am trying to connect 2 physical switches together with
in both of the hEXes, I make a bridge including eth2-5 which makes those physical ports transparent. so it followed to add each hEX’s zt1 i/f onto the bridge.
It's a layer 3 interface with it's own IP address, not something to be added to a bridge. You also wouldn't add a wireguard1 interface to a bridge for the same reason.
Or, if you look at normal ethernet interfaces such as ether2. If you don't add it to any bridge, then you can configure IP address directly on that standalone ether2 interface (with /ip address). But if you add ether2 to a bridge, you'll should no longer give that interface it's own IP address anymore, it's only acts as a pure layer 2 interface. In fact, ether2 no longer has an IP address, only the bridge has.
zerotier1 is a layer 3 interface with its own IP address in the range 172.24.0.0/16. Use it like a standalone interface.
Next step is to attempt to make DHCP operate through the zerotier “service”. (I think)
Ultimately I would like to use the zerotier like a long ethernet cable, to bridge my two mikrotik bridges as transparently as a cable would do this between two small switches on the workbench. so that I can have 1 subnet spread across 2 distant sites. linked by an IP tunnel
now that I write that down, I wonder why I am concerned with DHCP at all. if I can bridge the bridges transparently, then I just need to make sure I have only 1 dhcp server….. right?
Wireguard is really an L3 interface, however Zerotier is L2. To be exact it’s a strange amalgamation: it’s what I refer to as “routed Ethernet” which sound like something that shouldn’t exist. This means that peers announce their mac addresses, and keep tables on where to send packets. So it’s L2 with tricks around mitigating unknown unicast.
Of course an L2 interface can be used as a routed interface, and that’s the common usage, and Mikrotik’s docs show how to do this (and only this.)
There are some consideration when more than one address is present behind one peer, and in that case “bridge mode” has to be enabled for the peer. Mikrotik’s implementation fully supports this, both as client and controller.
A particular point is that Zerotier includes a full blown policy system - in essence the peers do not trust each other. This actually makes a lot of sense for ad-hoc type networks, such as someone on the other side of the world running a game server secured by ZT (and yes this happens with some frequency.)
Policies are applied on the client side, and dhcp is filtered - so no, dhcp will not work.
Determining that it’s L2 is sort of easy. An ip (L3) packet has an ip header, at L2 an additional ethernet header is added. L2 also implies bridging (switching) while IP packets are routed.
Being “transparent” for any/all traffic is a different property. What you’re saying that a switch only looks at the mac address (which is a fine principle) but for anything but the simplest dumb switch is not the case. Bridges have bridge firewalls, switches have switch rules. These are usually able to match into the packets and understand ip/tcp/udp headers just fine. Networking equipment has simply evolved from the initial principles.
Blocking dhcp is simply a security consideration, and zt provides alternative addressing allocation. It’s also not in the least illegal to run a dhcp server on the remote end. The subnets are fine to overlap, it’s only necessary to have non-overlapping address ranges.
This is a kind of common misconception. E.g. when you capture a packet on a genuine L3 interface like Wireguard, IPIP or PPPoE, you simply don’t have an ethernet header (or MAC address); just the naked IP packet.
When this packet has to be sent over Ethernet (an L2 interface) that’s when it’s equipped with the header, and this is the point that ARP or ND is performed.
EDIT: I can only agree that most packets go through various steps during its journey, understanding these steps meticulously is key to designing networking solutions.
The essential question here is about contrasting Wireguard which is purely L3, no doubt is possible: e.g. Mikrotik’s ui simply doesn’t allow a wg tunnel to become part of a bridge. Zerotier is L2, and it shouldn’t be surprising that it can be made part of a bridge. Wireguard is similar to ppp, Zerotier is ethernet.