EoIP over Back To Home between two routers, both behind CGNAT — ZeroTier relay drops broadcast

RouterOS Wireless Networking
1

I am trying to bridge two LANs at Layer 2 between two sites that have no possible cable path between them (about 70 m apart, reinforced concrete, one site is an underground garage). Both sites have mobile-only WAN, two separate SIMs from the same Italian carrier. A Tesla Wall Connector in the garage has to sit on the same flat L2 segment as a Powerwall Gateway in the house, because pairing and daily charge management rely on mDNS/broadcast discovery. A routed L3 VPN is not enough, and this has to be a permanent link.

**Setup**

● Site A (house): Chateau 5G R17 ax, RouterOS 7.23.3. Main router, bridge `bridge`, LAN 192.168.88.0/16, DHCP server on 192.168.88.1.

● Site B (garage): Chateau LTE12, RouterOS 7.23.3. Bridge `bridge1`, management address 192.168.88.254/16, no DHCP server.

● A FRITZ!Box 3490 in AP mode (DHCP disabled) cabled to the LTE12 provides WiFi to the Wall Connector.

● Both SIMs are behind CGNAT on IPv4. Both get a dynamic IPv6 /64, but the carrier blocks all inbound IPv6 — verified with explicit ICMPv6 and UDP accept rules in place, no reply on either global address.

**What I built**

ZeroTier on both routers as the transport, with an EoIP tunnel between the two ZeroTier addresses, bridged at both ends:

```

/interface eoip add name=eoip-box remote-address=10.81.124.187 tunnel-id=100 mtu=1400

/interface eoip add name=eoip-casa remote-address=10.81.124.250 tunnel-id=100 mtu=1400

```

The EoIP interface is a bridge port on both sides, so the two LANs form one flat segment with a single DHCP server on the house side. Broadcast is enabled on the ZeroTier network, multicast limit 32. MTU also tested at 1350 and 1300, with `clamp-tcp-mss=yes`.

**Bench test: passed.** With both routers side by side on a desk and the garage router on its own SIM (not a LAN cable, to avoid a bridge loop), a laptop on the FRITZ!Box WiFi got 192.168.88.183 from the house DHCP through the tunnel and reached the house LAN with 0% packet loss.

**Production: failed.** Once the LTE12 was installed in the garage, unicast still works — ICMP across the tunnel is fine both ways — but broadcast does not. No client on the garage side ever gets an address. A sniffer on the LTE12 shows the DHCP Discover frames leaving the garage bridge; the DHCP log on the house router shows nothing arriving.

**Cause:** the two ZeroTier peers never establish a direct path. The house node presents itself over IPv4 from behind CGNAT, the garage node over IPv6. The two families cannot meet, so everything goes through ZeroTier's public relays, and the relays do not carry broadcast/multicast. On the bench the two nodes were on the same LAN and peered directly, which is exactly why the bench test passed.

**Where I am now**

The plan is to replace ZeroTier with Back To Home as the transport, keeping the same EoIP-over-VPN architecture: BTH relays plain WireGuard, so an EoIP payload including broadcast should pass through untouched. I have read the thread here about creating a BTH user on one router, exporting its WireGuard config and importing it on the second router with `wg-import`, and it looks like exactly what I need.

Both routers are on 7.23.3. I am aware 7.24 was released a few days ago, and I have read the WireGuard fixes in it — endpoint domain name now printed by `wg-export`, whitespace handling in AllowedIPs during `wg-import`, wg-quick specific keys now ignored on import — which all look directly relevant to the export/import step this method depends on. I have not upgraded yet: the house router carries a home automation controller that has only just been stabilised, and I would rather not put a few-days-old release on it before the tunnel itself is proven. If the export/import step misbehaves on 7.23.3 I will upgrade, starting with the remote router.

Before I commit to it, a few questions to anyone who has run this in production:

1. How stable is the BTH relay over weeks for a permanent low-bandwidth link (a few kbit/s, 24/7)? Are there session timeouts or keepalive settings I should set from the start?

2. Any recommended EoIP MTU for EoIP encapsulated in WireGuard over LTE? I am starting from 1400 with MSS clamping, but I would rather begin from a value someone has actually measured.

3. Does anything change if both ends are behind CGNAT rather than just one? In my case neither router can be reached from outside.

4. One thing I ran into that others may want to avoid: with the remote end powered off, the EoIP interface left enabled as a bridge port kept changing state, and the RSTP topology-change notifications flushed the bridge MAC table every few minutes. A home automation controller on the same bridge lost connectivity roughly every five minutes until I disabled the port. I will be setting `edge=yes` on the tunnel port this time — is that the right approach, or would you use `protocol-mode=none` or a separate bridge instead?

5. Has anyone set up BTH site-to-site on 7.24 specifically? If those `wg-export` / `wg-import` fixes make the difference between a working and a broken import, that is worth knowing before starting rather than after.

Supout files and `/export` output from either router available if useful. Thanks.

2

Something else is wrong here in your IPv6 configuration. No carrier will waste resources to track individual connections (stateful!!!) to be able to "block inbound IPv6". This is not like IPv4 with CGNAT that requires tracking.

Even in the unlikely case when they really "block inbound IPv6", that blocking is soon gone once your two WG peers start sending handshakes. Just look at this:

Both ends know the IPv6 address and remote port of the other end. Site A has IP_A:Port_A, site B has IP_B:Port_B. Configure those settings in the WG peers' endpoint data, and each side will start sending UDP packets.

  • Site A: outgoing packets with source IP_A:Port_A destination IP_B:Port_B. The ISP on site A will not block this.
  • This packet arrives at site B and might be "blocked" in your theory (with ISP deploying full connection tracking, which is doubtful), however because
  • Site B: also sends outgoing packets with source IP_B:Port_B destination IP_A:Port_A. The ISP on site B will not block this. And if they deploy conntrack with inbound blocking, this will automatically create an entry that allows return packets with source IP_A:Port_A destination IP_B:Port_B. Which means:
  • Packets sent by site A towards site B are no longer blocked by ISP B. Similarly:
  • Packets sent by site B towards site A are no longer blocked. Because ISP A already sees outgoing packets with source IP_A:Port_A destination IP_B:Port_B, they will allow incoming packets with source IP_B:Port_B destination IP_A:Port_A.

So whatever inbound blocking, if it exists, will not be an obstacle at all. If IPv6 is properly configured at your two sites, you can establish a direct WG connection between them without any issue.

Using that direct IPv6 WG connection will be much better than whatever hack you try with the BTH relay.

3

I don't know if this is from some sort of AI, but it's dead wrong. In your case ZT only encounters unicast packets because it only transports the EoIP encapsulated packets. BTW, ZT relays have no problem with broadcast/multicast anyway.

4

Off Topic:

@lurker888, please stop the AI bashing, it's just so tiresome. People can be wrong without AI being involved, and simply attacking someone without explaining what's actually wrong isn't much help to the OP.

So if you have something useful to add, please help out and explain the technical issue and why. Otherwise, feel free not to post at all. Thank you! :folded_hands:
__

On Topic:

@MMX1 - Since your DHCP broadcast is encapsulated inside EoIP, from ZeroTier's point of view the traffic is just standard unicast IP traffic, so whether the ZeroTier path is DIRECT or RELAY doesn't matter at all.

Worth noting, ZeroTier itself can be configured as an L2 network, so there's actually no need for EoIP at all. And using ZeroTier Flow Rules, you can restrict what type of L2 traffic you want between the segments.

Anyway, since IP through your tunnel works, I'd first look at EoIP itself, especially MTU/fragmentation and the bridge path. A packet capture on the EoIP interfaces at both ends should tell you pretty quickly where the DHCP frame disappears.

@CGGXANNX is right that a direct IPv6 WireGuard tunnel would probably be easier. BUT there's an important thing to keep in mind with carrier-assigned IPv6/IPv4 addresses: they're mostly dynamic (especially on consumer plans). When the address changes, you'll need to update the WireGuard endpoint on both ends. One way is to use DDNS (for example Mikrotik IP Cloud) for the endpoints and a script that detects an IP address change and updates the WG endpoint with the new addresses. Note that while MikroTik IP Cloud is free, it isn't completely reliable and doesn't support dual-stack.

In this case, ZeroTier is a much better fit since it manages address changes and roaming automatically and can use both IPv4 and IPv6 at the same time.

Where I don't agree is the assumption that carriers wouldn't use stateful filtering for IPv6. That's perfectly normal, especially with MNOs, and there's no guarantee that the carrier is using exactly that kind of filtering. It could just as well be some other inbound/outbound policy or ACL. Typical ports filtered in consumer plans are outbound SMTP and inbound HTTP and HTTPS.

5

Okay, maybe the bashing is too much, but I did add to the discussion (in the same way you did,) by pointing out that the eoip encapsulated traffic is unicast anyway.

And now to the point. While ZT can absolutely do L2 bridging, there is an implicit (cannot-be-disabled) filter specifically for DHCP for security reasons. So if ZT bridging is used, both sides have to have DHCP servers - probably with non-overlapping addresses from the same subnet in this case.

6

@lurker888, that's simply not correct.

ZeroTier absolutely supports L2 bridging, including broadcast/multicast, and the official documentation explicitly mentions selectively blocking traffic like DHCP across the bridge if you don't want it there. That directly contradicts the claim that there's a mandatory, non-disableable DHCP filter requiring a DHCP server on both sides.

If you're going to correct someone this strongly, at least make sure the technical claim is actually correct.

7

I suspect this part may be missed. That either a more tailor ACL, or simply "accept all;" like it cause of the troubles.

I'll add that ZeroTier itself does not broadcast DHCP, ZeroTier "members" get their IP address from the ZT client. So even in bridge mode, you need you're own DHCP server which effect non-ZeroTier devices on the bridge.

And correct EoIP is not needed with ZT. It's useful on WG or BTH-WG to get L2 over the L3 WG tunnels.

8

Yep. Went for a deep dive, and indeed ZT's previously firm stance on this has changed. As far as I can tell, it was around 2022/23, and the discussions around this on discuss.zerotier... have been scrubbed. (Previously, the dhcp blocking flow rules were installed on the client "ZeroTier One" side in addition to what was pushed by the controller.)

Actually, now the controller even returns a "dhcp" flag that indicates whether dhcp requests/responses are expected to propagate through the fabric. Mikrotik even seems to expose this flag, although without documenting it...

So you're right, and the easiest setup would be to simply use ZT in bridge mode.

9

Yes, I only get dynamic addresses with my ISPs so the IPv6 tunnels between my sites all have dynamic IPv6 endpoints, but it's trivial to setup scheduled scripts that updates the endpoint IPv6 address in the peer entry (the DDNS domain is put in the comment of the peer, not in the endpoint field) when it no longer matches the resolved result. And services like No-Ip (what I am using) are very reliable, have short TTL, and also allow you to only have an AAAA record, without A record. The interruption on address changes is less than a couple of minutes and connectivity is re-established automatically.

Also, most of the time, the tunnel fixes itself even before your scripts updates the endpoint address if the ISP doesn't do shenanigans like blocking inbound IPv6. Because WG is built for roaming, the side that just got a new IPv6 address just has to send a handshake packet to the other side (that still has old IP) and that other side will automatically update the Current Endpoint value of the peer, and use that new address as destination for future packets.

Yes, but usually these kinds of blocking just do a stateless drop for the relevant destination ports (22, 25, 53, 123, 80, 443, etc...) when the destination address is within the address block allocated to customers, not with stateful tracking. There are ISPs in my country where Windows user are unable to synchronize the time because of this type of blocking.

10

Thanks everyone, this has been genuinely useful.

One thing I should correct on my side: we've been talking about DHCP as the test case, but the actual discovery protocol between the Tesla Wall Connector and the Powerwall Gateway is mDNS (multicast, 224.0.0.251 udp/5353), not broadcast DHCP. So a DHCP test may not tell me much either way. I'll be capturing mDNS.

The honest state of things is that I have never measured anything. I jumped straight to trying different transports without ever checking the EoIP path itself, which is probably why I've been going in circles. So before I build anything else, three tests, cheapest first:

  1. Can zerotier1 actually be added to a bridge on RouterOS? Empty test bridge, protocol-mode=none, no physical ports. Either it takes it or it doesn't.
  2. Does inbound IPv6 survive on my carrier? Cross ping6 between the two global addresses, both directions, after making sure icmpv6 is accepted in the input chain so I don't fool myself.
  3. EoIP MTU. Test /30 on the EoIP interfaces only, nothing bridged, then a do-not-fragment ping ladder from 1500 down.

None of these touch a bridge in production. Context for that: earlier I added an EoIP bridge port while the remote router was powered off, and the resulting RSTP topology change floods took down a home automation controller on the LAN. Not repeating that.

Does the order make sense, or is there something obvious I'm still missing? I'll post the numbers once I have them.

11

Thanks all — following up with actual numbers this time.

First, a correction to my own framing: we were all discussing DHCP, but the Wall Connector discovers the Powerwall Gateway via mDNS (multicast, udp/5353), not broadcast DHCP. That is what I should have been testing all along.

So I stopped trying different transports and measured the existing path instead. Setup: two Chateau routers (5G at home, LTE12 at the remote site), both RouterOS 7.23.3, both behind CGNAT, EoIP tunnel-id 100 over ZeroTier. Test /30 on the EoIP interfaces only, nothing bridged.

ZeroTier transport, remote to home, 43 packets: 0% loss, avg 124ms

ZeroTier transport, home to remote, 40 packets: 0% loss, avg 136ms

EoIP plain ping: 0% loss, avg 144ms

EoIP 1272 bytes with do-not-fragment: 0% loss

EoIP 1500 bytes with do-not-fragment: 0% loss, avg 170ms

So MTU is not the problem. I had set the EoIP to mtu=1300 at some point and never measured it — turns out 1500 goes through intact. Both ends are now at 1500.

I also tested whether the zerotier1 interface can actually be added to a bridge on RouterOS: empty test bridge, protocol-mode=none, no physical ports. Accepted without error. So L2 ZeroTier is at least possible, although that only proves a necessary condition, not that multicast really crosses the fabric.

Two things excluded then: the transport and the MTU. What is left unmeasured is multicast behaviour across the bridge, which is where I am going next with a capture on udp/5353 at both ends.

One more thing worth recording, since it burned me before. I had added the EoIP bridge port while the remote router was powered off, and the resulting RSTP topology change floods took down a home automation controller on the LAN. The fix turned out to be different per site: the remote router has a single active port, so protocol-mode=none is safe there, while the home router has four active ports, so I left RSTP on and set edge=yes on the tunnel port only.

Anything obvious I should capture beyond udp/5353 when I get to the multicast test?

12

Solved. Posting the full answer because the cause was nowhere near where I had been looking, and someone else will hit this.

Root cause: dhcp-snooping was enabled on the bridge at the remote site, with every bridge port at trusted=no. The bridge was silently dropping DHCP before it could ever reach the EoIP tunnel. One line fixed it:

/interface/bridge/set numbers=0 dhcp-snooping=no

Within seconds my phone, associated to the AP at the remote site, pulled 192.168.88.169 from the DHCP server at home, across EoIP over ZeroTier. Which is exactly what was supposed to happen all along.

What makes this worth writing up is how well it disguised itself. No device at the remote site could ever get an address - phone, laptop, Wall Connector, all of them. That looks exactly like a broken tunnel, so I spent months rebuilding the tunnel. It was never the tunnel.

The measurements, for the record, all taken before I found it:

ZeroTier transport, both directions, 83 packets total: 0% loss, avg 124 and 136ms
EoIP plain ping: 0% loss, avg 144ms
EoIP 1500 bytes with do-not-fragment: 0% loss, avg 170ms
zerotier1 added to an empty test bridge: accepted, no error

So the transport was fine, the MTU was fine (I had it clamped to 1300 for no measured reason - 1500 goes through intact), and L2 bridging over ZeroTier is possible. Every architectural theory I had was wrong.

Larsa, your first reply is what actually solved this: look at the EoIP itself, capture at both ends, see where the DHCP frame disappears. That was the right instruction and I should have followed it sooner. When I finally ran the capture at the receiving end, the DHCP requests were there on the local ports and on the bridge, and completely absent on the EoIP interface. Nothing was crossing. The picture was unambiguous.

One detail that nearly fooled me: I captured mDNS first, on the theory that it was the traffic that actually mattered for the Tesla discovery. mDNS crossed the tunnel perfectly - I could see multicast from the home LAN arriving at the remote site. Very reassuring, completely useless. dhcp-snooping filters DHCP specifically, so every other class of traffic looked healthy. If I had trusted the mDNS result I would still be looking.

Separate lesson from the same project, in case it saves someone a bad evening. Earlier on I added the EoIP bridge port while the remote router was powered off, and the RSTP topology change floods took down a home automation controller on the LAN. The fix turned out to be different per site: the remote router has a single active port, so protocol-mode=none is safe there, while the home router has four active ports, so I left RSTP running and set edge=yes on the tunnel port only. With that in place the bridge has now been up for over half an hour with no effect on the controller at all.

Thanks to everyone who weighed in. The useful lesson for me is that I kept redesigning the architecture when I should have been capturing packets.

13

Or... the unlearned lesson is why EoIP is used on top of ZeroTier — you have an extra layer and inefficency and potential MTU-related issues. When you already have native ethernet available from ZeroTier, so could have just carried VLANs over that directly if that was the intent behind EoIP. This avoid yet another layer of MAC/IP headers to reason about vs just trying to understand ZeroTier mechanism (like Flow Rules) better. So in some sense "redesigning the architecture" may in fact have been warranted here. Just my opinion.

14

Solved — and I owe thanks to everyone who took the time to reply.

Two of the replies changed the direction of the work. The correction about ZeroTier was right: EoIP encapsulates everything as unicast, so ZeroTier could not have been filtering broadcast, and my original diagnosis was simply wrong. That pushed me to stop guessing and start measuring. The suggestion to look at EoIP MTU and fragmentation turned out to be half of the actual answer.

For anyone who finds this thread later, the root cause was two independent problems:

  1. Firewall. The input chain had an accept rule for the old ZeroTier interface but none for the WireGuard interface that replaced it. GRE (protocol 47) was silently dropped by the default "drop all not coming from LAN" rule on both routers. The symptom is misleading: EoIP shows RUNNING, TX counters climb, RX stays at zero on both ends, and the bridge host table stays empty. /interface/monitor-traffic is what exposes it.
  2. MTU. EoIP was left at the 1500 default. The WireGuard interface has MTU 1420, and EoIP adds 42 bytes (28 GRE/IP + 14 Ethernet), so the usable ceiling is 1378. Anything larger was dropped without a trace — keepalives are small, so the tunnel looked healthy the whole time.

One more thing worth recording: the bridge on the home router had mtu=auto, so setting EoIP to 1378 caused the bridge to inherit it and push the lower MTU onto the whole LAN. It has to be pinned to 1500 explicitly.

Transport is now WireGuard over Back To Home instead of ZeroTier: 0% packet loss versus 7%, and much tighter jitter. edge=yes on both EoIP bridge ports keeps RSTP topology-change floods from reaching a home automation controller when the remote site powers down — tested with the remote end off for fifteen minutes, no disruption.

The Wall Connector now appears in the Tesla app next to the Powerwall. Thanks again to all of you.