Community discussions

MikroTik App
 
rizan
just joined
Topic Author
Posts: 3
Joined: Wed Oct 30, 2024 5:32 am

Multiple MikroTik on Zerotier Network

Thu Oct 31, 2024 7:19 am

Hi Guys,

I want to try the idea to make a quite big network subnet to make remote site in the same network with the office/main location.
So when I need to add new mikrotik on 1 or more new remote site, it would be easy since they are on the same subnet.
There will be some office laptop for mobile purpose with zerotier client installed with the assigned IP is 172.22.0.251~172.22.0.254, and those laptops would be able to access all the network at the main and remote sites.
I read some good example <viewtopic.php?t=183424>, but the case was to connect laptop to 1 mikrotik.

From LAPTOP MOBILE#1, I can connect to hub and remote1. But the winbox kept closing after I connect for some time.
From LAPTOP MOBILE#1, I can’t connect to the devices behind hub or remote1, even though it is using same subnet size /21.
Same thing happen on the laptop from network hub to remote1, vice versa.

Weird thing I observe on the on the remote1 traffic on zerotier1 interface, it is showing Tx 25Mbps or sometimes 40Mbps even though there are no download/upload between hub and remote1.

Is there any configuration that I might missed?

The IP address allocation will be such:
Zerotier managed route: 172.22.0.0/21 via 172.22.0.1
Zerotier auto IP : 172.22.0.251~172.22.0.254
hub/main: 172.22.0.1 with DHCP client 172.22.0.101~172.22.0.250
remote1: 172.22.1.1 with DHCP client 172.22.1.101~172.22.1.250


this is configuration from hub/main:
# 2024-10-31 14:42:44 by RouterOS 7.15.3
# software id = FB3Z-1N2N
#
# model = RB450Gx4
# serial number = HD4082KJSKS
/interface bridge
add add-dhcp-option82=yes admin-mac=18:FD:74:A4:46:52 auto-mac=no comment=\
    defconf dhcp-snooping=yes name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=eth1
set [ find default-name=ether2 ] name=eth2
set [ find default-name=ether3 ] name=eth3
set [ find default-name=ether4 ] name=eth4
set [ find default-name=ether5 ] name=eth5 poe-out=off
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=172.22.0.101-172.22.0.250
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/port
set 0 name=serial0
/queue type
add cake-diffserv=besteffort cake-flowmode=dual-srchost cake-nat=yes kind=\
    cake name=Cake-Tx-Down
add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-nat=yes kind=\
    cake name=Cake-Rx-Up
add kind=fq-codel name=fq-codel
add fq-codel-ecn=no kind=fq-codel name=fq-codel-ethernet-default
/queue interface
set eth1 queue=fq-codel
set eth2 queue=fq-codel-ethernet-default
set eth3 queue=fq-codel-ethernet-default
set eth4 queue=fq-codel-ethernet-default
set eth5 queue=fq-codel-ethernet-default
/zerotier
set zt1 comment="ZeroTier Central controller - https://my.zerotier.com/" \
    name=zt1 port=9993
/zerotier interface
add allow-default=yes allow-global=yes allow-managed=no disabled=no instance=\
    zt1 name=zerotier1 network=<zt-network>
/queue interface
set zerotier1 queue=fq-codel
/tool traffic-generator port
add interface=zerotier1 name=port1
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge port
add bridge=bridge comment=defconf interface=eth2
add bridge=bridge comment=defconf interface=eth3
add bridge=bridge comment=defconf interface=eth4
add bridge=bridge comment=defconf interface=eth5
add bridge=bridge interface=zerotier1
/interface bridge settings
set use-ip-firewall-for-pppoe=yes use-ip-firewall-for-vlan=yes
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=eth1 list=WAN
/ip address
add address=172.22.0.1/21 comment="don't delete!!!" interface=bridge \
    network=172.22.0.0
/ip dhcp-client
add comment=defconf interface=eth1
/ip dhcp-server network
add address=172.22.0.0/21 comment=defconf dns-server=172.22.0.1 domain=local \
    gateway=172.22.0.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=172.22.0.1 name=hub.local
add address=172.22.1.1 name=remote1.local
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip firewall service-port
set rtsp disabled=no
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=Australia/Brisbane
/system identity
set name=hub/main
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN


And this is the configuration from remote1:
# 2024-10-31 14:43:55 by RouterOS 7.15.3
# software id = GLA4-S9GI
#
# model = wAPGR-5HacD2HnD
# serial number = HG909YAEWXT
/interface bridge
add admin-mac=D4:01:C3:59:FF:C6 auto-mac=no comment=defconf dhcp-snooping=yes \
    name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=eth1
set [ find default-name=ether2 ] name=eth2
/interface lte
set [ find default-name=lte1 ] allow-roaming=no band="" disabled=yes \
    sms-read=no
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa2-psk mode=dynamic-keys name=local-profile \
    supplicant-identity=""
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX \
    disabled=no frequency=auto mode=ap-bridge name=wifi1 security-profile=\
    local-profile ssid=MikroTik
set [ find default-name=wlan2 ] band=5ghz-a/n/ac channel-width=\
    20/40/80mhz-XXXX disabled=no frequency=auto mode=ap-bridge name=wifi2 \
    security-profile=local-profile ssid=MikroTik
/ip pool
add name=default-dhcp ranges=172.22.1.101-172.22.1.250
/ip dhcp-server
add address-pool=default-dhcp comment="default" interface=bridge \
    name=defconf
/queue type
add kind=fq-codel name=fq-codel-ethernet-default
add cake-diffserv=besteffort cake-flowmode=dual-srchost cake-nat=yes kind=\
    cake name=Cake-Tx-Down
add cake-diffserv=besteffort cake-flowmode=dual-dsthost cake-nat=yes kind=\
    cake name=Cake-Rx-Up
add kind=fq-codel name=fq-codel
/queue interface
set eth1 queue=fq-codel-ethernet-default
set eth2 queue=fq-codel-ethernet-default
set wifi1 queue=fq-codel-ethernet-default
set wifi2 queue=fq-codel-ethernet-default
/zerotier
set zt1 comment="ZeroTier Central controller - https://my.zerotier.com/" \
    name=zt1 port=9993
/zerotier interface
add allow-default=yes allow-global=yes allow-managed=no disabled=no instance=\
    zt1 name=zerotier1 network=<zt-network>
/queue interface
set zerotier1 queue=fq-codel
/interface bridge port
add bridge=bridge comment="defconf: used for WAN connection." \
    disabled=yes interface=eth1 trusted=yes
add bridge=bridge comment=defconf interface=eth2
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=zerotier1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=lte1 list=WAN
add interface=eth1 list=WAN
/interface wireless access-list
add interface=wifi1 signal-range=-60..120 vlan-mode=no-tag
add interface=wifi2 signal-range=-60..120 vlan-mode=no-tag
/ip address
add address=172.22.1.1/21 comment="don't delete!!!" interface=bridge \
    network=172.22.0.0
/ip dhcp-server network
add address=172.22.0.0/21 comment=defconf dns-server=172.22.1.1 domain=local \
    gateway=172.22.1.1
/ip dns
set allow-remote-requests=yes cache-size=1024KiB
/ip dns static
add forward-to=172.22.0.1 match-subdomain=yes name=local type=FWD
add address=172.22.1.1 name=remote1.local
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=all
/ip firewall service-port
set rtsp disabled=no
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=Australia/Brisbane
/system identity
set name=remote1
/system logging
add topics=wireless,info
add topics=dhcp,info
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
Network Design.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4235
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multiple MikroTik on Zerotier Network

Sat Nov 02, 2024 10:48 pm

You got a few things going on there, but topology should work.

I don't have an instant answer on what's wrong, but couple things to check:

1. On the ZeroTier Controller (my.zerotier.com), did you set "allow bridging" on the Mikrotik members?

2. zerotier1 is not a member of either LAN or WAN interface list, and your using "/interface/bridge/settings/set ... use-ip-firewall-for-vlan=yes" ... I didn't study your firewall in depth, but overall it does seem firewall is blocking traffic. Add zerotier1 to the LAN interface-list may get things working, and if you need more security it's likely easier to add once it's working.

Also, when you loose connection, it could be ARP related. I'd keep on an eye on /ip/arp for that one and see what's generally going on. ARP and ZeroTier gets tricky, and since your bridging... even more tricky. #1 above may be related since I believe it's ARP why ZT need to know if it's bridged, although not 100% on that one.
 
rizan
just joined
Topic Author
Posts: 3
Joined: Wed Oct 30, 2024 5:32 am

Re: Multiple MikroTik on Zerotier Network

Tue Nov 05, 2024 1:16 pm

You got a few things going on there, but topology should work.

I don't have an instant answer on what's wrong, but couple things to check:

1. On the ZeroTier Controller (my.zerotier.com), did you set "allow bridging" on the Mikrotik members?

2. zerotier1 is not a member of either LAN or WAN interface list, and your using "/interface/bridge/settings/set ... use-ip-firewall-for-vlan=yes" ... I didn't study your firewall in depth, but overall it does seem firewall is blocking traffic. Add zerotier1 to the LAN interface-list may get things working, and if you need more security it's likely easier to add once it's working.

Also, when you loose connection, it could be ARP related. I'd keep on an eye on /ip/arp for that one and see what's generally going on. ARP and ZeroTier gets tricky, and since your bridging... even more tricky. #1 above may be related since I believe it's ARP why ZT need to know if it's bridged, although not 100% on that one.
I did setting the allow bridging from zerotier web for the mikrotik member.

I tried to put zerotier1 on the LAN list, but it's still the same. So I delete it again.

One thing that bothering me actually why the traffic on the zerotier1 interface was so high but it doesn't go anywhere. I checked the eth1 which is the WAN, the total traffic on the eth1 much smaller compared to traffic on the zerotier1 interface. I don't understand where that traffic is going.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1603
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: Multiple MikroTik on Zerotier Network

Tue Nov 05, 2024 2:18 pm

I might have missed or misunderstood something when I read the description and checked the config, but it seems like you’re using the same subnet for your local networks and ZeroTier, which can get tricky if you’re not careful. Are you planning to bridge (Layer 2/Ethernet) or route (Layer 3/IP) all the local networks together?

I'm assuming you've already checked out these links:
- https://docs.zerotier.com/integrating-physical-networks
- https://docs.zerotier.com/route-between-phys-and-virt/
- https://docs.zerotier.com/bridging/
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4235
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multiple MikroTik on Zerotier Network

Tue Nov 05, 2024 11:03 pm

What I'm not sure of is the effect of "use-ip-firewall-for-vlan=yes" as that could add another dimension to ZT bridging. IDK but I've never tested that option with ZT.
 
rizan
just joined
Topic Author
Posts: 3
Joined: Wed Oct 30, 2024 5:32 am

Re: Multiple MikroTik on Zerotier Network

Thu Nov 07, 2024 4:18 am

I might have missed or misunderstood something when I read the description and checked the config, but it seems like you’re using the same subnet for your local networks and ZeroTier, which can get tricky if you’re not careful. Are you planning to bridge (Layer 2/Ethernet) or route (Layer 3/IP) all the local networks together?

I'm assuming you've already checked out these links:
- https://docs.zerotier.com/integrating-physical-networks
- https://docs.zerotier.com/route-between-phys-and-virt/
- https://docs.zerotier.com/bridging/
Yes the goal is to get L2 bridge. I follow the guide from zerotier docs and from this forum. Works for 1 mikrotik, success connect mobile laptop on zerotier network to the network below the mikrotik. But didn't success when I start using 2 or more mikrotik.

What I'm not sure of is the effect of "use-ip-firewall-for-vlan=yes" as that could add another dimension to ZT bridging. IDK but I've never tested that option with ZT.
It seems it is already there in the default configuration. I didn't pay attention on that part.

Who is online

Users browsing this forum: McSee and 32 guests