Community discussions

MikroTik App
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

WireGuard site to site routing help

Fri Nov 08, 2024 4:34 am

Hi guys, Ive been self learning ROS for the past few month, please excuse my dumb questions
Here is my setup:
4 routers with latest 17.16 OS
Wireguard site to site tunnels between each site, working as it should.
One of the sites has a peer for mobile laptop and i can access that site from laptop, but not other sites.
Question: is it possible to access other sites via laptop wireguard peer?
Site A: LAN 192.168.21.0/24 WG interface 172.17.0.21/24 (this site has mobile wg peer)
Site B: LAN 192.168.22.0/24 WG interface 172.17.0.22/24
Site C: LAN 192.168.23.0/24 WG interface 172.17.0.23/24
and so on
mobile WG interface is 172.17.0.99/32 and its in the list of allowed IPs of all sites
what am i missing here?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Fri Nov 08, 2024 5:01 am

Wireguard is very flexible anything is possible..........

Which of the four routers is the peer Server for handshake??
 
anovojr
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Wed Nov 15, 2017 9:24 am
Location: Philippines
Contact:

Re: WireGuard site to site routing help

Fri Nov 08, 2024 8:17 am

To make the laptop able to reach the other sites, it sounds like you’ll need to update the routing on Site A to forward traffic from the laptop to Sites B, C, and so on. You may need to add routes to those other sites in the WireGuard config on the laptop or adjust Site A's firewall rules to allow forwarding to the other site subnets.
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Fri Nov 08, 2024 7:45 pm


Which of the four routers is the peer Server for handshake??
None of them are peer Server. Each site has peers to other sites.
Mobile peer is configured at site A
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Fri Nov 08, 2024 8:49 pm

Yes, they are all peers once a connection has been established........ but generally speaking the router acting as server for handshake will have the udp port open on the input chain for example.

Are you saying all routers have public IPs and open UDP ports for wireguard??
Can you post wireguard settings for all four please.
/interface wireguard
AND
/interface wireguard peers

(minus any public WANIP information, and keys)
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Sat Nov 09, 2024 4:26 am

Yes, they are all peers once a connection has been established........ but generally speaking the router acting as server for handshake will have the udp port open on the input chain for example.

Are you saying all routers have public IPs and open UDP ports for wireguard??
Can you post wireguard settings for all four please.
/interface wireguard
AND
/interface wireguard peers

(minus any public WANIP information, and keys)
Yes, each site has default port 13231 open in the input chain and subnet to subnet rules in forward chain. Each peer has IP of the other site and persistent keep alive set to 25 sec.
I didn’t learn ACL yet to limit to specific IP addresses only. And will also limit peers Allowed IPs to specific addresses that need to communicate with other sites instead of entire subnet.
I will export wg config file from one of the routers and post here later.
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Sat Nov 09, 2024 7:15 am

Can you post wireguard settings for all four please.
/interface wireguard
AND
/interface wireguard peers

(minus any public WANIP information, and keys)
Im not sure how you want me to post settings /interface wireguard and peers
I use Winbox v3.40
for new router setup i go to WireGuard tab select WG Import and upload config file below (keys and addresses edited), i could not find a way to add other values to config file like Name of peer or WG interface so I change it manually ones interface and peers are generated, also parsing more than two Allowed IPs from config is not working (bug?)
then i add address for wg interface and routes to all sites,
next is firewall rules
then i go to every site and manually add new peer for the site I just created and route
ones tested i add new peer in config file for next setup
[Interface]
ListenPort = 13231
PrivateKey = sdfgknsdlvksd?fV5XC/dfasRnEuisaj7fdgfdkgndfdad=

[Peer]
PublicKey = dfsmg858fV/2wlP3Y0fsvnslkvnl6UlcwL+YWqm/11ODYS4=
AllowedIPs = 10.0.17.14/32, 172.17.14.0/24
Endpoint = hfd985hnwtbd.sn.mynetname.net:13231
PersistentKeepalive = 25

[Peer]
PublicKey = e5698304jedg;/Q05YJZRrk8UrJBoxdxghsdgYGFrttTg=
AllowedIPs = 10.0.17.13/32, 172.17.13.0/24
Endpoint = hsgwtedn76j.sn.mynetname.net:13231
PersistentKeepalive = 25

[Peer]
PublicKey = e/Sa9ohBsfgsdfbsdfbLGjhfnKB6mbejk6Td3/rtgsdfbsf=
AllowedIPs = 10.0.17.99/32
Endpoint = :0

[Peer]
PublicKey = fwYsdfgsdufZpO8utFQsSU2Ehnkp47kjsdfgsdgsc=
AllowedIPs = 10.0.17.16/32, 172.17.16.0/24
Endpoint = 35.85.45.55:13231
PersistentKeepalive = 25

[Peer]
PublicKey = htyvjQPbA6ZwHyasdgsfgsdfIk3BefGfIpHQxfgxdf=
AllowedIPs = 10.0.17.15/32, 172.17.15.0/24
Endpoint = 65.75.25.15:13231
PersistentKeepalive = 25
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Sat Nov 09, 2024 2:56 pm

Yikes, I use no wizards, just do it manually LOL, and normally for keys one just puts "++++++" or something never the real keys.

Okay that gives me a bit of a sense of what you are doing.

Interesting, in 4 router scenario, its rare to have each one have a public IP.
Normally its one, so what is done is
ONE server Router at handshake
THREE client Routers at handshake

Now if two of the four have public IPs what I recommend is a backup separate wireguard network
In this case, assuming Primary router goes down, then make the second one with public IP the primary
ALTERNATE server Router at handshake
Two client Routers at handshake.

It would appear you have all four on same network, with all the same listening port in settings, all are listening on the same port in the input chain as well.
On the peer settings, each router has three peers.

I think I understand what you want to throw into the mix, you as admin via a remote laptop, for example, have connected to one of the routers and want the added ability to reach any subnet LAN or any other router for config purposes.
Not sure why you want to reach one through the other as you can simply DIRECTLY connect to the router you desire.
For example on wireguard app on my iphone, I would simply make four wireguard configs, one for each router............... but its possible so why not think about it.

Now if one wants to wireguard and connect to ONE router and then reach the other three, I know how to do that in the ONE server scenario, but this is different and will have to think about it.
By the way this is apparently called FULL MESH or ( full mess ) TOPOLOGY.
As long as there is no requirement for any local subnet users to use the internet of a different router, this should work just fine. The admiin on the laptop coming into any router depending on firewall rules can access internet if desired.......

LOGIC:
1. One connects to R1 via wireguard and am now at the LAN side of R1 ( used the ios wireguard app )
2. I want to reach R4, which has a specific wireguard IP ( via winbox likely ), or I use the gateway of a subnet on R4, to attempt to reach config of R4 ( via browser likely )
Routes
- R1 knows that the wireguard address of R4, so nothing needs to be made
- R1 knows nothing about remote subnet on R4, so one needs a route made to that subnet
FW Rules
One has to allow traffic that left the tunnel go back into the tunnel, which is what I call a relay rule..............

CONCLUSION/SOLUTION.

On each router add all the possible routes to other subnets.
ex. R1
add dst-address=lansubnetR2 gateway=wireguard1
add dst-address=lansubnetR3 gateway=wireguard1
add dst-address=lansubnetR4 gateway=wireguard1


On each router add a relay rule that allows traffic to exit the tunnel and then renter the tunnel.
add action=accept chain=forward comment="relay wg" in-interface=wireguard1 out-interface=wireguard1

Do this on all your routers and thus you as remote admin should be able to wireguard into any of the four specific routers and reach any other subnet or router to config.
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Sun Nov 10, 2024 7:49 am

Yikes, I use no wizards, just do it manually LOL, and normally for keys one just puts "++++++" or something never the real keys.

Okay that gives me a bit of a sense of what you are doing.

Interesting, in 4 router scenario, its rare to have each one have a public IP.
Normally its one, so what is done is
ONE server Router at handshake
THREE client Routers at handshake

Now if two of the four have public IPs what I recommend is a backup separate wireguard network
In this case, assuming Primary router goes down, then make the second one with public IP the primary
ALTERNATE server Router at handshake
Two client Routers at handshake.

It would appear you have all four on same network, with all the same listening port in settings, all are listening on the same port in the input chain as well.
On the peer settings, each router has three peers.

I think I understand what you want to throw into the mix, you as admin via a remote laptop, for example, have connected to one of the routers and want the added ability to reach any subnet LAN or any other router for config purposes.
Not sure why you want to reach one through the other as you can simply DIRECTLY connect to the router you desire.
For example on wireguard app on my iphone, I would simply make four wireguard configs, one for each router............... but its possible so why not think about it.

Now if one wants to wireguard and connect to ONE router and then reach the other three, I know how to do that in the ONE server scenario, but this is different and will have to think about it.
By the way this is apparently called FULL MESH or ( full mess ) TOPOLOGY.
As long as there is no requirement for any local subnet users to use the internet of a different router, this should work just fine. The admiin on the laptop coming into any router depending on firewall rules can access internet if desired.......

LOGIC:
1. One connects to R1 via wireguard and am now at the LAN side of R1 ( used the ios wireguard app )
2. I want to reach R4, which has a specific wireguard IP ( via winbox likely ), or I use the gateway of a subnet on R4, to attempt to reach config of R4 ( via browser likely )
Routes
- R1 knows that the wireguard address of R4, so nothing needs to be made
- R1 knows nothing about remote subnet on R4, so one needs a route made to that subnet
FW Rules
One has to allow traffic that left the tunnel go back into the tunnel, which is what I call a relay rule..............

CONCLUSION/SOLUTION.

On each router add all the possible routes to other subnets.
ex. R1
add dst-address=lansubnetR2 gateway=wireguard1
add dst-address=lansubnetR3 gateway=wireguard1
add dst-address=lansubnetR4 gateway=wireguard1


On each router add a relay rule that allows traffic to exit the tunnel and then renter the tunnel.
add action=accept chain=forward comment="relay wg" in-interface=wireguard1 out-interface=wireguard1

Do this on all your routers and thus you as remote admin should be able to wireguard into any of the four specific routers and reach any other subnet or router to config.
Why is it rare for 4 routers to have public IPs?

I guess it is a Mesh topology, I didnt plan it that way it just happened because when i added third site to original Server-Client setup i couldnt get routing to work on server side between clients , so i added a peer between clients bypassing the server, when it worked I did the same with site 4.
Mesh tunneling between sites works perfect for this client, because all of his locations are independent businesses and dont share data between locations or have centralized database. main purpose of tunnels is voip traffic between their pbxs. i can share unused sip trunks at any of the sites for outbound calls and send calls between locations via internal trunks bypassing telco provider and increasing outbound capacity.

i think i already have all possible routes to each site, but not the relay rule (I will research it)
Im thinking of adding travel router to mesh topology as a site 0 and also as you suggested mobile peer at each site for laptop in case travel router fails.

here is current configuration from 2 sites
[admin@somerouter] > export
# 2024-11-09 20:51:10 by RouterOS 7.16.1
# software id = xxxxxxxxx
#
# model = RB5009UG+S+
# serial number = xxxxxxxxxxxxx
/interface bridge
add admin-mac=48:xxxxxxxxxx:EA auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=WAN
set [ find default-name=ether2 ] comment=LAN
set [ find default-name=ether6 ] advertise=\
    10M-baseT-half,10M-baseT-full,100M-baseT-half,100M-baseT-full
set [ find default-name=ether8 ] comment="WiFi CapAC"
/caps-man interface
add disabled=yes mac-address=00:00:00:00:00:00 master-interface=none name=cap1 \
    radio-mac=00:00:00:00:00:00 radio-name=""
/interface wireguard
add comment="main site" listen-port=13231 mtu=1420 name=someWG
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=172.17.1.100-172.17.1.254
/ip dhcp-server
add address-pool=dhcp interface=bridge name=defconf
/snmp community
set [ find default=yes ] security=private
/system logging action
set 3 bsd-syslog=yes remote=172.17.1.115 remote-port=1514
/caps-man manager
set enabled=yes
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set accept-router-advertisements=yes disable-ipv6=yes
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/interface wifi capsman
set package-path="" require-peer-certificate=no upgrade-policy=none
/interface wireguard peers
add allowed-address=10.0.17.16/32,172.17.16.0/24 comment="Munster peer" \
    endpoint-address=xxxxxxxxxxxxx.xxxxx.com endpoint-port=13231 \
    interface=someWG name=peer-to-munster persistent-keepalive=25s \
    public-key="fwxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxXc="
add allowed-address=10.0.17.13/32,172.17.13.0/24,172.17.13.1/32 comment=\
    "Hickory tunnel" endpoint-address=xxxxxxxxxxxxx.sn.mynetname.net \
    endpoint-port=13231 interface=someWG name=peer-to-hickory \
    persistent-keepalive=25s public-key=\
    "CbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxTg="
add allowed-address=10.0.17.99/32,10.0.17.0/24 interface=someWG name=\
    peer-for-Dell-Laptop public-key=\
    "e/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxFE="
add allowed-address=10.0.17.15/32,172.17.15.0/24 disabled=yes \
    endpoint-address=xxxxxxxxxxxxx.sn.mynetname.net endpoint-port=13231 \
    interface=someWG name=peer-to-47th persistent-keepalive=25s public-key=\
    "htxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCA="
add allowed-address=10.0.17.14/32,172.17.14.0/24 endpoint-address=\
    xxxxxxxxxxxxx.sn.mynetname.net endpoint-port=13231 interface=someWG \
    name=peer-to-naperville persistent-keepalive=25s public-key=\
    "9axxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxS4="
/ip address
add address=172.17.1.1/24 comment=defconf interface=bridge network=172.17.1.0
add address=10.0.17.1/24 comment="someHq wireguard interface" interface=\
    someWG network=10.0.17.0
add address=xxx.xxx.xxx.xxx/22 interface=ether1 network=xxx.xxx.xxx.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dhcp-server network
add address=172.17.1.0/24 comment=defconf dns-server=8.8.8.8,8.8.4.4 gateway=\
    172.17.1.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8
/ip dns static
add address=172.17.1.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment="allow wireguard" dst-port=13231 \
    protocol=udp
add action=accept chain=input comment="winbox remote" dst-port=xxxx protocol=\
    tcp
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input src-address=10.0.17.0/24
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward src-address=10.0.17.0/24
add action=accept chain=forward src-address=10.0.17.1
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=\
    172.17.16.0/24
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=\
    172.17.15.0/24
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=\
    172.17.14.0/24
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=\
    172.17.13.0/24
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=\
    172.17.12.0/24
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=\
    172.17.11.0/24
add action=accept chain=forward dst-address=172.17.11.0/24 src-address=\
    172.17.1.0/24
add action=accept chain=forward dst-address=172.17.12.0/24 src-address=\
    172.17.1.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=\
    172.17.1.0/24
add action=accept chain=forward dst-address=172.17.14.0/24 src-address=\
    172.17.1.0/24
add action=accept chain=forward dst-address=172.17.15.0/24 src-address=\
    172.17.1.0/24
add action=accept chain=forward dst-address=172.17.16.0/24 src-address=\
    172.17.1.0/24
add action=accept chain=forward disabled=yes dst-address=172.17.1.0/24 \
    src-address=192.168.177.0/24
add action=accept chain=forward disabled=yes dst-address=192.168.177.0/24 \
    src-address=172.17.1.0/24
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=\
    out,none out-interface-list=WAN
/ip ipsec profile
set [ find default=yes ] dpd-interval=2m dpd-maximum-failures=5
/ip route
add comment="route to munster" disabled=yes distance=1 dst-address=0.0.0.0/0 \
    gateway=xxx.xxx.xxx.xxx pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment="route to munster" disabled=no distance=1 dst-address=\
    172.17.16.0/24 gateway=10.0.17.16 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment="route to hickory" disabled=no distance=1 dst-address=\
    172.17.13.0/24 gateway=10.0.17.13 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment="route to 47th" disabled=no distance=1 dst-address=172.17.15.0/24 \
    gateway=10.0.17.15 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=172.17.12.0/24 gateway=10.0.17.12 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=172.17.14.0/24 gateway=10.0.17.14 \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment="route to oakbrook" disabled=no distance=1 dst-address=\
    172.17.11.0/24 gateway=10.0.17.11 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/snmp
set location=home trap-version=3
/system clock
set time-zone-name=America/Chicago
/system identity
set name=SomeRouter
/system logging
add action=remote prefix=INFO topics=info
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=pool.ntp.org
/tool bandwidth-server
set authenticate=no enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool romon
set enabled=yes
/tool sniffer
set streaming-server=172.17.1.115:9000

site 2
# 2024-11-09 21:18:06 by RouterOS 7.16.1
# software id = xxxxxxx
#
# model = RB5009UG+S+
# serial number = xxxxxxxxxxx
/interface bridge
add admin-mac=78:9A:18:CB:13:B2 auto-mac=no comment=defconf name=bridge port-cost-mode=short
/interface ethernet
set [ find default-name=ether1 ] comment="WAN comcast"
set [ find default-name=ether4 ] comment="LTE Management"
set [ find default-name=ether6 ] comment=LinksysPAP2
set [ find default-name=ether7 ] comment="WAN2 LTE-Modem"
set [ find default-name=sfp-sfpplus1 ] auto-negotiation=no comment=CRS326
/interface wireguard
add listen-port=13231 mtu=1420 name=HickoryWG
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=172.17.13.100-172.17.13.249
/ip dhcp-server
add address-pool=dhcp interface=bridge lease-time=10m name=defconf
/system logging action
set 3 bsd-syslog=yes remote=172.17.1.115 remote-port=1513
/interface bridge port
add bridge=bridge comment=defconf interface=ether2 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether6 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether8 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=sfp-sfpplus1 internal-path-cost=10 path-cost=10
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes forward=no
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether7 list=WAN
/interface wireguard peers
add allowed-address=10.0.17.16/32,172.17.16.0/24,172.17.16.1/32 endpoint-address=xxxxxxxxxxx.sn.mynetname.net endpoint-port=13231 interface=HickoryWG \
    name=peer-to-munster persistent-keepalive=25s public-key="fwxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxXc="
add allowed-address=10.0.17.1/32,172.17.1.0/24 endpoint-address=xxxxxxxxxxx.sn.mynetname.net endpoint-port=13231 interface=HickoryWG name=\
    peer-to-highlandpark persistent-keepalive=25s public-key="Oyxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxno="
add allowed-address=10.0.17.15/32,172.17.15.0/24 disabled=yes endpoint-address=xxxxxxxxxxx.sn.mynetname.net endpoint-port=13231 interface=HickoryWG \
    name=peer-to-47th persistent-keepalive=25s public-key="htxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCA="
add allowed-address=10.0.17.14/32,172.17.14.0/24 endpoint-address=xxxxxxxxxxx.sn.mynetname.net endpoint-port=13231 interface=HickoryWG name=\
    peer-to-naperville persistent-keepalive=25s public-key="9axxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxS4="
/ip address
add address=172.17.13.1/24 comment=defconf interface=bridge network=172.17.13.0
add address=10.0.17.13/24 interface=HickoryWG network=10.0.17.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=30m update-time=no
/ip dhcp-client
add comment=defconf interface=ether1 use-peer-dns=no use-peer-ntp=no
add default-route-distance=5 interface=ether7 use-peer-dns=no use-peer-ntp=no
/ip dhcp-server network
add address=172.17.13.0/24 comment=defconf dns-server=172.17.13.1 gateway=172.17.13.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=172.17.13.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input dst-port=13231 protocol=udp
add action=accept chain=input dst-port=xxxx protocol=tcp
add action=accept chain=input comment="allow wg" src-address=10.0.17.0/24
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=drop chain=forward comment=www.star4live.com content=www.star4live.com
add action=accept chain=forward src-address=10.0.17.0/24
add action=accept chain=forward dst-address=172.17.1.0/24 src-address=172.17.13.0/24
add action=accept chain=forward dst-address=172.17.11.0/24 src-address=172.17.13.0/24
add action=accept chain=forward dst-address=172.17.12.0/24 src-address=172.17.13.0/24
add action=accept chain=forward dst-address=172.17.14.0/24 src-address=172.17.13.0/24
add action=accept chain=forward dst-address=172.17.15.0/24 src-address=172.17.13.0/24
add action=accept chain=forward dst-address=172.17.16.0/24 src-address=172.17.13.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=172.17.1.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=172.17.11.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=172.17.12.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=172.17.14.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=172.17.15.0/24
add action=accept chain=forward dst-address=172.17.13.0/24 src-address=172.17.16.0/24
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=output comment=www.star4live.com content=www.star4live.com
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat comment=NVR dst-port=84 in-interface-list=WAN protocol=tcp to-addresses=172.17.13.2 to-ports=80
add action=dst-nat chain=dstnat comment=NEC dst-port=88 in-interface-list=WAN protocol=tcp to-addresses=172.17.13.80 to-ports=88
add action=dst-nat chain=dstnat comment="NEC PC PRO" dst-port=8888 in-interface-list=WAN protocol=tcp to-addresses=172.17.13.80 to-ports=8888
add action=dst-nat chain=dstnat comment=NVR dst-port=8554 in-interface-list=WAN protocol=tcp to-addresses=172.17.13.2 to-ports=8554
add action=dst-nat chain=dstnat comment=NVR dst-port=8554 in-interface-list=WAN protocol=udp to-addresses=172.17.13.2 to-ports=8554
add action=dst-nat chain=dstnat comment=NVR dst-port=84 in-interface-list=WAN protocol=udp to-addresses=172.17.13.2 to-ports=80
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
/ip ipsec profile
set [ find default=yes ] dpd-interval=2m dpd-maximum-failures=5
/ip route
add disabled=no dst-address=172.17.1.0/24 gateway=10.0.17.1 routing-table=main suppress-hw-offload=no
add disabled=no distance=1 dst-address=172.17.16.0/24 gateway=10.0.17.16 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=172.17.11.0/24 gateway=10.0.17.11 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=172.17.12.0/24 gateway=10.0.17.12 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=172.17.14.0/24 gateway=10.0.17.14 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=172.17.15.0/24 gateway=10.0.17.15 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/system clock
set time-zone-autodetect=no time-zone-name=America/Chicago
/system identity
set name=HickoryRouter
/system logging
add action=remote prefix=INFO topics=system,info
add action=remote topics=interface
add action=remote topics=firewall
add action=remote topics=critical
add action=remote topics=error
add
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=pool.ntp.org
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool romon
set enabled=yes
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Sun Nov 10, 2024 4:13 pm

Well, That is the purpose of a Mesh topology, a remote device need only connect to one router and should be able to access ALL Lans subnets, and all routers for config purposes.

If doing it with single server 3 client routers, one connects to the single server with the remote device and then you can reach ALL Lan subnets and all router for config purposes.

You are creating a monster LOL
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1605
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: WireGuard site to site routing help

Sun Nov 10, 2024 5:33 pm

@Usbuild - Once you’ve made some progress and set up your WireGuard tunnels, you can start considering a true "mesh solution" where all nodes connect with each other. This setup makes the network more redundant in case any link goes down.
Wireguard Mesh.png

To avoid adding static routes, try using OSPF. It keeps track of all links and automatically adds the necessary routes to each node’s routing table. A helpful tip: always set up a separate WireGuard link for management on each router.

If you’re thinking of adding more nodes to your network, you might want to consider an SD-WAN solution with built-in MESH support like ZeroTier or Tailscale. ZeroTier is already built into all ARM-based devices from Mikrotik.
You do not have the required permissions to view the files attached to this post.
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Sun Nov 10, 2024 6:57 pm

I agree, It could quickly become a monster. That’s why I’m going to limit it by editing Allowed IPs and Forward chain to specific hosts.
I do believe there is an option to also limit Source Addresses in wireguard Input chain to known IPs, since all of them are dynamic it will break from time to time, but not often enough to become a nuisance. Actually, I don’t know if it will break tunnel when one the peers get new public IP, because both ends have persistent keep alive set, one of them should maintain a tunnel.

I like wireguard mesh topology a lot more vs OpenVPN server-clients setup I ones created… what a nightmare it was dealing with all routing tables, certificates, users and not to mention troubleshooting it all
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Sun Nov 10, 2024 6:58 pm

@Usbuild - Once you’ve made some progress and set up your WireGuard tunnels, you can start considering a true "mesh solution" where all nodes connect with each other. This setup makes the network more redundant in case any link goes down.

Wireguard Mesh.png

To avoid adding static routes, try using OSPF. It keeps track of all links and automatically adds the necessary routes to each node’s routing table. A helpful tip: always set up a separate WireGuard link for management on each router.

If you’re thinking of adding more nodes to your network, you might want to consider an SD-WAN solution with built-in MESH support like ZeroTier or Tailscale. ZeroTier is already built into all ARM-based devices from Mikrotik.
Isn’t it already a Mesh?
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1605
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: WireGuard site to site routing help

Sun Nov 10, 2024 7:17 pm

If each of your 4 nodes is connected to all the others (ie 6 tunnels in your config), then the answer is yes. But if the other nodes only connect to a central node, the answer is no.
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Sun Nov 10, 2024 7:27 pm

If each of your 4 nodes is connected to all the others (ie 6 tunnels in your config), then the answer is yes. But if the other nodes only connect to a central node, the answer is no.
yes, they all have peers to each other
I looked into OSPF and it seems like an overkill for this size network. static routes should be easier to manage and troubleshoot. But ill keep it in mind as an option, thanks for the tip.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1605
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: WireGuard site to site routing help

Sun Nov 10, 2024 7:57 pm

I wouldn’t call it overkill. OSPF is actually pretty easy to set up and used with the BFD option you get quick failover if a link goes down. You can always add OSPF later if you want, and you can run it on top of the static routes, which then act as backup routing.
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Mon Nov 11, 2024 12:44 am

I wouldn’t call it overkill. OSPF is actually pretty easy to set up and used with the BFD option you get quick failover if a link goes down. You can always add OSPF later if you want, and you can run it on top of the static routes, which then act as backup routing.
I will definitely look into it, but at the moment I dont understand how it works and how it could possibly add failover to a mesh topology? i dont have any other vpn service or second ISP with enough bandwidth to handle alternative routes
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Mon Nov 11, 2024 9:06 pm

@anav it all worked great until I added one more router to this mesh. I took a new rb5009 connected it directly to the modem with my laptop as the only host in subnet.
Router successfully joined wireguard mesh and I can connect to any mikrotik APs or CRS switches via winbox on any of the sites, except Mikrotik gateways itself and the worst part is, problem is inconsistent.
I left winbox on auto reconnect to site A, opened another session, connected to AP on site A via ROMON, discovered a router I’m trying to connect in order to see if any firewall rules are blocking connection and while doing it I noticed that winbox managed to establish connection after many retries. So nothing really changed, winbox finally reconnected after many attempts.
So winbox will eventually connect if I let it on retry. It doesn’t matter if I connect via private IP address or via FQDN.
What could be delaying connection?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Mon Nov 11, 2024 9:24 pm

Glad to hear it works no problem for 4 routers, perhaps 5 is a dark magic evil number. Did you name the new router LARSA by any chance? ;-PPP

To add a fifth router means
ON ALL FOUR ROUTERS you need to ADD another peer client

add allowed-peers=IPaddress#5/32,(anysubnets on said router 5), interface=wireguard public-key="======" comment="To LARSA:"
endpoint-address=publicIP#5 endpoint-port=as applicable keep-alive=25s.


/ip route
add dst-address=subnet_on_LARSA gateway=wireguard routing-table=main


/ip firewall
add listening port on input chain
add relay rule on forward chain
add access to local subnet from wireguard
add access for local subnet to wireguard

++++++++++++++++++++++++++++++++++++++++++++++++++++++
Then of course one needs to setup router 5 correctly...........
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Mon Nov 11, 2024 11:11 pm

@anav all of the above except relay rule is there.
I will add it, but how would it explain the issue.
I can reach any ip in remote subnet 172.17.13.0/24 but not the 172.17.13.1 ?
Ping from 172.17.14.1 to 172.17.13.1 works, but I can’t access it via winbox. Not true actually, I can access it but delay is few minutes…. I’m going to time it and see if delay is consistent.
It almost feels like my routing rules fail and that magical OSPF turns on and saves the day
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1605
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: WireGuard site to site routing help

Mon Nov 11, 2024 11:43 pm

I will definitely look into it, but at the moment I dont understand how it works and how it could possibly add failover to a mesh topology? i dont have any other vpn service or second ISP with enough bandwidth to handle alternative routes

Got it. Just want to add that OSPF isn’t really tied to other VPN services or multiple ISPs wan links.

It’s basically a “smart” tool (deamon) that runs on each router. You feed it with some basic info about your WireGuard links, and it takes care of adding routes to the routing table for you, ie no need to do it by hand. All the OSPF deamons talk to each other, sharing updates on how the WireGuard links are doing to always keep the best routes between all your nodes. If a WireGuard link breaks, OSPF will instantly update all the routing tables to keep things running smoothly, even if traffic has to take a detour around the broken link. Simple as that.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Tue Nov 12, 2024 4:55 am

I do have a question .
Do you use same receiving port on each router???

What I would do is use a different one on each router.

Router A - 111111 / Router B - 22222 / and so ON.

SO on R1

I would have input chain rule for upd port 11111
I would have listening port set at 13231 on wg settings
I would have allowed IPs of
r2client - endpoint port 22222
r3client - endpoint port 33333
etc..

SO on R2
I would have input chain rule for udp port 22222
I would have listening port set at 13232 on wg settings
I would have allowed IPs of
r1client - endpoint port 11111
r3client - endpoint port 33333
etc..

Rinse repeat, so that each router has a input chain rule for the incoming wireguard handshake and a different port one goes out with as source for initial handshake (misnamed listening port IMHO) ( the 13231 etcl...)

See if that helps with the speed issue.
Dont forget to add relay rule to Router 5
 
Usbuild
just joined
Topic Author
Posts: 20
Joined: Fri Jun 14, 2024 3:40 pm

Re: WireGuard site to site routing help

Tue Nov 12, 2024 5:39 am

@anav I disabled wireguard on all of them rebooted all routers and enabled wireguard. I can access any host instantly and wasn’t able to replicate the problem since.
When issue existed I timed delay and varied between 30 seconds and 2:50. Never more than 3 minutes.
I’ll be adding 3 more sites this week, so you’ll hear more from me.

I like your idea of assigning dedicated ports for each site. It should be easier to troubleshoot later.

Not sure if I need a relay rule in the case of mesh topology I’m not routing outside of wireguard interface or to the wan all connections are peer to peer. Unless I’m misunderstanding what relay does.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 21783
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WireGuard site to site routing help

Tue Nov 12, 2024 5:59 am


Not sure if I need a relay rule in the case of mesh topology I’m not routing outside of wireguard interface or to the wan all connections are peer to peer. Unless I’m misunderstanding what relay does.
Although a MESH, all connections are strictly peer to peer.
Each hop needs to be considered.
Take you as a laptop warrior at your coffee shop.
You wanted to take a look at a subnet on R1, so you selected your wireguard profile on your laptop for R1 and connected.
Then you browsed to a device on a subnet on R1.

Then you realized you wanted to look at the configuration of R3 , no need to change wireguard connection!!
So you open up winbox and type wireguardIPR3:endpoint port R3
Or quite reasonably you could type the gatewayIP of a sbunet on R3 gatewayIP:endpoint port R3

But you are in R1 not R3.
To reach R3, now there would have to be a firewall rule to allow your Road warrior wireguard address to reach R3 ( the router knows where to send the traffic due to our setup but you dont have permission)

Hence relay rule, come in on the tunnel but want to go back out into the tunnel, our relay rule does that BOOM!
No need for three or four or five separate allow rules in the forward chain from your laptop address to each wireguard peer.......

I mean you could make the same relay rule differently
instead of the standard on each router
add action=accept chain=forward in-interface=wireguard out-interface=wireguard
YOU COULD USE
add action=accept chain=forward src-address=WGAddressofLaptop out-interface=wireguard

however lets say you have a wireguard address for your smartphone, or ipad, etc, so thats another address,
now you could do it with a source address list vice a single address.
add action=accept chain=forward src-address-list=MyDevices out-interface=wireguard
BUT, simple works and dont need to keep track..........of extra lists......

Who is online

Users browsing this forum: No registered users and 20 guests