Inter sites Wireguard tunels and public access to services through WG

RouterOS General
1

Hi There

Im trying to setup a CHR using an azure sponsorship (can’t actually spend money besides buying the CHR license if this works).

ISSUEs TO SOLVE:

  • one of 2 buildings has losts its public IPv4 and we cant use IPv6 yet.
  • the other building has its public ipv4, but it will eventually lose it too (And it gets losts from time to time too)

IDEA: Setup a CHR, connect both sites to it and enable:

  1. Remote access for me and other peers who would connect to it using wireguard and gain access to inside IPs for buildings 1 and 2
  2. Setup port redirection on CHR so that i can publish some public facing services FROM the CHR that would end up answred from a server either on building 1 or 2.

Addresses / networks for each site:

Building1: physical mikrotik
10.0.2.0/24: local network
10.200.2.0/24: wg site 2 site network
10.0.10.0/24: wg peers network (for computers, not other routers)
10.0.2.254: router
10.200.2.254: WG address for the non cloud inter building WG network
10.0.10.254: WG address for the peers (computers, not routers) WG network
10.0.1.1: WG address for the azure WG network




Building2: physical mikrotik
10.0.3.0/24: local network
10.0.3.254: router
10.200.2.252: WG address for the non cloud inter building WG network
10.0.1.2: WG address for the azure WG network




Azure: CHR with demo license
10.0.0.4: azure interface (placed as WAN in the interface list)
10.0.1.0/24: WG network for inter building stuff
10.200.1.0/24: WG network for peers who want to access the CHR or the other 2 buildings.
10.0.1.254: WG address for the azure WG network
10.200.1.254: WG address for the azure wg peers (computers, not routers) network.

diagrama red.png
Routes for each site (so far):

Not sure if this is optimal (and i might need to learn OSPF and/or other more advanced routing tools? never touched the “routing” submenu).

Buidling1:

DST-ADDRESS GATEWAY DISTANCE

;;; Ruta test S2S casa roberto
0 Xs 172.17.118.0/24 wg_mcc_country 1
;;; Ruta a guardia x azure
1 Xs 10.0.3.0/24 10.0.1.2 1
DAd 0.0.0.0/0 181.16.78.1 1
;;; Ruta red interfaz azure
2 As 10.0.0.0/24 10.0.1.254 1
DAc 10.0.1.0/24 wg_mcc_azure 0
DAc 10.0.2.0/24 bridge-local 0
;;; Ruta Guardia sin azure
3 As 10.0.3.0/24 10.200.2.252 1
;;; Guardia por tailscale
4 As 10.0.3.111/32 10.0.2.20 1
DAc 10.0.10.0/24 wgpeers 0
;;; Ruta azure peers
5 As 10.200.1.0/24 10.0.1.254 1
DAc 10.200.2.0/24 wg_mcc_country 0
DAc 181.16.78.0/24 ether1-internet 0


/ip route
add comment="Ruta test S2S casa roberto" disabled=yes distance=1 dst-address=172.17.118.0/24 gateway= wg_mcc_country pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Ruta a guardia x azure" disabled=yes distance=1 dst-address=10.0.3.0/24 gateway=10.0.1.2 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment="Ruta red interfaz azure" disabled=no distance=1 dst-address=10.0.0.0/24 gateway=10.0.1.254 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment="Ruta azure peers" disabled=no distance=1 dst-address=10.200.1.0/24 gateway=10.0.1.254 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Ruta Guardia sin azure" disabled=no distance=1 dst-address=10.0.3.0/24 gateway=10.200.2.252 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Guardia por tailscale" disabled=no distance=1 dst-address=10.0.3.111/32 gateway=10.0.2.20 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10



DST-ADDRESS GATEWAY DISTANCE

;;; Ruta adm por azure
0 Xs 10.0.2.0/24 10.0.1.1 1
;;; ADM por tailscale
1 Xs 10.0.2.0/24 10.0.3.111 1
DAd 0.0.0.0/0 192.168.0.1 1
;;; Ruta red interfaz azure
2 As 10.0.0.0/24 10.0.1.254 1
DAc 10.0.1.0/24 wg_mcc_azure 0
;;; Ruta ADM por wg adm
3 As 10.0.2.0/24 10.200.2.254 1
DAc 10.0.3.0/24 bridge 0
;;; Ruta azure peers
4 As 10.200.1.0/24 10.0.1.254 1
DAc 10.200.2.0/24 wg_mcc_country 0
DAc 192.168.0.0/24 ether1-internet 0
DAc 192.168.44.0/24 bridge-guardias 0


/ip route
add check-gateway=ping comment="Ruta ADM por wg adm" disabled=no distance=1 dst-address=10.0.2.0/24 gateway=10.200.2.254 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment="Ruta red interfaz azure" disabled=no distance=1 dst-address=10.0.0.0/24 gateway=10.0.1.254 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment="Ruta azure peers" disabled=no distance=1 dst-address=10.200.1.0/24 gateway=10.0.1.254 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Ruta adm por azure" disabled=yes distance=1 dst-address=10.0.2.0/24 gateway=10.0.1.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment="ADM por tailscale" disabled=yes distance=1 dst-address=10.0.2.0/24 gateway=10.0.3.111 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10



Azure:

DST-ADDRESS GATEWAY DISTANCE

DAd 0.0.0.0/0 10.0.0.1 1
DAc 10.0.0.0/24 ether1 0
DAc 10.0.1.0/24 wg_mcc_azure 0
;;; Ruta a administraciF3n
0 As 10.0.2.0/24 10.0.1.1 1
;;; Ruta a guardia
1 As 10.0.3.0/24 10.0.1.2 1
DAc 10.200.1.0/24 wg-azure-peers 0
DAd 168.63.129.16/32 10.0.0.1 1
DAd 169.254.169.254/32 10.0.0.1 1


/ip route
add check-gateway=ping comment="Ruta a administraci\F3n" disabled=no distance=1 dst-address=10.0.2.0/24 gateway=10.0.1.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Ruta a guardia" disabled=no distance=1 dst-address=10.0.3.0/24 gateway=10.0.1.2 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10

Wireguard peers config for each site:

Building1:
To building2 - Allowed address: 10.200.2.252/32, 10.0.3.0/24
To Azure - Allowed address: 10.0.0.4/32, 10.0.1.0/24, 10.200.1.0/24
Building2:
To buildin1 - Allowed address: 10.200.2.254/32, 10.0.2.0/24
To azure - Allowed address: 10.0.0.4/32, 10.0.1.0/24, 10.200.1.0/24
Azure:
To building1 - Allowed address: 10.0.1.1/32, 10.0.2.0/24
To building2 - Allowed address: 10.0.1.2/32, 10.0.3.0/24

I initially disabled the building1-building2 wireguard connection, but i found my cloud tunnel to be somewhat slow (demo license applied) bunno if it was a thing of the moment or if i will be in trouble with it in the future.
Besides that, i have NO idea how i could enable both, and have the routers use the direct building to building tunnel and only jump to the cloud one if the direct connection is not available (i thought about using routes with different distance metrics, BUT, wireguard seems incompatible with having 2 peers with the “allowed address” for a same io/ip range (if i declare 10.0.2.0/24 in 2 peers, the last one that connects is the only one that works, it seems).

Besides that, i have it working.

Now, what i failed to get working:

Redirections to expose services to the public from the cloud

Port 4091: to building1 – 10.0.2.20:4091
Port 8444: to building2 – 10.0.3.111:443
Port 6080: to building2 – 10.0.3.111:80
Port 6021: to building2 – 10.0.3.111:21

Ideally, either from the public IP of azure or building 1, i need to expose the above prots and have them redirected to a specific internal IP, and then have communication return to internet from where it was received (either azure or building2, azure for everything would be ideal).

I allowed the needed ports in the Azure Firewall and then tried to use dst-nat rules on the CHR indicating to dst-nat from those external ports to the internal ones mentioned above, and to the internal IP mentioned there.

That did not work, but i was unable to test too much due to unforseen issues.

In any case, i would love some help/pointers to how i should go to get this last step working (And also recommendations on what to learn about OSFP or other routing stuff that could help me have a better setup, and also the backup/redundant tunnel/routes thing).

2

So MT CHR in the cloud as the way to reach both routers
a. for admin for config purposes
b. to forward ports to servers behind each router
c. for subnets on R1 to reach R2 and vice versa ??

In terms of port forwarding, does the server need to see the public IP of the external user hitting the CHR, or not important.

Are all 3 MT devices, if so post all three.

3

Hi!

a: yes (already working)
b: yes (not working)
c: yes (already working, albeit, with the local network of each building mikrotik right now not using the azure tunel but the direct WG tunnel between them until i loose public IP on building 1)

I am not sure if the servers need to know the public IP or not.
The server on building1 is used only by a cloud service it uploads data too (the service asked me for a fixed IP or a hostname, and they are using the mikrotik cloud hostname right now)

The server on building2 provides a web/app service for peaple (ports 80 and 443) and FTP port for the service provider.

I will ask both service providers if they need to know the public ip of the service useres or not.

the CHR is a mikrotik cloud router hosted in azure.
Building1 is a HEX S (the 760gs or something like that was the model)
Building2 is a hap ac lite.

Did you need to know that or should i post some/all config for each one?

4

Eventually will need to see all three configs.
/export file=anynameyouwish ( minus router serial number, any public WANIP information, keys etc. )

5

Is there any command to redact that info, or must it be done manually?

Thx!

6

Open in notepad++ and manually remove or modify then post with code tags.

7

Hi hi

Im back after getting some time to test this.

it is actually going from the Azure firewall to my CHR.

I removed the “from WAN” part of the nat rules, to test if with wireguard connected, the CHR correctly redirects me from ALL of its IP addresses to the remote targets (either making the rule point to the local firewalls which then re-redirect to the local computer, or i can get directly to the computer FROM the CHR). This worked.

I did some modifications to my rules so that they would LOG if they were being hit.

Log for rule for TCP port 6080:

nat6080 dstnat: in:ether1 out:(unknown 0), connection-state:new src-mac 12:34:56:78:9a:bc, proto TCP (SYN), 181.16.124.137:31615->10.0.0.4:6080, len 52

I still can’t get the CHR > local mikrotik > local computer (or directly CHR > local computer) thing to work.

I also added logging rules on the local mikrotiks, and used torch on the internet interfaces, and also checked the counters on the NAT rules.

If i try to get to my services FROM internet, i only see some activity on the CHR, nothing on the local TIKs.

I disabled all the RAW rules since i did not set those (nor do i understand them), just for this test.

I also made an INPUT and FORWARD rule on each local TIK just in case the NAT rule was not enough for some reason (it was enough when they had public IPv4 tho).

Finally, i am attaching the exported configurations.

Thanks in advance.
grdtik.rsc (35.8 KB)
admtik.rsc (38 KB)
CHRtik.rsc (4.3 KB)

8

Why two wireguard interfaces on CHR?

9

in my head, it was cleaner to have one for “site 2 stie stuff” and another for “client peers”.

Both my local mikrotiks had them too, but i think they are not disabled (because its imposible to connect to them)

10

Well you certainly have a plethora of options.
I would recommend to use ONE interface on the CHR and on both routers.
If you want separation from router to router traffic then you can use two different subnets on the CHR ( both with same interface )

Typically remote/mobile clients ( laptops, smartphones ) need to access
a. lan devices
b. internet
c. router for config purposes ( admin only ).

Typically LAN users need to access
a. other router Lan devices
b. router for config purposes ( admin only )

Was your intent for any router to connect to the internet of the CHR??? ( or just mobile/remote users )

11

There are 2 types (and actually just 2 users, LOL) of wireguard users:

  1. ME: i need to access everything
  2. User: just needs RDP access to 1 computer

I have configured my “client peer” for both interaces to test, so i could remove the second interface. (i am not sure what i gain or loose, but since both work, i can do that).

There is NO intention of using wireguard so that remote clients browse using the azure internet, we only need (and already achieved):

  1. interconnect both local sites
  2. gain remote access to both local sites

We need but can’t get working:

  1. Publish local services to the public using the Cloud CHR public IPV4
12

Well if thats the case, you simply need to port forward on the CHR into the tunnel.
I would use one wireguard tunnel for local router users and your mobile warrior and the other tunnel for port forwarded traffic as a natural separation.

So users have two things, DYNDNS or staticIP of CHR and port entry.
So on the CHR forward chain rules you need.
add chain=forward action=accept comment=“port forwarding” connection-nat-state=dstnat

( assuming CHR ip address is 10.10.10.1/24 interface=wg-PF network=10.10.10.0)
In the NAT rules
/ip firewall nat
add chain=dstnat action=dst-nat dst-address=CHR-IP-address dst-port=remoteServerPort1 to-address=remoteServerAddress1
add chain=dstnat action=dst-nat dst-address=CHR-IP-address dst-port=remoteServerPort2 to-address=remoteServerAddress2
etc..
Now the CHR needs to know where to find the Server Address and thus you need to add this to routes.
/ip route
add dst-address=ServerIP1 gateway=wg-PF routing-table=main
add dst-address=ServerIP2 gateway=wg-PF routing-table=main

Allowed IPs have to match…and this is where it becomes clear to the router which servers is related to which client peer.
The TWO entries on the CHR for same would be
hallowed=10.10.10.2/32,ServerAddress1,ServerAddress2 interface=wg-PF publi-key=“xxxxxxx” comment=RouterA portforwarding
allowedIps=10.10.10.3/32,ServerAddress3,ServerAddress4 interface=wg-PF publi-key=“yyyyyyy” comment=RouterB portforwarding

THERE ARE TWO SOLUTIONS to approach return traffic…
a. do the servers need to see the originators IP,
in other words incoming traffic can be any IP address
OR
b. or are you happy they only see the wireguard IP of the CHR…
in other words incoming traffic need only identify the wireguard subnet!!

The differences being in option a.
one has to have allowed IPs on each router of 0.0.0.0/
and either routing rule or mangles forcing all server traffic into the tunnel.

In option B. one has to sourcenat all the traffic going into the tunnel so it gets assigned a source address of the IP of the CHR wireguard interface.
Thus allowed IPs for each router is simply 10.10.10.0/24.



The wireguard settings on either router to the CHR for this wireguard network will be
allowedIps=0.0.0.0/0 interface=wg-other endpoint=chrIPaddress endpoint-port=PortAssigned persistent-keep-alive=35s comment=“For port forwarding”

The TWO entries on the CHR for same would be
hallowed=10.10.10.2/32,ServerAddress1,ServerAddress2 interface=wg-PF publi-key=“xxxxxxx” comment=RouterA portforwarding
allowedIps=10.10.10.3/32,ServerAddress3,ServerAddress4 interface=wg-PF publi-key=“yyyyyyy” comment=RouterB portforwarding

On each router

13

Hi!

Will check/test all of the above.

I am not sure if the services we have published need to identify each source or not.

I think its not relevant, but i will have to ask/test

14

Ok, after checking, i think most if not all of that was in place, except:

  1. The forward accept (not needed because there were literally no rules in the filter section, but i will leave it there)
  2. Managing the return traffic.

Since “option B” seemed simpler, i tried this:

add chain=srcnat action=masquerade dst-port=PORT protocol=6 (tcp) Out interface=“wg_interface”

This is working for accessing one of the services from internet.

Now, on the other site, i also have 1 server, and 3 ports.

I repeated the process, but in this case its not working. (nothing happens on the site router, i do see the connection using torch like before on the could router).

I did re-check my NAT rules and if i remove the “WAN” list restriction i can go to the CHR internal address (10.0.0.4:port or the WG one) from my computer while connected to wireguard, and i get correctly redirected to the internal server, so the port forwarding works, and rotues work too (that was already tested).

Did i go the correct route in order to source nat?

If i wanted to try the “mangle way”, how should i do it?

15

Scrap that.

My masquerade rule was not working because i was usint the internet facing port, and i needed to use the internal port where the server listens (and to wich the nat rule points)

This might be solved, but in case this breaks something else, i would still love to know about the mangle option.

Thank!