Aim: an off-site LEMP server on a family member’s residential internet connection (site B) with several vhosts, some of which are publicly accessible, some only privately (from devices on LAN or WireGuard subnets).
Limitations: MikroTik hEX RB750Gr3 (site B) behind ISP-provided modem/router whose NAT that cannot be disabled. Static WAN IP. Don’t need IPv6.
Network setup:
(site A) - not terribly relevant, but explains some of the other info further down
CGNAT
192.168.1.1 (ISP modem/router on its subnet)
192.168.1.100 (MikroTik hEX RB750Gr3 (site A) on ISP modem/router subnet)
192.168.40.1 (MikroTik hEX RB750Gr3 (site A) on its subnet)
10.66.66.4 (MikroTik hEX RB750Gr3 (site A) on wg_personal WireGuard subnet)
(site B)
Static WAN IP
192.168.0.1 (ISP modem/router on its subnet)
192.168.0.100 (MikroTik hEX RB750Gr3 (site B) on ISP modem/router subnet)
192.168.140.1 (MikroTik hEX RB750Gr3 (site B) on its subnet)
192.168.140.16 (LEMP server on MikroTik hEX RB750Gr3 (site B) subnet)
10.66.66.1 (LEMP server (site B) on wg_personal WireGuard subnet - it’s also a Pi-hole)
10.66.66.5 (MikroTik hEX RB750Gr3 (site B) on wg_personal WireGuard subnet)
Two sites connected via WireGuard VPN. LEMP server at site B is also a WireGuard peer that is publicly accessible from the internet, though that is not relevant to my issue I think. Port forwarding for HTTP, HTTPS, WireGuard (two peers) is properly set up in ISP modem/router. Worked as expected before I got the MikroTik router.
Output (some edits for relevance and privacy) of [admin@mikrotik-rb750gr3-site-B] > export compact hide-sensitive
/interface bridge
add admin-mac=48:A9:8A:42:3B:2F auto-mac=no comment=defconf name=bridge
/interface wireguard
add comment="Personal WireGuard VPN" listen-port=60597 mtu=1420 name=wg_personal
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=dhcp ranges=192.168.140.2-192.168.140.254
/ip dhcp-server
add address-pool=dhcp interface=bridge lease-time=10m name=defconf
/routing table
add comment="Table for personal WireGuard setup" disabled=no fib name=wg_personal_table
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=wg_personal list=WAN
/interface wireguard peers
(some omitted)
add allowed-address=10.66.66.4/32,192.168.1.0/24,192.168.40.0/24 comment="MikroTik RouterBoard hEX RB750Gr3 (site A)" interface=wg_personal persistent-keepalive=15s public-key="p9//SRqNOrB3FuXRODeoNbBrJzN72QDofwmTCmD0xCg="
add allowed-address=10.66.66.1/32 comment="HP ProLiant ML310e Gen8 v2 (eno1)" endpoint-address=192.168.140.16 endpoint-port=60590 interface=wg_personal persistent-keepalive=15s public-key="hQ/vQbvSrcdBMIZxgzeqOeK9OGF7bVE8Fz/iaMJAdmY="
/ip address
add address=192.168.140.1/24 comment=defconf interface=bridge network=192.168.140.0
add address=10.66.66.5/24 comment="wg_personal address" interface=wg_personal network=10.66.66.0
/ip dhcp-client
add comment=defconf interface=ether1 use-peer-dns=no
/ip dhcp-server lease
(Omitted. Everything gets a static lease on the site B subnet.)
/ip dhcp-server network
add address=192.168.140.0/24 comment=defconf dns-server=192.168.140.1 gateway=192.168.140.1 netmask=24
/ip dns
set allow-remote-requests=yes cache-max-ttl=1d servers=10.66.66.1
/ip dns static
(some omitted)
add address=10.66.66.4 name=mikrotik-rb750gr3-site-A-wg.example.com
add address=10.66.66.5 name=mikrotik-rb750gr3-site-B-wg.example.com
add address=192.168.40.1 name=mikrotik-rb750gr3-site-A-ethernet.example.com
add address=192.168.140.1 name=mikrotik-rb750gr3-site-B-ethernet.example.com
add address=192.168.140.16 name=hp-proliant-ml310e-gen8-v2-eno1.example.com
add cname=hp-proliant-ml310e-gen8-v2-eno1.example.com name=hp-proliant-ml310e-gen8-v2.example.com type=CNAME
add cname=hp-proliant-ml310e-gen8-v2.example.com name=freshrss.example.com type=CNAME
add cname=mikrotik-rb750gr3-site-A-ethernet.example.com name=mikrotik-rb750gr3-site-A.example.com type=CNAME
add cname=mikrotik-rb750gr3-site-B-ethernet.example.com name=mikrotik-rb750gr3-site-B.example.com type=CNAME
/ip firewall address-list
add address=10.66.66.0/24 comment="wg_personal subnet" list=private_ip
add address=(site B static WAN IP) comment="site B static WAN IP" list=private_ip
add address=192.168.0.0/16 comment="Class C private networks" list=private_ip
add address=192.168.140.16 comment="HP ProLiant ML310e Gen8 v2 (eno1)" disabled=yes list=via_wg_personal
add address=192.168.140.17 comment="HP ProLiant ML310e Gen8 v2 (eno2)" disabled=yes list=via_wg_personal
add address=192.168.140.16 comment="HP ProLiant ML310e Gen8 v2 (eno1)" list=dot_clients
add address=192.168.140.17 comment="HP ProLiant ML310e Gen8 v2 (eno2)" list=dot_clients
/ip firewall filter
add action=accept chain=input comment="allow WireGuard" dst-port=60597 protocol=udp
add action=accept chain=input comment="allow incoming traffic from private IPs" src-address-list=private_ip
add action=accept chain=forward comment="allow forwarding traffic from private IPs" src-address-list=private_ip
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related disabled=yes hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting dst-address-list=private_ip in-interface-list=LAN new-connection-mark=noVPN passthrough=yes
add action=accept chain=prerouting connection-mark=noVPN
add action=mark-routing chain=prerouting connection-nat-state=!dstnat in-interface-list=LAN new-routing-mark=wg_personal_table passthrough=no src-address-list=via_wg_personal
add action=mark-routing chain=prerouting disabled=yes dst-port=!853 new-routing-mark=wg_personal_table passthrough=no protocol=tcp src-address-list=dot_clients
add action=mark-routing chain=prerouting disabled=yes new-routing-mark=wg_personal_table passthrough=yes protocol=udp src-address-list=dot_clients
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=ether1
add action=masquerade chain=srcnat disabled=yes out-interface=wg_personal
add action=dst-nat chain=dstnat dst-port=80 in-interface-list=WAN protocol=tcp to-addresses=192.168.140.16 to-ports=80
add action=dst-nat chain=dstnat dst-port=80 in-interface-list=WAN protocol=udp to-addresses=192.168.140.16 to-ports=80
add action=dst-nat chain=dstnat dst-port=443 in-interface-list=WAN protocol=tcp to-addresses=192.168.140.16 to-ports=443
add action=dst-nat chain=dstnat dst-port=443 in-interface-list=WAN protocol=udp to-addresses=192.168.140.16 to-ports=443
add action=dst-nat chain=dstnat dst-port=60590 in-interface-list=WAN protocol=udp to-addresses=192.168.140.16 to-ports=60590
/ip route
add disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=wg_personal pref-src="" routing-table=wg_personal_table scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=192.168.1.0/24 gateway=wg_personal pref-src="" routing-table=wg_personal_table scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=192.168.40.0/24 gateway=wg_personal pref-src="" routing-table=wg_personal_table scope=30 suppress-hw-offload=no target-scope=10
/routing rule
add action=lookup disabled=no dst-address=10.66.66.0/24 src-address=192.168.140.0/24 table=wg_personal_table
add action=lookup disabled=no dst-address=192.168.1.0/24 src-address=192.168.140.0/24 table=wg_personal_table
add action=lookup disabled=no dst-address=192.168.40.0/24 src-address=192.168.140.0/24 table=wg_personal_table
add action=lookup-only-in-table disabled=no routing-mark=wg_personal_table table=wg_personal_table
/system identity
set name=mikrotik-rb750gr3-site-B
Nature of the problem: I don’t understand the basics of firewalling NAT. Vhosts for which NGINX is listening only on eno1 interface (ex. freshrss.example.com) are accessible, but those for who NGINX is listening on all interfaces (ex. www.example.com) are responded to by one of those for which NGINX is only listening on the eno1 interface. Unless I enable only vhosts for which NGINX is listening on all interfaces, then everything works as expected.
Requests: A high-level explanation of how I would accomplish what I want to accomplish. Frank criticism of the most questionable parts of my setup.