Routing outgoing SMTP from server beind MikroTik through two Wireguard tunnels

Hello,

this might be a noob question, because I am relatively inexperienced with MikroTik, so please have mercy with me :slight_smile:

I have a setup consisting of two small edge VPS Debian machines, a MikroTik RB960PGS and a big Debian server behind that.
The MikroTik Router is connected to the edge servers via Wireguard, the MikroTik Router is the Wireguard client.
The MikroTik Router is connected to the Internet via a bad ugly plastic router I got from my ISP, this device can not be used for anything else in this scenario.
On the edge servers I have HAProxy running, that forwards specific traffic to the big server via the Wireguard tunnels. Incoming traffic is solved for me in that regards.

But I want to run a full mail server on the big server, that was previously hosted on one edge server.
My problem with that is, that the SPF record of my mail domain should not change every time my ISP gives me a new IP address.
I want the fixed IP addresses of my edge servers in the SPF record. So my outgoing SMTP packages must be routed through the MikroTik Wireguard tunnels somehow.

I looked into several posts in this awesome forum, I tried to bundle my wg1 and wg2 interface into a routable interface and set a route on them.
Maybe it is something that I have to do on the edge server, but I have doubts about that. The MikroTik router should IMHO be the routing element here.
I did not manage to get it done properly, there where no outgoing packages traceable in all the tests I did.

Ideally I only want connections that are in the NEW state in TCP and are coming from port 25, 465 or 587 and the IP address 10.10.12.11 to be routed through one of the available Wireguard tunnels.
But it would be OK if all the traffic that comes from 10.10.12.11 would go through them.

To make stuff easier to understand I drew a picture of the setup, I hope that helps.
I also include my censored MikroTik export configuration.

[admin@MikroTik] > export hide-sensitive
# 2024-09-07 10:06:40 by RouterOS 7.15.3
# software id = JSRA-V7Q2
#
# model = RB960PGS
# serial number = REDACTED
/interface bridge
add admin-mac=48:8F:5A:90:XX:YY auto-mac=no comment=defconf name=bridge port-cost-mode=short
/interface ethernet
set [ find default-name=ether5 ] poe-priority=1
/interface wireguard
add comment=wireguard1 listen-port=37483 mtu=1420 name=wg1
add comment=wireguard2 listen-port=15568 mtu=1420 name=wg2
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add comment=ansible name=server-dhcp ranges=10.10.12.10-10.10.12.254
/ip dhcp-server
add address-pool=server-dhcp comment=ansible interface=bridge name=server-dhcp
/routing bgp template
set default disabled=no output.network=bgp-networks
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp1 internal-path-cost=10 path-cost=10
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add comment=wireguard1 interface=wg1 list=LAN
add comment=wireguard2 interface=wg2 list=LAN
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=0.0.0.0/0 comment=wireguard1 endpoint-address=444.444.444.444 endpoint-port=4199 interface=wg1 name=peer46 persistent-keepalive=25s public-key=\
    "REDACTED"
add allowed-address=0.0.0.0/0 comment=wireguard2 endpoint-address=555.555.555.555 endpoint-port=4199 interface=wg2 name=peer47 persistent-keepalive=25s public-key=\
    "REDACTED"
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=10.10.12.1/24 comment=ansible interface=bridge network=10.10.12.0
add address=10.10.10.2/24 comment=wireguard1 interface=wg1 network=10.10.10.0
add address=10.10.11.2/24 comment=wireguard2 interface=wg2 network=10.10.11.0
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server lease
add address=10.10.12.10 comment=ansible mac-address=D8:3A:DD:B6:ED:C0 server=server-dhcp
add address=10.10.12.11 comment=ansible mac-address=00:13:3B:10:A6:CC server=server-dhcp
/ip dhcp-server network
add address=10.10.12.0/24 comment=ansible dns-server=10.10.12.1 gateway=10.10.12.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=10.10.12.1 comment=ansible name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=accept chain=forward comment=internet in-interface-list=LAN out-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat comment=ansible dst-port=8001 in-interface-list=all protocol=tcp to-addresses=10.10.12.10 to-ports=443
add action=dst-nat chain=dstnat comment=ansible dst-port=4002 in-interface-list=all protocol=tcp to-addresses=10.10.12.10 to-ports=22
add action=dst-nat chain=dstnat comment=ansible dst-port=4001 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=22
add action=dst-nat chain=dstnat comment=ansible dst-port=4003 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=25
add action=dst-nat chain=dstnat comment=ansible dst-port=4004 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=110
add action=dst-nat chain=dstnat comment=ansible dst-port=4005 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=143
add action=dst-nat chain=dstnat comment=ansible dst-port=4006 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=465
add action=dst-nat chain=dstnat comment=ansible dst-port=4007 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=587
add action=dst-nat chain=dstnat comment=ansible dst-port=4008 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=993
add action=dst-nat chain=dstnat comment=ansible dst-port=4009 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=995
add action=dst-nat chain=dstnat comment=ansible dst-port=8002 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=80
add action=dst-nat chain=dstnat comment=ansible dst-port=5222 in-interface-list=all protocol=tcp to-addresses=10.10.12.11 to-ports=5222
/routing bfd configuration
add disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5
/system clock
set time-zone-name=Europe/Berlin
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

mikrotik_censored_export.txt (6.21 KB)
mikrotik_smtp.png
Thanks in advance
Martin

So I think I should use a routing table like this:

/routing table
add comment=ansible disabled=no fib name=wireguard
/ip firewall mangle
add action=mark-routing chain=prerouting comment=ansible dst-port=25 new-routing-mark=wireguard passthrough=yes protocol=tcp src-address=10.10.12.11
/ip route
add comment=ansible disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.10.11.1 routing-table=wireguard suppress-hw-offload=no

This makes all outgoing traffic on port 25 disappear into nirvana, and not reappear on the other side of the wireguard tunnel (10.10.11.2 ↔ 10.10.11.1).
I have these sysctl settings though:

net.ipv4.conf.wg0.forwarding = 1
net.ipv4.ip_forward = 1

Logging the Packages gives me:

prerouting: in:bridge out:(unknown 0), connection-state:new src-mac 00:13:3b:10:a6:cc, proto TCP (SYN), 10.10.12.11:43942->82.165.117.193:25, len 60

When I disable the route I get this second Log entry and the connection is established:

prerouting: in:bridge out:(unknown 0), connection-state:established,snat src-mac 00:13:3b:10:a6:cc, proto TCP (ACK), 10.10.12.11:60956->82.165.117.193:25, NAT (10.10.12.11:60956->192.168.178.86:60956)->82.165.117.193:25, len 52

Does anyone have any Ideas?

Best Regards
Martin

So it turned out, I was wrong about β€œThe MikroTik router should IMHO be the routing element here.”.
It was the Wireguard server all along, that was not forwarding the packets.

Setting these iptable routes on the Wireguard servers did the trick:

iptables -t nat -A POSTROUTING -s 10.10.0.0/16 -p tcp -m tcp --dport 25 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.10.0.0/16 -p tcp -m tcp --dport 465 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.10.0.0/16 -p tcp -m tcp --dport 587 -j MASQUERADE
iptables -t filter -A FORWARD -i wg0 -j ACCEPT
iptables -t filter -A FORWARD -i <eth0/ens3> -o wg0 -m conntrack --ctstate RELATED, ESTABLISHED -j ACCEPT

Obviously only in combination with these MikroTik settings:

/interface list
add comment=ansible name=wireguard
/routing table
add comment=ansible disabled=no fib name=wireguard
/interface list member
add comment=ansible interface=wg1 list=wireguard
add comment=ansible interface=wg2 list=wireguard
/ip address
add address=10.10.10.2/24 comment=wireguard1 interface=wg1 network=10.10.10.0
add address=10.10.11.2/24 comment=wireguard2 interface=wg2 network=10.10.11.0
/ip firewall filter
add action=accept chain=forward comment=ansible in-interface-list=LAN out-interface-list=wireguard routing-mark=wireguard
/ip firewall mangle
add action=mark-routing chain=prerouting comment=ansible dst-port=25,465,587 new-routing-mark=wireguard passthrough=yes protocol=tcp src-address=10.10.12.11
/ip firewall nat
add action=masquerade chain=srcnat comment=ansible ipsec-policy=out,none out-interface-list=wireguard routing-mark=wireguard
/ip route
add comment=ansible disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.10.11.1 routing-table=wireguard scope=30 suppress-hw-offload=no target-scope=10
add comment=ansible disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.10.10.1 routing-table=wireguard scope=30 suppress-hw-offload=no target-scope=10

Both routes now work … separately. If both are active at the same time, only the topmost route is used all the time.

So currently I can manually decide which of my Wireguard tunnels I use for outgoing SMTP traffic. That was my minimal goal.

If anyone of you know how to harden this setup against one of the Wireguard tunnels being down, that would make my day.

Cheers
Martin

Addendum: If you do this you should also set this:

iptables -t nat -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380

Without this TLS Handshakes with Mailservers may fail a lot, because the MTU of the Wireguard Interface is 1420 not 1500.