Dual WAN srcnat and dst-nat setup issue

RouterOS General
1

HI all,

Hoping you can help with what I thought would be more straightforward :slight_smile:

Context: Some networking experience. I have Comcast business (fixed IP) and am switching over to Xfinity (dynamic IP) consumer which is cheaper. I run my own mail server. My old linux firewall just died and I bought a RB5009 as replacement. I’m trying to get the basics set up with my mail server still working via comcast WAN and everything else going through the Xfinity side. I’ll post my current config down below but first what I’m trying to do will be described.

Interfaces:

  • ether1 = WAN = XFinity DHCP
  • ether7 = WANT = Comcast Fixed IP
  • ether5 = Internal LAN (on bridge)

Goals:

  1. Internet connections to ether7 (comcast) ip port 25 go to my internal mail server 192.168.4.12
  2. Outgoing connections from my mail server that aren’t destined for my local LAN, go out the ether7 (comcast) via srcnat
  3. Outgoing connections from everything else srcnat out the ether1 (Xfinity) interface

Current state:
I reset the router without applying the default script, and then manually added stuff from the default script one line at a time so I could understand what was being done and customize it. In my config post below, you’ll see some defconf comments even though I didn’t run the script, due to my copy / pasting.

I’ve put logging on all the firewall drop rules so I can see any drops, and also for connection attempts coming in for my mail server.
Web browsing works fine and gets src-nat’d out the xfinity ether1 interface.
Plex dst-nat via the xfinity ether1 works fine, checked with Plex app on my phone.

External incoming connections to my mail server don’t work, even though I’ve set up basic dst-nat. Firewall logs even show the incoming connections.

Since dst-nat works via xfinity ether1 (default route), I suspect I’ve got some routing problem for the Comcast dst-nat config. I’ve read a zillion pages on routing marks, mangling, routing tables, etc. and have become so confused as to what to set, how to make sure it stays fasttrack etc. that I’m completely lost. I’ve review the diagram in the docs a couple of times but still haven’t figured this out.

If someone can help me get the above fixed, then I’m hoping that knowledge will also help me solve goal 2.

Firewall log snippet for external connections to my mail server show that traffic is incoming and hitting my public IP, but beyond that the dst-nat seems to go to never never land and I can’t figure out how to log/debug that part:

2024-06-07T13:29:46.566730+00:00 _gateway firewall,info MWP-SMTP input: in:ether7.comcast out:(unknown 0), connection-state:new src-mac 68:ee:96:e1:e0:29, proto TCP (SYN), 149.72.35.160:64254->x.x.x.29:25, len 60
2024-06-07T13:29:48.076695+00:00 _gateway firewall,info MWP-SMTP input: in:ether7.comcast out:(unknown 0), connection-state:new src-mac 68:ee:96:e1:e0:29, proto TCP (SYN), 216.147.165.49:29892->x.x.x.29:25, len 60
2024-06-07T13:29:49.036669+00:00 _gateway firewall,info MWP-SMTP input: in:ether7.comcast out:(unknown 0), connection-state:new src-mac 68:ee:96:e1:e0:29, proto TCP (SYN), 66.231.93.60:49017->x.x.x.29:25, len 60

Config:
Note that I’ve hidden the public IP and plex ports, but the internal net numbers are real because I’ll be changing all that very soon and using VLANs and such. Just the basics for now :smiley:

/export terse
# jan/03/1970 19:35:52 by RouterOS 7.8
# software id = JW6F-K94A
#
# model = RB5009UG+S+
/interface bridge add name=mainbridge
/interface ethernet set [ find default-name=ether1 ] comment="New XFinity Modem" name=ether1.xfinity
/interface ethernet set [ find default-name=ether5 ] comment="Internal network" name=ether5.internal
/interface ethernet set [ find default-name=ether7 ] comment="Old Comcast Internet" name=ether7.comcast
/interface ethernet set [ find default-name=ether8 ] comment="Leave ether8 open for direct wire"
/interface ethernet set [ find default-name=sfp-sfpplus1 ] comment="New network trunk" name=sfp-sfpplus1.newnet
/interface list add name=WANlist
/interface list add name=LANlist
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/ip pool add name=dot4pool ranges=192.168.4.100-192.168.4.250
/ip dhcp-server add address-pool=dot4pool interface=mainbridge name=dot4dhcpsrvr
/interface bridge port add bridge=mainbridge interface=ether2
/interface bridge port add bridge=mainbridge interface=ether3
/interface bridge port add bridge=mainbridge interface=ether4
/interface bridge port add bridge=mainbridge interface=ether5.internal
/interface bridge port add bridge=mainbridge interface=ether6
/interface bridge port add bridge=mainbridge interface=sfp-sfpplus1.newnet
/ip neighbor discovery-settings set discover-interface-list=LANlist
/interface list member add comment="New XFinity" interface=ether1.xfinity list=WANlist
/interface list member add comment="Old Comcast" interface=ether7.comcast list=WANlist
/interface list member add interface=mainbridge list=LANlist
/interface list member add interface=ether8 list=LANlist
/ip address add address=192.168.4.254/24 interface=ether5.internal network=192.168.4.0
/ip address add address=x.x.x.x/30 interface=ether7.comcast network=x.x.x.28
/ip dhcp-client add comment="XFinity Finally!" interface=ether1.xfinity use-peer-dns=no use-peer-ntp=no
/ip dhcp-server network add address=192.168.4.0/24 dns-server=192.168.4.12,192.168.4.11 gateway=192.168.4.254 ntp-server=192.168.4.12,192.168.4.18,192.168.4.11
/ip firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=Drop2
/ip firewall filter add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
/ip firewall filter add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
/ip firewall filter add action=accept chain=input comment="External Plex Services" connection-state=new dst-port=31111 in-interface-list=WANlist log=yes log-prefix=MWPR
EFIX protocol=tcp
/ip firewall filter add action=accept chain=input comment="External SMTP Access" connection-state=new dst-port=25 in-interface=ether7.comcast log=yes log-prefix=MWP-SMT
P protocol=tcp
/ip firewall filter add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LANlist log=yes log-prefix=Drop5
/ip firewall filter add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
/ip firewall filter add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
/ip firewall filter add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
/ip firewall filter add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=Drop10
/ip firewall filter add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=
WANlist log=yes log-prefix=Drop11
/ip firewall nat add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WANlist
/ip firewall nat add action=dst-nat chain=dst-nat comment="Plex DNAT" dst-port=31111 in-interface-list=WANlist log=yes log-prefix=MWPREFIX protocol=tcp to-addresses=192
.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dst-nat comment="SMTP DNat" dst-address=x.x.x.29 dst-port=25 in-interface=ether7.comcast protocol=tcp to-addresses=192.168.4.12
/ipv6 firewall address-list add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
/ipv6 firewall address-list add address=::1/128 comment="defconf: lo" list=bad_ipv6
/ipv6 firewall address-list add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
/ipv6 firewall address-list add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
/ipv6 firewall address-list add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
/ipv6 firewall address-list add address=100::/64 comment="defconf: discard only " list=bad_ipv6
/ipv6 firewall address-list add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
/ipv6 firewall address-list add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
/ipv6 firewall address-list add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
/ipv6 firewall filter add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LANlist
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept HIP" protocol=139
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LANlist
/system logging add action=remote topics=firewall
/system logging add action=remote topics=debug
/system logging add action=rsyslog topics=info
/tool mac-server set allowed-interface-list=LANlist
/tool mac-server mac-winbox set allowed-interface-list=LANlist

Any help will be GREATLY appreciated :slight_smile:

Thanks,

David

2

(1) Address should be assigned to the bridge NOT ether5.
(2) Whats with 192.168.4.11/12 running some sort of pi server for DNS and ntp.
Some people do this but not sure there is any added value? Certainly NTP is better done through the router anyway,
while DNS has some better affect also forcing using dstnat rules but any browser can bypass those …

(3) Input chain is for traffic to the router. The servers are on the LAN so those should be removed.
In terms of PLEX, do users access plex from the external side by two different dyndns urls??

-Why WAN list, what happens when WAN1 goes down for example.

(4) Do local users access servers and if so HOW, by lan ip address??

(5) If not using IPV6, it should be disabled and firewall rules/address lists removed.

(6) You are going to have to apply mangle rules and routing infrastructure for your requirements.

(7) I dont see any existing routes for the two WANs. WHy???
( the wan with an IP adddress, fixed, should have a manual route entered, and the other one the same if IP DHCP client, default-route=no (which is better as the manual route is easier when things get a bit complex ).

(8) For ether8 recommend OFF Bridge access and do all config from here, First two steps already done, not on bridge and part of LANlist. Add an IP address such as 192.168.55.0/30 and then plug in any computer give it an IPV4 address of 192.168.55.2 and you are in.

(9) Do you want to be able to modify the config while away from the router, aka while travelling or anytime via smartphone etc… use wireguard.


+++++++++++++++++++++++++++++++++
Answer above questions will lead to further assistance.

3

Thanks for the prompt reply Anav.

Answers

1 - Fixed the IP assignment, now on the bridge. FYI it didn’t change the result.

2 - This is my old private network that has been around a long time. I run private DNS servers inside for resolving stuff I don’t want outside. I have separate DNS in the cloud for resolving my public host names. This is also the NTP case. The ntp servers actually are set to go to the firewall to get ntp. The old firewall ran ntp server that went out over the internet. My old firewall was VERY restrictive on outbound connections as well. I’m being more lenient on outbound now just because I’m desparate to get this running and receive email again.

3 - I’m not sure what you mean here. I had to accept the plex port on the public WAN side or dst-nat didn’t work. So I also did the same for smtp even though that didn’t work. If I remove the input rule for the plex server for example, the incoming dst-nat to it fails.

Think of it this way: before my firewall died I had one firewall on public IP. I had public access to my plex server and mail server via DNAT. Now with new firewall I also want to transition to XFinity. So public plex is working via XFinity ala dst-nat. I can’t move my mail server dst-nat over to Xfinity yet so it still needs to come in through comcast.

WAN list is because I have the two public ISP services currently and don’t want to change rules when I finally shut off comcast. Also it is the way the default configuration script sets things up so I copied that.

4 - yes by LAN ip address served up by the local DNS servers. If I am away / not on the local internet those server names (e.g. mail.x.com) resolve to the public IP on my firewall which dst-nats to the backend servers.

5 - noted. I’ll disable for now until I get a better handle on that. IPV6 only seems to help with some xbox games but that is SOOO low priority.

6 - yes, this is what I’m specifically asking about. I’ve tried various mangle things but can’t get the connection. I think I also need a separate route table and use in combo with mangle but this is what I haven’t figured out hehe

7 - the current route are the ones added when I turned on the dhcp client for the xfinity side, and the one automatically added when I added the IP address for comcast, all in the main route table. I haven’t done anything else yet because I’m trying to learn what I need for the mangling etc.

/ip route print
Flags: D - DYNAMIC; A - ACTIVE; c, s, d, y - COPY
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS      GATEWAY         DISTANCE
  DAd 0.0.0.0/0        <xfinitypublicip>           1
  DAc x.x.x.28/30  ether7.comcast         0
  DAc <xfinitynet>/23  ether1.xfinity         0
  DAc 192.168.4.0/24   mainbridge             0

8 - noted and will do

9 - noted, I’m thinking about that and have played with wiredguard and zerotier separately. After I get these basics working so I can breathe easily, then I’ll do more.

Again thanks for the prompt reply. I’m hoping this makes sense. If not I’ll draw a small diagram or something :slight_smile:

Warm regards,

David

4
  • yes the address sort of creates a route but to be complete one must make a manual route as it pertains to non-local traffic.
  • so you have dyndns Urls to both IPs.

To simplify,
Will make WAn1 Xfinity the primary route so all traffic will go out that WAN without special rules.
Will ensure that any traffic hitting WAN2 but heading for the LAN goes back out WAN2 ( incoming traffic to server for example )

Mostly changes shown:

/interface bridge add name=mainbridge

/interface ethernet set [ find default-name=ether8 ] comment="Leave ether8 open for direct wire" name=Off-Bridge

/interface list member add interface=Off-Bridge list=LANlist

/ip address add address=192.168.4.254/24 interface=mainbridge network=192.168.4.0

++++++++++++++++++++++
{default rules to keep}
/ip firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=Drop2
/ip firewall filter add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
/ip firewall filter add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
{admin rules}
/ip firewall filter add action=accept chain=input comment="accept all LAN traffic" in-interface-list=LANlist
/ip firewall filter add action=drop comment="drop all else" { put this rule in last }
+++++++++++++++++++++++++++++
{default rules to keep}
/ip firewall filter add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
/ip firewall filter add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
/ip firewall filter add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes connection-mark=no-mark
/ip firewall filter add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=Drop10
{admin rules}
/ip firewall filter add action=accept chain=forward comment="internet traffic" in-interface-list=LANlist out-interface-list=WANlist
/ip firewall filter add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat
/ip firewall filter add action=drop chain=forward comment="drop all else"
+++++++++++++++++++++++++++++++++++
/ip firewall address-list
add address=DYNDNSURL-XFINITY list=MYWAN1
+++++++++++++++++++++++++++++++++++++++++++++
/ip firewall nat add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WANlist
/ip firewall nat add action=dst-nat chain=dstnat comment="Plex DNAT" dst-port=31111 dst-address-list=MYWAN1 log=yes log-prefix=XF-PREFIX protocol=tcp to-addresses=192
.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dstnat comment="Plex DNAT" dst-port=31112 dst-address=x.x.x.29 log=yes log-prefix=COM-PREFIX protocol=tcp to-addresses=192
.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dstnat comment="SMTP DNat" dst-address=x.x.x.29 dst-port=25 protocol=tcp to-addresses=192.168.4.12

/routing table
add fib name=useWAN2

/ip firewall mangle
{ external traffic heading to server via WAN2 }
add chain=forward action=mark-connection connection-mark=no-mark in-interface=ether7.comcast
new-connection-mark=fromWAN2 passthrough=yes
add chain=prerouting action=mark-route connection-mark=fromWAN2
new-routing-mark=useWAN2 passthrough=no
{ to ensure you can ping and get a response from WAN2 }
add chain=input action=mark-connection connection-mark=no-mark in-interface=ether7.comcast
new-connection mark=viaWAN2 passthrough=yes
add chain=output action=mark-route connection-mark=viaWAN2
new-routing-mark=useWAN2 passthrough=no
/ip route
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=gatewayIP-Xfinity routing-table=main
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=gatewayIP-Comcast routing-table=main
add dst-address=0.0.0.0/0 gateway=gatewayIP-Comcast routing-table=useWAN2

/tool mac-server set allowed-interface-list=NONE
/tool mac-server mac-winbox set allowed-interface-list=LANlist

Summary.
All users will go out Xfinity.
All traffic coming in on WAN2 ( the email traffic ) will reach the server on the LAN and return traffic will go out the same WAN2.
Assuming there is no traffic ORIGINATING on the server heading out to the internet ????
PLEX is accessible on both WANS all the time. You will note that I used a different entry port argument just to make it clear for WAN2.
Chain for destination nat is dstnat ( action is dst-nat )

5

First off, I can’t thank you enough for the time you’ve spent helping me. You rock so let me know where to send the free drinks :sunglasses:

I’ve tried your config above and also tried different tweaks but dnat to the mail server (.12) via the comcast interface still isn’t working.
I can still hit the plex server just fine, I don’t need it on both interfaces because that requires client reconfig but thanks for the thought.
Note I’m using RouterOS 7.8 in case that wasn’t clear.

As to your question about traffic going OUT from internal via the Comcast interface, for now I do need to have smtp from the internal mail server (.12) go out via srcnat from the comcast interface so my SPF records work and mail doesn’t get bounced. I’ll migrate to a new approach on the xfinity later but for now I just have to get this going because we’ve been without mail for 4 days.

I’ll post my current config below in case you want to take a look to see if I missed anything. Again thanks for all the help and have a great weekend. I might try building a virtual CHR environment tomorrow morning and play so I don’t accidentally kill my Internet access lol.

Here’s my current config after your suggestions:

# jun/07/2024 22:40:15 by RouterOS 7.8
# model = RB5009UG+S+

/interface bridge add name=mainbridge
/interface ethernet set [ find default-name=ether1 ] comment="New XFinity Modem" name=ether1.xfinity
/interface ethernet set [ find default-name=ether5 ] comment="Internal network" name=ether5.internal
/interface ethernet set [ find default-name=ether7 ] comment="Old Comcast Internet" name=ether7.comcast
/interface ethernet set [ find default-name=ether8 ] comment="Leave ether8 open for direct wire" name=ether8.offbridge
/interface ethernet set [ find default-name=sfp-sfpplus1 ] comment="New network trunk" name=sfp-sfpplus1.newnet

/interface list add name=WANlist
/interface list add name=LANlist

/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik

/routing table add fib name=useWAN2

/system logging action set 3 bsd-syslog=yes remote=192.160.4.60
/system logging action add name=rsyslog remote=192.168.4.60 target=remote

/interface bridge port add bridge=mainbridge interface=ether2
/interface bridge port add bridge=mainbridge interface=ether3
/interface bridge port add bridge=mainbridge interface=ether4
/interface bridge port add bridge=mainbridge interface=ether5.internal
/interface bridge port add bridge=mainbridge interface=ether6
/interface bridge port add bridge=mainbridge interface=sfp-sfpplus1.newnet

/ip neighbor discovery-settings set discover-interface-list=LANlist

/interface list member add comment="New XFinity" interface=ether1.xfinity list=WANlist
/interface list member add comment="Old Comcast" interface=ether7.comcast list=WANlist
/interface list member add interface=mainbridge list=LANlist
/interface list member add interface=ether8 list=LANlist

/ip address add address=192.168.4.254/24 comment="Internal Gateway" interface=mainbridge network=192.168.4.0
/ip address add address=<comcast-ip>/30 comment="Comcast Public IP" interface=ether7.comcast network=x.x.x.28

/ip dhcp-client add comment="XFinity Finally!" interface=ether1.xfinity use-peer-dns=no use-peer-ntp=no

/ip dns set servers=192.168.4.12 comment="Resolve using internal servers - private host names"

# Just hard coding the IP that came from xfinity dhcp client for now so I can test it
/ip firewall address-list add address=<xfinity-dynamic-ip> list=MYWAN1

/ip firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=input comment="defconf: drop invalid - MWD-Drop2" connection-state=invalid log=yes log-prefix=MWD-Drop2
/ip firewall filter add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
/ip firewall filter add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
/ip firewall filter add action=accept chain=input comment="Accept all LAN traffic" in-interface-list=LANlist
/ip firewall filter add action=accept chain=input comment="External Plex Services" connection-state=new dst-port=31111 in-interface=ether1.xfinity log=yes log-prefix=MWPREFIX protocol=tcp
/ip firewall filter add action=accept chain=input comment="External SMTP Access" connection-state=established,new dst-port=25 in-interface=ether7.comcast log=yes log-prefix=MWP-SMTP protocol=tcp
/ip firewall filter add action=drop chain=input comment="defconf: drop all not coming from LAN - MWD-Drop5" in-interface-list=!LANlist log=yes log-prefix=MWD-Drop5
/ip firewall filter add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
/ip firewall filter add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
/ip firewall filter add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-mark=no-mark connection-state=established,related hw-offload=yes
/ip firewall filter add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log=yes log-prefix=Drop10
/ip firewall filter add action=accept chain=forward comment="internet traffic" in-interface-list=LANlist out-interface-list=WANlist
/ip firewall filter add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat
/ip firewall filter add action=drop chain=forward comment="drop all else" log-yes log-prefix=Drop11

/ip firewall mangle add action=mark-connection chain=forward connection-mark=no-mark in-interface=ether7.comcast new-connection-mark=fromWAN2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=fromWAN2 new-routing-mark=useWAN2 passthrough=no
/ip firewall mangle add action=mark-connection chain=input connection-mark=no-mark in-interface=ether7.comcast new-connection-mark=viaWAN2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=viaWAN2 new-routing-mark=useWAN2 passthrough=no

/ip firewall nat add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WANlist
/ip firewall nat add action=dst-nat chain=dst-nat comment="Plex DNAT" dst-address-list=MYWAN1 dst-port=31111 in-interface=ether1.xfinity log=yes log-prefix=MWP-XFIN protocol=tcp to-addresses=192.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dst-nat comment="SMTP DNat" dst-address=<comcast-public-ip> dst-port=25 in-interface=ether7.comcast protocol=tcp to-addresses=192.168.4.12


/ip route add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=<xfinity-dynamic-gateway-ip> routing-table=main
/ip route add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=<comcast-gateway-ip> routing-table=main
/ip route add dst-address=0.0.0.0/0 gateway=gateway=<comcast-gateway-ip>  routing-table=useWAN2

/ipv6 settings set disable-ipv6=yes

/tool mac-server set allowed-interface-list=none
/tool mac-server mac-winbox set allowed-interface-list=LANlist

Warm regards,

David

6

I forgot to mention one other thing. I’m seeing counters increase on the input and output mangle rules when external servers are trying to deliver mail inbound, but counters are not increasing on the forward or prerouting rules. I don’t know if this matters.
Screenshot_20240608_001309.png

7

Okay so you are saying the Mail Server originates traffic outbound and it has to go out WAN2.
You didnt notice but there is no need for interface on the dstnat rule for comcast, it should be removed.

In that case lets adjust the mangle rules.

{Can be first rule, ensuring Server originated traffic goes out WAN2)
/ip firewall mangle add action=mark routing src-address=192.168.192.168.4.12 dst-port=25 protocol=tcp new-routing-mark=useWAN2 passthrough=no
/ip firewall mangle add action=mark-connection chain=forward connection-mark=no-mark in-interface=ether7.comcast new-connection-mark=fromWAN2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=fromWAN2 new-routing-mark=useWAN2 passthrough=no
/ip firewall mangle add action=mark-connection chain=input connection-mark=no-mark in-interface=ether7.comcast new-connection-mark=viaWAN2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=viaWAN2 new-routing-mark=useWAN2 passthrough=no

/ip firewall nat add action=masquerade chain=srcnat comment=“defconf: masquerade” ipsec-policy=out,none out-interface-list=WANlist
/ip firewall nat add action=dst-nat chain=dst-nat comment=“Plex DNAT” dst-address-list=MYWAN1 dst-port=31111 in-interface=ether1.xfinity log=yes log-prefix=MWP-XFIN protocol=tcp to-addresses=192.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dst-nat comment=“SMTP DNat” dst-address= dst-port=25 protocol=tcp to-addresses=192.168.4.12

8

I’ve spent another 15 hours on this and am really bummed and frustrated. Our mail has been down so long now it is bouncing sigh.

Anyway, I couldn’t get anything we tried to work so I just tried to get back to as simple as possible and move back to just using the Comcast ISP. I’ve disabled the XFinity IP and removed all of the mangle, route table, etc. rules in a desperate attempt to simplify enough that I can receive incoming SMTP through dst-nat.

Below is my new config. I’m pulling my hair out now because the dst-nat isn’t working even in this simple setup. It shows 0 packets in the dst-nat table even though I allow incoming to port 25 on my public IP.

Any ideas why this isn’t working? This is something that was trivially easy to do with my previous firewall before it died so I’ve got to think I’m doing something really stupid or missing some little check box. With this config nothing is being shown as dropped in the firewall logs.

I even remove all the static routes I had (except setting the default gateway to comcast) so I would just be using router defaults.

Thanks again for all the assistance.

Here are the routes:

/ip route print
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - COPY
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS      GATEWAY         DISTANCE
0  As 0.0.0.0/0        50.246.19.30           3
  DAc 50.246.19.28/30  ether7.comcast         0
  DAc 192.168.4.0/24   mainbridge             0




# jun/10/2024 01:10:01 by RouterOS 7.8
#
# model = RB5009UG+S+

/interface bridge
add name=mainbridge

/interface ethernet
set [ find default-name=ether1 ] comment="New XFinity Modem" disabled=yes \
    name=ether1.xfinity
set [ find default-name=ether5 ] comment="Internal network" name=\
    ether5.internal
set [ find default-name=ether7 ] comment="Old Comcast Internet" name=\
    ether7.comcast
set [ find default-name=ether8 ] comment="Leave ether8 open for direct wire" \
    name=ether8.offbridge
set [ find default-name=sfp-sfpplus1 ] comment="New network trunk" name=\
    sfp-sfpplus1.newnet
    
/interface list
add name=WANlist
add name=LANlist

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/ip pool
add name=dot4pool ranges=192.168.4.100-192.168.4.250

/ip dhcp-server
add address-pool=dot4pool interface=mainbridge name=dot4dhcpsrvr

/system logging action
set 3 bsd-syslog=yes remote=192.160.4.60
add name=rsyslog remote=192.168.4.60 target=remote

/interface bridge port
add bridge=mainbridge interface=ether2
add bridge=mainbridge interface=ether3
add bridge=mainbridge interface=ether4
add bridge=mainbridge interface=ether5.internal
add bridge=mainbridge interface=ether6
add bridge=mainbridge interface=sfp-sfpplus1.newnet

/ip neighbor discovery-settings
set discover-interface-list=LANlist

/ipv6 settings
set disable-ipv6=yes

/interface list member
add comment="New XFinity" interface=ether1.xfinity list=WANlist
add comment="Old Comcast" interface=ether7.comcast list=WANlist
add interface=mainbridge list=LANlist
add interface=ether8.offbridge list=LANlist

/ip address
add address=192.168.4.254/24 comment="Old Gateway IP" interface=mainbridge \
    network=192.168.4.0
add address=<comcast-public-ip>/30 comment="Comcast public IP." interface=\
    ether7.comcast network=<comcast-network-address>

/ip dhcp-client
add comment="XFinity Finally!" default-route-distance=3 interface=\
    ether1.xfinity use-peer-dns=no use-peer-ntp=no

/ip dhcp-server network
add address=192.168.4.0/24 dns-server=192.168.4.12,192.168.4.11 gateway=\
    192.168.4.254 ntp-server=192.168.4.12,192.168.4.18,192.168.4.11
    
/ip dns
set servers=192.168.4.12

/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid - MWD-Drop2" \
    connection-state=invalid log=yes log-prefix=MWD-Drop2
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="Accept all LAN traffic" \
    in-interface-list=LANlist
add action=accept chain=input comment="External Plex Services" \
    connection-state=new disabled=yes dst-port=30103 in-interface=\
    ether1.xfinity log=yes log-prefix=MWPREFIX protocol=tcp
add action=accept chain=input comment="SMTP From Comcast" connection-state=\
    established,related,new dst-address=<comcast-public-ip> dst-port=25 \
    in-interface=ether7.comcast log-prefix=MWPREFIX protocol=tcp
add action=drop chain=input comment=\
    "defconf: drop all not coming from LAN - MWD-Drop5" in-interface-list=\
    !LANlist log=yes log-prefix=MWD-Drop5
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid log=yes log-prefix=Drop10
add action=accept chain=forward comment="internet traffic" in-interface-list=\
    LANlist out-interface-list=WANlist
add action=drop chain=forward comment="drop all else" log=yes log-prefix=\
    Drop11
    
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WANlist
add action=dst-nat chain=dst-nat comment="SMTP DNat" dst-address=<comcast-public-ip> \
    dst-port=25 protocol=tcp to-addresses=192.168.4.12
    
/ip route
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=<comcast-gateway-ip> \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
    
/system clock
set time-zone-name=America/New_York

/system identity
set name=ProdFW1

/system logging
add action=remote topics=firewall
add action=remote topics=debug
add action=rsyslog topics=info

/system ntp client
set enabled=yes

/system ntp client servers
add address=pool.ntp.org

/tool mac-server
set allowed-interface-list=LANlist

/tool mac-server mac-winbox
set allowed-interface-list=LANlist

/tool romon
set enabled=yes id=00:00:00:xx:xx:xx

/tool sniffer
set filter-interface=all filter-ip-protocol=tcp filter-port=smtp \
    streaming-enabled=yes streaming-server=192.168.4.78
9

without looking at the config, suspect ISPs are blocking port 25.
Will look at it later today.

10

The ISP in this case is Comcast business and is definitely not blocking port 25… This is the same ISP I’ve used for years. I have a firewall input chain rule that accepts port 25 coming in to my public IP on the comcast interface. If I turn on logging I can see the connections being accepted.

Also, I have a watch on the logs grepping for anything port 25 related and don’t see anywhere the firewall is dropping anything with port 25.

11

I added some addition logging and show that TCK ACK RST (reset) is being returned from my firewall out the public IP.

Here is the log:

_gateway firewall,info MWP-Comcast-Out-Port-25 output: in:(unknown 0) out:ether7.comcast, connection-state:established proto TCP (ACK,RST), :25->167.89.105.40:40860

This seems to be a further indication that my dst-nat rule for some reason isn’t working. My mail server on 192.168.4.12 is working and accepting connections on port 25. I can telnet to .12 from my firewall and it connects. For some reason my dst-nat isn’t working.

12

You didnt follow my firewall forward chain rules. Missing KEY RULE!!

/ip firewall filter
…
…
add action=accept chain=forward comment=“internet traffic” in-interface-list=
LANlist out-interface-list=WANlist
add action=accept chain=forward comment=“port forwarding” connection-nat-state=dstnat
add action=drop chain=forward comment=“drop all else” log=yes log-prefix=Drop11

13

I must have accidentally deleted it when I was trying to clean up everything. Dangit!! :blush: Note that I had that rule in the previous version of my config I posted and it wasn’t working, but I canned it somehow when I was trying to get everything simple.

So much thanks for your help!