2 WAN Port Forwarding with Multiple IP Public

SnappR · Wed Jul 25, 2018 3:41 am

hey there,
i've some issue with my setup where i use 2 WAN Port Forwarding.
so let's say the setup are like this example :

WAN1(eth1) : 11.11.11.11/27(A), 11.11.11.12/27(B), 11.11.11.13/27(C) [gw: 11.11.11.1/27]
WAN2(eth2) : 33.33.33.11/27(A), 33.33.33.12/27(B), 33.33.33.13/27(C) [gw: 33.33.33.1/27]
LAN(eth3) : 10.25.25.0/24
Server Port Forward :
IP(B) TCP port 2018 -> 10.25.25.5 port 80
IP(C) TCP port 80 -> 10.25.25.200 port 80

i'd like to be able to port forward from each WAN static IP to the same internal LAN Server,
and i've configured it like this :

Source NAT (and i'm not using masquerade)

/ip firewall nat
add chain=srcnat out-interface=WAN1 src-address=10.25.25.5 action=src-nat to-address=11.11.11.12
add chain=srcnat out-interface=WAN2 src-address=10.25.25.5 action=src-nat to-address=33.33.33.12.12
add chain=srcnat out-interface=WAN1 src-address=10.25.25.200 action=src-nat to-address=11.11.11.13
add chain=srcnat out-interface=WAN2 src-address=10.25.25.200 action=src-nat to-address=33.33.33.13

Ip Route

/ip route
add dst-address=0.0.0.0/0 gateway=11.11.11.1 distance=1
add dst-address=0.0.0.0/0 gateway=33.33.33.1 distance=2

Port Forward Rules

/ip firewall nat
add chain=dstnat action=dst-nat in-interface=WAN1 dst-address=11.11.11.12 dst-port=2018 to-address=10.25.25.5 to-ports=80
add chain=dstnat action=dst-nat in-interface=WAN2 dst-address=33.33.33.12 dst-port=2018 to-address=10.25.25.5 to-ports=80
add chain=dstnat action=dst-nat in-interface=WAN1 dst-address=11.11.11.13 dst-port=80 to-address=10.25.25.200 to-ports=80
add chain=dstnat action=dst-nat in-interface=WAN2 dst-address=33.33.33.13 dst-port=80 to-address=10.25.25.200 to-ports=80

Mark Connection

/ip firewall mangle
add chain=prerouting in-interface=WAN1 action=mark-connection new-connection-mark=WAN1_Conn
add chain=prerouting in-interface=WAN2 action=mark-connection new-connection-mark=WAN2_Conn

Mark Route

/ip firewall mangle
add chain=prerouting in-interface=LAN connection-mark=WAN1_Conn action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting in-interface=LAN connection-mark=WAN2_Conn action=mark-routing new-routing-mark=to_WAN2

Route with Mark Routing

/ip route
add dst-address=0.0.0.0/0 routing-mark=to_WAN1 gateway=11.11.11.1
add dst-address=0.0.0.0/0 routing-mark=to_WAN2 gateway=33.33.33.1

everything looks fine with WAN1, but the problem exist in WAN2.. so, if DAC Route from IP in WAN2 has pref.source of 33.33.33.11,
then i can't access to other 2 IP's of WAN2 (33.33.33.12 & .13), only pref.source IP of WAN2 could port forward and give response from outside request.

what am i missing here btw? hope anyone could help~
Thanks~

pukkita · Wed Jul 25, 2018 11:34 am

For dual wan use you need to further use mangle to:

- keep track of connections, so that what enters via one WAN, exits via the same one
- Steer/balance traffic towards the two WANs

Even if you aren't using both WANs for general internet traffic, (e.g. general traffic to Internet exits via a single one) but want to be able to forward ports selectively from either of both WANs to internal hosts, you'll need more mangle.

Have a look at this greatly detailed and explained Tomas Kirnak presentation, link to the slides PDF on the top of the comments.

This being said, I see you're forwarding ports 80, so I'll assume you need this for web servers.

If this is the case, I'd follow a different approach. I would forward public port 80 (any public IP) to a single internal http reverse proxy; this way it will be the reverse proxy which will forward internally to the proper server IP depending on domain.

Simpler config, and simpler management, with all the relevant L7 config on a single place: the reverse proxy config.

sindy · Wed Jul 25, 2018 10:29 pm

If we stay in the frame of your current setup, what you say sounds like a bug to me. Normally, the pref-src value of a route is only ever used for locally originated packets, i.e. those sent by the Mikrotik itself. It is a job of connection tracking to remember the original destination address to which the initial packet of the dst-nated connection has arrived, and to "un-dst-nat" the source address of the response packets back to that one as they leave the machine. And there is no way to fix (nor break) that manually because even if you'd note down not only the in-interface of the initial packet establishing that connection but also its dst-address into the connection mark you'd assign, and you would want to forcifully src-nat the response packets while sending them out that WAN, its simply wouldn't happen because both dst-nat and src-nat treatments are assigned to the whole connection just once, when the initial packet is handled.

Another possibility would be that you have an unlimited action=fasttrack-connection rule active in /ip firewall filter and this mistake miraculously didn't manifest itself by more harm than this (as fasttracking makes most packets of a connection skip the mangle rules, so they don't get routing-marked and take the wrong path).

SnappR · Thu Jul 26, 2018 6:05 am

@pukkita : actually, my real setup isn't limited for web servers only, i have like sftp and others too..
i've implemented http reverse proxy too but i'm not applying it to all of my web servers for some reasons~

i need to make full use of the Public IP's since from the old setup, all of the public IP's has been registered/whitelisted by partners/customers and need a lot of time if i want to re-register it.
from the old setup, each public ip has it's own router.. now i put it all together in 1 CCR, and this problem happen .-.

@sindy : i'm not using fasttrack-connection since i'm not familiar with it yet, i've read some of it about skipping some mangle etc.. i'm not ready to use it~ ^-^'
so is this really a bug? it works normally with only 2 WAN and 2 public IP.. but it don't with multiple alias, atleast for WAN2 only.. still, i need it to work fully.. >.<

is there something else i could try to make it works?

sindy · Thu Jul 26, 2018 8:48 am

You may try to capture traffic on the affected WAN interface using /tool sniffer, depending on the volume of the traffic either directly into a file on the machine and then downolad and open the file using Wireshark, or you may stream a copy of each captured packet to an IP address in the LAN subnet where the Wireshark itself will be capturing (in this case, the packets come TZSP-encapsulated). If you capture to a file, or dedicate an interface and subnet for the wireshark machine if streaming, you can capture on both the LAN interface and the WAN interface to see where the server sends the response and what the Mikrotik does with it.

Not long ago another user here had a similarly mysterious issue even without having multiple addresses on the WAN interface, and the solution turned out to be an upgrade without any particular changelog item nearly resembling what was happening there. In his case, the un-dst-nated packet simply were not there, but I admit we did not try to capture both WANs simultaneously, so maybe after the un-dst-nat, the responses took the other WAN rather than being dropped.

pukkita · Thu Jul 26, 2018 11:02 am

I see...

Agree with Sindy, this looks like a bug, your config looks fine to me with regards to being able to connect to other IPs but from the src/dst-natted ones.

ROS version? Routerboard firmware version? How do you config the IPs, directly on the WAN interfaces (no private transit or loopback?) Do the external packets towards the other IPs reach the router?

SnappR · Wed Aug 01, 2018 10:22 am

hey guys,

@pukkita the public IP is directly configured in interface, eth1 & eth2
as i said before, i could access all 3 IP's in WAN1 but only 1 IP in WAN2(IP that automatically set as pref.source in DAC route)

i still don't have time to capture traffic from the WAN with /tool sniffer, i'll update soon when i could..
anyway.. my ROS version is 6.42.3 tilegx, CCR1016~ i'll try to upgrade the ROS version to 6.42.6 later and check it again if the problem still there or not~

2 WAN Port Forwarding with Multiple IP Public

2 WAN Port Forwarding with Multiple IP Public

Re: 2 WAN Port Forwarding with Multiple IP Public

Re: 2 WAN Port Forwarding with Multiple IP Public

Re: 2 WAN Port Forwarding with Multiple IP Public

Re: 2 WAN Port Forwarding with Multiple IP Public

Re: 2 WAN Port Forwarding with Multiple IP Public

Re: 2 WAN Port Forwarding with Multiple IP Public

Who is online