Dual wan setup troubles

Again, try to find out if there are any replies and where are they going to. Add this at the beginning (x.x.x.246 is your WAN2 address):

/ip firewall mangle
add action=log chain=output protocol=icmp src-address=x.x.x.246

If ping requests are not blocked by RP filter and they are accepted in input chain, then router must be trying to reply to them. If this rule doesn’t catch anything, then try without src-address option (even though it shouldn’t use source different address).

mangle output log with src-address as WAN2 is clear I leave it couple of minutes while pinging from outside - nothing.
If I remove src-address, then I have many ICMP replies but only from my ovpn tunels and my main WAN1 - SFP1, nothing about ether8 (WAN2)

19:16:34 firewall,info output: in:(none) out:sfp1, proto ICMP (type 8, code 0), x.x.x.58->x.x.x.57, NAT x.x.x.58->x.x.x.57, len 56 
19:16:34 firewall,info output: in:(none) out:sfp1, proto ICMP (type 8, code 0), x.x.x.58->x.x.x.57, NAT x.x.x.58->x.x.x.57, len 56 
19:16:34 firewall,info output: in:(none) out:ovpn-1, proto ICMP (type 3, code 1), 192.168.11.1->192.168.3.204, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-1, proto ICMP (type 3, code 1), 192.168.11.1->192.168.3.204, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.23, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.23, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.191, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.191, len 124

Nothing going to WAN2 is expected (you would have seen it in torch before), but I’d think that there would be replies going somewhere, even if it was for some reason wrong way.

You’ll have to go step by step and check where it fails. For example, you have input rule to accept pings, so verify that it really accepts your pings to WAN2 (either enable log=yes for it, or add another rule just for the source address you’re testing it from).

There must be some logical explanation.

I deleted all mangle rules, rebooted mikrotik and voila everything start working, I have ping on WAN2
this is so strange for me because everything working without any mangle and mark rules :astonished: even port forward on WAN2 to internal lan server working
so I’m asking myself if I need these rules if everything works, will see tomorrow when people come back to work.

Check your WAN1 then. :slight_smile: Because normally two WANs don’t “just work”.

I have not been through all you mangle rules but on your routes there is an issue.
Under mangle rules you have marked traffic as to_WAN1 and to_WAN2. in your ip route

 add disabled=yes distance=1 gateway=x.x.x.245 routing-mark=from_WAN2
add disabled=yes distance=1 gateway=x.x.x.57 routing-mark=to_WAN1

the first rule does not match your mangle rules.
The main purpose of your mangle rules in this situation is to ensure your traffic goes out the same connection it comes in through. I currently have 3 wan connections. I am running Round robin DNS so my users connect to a domain name and this points them at one of my 3 public ip’s. I have dstnat to servers on the inside. I also have failover on the browsing for the users on the inside. works well.

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="Srcnat WAN1" dst-address=0.0.0.0/0 out-interface=sfp1 
add action=masquerade chain=srcnat comment="Srcnat WAN2" dst-address=0.0.0.0/0 out-interface=ether8


/ip firewall mangle
add action=mark-routing chain=prerouting comment="pfw WAN1, out WAN1" connection-mark=WAN1_pfw in-interface-list=Lan+ETH5 new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=prerouting comment="pfw WAN2, out WAN2" connection-mark=WAN2_pfw in-interface-list=Lan+ETH5 new-routing-mark=WAN2_traffic passthrough=no
add action=mark-connection chain=input comment="in WAN1, out WAN1" in-interface=sfp1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input comment="in WAN2, out WAN2" in-interface=ether8 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=forward comment="pfw WAN1, out WAN1" connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=yes
add action=mark-connection chain=forward comment="pfw WAN2, out WAN2" connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=yes
add action=mark-routing chain=output comment="in WAN1, out WAN1" connection-mark=WAN1_conn new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=output comment="in WAN2, out WAN2" connection-mark=WAN2_conn new-routing-mark=WAN2_traffic passthrough=no

/ip route
add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=WAN1_traffic
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=WAN2_traffic
add check-gateway=ping distance=1 gateway=x.x.x.57
add check-gateway=ping distance=2 gateway=x.x.x.245

so I think I can say that my problem is solved :smiley: I’ve done this in Sob way (short version):

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN2 passthrough=no

after yesterday mikrotik reboot everything start to working, I have now connection to two wans, I’ve done already port forwarding on WAN2 everything is working.
I had today morning still problem with access from remote ipsec to my lan server, but after enabled Sob suggestion:

/ip firewall mangle
add action=accept chain=prerouting ipsec-policy=in,ipsec

my problem is gone.

Thank you Sob and other guys for your great support.

life is not so simple, because new problems occurs :slight_smile:
when I do port forwarding on my WAN2 to my lan on mikrotik it works without problems, but when I need to do port forwarding to remote OVPN network I don’t get connection:

add action=dst-nat chain=dstnat dst-address=87.X.X.153 dst-port=3315 protocol=tcp to-addresses=192.168.3.38 to-ports=22

this host is accessible from mikrotik.

Zuku can you explain exactly what you are wanting to do with this line

add action=dst-nat chain=dstnat dst-address=87.X.X.153 dst-port=3315 protocol=tcp to-addresses=192.168.3.38 to-ports=22

is 87.X.X.153 an ip address within the range of you wan addresses. Is 192.168.3.38 on your lan?

address 87.X.X.153 is one of 16 public addresses on my newly added WAN2 on mikrotik,
every of this Public addresses will be redirected to my servers on LAN but also on remote ovpn subnets connected to my mikrotik.
So let’s say:
87.X.X.151 → 10.1.0.140 (Mikrotik-Lan) this DST_NAT working
87.X.X.153 → 192.168.3.38 (remote ovpn subnet) DST_NAT not working
87.X.X.154 → 192.168.5.120 (remote ovpn subnet) DST_NAT not working

from mikrotik routing to all remove ovpn networks working.

Confirm you have routes on both sides on the Tunnel and you can access the server from your lan currently.

I have access to every my remote network, problem is I think with mangle because I have only this for port forward:

add action=mark-routing chain=prerouting comment="PFW WAN IN -- > WAN OUT" connection-mark=WAN1 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN2 passthrough=no

so reply for port forwarding are coming from remote network 192.168.3.0 192.168.5.0 but are not market to out proper WAN - am I right?

There can be more than one problem.

You found one by yourself, if you mark routing only for packets from Lan+ETH5, then packets coming from elsewhere won’t get their routing marked. You can either extend interface list to include VPNs, or do the opposite, make a list of interfaces where you don’t want to mark routing and use it with in-interface-list=!, so the rule will apply to all others.

Second problem is that you can’t usually just forward port to remote network, because when target device’s default gateway is different from tunnel, it will send replies there and it can’t work. To make replies go back the same way, you must either srcnat them at your end, so it will look like the source is your end of tunnel, or you need the same kind of dual-WAN config at remote end (where second “WAN” is tunnel).

hmm before I connected WAN2 to mikrotik this link was in my linux box which do port forwarding, and with this config where 10.1.0.1 is Mikrotik LAN GW:

route add -net 192.168.3.0 netmask 255.255.255.0 gw 10.1.0.1 eth0
route add -net 192.168.5.0 netmask 255.255.255.0 gw 10.1.0.1 eth0
iptables -t nat -A PREROUTING -p tcp -d 87.x.x.153 --dport 3315 -j DNAT --to 192.168.3.38:22
iptables -t nat -A PREROUTING -p tcp -d 87.x.x.153 --dport 3317 -j DNAT --to 192.168.3.35:22
iptables -t nat -A PREROUTING -p tcp -d 87.x.x.153 --dport 3316 -j DNAT --to 192.168.5.124:22

it forwarded every port to lan and remote subnets without problems, and mikrotik configuration was the same as today.

I don’t know the config of all involved devices and networks. In fact, I know only a tiny part of one device’s config. It’s very possible that you have something different than I thought.

What I was describing is when you have:

[optional local LAN] — [router / VPN server] — tunnel — [remote router] — [device in remote LAN]

Everything works great between routers and LANs, you can connect from anywhere to anywhere. But if you forward port from VPN server’s public address to device in remote LAN, it doesn’t work. Because usually the remote network connects to internet using own connection and tunnel is only for traffic to/from other LAN. When a client with address 1.2.3.4 connects to port on VPN server, which is forwarded to other LAN, this direction is ok, it goes through tunnel, reaches remote device, it sends a reply back … but the remote router sees that packet destination is 1.2.3.4 and route to that is via its default gateway. But when it sends it there, connection fails, because client expects packet from VPN server’s address. So it’s important to make sure that reply packets go back via tunnel to VPN server and it forwards them to client with right (own) source address.

yes I understand how it works, but wonder how could this work earlier through my linux box which was part of my mikrotik LAN subnet, it has 10.1.0.36 address, and it’s own WAN and on this WAN do port forwarding even for remote subnets

I added to my interface list “Lan+Eth5” two ovpn tunnels, but port forwarding to remote networks behind them unfortunately not working.

It could work if Linux box used srcnat for forwarded connections. That way packets would look to all other devices as coming from 10.1.0.36, and they would know where it is. But with original source address kept (e.g. my example 1.2.3.4), all other devices would send it wrong way, not a single one would know that replies should be routed back to Linux box. It can be done without NAT, but not without other rules (marking connections by source and routing replies back there).

but on this second WAN used for port forwarding this block of 16 Public IP are used only for forwarding nothing more so I don’t use NAT (masquerade) on these addresses.
So should I use NETMAP or something to do 1:1 translation ?

It depends on direction. If you use those addresses only for forwarding ports elsewhere, you don’t need srcnat to internet, because you won’t have outgoing connections and replies to incoming one will be handled automatically. But you might need srcnat for forwarded ports, i.e. from internet to internal addresses. Because unless those .3.x and .5.x devices have tunnel as their default gateway, they would be sending replies the wrong way.

I’m not sure if we understand each other completely, so try a quick experiment (masquerade connections to remote networks when they have some mark, i.e. came from one of WANs):

/ip firewall nat
add action=masquerade chain=srcnat connection-mark=!no-mark dst-address=192.168.3.0/24
add action=masquerade chain=srcnat connection-mark=!no-mark dst-address=192.168.5.0/24