Dual wan setup troubles

zuku · Sun Aug 27, 2017 6:05 pm

Hi,
I'm trying to add to mikrotik second WAN, because ISP gave me on this WAN link another subnet block of 16 Public IP addresses, these IP should forward for my internal services on LAN, but now I'm stuck with problem even accessing my mikrotik through this second WAN, if I figure this out then will step to port forwarding on this WAN. This is my setup:
LAN - sfp-sfpplus1
DMZ - ETH5
WAN1 - sfp - x.x.x.58/29
WAN2 - ETH8 x.x.x.246/30, routable IP block on this link is x.x.x.145/28
ROS ver 6.37.5

so I've configure already masquerade:

add action=masquerade chain=srcnat out-interface=sfp1
add action=masquerade chain=srcnat out-interface=ether8

mangle:

add action=mark-connection chain=prerouting comment="WAN1 IN -- > WAN1 OUT" in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-connection chain=prerouting comment="WAN2 IN -- > WAN2 OUT" in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN2 IN -- > WAN2 OUT" connection-mark=WAN2_pfw in-interface-list=Lan+ETH5 new-routing-mark=\
    to_WAN2 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN1 IN -- > WAN1 OUT" connection-mark=WAN1_pfw in-interface-list=Lan+ETH5 new-routing-mark=\
    to_WAN1 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=no

and routing:

add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=to_WAN2
add comment=WAN1 distance=1 gateway=x.x.x.57
add comment=WAN2 distance=2 gateway=x.x.x.245

But after that, I can ping only WAN1 from outside, pings to WAN2 x.x.x.246 didn't respond - so what's wrong with my setup?

My second question how should I configure this IP addresses block, should I simply add this IP block x.x.x.145/28 as secondary address to WAN2?

pukkita · Sun Aug 27, 2017 6:30 pm

Have a look at Tomas Kirnak's Load Balance / Mangle Deep Dive presentation.

Your problem comes from the fact you're not marking on input chain.

zuku · Mon Aug 28, 2017 9:11 am

I've changed my setup with this provided on Tomas Kirnak presentation:

add action=mark-connection chain=input comment="WAN1 IN -- > WAN1 OUT" in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-connection chain=input comment="WAN2 IN -- > WAN2 OUT" in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN2 IN -- > WAN2 OUT" connection-mark=WAN2_pfw in-interface=sfp-sfpplus1 \
    new-routing-mark=to_WAN2 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN1 IN -- > WAN1 OUT" connection-mark=WAN1_pfw in-interface=sfp-sfpplus1 \
    new-routing-mark=to_WAN1 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=no

but still can't ping WAN2 x.x.x.246 from outside, I get request timeout, and on mikrotik firewall connections I don't have any connection marks with "WAN2"

mangle on WAN2 are the same like on WAN1 but counters on WAN2 mangle rules do not increase at all, doesn't this look strange?

zuku · Tue Aug 29, 2017 6:21 pm

could anyone help me with that?

pukkita · Tue Aug 29, 2017 7:37 pm

Follow the presentation mangle to the letter until you get a solid grasp on it, before jumping into customizations.

Order, and following it to the utmost detail is really important.

zuku · Tue Aug 29, 2017 10:20 pm

I'm done with tutorial I can't add this load balancing part:

add chain=prerouting connection-mark=LAN->WAN src-address-list=LAN
action=mark-routing new-routing-mark=to_WAN1
comment="Load-Balancing here"

because then all my ovpn tunnels disconnecting and I loose connectivity with my remote ovpn networks.

My mangle looks now:

/ip firewall mangle
add action=accept chain=prerouting dst-address-list=Connected src-address-list=Connected
add action=mark-connection chain=input comment="WAN - > ROS" connection-mark=no-mark in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-connection chain=forward comment="WAN -> LAN" connection-mark=no-mark connection-state="" in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=no
add action=mark-connection chain=forward connection-mark=no-mark connection-state="" in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN1_pfw new-routing-mark=to_WAN1 passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=WAN2_pfw new-routing-mark=to_WAN2 passthrough=no src-address-list=LAN
add action=mark-connection chain=prerouting comment="LAN -> WAN" connection-mark=no-mark dst-address-list=!Connected dst-address-type=!local new-connection-mark=LAN->WAN passthrough=no \
    src-address-list=LAN
add action=mark-connection chain=prerouting comment="Sticky connections" connection-mark=LAN->WAN new-connection-mark=Sticky_WAN1 passthrough=no routing-mark=to_WAN1
add action=mark-connection chain=prerouting connection-mark=LAN->WAN new-connection-mark=Sticky_WAN2 passthrough=no routing-mark=to_WAN2
add action=mark-routing chain=prerouting connection-mark=Sticky_WAN1 new-routing-mark=to_WAN1 passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=Sticky_WAN2 new-routing-mark=to_WAN2 passthrough=no src-address-list=LAN

when pinging mikrotik WAN2 still reguest timeout, and any connection mark with "WAN2" doesn't show in firewall connections.

Sob · Fri Sep 01, 2017 4:49 am

I didn't study everything in detail, but if alternative routing tables give you problems with reaching some destinations, then either don't mark routing for those destinations, or make sure that routes to them are always looked up in right routing table, e.g.:

/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main

zuku · Fri Sep 01, 2017 9:02 am

I giving up with this, because even this first rule:

add action=mark-connection chain=input connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2->ROS passthrough=no

should show on firewall connections connection mark "WAN2->ROS" when I ping mikrotik to WAN2 from outside, but it didn't get any "WAN2->ROS" mark, so is clear that next all my mangles do not work.
I've even upgraded mikrotik to latest bugfixes ROS but still the same.

Sob · Sun Sep 03, 2017 5:35 am

I had a better look at whole thread, and even your initial config should have worked. Are you sure that you for example don't block stuff from second WAN in /ip firewall filter?

It also means that I have to disagree with pukkita's original suggestion (sorry

). You don't need to mark connections in input, doing so in prerouting is fine. And you don't need to mark them in forward either. Just prerouting alone is enough (unless you want to have different marks for connections to and through router, for some reason). This is minimal config covering both traffic to router and through router:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=<WAN1> new-connection-mark=wan1
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=<WAN2> new-connection-mark=wan2
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> new-routing-mark=wan1
add action=mark-routing chain=prerouting connection-mark=wan2 in-interface=<LAN> new-routing-mark=wan2

Passthrough option yes or no does not matter in this simple config.

To save rules with multiple LANs, you can change last two rules (instead of duplicating them for each LAN) to:

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=wan1 new-routing-mark=wan1
add action=mark-routing chain=prerouting connection-mark=wan2 new-routing-mark=wan2

This will break routing, but it can be fixed using this:

/ip route rule
add action=lookup-only-in-table dst-address=<LAN subnet 1> table=main
add action=lookup-only-in-table dst-address=<LAN subnet 2> table=main
...

Btw, in linked presentation, the whole "sticky connections" part seems to be unnecessarily complicated. I hope I'm not missing something (it's a little late here...), but it seems to me that it only makes sense when you insist on different marks for incoming and outgoing connections (not sure why). If you don't need that, then with the above simple config, you can mark new outgoing connections with wan1 or wan2 mark and you have the same thing. Or don't mark them at all and they'll get correctly marked by first reply packet coming from WAN.

zuku · Tue Sep 05, 2017 8:48 am

OK so I have changed setup with yours, and even rebooted mikrotik:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN2 passthrough=no

I have reviewed my firewall, on top of it I have enabled ICMP no matter of what incoming interface:

add action=accept chain=input comment="enable: pings" limit=50/5s,2 protocol=icmp

But still have request timeout when I try to ping WAN2 from outside, on firewall connections nothing with connection mark "WAN2" while pinging:

Even if I disable my forward ind input drop rules at the bottom of my firewall list:

add action=drop chain=forward comment="block forward" disabled=yes
add action=drop chain=input comment="block input" disabled=yes

Now something more, I have three remote networks over ovpn tunnels, and three over ipsec tunnels, users over ipsec tunnel connecting to my server on LAN, but after enabled this your setup these ipsec remote users loose connectivity to this server, until I disable this:

add disabled=yes distance=1 gateway=X.X.X.57 routing-mark=to_WAN1

So I think is any problem with data flow throught the router, packets get stucks somewhere?

Sob · Tue Sep 05, 2017 7:12 pm

Stupid question first, the second connection definitely works (when you e.g. connect different router there), right?

Next, did you perhaps do something with rp-filter in /ip settings? Option "strict" is known to not play nice with multiple WANs.

I'm not sure what else it could be.

With tunnels, ovpn ones will probably need a routing rule for remote subnets. With IPSec it depends on what you have. If it's transport mode with some internal tunnel (ipip, ...), it should be the same as ovpn. If it's tunnel mode, I'd guess that it should just work. I never tried it with multiple WANs, but outside encrypted packets should work well with the rules. Incoming inside packets (stuff going through tunnel) will get marked with WANx mark, but as long as the tunnel itself works, it should not matter. But you can try to make an exception for them (add accept mangle rules before the others with in-interface=<wanx> ipsec-policy=in,ipsec, and you can also try to not mark routing for ipsec-policy=out,ipsec packets). But in any case, first step is to solve ping on WAN2.

zuku · Tue Sep 05, 2017 7:23 pm

Yes, link that I'm trying connect as WAN2 working for 100% I switched it to my other tp-link router and everything works there.
Now I see in IP --> setting that I have RP filter - Strict .
My IPSEC are in tunnel mode, and my OVPN tunnels have OSFP so do not have any route roules.

Sob · Tue Sep 05, 2017 9:15 pm

Then you know what to do, first change RP filter to get second WAN going, and then you'll see if tunnels need any further help.

zuku · Wed Sep 06, 2017 9:08 am

RP Filter changed to "no" or "loose"
but still can't ping WAN2 IP, any "WAN2" connections marks doesn't show in mikrotik firewall connections while pinging.

Remote IPSEC networks doesn't have access to my server, it's java based application on this server, application login page doesn't show for them but when I disable IP route with mangle "to_WAN1":

add disabled=yes distance=1 gateway=X.X.X.57 routing-mark=to_WAN1

then bum login page shows immediately.

Sob · Wed Sep 06, 2017 3:48 pm

Either use Tools->Torch on WAN2 interface to see what's going on there, or (maybe better, because you won't miss anything) add some logging rules for icmp (in prerouting, input, output and postrouting) and watch what happens with your ping packets, if they are any replies and where are they going.

The second part is strange too. If you still have routes configured like in first post:

add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=to_WAN2
add comment=WAN1 distance=1 gateway=x.x.x.57
add comment=WAN2 distance=2 gateway=x.x.x.245

then difference bettween disabled and enabled to_WAN1 route should be none. If it does something, then it means that connections (and then routing) must be getting marked, otherwise router would ignore to_WAN1 routing table. But there's no difference between this and main one, both have gateway=x.x.x.57.

You can try to not mark connections from IPSec (put it before other rules):

/ip firewall mangle
add action=accept chain=prerouting ipsec-policy=in,ipsec

Wed Sep 06, 2017 3:59 pm

I counted 14 mangle rules in export but screen shot shows more. Is it possible that FastTrack is enabled and dynamic Mangle rules are added?

dgnevans · Wed Sep 06, 2017 4:25 pm

Can you do an export of your ip route as well.

zuku · Wed Sep 06, 2017 4:53 pm

when I enable torch o WAN2 I see that something is going on here, I see my incoming connection from 83.x.x.130, but this not shows as connection mangle mark "WAN2"

additionally I see that counters on WAN2 prerouting increasing this time, earlier I didn't have that:

I have only something like this about mangle dynamic:

PatrickDunn · Wed Sep 06, 2017 4:54 pm

Worth buying cause of good support.

zuku · Wed Sep 06, 2017 5:07 pm

Can you do an export of your ip route as well.

sure:

/ip route
add check-gateway=ping distance=1 dst-address=8.8.4.4/32 gateway=x.x.x.57 routing-mark=google
add disabled=yes distance=1 gateway=x.x.x.245 routing-mark=from_WAN2
add disabled=yes distance=1 gateway=x.x.x.57 routing-mark=to_WAN1
add distance=1 gateway=x.x.x.245 routing-mark=to_WAN2
add comment=WAN1 distance=1 gateway=x.x.x.57
add comment=WAN2 distance=2 gateway=x.x.x.245
add distance=1 dst-address=10.5.0.0/24 gateway=sfp-sfpplus1
add distance=1 dst-address=10.20.0.0/22 gateway=sfp-sfpplus1
add distance=1 dst-address=192.168.0.0/24 gateway=sfp-sfpplus1
add distance=1 dst-address=192.168.1.0/24 gateway=sfp-sfpplus1

these last local subnets are remote IPSEC (tunnel) subnets - these have problem accesing my LAN server described earlier.
I have also three ovpn subnets and here is enabled OSPF so routes are dynamic.

Sob · Wed Sep 06, 2017 6:27 pm

Again, try to find out if there are any replies and where are they going to. Add this at the beginning (x.x.x.246 is your WAN2 address):

/ip firewall mangle
add action=log chain=output protocol=icmp src-address=x.x.x.246

If ping requests are not blocked by RP filter and they are accepted in input chain, then router must be trying to reply to them. If this rule doesn't catch anything, then try without src-address option (even though it shouldn't use source different address).

zuku · Wed Sep 06, 2017 8:27 pm

mangle output log with src-address as WAN2 is clear I leave it couple of minutes while pinging from outside - nothing.
If I remove src-address, then I have many ICMP replies but only from my ovpn tunels and my main WAN1 - SFP1, nothing about ether8 (WAN2)

19:16:34 firewall,info output: in:(none) out:sfp1, proto ICMP (type 8, code 0), x.x.x.58->x.x.x.57, NAT x.x.x.58->x.x.x.57, len 56 
19:16:34 firewall,info output: in:(none) out:sfp1, proto ICMP (type 8, code 0), x.x.x.58->x.x.x.57, NAT x.x.x.58->x.x.x.57, len 56 
19:16:34 firewall,info output: in:(none) out:ovpn-1, proto ICMP (type 3, code 1), 192.168.11.1->192.168.3.204, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-1, proto ICMP (type 3, code 1), 192.168.11.1->192.168.3.204, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.23, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.23, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.191, len 124 
19:16:34 firewall,info output: in:(none) out:ovpn-2, proto ICMP (type 3, code 1), 192.168.13.1->192.168.5.191, len 124

Sob · Wed Sep 06, 2017 9:54 pm

Nothing going to WAN2 is expected (you would have seen it in torch before), but I'd think that there would be replies going somewhere, even if it was for some reason wrong way.

You'll have to go step by step and check where it fails. For example, you have input rule to accept pings, so verify that it really accepts your pings to WAN2 (either enable log=yes for it, or add another rule just for the source address you're testing it from).

There must be some logical explanation.

zuku · Wed Sep 06, 2017 11:40 pm

I deleted all mangle rules, rebooted mikrotik and voila everything start working, I have ping on WAN2
this is so strange for me because everything working without any mangle and mark rules

even port forward on WAN2 to internal lan server working
so I'm asking myself if I need these rules if everything works, will see tomorrow when people come back to work.

Sob · Wed Sep 06, 2017 11:53 pm

Check your WAN1 then.

Because normally two WANs don't "just work".

dgnevans · Thu Sep 07, 2017 9:50 am

I have not been through all you mangle rules but on your routes there is an issue.
Under mangle rules you have marked traffic as to_WAN1 and to_WAN2. in your ip route

 add disabled=yes distance=1 gateway=x.x.x.245 routing-mark=from_WAN2
add disabled=yes distance=1 gateway=x.x.x.57 routing-mark=to_WAN1

the first rule does not match your mangle rules.
The main purpose of your mangle rules in this situation is to ensure your traffic goes out the same connection it comes in through. I currently have 3 wan connections. I am running Round robin DNS so my users connect to a domain name and this points them at one of my 3 public ip's. I have dstnat to servers on the inside. I also have failover on the browsing for the users on the inside. works well.

dgnevans · Thu Sep 07, 2017 11:54 am

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="Srcnat WAN1" dst-address=0.0.0.0/0 out-interface=sfp1 
add action=masquerade chain=srcnat comment="Srcnat WAN2" dst-address=0.0.0.0/0 out-interface=ether8


/ip firewall mangle
add action=mark-routing chain=prerouting comment="pfw WAN1, out WAN1" connection-mark=WAN1_pfw in-interface-list=Lan+ETH5 new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=prerouting comment="pfw WAN2, out WAN2" connection-mark=WAN2_pfw in-interface-list=Lan+ETH5 new-routing-mark=WAN2_traffic passthrough=no
add action=mark-connection chain=input comment="in WAN1, out WAN1" in-interface=sfp1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input comment="in WAN2, out WAN2" in-interface=ether8 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=forward comment="pfw WAN1, out WAN1" connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=yes
add action=mark-connection chain=forward comment="pfw WAN2, out WAN2" connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=yes
add action=mark-routing chain=output comment="in WAN1, out WAN1" connection-mark=WAN1_conn new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=output comment="in WAN2, out WAN2" connection-mark=WAN2_conn new-routing-mark=WAN2_traffic passthrough=no

/ip route
add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=WAN1_traffic
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=WAN2_traffic
add check-gateway=ping distance=1 gateway=x.x.x.57
add check-gateway=ping distance=2 gateway=x.x.x.245

zuku · Thu Sep 07, 2017 4:34 pm

so I think I can say that my problem is solved

I've done this in Sob way (short version):

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN2 passthrough=no

after yesterday mikrotik reboot everything start to working, I have now connection to two wans, I've done already port forwarding on WAN2 everything is working.
I had today morning still problem with access from remote ipsec to my lan server, but after enabled Sob suggestion:

/ip firewall mangle
add action=accept chain=prerouting ipsec-policy=in,ipsec

my problem is gone.

Thank you Sob and other guys for your great support.

zuku · Fri Sep 08, 2017 12:48 pm

life is not so simple, because new problems occurs

when I do port forwarding on my WAN2 to my lan on mikrotik it works without problems, but when I need to do port forwarding to remote OVPN network I don't get connection:

add action=dst-nat chain=dstnat dst-address=87.X.X.153 dst-port=3315 protocol=tcp to-addresses=192.168.3.38 to-ports=22

this host is accessible from mikrotik.

dgnevans · Fri Sep 08, 2017 1:26 pm

Zuku can you explain exactly what you are wanting to do with this line

add action=dst-nat chain=dstnat dst-address=87.X.X.153 dst-port=3315 protocol=tcp to-addresses=192.168.3.38 to-ports=22

is 87.X.X.153 an ip address within the range of you wan addresses. Is 192.168.3.38 on your lan?

zuku · Fri Sep 08, 2017 1:29 pm

address 87.X.X.153 is one of 16 public addresses on my newly added WAN2 on mikrotik,
every of this Public addresses will be redirected to my servers on LAN but also on remote ovpn subnets connected to my mikrotik.
So let's say:
87.X.X.151 --> 10.1.0.140 (Mikrotik-Lan) this DST_NAT working
87.X.X.153 --> 192.168.3.38 (remote ovpn subnet) DST_NAT not working
87.X.X.154 --> 192.168.5.120 (remote ovpn subnet) DST_NAT not working

from mikrotik routing to all remove ovpn networks working.

dgnevans · Fri Sep 08, 2017 1:55 pm

Confirm you have routes on both sides on the Tunnel and you can access the server from your lan currently.

zuku · Fri Sep 08, 2017 2:18 pm

I have access to every my remote network, problem is I think with mangle because I have only this for port forward:

add action=mark-routing chain=prerouting comment="PFW WAN IN -- > WAN OUT" connection-mark=WAN1 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN2 passthrough=no

so reply for port forwarding are coming from remote network 192.168.3.0 192.168.5.0 but are not market to out proper WAN - am I right?

Sob · Fri Sep 08, 2017 5:00 pm

There can be more than one problem.

You found one by yourself, if you mark routing only for packets from Lan+ETH5, then packets coming from elsewhere won't get their routing marked. You can either extend interface list to include VPNs, or do the opposite, make a list of interfaces where you don't want to mark routing and use it with in-interface-list=!<list>, so the rule will apply to all others.

Second problem is that you can't usually just forward port to remote network, because when target device's default gateway is different from tunnel, it will send replies there and it can't work. To make replies go back the same way, you must either srcnat them at your end, so it will look like the source is your end of tunnel, or you need the same kind of dual-WAN config at remote end (where second "WAN" is tunnel).

zuku · Fri Sep 08, 2017 5:39 pm

hmm before I connected WAN2 to mikrotik this link was in my linux box which do port forwarding, and with this config where 10.1.0.1 is Mikrotik LAN GW:

route add -net 192.168.3.0 netmask 255.255.255.0 gw 10.1.0.1 eth0
route add -net 192.168.5.0 netmask 255.255.255.0 gw 10.1.0.1 eth0
iptables -t nat -A PREROUTING -p tcp -d 87.x.x.153 --dport 3315 -j DNAT --to 192.168.3.38:22
iptables -t nat -A PREROUTING -p tcp -d 87.x.x.153 --dport 3317 -j DNAT --to 192.168.3.35:22
iptables -t nat -A PREROUTING -p tcp -d 87.x.x.153 --dport 3316 -j DNAT --to 192.168.5.124:22

it forwarded every port to lan and remote subnets without problems, and mikrotik configuration was the same as today.

Sob · Fri Sep 08, 2017 6:24 pm

I don't know the config of all involved devices and networks. In fact, I know only a tiny part of one device's config. It's very possible that you have something different than I thought.

What I was describing is when you have:

[optional local LAN] --- [router / VPN server] --- tunnel --- [remote router] --- [device in remote LAN]

Everything works great between routers and LANs, you can connect from anywhere to anywhere. But if you forward port from VPN server's public address to device in remote LAN, it doesn't work. Because usually the remote network connects to internet using own connection and tunnel is only for traffic to/from other LAN. When a client with address 1.2.3.4 connects to port on VPN server, which is forwarded to other LAN, this direction is ok, it goes through tunnel, reaches remote device, it sends a reply back ... but the remote router sees that packet destination is 1.2.3.4 and route to that is via its default gateway. But when it sends it there, connection fails, because client expects packet from VPN server's address. So it's important to make sure that reply packets go back via tunnel to VPN server and it forwards them to client with right (own) source address.

zuku · Fri Sep 08, 2017 7:25 pm

yes I understand how it works, but wonder how could this work earlier through my linux box which was part of my mikrotik LAN subnet, it has 10.1.0.36 address, and it's own WAN and on this WAN do port forwarding even for remote subnets

I added to my interface list "Lan+Eth5" two ovpn tunnels, but port forwarding to remote networks behind them unfortunately not working.

Sob · Fri Sep 08, 2017 7:46 pm

It could work if Linux box used srcnat for forwarded connections. That way packets would look to all other devices as coming from 10.1.0.36, and they would know where it is. But with original source address kept (e.g. my example 1.2.3.4), all other devices would send it wrong way, not a single one would know that replies should be routed back to Linux box. It can be done without NAT, but not without other rules (marking connections by source and routing replies back there).

zuku · Mon Sep 11, 2017 11:40 am

but on this second WAN used for port forwarding this block of 16 Public IP are used only for forwarding nothing more so I don't use NAT (masquerade) on these addresses.
So should I use NETMAP or something to do 1:1 translation ?

Sob · Wed Sep 13, 2017 4:08 am

It depends on direction. If you use those addresses only for forwarding ports elsewhere, you don't need srcnat to internet, because you won't have outgoing connections and replies to incoming one will be handled automatically. But you might need srcnat for forwarded ports, i.e. from internet to internal addresses. Because unless those .3.x and .5.x devices have tunnel as their default gateway, they would be sending replies the wrong way.

I'm not sure if we understand each other completely, so try a quick experiment (masquerade connections to remote networks when they have some mark, i.e. came from one of WANs):

/ip firewall nat
add action=masquerade chain=srcnat connection-mark=!no-mark dst-address=192.168.3.0/24
add action=masquerade chain=srcnat connection-mark=!no-mark dst-address=192.168.5.0/24

omersiar · Fri Apr 20, 2018 4:20 pm

/ip settings
set accept-source-route=yes allow-fast-path=no route-cache=no rp-filter=loose tcp-syncookies=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="Srcnat WAN1" dst-address=0.0.0.0/0 out-interface=sfp1 
add action=masquerade chain=srcnat comment="Srcnat WAN2" dst-address=0.0.0.0/0 out-interface=ether8


/ip firewall mangle
add action=mark-routing chain=prerouting comment="pfw WAN1, out WAN1" connection-mark=WAN1_pfw in-interface-list=Lan+ETH5 new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=prerouting comment="pfw WAN2, out WAN2" connection-mark=WAN2_pfw in-interface-list=Lan+ETH5 new-routing-mark=WAN2_traffic passthrough=no
add action=mark-connection chain=input comment="in WAN1, out WAN1" in-interface=sfp1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input comment="in WAN2, out WAN2" in-interface=ether8 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=forward comment="pfw WAN1, out WAN1" connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=yes
add action=mark-connection chain=forward comment="pfw WAN2, out WAN2" connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=yes
add action=mark-routing chain=output comment="in WAN1, out WAN1" connection-mark=WAN1_conn new-routing-mark=WAN1_traffic passthrough=no
add action=mark-routing chain=output comment="in WAN2, out WAN2" connection-mark=WAN2_conn new-routing-mark=WAN2_traffic passthrough=no

/ip route
add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=WAN1_traffic
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=WAN2_traffic
add check-gateway=ping distance=1 gateway=x.x.x.57
add check-gateway=ping distance=2 gateway=x.x.x.245

Hello @dgnevans

Is this a working example?

Who is online