Dual wan setup troubles

Hi,
I’m trying to add to mikrotik second WAN, because ISP gave me on this WAN link another subnet block of 16 Public IP addresses, these IP should forward for my internal services on LAN, but now I’m stuck with problem even accessing my mikrotik through this second WAN, if I figure this out then will step to port forwarding on this WAN. This is my setup:
LAN - sfp-sfpplus1
DMZ - ETH5
WAN1 - sfp - x.x.x.58/29
WAN2 - ETH8 x.x.x.246/30, routable IP block on this link is x.x.x.145/28
ROS ver 6.37.5

so I’ve configure already masquerade:

add action=masquerade chain=srcnat out-interface=sfp1
add action=masquerade chain=srcnat out-interface=ether8

mangle:

add action=mark-connection chain=prerouting comment="WAN1 IN -- > WAN1 OUT" in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-connection chain=prerouting comment="WAN2 IN -- > WAN2 OUT" in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN2 IN -- > WAN2 OUT" connection-mark=WAN2_pfw in-interface-list=Lan+ETH5 new-routing-mark=\
    to_WAN2 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN1 IN -- > WAN1 OUT" connection-mark=WAN1_pfw in-interface-list=Lan+ETH5 new-routing-mark=\
    to_WAN1 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=no

and routing:

add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=to_WAN2
add comment=WAN1 distance=1 gateway=x.x.x.57
add comment=WAN2 distance=2 gateway=x.x.x.245

But after that, I can ping only WAN1 from outside, pings to WAN2 x.x.x.246 didn’t respond - so what’s wrong with my setup?

My second question how should I configure this IP addresses block, should I simply add this IP block x.x.x.145/28 as secondary address to WAN2?

Have a look at Tomas Kirnak’s Load Balance / Mangle Deep Dive presentation.

Your problem comes from the fact you’re not marking on input chain.

I’ve changed my setup with this provided on Tomas Kirnak presentation:

add action=mark-connection chain=input comment="WAN1 IN -- > WAN1 OUT" in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-connection chain=input comment="WAN2 IN -- > WAN2 OUT" in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN2 IN -- > WAN2 OUT" connection-mark=WAN2_pfw in-interface=sfp-sfpplus1 \
    new-routing-mark=to_WAN2 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=no
add action=mark-routing chain=prerouting comment="Port Forward WAN1 IN -- > WAN1 OUT" connection-mark=WAN1_pfw in-interface=sfp-sfpplus1 \
    new-routing-mark=to_WAN1 passthrough=no
add action=mark-connection chain=forward connection-state=new in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=no

but still can’t ping WAN2 x.x.x.246 from outside, I get request timeout, and on mikrotik firewall connections I don’t have any connection marks with “WAN2”

mangle on WAN2 are the same like on WAN1 but counters on WAN2 mangle rules do not increase at all, doesn’t this look strange?

could anyone help me with that?

Follow the presentation mangle to the letter until you get a solid grasp on it, before jumping into customizations.

Order, and following it to the utmost detail is really important.

I’m done with tutorial I can’t add this load balancing part:

add chain=prerouting connection-mark=LAN->WAN src-address-list=LAN
action=mark-routing new-routing-mark=to_WAN1
comment="Load-Balancing here"

because then all my ovpn tunnels disconnecting and I loose connectivity with my remote ovpn networks.

My mangle looks now:

/ip firewall mangle
add action=accept chain=prerouting dst-address-list=Connected src-address-list=Connected
add action=mark-connection chain=input comment="WAN - > ROS" connection-mark=no-mark in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-connection chain=forward comment="WAN -> LAN" connection-mark=no-mark connection-state="" in-interface=sfp1 new-connection-mark=WAN1_pfw passthrough=no
add action=mark-connection chain=forward connection-mark=no-mark connection-state="" in-interface=ether8 new-connection-mark=WAN2_pfw passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN1_pfw new-routing-mark=to_WAN1 passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=WAN2_pfw new-routing-mark=to_WAN2 passthrough=no src-address-list=LAN
add action=mark-connection chain=prerouting comment="LAN -> WAN" connection-mark=no-mark dst-address-list=!Connected dst-address-type=!local new-connection-mark=LAN->WAN passthrough=no \
    src-address-list=LAN
add action=mark-connection chain=prerouting comment="Sticky connections" connection-mark=LAN->WAN new-connection-mark=Sticky_WAN1 passthrough=no routing-mark=to_WAN1
add action=mark-connection chain=prerouting connection-mark=LAN->WAN new-connection-mark=Sticky_WAN2 passthrough=no routing-mark=to_WAN2
add action=mark-routing chain=prerouting connection-mark=Sticky_WAN1 new-routing-mark=to_WAN1 passthrough=no src-address-list=LAN
add action=mark-routing chain=prerouting connection-mark=Sticky_WAN2 new-routing-mark=to_WAN2 passthrough=no src-address-list=LAN

when pinging mikrotik WAN2 still reguest timeout, and any connection mark with “WAN2” doesn’t show in firewall connections.

I didn’t study everything in detail, but if alternative routing tables give you problems with reaching some destinations, then either don’t mark routing for those destinations, or make sure that routes to them are always looked up in right routing table, e.g.:

/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main

I giving up with this, because even this first rule:

add action=mark-connection chain=input connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2->ROS passthrough=no

should show on firewall connections connection mark “WAN2->ROS” when I ping mikrotik to WAN2 from outside, but it didn’t get any “WAN2->ROS” mark, so is clear that next all my mangles do not work.
I’ve even upgraded mikrotik to latest bugfixes ROS but still the same.

I had a better look at whole thread, and even your initial config should have worked. Are you sure that you for example don’t block stuff from second WAN in /ip firewall filter?

It also means that I have to disagree with pukkita’s original suggestion (sorry :wink:). You don’t need to mark connections in input, doing so in prerouting is fine. And you don’t need to mark them in forward either. Just prerouting alone is enough (unless you want to have different marks for connections to and through router, for some reason). This is minimal config covering both traffic to router and through router:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=<WAN1> new-connection-mark=wan1
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=<WAN2> new-connection-mark=wan2
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> new-routing-mark=wan1
add action=mark-routing chain=prerouting connection-mark=wan2 in-interface=<LAN> new-routing-mark=wan2

Passthrough option yes or no does not matter in this simple config.

To save rules with multiple LANs, you can change last two rules (instead of duplicating them for each LAN) to:

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=wan1 new-routing-mark=wan1
add action=mark-routing chain=prerouting connection-mark=wan2 new-routing-mark=wan2

This will break routing, but it can be fixed using this:

/ip route rule
add action=lookup-only-in-table dst-address=<LAN subnet 1> table=main
add action=lookup-only-in-table dst-address=<LAN subnet 2> table=main
...

Btw, in linked presentation, the whole “sticky connections” part seems to be unnecessarily complicated. I hope I’m not missing something (it’s a little late here…), but it seems to me that it only makes sense when you insist on different marks for incoming and outgoing connections (not sure why). If you don’t need that, then with the above simple config, you can mark new outgoing connections with wan1 or wan2 mark and you have the same thing. Or don’t mark them at all and they’ll get correctly marked by first reply packet coming from WAN.

OK so I have changed setup with yours, and even rebooted mikrotik:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=sfp1 new-connection-mark=WAN1 passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether8 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=to_WAN2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface-list=Lan+ETH5 new-routing-mark=to_WAN2 passthrough=no

I have reviewed my firewall, on top of it I have enabled ICMP no matter of what incoming interface:

add action=accept chain=input comment="enable: pings" limit=50/5s,2 protocol=icmp

But still have request timeout when I try to ping WAN2 from outside, on firewall connections nothing with connection mark “WAN2” while pinging:

Even if I disable my forward ind input drop rules at the bottom of my firewall list:

add action=drop chain=forward comment="block forward" disabled=yes
add action=drop chain=input comment="block input" disabled=yes

Now something more, I have three remote networks over ovpn tunnels, and three over ipsec tunnels, users over ipsec tunnel connecting to my server on LAN, but after enabled this your setup these ipsec remote users loose connectivity to this server, until I disable this:

add disabled=yes distance=1 gateway=X.X.X.57 routing-mark=to_WAN1

So I think is any problem with data flow throught the router, packets get stucks somewhere?

Stupid question first, the second connection definitely works (when you e.g. connect different router there), right?

Next, did you perhaps do something with rp-filter in /ip settings? Option “strict” is known to not play nice with multiple WANs.

I’m not sure what else it could be.

With tunnels, ovpn ones will probably need a routing rule for remote subnets. With IPSec it depends on what you have. If it’s transport mode with some internal tunnel (ipip, …), it should be the same as ovpn. If it’s tunnel mode, I’d guess that it should just work. I never tried it with multiple WANs, but outside encrypted packets should work well with the rules. Incoming inside packets (stuff going through tunnel) will get marked with WANx mark, but as long as the tunnel itself works, it should not matter. But you can try to make an exception for them (add accept mangle rules before the others with in-interface= ipsec-policy=in,ipsec, and you can also try to not mark routing for ipsec-policy=out,ipsec packets). But in any case, first step is to solve ping on WAN2.

Yes, link that I’m trying connect as WAN2 working for 100% I switched it to my other tp-link router and everything works there.
Now I see in IP → setting that I have RP filter - Strict .
My IPSEC are in tunnel mode, and my OVPN tunnels have OSFP so do not have any route roules.

Then you know what to do, first change RP filter to get second WAN going, and then you’ll see if tunnels need any further help.

RP Filter changed to “no” or “loose”
but still can’t ping WAN2 IP, any “WAN2” connections marks doesn’t show in mikrotik firewall connections while pinging.

Remote IPSEC networks doesn’t have access to my server, it’s java based application on this server, application login page doesn’t show for them but when I disable IP route with mangle “to_WAN1”:

add disabled=yes distance=1 gateway=X.X.X.57 routing-mark=to_WAN1

then bum login page shows immediately.

Either use Tools->Torch on WAN2 interface to see what’s going on there, or (maybe better, because you won’t miss anything) add some logging rules for icmp (in prerouting, input, output and postrouting) and watch what happens with your ping packets, if they are any replies and where are they going.

The second part is strange too. If you still have routes configured like in first post:

add check-gateway=ping distance=1 gateway=x.x.x.57 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=x.x.x.245 routing-mark=to_WAN2
add comment=WAN1 distance=1 gateway=x.x.x.57
add comment=WAN2 distance=2 gateway=x.x.x.245

then difference bettween disabled and enabled to_WAN1 route should be none. If it does something, then it means that connections (and then routing) must be getting marked, otherwise router would ignore to_WAN1 routing table. But there’s no difference between this and main one, both have gateway=x.x.x.57.

You can try to not mark connections from IPSec (put it before other rules):

/ip firewall mangle
add action=accept chain=prerouting ipsec-policy=in,ipsec

I counted 14 mangle rules in export but screen shot shows more. Is it possible that FastTrack is enabled and dynamic Mangle rules are added?

Can you do an export of your ip route as well.

when I enable torch o WAN2 I see that something is going on here, I see my incoming connection from 83.x.x.130, but this not shows as connection mangle mark “WAN2”

additionally I see that counters on WAN2 prerouting increasing this time, earlier I didn’t have that:

I have only something like this about mangle dynamic:

Worth buying cause of good support.

sure:

/ip route
add check-gateway=ping distance=1 dst-address=8.8.4.4/32 gateway=x.x.x.57 routing-mark=google
add disabled=yes distance=1 gateway=x.x.x.245 routing-mark=from_WAN2
add disabled=yes distance=1 gateway=x.x.x.57 routing-mark=to_WAN1
add distance=1 gateway=x.x.x.245 routing-mark=to_WAN2
add comment=WAN1 distance=1 gateway=x.x.x.57
add comment=WAN2 distance=2 gateway=x.x.x.245
add distance=1 dst-address=10.5.0.0/24 gateway=sfp-sfpplus1
add distance=1 dst-address=10.20.0.0/22 gateway=sfp-sfpplus1
add distance=1 dst-address=192.168.0.0/24 gateway=sfp-sfpplus1
add distance=1 dst-address=192.168.1.0/24 gateway=sfp-sfpplus1

these last local subnets are remote IPSEC (tunnel) subnets - these have problem accesing my LAN server described earlier.
I have also three ovpn subnets and here is enabled OSPF so routes are dynamic.