Dual WAN routing policy + mangle rules [not working]

zakai · Mon Jul 15, 2013 10:44 am

Hi guys,

I have 2 WAN connections, namely ISP1 (1.1.1.1) and ISP2 (2.2.2.1), each on a different MT interface (ISP1 respectively ISP2). On the LAN side I have a few private subnets, 10.0.1.0/24 (interface LAN1) and 10.0.2.0/24 (LAN2). I also have dst-nat for both public IPs into the LANs.

My goal is to use the ISP1 as the default outgoing interface and use ISP2 only as backup. In the same time if flows come on either ISP1 or ISP2 interfaces I want to use the same interface for outgoing packets.

I've figured out these steps (please correct me if I'm wrong):

1. mark the incoming connections in order to be able to track on which interface they arrived and use the routing mark to select different lookup tables

2. mark the outgoing connections (originating from LAN) with the routing mark for default table that I want to use (in order to exit the default ISP1 interface)

3. add 2 route rules, one for each routing-mark to lookup in the desired tables

4. populate the routing tables corresponding to each routing mark

5. add the src and dst nat rules

Now I've done all this except it's not working, that is I cannot exit to the internet. So obviously there's something I didn't do or did wrong. I suspect the mangle rules are the reason. Below is my actual configuration:

ros code

/ip address
add address=1.1.1.1/30 comment="ISP1, default" disabled=no interface=ISP1 network=1.1.1.0
add address=2.2.2.1/30 comment="ISP2, backup" disabled=no interface=ISP2 network=2.2.2.0
add address=10.0.1.1/24 comment="LAN1" disabled=no interface=LAN1 network=10.0.1.0
add address=10.0.2.1/24 comment="LAN2" disabled=no interface=LAN2 network=10.0.2.0

/ip firewall mangle
add action=mark-connection chain=prerouting disabled=no in-interface=ISP1 new-connection-mark=ISP1 passthrough=no
add action=mark-connection chain=prerouting disabled=no in-interface=ISP2 new-connection-mark=ISP2 passthrough=no
add action=mark-routing chain=prerouting comment="LAN packets marked with 'ISP2' are stamped with routing mark 'to_ISP2'" \
    connection-mark=ISP2 disabled=no src-address-list=lans new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting comment="LAN packets marked with 'ISP1' are stamped with routing mark 'to_ISP1'" \
    connection-mark=ISP1 disabled=no src-address-list=lans new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting comment="LAN packets not previously marked are by default stamped with routing mark 'to_ISP1'" \
    disabled=no new-routing-mark=to_ISP1 passthrough=no src-address-list=lans
	
/ip firewall nat
add action=masquerade chain=srcnat disabled=no dst-address-list=!lans src-address-list=lans

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=2.2.2.2 routing-mark=to_ISP2 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=1.1.1.2 routing-mark=to_ISP2 scope=30 target-scope=10
add disabled=no distance=1 dst-address=10.0.1.0/24 gateway=LAN1 routing-mark=to_ISP2 scope=30 target-scope=10
add disabled=no distance=1 dst-address=10.0.2.0/24 gateway=LAN2 routing-mark=to_ISP2 scope=30 target-scope=10

add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=1.1.1.2 routing-mark=to_ISP1 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=2.2.2.2 routing-mark=to_ISP1 scope=30 target-scope=10
add disabled=no distance=1 dst-address=10.0.1.0/24 gateway=LAN1 routing-mark=to_ISP1 scope=30 target-scope=10
add disabled=no distance=1 dst-address=10.0.2.0/24 gateway=LAN2 routing-mark=to_ISP1 scope=30 target-scope=10

/ip route rule
add action=lookup disabled=no routing-mark=to_ISP1 table=to_ISP1
add action=lookup disabled=no routing-mark=to_ISP2 table=to_ISP2

/ip firewall address-list
add list=lans address=10.0.1.0/24
add list=lans address=10.0.2.0/24

Help is greatly appreciated, thanks!

faisali · Tue Jul 16, 2013 9:43 pm

-------
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=2.2.2.2 routing-mark=to_ISP2 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=1.1.1.2 routing-mark=to_ISP2 scope=30 target-scope=10
----------

Possible Typo ? Shouldn't the routing-mark be to_ISP2 and to_ISP1 ?

---------------------------------------------------
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=1.1.1.2 routing-mark=to_ISP1 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=2.2.2.2 routing-mark=to_ISP1 scope=30 target-scope=10
---------------------------------------------------

Similar in above ....

zakai · Tue Jul 16, 2013 10:01 pm

Hi faisali,

thanks for your input but it's not a typo though.

The routing-mark defines the table where the route will be. So I have 2 tables, both containing the connected routes + the default routes.

In table 'to_ISP1' I'll have the connected routes, one default route with a distance of 1 to ISP1 and one default failover to ISP2 with a distance of 10. All these 4 routes have to be in the same table so when a packet (belonging to a connection marked with 'ISP1') comes to the routing decision, it will use the routes in the 'to_ISP1' table as per route rule.
The same approach goes for table 'to_ISP2' but with the interchanged distance.

Any other hints guys? I'll make another test tomorrow and hopefully I'll figure it out...

gazzamit · Sat Jul 27, 2013 12:42 am

Hi,

Did you ever get this to work. I'm have problems with policy routing discussed in another thread?

Regards, Gary

Proxx · Sun Dec 01, 2013 5:30 pm

I would say that the problem is here:

ros code

add action=mark-connection chain=prerouting disabled=no in-interface=ISP1 new-connection-mark=ISP1 passthrough=no
add action=mark-connection chain=prerouting disabled=no in-interface=ISP2 new-connection-mark=ISP2 passthrough=no

After marking connection with new-connection-mark ISP1 or ISP2, packets leave mangle table. Therefore they never reach routing marking

ros code

add action=mark-routing chain=prerouting comment="LAN packets marked with 'ISP2' are stamped with routing mark 'to_ISP2'" \
    connection-mark=ISP2 disabled=no src-address-list=lans new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting comment="LAN packets marked with 'ISP1' are stamped with routing mark 'to_ISP1'" \
    connection-mark=ISP1 disabled=no src-address-list=lans new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting comment="LAN packets not previously marked are by default stamped with routing mark 'to_ISP1'" \
    disabled=no new-routing-mark=to_ISP1 passthrough=no src-address-list=lans

So the soultion is to change passthrough to yes in connection-marking so packets continue trough the mangle table.

ros code

add action=mark-connection chain=prerouting disabled=no in-interface=ISP1 new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=prerouting disabled=no in-interface=ISP2 new-connection-mark=ISP2 passthrough=yes

tania · Tue Feb 18, 2014 1:56 pm

http://mikrotikroutersetup.blogspot.com ... outer.html

herafi · Mon Sep 14, 2015 5:50 pm

Dear zakai,

I have the same problem, does it solved?

rgpmikrotik · Sat Apr 30, 2016 12:21 am

I have the same problem for the incoming traffic, have you solve this?

schadom · Wed Aug 16, 2017 1:32 am

i have the same problem. any solution?

justanotherhuman · Mon Jun 18, 2018 10:24 pm

i have the same problem. any solution?

Check also /ip settings !
The rp-filter=loose might be your friend!

Cheers!

zivtal · Fri Jul 27, 2018 12:51 am

I made a script that doing it, after running the script you should run from terminal: $WanInOut <add/remove> <wan interface name> <local lan interface name>

This is the script:

# WanInOut.rsc 06/03/2018
:global WanInOut do={
    :if ([len $1]=0) do={:error "example: \$WanInOut <add/remove> <wan interface name> <local lan interface name>";}
    :local todo [:tostr $1];
    :local wans [:tostr $2];

    # If no wan interface input detected
    :if ([:len $wans]=0) do={
        # search for isp connections
        :foreach idn in=[/interface find where type="pppoe-out" and disabled=no or type="l2tp-out" and disabled=no or type="ppp-out" and disabled=no or type="sstp-out" and disabled=no] do={
            if ([:typeof $wans]="nothing") do={
                :set $wans ([/interface get value-name=name $idn]);
            } else={
                :set $wans ($wans.",".[/interface get value-name=name $idn]);
            }
        }
        # search for dhcp clients
        :foreach idn in=[/ip dhcp-client find where add-default-route="yes" and status="bound" and disabled=no] do={
            :if ([/interface get value-name=type $idn]="ether") do={
                :if ([:typeof $wans]="nothing") do={
                    :set $wans ([/ip dhcp-client get value-name=interface $idn]);
                } else={
                    :set $wans ($wans.",".[/ip dhcp-client get value-name=interface $idn]);
                }
            }
        }
        # search for lte connections
        :foreach idn in=[/interface find where type="lte" and disabled=no] do={
            if ([:typeof $wans]="nothing") do={
                :set $wans ([/interface get value-name=name $idn]);
            } else={
                :set $wans ($wans.",".[/interface get value-name=name $idn]);
            }
        }
    }

    # If no lan interface input detected
    :local lans [:tostr $3];
    if ([:len $lans]=0) do={
        # search for bridges
        :foreach idn in=[/interface find where type="bridge" and disabled=no] do={
            if ([:typeof $lans]="nothing") do={
                :set $lans ([/interface get value-name=name $idn]);
            } else={
                :set $lans ($lans.",".[/interface get value-name=name $idn]);
            }
        }
    }

    :local wanip do={:if ([:len [/ip dhcp-client find where interface=[:tostr $1]]]>0) do={:return [/ip dhcp-client get [find interface=[:tostr $1]] gateway];} else={:return [$1];}}
    :local lanip do={ :return [:pick [:tostr [/ip address get [find where interface=[:tostr $1]] value-name=address]] 0 [:find [:tostr [/ip address get [find where interface=[:tostr $1]] value-name=address]] "/"]] }
    :local psfw $4;
    
    :if ($todo="add") do={
        foreach wan in=[:toarray $wans] do={
            :local wid [:pick [:tostr [/interface find name=$wan]] 1 ([:len [:tostr [/interface find name=$wan]]])];
            foreach lan in=[:toarray $lans] do={
                # rollback
                :local lid [:pick [:tostr [/interface find name=$lan]] 1 ([:len [:tostr [/interface find name=$lan]]])];
                :if (([:len [/ip firewall nat find where out-interface=$lan and action=masquerade and comment="rollback_$lid"]]=0) and ([:len [/ip address find interface=$lan]]>0)) do={
                    /ip firewall nat add chain=srcnat out-interface=$lan action=masquerade src-address=[/ip address get value-name=address [/ip address find interface=$lan]] comment="rollback_$lid"
                    :log info "'$lan' firewall nat masquerade ('rollback') has been created.";
                } else={
                    :log warning "'$lan' firewall nat masquerade ('rollback') exists.";
                }
                # redirect outgoing traffic in local network to wan(s)
                :if ([:len [/ip firewall nat find where out-interface=$wan and action=masquerade]]=0) do={
                    /ip firewall nat add chain=srcnat out-interface=$wan action=masquerade comment="enable internet access via '$wan'"
                    :log info "'$wan' firewall nat masquerade has been created.";
                } else={
                    /ip firewall nat set comment="enable internet access via '$wan'" [find where out-interface=$wan and action=masquerade];
                    :log warning "'$wan' firewall nat masquerade exists.";
                }
                # redirect incoming traffic in wan(s) to the local network
                :if ([:typeof $psfw]!="nothing") do={
                    :foreach pfwn in=[:toarray $psfw] do={
                        :if ([:len [/ip firewall nat find where chain=dstnat and protocol=tcp and dst-port=$pfwn and in-interface=$wan and action=dst-nat and to-addresses=[$lanip $lan] and to-ports=$pfwn]]=0) do={
                            /ip firewall nat add chain=dstnat protocol=tcp dst-port=$pfwn in-interface=$wan action=dst-nat to-addresses=[$lanip $lan] to-ports=$pfwn comment="redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan]:$pfwn)"
                            :log info "firewall nat redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan]:$pfwn) was created.";
                        } else={
                            /ip firewall nat set comment="redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan]:$pfwn)" [find where chain=dstnat and protocol=tcp and dst-port=$pfwn and in-interface=$wan and action=dst-nat and to-addresses=[$lanip $lan] and to-ports=$pfwn]
                            :log warning "firewall nat \"redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan]:$pfwn)\" exists.";
                        }
                    }
                } else={
                    :if ([:len [/ip firewall nat find where chain=dstnat and in-interface=$wan and action=dst-nat and to-addresses=[$lanip $lan]]]=0) do={
                        /ip firewall nat add chain=dstnat in-interface=$wan action=dst-nat to-addresses=[$lanip $lan] comment="redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan])"
                        :log info "firewall nat \"redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan])\" was created.";
                    } else={
                        /ip firewall nat set comment="redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan])" [find where chain=dstnat and in-interface=$wan and action=dst-nat and to-addresses=[$lanip $lan]]
                        :log warning "firewall nat \"redirect incoming traffic from '$wan' to '$lan' ($[$lanip $lan])\" exists.";
                    }
                }
                # what comes from specific wan, gets out from specific same wan
                :if ([:len [/ip firewall mangle find where action=mark-connection and chain=input and in-interface=$wan and new-connection-mark="conn_$wid"]]=0) do={
                    /ip firewall mangle add action=mark-connection chain=input in-interface=$wan new-connection-mark="conn_$wid" dst-address-type=!local passthrough=yes comment="incoming from '$wan' outgoing to '$wan'"
                    :log info "firewall mangle \"incoming from '$wan' outgoing to '$wan'\" was created.";
                } else={
                    /ip firewall mangle set dst-address-type=!local passthrough=yes comment="incoming from '$wan' outgoing to '$wan'" [find where action=mark-connection and chain=input and in-interface=$wan and new-connection-mark="conn_$wid"]
                    :log warning "firewall mangle \"incoming from '$wan' outgoing to '$wan'\" exists.";
                }
                :if ([:len [/ip firewall mangle find where action=mark-routing and chain=output and connection-mark="conn_$wid" and new-routing-mark="traffic_$wid"]]=0) do={
                    /ip firewall mangle add action=mark-routing chain=output connection-mark="conn_$wid" new-routing-mark="traffic_$wid" passthrough=no comment="outgoing to '$wan' incoming from '$wan'"
                    :log info "firewall mangle \"outgoing to '$wan' incoming from '$wan'\" was created.";
                } else={
                    /ip firewall mangle set passthrough=no comment="outgoing to '$wan' incoming from '$wan'" [find where action=mark-routing and chain=output and connection-mark="conn_$wid" and new-routing-mark="traffic_$wid"];
                    :log warning "firewall mangle \"outgoing to '$wan' incoming from '$wan'\" exists.";
                }
                # forwards from specific wan, gets out from specific same wan
                :if ([:len [/ip firewall mangle find where action=mark-connection and chain=forward and in-interface=$wan and new-connection-mark="pfw_$wid"]]=0) do={
                    /ip firewall mangle add action=mark-connection chain=forward in-interface=$wan connection-state=new new-connection-mark="pfw_$wid" passthrough=no comment="forwards incoming from '$wan' outgoing to '$wan'"
                    :log info "firewall mangle \"forwards incoming from '$wan' outgoing to '$wan'\" was created.";
                } else={
                    /ip firewall mangle set connection-state=new passthrough=no comment="forwards incoming from '$wan' outgoing to '$wan'" [find where action=mark-connection and chain=forward and in-interface=$wan and new-connection-mark="pfw_$wid"]
                    :log warning "firewall mangle \"forwards incoming from '$wan' outgoing to '$wan'\" exists.";
                }
                :if ([:len [/ip firewall mangle find where chain=prerouting and in-interface=$lan and connection-mark="conn_$wid" and action=mark-routing and new-routing-mark="traffic_$wid"]]=0) do={
                    /ip firewall mangle add chain=prerouting in-interface=$lan connection-mark="conn_$wid" action=mark-routing new-routing-mark="traffic_$wid" comment="prerouting incoming from '$lan' outgoing to '$wan'"
                    :log info "firewall mangle \"prerouting incoming from '$lan' outgoing to '$wan'\" was created.";
                } else={
                    /ip firewall mangle set comment="prerouting incoming from '$lan' outgoing to '$wan'" [find where chain=prerouting and in-interface=$lan and connection-mark="conn_$wid" and action=mark-routing and new-routing-mark="traffic_$wid"];
                    :log warning "firewall mangle \"prerouting incoming from '$lan' outgoing to '$wan'\" exists.";
                }
                if ([:len [/ip firewall mangle find where chain=prerouting and in-interface=$wan and action=mark-connection and new-connection-mark="conn_$wid"]]=0) do={
                    /ip firewall mangle add chain=prerouting in-interface=$wan action=mark-connection new-connection-mark="conn_$wid" comment="prerouting incoming from '$wan' outgoing to '$lan'"
                    :log info "firewall mangle \"prerouting incoming from '$wan' outgoing to '$lan'\" was created.";
                } else={
                    /ip firewall mangle set comment="prerouting incoming from '$wan' outgoing to '$lan'" [find where chain=prerouting and in-interface=$wan and action=mark-connection and new-connection-mark="conn_$wid"]
                    :log warning "firewall mangle \"prerouting incoming from '$wan' outgoing to '$lan'\" exists.";
                }
                # routing rules for wans' traffic
                :if ([:len [/ip route find where dst-address=0.0.0.0/0 and gateway=[$wanip $wan] and distance=1 and routing-mark="traffic_$wid"]]=0) do={
                    /ip route add dst-address=0.0.0.0/0 gateway=[$wanip $wan] distance=1 routing-mark="traffic_$wid" comment="routing rules for '$wan' traffic"
                    :log info "firewall mangle \"routing rules for '$wan' traffic\" was created.";
                } else={
                    /ip route set comment="routing rules for '$wan' traffic" [find where dst-address=0.0.0.0/0 and gateway=[$wanip $wan] and distance=1 and routing-mark="traffic_$wid"];
                    :log warning "firewall mangle \"routing rules for '$wan' traffic\" exists.";
                }
            }
        }
        :set $i ($i+1);
    }
    :if ($todo="remove") do={
        foreach wan in=[:toarray $wans] do={
            :local wid [:pick [:tostr [/interface find name=$wan]] 1 ([:len [:tostr [/interface find name=$wan]]])];
            foreach lan in=[:toarray $lans] do={                
                # redirect incoming traffic in wan(s) to the local server
                /ip firewall nat remove [find where chain=dstnat and protocol=tcp and dst-port=$pfwn and in-interface=$wan and action=dst-nat and to-addresses=[$lanip $lan] and to-ports=$pfwn]
                /ip firewall nat remove [find where chain=dstnat and in-interface=$wan and action=dst-nat and to-addresses=[$lanip $lan]]
                # rollback
                :local lid [:pick [:tostr [/interface find name=$lan]] 1 ([:len [:tostr [/interface find name=$lan]]])];
                /ip firewall nat remove [find where out-interface=$lan and action=masquerade and comment="rollback_$lid"];
                # what comes from specific wan, gets out from specific same wan
                /ip firewall mangle remove [find where action=mark-connection and chain=input and in-interface=$wan and new-connection-mark="conn_$wid"]
                /ip firewall mangle remove [find where action=mark-routing and chain=output and connection-mark="conn_$wid" and new-routing-mark="traffic_$wid"]
                # port forwards from specific wan, gets out from specific same wan
                /ip firewall mangle remove [find where action=mark-connection and chain=forward and in-interface=$wan and new-connection-mark="pfw_$wid"]
                /ip firewall mangle remove [find where action=mark-routing and chain=prerouting and in-interface=$lan and connection-mark="pfw_$wid" and new-routing-mark="traffic_$wid"]
                /ip firewall mangle remove [find where chain=prerouting and in-interface=$lan and connection-mark="conn_$wid" and action=mark-routing and new-routing-mark="traffic_$wid"]
                /ip firewall mangle remove [find where chain=prerouting and in-interface=$wan and action=mark-connection and new-connection-mark="conn_$wid"]
                # routing rules for wans' traffic
                /ip route remove [find where dst-address=0.0.0.0/0 and gateway=[$wanip $wan] and distance=1 and routing-mark="traffic_$wid"]
            }
        }
    }
}