ROS 6.49.6 How to fix or improve config for Recursive Routing with PBR?

Hello Everyone!

Another query here that needs your expertise. May I ask for your analysis and suggestions please on how to fix (if there is something wrong) or improve below recursive routing configuration? This sample config was made by someone and I want to understand how this config works. I don’t understand why there is no target scope set on the default routes. And what are those virtual routes for? Also, with the policy based routing included, would the traffic under routing-marks “to-gw2” or “to-gw3” automatically rerouted to other ISP should the corresponding assigned ISP went down?
Note: ISP1 and ISP2 have public static IP addresses ,hence, I replaced them with xxx.xxx for security purposes.


/ip route
add comment=“ISP1 route” distance=1 gateway=8.8.8.8 routing-mark=to-gw1 scope=10
add comment=“ISP2 route” distance=1 gateway=10.2.2.2 routing-mark=to-gw2 scope=10
add comment=“ISP3 route” distance=1 gateway=208.67.222.222 routing-mark=to-gw3 scope=10

add comment=“default route ISP1” distance=1 gateway=8.8.8.8 scope=10
add comment=“failover route ISP2” distance=2 gateway=10.2.2.2 scope=10
add comment=“failover route ISP3” distance=3 gateway=8.8.4.4 scope=10

add comment=“monitoring ip ISP1” distance=1 dst-address=9.9.9.9/32 gateway=xxx.xxx.178.121 scope=10
add comment=“monitoring ip ISP2” distance=1 dst-address=1.1.1.1/32 gateway=xxx.xxx.181.89 scope=10
add comment=“monitoring ip ISP3” distance=1 dst-address=8.26.56.26/32 gateway=192.168.1.1 scope=10

add check-gateway=ping comment=“virtual route ISP1” distance=1 dst-address=8.8.8.8/32 gateway=
9.9.9.9 scope=10
add check-gateway=ping comment=“virtual route ISP2” distance=1 dst-address=10.2.2.2/32 gateway=
1.1.1.1 scope=10
add check-gateway=ping comment=“virtual route ISP3” distance=1 dst-address=208.67.222.222/32
gateway=8.26.56.26 scope=10

anyone can help please? thank you in advance.

The most logical thing you need to do is write on the topic where you found those rules,
or ask whoever wrote it.
On the forum are already present dozen of examples and some are well explained.

Thank you for the response. This existing routing config in a running (production) MT was done by someone who is no longer reachable right now, hence, the non-techie owner of the network has ask me for help to check if the said config is correct or needs to improve.

Would appreciate if you can point me to specific links with almost same scenario (3 ISPs and 3 routing-mark assigned to each WAN) using PBR and Recursive Routing failover.
Thanks.

You can use search function on forum, is free

https://forum.mikrotik.com/viewtopic.php?t=182373

Thank you very much Sir.

Hello Sir,

After reading your routing guide, I was able to come up with this trial config. Would appreciate much if you can help me check for possible errors before I will implement this to the production MT and change its current routing configuration.

By the way, the expectation is that all three marked-routing traffic should maintain their internet access even if their default assigned ISP/WAN is down.

for example:

TRAFFIC#1: “to-gw1 traffic to ISP1” or “to ISP2 if ISP1 is down” or to ISP3 if both ISP1 & ISP2 are down.

TRAFFIC#2: “to-gw2 traffic to ISP2” or “to ISP1 if ISP2 is down” or “to ISP3 if both ISP1 & ISP2 are down”

TRAFFIC#3: “to-gw3 traffic to ISP3” or “to ISP1 if ISP3 is down” or “to ISP2 if both ISP3 & ISP1 are down”

Thank you.




+++TRIPLE WAN FAILOVER THREE RECURSIVE ROUTES (NESTED) WITH PBR+++

/ip firewall address-list
add address=10.100.100.0/24 list=STANDARD
add address=10.100.200.0/24 list=STANDARD
add address=10.200.123.0/24 list=VIP
add address=172.20.234.0/24 list=VIP
add address=192.168.111.0/24 list=OTHERS
add address=192.168.222.0/24 list=OTHERS

/ip firewall mangle
add action=mark-routing chain=prerouting comment=To-ISP1 new-routing-mark=to-gw1 passthrough=no src-address-list=STANDARD
add action=mark-routing chain=prerouting comment=To-ISP2 new-routing-mark=to-gw2 passthrough=no src-address-list=VIP
add action=mark-routing chain=prerouting comment=To-ISP3 new-routing-mark=to-gw3 passthrough=no src-address-list=OTHERS
++++++++++++++++++++++++++

/ip route
add comment=“Virtual Route ISP1” dst-address=0.0.0.0/0
gateway=10.1.1.1 scope=10 target-scope=14
+++++++++++++++++++++
add comment=“Monitoring ISP1-A” check-gateway=ping dst-address=10.1.1.1/32
gateway=8.8.8.8 scope=10 target-scope=13
add comment=“Primary Route ISP1-A” distance=1 dst-address=8.8.8.8/32
gateway=XXX.XXX.181.89 scope=10 target-scope=12
add comment=“Primary Route ISP1-A to-GW1” distance=1 dst-address=8.8.8.8/32
gateway=XXX.XXX.181.89 routing-mark=to-gw1 scope=10 target-scope=12
++++++++++++++++++++++
add comment=“Monitoring ISP1-B” check-gateway=ping dst-address=10.1.1.1/32
gateway=76.76.2.0 scope=10 target-scope=13
add comment=“Primary Route ISP1-B” distance=1 dst-address=76.76.2.0/32
gateway=XXX.XXX.181.89 scope=10 target-scope=12
add comment=“Primary Route ISP1-B to-GW1” distance=1 dst-address=76.76.2.0/32
gateway=XXX.XXX.181.89 routing-mark=to-gw1 scope=10 target-scope=12
++++++++++++++++++++++++++++

add comment=“Virtual Route ISP2” dst-address=0.0.0.0/0
gateway=10.2.2.2 scope=10 target-scope=14
+++++++++++++++++++++
add comment=“Monitoring ISP2-A” check-gateway=ping dst-address=10.2.2.2/32
gateway=9.9.9.9 scope=10 target-scope=13
add comment=“Secondary Route ISP2-A” distance=2 dst-address=9.9.9.9/32
gateway=192.168.1.1 scope=10 target-scope=12
add comment=“Secondary Route ISP2-A to-GW2” distance=2 dst-address=9.9.9.9/32
gateway=192.168.1.1 routing-mark=to-gw2 scope=10 target-scope=12
++++++++++++++++++++++
add comment=“Monitoring ISP2-B” check-gateway=ping dst-address=10.2.2.2/32
gateway=208.67.222.222 scope=10 target-scope=13
add comment=“Secondary Route ISP2-B” distance=2 dst-address=208.67.222.222/32
gateway=192.168.1.1 scope=10 target-scope=12
add comment=“Secondary Route ISP2-B to-GW2” distance=2 dst-address=208.67.222.222/32
gateway=192.168.1.1 routing-mark=to-gw2 scope=10 target-scope=12
+++++++++++++++++++++++++

add comment=“Virtual Route ISP3” dst-address=0.0.0.0/0
gateway=10.3.3.3 scope=10 target-scope=14
+++++++++++++++++++++
add comment=“Monitoring ISP3-A” check-gateway=ping dst-address=10.3.3.3/32
gateway=1.1.1.1 scope=10 target-scope=13
add comment=“Tertiary Route ISP3-A” distance=3 dst-address=1.1.1.1/32
gateway=XXX.XXX.178.121 scope=10 target-scope=12
add comment=“Tertiary Route ISP3-A to-GW3” distance=3 dst-address=1.1.1.1/32
gateway=XXX.XXX.178.121 routing-mark=to-gw3 scope=10 target-scope=12
++++++++++++++++++++++
add comment=“Monitoring ISP3-B” check-gateway=ping dst-address=10.3.3.3/32
gateway=76.76.19.19 scope=10 target-scope=13
add comment=“Tertiary Route ISP3-B” distance=3 dst-address=76.76.19.19/32
gateway=XXX.XXX.178.121 scope=10 target-scope=12
add comment=“Tertiary Route ISP3-B to-GW3” distance=3 dst-address=76.76.19.19/32
gateway=XXX.XXX.178.121 routing-mark=to-gw3 scope=10 target-scope=12

are the sequence of the entries/lines correct? does the order/sequence in /ip routes matters like those in /ip filter and mangle rules?

how about the distance, am I not missing something?

i hope someone can help me look and check my routing config. thank you.

Will look at this tomorrow, too fried tonight to make sense of it, but good questions!!!

Thank you Sir Anav.
Aside from your guide/tutorial, i also read Chupaka’s thread and other similar threads regarding recursive failover routing.
Most of the examples are dual WAN only, hence, i need to be sure if what i am doing for a triple WAN recursive failover is correct.
Unfortunately, i don’t have a bench MT units to experiment right now.

In addition to my questions above, i noticed in your sample config that the target-scope of the secondary ISP (back-up) is 30. May i know what’s the rational explanation why it should be target-scope=30 and not 20 or 17 or 15 or lower? In a Triple WAN recursive failover scenario, what should be the target scope of the secondary and tertiary routes? In my Triple WAN recursive (nested) config above, the scope and target-scope settings are just identical to each other for the three ISP routes. Is my config wrong?

Also, with regards to the distances, I only set the distance (based on priority 1, 2, 3) on the resolving routes and not on the monitoring (recursive) routes. Is my config correct?
What is the difference between simply using distances 1, 2 & 3 for the primary, secondary and tertiary routes respectively and by using other higher numbers for the distance like for example 5, 10, 15 or 1, 10, 20 or 100, 200, 300? does the numbers really matter? does it affect the length of time to shift from a broken route to a good route?

Looking forward to your analysis and recommendations.
Thank you and have a good sleep.

Just to confirm you have three groups of users, each should use a different wan as their primary.
There is no concern for incoming traffic externally originated, this setup is for outgoing traffic originated on the LANs?

Also , what I dont understand is why are you mangling if all the groups of users involved are whole subnets?
One should be able to avoid mangling and use routing rules etc…

Are all the addresses on the lists above local subnets behind the router ? Assuming yes.

ANother question, do any of the users need to access other subnets on the router. Since we are directing all users out a specific wan, we need to know if there any exceptions, when going outside ones own subnet. For example is their a shared printer or a local server that users will be crossing over in terms of subnets to get to…???

Yes Sir, users were grouped into three and each group has assigned primary route/WAN/ISP to access the internet. And yes the concern is the traffic from LANs going to the internet.

I am sorry but i really don’t fully understand yet how to use the routing rules in lieu to mangle rules. most of the video tutorials i found in youtube are using firewall address list and then mangle rules. would highly appreciate if you can teach me how to properly do it (routing rules) to be able to separate each group of users.

Yes, all addresses in the address lists are local subnets behind the router.

No shared printers nor servers, preferably each user should not be able to communicate each other locally.

Yes Sir, users were grouped into three and each group has assigned primary route/WAN/ISP to access the internet. And yes the concern is the traffic from LANs going to the internet.

I am sorry but i really don’t fully understand yet how to use the routing rules in lieu to mangle rules. most of the video tutorials i found in youtube are using firewall address list and then mangle rules. would highly appreciate if you can teach me how to properly do it (routing rules) to be able to separate each group of users.

Yes, all addresses in the address lists are local subnets behind the router.

No shared printers nor servers, preferably each user should not be able to communicate each other locally.

by the way, i am using RouterOS 6.49.6

i can’t really figure out where to look that /routing rules tab

what i have found is only /routing with below sub categories
BFD
BGP
Filters
MME
OSPF
Prefix List
RIP

i can’t find the Rules tab under Routing.

In v6 it’s in IP->Routes on Rules tab.

You can use any numbers for distance as long as they are separated.
I setup the scope and target scope numbers such that they are also legal for Vers7 when you switch.
The only difference between V6 and V7 for this particular config would be the need to create 3 tables and
on the second IP route for each ISPX, use table= vice routing-mark=

VERSION 6

dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 table=main  distance=5
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 routing-mark=ISP1_route distance=5
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13 
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12

..................................................................................................................................................

dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 table=main distance=10
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 routing-mark=ISP2_route  distance=10
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=SecondarISP-gatewayIP scope=10 target-scope=12
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13 
add dst-address=1.0.0.1/32 gateway=SecondaryISP-gatewayIP scope=10 target-scope=12

......................................................................................................................................................

dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 table=main distance=20
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 routing-mark=ISP3_route  distance=20
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=TertiaryISP-gatewayIP scope=10 target-scope=12
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13 
add dst-address=1.0.0.1/32 gateway=TertiaryISP-gatewayIP scope=10 target-scope=12

/ip routing rules
add src-address=subnetA  action=lookup table=ISP1_route comment=standard
add src-address=subnetB  action=lookup table=ISP1_route comment=standard
add src-address=subnetC  action=lookup table=ISP2_route comment=vip
add src-address=subnetD  action=lookup table=ISP2_route comment=vip
add src-address=subnetC  action=lookup table=ISP3_route comment=others
add src-address=subnetD  action=lookup table=ISP3_route comment=others

..

This should do it. All standard users will follow the route rules and go out ISP1, all vip users will go out ISP2 and all others users will go out ISP3.
Case1: IF ISP1 goes down the router will look for the next available route and will find it in the main table and since ISP2 is lower in distance it will be chosen and if ISP2 is not available it will choose ISP3. IF ISP2 comes back on line the users will be directed back through ISP2 and if ISP1 comes back on line the users will go to their original ISP1.
Case2: IF ISP2 goes doen the router will look for the next available route and it will be ISP1, and then ISP3 etc.
Case3: IF ISP3 goes down, the router will look for the next available route it will e ISP1, and then ISP2 etc.


VERSION 7 Differences.

/ip tables
add name=ISP1-route fib
add name=ISP2-route fib
add name=ISP3-route fib

dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 **table=**ISP1_route distance=5
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 **table=**ISP2_route distance=10
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 **table=**ISP3_route distance=20

Thank you very much for pointing me to the right tab Sir. I surely overlooked it.

Now i just need to understand how to properly use the Routing Rules to replace my Mangle Rules.

By the way, may i know why Anav and the other experts here preferred to use the routing rules instead of mangling? What are the advantages and disadvantages between the two?

jajajaja I’m not an expert and that is the reason I personally avoid mangling, its more complex and since normally one cannot use fastrack with mangling, (sometimes one can work around mangling but often not) and thus performance will be slower (although probably a home user would never notice). I use it when I dont have another choice. Keep it simple!!

rules.jpg

Hi Sir, noted on this. Will try to implement this config ASAP once my client (the network owner) has already sent an advisory to their subscribers for a scheduled system maintenance. This is to ensure the end users are aware of a possible service downtime.

Just a little clarification, i have noticed you only used the same set of external dns or ip addresses (9.9.9.9 & 1.0.0.1) for all the three WAN routes as well as the bogus IP address (10.10.10.10). What if for some weird instances, those two dns ip addresses are not available or down? Is it ok if we use different set of dns addresses for each WAN route like what i did in my trial config posted earlier ( 2 dns addreses and 1 bogus address per WAN) What would be the disadvantage of using different sets of external ip addresses for the recursive route?