modifying route distance dual wan

cyayon · Thu Dec 29, 2022 10:32 am

Hi,

I am going to be crazy with this issue.
I have 2 wan :
WAN1 primary : dhcp-client inside a bridge (to do bridge-filters and modifying COS, required by ISP)
WAN2 secondary : standard interface to a tier router

WAN1 default route distance is 1
WAN2 default route distance is 2

/ping address=8.8.8.8 count=3 interface=wan1 -> ok
/ping address=8.8.8.8 count=3 interface=wan2 -> ok
I can confirm that it worked as expected because latencies are differents one each link and it match.

/ip route set [find where dst-address=0.0.0.0/0 and gateway=<wan1_gw>] distance=10
/ping address=8.8.8.8 count=3 interface=wan1 -> KO
/ping address=8.8.8.8 count=3 interface=wan2 -> ok

/ip route set [find where dst-address=0.0.0.0/0 and gateway=<wan1_gw>] distance=1
/ping address=8.8.8.8 count=3 interface=wan1 -> ok
/ping address=8.8.8.8 count=3 interface=wan2 -> ok

Why ?

I tried routing tables and rules, mangle output/prerouting, etc... No way.
Perhaps a same issue with recursive routing to detect wan failure do not work... Or i didn't manage to make it work...

I am using a CCR2004 on ROS 7.6, i tried ROS 7.7rc3. no change

thanks

pcunite · Thu Dec 29, 2022 3:51 pm

This can be tricky to get correct. I may do an entire article series on this someday. At the moment, forum member anav has an extensive resource here. See section I. IP ROUTE - Multi-WAN. If you prefer a video format, this example is useful.

Take it slow understanding this concept. I know you want it all figured out now, but its going to take a while to get it and adapt to MikroTik's way of doing things.

sindy · Thu Dec 29, 2022 4:00 pm

Just a suggestion for investigation, as I could only test part of it on a remote device I cannot afford to tamper with too much.

If you specify an interface as a parameter of the ping command, RouterOS doesn't look for the best route out of those whose gateway interface is the specified one, but it sends an ARP request from that interface, and if it gets a response, it sends the echo request packet to the MAC address returned. But it seems that before doing that, it queries the normal routing, and if the gateway interface indicated by normal routing is the one specified for the ping, it sends the echo request to the MAC address of the corresponding gateway without first sending an ARP request for the actual destination of the ping.

So the difference between your wan1 and wan2 may be the support of the proxy-arp functionality on the adjacent routers (gateways) - if my assumption is correct, the one connected to wan2 supports it and the one connected to wan1 doesn't. So ping with interface=wan1 only works if wan1 is the gateway interface of the active default route.

Sniffing on your system in both states (wan1 distance=1 and wan1 distance=10) will show you whether my assumption is correct.

cyayon · Thu Dec 29, 2022 5:50 pm

Just a suggestion for investigation, as I could only test part of it on a remote device I cannot afford to tamper with too much.

If you specify an interface as a parameter of the ping command, RouterOS doesn't look for the best route out of those whose gateway interface is the specified one, but it sends an ARP request from that interface, and if it gets a response, it sends the echo request packet to the MAC address returned. But it seems that before doing that, it queries the normal routing, and if the gateway interface indicated by normal routing is the one specified for the ping, it sends the echo request to the MAC address of the corresponding gateway without first sending an ARP request for the actual destination of the ping.

So the difference between your wan1 and wan2 may be the support of the proxy-arp functionality on the adjacent routers (gateways) - if my assumption is correct, the one connected to wan2 supports it and the one connected to wan1 doesn't. So ping with interface=wan1 only works if wan1 is the gateway interface of the active default route.

Sniffing on your system in both states (wan1 distance=1 and wan1 distance=10) will show you whether my assumption is correct.

Hi,

from a another router (a custom archlinux based router), it works as expected and i have no issue at all.
In both case, the adjacent router is the same.

Why on mikrotik it does not work ?
Do i have something special to do to allow just a ping for an interface even after changing the distance/metric ?

thanks

sindy · Thu Dec 29, 2022 6:22 pm

from a another router (a custom archlinux based router), it works as expected and i have no issue at all.
In both case, the adjacent router is the same.

Why on mikrotik it does not work ?

One possibility I can think of is that the ping with specification of interface works different on the two operating systems. Another possibility is that whereas Mikrotik definitely ignores the ICMP router advertisements possibly sent by the gateway router, the archlinux may use them to associate a gateway with an interface, so it knows what gateway to use for packets sent from "wan1" even though the gateway doesn't support proxy-arp.

Do i have something special to do to allow just a ping for an interface even after changing the distance/metric ?

I would just use routing-table instead of interface as a parameter of the ping.

Also, I wouldn't jump to any extensive conclusions based on the behaviour you observe - if sniffing confirms that when the route via wan1 is not active (because its distance is too high), RouterOS sends ARP requests for 8.8.8.8 via wan1 and gets no response, it says nothing more than that it indeed behaves that way. The issue with recursive routing is highly unlikely to caused by this.

To let the forum analyse your actual issue with failover, you have to provide the export of the configuration and explain what exactly behaves different than you expect.

cyayon · Thu Dec 29, 2022 6:40 pm

Hi,

thanks for your answer.

here an export.

how to define a routing-table with ping command on ROS ?

sindy · Thu Dec 29, 2022 7:31 pm

how to define a routing-table with ping command on ROS ?

Oops... not possible in ROS 7 (hopefully it's a temporary state). So you'd have to use a routing rule matching on a src-address (the one attached to the wan in question) and specify a src-address as a parameter of the ping.

In the export I cannot see any signs of an advanced failover setup for the wans - the only failover assumed is from wan1 (Orange via bridge, distance of the default route added by DHCP is 1) to wan2 (distance of the manually configured default route is 3) if wan1 physically fails. But the thing is that the way it is configured (due to the caprice by Orange), the route via wan1 will not go down even if you physically disconnect the Ethernet/SFP interface, because in RouterOS, the bridge interface remains up even if all member interfaces of that bridge are down. So the route via wan1 will only disappear from the routing table once the DHCP lease expires, which may take hours after the physical interface goes down.

So you have to monitor the actual transparency of the uplink via wan1 all the way to the internet, using either the scriptless failover based on recursive next-hop search or some scripted solution.

cyayon · Thu Dec 29, 2022 7:59 pm

Thanks,

Before configuring something more advanced i am trying to check something simple.
I have followed recursive routing doc before (not is this export), but it didn’t work at all.

The routes were flapping ok/KO every 2 or 3 minutes.
After that I was going to do failover with a script as a workaround, when the interface fail (or isp), I would like to update distance of the failed interface route to something superior to promote the other one. That is the story.

Could you please confirm me the configuration in my context to make recursive routing work ?

Thanks.

sindy · Thu Dec 29, 2022 8:34 pm

Could you please confirm me the configuration in my context to make recursive routing work ?

Sorry, I don't understand what you ask (and my French is really bad).

Your current configuration doesn't contain anything related to recursive routing, so I cannot confirm anything.

To make it work, you have to add the recursive hierarchy of two routes - one to a /32 destination address, some "canary" one (such as 8.8.8.8) out there in the internet that uses the actual wan1 gateway as gateway, and another one to 0.0.0.0/0 that uses the canary address as a gateway. The scope of the former one must be lower than the default one of 30 (I usually set 10), the target-scope of the latter must be one higher than the scope of the former, and the latter must have check-gateway set to ping.

This ensures that RouterOS keeps pinging the canary address, and the latter route is only active if it receives responses to that ping. The relationship between the target-scope of the latter and the scope of other routes than the "former" one ensures that if the former one becomes inactive, the test pings will not use any other route, such as the one via wan2. And the recursion ensures that packets matching the latter route will actually be sent via the gateway of the former route.

But in your case, as the gateway of wan1 is obtained via DHCP, you have to either be sure that Orange will assign always the same gateway address, or you have to add a script to the DHCP client that will update the "former" route each time the gateway address changes.

pcunite · Thu Dec 29, 2022 8:43 pm

Could you please confirm me the configuration in my context to make recursive routing work ?

Watch the video I linked to. Watch slowly.

cyayon · Fri Dec 30, 2022 9:52 am

Hi,

i have removed all mangle firewall rules, removed all default gateway and disabled on dhcp-client add-default-route (ISP1).

Then i executed the following (80.11.60.1 is main default route ISP1/dhcp and 192.168.6.1 is LTE failover router) :

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=10 target-scope=12
add distance=3 dst-address=1.1.1.1/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=10 target-scope=12
add distance=4 dst-address=8.8.8.8/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=5 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12
add distance=5 dst-address=9.9.9.9/32 gateway=80.11.60.1 scope=10 target-scope=11
add comment=Failover distance=10 dst-address=0.0.0.0/0 gateway=192.168.6.1 scope=10 target-scope=30

Is it correct ?

For now, the link/route table seems stable, i will check longer...

In case of a link failure from ISP1, the failover is automatic as i understand.
But the rollback recover is also automatic or do i have to script something ?

if i would like to make sure that 80.11.60.1 never change and prefer a dhcp script, do you have an example to found gateway from dhcp-client please ?

thanks.

sindy · Fri Dec 30, 2022 12:56 pm

Is it correct ?

The general idea is OK, but the implementation details are not. The purpose of scope and target-scope is to define the hierarchy of the routes for the recursive next-hop search or, in another words, to prevent looping. So set the scope of all the routes with dst-address=0.0.0.0/0 to the default 30, otherwise they could use each other in the recursion.

Just to avoid a misconception - the distance parameter only differentiates between routes with the same dst-address and routing-table. So there is no need to set it differently for each of the /32 routes, unless you do it to make Winbox sort them next to the /0 routes that use them as carrier ones.

In case of a link failure from ISP1, the failover is automatic as i understand.
But the rollback recover is also automatic or do i have to script something ?

Yes and no depending on your traffic. Active TCP connections will fail at each change; depending on the applications that use them, some will get re-established automatically and some will have to be re-established manually. UDP connections that are periodically updated, such as IPsec or SIP connections, need to be removed using a script after a failover, because the router itself or the device in its LAN keep sending packets even if no responses come back, and the router sets the source address of these packets to the one of the WAN through which the respective connection has been established initially, unless that address has been lost (due to interface down or expiration of DHCP lease) and they have been src-nated using a masquerade rule.

if i would like to make sure that 80.11.60.1 never change and prefer a dhcp script, do you have an example to found gateway from dhcp-client please ?

Something like
/ip dhcp-client set [find where interface=bridge-wan1] script=":if (\$bound=1) do={\
\n /ip route set [find where dst-address~\"/32\" scope=10] gateway=\$\"gateway-address\"\
\n}"

To check that it really works (I have adjusted it to your config and I may have made a mistake), set the gateway of one of the routes to some bogus IP address like 10.22.33.44, and do
/ip dhcp-client release [find interface=bridge-wan1]

If the gateway you have changed changes back to the correct one, it works.

cyayon · Fri Dec 30, 2022 1:24 pm

Hi,
Many thanks for your answer.

Do I have to check-gateway ping the failover wan too ?

Just to make sure, could you please my implementation ?

Thanks !

sindy · Fri Dec 30, 2022 1:50 pm

Do I have to check-gateway ping the failover wan too ?

You don't exactly have to, but any backup solution is almost useless if it is not monitored - as it stays unused for months or even years, it may silently fail and when the primary one fails, the backup is not available. I prefer to use the Telegram app to send notifications about the state of SOHO networks to their users, but of course you can use e-mail as well.

For the operation itself, monitoring of the state of the WAN of last resort makes no sense, as there is no further backup the router could use if it fails. But there are setups with multiple WANs where each of them is the primary one for some class of traffic and a backup one for another class, and there you obviously have to monitor all of them to facilitate a failover.

Just to make sure, could you please my implementation ?

Sure I can have a look at the export of your current configuration, but it will be much more useful if you check it practically first (using /interface/disable ether1 and/or by adding firewall rules dropping the test pings to chain output of firewall filter). E.g. if you use /ip/firewall/filter add chain=output dst-address=1.1.1.1 protocol=icmp action=drop, you will imitate that 1.1.1.1 doesn't respond, so the /0 route via 1.1.1.1 will become inactive and the route via 8.8.8.8 will take over.

cyayon · Fri Dec 30, 2022 2:09 pm

I will post here my last export. As soon as i have access later today…

Could you please confirm that it is better to do this :
dst-address=0.0.0.0/0 -> scope=30
dst-address=x.x.x.x/32 -> distance=3

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=30 target-scope=12
add distance=3 dst-address=1.1.1.1/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=30 target-scope=12
add distance=3 dst-address=8.8.8.8/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=5 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=30 target-scope=12
add distance=3 dst-address=9.9.9.9/32 gateway=80.11.60.1 scope=10 target-scope=11
add comment=Failover distance=10 dst-address=0.0.0.0/0 gateway=192.168.6.1 scope=30 target-scope=30

But I thank that i must set target-scope > scope ?
Or I make a mistake ?

Thanks

sindy · Fri Dec 30, 2022 2:52 pm

Could you please confirm that it is better to do this :
dst-address=0.0.0.0/0 -> scope=30

Yes.

dst-address=x.x.x.x/32 -> distance=3

I don't say that having the distance values of the /32 routes as you had them before is wrong; what I say is that it doesn't matter what those distance values are because those /32 routes differ in dst-address.

But I thank that i must set target-scope > scope ?

The relation between target-scope and scope of the same route is irrelevant; what matters is that the target-scope of a "client" route was at least one higher than the scope of the "server" route, see a hypothetic example below. Red and blue indicate the related elements, gray indicates default values (that are normally not shown in export):
/ip route
add dst-address=8.8.8.8 gateway=192.168.1.1 scope=10 target-scope=10
add dst-address=8.8.8.255 gateway=8.8.8.8 check-gateway=ping scope=11 target-scope=11
add dst-address=0.0.0.0/0 gateway=8.8.8.255 scope=30 target-scope=12

In this example, the route to 8.8.8.8/32 via an actual gateway is the "bottommost server" one, and the route to 0.0.0.0/0 via 8.8.8.255 is the "topmost client" one. In this static case, the address 8.8.8.255 is only a linking element - nothing actually uses it (but it must be an address to which you never need to send any actual traffic). In a dynamic routing environment, which is the actual reason why the recursive next-hop search has been implemented, things may be much more complex. In short, a router somewhere at the border of your network advertises that it has a gateway to some destination subnet, but you have no common subnet with that border router, so to get the traffic for that destination subnet to that border router for delivery, you have to send it to some adjacent router that does have a way to forward it to the border one.

cyayon · Fri Dec 30, 2022 3:43 pm

Ok, i think i understand.

in this implementation, if i understand correctly everything is fine :

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=30 target-scope=12
add distance=3 dst-address=1.1.1.1/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=30 target-scope=12
add distance=3 dst-address=8.8.8.8/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=5 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=30 target-scope=12
add distance=3 dst-address=9.9.9.9/32 gateway=80.11.60.1 scope=10 target-scope=11
add comment=Failover distance=10 dst-address=0.0.0.0/0 gateway=192.168.6.1 scope=30 target-scope=30

explanation for 1.1.1.1 :
server route : add distance=3 dst-address=1.1.1.1/32 gateway=80.11.60.1 scope=10 target-scope=11
client route : add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=30 target-scope=12
client target scope : 12 > server scope : 10 - OK

right ?

but i do not understand why you choose in your example to add a route to 8.8.8.255 ? I know that i will never have to send traffic to this one, but why not only using dst-address=8.8.8.8 abd dst-address=0.0.0.0/0 routes ?

thanks !

sindy · Fri Dec 30, 2022 5:08 pm

right ?

Right.

but i do not understand why you choose in your example to add a route to 8.8.8.255 ?

Mostly to illustrate that there may be multiple levels of recursion and how they are related. But a typical approach when using multiple "canary addresses" looks as follows:

dst-address=1.1.1.1/32 gateway=192.168.1.1 scope=10
dst-address=8.8.8.8/32 gateway=192.168.1.1 scope=10

dst-address=10.22.33.44/32 gateway=1.1.1.1 target-scope=11 scope=11 check-gateway=ping
dst-address=10.22.33.44/32 gateway=8.8.8.8 target-scope=11 scope=11 check-gateway=ping

dst-address=0.0.0.0/0 gateway=10.22.33.44 target-scope=12

The goal of using the intermediate route to a fictious /32 destination is to have only a single default route per each WAN. This simplifies configurations where multiple routing tables are used (e.g. "prefer wan1" and "prefer wan2"). For your simple case, your approach with multiple default routes is more efficient as you have 6 routes in total for 3 canary addresses, whereas the typical approach needs 7. For two routing tables, it's 9 routes with your approach and 8 with the typical one.

cyayon · Fri Dec 30, 2022 8:06 pm

Many thanks for your answer !

I will test and keep you informed

Just a last questions

When wan1 go down, the failover link wan2 will take over.
But, will I be able to join 8.8.8.8, 1.1.1.1. And 9.9.9.9 from wan2 ?
If not, i will have to choose others ip addresses …

What does check-gateway arp and bfd ?

Thanks.

anav · Fri Dec 30, 2022 8:50 pm

Canary??
I call it flat vs nested.........
Sindys routes look bang on.........

This is what I was informed were the basic tenants of recursive in vers 7.
TWO RULES OF THUMB (scope & target scope):
First Rule. The resolving route (DIRECT - connected route) with dst-address TO the "real WWW IP (dns site)" and with local ISP gateway IP, has Target-Scope=X and the recursive route (INDIRECT - external route) with gateway IP VIA the "real work WWW gateway IP (dns site)" has Target-Scope=X+1. In other words, the farther one gets from the router, the TS increases by one.

Second Rule. Between the same two routes being compared, the Direct , connected route, with local ISP gateway IP (resolving route) has to have a SCOPE that is equal to or less than the TARGET SCOPE of the recursive route. In other words, the scope of the route must be equal or less than the target scope of the next farthest route.

EX.
FARTHEST ROUTE: SCOPE= (doesnt matter) / TARGET SCOPE=Y+2 (recursive route)
CLOSER ROUTE: SCOPE= Y+2 or less / TARGET SCOPE=Y+1 (recursive route)
CLOSEST ROUTE: SCOPE=Y+1 or less / TARGET SCOPE=Y (gateway=ISP, resolving route)
INTERNAL ROUTE: ( within router, scope is not used, no recursive action at all )

Thus,
A FLAT two recursive setup would look like......... So we only have a Y+1 scenario.
/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11
+++++++++++++++++++
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12
add distance=4 dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11
+++++++++++++++++++
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

Note using scope=10 for all primary associated routes is 'safe'.

=====================================================

Lets do it nested! In this case we use a fictitious _ ahhh this is what Sindy means by canary!! address to force the router to resolve it via two recursive routes.
Where 10.10.10.10 is an address/gateway that is private but does not exist on the router........

/ip route
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14
++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
+++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
+++++++++++++++
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

Note: All Primary routes have same distance, only the last Secondary route has a higher distance.
Note: Again using scope=10 is 'safe' all the way round.

Caveat: Since these rules were communicated there may have been some nuances or discoveries that I am not aware of, in which case would have to update my page.

+++++++++++++
Last word I tried to help someone with three WANS, and the level of complexity jumps considerably such that one needs an excel table to keep track of all required routes. More than you think!

cyayon · Fri Dec 30, 2022 9:20 pm

Ok,

Just to be sure...

With my gateways and context :

The nested following version :

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14
add check-gateway=ping dst-address=10.10.10.10/32 gateway=8.8.8.8 scope=10 target-scope=13
add dst-address=8.8.8.8/32 gateway=80.11.60.1 scope=10 target-scope=12
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=80.11.60.1 scope=10 target-scope=12
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.1.1.1 scope=10 target-scope=13
add dst-address=1.1.1.1/32 gateway=80.11.60.1 scope=10 target-scope=12
add comment=Failover distance=10 dst-address=0.0.0.0/0 gateway=192.168.6.1 scope=10 target-scope=30

will work as expected, and is BETTER than the following flat :

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=30 target-scope=12
add distance=3 dst-address=1.1.1.1/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=30 target-scope=12
add distance=3 dst-address=8.8.8.8/32 gateway=80.11.60.1 scope=10 target-scope=11
add check-gateway=ping distance=5 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=30 target-scope=12
add distance=3 dst-address=9.9.9.9/32 gateway=80.11.60.1 scope=10 target-scope=11
add comment=Failover distance=10 dst-address=0.0.0.0/0 gateway=192.168.6.1 scope=30 target-scope=30

Right ?

ROS will switch to Failover gateway (wan2 192.168.6.1), only if 1.1.1.1 8.8.8.8 and 9.9.9.9 are unreachable.

But will i be able to join 1.1.1.1 8.8.8.8 and 9.9.9.9 when wan1 is down and wan2 take the default gateway ?

Thanks !

sindy · Fri Dec 30, 2022 9:31 pm

But, will I be able to join 8.8.8.8, 1.1.1.1. And 9.9.9.9 from wan2 ?

No. You can use 8.8.4.4 instead of 8.8.8.8, 1.0.0.1 instead of 1.1.1.1, and 149.112.112.112 instead of 9.9.9.9 - same companies, different addresses.

What does check-gateway arp and bfd ?

arp means that ROS uses ARP protocol rather than ICMP echo to check the availability of the gateway, which means that the gateway must be reachable via an L2 interface, hence this mode cannot be used to check that the uplink is transparent all the way to the internet. Even in the unlikely case that the nearest Orange's router acting as the gateway for your one supports proxy-arp mode, if you'd be checking e.g. 8.8.8.8 using check-ping=arp, it would be the nearest Orange that would respond. So in case of an issue somewhere inside Orange's network, your router would keep using wan1 but would get nowhere.
bfd means that ROS uses Bi-directional Fault Detection to check the availability of the gateway; whereas ping and arp are served by the lowest layers of the networking stack so they usually work even if the device is almost dead, BFD is an application over UDP so if a device responds on BFD, it is usually a better indication that it is in good condition than ARP or ping. But it can only be used if the gateway supports it, so it is out of question for this application where we use anycast addresses of public DNS servers because they have to stay at fixed IP addresses intrinsically (a DNS server that would need another DNS server to resolve its domain name to an IP number would be pretty useless) and are designed to be redundant (which unfortunately doesn't mean they never fail - I had a case recently where 8.8.8.8 was unreachable from multiple devices accross Europe for several minutes, so using multiple canary addresses belonging to different providers is a good way to avoid false alarms).

sindy · Fri Dec 30, 2022 9:42 pm

In this case we use a fictitious _ ahhh this is what Sindy means by canary!! address to force the router to resolve it via two recursive routes.

Nope. What @sindy means by canary is an address that is used as an indicator that the uplink is working all the way to the internet. The term is not mine, and it is based on the fact that before electronic gas detectors became available, miners used canaries as biological indicators that the air is safe to breathe. If all canary addresses associated to a given uplink stop responding, you know that internet cannot be reached via that uplink.

The "fictious" gateway cannot be checked by pinging, thus it cannot be used as a canary one.

cyayon · Fri Dec 30, 2022 9:46 pm

Many thanks to both.

I do not really understand why nested is better (or not) than the flat ?

Nested seems more readable, but is it more efficient ?

sindy · Fri Dec 30, 2022 9:49 pm

will work as expected, and is BETTER than the following flat :

In your simple application, the nested approach is not better than the flat one, it is just different. In more complicated applications, the nested one becomes better because it makes the overall configuration simpler than the flat one.

Nested seems more readable, but is it more efficient ?

It is equally efficient. The recursion is not reiterated for every single packet - when you use /ip/route/print detail, you'll see something like gateway-status=8.8.8.8 recursive via 192.168.1.1 ether1.

cyayon · Fri Dec 30, 2022 10:02 pm

Thanks !
I Will test tomorrow…

anav · Fri Dec 30, 2022 10:17 pm

Hence the popular term canary in a coal mine LOL. Thanks!

cyayon · Sat Dec 31, 2022 2:20 am

Thé configs you post here seems to work as expected. Many thanks for that !!!

I do not understand why the config in the official guide https://help.mikrotik.com/docs/pages/vi ... d=26476608 do not work at all.
I followed it and gateways were flapping ok/ko every 2 or 3 minutes.
What am I missing ?

anav · Sat Dec 31, 2022 2:31 am

Dont care what they post, its often generic and one needs to adapt to ones particular needs......

cyayon · Sat Dec 31, 2022 7:49 am

Ok thanks.

If I need inbound connections from internet via ISP1 and ISP2, do
I need firewall mangling rules ? For rouie and conn marking.
If yes, prerouting only I suppose, not output ?

Thanks.

sindy · Sat Dec 31, 2022 8:26 am

Yes.

Does your LTE backup have a public IP address? If not, you cannot get inbound connections on it, unless you create a tunnel through it to some external router that does have a public address.

As for prerouting/output, the latter would be necessary to translate the connection mark to a routing mark to responses from the router itself, and for that, you can use a routing rule instead if the address of wan 2 is static.

cyayon · Sat Dec 31, 2022 9:40 am

Yes my LTE have an external address. It is not fixed. But the Mikrotik will connected to the LTE router which have a fixed address (not a bridge).
On my current router (a custom Archlinux based), I have only mangle prerouting rules and routing rules and it works.

For the Mikrotik, If i put anything together I have this :

/ip routing table
add name=ISP1_route fib
add name=ISP2_route fib

/ip route
add check=gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=ISP1_route
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main

add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=main
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=ISP1_route
add distance=4 dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main

add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=ISP2_route

/ip firewall mangle
add chain=prerouting in-interface=WAN1 connection-state=new action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN1_conn action=mark-routing new-routing-mark=ISP1_route
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN2_conn action=mark-routing new-routing-mark=ISP2_route

/ip routing rule
routing-mark=ISP1_route action=lookup table=main
routing-mark=ISP2_route action=lookup table=main
src-address=PrimaryISP-gatewayIP action=lookup table=ISP1_route
src-address=SecondaryISP-gatewayIP action=lookup table=ISP2_route

do i missing something ?

Thanks

sindy · Sat Dec 31, 2022 10:54 am

In the ISP1_route table, there is no point to use the recursion to monitor the uplink state - this table is used for responses that must use wan1 and if wan1 doesn't work, sending those packets via wan2 won't help. So the default route in ISP1_route can use directly PrimaryISP-gatewayIP as gateway and no check-gateway is necessary.

The mangle rules seem OK to me (they could be slightly optimized for throughput but let's make it work first and then care about perfection), but the routing rules are a total miss.

The first two ones should actually do nothing in RouterOS > 7.5 or so - before, routing rules could overwrite the routing table name assigned by mangle rules, but since 7.5 (or 7.4?) this has changed (along with a few other related things). Before 7.5, these rule would revert the routing table assignment by mangle rules and force the packets to use main.

The last two ones are correct except that the src-address must be the own address of the router on the respective WAN interface, not the address of the gateway router.

cyayon · Sat Dec 31, 2022 11:04 am

Thanks.

ip routing table
add name=ISP1_route fib
add name=ISP2_route fib

/ip route
add check=gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12 table=ISP1_route
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main

add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=main
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway= PrimaryISP-gatewayIP scope=10 target-scope=12 table=ISP1_route
add distance=4 dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main

add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=ISP2_route

/ip firewall mangle
add chain=prerouting in-interface=WAN1 connection-state=new action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN1_conn action=mark-routing new-routing-mark=ISP1_route
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN2_conn action=mark-routing new-routing-mark=ISP2_route

/ip routing rule
———- is it necessary ???? —————
routing-mark=ISP1_route action=lookup table=main
routing-mark=ISP2_route action=lookup table=main
—————————————————-
src-address=PrimaryInterfaceIP action=lookup table=ISP1_route
src-address=SecondaryInterfaceIP action=lookup table=ISP2_route

To make sure i understand correctly, could you please correct them in place ?

sindy · Sat Dec 31, 2022 11:09 am

/ip routing rule
src-address=own-IP-at-wan1 action=lookup table=ISP1_route
src-address=own-IP-at-wan2 action=lookup table=ISP2_route

cyayon · Sat Dec 31, 2022 11:12 am

Thanks !

I didn’t understand what you say about this

/ip routing rule
———- is it necessary ???? —————
routing-mark=ISP1_route action=lookup table=main
routing-mark=ISP2_route action=lookup table=main
—————————————————-

And if i do this

add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12 table=ISP1_route

The route is marked as unreachable by ROS

sindy · Sat Dec 31, 2022 11:40 am

/ip routing rule
———- is it necessary ???? —————
routing-mark=ISP1_route action=lookup table=main
routing-mark=ISP2_route action=lookup table=main
—————————————————-

No, just remove these rules.

And if i do this
...
The route as unreachable

Setting check-gateway to none should help, I hazily remember it worked strange for non-recursive routes.

cyayon · Sat Dec 31, 2022 11:45 am

Ok i removed check-gateway and it worked.

Are you sure that there is no need for this route to be recursive because in Anav guide and milkrotik help site, these exist and are recursive.

If it is not necessary, I suppose I could also keep only one (the first) table=route_ISP1 distance=3 ?
Do i need the distance param in this case ?

Finally, is it necessary to change rp_filter from no to loose ?

Thanks

sindy · Sat Dec 31, 2022 12:48 pm

Are you sure that there is no need for this route to be recursive because in Anav guide and milkrotik help site, these exist and are recursive.

Don't trust my opinion, trust mere logic:
What is the purpose of the recursive routing in your case? To let the traffic use a backup path if the primary one becomes unavailable.
Does rerouting the response packets of connections that have been initiated from the internet to wan1 via wan2 save those connections? Well, yes - in the unlikely case that the responses sent via wan2 reach the client and their source address remains the one of wan1, and that only the wan1->internet direction is broken so the subsequent packets from the client to the IP of wan1 do reach your router.
Your LTE router does NAT, so the first premise (keeping the source address of wan1 for packets sent via wan2) won't be met.

If it is not necessary, I suppose I could also keep only one (the first) table=route_ISP1 distance=3 ?
Do i need the distance param in this case ?

No. You only need the distance parameter to define the mutual priority of routes with identical routing-table and dst-address parameters.

Finally, is it necessary to change rp_filter from no to loose ?

Yes, sure. strict means that packets that come in through any other interface than through which responses to them would be sent are dropped, and the rp-filter cannot use any kind of policy routing when evaluating this. But the benefits of rp-filter=strict are overrated anyway.

cyayon · Sat Dec 31, 2022 12:59 pm

Ok, many thanks again.

In conclusion only one default gateway route with no distance param in each route table != main.
Rp_filter = no->loose

Thanks !

sindy · Sat Dec 31, 2022 1:31 pm

with no distance param in each route table != main.

"no distance parameter" actually means distance=1. All parameters have some value, but like with many other vendors, configuration export doesn't show parameters with default values unless explicitly asked to do so (using the verbose parameter)

Rp_filter = no->loose

Sorry, I have msunderstood your question. Both no and loose are OK for policy routing, only strict breaks it.

cyayon · Sat Dec 31, 2022 1:58 pm

Thanks. I understood

cyayon · Sat Dec 31, 2022 4:33 pm

i just applied everything.

first problem : i cannot ping from the router itself and from LAN devices my 2 external public IP addresses given by my ISPs (from wan1 and wan2).

I think it is a routing issue, my firewall rules drop ICMP :

DROP input: in:wan2 out:(unknown 0), connection-mark:conn-wan2 connection-state:new src-mac xxxx, proto ICMP (type 8, code 0), WAN1_IP->WAN2_IP, len 84

pinging WAN1_IP should come from WAN1 not from WAN2 ?

EDIT:
as soon as i DISABLE the following routing rules, it works, i can ping WAN1_IP from LAN :
/routing rule
add src-address=80.11.60.214 action=lookup table=route-wan1
add src-address=192.168.6.254 action=lookup table=route-wan2

Do I miss something ?

thanks.

sindy · Sat Dec 31, 2022 6:14 pm

first problem : i cannot ping from the router itself and from LAN devices my 2 external public IP addresses given by my ISPs (from wan1 and wan2).

The current routing rules say that whatever is sent from one of the WAN addresses must use one of the two ISPx_route tables, and these tables have no routes but the default one, so the response to a ping coming from a private address in the LAN subnet are sent to the respective WAN gateway. So to let the router respond to these pings, you have to place another routing rule before the two existing ones:
dst-address=192.168.0.0/16 action=lookup table=main
(if you need a finer granularity, you need more routing rules or you have to switch to a mangle rule matching on an address lists or on an interface list; both ways mean more CPU load per packet).

pinging WAN1_IP should come from WAN1 not from WAN2 ?

Indeed a ping request towards the IP of wan 1 from the IP of wan 2 should arrive to wan 1, but the log from the firewall shows that it was a ping towards the IP of wan 2 from the IP of wan 1, so it looks fine to me.

Your firewall rules in chain input are indeed tight - there is no "accept icmp" one, and the final one is "drop everything". However, in the export you've posted before, there is log-prefix=TRASH_INPUT, and I can't see that on the log row you've posted now, so maybe you've made some changes also in the firewall in the meantime?

cyayon · Sat Dec 31, 2022 6:54 pm

ok, i have added all these rules :
/routing rule
add src-address=80.11.60.214 action=lookup table=route-wan1
add src-address=192.168.6.254 action=lookup table=route-wan2
add dst-address=192.168.0.0/16 action=lookup table=main

but, it didn't work.

no ping from lan (192.168.0.0/16 match) to WAN1_IP.
as soon as i disable them, ping is OK.

Then, i have remove all routing rules and re-add them in a different order :
/routing rule
add dst-address=192.168.0.0/16 action=lookup table=main
add src-address=80.11.60.214 action=lookup table=route-wan1
add src-address=192.168.6.254 action=lookup table=route-wan2

and it worked.
ping from lan to WAN1_IP is OK.

I understant, the order is important.

but i disable all routing rules and everything seems to work fine, is it really necessary to add them ?
perhaps there will be some use-case i missed ...

thanks

sindy · Sat Dec 31, 2022 7:32 pm

but i disable all routing rules and everything seems to work fine, is it really necessary to add them ?

The only reason to add the routing rules is that the router itself would respond to incoming requests from the internet using the proper uplink. This basically boils down to pinging it from the internet and connecting VPN clients (or peers) to it.

cyayon · Sat Dec 31, 2022 7:50 pm

Ok i understand. Many many thanks for your time and precious help.
I wish you an happy new year !

See you next year

pcunite · Thu Feb 02, 2023 12:29 am

In the ISP1_route table, there is no point to use the recursion to monitor the uplink state - this table is used for responses that must use wan1 and if wan1 doesn't work, sending those packets via wan2 won't help. So the default route in ISP1_route can use directly PrimaryISP-gatewayIP as gateway and no check-gateway is necessary.

Testing with ROS v7.7 and four WAN links, I needed check-gateway enabled when I had a Server set to only use a particular WAN link and that link goes down. So, it seems there is some validity to monitoring that a route with a table parameter. See my example here. Discussion is open.

sindy · Fri Feb 03, 2023 6:21 pm

I'm not sure we are at the same page. First, we don't discuss the case where there is no NAT and you can use dynamic routing protocols to advertise your subnets to multiple neighbors - we are dealing with the "home power user" scenario where there is one or multiple layers of NAT between each WAN and the internet, right?

In this case, in my understanding, it only makes sense to monitor a gateway if it makes sense to send a packet, which would normally be sent via that gateway, via another gateway. Normally, a mid-connection packet that comes from another source address than previous packets belonging to the same connection is dropped by the recipient, because it cannot be identified as one belonging to that connection. Exceptions exist, e.g. Wireguard can adjust to change of peer's address and port dynamically. But for TCP, where the unique combination of local and remote addresses and ports is used as an index to the list of sessions, sending a packet via another gateway than the previous ones is useless.

Due to this, for any connections initiated from the internet, the upstream traffic (responses) must be sent via the same WAN via which the initial request has arrived. If no connections can be initiated from the LAN side via a given WAN, there is no need to monitor the state of its gateway.

If a particular WAN is used as a path of last resort (i.e. only if all other ones are unavailable), there is also no point in monitoring its gateway, because in such a case, there is no way to deliver the packets if you find that the gateway of this particular WAN is down.

cyayon · Thu Jul 13, 2023 9:49 am

Hi,

I encounter a strange issue today.
I had a failure on my primary link and another failure on my dynamic ip update script for my secondary link...

When the primary link came back, i tested my secondary link input rules and routes (when my 2 links are up and running).
A simple curl from an internet test server to my secondary ip link (LTE) took very long time (at least 20-30s) to answer.
However, output (NAT) from LAN to internet via the secondary link do not have any issue and work as expected.
Of course i do not have any issue on my primary link (input/output).

I defined a simple route in main table to force output to the server where i tested input (curl) and after that, it worked as expected. The requests on the secondary link answer very quick as expected.
In conclusion, there seems to be an routing issue, but i do not understand where i made a mistake.

My router is a CCR2116 with RouterOS 7.7.
Here is the config in attachment, could you please check if everything is fine ?

EDIT1 : i tried to use tools/Torch to check, and as soon as i enable Torch, everything work perfectly ! I stop Torch issue come back again !
EDIT2 : upgraded to the last 7.10.2, no change. I even tried to completely disable recursive monitoring for primary link, (and only use distance) no change
EDIT3 : in real failover (when primary link is really down), no issue, the secondary ip (failover link) work and answer very quickly
EDIT4 : using packet sniffer tool has the same effect that Torch. As soon as i enable, it works perfectly. It seems that it is the promiscious mode that make it works. VERY strange...
EDIT5 : opened a support ticket SUP-121920

thanks !

cyayon · Tue Jul 18, 2023 1:11 pm

Hi,

Nobody with this strange issue ?
Thanks

anav · Tue Jul 18, 2023 4:12 pm

Post the full config and I will have a look, wont waste my time on half the evidence

cyayon · Tue Jul 18, 2023 5:08 pm

Hello,

Here is the config. I just removed some sensitive informations.

thanks.

cyayon · Sat Jul 29, 2023 2:31 am

Hi,
Nobody have an idea for this issue ?
No answer from the support

Just an idea, could it be related to fasttrack ?
I read somewhere that Fasttrack and mangle dual-wan are incompatible, is it a mistake ?
If fasttrack is the issue, is there a workaround to keep fasttrack only for some packets of and keep mangle / routing rules/tables ?
Perhaps with some additional filters on fasttrack rule (exclude input interface or not marked packets)?

Thanks

sindy · Sat Jul 29, 2023 9:21 am

I read somewhere that Fasttrack and mangle dual-wan are incompatible, is it a mistake ?

The very principle of fasttrack operation is that the fasttracked packets bypass some stages of packet processing, mangle is just one of these stages.

is there a workaround to keep fasttrack only for some packets of and keep mangle / routing rules/tables ?

Sure there is: viewtopic.php?p=659676#p659676

cyayon · Sat Jul 29, 2023 10:42 am

Hi,

Many thanks Sindy for your answer.

I thank I was using mangle rules right, but as an evidence, it is not...
My issue is for inbound packets from internet via WAN2 (when WAN1 is NOT down).

I do not understand where I made a mistake :
vlan832-orange1 is wan1
ether2.wan2 is wan2
interface-list WAN contain wan1 and wan2

current version :

/ip firewall mangle add action=mark-connection chain=input connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=input connection-state=new in-interface=ether2.wan2 new-connection-mark=conn-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan1 in-interface-list=!WAN new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan2 in-interface-list=!WAN new-routing-mark=route-wan2 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=ether1.wan2 new-connection-mark=conn-wan2 passthrough=yes

a reworked version based on your post but it do not work :

/ip firewall mangle

add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface-list=WAN   action=accept # download packets MUST NOT be routing-marked

add chain=prerouting connection-mark=conn-wan1 action=mark-routing new-routing-mark=route-wan1 # passthrough=no is a default behaviour but you can state it explicitly
add chain=prerouting connection-mark=conn-wan2 action=mark-routing new-routing-mark=route-wan2 # same like above
add chain=prerouting action=mark-connection connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
add chain=prerouting action=mark-connection connection-state=new in-interface=ether1.wan2     new-connection-mark=conn-wan2 passthrough=yes

add chain=output action=mark-routing connection-state=new connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
add chain=output action=mark-routing connection-state=new connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

thanks for your help !

cyayon · Sat Jul 29, 2023 11:34 am

Ok,
I finally managed to made it work with the following rules and fasttrack additional condition : connection-mark=no-mark.

/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan1 in-interface-list=!WAN new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan2 in-interface-list=!WAN new-routing-mark=route-wan2 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=ether1.wan2 new-connection-mark=conn-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

I suppose it is not optimized, could you please check ?
thanks !

sindy · Sat Jul 29, 2023 11:43 am

I'm not sure I understand your intentions properly, so just some points:

in prerouting, you first assign the routing-mark to packets that did not come via WAN, and only the next rule assigns the connection-mark depedning on in-interface. From this, and from the absence of any rules distributing the traffic among the two WANs, I deduce that you only want to keep the connection on the WAN through which it has been initially established, and respond incoming connections from the internet using the correct WAN. Is that a correct understanding?
in both output and prerouting, you translate a connection-mark conn-wanX to routing-mark route-wanX. But whilst in input, you assign the connection-mark conn-wan2 based on in-interface sfp2.wan1, in prerouting, you assign it based on in-interface ether1.wan2. So one of these must be wrong, most likely the one in input as none of your routes has a gateway in 192.168.4.0/24.
from the moment a connection gets fasttracked, all its packets use the routes in table main (except those few that RouterOS chooses not to fasttrack). So if you want to use fasttracking e.g. only for connections that use WAN 1, table main must not contain any route via WAN 2, which effectively means that you need some other table than main to handle the initial packets of all connections so that the connection could be established via ether WAN depending on their current state. Once a WAN interface is chosen by routing using that "initial" table, you assign a connection-mark to the connection initiated by that packet, which will later be translated to a routing-mark. It is enough to assign the connection-mark conn-wan2 as packets belonging to connections without any connection-mark will not get any routing-mark even if they pass via mangle, so they will be routed using table main. And the action=fasttrack-connection rule must match on connection-mark=no-mark or connection-mark=!conn-wan2 to prevent it from fasttracking connections that must use WAN 2.
just for laughs, this is a chef-d'œuvre:
/ip route add comment="main table" ... routing-table=route-wan1

cyayon · Sat Jul 29, 2023 12:14 pm

Thanks for your answer.

1. YES (but I don't know if it a good pratice)
2. in the last version, I removed the input mangle rules and it works (but I think this not optimized)
3. I do not understand everything, could you please give me a small example ?
4. for a moment, I thank I made a real mistake in my configuration

But it is just a funny comment to remind me it is the route-table wan1 (which is main wan - not the failover).

What do you think about these rules ? are they right and more optimized ?
I suppose because of the two first rules which are not passthrough on established

/ip firewall mangle
add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface-list=WAN   action=accept # download packets MUST NOT be routing-marked

add chain=prerouting action=mark-routing connection-mark=conn-wan1 new-routing-mark=route-wan1 
add chain=prerouting action=mark-routing connection-mark=conn-wan2 new-routing-mark=route-wan2

add chain=prerouting action=mark-connection connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
add chain=prerouting action=mark-connection connection-state=new in-interface=ether1.wan2     new-connection-mark=conn-wan2 passthrough=yes

add chain=output action=mark-routing connection-state=new connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
add chain=output action=mark-routing connection-state=new connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

ÉDIT : Just to clarify, what I would like to do is relatively simple.
For outbound (lan to internet) use wan1 (main) and wan2 ONLY if wan1 is down (check wan1 with recursive routing…)
For inbound (internet to lan), wan1 and wan2 could be used from internet. I self-host some services…

sindy · Sat Jul 29, 2023 12:42 pm

1. YES (but I don't know if it a good pratice)

It is OK if it is your intention (to keep a connection on the secondary WAN even if the primary one recovers).

2. in the last version, I removed the input mangle rules and it works (but I think this not optimized)

Traffic that uses input chain also uses the output chain. So once you have removed the mangle rules from input, the mangle rules in output became useless. Own outgoing traffic of the router is only handled by output and postrouting chains in mangle, not by the prerouting one.

3. I do not understand everything, could you please give me a small example ?

The rough idea is to create a routing table initial that will contain the two topmost recursive default routes (via WAN 1 and via WAN 2, preferring WAN 1), and to use mangle rules in output and in prerouting to assign a routing-mark initial to packets that match connection-state=new so that they would use that table.

There must be a lot of exceptions in these rules - in order that the monitoring of the canary addresses worked properly, you either have to exclude these addresses from the mangle rule in output, or you must move the complete set of recursive routes to table initial as well. And you have to prevent traffic from the router itself to devices in LAN from getting the routing-mark initial as well.
Routing table main will only contain the default route via WAN 1's gateway and all the dynamically created routes to LAN subnets.

Routing table route-wan2 will only contain the default route via WAN 2's gateway.

cyayon · Sat Jul 29, 2023 12:51 pm

Thanks.

The output rules i have are for handle traffic of the router itself.
I edited my last post with new mangles rules. Did you see them and what do you think about ?

Thanks.

cyayon · Sat Jul 29, 2023 5:11 pm

Ok, here are my updated rules.

For memory, I am trying to achieve these goals :
-For outbound (lan to internet) use wan1 (main) and wan2 ONLY if wan1 is down (check wan1 with recursive routing…)
-For inbound (internet to lan), wan1 and wan2 could be used from internet. I self-host some services…

first, the fasttrack updated to keep only connection-mark=no-mark

/ip firewall filter add action=fasttrack-connection chain=forward comment="accept established,related" connection-mark=no-mark connection-state=established,related,untracked hw-offload=yes

second, the mangle rules

/ip firewall mangle add action=accept chain=prerouting comment="nomark related" connection-mark=no-mark connection-state=established,related
/ip firewall mangle add action=accept chain=prerouting comment="WAN related" connection-state=established,related in-interface-list=WAN 
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=no
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=no
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=ether1.wan2 new-connection-mark=conn-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

After some tests, everything seems to work as expected.
I am able to reach from internet my hosted services from wan1 and wan2 (when my two connections are up and running)
When wan1 failed, wan2 take the relay (from lan to internet). Of course, in this case I always can reach my services from wan2.
When wan1 recover, it returns as the main gateway as expected.

As soon as I remove the condition "connection-mark=no-mark" on my fasttrack filter rule, the issue came back again and I cannot reach my services from internet on my secondary link wan2 when my two wan1 and wan2 are up and running at the same time. But, I understand, it is normal. Then i restore the condition "connection-mark=no-mark".

What do you think about my rules ?
Is there a workaround to keep fasttrack on all packets (without "connection-mark=no-mark" ) and have multi-wan routing/connection mark ?

thanks

anav · Sat Jul 29, 2023 11:24 pm

These four rules go together well. They basically ensure any traffic arriving on a particular WAN, leaves on the same WAN.

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1.wan2 new-connection-mark=conn-wan2 passthrough=yes

/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=no
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=no

In terms of FAILOVER this is done in routes ( simple single flat recursive ).

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=11 target-scope=11
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

OR dual nested recursive..........

/ip route
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14
++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
+++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
+++++++++++++++
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

+++++++++++++++++++++++++++++++++++++++

Dont forget the extra routes required for incoming originated traffic........
/ip routes
add dst-address=0.0.0.0/0 gateway=ISP1gatewayip routing-table=route-wan1
add dst-address=0.0.0.0/0 gateway=ISP2gatewayip routing-table=route-wan2

cyayon · Sat Jul 29, 2023 11:55 pm

Thanks !

I thank it was a good practice to begin mangle with the following 2 rules to skip already established.

/ip firewall mangle add action=accept chain=prerouting comment="nomark related" connection-mark=no-mark connection-state=established,related
/ip firewall mangle add action=accept chain=prerouting comment="WAN related" connection-state=established,related in-interface-list=WAN

Note that my current recursive routing is a little different :

/ip route add check-gateway=ping comment="main recursive" disabled=no distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
/ip route add comment="main table" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=80.11.60.1 pref-src="" routing-table=route-wan1 scope=10 suppress-hw-offload=no target-scope=12
/ip route add comment="main recursive check1" disabled=no distance=3 dst-address=1.0.0.1/32 gateway=80.11.60.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11

/ip route add check-gateway=ping comment="main recursive" disabled=no distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.10 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
/ip route add comment="main recursive check2" disabled=no distance=4 dst-address=9.9.9.10/32 gateway=80.11.60.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11

/ip route add check-gateway=ping comment="main recursive" disabled=no distance=5 dst-address=0.0.0.0/0 gateway=8.8.4.4 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
/ip route add comment="main recursive check3" disabled=no distance=5 dst-address=8.8.4.4/32 gateway=80.11.60.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11

/ip route add comment=failover disabled=no distance=9 dst-address=0.0.0.0/0 gateway=192.168.6.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=30
/ip route add comment="failover table" disabled=no distance=9 dst-address=0.0.0.0/0 gateway=192.168.6.1 pref-src="" routing-table=route-wan2 scope=10 suppress-hw-offload=no target-scope=30

And it seems to work…

Who is online