modifying route distance dual wan

sindy · December 31, 2022, 11:31am

“no distance parameter” actually means distance=1. All parameters have some value, but like with many other vendors, configuration export doesn’t show parameters with default values unless explicitly asked to do so (using the verbose parameter)

Sorry, I have msunderstood your question. Both no and loose are OK for policy routing, only strict breaks it.

cyayon · December 31, 2022, 11:58am

Thanks. I understood

cyayon · December 31, 2022, 2:33pm

i just applied everything.

first problem : i cannot ping from the router itself and from LAN devices my 2 external public IP addresses given by my ISPs (from wan1 and wan2).

I think it is a routing issue, my firewall rules drop ICMP :

DROP input: in:wan2 out:(unknown 0), connection-mark:conn-wan2 connection-state:new src-mac xxxx, proto ICMP (type 8, code 0), WAN1_IP->WAN2_IP, len 84

pinging WAN1_IP should come from WAN1 not from WAN2 ?

EDIT:
as soon as i DISABLE the following routing rules, it works, i can ping WAN1_IP from LAN :
/routing rule
add src-address=80.11.60.214 action=lookup table=route-wan1
add src-address=192.168.6.254 action=lookup table=route-wan2

Do I miss something ?

thanks.

sindy · December 31, 2022, 4:14pm

The current routing rules say that whatever is sent from one of the WAN addresses must use one of the two ISPx_route tables, and these tables have no routes but the default one, so the response to a ping coming from a private address in the LAN subnet are sent to the respective WAN gateway. So to let the router respond to these pings, you have to place another routing rule before the two existing ones:
dst-address=192.168.0.0/16 action=lookup table=main
(if you need a finer granularity, you need more routing rules or you have to switch to a mangle rule matching on an address lists or on an interface list; both ways mean more CPU load per packet).

Indeed a ping request towards the IP of wan 1 from the IP of wan 2 should arrive to wan 1, but the log from the firewall shows that it was a ping towards the IP of wan 2 from the IP of wan 1, so it looks fine to me.

Your firewall rules in chain input are indeed tight - there is no “accept icmp” one, and the final one is “drop everything”. However, in the export you’ve posted before, there is log-prefix=TRASH_INPUT, and I can’t see that on the log row you’ve posted now, so maybe you’ve made some changes also in the firewall in the meantime?

cyayon · December 31, 2022, 4:54pm

ok, i have added all these rules :
/routing rule
add src-address=80.11.60.214 action=lookup table=route-wan1
add src-address=192.168.6.254 action=lookup table=route-wan2
add dst-address=192.168.0.0/16 action=lookup table=main

but, it didn’t work.

no ping from lan (192.168.0.0/16 match) to WAN1_IP.
as soon as i disable them, ping is OK.

Then, i have remove all routing rules and re-add them in a different order :
/routing rule
add dst-address=192.168.0.0/16 action=lookup table=main
add src-address=80.11.60.214 action=lookup table=route-wan1
add src-address=192.168.6.254 action=lookup table=route-wan2

and it worked.
ping from lan to WAN1_IP is OK.

I understant, the order is important.

but i disable all routing rules and everything seems to work fine, is it really necessary to add them ?
perhaps there will be some use-case i missed …

thanks

sindy · December 31, 2022, 5:32pm

The only reason to add the routing rules is that the router itself would respond to incoming requests from the internet using the proper uplink. This basically boils down to pinging it from the internet and connecting VPN clients (or peers) to it.

cyayon · December 31, 2022, 5:50pm

Ok i understand. Many many thanks for your time and precious help.
I wish you an happy new year !

See you next year

pcunite · February 1, 2023, 10:29pm

Testing with ROS v7.7 and four WAN links, I needed check-gateway enabled when I had a Server set to only use a particular WAN link and that link goes down. So, it seems there is some validity to monitoring that a route with a table parameter. See my example here. Discussion is open.

sindy · February 3, 2023, 4:21pm

I’m not sure we are at the same page. First, we don’t discuss the case where there is no NAT and you can use dynamic routing protocols to advertise your subnets to multiple neighbors - we are dealing with the “home power user” scenario where there is one or multiple layers of NAT between each WAN and the internet, right?

In this case, in my understanding, it only makes sense to monitor a gateway if it makes sense to send a packet, which would normally be sent via that gateway, via another gateway. Normally, a mid-connection packet that comes from another source address than previous packets belonging to the same connection is dropped by the recipient, because it cannot be identified as one belonging to that connection. Exceptions exist, e.g. Wireguard can adjust to change of peer’s address and port dynamically. But for TCP, where the unique combination of local and remote addresses and ports is used as an index to the list of sessions, sending a packet via another gateway than the previous ones is useless.

Due to this, for any connections initiated from the internet, the upstream traffic (responses) must be sent via the same WAN via which the initial request has arrived. If no connections can be initiated from the LAN side via a given WAN, there is no need to monitor the state of its gateway.

If a particular WAN is used as a path of last resort (i.e. only if all other ones are unavailable), there is also no point in monitoring its gateway, because in such a case, there is no way to deliver the packets if you find that the gateway of this particular WAN is down.

cyayon · July 13, 2023, 6:49am

Hi,

I encounter a strange issue today.
I had a failure on my primary link and another failure on my dynamic ip update script for my secondary link…

When the primary link came back, i tested my secondary link input rules and routes (when my 2 links are up and running).
A simple curl from an internet test server to my secondary ip link (LTE) took very long time (at least 20-30s) to answer.
However, output (NAT) from LAN to internet via the secondary link do not have any issue and work as expected.
Of course i do not have any issue on my primary link (input/output).

I defined a simple route in main table to force output to the server where i tested input (curl) and after that, it worked as expected. The requests on the secondary link answer very quick as expected.
In conclusion, there seems to be an routing issue, but i do not understand where i made a mistake.

My router is a CCR2116 with RouterOS 7.7.
Here is the config in attachment, could you please check if everything is fine ?

EDIT1 : i tried to use tools/Torch to check, and as soon as i enable Torch, everything work perfectly ! I stop Torch issue come back again !
EDIT2 : upgraded to the last 7.10.2, no change. I even tried to completely disable recursive monitoring for primary link, (and only use distance) no change
EDIT3 : in real failover (when primary link is really down), no issue, the secondary ip (failover link) work and answer very quickly
EDIT4 : using packet sniffer tool has the same effect that Torch. As soon as i enable, it works perfectly. It seems that it is the promiscious mode that make it works. VERY strange…
EDIT5 : opened a support ticket SUP-121920

thanks !
router1 (1).rsc (5.69 KB)

cyayon · July 18, 2023, 10:11am

Hi,

Nobody with this strange issue ?
Thanks

anav · July 18, 2023, 1:12pm

Post the full config and I will have a look, wont waste my time on half the evidence

cyayon · July 18, 2023, 2:08pm

Hello,

Here is the config. I just removed some sensitive informations.

thanks.
router1.rsc (58.8 KB)

cyayon · July 28, 2023, 11:31pm

Hi,
Nobody have an idea for this issue ?
No answer from the support

Just an idea, could it be related to fasttrack ?
I read somewhere that Fasttrack and mangle dual-wan are incompatible, is it a mistake ?
If fasttrack is the issue, is there a workaround to keep fasttrack only for some packets of and keep mangle / routing rules/tables ?
Perhaps with some additional filters on fasttrack rule (exclude input interface or not marked packets)?

Thanks

sindy · July 29, 2023, 6:21am

The very principle of fasttrack operation is that the fasttracked packets bypass some stages of packet processing, mangle is just one of these stages.

Sure there is: http://forum.mikrotik.com/t/static-default-route-im-missing-something/119183/9

cyayon · July 29, 2023, 7:42am

Hi,

Many thanks Sindy for your answer.

I thank I was using mangle rules right, but as an evidence, it is not…
My issue is for inbound packets from internet via WAN2 (when WAN1 is NOT down).

I do not understand where I made a mistake :
vlan832-orange1 is wan1
ether2.wan2 is wan2
interface-list WAN contain wan1 and wan2

current version :

/ip firewall mangle add action=mark-connection chain=input connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=input connection-state=new in-interface=ether2.wan2 new-connection-mark=conn-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan1 in-interface-list=!WAN new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan2 in-interface-list=!WAN new-routing-mark=route-wan2 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=ether1.wan2 new-connection-mark=conn-wan2 passthrough=yes

a reworked version based on your post but it do not work :

/ip firewall mangle

add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface-list=WAN   action=accept # download packets MUST NOT be routing-marked

add chain=prerouting connection-mark=conn-wan1 action=mark-routing new-routing-mark=route-wan1 # passthrough=no is a default behaviour but you can state it explicitly
add chain=prerouting connection-mark=conn-wan2 action=mark-routing new-routing-mark=route-wan2 # same like above
add chain=prerouting action=mark-connection connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
add chain=prerouting action=mark-connection connection-state=new in-interface=ether1.wan2     new-connection-mark=conn-wan2 passthrough=yes

add chain=output action=mark-routing connection-state=new connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
add chain=output action=mark-routing connection-state=new connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

thanks for your help !

cyayon · July 29, 2023, 8:34am

Ok,
I finally managed to made it work with the following rules and fasttrack additional condition : connection-mark=no-mark.

/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan1 in-interface-list=!WAN new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=conn-wan2 in-interface-list=!WAN new-routing-mark=route-wan2 passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-state=new in-interface=ether1.wan2 new-connection-mark=conn-wan2 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

I suppose it is not optimized, could you please check ?
thanks !

sindy · July 29, 2023, 8:43am

I’m not sure I understand your intentions properly, so just some points:

in prerouting, you first assign the routing-mark to packets that did not come via WAN, and only the next rule assigns the connection-mark depedning on in-interface. From this, and from the absence of any rules distributing the traffic among the two WANs, I deduce that you only want to keep the connection on the WAN through which it has been initially established, and respond incoming connections from the internet using the correct WAN. Is that a correct understanding?
in both output and prerouting, you translate a connection-mark conn-wanX to routing-mark route-wanX. But whilst in input, you assign the connection-mark conn-wan2 based on in-interface sfp2.wan1, in prerouting, you assign it based on in-interface ether1.wan2. So one of these must be wrong, most likely the one in input as none of your routes has a gateway in 192.168.4.0/24.
from the moment a connection gets fasttracked, all its packets use the routes in table main (except those few that RouterOS chooses not to fasttrack). So if you want to use fasttracking e.g. only for connections that use WAN 1, table main must not contain any route via WAN 2, which effectively means that you need some other table than main to handle the initial packets of all connections so that the connection could be established via ether WAN depending on their current state. Once a WAN interface is chosen by routing using that “initial” table, you assign a connection-mark to the connection initiated by that packet, which will later be translated to a routing-mark. It is enough to assign the connection-mark conn-wan2 as packets belonging to connections without any connection-mark will not get any routing-mark even if they pass via mangle, so they will be routed using table main. And the action=fasttrack-connection rule must match on connection-mark=no-mark or connection-mark=!conn-wan2 to prevent it from fasttracking connections that must use WAN 2.
just for laughs, this is a chef-d’œuvre:
/ip route add comment=“main table” … routing-table=route-wan1

cyayon · July 29, 2023, 9:14am

Thanks for your answer.

YES (but I don’t know if it a good pratice)
in the last version, I removed the input mangle rules and it works (but I think this not optimized)
I do not understand everything, could you please give me a small example ?
for a moment, I thank I made a real mistake in my configuration But it is just a funny comment to remind me it is the route-table wan1 (which is main wan - not the failover).

What do you think about these rules ? are they right and more optimized ?
I suppose because of the two first rules which are not passthrough on established

/ip firewall mangle
add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface-list=WAN   action=accept # download packets MUST NOT be routing-marked

add chain=prerouting action=mark-routing connection-mark=conn-wan1 new-routing-mark=route-wan1 
add chain=prerouting action=mark-routing connection-mark=conn-wan2 new-routing-mark=route-wan2

add chain=prerouting action=mark-connection connection-state=new in-interface=vlan832-orange1 new-connection-mark=conn-wan1 passthrough=yes
add chain=prerouting action=mark-connection connection-state=new in-interface=ether1.wan2     new-connection-mark=conn-wan2 passthrough=yes

add chain=output action=mark-routing connection-state=new connection-mark=conn-wan1 new-routing-mark=route-wan1 passthrough=yes
add chain=output action=mark-routing connection-state=new connection-mark=conn-wan2 new-routing-mark=route-wan2 passthrough=yes

ÉDIT : Just to clarify, what I would like to do is relatively simple.
For outbound (lan to internet) use wan1 (main) and wan2 ONLY if wan1 is down (check wan1 with recursive routing…)
For inbound (internet to lan), wan1 and wan2 could be used from internet. I self-host some services…

sindy · July 29, 2023, 9:42am

It is OK if it is your intention (to keep a connection on the secondary WAN even if the primary one recovers).

Traffic that uses input chain also uses the output chain. So once you have removed the mangle rules from input, the mangle rules in output became useless. Own outgoing traffic of the router is only handled by output and postrouting chains in mangle, not by the prerouting one.

The rough idea is to create a routing table initial that will contain the two topmost recursive default routes (via WAN 1 and via WAN 2, preferring WAN 1), and to use mangle rules in output and in prerouting to assign a routing-mark initial to packets that match connection-state=new so that they would use that table.

There must be a lot of exceptions in these rules - in order that the monitoring of the canary addresses worked properly, you either have to exclude these addresses from the mangle rule in output, or you must move the complete set of recursive routes to table initial as well. And you have to prevent traffic from the router itself to devices in LAN from getting the routing-mark initial as well.
Routing table main will only contain the default route via WAN 1’s gateway and all the dynamically created routes to LAN subnets.

Routing table route-wan2 will only contain the default route via WAN 2’s gateway.