Netwatch duplicate ip route

Hello,

I’m facing a weird issue on some of my router (all 7.16.2), here is the behavior I aim to have :
My router is the .253/24 of the network, and the gateway sent from the DHCP server. My local gateway is the .254 while an access to network is available, I have a static route to the 1st host through .254 so even my gateway can’t match on it.
If the .254 is not able to connect me to internet I disable my local gateway to the .254 and I use another route to go.

My netwatch looks like :

/tool netwatch
add disabled=no down-script="/ip route disable [find dst-address=0.0.0.0/0 gateway=10.144.26.254 routing-table=VRF-Pro ]\r\
    \n/ip route disable [find dst-address=0.0.0.0/0 gateway=10.142.26.254 routing-table=VRF-Wifi ]\r\
    \n/ip route disable [find dst-address=0.0.0.0/0 gateway=10.141.26.254 routing-table=main ]\r\
    \n/ip address disable [find address=[:tostr 10.141.26.248/24] interface=vlan-data-vrrp-master]" host=HOST-FOR-PROBE http-codes="" interval=6s name=test-netrouter packet-count=3 src-address=10.141.26.253 test-script="" thr-max=2s type=icmp \
    up-script="/ip route enable [find dst-address=0.0.0.0/0 gateway=10.144.26.254 routing-table=VRF-Pro ]\r\
    \n/ip route enable [find dst-address=0.0.0.0/0 gateway=10.142.26.254 routing-table=VRF-Wifi ]\r\
    \n/ip route enable [find dst-address=0.0.0.0/0 gateway=10.141.26.254 routing-table=main ]\r\
    \n/ip address enable [find address=[:tostr 10.141.26.248/24] interface=vlan-data-vrrp-master]"

However every time it goes UP/DOWN my route act as the following :

/ip route print where static disabled=no
Flags: I - INACTIVE, A - ACTIVE; s - STATIC; H - HW-OFFLOADED; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#      DST-ADDRESS       GATEWAY        DISTANCE
1 As + 0.0.0.0/0         10.141.26.254         1
1 As + 0.0.0.0/0         10.141.26.254         1
1 As + 0.0.0.0/0         10.141.26.254         1
1 As + 0.0.0.0/0         10.141.26.254         1
2 As + 9.9.9.9/32        10.141.26.254         1
2 As + 9.9.9.9/32        10.141.26.254         1
2 As + 9.9.9.9/32        10.141.26.254         1
2 As + 9.9.9.9/32        10.141.26.254         1
3 As + PROBEHOST1/32    10.141.26.254         1
3 As + PROBEHOST1/32    10.141.26.254         1
4 As + PROBEHOST2/32  10.141.26.254         1
4 As + PROBEHOST2/32  10.141.26.254         1
5 IsH  0.0.0.0/0         10.144.26.254         1
6 IsH  0.0.0.0/0         10.142.26.254         5
6 IsH  0.0.0.0/0         10.142.26.254         5
6 IsH  0.0.0.0/0         10.142.26.254         5
6 IsH  0.0.0.0/0         10.142.26.254         5

Does someone already have seen a behavior like this ? The only way to clean the routing table is to reboot. I can’t upgrade the router as 7.16.2+ version have the weird behavior on conntrack and VRRP, and i can’t downgrade under 7.15.2 as its the factory version… I feel kinda stuck as the end client want a smooth backup.

Well, you should post the output of “/ip route print where static” (without the disabled=no) TWO times.
Once after the up-script has run and once after the down-script has run.

Post also the plain:

/ip route export

Set aside the change of address, the script disables and enables (should disable/enable) three routes to 0.0.0.0/0 in 3 different tables:

  1. routing-table=VRF-Pro
  2. routing-table=VRF-Wifi
  3. routing-table=main

Yet in the /ip route print where static disabled=no you have 4 of them (with distance 1)?

And you also have 5 IsH ones, of which 4 with distance 5?

I can understand that the ones with distance 5 can be IsH, but the one with distance 1?

Hello,

Before openning the post I shreded hard my configuration so I don’t have any issue of a misconfiguration from my side, as the routing-table main is the most important to backup.
And the for disabled route, its just leftover from my test.

But here is the full IP Route export and ip route print where static :

/ip route 
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.141.26.254 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=0.0.0.0/0 gateway=10.142.26.254 routing-table=VRF-Wifi scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.144.26.254 routing-table=VRF-Pro scope=30 suppress-hw-offload=no target-scope=10
add disabled=no dst-address=PROBE-HOST3/32 gateway=10.141.26.254 routing-table=main suppress-hw-offload=no
add disabled=no distance=1 dst-address=PROBE-HOST1/32 gateway=10.141.26.254 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=yes distance=2 dst-address=PROBE-HOST1/32 gateway=VISP-FTTH-Main routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=PROBE-HOST2/32 gateway=10.141.26.254 routing-table=main scope=30 suppress-hw-offload=no target-scope=10


/ip route pr detail where static 
Flags: D - dynamic; X - disabled, I - inactive, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, i - is-is, d - dhcp, v - vpn, m - modem, y - bgp-mpls-vpn; H - hw-offloaded; + - ecmp 
 0  Xs   dst-address=PROBE-HOST1/32 routing-table=main gateway=VISP-FTTH-Main immediate-gw=VISP-FTTH-Main distance=2 scope=30 target-scope=10 suppress-hw-offload=no 
 1  As + dst-address=0.0.0.0/0 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 1  As + dst-address=0.0.0.0/0 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 1  As + dst-address=0.0.0.0/0 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 1  As + dst-address=0.0.0.0/0 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 2  As + dst-address=PROBE-HOST3/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 2  As + dst-address=PROBE-HOST3/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 2  As + dst-address=PROBE-HOST3/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 2  As + dst-address=PROBE-HOST3/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 0  Xs   dst-address=PROBE-HOST1/32 routing-table=main gateway=VISP-FTTH-Main immediate-gw=VISP-FTTH-Main distance=2 scope=30 target-scope=10 suppress-hw-offload=no 
 3  As + dst-address=PROBE-HOST1/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 3  As + dst-address=PROBE-HOST1/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 4  As + dst-address=PROBE-HOST2/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 4  As + dst-address=PROBE-HOST2/32 routing-table=main gateway=10.141.26.254 immediate-gw=10.141.26.254%vlan-data-vrrp-master distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 5  IsH  dst-address=0.0.0.0/0 routing-table=VRF-Pro gateway=10.144.26.254 immediate-gw="" distance=1 scope=30 target-scope=10 suppress-hw-offload=no 
 6  IsH  dst-address=0.0.0.0/0 routing-table=VRF-Wifi gateway=10.142.26.254 immediate-gw="" distance=5 scope=30 target-scope=10 suppress-hw-offload=no 
 6  IsH  dst-address=0.0.0.0/0 routing-table=VRF-Wifi gateway=10.142.26.254 immediate-gw="" distance=5 scope=30 target-scope=10 suppress-hw-offload=no 
 6  IsH  dst-address=0.0.0.0/0 routing-table=VRF-Wifi gateway=10.142.26.254 immediate-gw="" distance=5 scope=30 target-scope=10 suppress-hw-offload=no 
 6  IsH  dst-address=0.0.0.0/0 routing-table=VRF-Wifi gateway=10.142.26.254 immediate-gw="" distance=5 scope=30 target-scope=10 suppress-hw-offload=no

And for the “VRF-WIFI” interface, its currently disable as I just aim to make the router work in a first place. And the host 10.144.26.254 is not up yet (for the IsH route #5).

Main issue is that i can’t make any edit on the route with #1 as it keep being duplicated…

Yep, now what is happening is clear, how to change this behaviour is another thing.
Pairing the routes in /ip route export with those in /ip route print (simplified) and matching them with the netwatch script lines (see the attached image) it seems to me clear that each time the down-script is run the interfaces are not always disabled, and when the up-script is run somehow the routes are re-added (creating the duplicates).
BUT the results of the SAME netwatch line that changes (should change) the enabled status of 4 different routes is not always the same for all the 4, possibly because when the script is run some interfaces are duplicated more than the others.
The two lines that “catch” only one route also seemingly behave differently, the VRF-Pro one seemingly works as intended, while the VRF-WiFI one creates the duplicates.
At first sight I though that the issue was with the command catching multiple routes, but since also one the “single” ones behaves the same, it is something else.
I would suspect that the disable action doesn’t take effect immediately because of whatever (established connections?) and when the down-script/up-script are run in a too short time interval the duplications are created, but this is only a guess, not that I know or understand what is actually happening.

Are you using src-nat or masquerade in /ip firewall nat?

The first usually needs to have the connection tables cleared, whilst the latter should do everything automatically, see the (seemingly unrelated) discussion starting here:
http://forum.mikrotik.com/t/simpler-failover-for-two-gateways-i-found-working/169108/1
enale_disable_routes.jpg

No there is no SRC-NAT on this router/masquerade as the nat is global for all the client have a firewall above and i’m just here to gather and route the lan2lan.

As I saw on the linked topic, the route selected through the find with the comment, I will edit my netwatch to check if its working better this way, I will update tomorow.

As update and few more test, having a netwatch with the find on the comment seems to have helped.

I will keep it in mind with the following “”“feature”“” :

  • Have a find on a ip route enable for 3 VRF routing table break the enable and duplicate routes… making them impossible to delete or disable easily.
  • I should apply way more often the Keep It Simple Stupid concept… I would avoid so many strange behavior.


    Thank you Jaclaz for the help :slight_smile:

You are welcome :slight_smile: , happy you found a way out.

Only thinking aloud, but I would try to change the distance of the routes (as opposed to disabling them).
This way you would have another matcher (the distance).
In the down-script you would change the distance from 1 to 10 (or from 5 to 50), and in the up-script you would revert them.
BUT no idea if it would change the behaviour.
Or if some other field can be mis-used as a matcher.

Personally I am not a fan of using comment as a matcher as the future you (in 6 months time, or 1 year) may forget that the comment needs to be “carved in stone” and change it, after all it is just a comment .. :wink: .