Community discussions

MikroTik App
 
vmsh0
just joined
Topic Author
Posts: 8
Joined: Tue Nov 20, 2018 9:59 pm

Recursive routing breaks interface-specific locally-originated traffic.

Wed Sep 22, 2021 5:13 pm

Hey everyone.
I'm using a script to provide automatic failover. To make it work without having to bake all the routing information into the script, is with the following recursive routing setup, divided into three levels:
  1. 0.0.0.0/0 routes for each gateway, for each routing table needed (incl. the default table). Their gateway is set to 127.100.1.ID, where ID is a number unique for each uplink. These rules have scope=30 target-scope=10, and their distance is dynamically updated by the failover script. So, basically, traffic will use the 127.100.1.ID decided by the failover script.
  2. the target for level 1. These are either dynamic 127.100.1.ID PPP routes for PPP links (the route address is set by the PPP profile remote-address option), or DHCP-script created static routes of the form 127.100.1.ID with gateway set to the gateway provided by the DHCP server, for DHCP links. These have scope=10 target-scope=10. Their distance doesn't matter as they are all disjunct.
  3. only for DHCP links: the dynamic entries for the DHCP addresses, added automatically when the address is assigned. These will contain both the router address and the DHCP-provided gateway address, which has been set as the gateway for the Level 2 rule. Their gateway obviously is the DHCP interface (e.g. etherX).

Now, the setup works perfectly for PPP interfaces. However, it breaks in two ways for DHCP/eth interfaces:
  1. when the DHCP script updates the level 2 route with a new gateway address, the recursion on level 1 is not recomputed. However, this is very easy to solve, by just disabling and re-enabling the level 1 rules for the particular level 2 gateway in the DHCP script
  2. RouterOS is not "smart enough" to work out directly-connected networks in this situation, for interface-specific locally-originated traffic. For example, if I try to ping 8.8.8.8 from a DHCP interface set up like this, RouterOS sends an ARP request which obviously goes unanswered. If I disable ARP for the interface, RouterOS just sends out all packets by setting the destination MAC address to its own, so obviously the gateway doesn't pick those up. However, if I add a static ARP entry for 8.8.8.8 with the gateway MAC address it works! (meaning that the gateway works just fine).

Let me give you a more practical example of how the routes might look like for a DHCP interface (omitting additional routing tables - should be irrelevant). Let's say the router is assigned 192.0.2.1/30 and the gateway is 192.0.2.2/30, on interface ether8. The routing table would be set up as follows:
Route         Gw           Gw (as computed recursively by RouterOS)    Note
0.0.0.0/0     127.100.1.5  127.100.1.5 recursive via 192.0.2.2 ether8  Level 1, static route
127.100.1.5   192.0.2.2    192.0.2.2 reachable ether8                  Level 2, created by DHCP script
192.0.2.0/30  ether8       ether8                                      Level 3, automatic route for address
As you can see, the default route is resolved just fine through the latter two routes. However, as mentioned, this doesn't work, unless the router is tricked into thinking that the address he's trying to talk to is directly connected. These are the solutions I tried:
  • Static ARP entries. However, I can't have 2^32 of them :)
  • Disabling ARP, wrapping the interface inside a bridge, and using bridge NAT to DNAT all outgoing packets, rewriting the destination MAC address to the address of the gateway. This should work in theory, but DNAT doesn't support specifying the output bridge, so it's probably too resource-intensive for having a simple backup connection on standby 99% of the time. Furthermore, it doesn't work in practice: it appears that locally-originated packets are not matched by the bridge firewall rules, even when disabling hardware offloading.
  • Leaving ARP enabled, wrapping the interface inside a bridge, and using bridge firewall to craft ARP-replies for any IP address that the router requests. Again, this should work, but locally-originated packets seem not to be matched.
This leaves me with a DHCP connection that I can't use, which is disappointing! Please note that it doesn't look like the problem is that there are three levels of recursion: the problem stays even with the standard "default gateway" rule generated by the DHCP client (if enabled). It seems to me that, simply, when a packet is locally-generated and interface-specific, a route is selected based on the chosen interface, and the gateway address is just ignored. Which means that when L3->L2 resolution takes place, the only information available is "8.8.8.8 is reachable on ether8", instead of "8.8.8.8 is reachable on ether8 through 192.0.2.2"; so 8.8.8.8 (rightly, from the point of view of the L3->L2 resolution mechanism) gets ARPed instead of 192.0.2.2.

Now, why do PPPs work nonetheless? That's easy: packets exiting a PPP interface don't use L3->L2 resolution, as you only have one peer by definition (i.e. there's no L2 inside the tunnel). They are just thrown out.

Now, how to fix this? I honestly have no clue. I am inclined to think that there might be some firewall-fu that could be done. But even if there is, I'm not going to do it. This is quite simply a bug! Chosing the outbound interface should not result in information from the routing table being ignored.

Any ideas? (No, not the loopback trick :)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Recursive routing breaks interface-specific locally-originated traffic.

Wed Sep 22, 2021 5:36 pm

A network diagram would help.
/export hide-sensitive file=anynameyouwish
 
vmsh0
just joined
Topic Author
Posts: 8
Joined: Tue Nov 20, 2018 9:59 pm

Re: Recursive routing breaks interface-specific locally-originated traffic.

Wed Sep 22, 2021 6:06 pm

A network diagram would help.
/export hide-sensitive file=anynameyouwish
The network diagram is literally just a RouterBOARD connected directly to a gateway which hands out addresses via DHCP. I apologize for not providing it, I haven't good any tools handy atm.

The configuration is 99% useless stuff and 1% what I just described, but it does contain the scripts, which are probably handy to reproduce this. I attached it.
Last edited by vmsh0 on Wed Sep 22, 2021 8:53 pm, edited 1 time in total.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Recursive routing breaks interface-specific locally-originated traffic.

Wed Sep 22, 2021 6:27 pm

Yeah, I find your config one big confusing mess.
Probably you haveve some very legitimate and interesting things going within the router as its very complex.
I am not sure so much inter router routing is required if on subnets.................
but not sure why LTE is part of the bridge ports? Can it be your providing a Wan within a LAN ??
( I certainly cannot begin to fathom the src nat rules for example )

Personally would use another vlan instead of using bridge for DHCP and its subnet............
But the biggest issue I have is the scale of bloated firewall rules for protection but
you dont have the basics of any input chain firewall ???
 
vmsh0
just joined
Topic Author
Posts: 8
Joined: Tue Nov 20, 2018 9:59 pm

Re: Recursive routing breaks interface-specific locally-originated traffic.

Wed Sep 22, 2021 9:01 pm

Yeah, I find your config one big confusing mess.
Probably you haveve some very legitimate and interesting things going within the router as its very complex.
I am not sure so much inter router routing is required if on subnets.................
but not sure why LTE is part of the bridge ports? Can it be your providing a Wan within a LAN ??
( I certainly cannot begin to fathom the src nat rules for example )

Personally would use another vlan instead of using bridge for DHCP and its subnet............
But the biggest issue I have is the scale of bloated firewall rules for protection but
you dont have the basics of any input chain firewall ???
Just for reference, 80% of the configuration is straight from MikroTik's, including >90% of the bloated rules. I mean, of the firewall configuration. The rest is simply stripped of context: as I mentioned, I had to remove some sensitive stuff. But then again, all of this is irrelevant, because I didn't ask you if my configuration was to your liking. What's worse, is that you didn't even look at the config all that well. You say I don't have an input chain. That's entirely correct: as you can (could) see, the bad stuff is statelessly dropped before reaching the stateful parts of firewall processing.

Anyway, I have now removed it because... it's just not relevant. I shouldn't have posted it in the first place, but it seemed disrespectful because I thought you were trying to help me. If anyone would like a minimum configuration for the convenience of quick testing, I will do my best to find some time and provide it.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Recursive routing breaks interface-specific locally-originated traffic.

Wed Sep 22, 2021 9:43 pm

Assuming you understood all the items you copied and stuck into your config, then you will have to wait for someone else that has a much higher level of knowledge of your settings to assist.
I prefer the default firewall rules and a clean config, not much else because in 98% of cases thats all that is required.
Each to his own.....

Who is online

Users browsing this forum: BinaryTB, Bing [Bot], Google [Bot] and 81 guests