Mangle policy based routing

[lurker888]

If I follow the packet flow I indeed now understand that packets originating from the router will not pass the prerouting chain. The do pass the output and postrouting chain, but that is after the ‘routing decision’, hence of no use. Makes sense!

Glad you understand. But I’m not sure you do so completely. When an output packet is generated it goes straight to the “routing decision” stage. It doesn’t pass any sort of mangle chain before that. This has two consequences:

• In tems of routing: If by some other means (application level software, TCP stack, etc.) the packet has a routing mark, then it will be observed, but otherwise the routing decision is based on the main table. However then it passes the “postrouting” chain where a routing mark can be assigned. Then there is a “routing adjustment” phase where routing is corrected based on any marks applied in postrouting. So if you mark your packets in “postrouting output”, then the packet will undergo a next round of routing, and all will work out well for you: the packet will exit according to the routing rules/tables/vrfs that you would want. (BTW This leads to a strange behavior that if there is no route to a given dst in the main table, the packet will be instantly blackholed, even if - were it to go through postrouting and routing adjustment - it could be routed successfully. So a route in the main has to exist, even if it points nowhere Btw of btw: There is actually a remark about this in the Mikrotik docs, but if you don’t know the full story, you would never notice.)

• The other - and in your current case more important - story concerns the source address of packets. Source addresses are normally assigned either (1) by the application layer (e.g. a DNS server receives a UDP query, it should clearly specify (to the kernel) as the source address of the reply the address the query was received on), (2) by other networking layers, e.g. TCP will originate packets from the address the SYN was received on, or from which the SYN was sent during the handshake. But if still a packet has no source address (often referenced as the “address of last resort”) the packet acquires the “pref src address” of the route it takes. And here’s the strangeness: this assignment happens during the initial “routing decision” and is not (can’t be) corrected in the “routing adjustment” phase. Wireguard - by its design - doesn’t assign source addresses to its outgoing packets, which leads to a situation where if the packet is to be redirected in the “mangle postrouting” chain, it will go out on the correct interface, but with the wrong source address. Happy world.
  
  
  

I also now understand that Wireguard is a bit different from other interfaces.

A few small questions for my understanding:

• you are referring to the Linux iptabels diagram. That is indeed much easier to read/decode. But how do you know that Mikrotik is following the iptables implementation?

Because it is. If you look closely, they line up kind of perfectly. Just disregard the “security” chains, because they are only there when SELinux is used. Use both, whichever is clearer for the particular issue you are trying to resolve.

There is one mismatch between the diagrams. The Mikrotik diagram is missing the (seldom used) “input src-nat” stage. This is an error in their diagram; their routers actually apply it correctly, as it is written in the kernel codebase.

• below you mention “As is quite clear from the diagram referenced.”. I assume you refer to the iptables diagram. Unfortunately I do not understand/see that. Could you nudge me in the right direction?

The other is wrt wireguard. Wireguard has a bit of a strange behavior in that it selects the source address for its packets based only on the main routing table. (As is quite clear from the diagram referenced.) And thus while routing is altered as a result of mangling later (and your packet is transmitted on the intended interface), its source address is incorrect.

• neither do I understand from the above Mikrotik diagrams how Wireguard fits in. Do you?

I hope the above explanation clears up the quoted part of my post.

As to how wireguard fits in. Wireguard is an application level service (although implemented in kernel space), so packets to it enter at I and exit at J, its output packets enter at K and exit at L.

But I guess you are both referring to the same issue regarding dual Wireguard interfaces on dual WAN. I hope I can figure it out do create the

… ahem … creative natting

thing.

It’s not totally clear from your description, but I gather that both connections are initiated from your device. (I.e. it does not have to contend with incoming wireguard connections.) In this case - if you are willing to follow my recommendation of using a single wireguard tunnel that is automatically rerouted in case of failover - the “creative natting” is not necessary. (Automatically in this sense means that it will use the active default route and no special configuration is needed, so if your other traffic is working correctly - so will the wg tunnel)