Static Default Route - I'm missing something

Sure there is. But to do that, it is not sufficient to use only routing marks, you have to use connection marking. And once you do, it is best to base routing marking on connection marking as well.

The action=fasttrack-connection rule in /ip firewall filter matches on both upload (LAN to WAN) and download (WAN to LAN) packets. So you have to mark the whole connection, i.e. all its packets in both directions, as you mangle its initial packet if you want to use non-default handling for that connection. Connections you don’t mark when they begin will be handled the default way.

Therefore, start by identifying (using a qualified guess or mangle rule statistics depending on your particular case) the kind of traffic which has the highest volume. This is the kind of traffic you want to use default handling for, so that you wouldn’t need to assign routing marks to it, so that you could use fasttracking for it. All other kinds of traffic will get some connection marks and, eventually, also routing marks based on them.

Then, you add a match condition connection-mark=no-mark to the condition list of the action=fasttrack-connection chain=forward connection-state=established,related rule in /ip firewall filter. Mission completed, only packets without any connection mark will now be fasttracked, so once you set up the connection marking rules properly, you can enable this action=fasttrack-connection rule.

Now the details.

  • You must assign (or not) the connection mark only when mangling the initial packet of each connection, so the action=mark-connection rules must conform to connection-state=new match condition; this can be ensured either by placing that condition into each rule or by preventing non-conformant packet from reaching that rule
  • for upload packets you need to add a routing mark also to the initial packet which has just been connection-marked. As you cannot assign both routing mark and connection mark using a single rule, the action=mark-connection rules must not stop the handling of the packet in prerouting, hence add passthrough=yes to them
  • you want to connection-mark each connection’s initial packet only once, so if you use more than one non-default handling and hence need more than one connection mark, it is always better to put a connection-mark=no-mark match condition to all action=mark-connection rules except the first one to avoid hard to understand mistakes. Normally you are used to that if rule N matches on src-address=192.168.0.3, rule N+1 just below it may match on src-adddress=192.168.0.1-192.168.0.9 and it will never actually act on packet with source address 192.168.0.3 because rule N won’t let the packet with that source address ever reach rule N+1. This usual behaviour doesn’t work if rule N has passthrough=yes, which is necessary so that the packet could reach some later action=mark-routing rule.

To speed up the processing as much as possible also for packets which are not fasttracked, you need to process each packet by the least possible number of rules in mangle, which means that the action=mark-routing rules should be as close as possible to the beginning of the chain, but at the same time you need them after the action=mark-connection rules, which simply means you have to put them to the chain twice.

So it’s time for an example. Assuming you have one default handling and two non-default ones, your mangle rules would like as follows:

/ip firewall mangle
add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface=your-wan action=accept # download packets MUST NOT be routing-marked
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A # passthrough=no is a default behaviour but you can state it explicitly
add chain=prerouting connection-mark=handling-B action=mark-routing new-routing-mark=handling-B # same like above

#only initial packets of connections (plus some garbage) get here past the rules above
add chain=prerouting ...list of classifying match conditions for handling A... connection-state=new action=mark-connection new-connection-mark=handling-A passthrough=yes
add chain=prerouting ...list of classifying match conditions for handling B... connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-B passthrough=yes

#initial packets of connections which evaded both the rules above get here with no connection mark; we just repeat the mark-routing rules above
add chain=prerouting connection-state=established,related in-interface=your-wan action=accept # download packets MUST NOT be routing-marked
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A
add chain=prerouting connection-mark=handling-B action=mark-routing new-routing-mark=handling-B

NB: connection marks, routing marks and packet marks use separate name spaces, so connection mark XYZ has no relationship to routing mark XYZ unless a mangle rule assigns the latter based on the former.

EDIT to have everything in a single post: if the various handlings are used to control multi-WAN arrangements, and if the router itself or devices on its LAN should act as servers accessible by clients in the internet, it is necessary to assign connection marks also to initial packets coming in via the WAN interface, so that the response packets would be routed through the same WAN interface. The default handling may use one of the WAN interfaces as outbound route; in such a case, connections initiated by packets coming in via that WAN interface get no connection mark. EDIT 2: fixed the template rules to properly handle the scenario where connections are initiated from the WAN side - thanks @Braddock. Also, see post #30 for a more compact approach.