Hey guys

So here's my question in a bit more detailed way:
Lets say we have a routing table as follow: Route C and D are connected routes.
We will also assume both Route A and Route B could be used to connect to 192.168.3.1

The question is that how Route B, being valid or invalid, affects the already established connections to 192.168.3.1

If Route B is valid and we establish a connection, the connection goes straight to Route B. and if meanwhile, Route B becomes invalid, the connection breaks immediately and new connection have to be made which goes through Route A.
But if then Route B becomes available again, those connections that are already established through Route A while Route B was unavailable, wont break immediately but with a delay (between a minute or two it appears) . Now i can only associate that with routing cache.

Here's what I think is happening: If the packets are going through Route B and it suddenly becomes unavailable, since it was the active Route, FIB will have to do the routing decision again. but if they are going through Route A and Route B suddenly becomes available again, since Route A is still valid, FIB will not refresh the cache immediately but only after the routing cache entry expires so new routing decision could made .

It also seems such situation, does not apply to ping. and ping packets, always travel the right Routes in time. which could suggest, routing cache does not apply to icmp packets.

Now my question is, am i right and is this what's happening? or did i miss something?
And exactly how long does it take till routing cache expires and FIB starts to do routing decision again?

PS: Although it seems quite obvious, my goal was to create a fail-over system. and it appears that its working. but i would like to know more about this behavior.

Thanks in advance

The functionality that you are describing is called connection tracking. That is in "/ip firewall connection-tracking". You can set the timeouts for certain kinds of traffic in there. But yes basically what happens is the router remembers what route it used for a connection, and it will continue to use that route for all packets in that connection. Once that session is closed it will switch back to the main route that you have in place.

You can clear out this table with a script to force it to switch over like when your main route fails if it is desired.

Hey Feklar. thank you for your reply, i appreciate it

connection-tracking was actually my first thought as well. since its clearly the case for nat and mangle rules, and its enabled in my router. and you are also right that if i close the connection, the new connection goes through the right route. however, there is something that doesn't add up. if its connection-tracking that is doing the routing cache, all the packets from an active connection, should go through the same route, without exception. but as i said, even active connection's being drop by router after couple of minutes.
to make it more clear, i did another testing:

Lets say we are downloading ubuntu
at the time of writing, gb.releases.ubuntu.com points to 194.169.254.10 . as soon as you add a route for that destination, the active connection, will be stuck. I also tested that by adding a route for that destination with CHECK_GATEWAY enabled, and tried to make it valid/invalid by blocking icmp requests to the specified gateway. same result.

Now that tells us that at the very least, the routing cache is being updated even for packets in the same connection regularly. which could suggest, connection-tracking either got nothing to do with routing cache or when it comes to routing, connection-tracking updates its cache even for packets belonging to an existing connection.

So it seems routing cache for tcp packets, just like icmp ones, is being updated pretty fast. the behavior about my original post, which takes up to couple of minutes for the router to drop the connection from an old routing table, might have something to do with the fact that the established connection is udp

Please let me know your thoughts

I'd be VERY surprised if connection tracking is used for route-cache... The one has nothing to do with the other as far as I am aware...

Hi Chris

Yes, it does appear that they are different things. as it wouldn't probably make so much sense to have 2 caching system for routing. there's already routing cache, why would you want to introduce another one in connection-tracking ?

From the manual:

Results of routing decision are remembered in the routing cache. This is done to improve forwarding performance. When another packet with the same source address, destination address, source interface, routing mark and ToS is routed, cached results are used.

But it does seem that the cache is being refreshed from time to time. it would be really good to know how exactly the system works. and why i don't get the same result with udp as i get with tcp or icmp when i change the routing table.

Edit: Found This article, Its a great read, explaining how routing cache works in-depth. it would be nice if we were able to adjust some of those parameters to match our networks. Also, there are some cool Routing tools that would be really handy if Mikrotik added them. specially 'ip route flush' and 'ip -s route show cache'