We are running a number of MikroTik routers in a central office and branch offices.
There is a partial mesh of tunnels between the routers (branch offices all connect to central office but there are some tunnels between branch offices).
Sometimes there are 3 tunnels between the same routers: one GRE/IPsec, one GRE6/IPsec, and one L2TP/IPsec.
eBGP is used to distribute routes and arrange failover. the L2TP/IPsec tunnels are over LTE and need to have a lower priority (only used for traffic when other paths have failed). I have used local-pref and bgp-prepend to implement that, but the result is the same.
What I observe is that this system, which worked well with one central office and 3 branch offices, is failing now that we have added more branch offices.
It appears that at some time, BGP considers “enough is enough” and on some connections the “number of prefixes” remains at zero, instead of the 22 that it should be for our network.
Looking in the route table, indeed the routes via that tunnel are not present. Although the BGP connection is up.
Sometimes, by forcing the connection down/up this can be “fixed” but it is unstable.
It looks like there is some limit on the number of “available but not active paths” in BGP, but I have been unable to locate such a parameter (to increase it).
What could this be?
I tried to use route filters to filter some routes, e.g. to accept routes between branch offices only for bgp-path-len=1, because that would be sufficient for the use case, but it does not help. The extra routes remain in the table, but are no only inactive but also in “filltered” status. That does not appear to make room for more routes.
The problem arising from the above is that when a primary tunnel (over fiber or DSL) fails, sometimes the backup tunnel has no active routes so it does not function as a backup, at least not immediately.
This worked OK before, and I am not sure what really triggered the problem: expansion of the network, or upgrade of RouterOS (currently at 7.16).
What can I do to have all prefixes accepted on all connections?