The route cache feature was removed from Linux in version 3.6, apparently because it was not very effective and it caused
problems and vulnerabilities all the time. However, RouterOS is still using an older kernel. Maybe MikroTik could
consider backporting the removal to their older kernel, or speed up the migration to a more recent kernel.
This is of course all “planned for RouterOS v7”, which will have a new kernel and thus no more route cache.
But we know that is postponed all the time.
As I have written before, I think they should release a new version with updated versions of all the components that are
used without major modifications, and then introduce the new features incrementally during minor releases and optional
packages, instead of the promised “you’ll get v7 and it will solve all problems because major subsystems have been
rewritten from scratch”. That model just will never work, and result in indefinite postponement of v7.
Of course by now they can not even release a v7.0 that is just a v6.xx with updated kernel and tools, because of the
highly risen expectations…
In fact, in my experience setting it to “no” tends to make the route cache fill faster.
Out of nearly 200 ROS devices on our network, only three have this problem of continuous route cache filling. They are all different hardware (RB2011, OmniTik ac, RB433GL) performing different network roles, none carrying an enormous amount of network traffic. They can be running any ROS version from 6.39 to the present, and the most it seems to affect is the rate of filling. I have seen one take two weeks, and I have seen one take five hours.
The frustrating part is that our only option of dealing with this condition, even if we see it coming, is to reboot the device.
I’ve been adding a new supout to my correspondence with Mikrotik almost daily, along with regular appeals to implement:
/ ip route cache clear
I have yet to receive a response that addresses this idea, or even acknowledges that this is a problem that they need to look into. I can’t see how a “clear” command would be difficult to implement–and even if doing so were to take the router offline for 30 seconds to repopulate its local routes and OSPF to settle down again, that’s still a big improvement over rebooting.
Replacing the hardware with similar or identical devices has not eliminated the problem. It’s also unclear whether this problem started as soon as the devices were installed, or whether it developed over time. We also have devices that used to have this problem, that no longer do, without any hardware or configuration changes. For these reasons, we can’t rule out interactions with other devices (e.g., OSPF) nearby, as a cause.
The three devices having this problem today are all in production, so we aren’t in a position to perform extensive experiments. Devices on the test bench with identical configurations don’t exhibit this problem, but we also aren’t prepared to duplicate the hardware environment and traffic encountered.
My idea to address the issue was to move from a VM to a physical hardware, but I have the issue in both cases.
My device is a OVPN server with 10Mb/s avg traffic. It fills in less than a week.
From my experiments, when the cache reaches 25% (both VM and Physical HW, that’s strange) the bug arise.
The only workaround is to schedule the following running every night:
:local uptime [/system resource get uptime];
:local date [/system clock get date];
:local time [/system clock get time];
:local identity [/system identity get name];
:local freememory [/system resource get free-memory];
:local cpuload [/system resource get cpu-load];
:local freehdspace [/system resource get free-hdd-space];
:local cache [/ip route cache get cache-size];
:local maxcache [/ip route cache get max-cache-size];
:global datum [/system clock get date];
:local percentused ((100 * [/ip route cache get cache-size]) / [/ip route cache get max-cache-size]);
:log info "RouteCacheUsed: $percentused %";
:if ($percentused > 17) do={
/tool e-mail send server="smtp.gmail.com" port=587 start-tls=yes user="XXXXXXXXX" password="XXXXXXX" from="XXXXXX@gmail.com" to="XXXXXXX@gmail.com" subject="Cache ip4you server full" body="Date: $date, Time: $time\n\nResources\n\nuptime: $uptime\nfree memory:$freememory\ncpu load: $cpuload\nfree hdd space: $freehdspace\ncache: $cache\nmax cache: $maxcache\nPercent used: $percentused %"
:log info "reboot";
/system reboot;
};
I have several routers, including CCR1009, operating in various different scenario’s which do lots of routing and also VPN (GRE and L2TP) and I have never seen this problem.
Checking the CCRs in operation the use of the route cache is very small compared to the size.
I think there must be something particular to the configuration or usage of the affected routers.
Our other 150+ ROS devices don’t have this problem either. Most of them operate for weeks and months at a time, and their route caches never come close to 1,000.
But three of them do. And as far as we can tell, none of them are configured in any unusual way; use any ROS feature that the other 150+ don’t use; perform any particular function or carry any particular traffic that sets them apart from the rest. If there were anything distinctive about them, that’s where I’d be looking. But even near-identical ROS devices nearby–the OmniTik on the roof of the next building, performing exactly the same functions; the ROS device at the other end of the 5GHz link, carrying exactly the same traffic–don’t exhibit this problem.
So, until someone can figure out what particular to the configuration or usage is affecting them, our only option is to reboot when necessary; if possible, at an hour when it will not cut off someone’s phone call or leave them offline for a couple frustrating minutes.
this is a very good news, I was not aware of this, thanks.
Do you think there is a better workaround (I mean script so schedule frequently) to avoid rebooting when cache reaches some size?
As a follow-up, we have identified two causes for our route-cache problem: one of which fills the cache quickly, and one which we believe fills the cache slowly:
We found two RBs with the same OSPF router-id in the same area. Resolving that conflict vastly reduced the rate at which certain route caches were filling: from a matter of hours to a matter of weeks.
We suspect, but have yet to confirm, that our remaining route-cache problem is caused by using dynamic WDS on links that drop and reconnect every hour or two.
Is there any plans to resolve it in near future?
Routers can’t even work 1 week with many ovpn tunnels… ip route cache get full and router needs restart… @Mikrotik please react.
Hello, I’m running ROS 6.45.6 on CCR1009 and problem still persist… the cache increase when the OVPN clients try to connect, for example if add various clients (in my case more than 100 PPP > Secrets) and disable, the clients try to connect, this start to increase the route cache, the unique solution is a script to reboot the CCR when the cache is running low.
Please fix this issue. my clients don’t stop crying.
I have some equipment (CCR, 4011, 2011) that all have active PPP services (openvpn / pptp ..etc).
We noticed that even using version 6.47.7 which until the day of this post is the latest stable version available, we still find problems with the uncontrolled increase of the routing cache.
Does anyone know of a possible correction date or a possible trick to solve the problem (without a reboot).
Even on 6.47.8, we’re still getting cache overflows on the same devices. They tend to be close (topologically) to radio links that are a bit unstable, which generate numerous OSPF updates.
It was quite a while ago that we last disabled route caching, and found that the cache filled up FASTER than when it was on.
We now run a script hourly on those RBs to notify us when the route cache has exceeded a threshold, and automatically reboot at 3am. So we know the problem is still happening, but (fortunately!) on devices that aren’t mission critical.