Neither the ‘ip’ nor ‘route’ commands appear to be part of the busybox binary that MikroTik ships with RouterOS, which is why I am interacting with the ‘proc’ virtual filesystem directly. I have plans to build and try a more complete, statically-linked busybox binary that includes the ‘ip’ command, though I have not checked yet to see how complete busybox’s version of ‘ip’ is.
– Nathan
EDIT: Update:
I now have a statically-linked busybox binary that includes both ‘ip’ and ‘route’. The busybox version of ‘route’ doesn’t support the -C parameter, but ‘ip route show cache’ does work, and if I run that on the box in question while the route cache size is going berserk, it doesn’t show anything abnormal: route cache size says 10000+ entires, but ‘ip route show cache’ only shows the 5 or so routes that I would expect to see on this particular router. So that’s interesting.
The problem (well, at least, my problem) is definitely related to MikroTik’s new PPP code, though. I’ll be attempting to put together a step-by-step method of reproducing the bug once I have it 100% nailed down, but right now it appears to be triggered if you are running a PPP-based server (PPTP, L2TP, SSTP, maybe even PPPoE, etc.) and you have several connections to it go up and down. Eventually, after one of the PPPs disconnect, it seems like there might be some kind of race condition that occurs when it tries to tear the PPP interface down. ‘dmesg’ output shows a huge number of messages like this on my router shortly after the route cache starts spinning out of control:
unregister_netdevice: waiting for ppp1 to become free. Usage count = 1
unregister_netdevice: waiting for ppp1 to become free. Usage count = 1
unregister_netdevice: waiting for ppp1 to become free. Usage count = 1
unregister_netdevice: waiting for ppp1 to become free. Usage count = 1
[...]
…and something just keeps repeating that message over and over again. This is despite the fact that ‘ppp1’ as an interface no longer exists:
# busybox ifconfig ppp1
ifconfig: ppp1: error fetching interface information: Device not found
…so it’s trying to unregister a device that doesn’t exist?
More bad news: once it gets to this state, it appears that it is impossible to flush the cache of the entries being added to it (I assume by PPP?). It would seem that something in the PPP subsystem has a lock on those entries and they can’t be freed. Tweaking with the route cache garbage collector values doesn’t make a difference, either…the number doesn’t go down. I know that those proc/sysctl values actually work because I tested them on a MikroTik with a fairly large route cache, but one that wasn’t spiraling out of control, and was able to successfully flush the cache and visibly see the garbage collector behavior change. Once this particular bug is triggered, however, the only thing that can cure it is a reboot. I even tried killing the ppp and ppp-worker processes, but although they cleanly exited after being sent SIGTERM, the route cache remained bloated and the “unregister_netdevice” console errors continued apace.
Finally, it may have something to do with MPPE. I notice that after the problem starts, the ‘ppp_mppe’ kernel module shows that it is in-use by something and cannot be unloaded, even after I have terminated all PPP tunnels and shutdown PPP services:
# lsmod | busybox grep mppe
ppp_mppe 5585 6 - Live 0x90d5d000
This might just be another symptom, though, rather than a cause, especially if people are also running into this problem with SSTP, which should have no use for MPPE.
EDIT 2: Well, now this is interesting. I think I have managed to find a way to reproduce a version of this bug, but now that I’ve done so, I tried manually flushing the route cache again, and this time doing so has an effect. It will continue to grow and grow on its own even after a flush, but executing the flush actually works this time and clears the cache (temporarily). Weird.