Issues related to upgrading, downgrading and infected routers

Hello Everyone,

I work for a company that has over 1000 Mikrotik routers deployed for clients across North America. After we recently became aware of the vulnerabilities in older firmware versions, we started to upgrade our routers in small batches. Initially, we had come accross some that did seem to be infected, but upgrading seemed to resolve the issue, as the firmware notes said it would. But we noticed that after upgrading some of the properties that were previously having no issues started to have seemingly random problems. We had some that would not hand out DHCP addresses, some would see much higher CPU usage than prior to upgrading, some were not allowing devices to authenticate on the hotspot, etc.

Over the last few days now, we have come across some routers that seem to have been infected and had web proxy settings configured, this is something we do not normally use. Upgrading the firmware does not remove this, and we have to disable and remove these settings manually. This is also when we noticed that our splash page for the hotspot was being blocked, anyone who hit the walled garden would get an “error: forbidden” message. We also recently noticed that scripts would be added and scheduled, and we would see the following message in the log:

fetch: file "autosupout1.rif"

All of this seems to persist after firmware upgrade. We have had some sites become so problematic that we would have to downgrade afterwards.

We are seeing these issues across mostly tile, mipsbe, and ppc devices.

Can anyone offer any insight on this?

ROS and firmware upgrades only patch the vulnerabilities through which routers got infected. If infection changed configuration or added scripts, then you need to review them manually. There’s no way upgrade could determine which part of setup is intended by legitimate admin and which part is added by"other" admin.

Some weird behaviour can be also result of upgrade to much newer version of ROS. Upgrade process is supposed to migrate configuration as well but in case of some advanced setup automagic can fail. Miserably.

Might not be easy in your situation as I assume devices are scattered all over the place, but if you suspect device has been compromised, best is to use netinstall which will reformat the drive and place fresh ros / config on.