I haven’t seen any chatter about this on these forums or elsewhere…
Just tonight we discovered a multitude of RouterOS devices on our network – mostly customer devices, so far only observed on MIPS architecture – that appear to be infected with something. The routers themselves are generating hundreds of outbound connections every second to random IP addresses, targeting telnet (TCP port 23), TR-069 (TCP port 7547), and WINBOX!!! (TCP 8291). I have confirmed that this traffic is not originating from a device on the customers’ LANs and then getting NATted…it is coming from the router itself and is successfully blocked using firewall rules in the “output” chain.
Rebooting the device does not stop it, so whatever it is has managed to write itself to flash, as well as insert itself into the OS startup/boot process. So it knows its way around RouterOS internals, it isn’t phased by the CPU architecture, it seems to know of some RouterOS vulnerability that it is exploiting somehow, and it targets Winbox.
The two devices I have examined remotely in detail are both running 6.34.2. (I know this is not current, but it is a customer device and we have not been forcing customers to update the software on their routers. This might have to change.)
I am hoping to collect a hardware sample soon, which will allow me to investigate the “worm” more in-depth and figure out where it has managed to hook itself into the OS. But maybe whatever this “worm” is is already known (and whatever security vulnerability still existed in 6.34.2 has since been patched), and I am just not up-to-date on the news. If anyone has any information about this I would really appreciate any leads.
I haven’t heard about that yet but it’s certainly worth investigating. Do you happen to have a Packet Sniffer output file that contains the outbound connection you could share?
Thanks. It’s not like I don’t know how to do a Netinstall. But we are potentially talking about a few hundred devices here. Anyway, the real question is not how to recover from it, but what this worm is precisely, and how to protect ourselves from it. For all I know, it could be exploiting a zero-day that hasn’t been fixed in a current version of RouterOS. In that case, Netinstalling a few hundred devices is going to be a big fat waste of time, don’t you think? They will all just become infected again.
I’m also surprised that I haven’t heard anything from anyone else being hit by something similar…
Hello Nathan,
we have seen it last night too, mostly on 6.34.x version, SXTs/LHGs - mipsbe.
I managed first to block the telnet traffic to and from them and then I updated them to latest current version 6.41.3.
All affected Mikrotik routers had public IPs and exposed common service ports (80,21,22,23, etc) to the internet but whatever it was it didn’t or couldn’t change their login data.
I was very scared until I discovered what caused this - a year ago one of our partners forgot to apply the common firewall rules to a batch of CPEs and nobody noticed it until now.
The attack had dramatic effect on our radius server but AFAIK it caused no other damage.
We are following logs and so far it looks like that this is over.
So solution is just common sense - always use regular firewall rules and updated ROS.
yes, there clearly is a problem!
we are not affected because we do not have management ports open to the internet, but scanning the internet link shows a lot of
incoming connects to port 8291 from wildly varying addresses. this clearly points to a worm.
(the usual scans from the self-appointed “security scanners” and “researches” have always been there but this is different)
Just a side note. Even I have never saw such behaviour I always populate the output firewall chain by similar rules like in input with final drop rule.
As it was mentioned long time ago, the consistency check or downgrade / upgrade exercise should delete everything else than installed packages.
Mikrotik always stated that there is no way to run custom code in ros. Hope this was not proved as a lie by this.
I hope also that the bad code is not part of standard ros packages distribution…
Of course we have always known this is just a naive claim. There is no way they can back that up, because in all software there are bugs and those could
be used to circumvent security measures. That it did not happen yet does not mean it is impossible, maybe it has not been tried or they did not notice it.
Also, there is little transparency. At the minimum, there should be some list of vulnerabilities and version where they have been fixed, so that users can
quickly judge if existing installations are at risk. “always install the latest version” does not cut it, because there are developments as well, and newer
versions do not always have the functionality that is required or they require further work before they can be deployed (e.g. version 6.41)
At the moment I see a large flow of incoming connects from residential addresses, which clearly points to a botnet. It is different from the always-present
scans from virtual servers done by the wellknown losers. It can no longer be denied, and all previous attempts at getting things rectified have mostly
gone to waste due to the denial of there being a problem. Let’s see what happens now.
Yeah, it has begun… I have 3 routers on remote locations infected. I cant access the router physically so i needed to fix it the time being.
After some attemps i got it under control, adding some filter rules to my firewall and blocking everyone except my ip(s).
Here are the rules, hope it helps someone. Replace YOURIP. Better turn on safe mode before typing.
Forwarding is not affected by that rules.
We were seeing what sounds like it may be the same exploit yesterday. I noticed telnet login attempts from some of our Mikrotiks to other Mikrotiks on our network. When I went to the source box, I could see in the logs something to the effect of "fetch: downloaded .i " (I apologize, I didn’t note the exact verbiage, but it did reference a file named “.i”). So, my guess is the .i file was some kind of script that was able to run on these routers and then start attempting telnet attempts. We believe that it was trying logins with common user names and password combinations (i.e., admin, root, ubnt, Administrator, etc.), as we were seeing these login attempts to other Mikrotiks on our network. It looked like the script was probably trying addresses close to it’s own IP address and then expanding the addresses outward from there.
We had another unrelated network (one of our admins has another network he runs) where they were seeing the same issue. On our network, we had about a half dozen machines that appeared to be exploited. All of them had http service enabled (a bad oversight on my part) and almost all of them were running older routerOS (typically 6.27 to 6.35.4). On the other network they had at least a dozen machines and they all had older RouterOS and http enabled) However, one of the compromised machines on my network was running 6.39.3. We had believed that this was most likely from a known security bug in RouterOs prior to 6.38.5 (What’s new in 6.38.5 (2017-Mar-09 11:32): !) www - fixed http server vulnerability;), although the 6.39.3 exploited machine has us concerned that it may be something different. We did not have any machines exploited that were not running the http service, so we’re mostly convinced that was the entry point.
We had one machine that did appear to continue running the script after disabling http and rebooting. So, it does seem like it was something that was embedded in the boot process. We’ve since disabled http on all machines and upgraded all pre 6.38.5 routers to 6.41.3 (we still need to upgrade some of our 6.39.3 machines, but we needed to fix the most vulnerable first). We also installed input and output chain filters to all machines to block connections from outside our network, so that will hopefully mitigate some of these issues.
hey, thx for the thread. our network had a major attack today as well. it seems like they opened some devices via the http port (quite a old firmware - damn) and they tried to spread or access by brute forcing mikrotik neighbours. quite a strange behaviour, because they could access passwords and we have the same password for all devices.
i did a portscan and one of the devices has port 61267 open. i am trying to figure out a solution at the moment.
I’m in the middle of some other things at the moment but me see what I can do about remote access for you in a bit here. Do you want me to just e-mail support@ or message you some other way?
srosen,
Interesting. I don’t think HTTP is the vector (though I could be wrong), mostly because I don’t see infected devices making any attempts to contact other IPs via HTTP…just telnet, TR-069, and Winbox. In our case we haven’t seen any kind of visible script, either on the (exposed) filesystem or in ‘/system script’, nor are there any active scripts/jobs running nor logged in users/sessions that are unaccounted for.
Interestingly, I just ran across a device running 6.37.4 that also showed signs of infection, but this time they disappeared after a reboot.
I have a /16 subnet on internet that I can trace, and I see about 100 TCP SYN requests per second to port 8291 to this network.
(from many different IP addresses on internet to random addresses in this network)
These are blocked in the firewall of the router towards that network.
I have tried to telnet back to a couple of those addresses and several times a RouterOS logon came back.
Looking up reverse for those addresses shows them to be residential.
For clarity, we were seeing telnets from the compromised machines (failed login attempts). What I was thinking was that the initial exploit that allowed the code to run was an exploit of an http security hole. All of the machines that were compromised had http (i.e., the www service on the Mikrotik) enabled - no machines that had this service disabled were compromised.
As far as the “script” - that may not have been good terminology. What I know is that the logs on the compromised machines showed the ".i " file having been downloaded. My assumption was that it was running some kind of “process” (be it a script process or some code kept in memory only), but clearly the “.i” file had something to do with the exploit. I just checked the scripts on one of the machines that had previously been compromised, and there’s no script by that name on it now, and there weren’t any files on it by that name at the time of the exploit, either.
If you find another compromised machine (hopefully you don’t), you may want to check the logs to see if you see anything like the “fetch download .i file” messages that I had seen.
Can others please also check if upgrade to 6.40+ (or even latest RC) fixes this? We did implement additional internal checks and checksums in later versions to prevent any 3rd party files from being copied into any router, even if it has no password and firewall
I checked my networks several times during the time we talk here about this and still I can not find any signs of the infection. At least so far. The oldest devices are around 6.25, the newest around 6.40 and all do as usual without any strange traffic. Just time to time some login attempts to admin user without success, because there is no such user at all…
Thanks. It’s not like I don’t know how to do a Netinstall. But we are potentially talking about a few hundred devices here. Anyway, the real question is not how to recover from it, but what this worm is precisely, and how to protect ourselves from it. For all I know, it could be exploiting a zero-day that hasn’t been fixed in a current version of RouterOS. In that case, Netinstalling a few hundred devices is going to be a big fat waste of time, don’t you think? They will all just become infected again.