7.24.2 [stable] is released!

Announcements
122

< mode edit: moved to move appropriate place >

Hello,

I would like to report what appears to be a routing regression on RouterOS 7.24.2 affecting the CCR2216-1G-12XS-2XQ.

DEVICE
Model: CCR2216-1G-12XS-2XQ
RouterOS: 7.24.2
switch1 configured with l3-hw-offloading=yes

TOPOLOGY (simplified)
The router is a BGP edge router for a small ISP. It has one upstream transit link and several downstream routers reachable over VLAN sub-interfaces on SFP28 ports belonging to switch1. A public /24 is announced upstream, with a blackhole route for that /24 present on the edge router. Individual addresses and small subnets inside that /24 are routed to the downstream routers via static routes.

SYMPTOM
After upgrading to 7.24.2, every static /32 host route pointing to a VLAN interface on switch1 stopped forwarding transit traffic. Static routes with shorter prefixes (/27, /28, /29) over the exact same interfaces, to the exact same gateways, continued working normally.

The failure was silent. BGP stayed up, all PPPoE customers had working Internet access, and the downstream routers were fully operational. Only the individual host addresses used for device management and for a few NAT translations became unreachable from outside the network.

OBSERVED PATTERN

Failed - all showed the H (hardware-offloaded) flag:
six static /32 routes, each egressing through a VLAN interface on switch1

Worked - also showed the H flag, same interfaces, same gateways:
/27, /28 and /29 static routes

Worked - /32 routes that did NOT carry the H flag, because their egress
interface is a standalone ethernet port outside switch1

The pattern was completely consistent. Prefix length alone was not the trigger, and hardware offloading alone was not the trigger. Only the combination of a /32 route and a switch1 egress interface failed.

DIAGNOSTIC EVIDENCE

  1. Ping from the router itself to an affected address: success, ~300 microseconds, 0% loss.

  2. Ping from the router with src-address forced to the public transit address, so the downstream router had to use its default route to reply: success. This confirmed the downstream router, its addressing and its return path were all correct.

  3. Ping to the same address from an external host on another continent: 100% loss.

  4. Traceroute from two independent external locations terminated at the upstream provider's final hop, one hop before this router.

  5. Ping from the same external host to a different address inside the same announced /24, but routed through an ethernet port outside switch1: success. This ruled out any BGP announcement problem or upstream routing issue.

Points 1 and 3 together indicate that the CPU forwarding path resolves these routes correctly while the hardware path does not. Packets appear to be dropped inside the switch chip before reaching the CPU.

One hypothesis: the /32 entries are not being installed correctly in the offload table, so hardware falls through to the covering /24 blackhole route and discards the traffic. The software routing table shows the /32 routes as active and correct throughout.

WORKAROUND

Setting suppress-hw-offload=yes on each affected route restored reachability immediately and completely:

/ip route set [find dst-address="x.x.x.x/32"] suppress-hw-offload=yes

The H flag disappeared and external reachability returned with 0% packet loss on the first attempt. The same change was applied to all six affected routes with identical results.

This is acceptable for low-volume management traffic, but it is a workaround. Host routes should not require hardware offloading to be disabled.

QUESTIONS

  1. Is this a known regression in the 7.24 branch, and is a fix planned?
  2. Is there a release where /32 routes offload correctly on this platform, that we could roll back to if needed?
  3. Is the issue specific to /32 routes that have a covering blackhole route, or does it affect host routes more generally?

I can supply a supout.rif file and a full configuration export privately if that would help with diagnosis.

Best regards,
Riad Shabani
System Administrator

123

So you’re preaching lack of transparency as a good thing?

This is exactly how it shouldn’t be handled, and it only backfires at them. Why not put out all the details on the get go, and be explicit and upfront with your customers? Why is the vagueness?

124

There is still a problem with the display of the number of prefixes on a BGP connection.

That has been the case for a long time, but what seems new is that it now corrects itself when the route table is displayed. So in connections you see "0 prefixes", then you look in the route table what is going on, and now on the next F5 refresh the number of prefixes is correct.

125

I only consider it as "transparent enough".....in my opinion....yes, it might be a bit more detailed, but I would not consider this as "a big issue".

126

ou’re once again mixing Apples with oranges.

Users failure to follow best practices does not equal the failure of responsible disclosure by the vendor.

The complaint you see here is not about the existence of the vulnerability, that’s nothing special on its own and it happens to any company and product. And it’s also not about best practices. The complaint is rather mikrotiks way of handling the disclosure.

What a crock of crap. Any device is vulnerable on the internet if one leaves Device management open to the internet. I have never been concerned about any security issues on my mT, as good practices remove the possibility of hacking. What companies do is try to limit the dangers when people DO NOT follow good security practices and malicious actors figure out to then use this to gain control of a router.
So really all they are doing it trying to make it harder for bad actors to access poorly administered routers.

As for telling you what the vulnerabilities, are, none of your business. When its prudent to release information Normis will. I trust their judgment on what is best to maintain good security for all the product lines and for the RoS.

The problem is not solely for RoS but for all vendors, and there is a CVE process, if you dont like it, too bad. It is what it is. I am in no rush to upload the new firmware, but will do so in the next couple of days when the server loads are less.

127

Well, that's a fake argument.
It doesn't take days to download. So why wait ?

I did almost 120 devices yesterday, longest I saw was still below 2 minutes.

128

"You sound like a b*tch, b*tch"

Every device can be vulnerable regardless whether it connects to the internet or not.
Exposing it to the internet widens the attack surface. So does having device management at all.

This statement is fundamentally false.

Good practices don't eliminate vulnerabilities, they reduce the attack surface. And yet, there are real life examples such as memory corruption in low level routines handling the incoming/outgoing packet pipelines that could still be used to exploit the device, regardless of how many good practices you did or did not follow.

Once again, this claim is ridicules.

Any networking device can be vulnerable and exploited regardless of its configuration. At some point it has to actually process traffic, otherwise it'd be useless. See my example above - depending on where the bug sits, there may be nothing you can do except preventing traffic from reaching the device entirely.

But heck, let's give you another example - what if a VPN server is vulnerable? is that a legitimate use case for you? Should mikrotik fix it, or would you claim its just a "poorly administrated router"?

No, in fact they don't do nearly enough to make exploitation harder even on pristinely configured devices, because they aren't following basic security practices in their own operating system.

Where is ASLR? Code signing? Sandboxing? kernel backed auth tokens? I'm not even talking about cutting edge security here, it's basic stuff that would've made exploitation noticeably harder.

I asked them to comply with industry standards, meaning providing information about the affected components, the impact and obviously to be upfront about active exploitation in the wild. No one asked for a POC or a white paper explaining anything about the exploitation itself, nor code snippets showcasing the vulnerable code paths.

And as an owner of their product, it sure as hell is my business. I need to assess my own situation to know if I was exposed and possibly compromised, investigate what happened and take immediate action, or maybe I'm safe and don't need to prioritize it, and can just update in my own time.

Or, maybe when they are being pressed by customers to eventually do the right thing like it was demonstrated here.

It's OK to admit that you aren't versed enough in actual security and delegate the responsibility to others. However, doing so blindly and arguing with others on the matter isn't wise.

What problem? Lack of transparency? yes, some companies tried it. It usually ends up with pressure being applied on them and building to a point where it causes bad PR. Then they cave in and comply.

How does that have any relation to what is being discussed? Do you even know what a CVE actually is? It's literally just an identifier assigned to vulnerabilities. I have quite a few of them to my name.

Pressure, bad PR and paying customers have the power to make a company change for the better.

Interesting. How do you know you aren't vulnerable though? Is it because people pressured mikrotik to tell us what services are affected? Isn't it ironic how you assess your own urgency on exactly the kind of information you back mikrotik for withholding?

129

Responsible disclosure is precisely how Mikrotik approached the issue.
Well said.

130

No, CERT was forced to disclose it because mikrotik failed to provide the minimum required information on time, forcing us, the community to reverse engineer and uncover what happened.

Why we are publishing now

We are publishing this information on an accelerated schedule because the patched RouterOS packages are already public, and their comparative analysis has allowed the community to reconstruct some of the fixed bugs. We limit the description to the information administrators need and do not publish exploit code or details that would make automating attacks easier.

131

Just a question - could you provide some example of "cutting edge security" in terms of vendors/products? I am asking because I tried to compare vulnerabilities across manufacturers providing comparable products (meant usable for our systems) and so far all of them has more vulnerabilities with higher CVSS score (9,8 - 10) while they are all considered "enterprise" - Cisco, PaloAlto, Fortinet, SonicWall...

EDIT: ant to be honest, based on recent details here (https://app.opencve.io/cve/?vendor=mikrotik) it really seems that it is not a single vulnerability that could be communicate the way you expect, but probably multiple vulnerabilities discovered probably by some AI analysis (based on number of vulnerabilities in a short time)...so if MT says "it's important security update.....update now, talk later".....it really seems that keep talking just provide attacker time to break into your device.

132

Do we see any evidence of persistence except the created "ops" user like

  • if login with "ops" was successful, what did they do
  • router os config changes after login
  • anything changed on the underlying linux os, did they get root shell?
  • rootkit-like behaviour?
133

Reported abuse @ leaseweb.com for IP 82.192.72.4 / ASN AS60781

134

That is the same as writing your complaint into /dev/null

135

Thank you for your comment.
Probably thats the reason why the IP is still online and still has positive reputation.
I have good experience with abuse reporting.

136

Please stop spreading misinformation

137

I often get this question from people outside of my field of expertise (which isn't routers), and the simple analogy I explain it with is the following:
Imagine you have 2 safes, one is used by a cashier under the register, while the other is located in a bank.

Both are by definition breakable, however they require significantly different amount of time, money and effort to be broken open, leading to a completely different scale of compromise and threat actors.

One invites casual robbers, while the other a team of experts working together to plan and facilitate a high target robbery.

If we now take this to software, the same thing applies.
You have 2 different platforms, same primitives.

On platform A, same memory corruption gives you an immediate arbitrary code execution in a highly privileged process with no meaningful isolation, and than forking into your own payload, because why not, it doesn't cost a thing. One bug to fully compromise the system.

On platform B, exactly the same memory corruption may only give execution inside a heavily sandboxed, low privileged process. Requiring another vulnerability to leak and calculate the slide to defeat ASLR, another one to escape the sandbox, and potentially additional vulnerabilities to attack the kernel and disable code signing if you wish, all while crafting highly specialized ROP chains to escape and bypass different security measures.

So while platform A generated a single vulnerability, and required maybe a work day to exploit by a single researcher, platform B required multiple vulnerabilities, carful planning and development of a far more complex strategy, requiring a team of researchers weeks or months to exploit.

Same initial vulnerability, different amounts of effort to exploit, which leads to different scale of attack, and different threat actors.

That's why counting CVEs doesn't really tell you much about which platform is in practice more secure. Platform B may end up with more vulnerabilities just because a successful attack requires longer and more complex chains. What matters is how much the platform limits what you can achive with a single vulnerability and the amount of effort needed to actually exploit it.

But theres another side to it happening in parallel. Not all CVE's listed are used in an active attack. Many vendors and security companies invest huge amounts of money and effort to proactively find, fix and disclose vulnerabilities, offering bug hunters bounties to attract additional, outside workforce into this effort. This would result in more CVEs, but in practice it leads to a more secure system.

I reversed the patches myself within a few hours just for fun, like many others. If I were to spend tokens on it - it would've likely taken somewhere between 30 to 60 minutes. So no, you're not providing a thing attackers wouldn't be able to reverse once patches are out. You're only depriving those who want to defend themselves from being able to do so wisely.

138

How is it false?
Mikrotik failed to provide basic information and pushed the community to reverse engineer the patches - some decided to make it public, effectively releasing far more information than needed, leading CERT to accelerate their own publication of the matter.

139

That actually doesn't answer my question "who is significantly better". Vulnerability with CVSS score 9.8+ basically means that unauthenticated user might get full access to the device/system. All mentioned vendors had these vulnerabilities so in this case the number of them and the severity has been much higher compared to Mikrotik. And because there are milions of them running and SOHO devices might be more attractive for botnet, I really don't think that the relative low number of vulnerabilities of Mikrotik/RouterOS could be explained by low number of systems or low attractivity for the "bad" guys...

140

While it avoids naming a specific vendor, it does provide the framework by which one should assess it.

And just to be clear, Platform A and B do not translate to any vendor but rather represent 2 extremes to illustrate the difference.

No, a 9.8 score means it is isolated to the affected scope.

Once again, see Platform A vs Platform B examples. Same vulnerability, same score. Different outcomes.
This is because the complexity the score represents is based on things that may make it unreliable, for example basing it on race conditions will lower the score, because it represents higher complexity and lower reliability. However, the score excludes standard defensive mitigations as I listed in example B, including the requirement of additional vulnerabilities required to exploit.

So in practice, same score may need more time and effort to be exploited in one system than in the other.

As for the difference in numbers, just from a quick look at Cisco's security advisories I immediately spotted that a good amount of them were discovered by internal reviews, meaning proactive research.
How many vulnerabilities were uncovered by proactively reviewing routeros and were reported to the public? Again, that's just one specific explanation. I really did not dig any further.

Although number of systems does play a role in attracting botnet campaigns, this kind of attacks burns through vulnerabilities, so they would rather target those that require less of them and are easier to replace, rather than systems that have significant mitigations in place and cost tens to hundreds of thousands of dollars to replace.

141

You forgot one LITTLE difference: one of your safes costs <$100, another >$1.000.000.