Community discussions

 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:19 pm

If your networks are so huge and yet you have failed to scale your infrastructure accordingly so that updating them is not manageable, and yet you have money to change over many of your devices just tells me you have other issues to overcome before changing equipment over.

You don't have the bigger picture here. This has nothing to do with scaling. We actually regularly update most of our network, semi-automated. But only after long term testing, because what is considered stable by Mikrotik is not considered stable by us. And sitting around on a bug like that for a year without even doing anything about it is negligence by Mikrotik.

Besides the fact that you actually believe that a proven long term vendor exists.............

When i say proven long term, then that refers to equipment we have tested long term in an aim to replace quite a bit of our Mikrotik infrastructure. This didn't just happen yesterday. The bug above just accelerates some of the replacements. Don't take my post out of context. I said, a vendor that is proven long term (by us). Not in general.

You can have a look on the left there .. under my avatar. I didn't just sign up here yesterday. I've been 12 years longer on this forum than you.

/M
Communication is the beginning of understanding
-- AT&T
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:39 pm

The ugly baby is how this has been dealt with over time, it isn't anyones process or workflow. Read my previous post. I stand by my statement - this has been handled poorly on all sides and because of this unnecessarily bad and totally avoidable mis-handling if it, we're forced to treat it like a zero-day. My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power. Therefor, since there has been a compressed timeline, for me, mitigation is the solution. I am in no way suggesting that anyone run untested software, we all know that MT is just as well known for [re-]introducing bugs in "stable" releases as it is for being economical. If you refer to my previous post, nowhere do I suggest doing anything close to that. What is disappointing is that we don't have enough folks working on a mitigation plan.
What I am advocating against is knee-jerk tactics, it's painting over the rust until such time as the metal can be repaired permanently. If disabling v6 is your solution, you do you. I would never suggest that someone has a workflow that is "wrong" if it works for them. Personally, I would rather spend the time necessary to operate in a degraded yet mitigated state than spend the cycles yanking an entire routed protocol out of a network to be re-enabled later, or never. My main point is that people should be working on as sane of a temporary solution as possible as opposed to right away throwing hands up and shutting things off, although if that's the last ditch effort, then that's what to do. My second point is that the author of the bug reports seems far more interested in poking things with a stick and giving his conference talk rather than providing insight into a possible temporary solution, which may impact the weight of his presentation.
By mining previous threads we have basic knowledge of what the issues are despite the author has being less than helpful about providing details. And again, it is implied that @IPANetEngineer has worked out a likely solution that is being tested more thoroughly as we speak.

nb


Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure.

It is actually not.

MT having a fix ready before disclosure is not enough. The fix needs to be ready, it needs to be in a stable release tree, it needs to tested and validated and customers need time to implement it.

@maznu has provided a clear timeline of when this was reported to Mikrotik, when the CVEs were issued and Mikrotik has ignored this until an imminent threat of it going public came about. If they had used the time from the report and the creation of the CVEs to mitigate this, there would have been plenty of time.

So in my case, it means IPv6 will be removed from all Mikrotik gear, because the risk of loss of business is too great. And where necessary, the gear will have been replaced with gear from other vendors, that's long term proven. Even if Mikrotik released a fix in time, there would be no time for testing. I would literally have to roll it out across 300+ devices in the core network. We have near to 10k Mikrotik devices in our network, if every one of them needs to be updated. This is not something you do in a few hours .. or days. And automated updates are not an option. The quality control of the release trees supplied by Mikrotik are not good enough. We aren't even on the current long-term release yet, as it has bugs we can't live with.

And the issue is, that it is not the first time, that Mikrotik has handled an issue this way. It is every single time. They only react, when issues become public knowledge.

/M
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:41 pm

My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power.

But this is where you are getting me wrong:

I'm not shutting IPv6 off on our network. We have been providing IPv6 to endusers since 2008. And even longer on the infrastructure. I'm turning it off on everything Mikrotik, because it is too big a risk.

And a good chunk of CCRs will have to replaced with alternative gear, not only because of this bug, but also because Mikrotik does not implemented features needed by the industry. I have tickets going back to before 2007 for some stuff that still isn't implemented. Can't even get a timeline.

There is plenty of hardware that is not affected.

/M
Communication is the beginning of understanding
-- AT&T
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:47 pm

despite the author has being less than helpful about providing details.
I have provided MikroTik with every detail at every step of the way.

I cannot provide anyone else with any more detail at all as this would literally give them the means to carry out the attack.

I have not shared any mitigation options because I do not believe there are any which do not involve significant expenditure of time and resources — something that many MikroTik users may not be able to afford.

I remain convinced that the fix can only come from MikroTik.
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:50 pm

Fair point, and great job on providing v6 back that far - few ISPs have that foresight. It's hard to infer context from a forum, and this thread runs the line of going "full-on UBNT forum" as it touches on a lot of peoples long held beliefs. Again, people should do what works for them. I'm just disappointed in the handling of this from soup to nuts.

nb
My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power.

But this is where you are getting me wrong:

I'm not shutting IPv6 off on our network. We have been providing IPv6 to endusers since 2008. And even longer on the infrastructure. I'm turning it off on everything Mikrotik, because it is too big a risk.

And a good chunk of CCRs will have to replaced with alternative gear, not only because of this bug, but also because Mikrotik does not implemented features needed by the industry. I have tickets going back to before 2007 for some stuff that still isn't implemented. Can't even get a timeline.

There is plenty of hardware that is not affected.

/M
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:58 pm

No, I get it, it's not my first rodeo with zero-day or high priority CVE, or with giving talks on any number of high sensitivity or previously embargoed subjects. I agree that you have said you provided the details to them and that handling has also been poor from their side, I trust that happened in one way or another. As far as the CVE, At least one of them is easily replicated using off the shelf tools, and is a pretty widely traveled technique (which frankly is a low hanging fruit in a pen-tester's toolbox). The other, I believe, is fairly straightforward (albeit egregious) as well, but again it's being pieced together with prior details in old threads. Likely not solvable at scale without a patch, but at least operationally minimized from the outside given the right strategy.

Having spoken my opinion I'm bowing out of this thread. My efforts are better spent on the mitigation plan.

nb
despite the author has being less than helpful about providing details.
I have provided MikroTik with every detail at every step of the way.

I cannot provide anyone else with any more detail at all as this would literally give them the means to carry out the attack.

I have not shared any mitigation options because I do not believe there are any which do not involve significant expenditure of time and resources — something that many MikroTik users may not be able to afford.

I remain convinced that the fix can only come from MikroTik.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
mstead
Member Candidate
Member Candidate
Posts: 112
Joined: Sat Mar 04, 2006 2:41 am

Re: UKNOF 43 CVE

Sat Mar 30, 2019 7:17 pm

I would make a good guess that the technique involves sending lots and lots of oversized neighbor discovery packets with the target IP of the victim. Easy to craft and while I never tested it, most likely could run the victim out of memory space if done at a high enough rate.
 
thantoldo
just joined
Posts: 22
Joined: Tue Apr 10, 2012 10:08 pm

Re: UKNOF 43 CVE

Sat Mar 30, 2019 8:10 pm

Meanwhile, Cisco dealing with its latest security vulnerabilities https://twitter.com/RedTeamPT/status/11 ... 6657238016
It can be much worse, guys..
Attitude to security should never be like ‘The squeaky wheel gets the grease’. Maznu just made it squeak!
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 729
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 9:02 pm

I would like to note that Buraglio isn't exactly green. He runs one of the largest capacity networks on the planet. I wouldn't dismiss his statements.
-----
Mike Hammett

The Brothers WISP
 
aussiewan
just joined
Posts: 24
Joined: Wed Sep 07, 2011 5:28 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:03 am

If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:16 am

If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
An alternative that doesn't necessarily mean we need to disclose vulnerabilities to each other or pass around weaponised PoCs is that MikroTik could send us the beta and we all bash it with our labs on Monday morning…?
Marek
 
aussiewan
just joined
Posts: 24
Joined: Wed Sep 07, 2011 5:28 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:36 am

@maznu - While I agree that sending around this potentially dangerous information has risks, I think that you have proven that you're more interested in a fix than having every IPv6-enabled MikroTik device in the world start spontaneously rebooting. Many others here are in the same boat. To clarify, I never intended it to sound like I wanted the information posted here, just shared directly between those who already have the information. I don't want to add to your workload, maznu, but it would probably make sense if those who have found methods sent you a "is this how you did it?" type message, if you're happy to receive them.

But yes, having another beta to stress-test sooner rather than later would also be fantastic!
If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
An alternative that doesn't necessarily mean we need to disclose vulnerabilities to each other or pass around weaponised PoCs is that MikroTik could send us the beta and we all bash it with our labs on Monday morning…?
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:42 am

Ideally this is how I would handle this. Again, we're super late in the game.
1. disclosed the environment hardware, in detail, that was used to test and confirm the the issue in.
2. have both validated it with a trusted, embargoed outside source(s). Ideally one is the vendor, clearly that didn't happen. 3rd party validation is pretty important either way as it proves a problem can be repeated independently.
3. If unable to define and at least indicate that there is a remediation, leverage outside trusted source to make sure that’s not possible. This seems to be kept between the person who discovered it and Mikrotik, who we *know* isn’t going to say anything. If getting credit for it is that important, agree upon that beforehand, it's doable and pretty straightforward with the right trusted sources.
A problem like this that is as egregious as described needs to be independently validated and ideally a remediation found. If there isn’t one, fine, Mikrotik has their head on the block, but we don’t know, and that’s the problem. At this point it would be sublime if @manzu could work with someone under embargo to validate this independently.

nb
If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
An alternative that doesn't necessarily mean we need to disclose vulnerabilities to each other or pass around weaponised PoCs is that MikroTik could send us the beta and we all bash it with our labs on Monday morning…?
ForwardingPlane, LLC
https://www.forwardingplane.net
 
AlexS
Member Candidate
Member Candidate
Posts: 257
Joined: Thu Oct 10, 2013 7:21 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 9:49 am

My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power.

But this is where you are getting me wrong:

I'm not shutting IPv6 off on our network. We have been providing IPv6 to endusers since 2008. And even longer on the infrastructure. I'm turning it off on everything Mikrotik, because it is too big a risk.

And a good chunk of CCRs will have to replaced with alternative gear, not only because of this bug, but also because Mikrotik does not implemented features needed by the industry. I have tickets going back to before 2007 for some stuff that still isn't implemented. Can't even get a timeline.

There is plenty of hardware that is not affected.

/M
Have to agree with the above. I have been a Mikrotik user for 8+ years - loved the devices at first. still like them now. But I hate being let down by the developers...
So some grips - apologies if these have been addresses - its been a while since i followed the forums
* single threaded BGP ... sigh
* 1G max through put for single stream tcp on ccr.. I believe this is again a cpu (single vs multi) issue on the ccr1072. - again going to be fixed in V7
* ability to share session state between routers - i raised this as feature when they announced it in linux. just add the daemon and it does the work
* basically V7 ... how long.... OMG.

So like above, I am slowly decreasing my usage of mikrotik.. This is just natural progression don't keep up with features others are offering.

But - I do love the interface and the tools...
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:02 am

now that we know what command to run
@IPANetEngineer: do you want to compare notes now that we are probably on the same page? Prompted by something MikroTik told me last thing on Friday about the nature of the underlying problem, and following my own research last night, I've got some good news to share — so it would be great to know if we're now independently testing and trying to mitigate the same things.

If you were leveraging the tool to attack subnet 2001:db8:3::/64 using the interface eth2, there is a command you would type. Let's assume you don't need to "sudo", and that the binary is in your $PATH (i.e. you don't need /usr/whatever/ at the beginning). You know the command that you would type (without any other options) to do this attack.

echo "command you would type" | md5sum

The first four characters of your output should match mine - I get "eaba". And if your first four characters match mine, then you now know that we are almost certainly using the same exploits.

If you want to work together on this, so that I know that we are both "fully weaponised", what are the next four characters of the hash? Drop me a private message, DM, email — something a bit more private so that we can start swapping notes.
Marek
 
User avatar
jprietove
Trainer
Trainer
Posts: 88
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:03 am

I've done my own investigation and I think I can reproduce the problem.
First it is important to note that the target of the attack needs not to be the Mikrotik itself: if it is forwarding traffic from an attack, its memory can exhaust and eventually it will reboot.

So my lab is similar to this: an attacker connected to a quagga router. This quagga router is connected to a chr and then this chr connected to another quagga that is connected to the target network.

When I launch the attack, the chr reboots but the other routers are not affected by the attack.

Firewall rules seems not to be effective.

But if I increase the chr memory from about 300 MiB to 3000 MiB the router seems to be ok: the free memory goes between 2200 and 2400.

As my lab is made in gns3 maybe it can't be the same as IRL with Mikrotik hardware so I plan to make some tests tomorrow with several ccr1009.

But I would ask @maznu if he can check what happens if he increase the ram. In his Twitter video he has about 300 MiB. And I would ask him if he has tested with routerboards or only chr.

It would be great if this thread keeps constructive and anyone who wants to cry or complain start another thread so we can silent it

Enviado desde mi Redmi 3 mediante Tapatalk

 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:18 am

When I launch the attack, the chr reboots but the other routers are not affected by the attack.

Firewall rules seems not to be effective.

But if I increase the chr memory from about 300 MiB to 3000 MiB the router seems to be ok: the free memory goes between 2200 and 2400.

As my lab is made in gns3 maybe it can't be the same as IRL with Mikrotik hardware so I plan to make some tests tomorrow with several ccr1009.

But I would ask @maznu if he can check what happens if he increase the ram. In his Twitter video he has about 300 MiB. And I would ask him if he has tested with routerboards or only chr.
This sounds almost exactly the same as what MikroTik will be fixing on Monday.

I have tested with non-CHR equipment before, with similar results. e.g. I crashed a TILE router with full transit tables loaded into memory (i.e. it already had a significant part of its memory allocated); and crashed all the mmips, mipsbe, and arm equipment I had available around me when I originally reported.

I will assume you are using the same tools I am, so @ jprietove: can you please perform the same thing as in my post immediately above yours? This way we all know we are talking about the same kind of attack.

What would be characters 9, 10, 11, 12 of the md5sum?
Last edited by maznu on Sun Mar 31, 2019 11:21 am, edited 2 times in total.
Marek
 
MichaelHallager
just joined
Posts: 17
Joined: Sun May 13, 2018 8:12 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:32 am

I can report I had (now disabled) IPv6 connectivity and a few days ago my router rebooted for no obvious reason. A couple of days later I was alerted to this issue.

As a consequence, I am now assuming the exploit is out there in the wild and is being used.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:50 am

As a consequence, I am now assuming the exploit is out there in the wild and is being used.
Thanks for this information, @MichaelHallager. I've saw something similar several times in the first two weeks of March this year, and advised MikroTik on 2019-03-15 about this, asking for urgent action.

At this stage I do not think it is part of a deliberate "attack campaign" — possibly it is incidental, collateral damage in some other activity on the Internet. But I cannot help worrying that it is reconnaissance — my worst fear would be a bad actor is looking to find vulnerable targets to shake down in an extortion campaign.

Do you have logs showing you why the router crashed? You can usually see that immediately after you log in.
Marek
 
MichaelHallager
just joined
Posts: 17
Joined: Sun May 13, 2018 8:12 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:57 am

I have been spreading the word around in other forums.

If it's of any interest / help I am happy to act as a remote test case providing no harm is done.
 
User avatar
jprietove
Trainer
Trainer
Posts: 88
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 12:01 pm

This sounds almost exactly the same as what MikroTik will be fixing on Monday.

What would be characters 9, 10, 11, 12 of the md5sum?
Sorry @maznu but I don't get the same md5sum you expected. Maybe mine is a different but correlated attack
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 12:03 pm

I have been spreading the word around in other forums.

If it's of any interest / help I am happy to act as a remote test case providing no harm is done.
At this stage, my best advice would be that people monitor the memory usage on their routers and graph it. If your memory usage is stable for many weeks, and then increases by several hundred Mb within a few minutes — possibly causing a crash as a result — that is one of the signs I would expect to see during an attack.
Marek
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 12:07 pm

Sorry @maznu but I don't get the same md5sum you expected. Maybe mine is a different but correlated attack
It is possible we are using different tools to trigger the same issue — there is more than one way to make some IPv6 packets. ;-)

If you're happy to discuss in private anyway, please drop me an email? Or DM me on Twitter (I just followed you).
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 6:56 pm

Can anyone verify in what order uRPF and route cache writes are processed? I suspect this is largely a solved problem, this was an issue in the early days of IPv6.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 8:28 pm

I can replicate both issues. This is very, vey easy to execute using simple, opensource pen-testing tools and is pretty effective at making a box reload, or under smaller load stop transiting traffic until the event stops. I do not consider this a security flaw at all. It's a very unfortunate implementation deficiency. I have not yet been able to replicate it with TCP or UDP, but I am not sure it can't be done as it has much less to do with the traffic type and more to do with the handling of particular caching functions. Mikrotik has called this neither a bug or vuln, and they're not exactly wrong, but it's an issue that needs addressed regardless, for sure, given it's extremely simple and low bar to execute.
A known mitigation strategy that won't work:
1. Netflow based detection and null routing where the flow generation is done on the devices under load. They don't export flow data once the event starts, and even if they did the 5m rollover of most flow data tools is far too long. FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write.

Still testing others.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 8:50 pm

FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write.
EDIT: with only:

* a route back to the attacker
* and only a default null route in my victim router's table
* (besides dynamic routes from link local addresses)

my lab looks like it will still OOM.

(the attack is coming from a single IPv6 address)
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 9:10 pm

If I understand correctly, you null route the offending host /128 (or /64) and the exhaustion still occurs, correct? If you null route the attacker IP address on the device that is transiting the traffic, does the OOM still occur? I am assuming that all routers are a mikrotik? I am working on this now.
FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write.
EDIT: with only:

* a route back to the attacker
* and only a default null route in my victim router's table
* (besides dynamic routes from link local addresses)

my lab looks like it will still OOM.

(the attack is coming from a single IPv6 address)
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 9:52 pm

Dumb question, have you validated that this is remotely exploitable outside of a contained lab?
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
davidcx
just joined
Posts: 16
Joined: Tue Nov 14, 2017 7:06 pm

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:40 pm

Mikrotik have publicly disclosed the details of the vulnerability, on a Sunday, in a way that a child could exploit it - before even providing a fixed beta, let alone a stable release version, let along giving us time to test and deploy it.

Truly despicable behaviour there Mikrotik. Do you have no respect for your customers at all?

-davidc
 
mstead
Member Candidate
Member Candidate
Posts: 112
Joined: Sat Mar 04, 2006 2:41 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:52 pm

Mikrotik have publicly disclosed the details of the vulnerability, on a Sunday, in a way that a child could exploit it - before even providing a fixed beta, let alone a stable release version, let along giving us time to test and deploy it.

Truly despicable behaviour there Mikrotik. Do you have no respect for your customers at all?

-davidc
Where is this posted? I did a quick search and didn't find anything.
 
bmann
just joined
Posts: 8
Joined: Sat Jan 05, 2013 2:10 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:17 am

I would say that MikroTik has good products for many people. For consumers it is great product.
You pay for a gear and then you can run it several years with updated RouterOS version and without any additional cost.
And in fact the issues are fixed in time. There is not many vendors doing this.

The reaction of MikroTik team is not probably ideal every time. But
How much does the gear cost with comparison with BIG network vendors?
How much do you pay for support per year?

I think that MikroTik helped to build a business for many ISPs and for many people. You know how they work, the benefits and limitations.
If your business is to big and MikroTik does not fit well anymore then you may choose another vendor with more powerful devices, support services
and of course with some costs for it.
 
MichaelHallager
just joined
Posts: 17
Joined: Sun May 13, 2018 8:12 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:27 am

@bmann has made some very good points which I can relate to.

I come from the Cisco camp and I was amazed when I bought my RB1100AHx4 what I was getting for the money... and it's made in Latvia, not China!

Personally, I think Mikrotik products are possibly a bit too cheap and I would be happy to pay a few extra bucks if it meant they were properly resourced so we don't have these types of issues.
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:30 am

Personally, I think Mikrotik products are possibly a bit too cheap and I would be happy to pay a few extra bucks if it meant they were properly resourced so we don't have these types of issues.
The reason, that here isn't such an option or a service contract is, that they'd have to take responsibility then and also implement features that customers require, instead of choosing themselves what they see fit to fix and implement.

I would have no problem paying a service contract, IF that would give me what I needed.

I have a good mix of Cisco, Mikrotik, Ceragon, Radwin and other gear. Cisco is just too expensive for some stuff and Mikrotik doesn't take things serious enough with other things.

/M
Last edited by marlow on Mon Apr 01, 2019 12:31 am, edited 1 time in total.
Communication is the beginning of understanding
-- AT&T
 
User avatar
davidcx
just joined
Posts: 16
Joined: Tue Nov 14, 2017 7:06 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:30 am

Where is this posted? I did a quick search and didn't find anything.
viewtopic.php?p=724264#p724238
 
proximus
Member Candidate
Member Candidate
Posts: 107
Joined: Tue Oct 04, 2011 1:46 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:49 am

Dumb question, have you validated that this is remotely exploitable outside of a contained lab?
.
That is certainly a critical question that needs to be answered and understood. The extraordinary amount of "pre-show publicity" has lead many to form strong opinions and responses before the full facts are understood.
 
msatter
Forum Guru
Forum Guru
Posts: 1124
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:52 am

@bmann has made some very good points which I can relate to.

I come from the Cisco camp and I was amazed when I bought my RB1100AHx4 what I was getting for the money... and it's made in Latvia, not China!

Personally, I think Mikrotik products are possibly a bit too cheap and I would be happy to pay a few extra bucks if it meant they were properly resourced so we don't have these types of issues.
I think assembled is a more fitting description.

https://www.made-in-china.com/manufactu ... rotik.html
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 6:35 am

I have come pretty close to being able to exhaust the route cache "in the wild" (a controlled real network built for this purpose), meaning on gear attached to a public network. I am sure I can do it, but I want to know who else tried this. There is an old thread that implies some of this was triggered by "real traffic", and I have definitely seen an uptick in ipv6 scanning that impacts major vendor router platforms to the point that they drop traffic and spike CPUs. This feels pretty darned similar, and not the catastrophic event horizon it's being played out to be. It's still bad, it stinks, and it's definitely been handled in a way that is less than optimal. I realize I am being a tad pedantic, but it's important to call a duck a duck in order to handle it in the correct manner. This is not a security vulnerability as I would describe it. There is no remote or local execution of code, no raised privileges, no mechanism of compromise. I don't think this is a software bug, either. It's close, but in reality what this is is a poorly implemented protocol that had what I can only believe was a poverty of testing before release and as such allows for a remotely triggered denial of service.
Denial of service isn't necessarily malicious, it just means exactly what it says: service is denied. As mentioned earlier in the thread, this is like the old ping of death issue: a poorly executed stack.

I'd like to know who has executed this in the wild and how far back this goes in their code train. I don't need to know how - I already have that sorted - I just want to know who and when. I'll bet dollars to donuts this has always been there. I've seen behavior like this before and I have seen denial of services a lot like this before (think exhausting the state table of a stateful packet filtering device). Easy pickings, low hanging fruit.
Dumb question, have you validated that this is remotely exploitable outside of a contained lab?
.
That is certainly a critical question that needs to be answered and understood. The extraordinary amount of "pre-show publicity" has lead many to form strong opinions and responses before the full facts are understood.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
xrobau
just joined
Posts: 5
Joined: Wed Jan 03, 2018 4:18 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 7:39 am

> This is not a security vulnerability as I would describe it.

Then what EXACTLY would you call the ability to stop your router from routing, remotely, without authentication? Because we in the Infosec world call that an Unauthenticated Denial of Service. (Which is a bad thing)

Look, MikroTik have dropped the ball on this. That they haven't gone 'Shit, yes, we made a mistake, we're sorry, and we're making (these changes) to make sure it doesn't happen again' is MORE telling.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24002
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Mon Apr 01, 2019 9:13 am

People above you in this thread are saying that there should be separate terminology between denial of service (CPU high, out of memory, crash etc) and something that allows attacker to gain access to devices, steal credentials, install malware and read private data. You are shoving them both under one term. I'm not expert, but at least that is what I mean when I said Denial of Service and Vulnerability are different things.
No answer to your question? How to write posts
 
msatter
Forum Guru
Forum Guru
Posts: 1124
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:15 am

The beta released today, addresses IPv6 route cache using more memory than available.

MAJOR CHANGES IN v6.45:
----------------------
!) ipv6 - fixed soft lockup when forwarding IPv6 packets;
!) ipv6 - fixed soft lockup when processing large IPv6 Neighbor table;
----------------------
Changes in this release:
*) ipsec - properly drop already established tunnel when address change detected;
*) ipv6 - adjust IPv6 route cache max size based on total RAM memory;
*) smb - fixed possible buffer overflow;
Two RB760iGS (hEX S) in series. One does PPPoE/IKEv2 and the other does the rest of the tasks.
Running:
RouterOS 6.46Beta / Winbox 3.19 / MikroTik APP 1.2.8
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
jprietove
Trainer
Trainer
Posts: 88
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:22 am

I have just tested this beta and I confirm that with 300 Mb RAM the router's memory doesn't fill. A CHR with 300 Mb of RAM with OSPF-v3 has 237 Mb of free-memory and during the attack it keeps on around 200 Mb.

Hopefully this fix will be in long-term and current branches soon.
 
p1p1p1
just joined
Posts: 22
Joined: Sat Feb 23, 2013 1:53 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:46 am

hmm testing beta in my sandbox routers, waiting for LONG TERM ...
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:19 pm

It has to filter to 6.43 long term first.

Too many problems in 6.44.

/M


Communication is the beginning of understanding
-- AT&T
 
cantanko
newbie
Posts: 28
Joined: Mon Apr 05, 2010 12:53 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:20 pm

People above you in this thread are saying that there should be separate terminology between denial of service (CPU high, out of memory, crash etc) and something that allows attacker to gain access to devices, steal credentials, install malware and read private data. You are shoving them both under one term. I'm not expert, but at least that is what I mean when I said Denial of Service and Vulnerability are different things.

I prefix this with the fact that I am a complete rank-amateur compared to some here, although I have to concur with Normis - the neighbour cache exhaustion issue is a denial-of-service, plain and simple. What is irksome is that it's a device-level, not protocol-level, DoS. If you're dual-stack on the same device, when you get scanned both of your stacks go away for the amount of time it takes for the router to reboot.

I also consider the fact that this issue has been knocking around for at least the last six years is what has many people frustrated (a nice discussion of it from March 2013 here).

Admittedly, I never bothered checking it either, but you kinda hope you don't have to...
 
enzain
just joined
Posts: 20
Joined: Wed Jan 17, 2018 9:15 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 2:52 pm

This may be the final straw. Time to start researching other products.
You always can use cisco:)
 
CTSsean
just joined
Posts: 23
Joined: Fri Sep 15, 2017 12:56 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:16 pm

We have near to 10k Mikrotik devices in our network, if every one of them needs to be updated. This is not something you do in a few hours .. or days.

...

And the issue is, that it is not the first time, that Mikrotik has handled an issue this way. It is every single time. They only react, when issues become public knowledge.

/M
Um, there are more than a few ways to automate the roll out of a config change or RouterOS package. Unimus or Ansible make this easy. Example PDF: https://mum.mikrotik.com/presentations/ ... 842532.pdf


As for how Mikrotik handled the issue... I suspect the blame is on both sides where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE. The issue is at kernal level and trying to re-write kernal code is not fun. Mikrotik has handled things better than most, but I think this RouterOS v6 is just becoming too long in tooth at this point. I'm a Mikrotik fanatic and everything lately is about fixing RouterOS6 when it needs be left behind and a newer kernal needs to be rolled up, even if they have to abandon Tile for RouterOS7.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24002
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:20 pm

It's true, MikroTik should have handled it better. It was mistakenly filed as just another ipv6 bug, which there are more of, until the author complained about lack of action. Then we looked at it again, and started to look for alternative solutions to handle it (because kernel change is not possible in v6). But in the end, the issue is fixed finally.

P.S.: we don't rely on kernel for TILE support, so new kernel doesn't affect it.
No answer to your question? How to write posts
 
BRMateus2
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Thu Oct 26, 2017 11:18 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:46 pm

Linux 5 is there, an stable kernel with tons of fixes from very dangerous bugs of 4.14-4.16 (BFQ with SCSI kernel locks (can corrupt the disk), Ryzen bugs ie suspend locks).
This is an DDoS kind, not something an dedicated blackhole subnet or country couldn't handle.

Good for Mikrotik and Normis telling details, please continue that attitude from now on, would be less of a rage earlier, good handling anyway.
 
enzain
just joined
Posts: 20
Joined: Wed Jan 17, 2018 9:15 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:48 pm

We have nine CCR1072 that can't receive full tables because they literally froze processing them.
Sorry, what?

How many tables on 1072?
I use 1036 - and this work with 2 FV tables
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 729
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:10 pm

We have nine CCR1072 that can't receive full tables because they literally froze processing them.
Sorry, what?

How many tables on 1072?
I use 1036 - and this work with 2 FV tables
Don't do full tables on CCRs. They are terrible at it.
-----
Mike Hammett

The Brothers WISP

Who is online

Users browsing this forum: No registered users and 49 guests