Community discussions

 
User avatar
dynek
Member Candidate
Member Candidate
Topic Author
Posts: 193
Joined: Tue Jan 21, 2014 10:03 pm

UKNOF 43 CVE

Wed Mar 27, 2019 4:08 pm

Hey,

Just discovered: https://indico.uknof.org.uk/event/46/contributions/667/
During some research which found CVE-2018-19298 (MikroTik IPv6 Neighbor Discovery Protocol exhaustion), I uncovered a larger problem with MikroTik RouterOS’s handling of IPv6 packets. This led to CVE-2018-19299, an unpublished and as yet unfixed (despite almost one year elapsing since vendor acknowledgement) vulnerability in RouterOS which allows for remote, unauthenticated denial of service. Unpublished… until UKNOF 43!
Any comment Mikrotik ? :-)

Thanks
 
User avatar
bigcw
Member Candidate
Member Candidate
Posts: 105
Joined: Mon Sep 08, 2014 2:38 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 1:43 pm

I've been talking to Marek (the presenter) this morning. In a nutshell, if you run v6 on a public-facing interface, you're f***ed come 9th April. Every script kiddie out there can remotely crash your router, and do it over and over again. The only solution is to disable ipv6, not even firewalling will help here.

Mikrotik: I hope you have every developer you have available working on a fix for this. The consequences of you not having a patch in time for the 9th do not bear thinking about.
Ecom International Network - Operators of AS61337 with POPs in Europe and North America - www.ecomltd.co.uk
Colocker Data Centre - The data centre with a difference! - www.colocker.com
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Thu Mar 28, 2019 1:50 pm

We are aware of this issue and are working on it.
No answer to your question? How to write posts
 
User avatar
bigcw
Member Candidate
Member Candidate
Posts: 105
Joined: Mon Sep 08, 2014 2:38 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 2:07 pm

I am not convinced that your statement is acceptable, Normis. This is a serious issue that could destroy many businesses and cost millions. Given the gravity of the situation, I would expect at the very least:

1. Reassurance that you are taking the matter seriously (unlike the past year where it has been ignored)
2. A statement to the effect that you are putting every bit of development effort available to you towards identifying the source of this problem
3. A guarantee that a fix will be available in good time before this knowledge is made public

I look forward to your response.
Ecom International Network - Operators of AS61337 with POPs in Europe and North America - www.ecomltd.co.uk
Colocker Data Centre - The data centre with a difference! - www.colocker.com
 
User avatar
dynek
Member Candidate
Member Candidate
Topic Author
Posts: 193
Joined: Tue Jan 21, 2014 10:03 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 2:18 pm

…and sadly @mikrotik_com continue to stonewall me saying this remote unauthenticated denial of service is a “bug” not a “security vulnerability” — which is probably why they haven’t prioritised it for the last 50 weeks.
https://twitter.com/maznu
 
User avatar
bigcw
Member Candidate
Member Candidate
Posts: 105
Joined: Mon Sep 08, 2014 2:38 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 2:26 pm

…and sadly @mikrotik_com continue to stonewall me saying this remote unauthenticated denial of service is a “bug” not a “security vulnerability” — which is probably why they haven’t prioritised it for the last 50 weeks.
https://twitter.com/maznu
My point exactly. Marek was kind enough to show me a video demonstrating the problem (I promised I would not share otherwise I would post here). It is very much as bad as it sounds. Why are you not taking responsibility, Mikrotik?
Ecom International Network - Operators of AS61337 with POPs in Europe and North America - www.ecomltd.co.uk
Colocker Data Centre - The data centre with a difference! - www.colocker.com
 
User avatar
pcunite
Forum Veteran
Forum Veteran
Posts: 945
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: UKNOF 43 CVE

Thu Mar 28, 2019 3:28 pm

I highly recommend MikroTik look into implementing something like the Safe 4.5 Lean-Agile framework for their company. It will help to get a handle on the continuous release cycle that is their type of company. This is a business process for how to organize, coordinate, and manage simultaneous hardware and software releases.
 
jamesmck
just joined
Posts: 1
Joined: Thu Jan 31, 2019 3:14 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 6:14 pm

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
 
highonsnow
newbie
Posts: 34
Joined: Fri Oct 19, 2007 3:30 am

Re: UKNOF 43 CVE

Thu Mar 28, 2019 6:34 pm

This is going to need every possible resource. Don't let this one slide guys.
 
jrpaz
Frequent Visitor
Frequent Visitor
Posts: 83
Joined: Wed Jun 05, 2013 5:54 am

Re: UKNOF 43 CVE

Thu Mar 28, 2019 7:12 pm

The fix is in v7 guys c'mon
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2291
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: UKNOF 43 CVE

Thu Mar 28, 2019 7:28 pm

We are aware of this issue and are working on it.
No info on blog.mikrotik.com
Will the patch be released this week?
LAN, FTTx, Wireless. ISP operator
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Thu Mar 28, 2019 7:59 pm

So far, Mikrotik communicated on the blog after the release of a fix.
 
User avatar
adhielesmana
Trainer
Trainer
Posts: 23
Joined: Tue Aug 25, 2009 11:01 am
Location: Monrovia, Liberia
Contact:

Re: UKNOF 43 CVE

Thu Mar 28, 2019 8:29 pm

Oh Shi'''t my whole isp core network & cpe run public ipv6 with mikrotik.
No choice, Im have to disable my ipv6 then
 
storp
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Tue Nov 24, 2015 2:53 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 8:36 pm

We are aware of this issue and are working on it.
You might need to put in some overtime hours to get this fixed! ;) Free pizza to the programmers is always a recipe for success :)

To be serious, I really hope there will be a fix out in time. Personally I just can disable IPv6 but for many that isn't an option.
 
neutronlaser
Member Candidate
Member Candidate
Posts: 212
Joined: Thu Jan 18, 2018 5:18 pm

Re: UKNOF 43 CVE

Thu Mar 28, 2019 8:58 pm

I've been talking to Marek (the presenter) this morning. In a nutshell, if you run v6 on a public-facing interface, you're f***ed come 9th April. Every script kiddie out there can remotely crash your router, and do it over and over again. The only solution is to disable ipv6, not even firewalling will help here.

Mikrotik: I hope you have every developer you have available working on a fix for this. The consequences of you not having a patch in time for the 9th do not bear thinking about.
No foul language please
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 993
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: UKNOF 43 CVE

Thu Mar 28, 2019 10:08 pm

Would somebody please post some additional information about this.
I need to understand what is the problem, the potential impact and what vulnerabilities are possible.
Where can I find information to read/learn about this?

North Idaho Tom Jones
 
highonsnow
newbie
Posts: 34
Joined: Fri Oct 19, 2007 3:30 am

Re: UKNOF 43 CVE

Thu Mar 28, 2019 11:46 pm

We've dropped IPv6 transit across our whole ISP network until this is resolved. Outrageous.
 
cmurrayis
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri May 15, 2009 4:31 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 1:21 am

No Choice - IPv6 Peers disabled.
 
fredyz
just joined
Posts: 6
Joined: Wed Jun 13, 2018 7:07 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 1:31 am

That's normal standard from Mikrotik when they are faced with a problem to resolve and they have no idea. They shoot the messenger, they don't like, they refuse to take the responsibility and probably they believe they have no obligation to give any reasonable satisfaction to their public. They might think they may only do that "if they feel like". It is their culture, they never change and as more people bring issues both with the product and with their culture more "ego-hurted" they become.

That explains well this type of statement you could read, kind of: "We are aware of this but we are not giving you any satisfaction, any timeframes, any workaround or any information it may be useful to you in the meantime because we don't feel that necessary and we are bothered to deal with this".

It's a total lack of organization and care, that is not from now, but for quiet a while and seems to come from the top.
I am not convinced that your statement is acceptable, Normis. This is a serious issue that could destroy many businesses and cost millions. Given the gravity of the situation, I would expect at the very least:

1. Reassurance that you are taking the matter seriously (unlike the past year where it has been ignored)
2. A statement to the effect that you are putting every bit of development effort available to you towards identifying the source of this problem
3. A guarantee that a fix will be available in good time before this knowledge is made public

I look forward to your response.
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 993
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 1:44 am

Hey , come on now … Please - let's not be negative about/to Mikrotik

I'm sure that Mikrotik will release an upgrade to resolve this IPv6 issue (hopefully in time).
Mikrotik is the cost-effective solution for smaller-Carrier-Grade ISPs and businesses.

I for one - welcome any information that helps me operate my ISP business.

North Idaho Tom Jones
 
shanecaznet
just joined
Posts: 15
Joined: Thu Jul 13, 2017 10:52 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:33 am

We are aware of this issue and are working on it.
Oh... good.... only a year late.

I can live with what I consider a priority and what Mikrotik considers a priority not aligning (read: multi threaded BGP etc). However, I cannot live with a known security problem being ignored for so long.

Security should always be number 1, above all else, no exceptions.

This may be the final straw. Time to start researching other products.
 
jimmer
just joined
Posts: 3
Joined: Wed Mar 06, 2019 10:06 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 5:36 am

The fact that a bug thats now a year old, a critical bug no less, is being taken so casually by Mikrotik has me concerned with my investment in the products, I like many others here will be looking at alternative platforms should it not be fixed within the next few days, turning off IPv6 is not a mitigation option for us, A resolution to this issue should have the priority it deserves or at the very least a filter to mitigate the Denial of service that this issue could impose.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 8:28 am

Would somebody please post some additional information about this.
I need to understand what is the problem, the potential impact and what vulnerabilities are possible.
Where can I find information to read/learn about this?
MikroTik acknowledged this issue on 2018-04-20.

To learn more about it: I am presenting at UKNOF 43 on 2019-04-09 (April 9th), and there will be a live stream.

MikroTik support was made aware of my intention to speak at UKNOF on 2019-03-04, which is when UKNOF accepted my talk. This gave MikroTik over a month of notice that I intended to discuss these issues.

Since 2019-03-04 I have told MikroTik that I believe there is exploitation in the wild already, and that they should reprioritise their efforts to fix this.

I am not aware of any workarounds or mitigations any of us can use.

Despite my repeated pleas for this to be treated as a security issue, everyone I have interacted with at MikroTik says the same. Even normis has stated it is not a "vulnerability" in MikroTik's eyes — it is just a "bug".
Last edited by maznu on Fri Mar 29, 2019 9:51 am, edited 1 time in total.
Marek
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:26 am

That's normal standard from Mikrotik when they are faced with a problem to resolve
That is not true at all! We have always reacted to issues quickly, all the previous vulnerabilities have been fixed within hours or days time.

Even in this case, we did reproduce and acknowledge the issue. In this case though, the cause is very low level and is not simple to fix. The person who submitted it, has given a publication date. We have therefore a target date before which the issue must be fixed. In fact, most CVE reports have a release date set, before which the software company can fix the issue. The date has not yet come to pass.
No answer to your question? How to write posts
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:45 am

That's normal standard from Mikrotik when they are faced with a problem to resolve
That is not true at all! We have always reacted to issues quickly, all the previous vulnerabilities have been fixed within hours or days time.

Even in this case, we did reproduce and acknowledge the issue. In this case though, the cause is very low level and is not simple to fix. The person who submitted it, has given a publication date. We have therefore a target date before which the issue must be fixed. In fact, most CVE reports have a release date set, before which the software company can fix the issue. The date has not yet come to pass.
Thanks for the update Normis. Since you mentioned in the other thread this is a kernel issue and very difficult to patch, do you expect there to be a fix within the next 10 days?
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
thobias
just joined
Posts: 22
Joined: Thu Nov 30, 2017 8:45 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:53 am

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
I also would like to know this.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:56 am

We aim to fix the issue before the mentioned publication date.
No answer to your question? How to write posts
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:57 am

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
I also would like to know this.
If you cannot route IPv6 packets, you should be safe.
Marek
 
User avatar
vecernik87
Long time Member
Long time Member
Posts: 648
Joined: Fri Nov 10, 2017 8:19 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:58 am

Quote from second thread:
Yes, it is kernel level and is very hard to fix, since RouterOS v6 has an older kernel version and we can't just change the kernel.
Is that v7 announcement? :D Hurray!
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 9:58 am

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
I also would like to know this.
Even if you haven't configured it, you still have a link-local address. An attacker in your network can target that address.
No answer to your question? How to write posts
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:01 am

Would somebody please post some additional information about this.
I need to understand what is the problem, the potential impact and what vulnerabilities are possible.
Where can I find information to read/learn about this?
I am not aware of any workarounds or mitigations any of us can use.
I believe I saw you comment that this can't be mitigated in MIkroTik at Layer3.

What about using a MikroTik router at Layer 2 (or a non-MikroTik) inline in bridge mode before the Internet connection and using the firewall to filter out whatever is in the crafted packet that creates the issue? I'm assuming something is getting set in the header to cause this which means we ought to be able to apply the same types of techniques that are used to detect and mitigate zero day attacks by WAFs, DDoS filters, L7 firewalls, etc. Even if a non-MikroTik solution is required until the patch is released.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:02 am

We aim to fix the issue before the mentioned publication date.
That is very welcome news, normis.

If you or your developers wish to contact me privately for any further information, you've got my email address.

Good luck!
Marek
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:05 am

I believe I saw you comment that this can't be mitigated in MIkroTik at Layer3.

What about using a MikroTik router at Layer 2 (or a non-MikroTik) inline in bridge mode before the Internet connection and using the firewall to filter out whatever is in the crafted packet that creates the issue? I'm assuming something is getting set in the header to cause this which means we ought to be able to apply the same types of techniques that are used to detect and mitigate zero day attacks by WAFs, DDoS filters, L7 firewalls, etc. Even if a non-MikroTik solution is required until the patch is released.
I do not believe that this approach will be useful to mitigate the problem.

Sorry, you will have to wait for disclosure at UKNOF 43 for full technical details.
Marek
 
thobias
just joined
Posts: 22
Joined: Thu Nov 30, 2017 8:45 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:20 am

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
I also would like to know this.
Even if you haven't configured it, you still have a link-local address. An attacker in your network can target that address.
Would below be enough to mitigate it?
>ipv6 export
>/ipv6 nd
>set [ find default=yes ] disabled=yes
>/ipv6 settings
>set accept-redirects=no accept-router-advertisements=no forward=no
and then remove all addresses from ipv6->addresses?
I would like to not have to disable the package and reboot untill a fix is released and tested for a while.
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:52 am

That's normal standard from Mikrotik when they are faced with a problem to resolve
That is not true at all! We have always reacted to issues quickly, all the previous vulnerabilities have been fixed within hours or days time.

Even in this case, we did reproduce and acknowledge the issue. In this case though, the cause is very low level and is not simple to fix. The person who submitted it, has given a publication date. We have therefore a target date before which the issue must be fixed. In fact, most CVE reports have a release date set, before which the software company can fix the issue. The date has not yet come to pass.
You've had close to a year to work on a fix...

The matter way reported to you last year but from what I am seeing, you did not do much about it and it's only since the threat of Mikrotik being "outed" on a live stream that can be seen around the world, that you are doing something about it.

What if some kid who can't get a date on a Saturday night figures this out by himself and starts letting his or her script loose before 9 April, "just because they can" ?

This is a very serious issue that needs to be addressed as a matter of urgency. There are ISPs out there with tens of thousands of routers that need to be patched. It takes us between a week and 10 days to roll a RouterOS update out over our network. We have less than 10,000 customers. How is a company with 50,000 hAPs in the field going to do this within the time frame?

On 2 April I have two choices:

Either roll out a patch, or

/sy package print
/sy package disble ipv6
/sy reb
y

I run what could very well be the largest IPv6 network in South Africa. Disabling IPv6 would be very unfortunate. We're already not using Mikrotik for new wireless installationsd. It would be sad if we also stopped using Mikrotik for routing.
 
User avatar
macgaiver
Forum Guru
Forum Guru
Posts: 1721
Joined: Wed May 18, 2005 5:57 pm
Location: Sol III, Sol system, Sector 001, Alpha Quadrant

Re: UKNOF 43 CVE

Fri Mar 29, 2019 12:15 pm


You've had close to a year to work on a fix...
If you take a look at original post there is a link and quote.
From what i understand original CVE, was not considered a vulnerability until 2nd one come along. And details of that will be revealed on given date.
So all this "close to year" shouting is overestimation. So i suggest to keep calm and wait for release, as MikroTik admitted 2nd CVE as vulnerability.
With great knowledge comes great responsibility, because of ability to recognize id... incompetent people much faster.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 12:19 pm

So all this "close to year" shouting is overestimation. So i suggest to keep calm and wait for release, as MikroTik admitted 2nd CVE as vulnerability.
Second "bug" was acknowledged by MikroTik on 2018-04-20.
Marek
 
Dude2048
Frequent Visitor
Frequent Visitor
Posts: 93
Joined: Thu Sep 01, 2016 4:04 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 12:47 pm

Quote from second thread:
Yes, it is kernel level and is very hard to fix, since RouterOS v6 has an older kernel version and we can't just change the kernel.
Is that v7 announcement? :D Hurray!
You have made my day!
 
the.max
just joined
Posts: 9
Joined: Sun Apr 01, 2007 3:47 pm
Location: Czech Republic, Bilina
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 12:58 pm

The fix is in v7 guys c'mon
Ros 7 is like the Yeti or Mrs. Colombo. Everyone talks about it, but nobody has ever seen it.
Gentoo linux
 
mkx
Forum Guru
Forum Guru
Posts: 3173
Joined: Thu Mar 03, 2016 10:23 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 1:14 pm

For those who won't notice it otherwise: MT just announced ROS 6.45 beta version which includes fix for these two issues.

Hopefully fix will land in other (stable and long term) branches shortly.
BR,
Metod
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 1:35 pm

For those who won't notice it otherwise: MT just announced ROS 6.45 beta version which includes fix for these two issues.

Hopefully fix will land in other (stable and long term) branches shortly.
CVE-2018-19299 is not fixed in 6.45beta22, I am afraid.
Marek
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Mar 29, 2019 2:16 pm

Hey maznu

Would you mind posting a link to your presentation / video on this forum once it's presented?
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 2:17 pm

For those who won't notice it otherwise: MT just announced ROS 6.45 beta version which includes fix for these two issues.

Hopefully fix will land in other (stable and long term) branches shortly.
CVE-2018-19299 is not fixed in 6.45beta22, I am afraid.
Please clarify
No answer to your question? How to write posts
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:00 pm

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish.
No answer to your question? How to write posts
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:00 pm

For those who won't notice it otherwise: MT just announced ROS 6.45 beta version which includes fix for these two issues.

Hopefully fix will land in other (stable and long term) branches shortly.
CVE-2018-19299 is not fixed in 6.45beta22, I am afraid.
Please clarify

https://www.youtube.com/watch?v=TzC-JVjMK8k

And if you need a diagram of the lab:

https://twitter.com/maznu/status/1111589812111319041

You already know the commands I'm running to launch the attack, because I supplied them to you in emails on 2018-04-17.
Last edited by maznu on Fri Mar 29, 2019 3:05 pm, edited 1 time in total.
Marek
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:03 pm

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish.
There has been plenty of communications on this matter, normis. The most recent, specifically about what I plan to publish, was an email in Ticket#2018040822000592 on 2019-03-04, which Martins S subsequently replied to saying that you had no update:
The UK Network Operators' Forum has accepted my talk about this subject: "Scanning IPv6 Address Space… and the remote vulnerabilities it uncovers"

https://indico.uknof.org.uk/event/46/co ... s/speakers

I shall be discussing IPv6 neighbor discovery exhaustion, and also how RouterOS will crash when routing IPv6 packets, i.e. both vulnerabilities I have disclosed to MikroTik in April 2018, currently unpublished as CVE-2018-19298 and CVE-2018-19299.

Do you think that MikroTik will have an update about these vulnerabilities that I can include in my presentation on April 9th?
I would be more than happy to send you my slides… just drop me an email. It's on the ticket.
Last edited by maznu on Fri Mar 29, 2019 3:06 pm, edited 1 time in total.
Marek
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:06 pm

We fixed the crashes that were reported to us. You said, we have not fixed "The CVE". I don't know what you will publish in the CVE. You have only provided a video that doesn't help at all. If you can reproduce an issue that we can't reproduce, please email support and describe the method you used now, after beta 22.
No answer to your question? How to write posts
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:09 pm

We fixed the crashes that were reported to us. You said, we have not fixed "The CVE". I don't know what you will publish in the CVE. You have only provided a video that doesn't help at all.
The CVE, CVE-2018-19299, was communicated to you in October 2018. It is literally just the number that MITRE assigned when I registered two CVEs for two separate issues:

CVE-2018-19298 = NDP exhaustion
CVE-2018-19299 = IPv6 routing exhaustion

You know the details. They're on your own tickets. The commands you need to run to trigger the exploit were given to you last year.

Let's stop playing this out in public. Drop me an email. I actually want you to fix this, not turn it into a pantomime, please.
Marek
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:19 pm

This version fixes:
1) Soft lockup when IPv6 router is forwarding IPv6 packets;
2) Soft lockup when the router is forwarding packets to a local network (directly connected) due to large IPv6 Neighbor table.

We are still working on improvements for IPv6 Neighbor table processing in userspace which can lead up to OOM condition.
Since CVE detials are not yet published, we did assume that CVE targets software lockup (#2) which we did fix in this release.
No answer to your question? How to write posts
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:23 pm

This version fixes:
1) Soft lockup when IPv6 router is forwarding IPv6 packets;
2) Soft lockup when the router is forwarding packets to a local network (directly connected) due to large IPv6 Neighbor table.

We are still working on improvements for IPv6 Neighbor table processing in userspace which can lead up to OOM condition.
Since CVE detials are not yet published, we did assume that CVE targets software lockup (#2) which we did fix in this release.
I've sent through all the details in a new ticket, Ticket#2019032922005182, which I hope includes all the information you need collated into one place.

I'm very glad you've prioritised this issue with your development team, and I look forward to testing releases that address the problem soon.
Marek
 
msatter
Forum Guru
Forum Guru
Posts: 1277
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: UKNOF 43 CVE

Fri Mar 29, 2019 3:35 pm

Thanks Maznu for finding this and reporting it to Mikrotik. Good to see that the communications is up-to-speed now so that Mikrotik can handle this correctly and in time for us Mikrotik device owners.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2291
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:28 pm

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish.
Normis - "Maznu" wrote the dates and when sent it to you. Ticket numbers... Description... All this a year ago. What do you need more?

Maznu - Thank you for your comments and activity
Last edited by honzam on Fri Mar 29, 2019 4:30 pm, edited 1 time in total.
LAN, FTTx, Wireless. ISP operator
 
highonsnow
newbie
Posts: 34
Joined: Fri Oct 19, 2007 3:30 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:30 pm

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish.
"maznu" wrote the dates he sent it to you. Ticket numbers. Description... All this a year ago. What do you need more?

Maznu - Thank you for your comments
It's quite clear that this wasn't given much priority.. only since there was a fire lit under the subject did it get pounded into a beta
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:43 pm


Normis - "Maznu" wrote the dates and when sent it to you. Ticket numbers... Description... All this a year ago. What do you need more?
No. He did not send proof of concept for all issues, just a generic report about a crash. When he now said that CVE number such and such is not fixed, It was not clear, since we don't know what he will publish in that CVE. There is not a single issue, there are multiple issues, we fixed most, now he has stumbled upon another (memory leak in some other condition). We are fixing the others as well.
No answer to your question? How to write posts
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:46 pm

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.
No answer to your question? How to write posts
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:53 pm

Can I run this on my edge routers and use my /32 or do I need to do it on each router?

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5940
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 4:56 pm

It should be enough to limit on edge router, since it already limits to 2 new connections every second, unless routers further along the path have less than 100MB free ram, then probably you will need to limit even more on that specific router.
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 993
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 5:01 pm

I think we should listen to what he has to say …

many many many many many years ago, I had both Microsoft and Sun Microsystems on the phone telling both of them I could lockup their Internet connected computers. They did not believe me so with both of them online in a conference call to both, I locked up the computers on the IP addresses they told me to try. Well - that little easy lockup trick is called "Ping of Death".
Yup - that was me that discovered it and worked with both Sun and Microsoft verify they later fixed their TCP/IP stack buffer-over-run problem.

Another one with Microsoft that would totally lockup a Windows computer was the left over PIP "ConCon" ( a carry over from the original CPM operating system ).

The point I am trying to make is that if somebody says there is a vulnerability, I wish they would get together in a live on-line conference and show their stuff and prove their point - prior to releasing the vulnerability to the general public.

North Idaho Tom Jones
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 993
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 5:10 pm

I think we should listen to what he has to say …

many many many many many years ago, I had both Microsoft and Sun Microsystems on the phone telling both of them I could lockup their Internet connected computers. They did not believe me so with both of them online in a conference call to both, I locked up the computers on the IP addresses they told me to try. Well - that little easy lockup trick is called "Ping of Death".
Yup - that was me that discovered it and worked with both Sun and Microsoft verify they later fixed their TCP/IP stack buffer-over-run problem.

Another one with Microsoft that would totally lockup a Windows computer was the left over PIP "ConCon" ( a carry over from the original CPM operating system ).

The point I am trying to make is that if somebody says there is a vulnerability, I wish they would get together in a live on-line conference and show their stuff and prove their point - prior to releasing the vulnerability to the general public.


Question - Am I correct in guessing this is something like "Packet of death in IPv6" , pretty much the same issue in IPv6 that is called Ping of Death in IPv4 ?

North Idaho Tom Jones
Last edited by TomjNorthIdaho on Fri Mar 29, 2019 5:12 pm, edited 1 time in total.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 5:12 pm

No. He did not send proof of concept for all issues, just a generic report about a crash. When he now said that CVE number such and such is not fixed, It was not clear, since we don't know what he will publish in that CVE. There is not a single issue, there are multiple issues, we fixed most, now he has stumbled upon another (memory leak in some other condition). We are fixing the others as well.
I sent support@mikrotik.com two tickets in April 2018 about two different issues. Later, in November, I advised MikroTik that these had now been allocated two different CVE numbers.

Both tickets in April were acknowledged and accepted as deficiencies by support@mikrotik.com, and I had given you the exact commands I had used so that you could reproduce them. Your team wrote back to my saying:
We have tested the same on our local network and managed to reproduce the same problem. We will see what we can do about this problem and improve IPv6 functionality in future RouterOS releases.

Your report and input into reproducing this problem is higly appreciated.
In those initial emails, almost a year ago, I even gave your support team my ideas about what I thought the problems were and — guess what! — your support team just told me about an hour ago that they believe the they've now found some problems to be — and they are exactly what I said in April 2018.
Last edited by maznu on Fri Mar 29, 2019 5:53 pm, edited 1 time in total.
Marek
 
dottxt
just joined
Posts: 15
Joined: Sun Feb 02, 2014 5:53 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 5:20 pm

For CVE-2018-19299, Are systems that do not have IPv6 connection tracking enabled affected?
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 5:23 pm

For CVE-2018-19299, Are systems that do not have IPv6 connection tracking enabled affected?
Yes.
Marek
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 6:15 pm

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.
Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.

I'm repeating the test several times just to make sure we have it right. Should this mitigate both CVEs?
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 6:23 pm

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.
May I please add "discovered independently by a third party" to the timeline and credit you during my UKNOF 43 talk, then?
Marek
 
eles
just joined
Posts: 4
Joined: Tue Jun 18, 2013 6:20 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 6:43 pm

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.
May I please add "discovered independently by a third party" to the timeline and credit you during my UKNOF 43 talk, then?
The post @IPANetEngineer was responding to, was Normis's.
--
MTCNA #1905NA3991
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 6:45 pm

I've been watching this quietly since it started. One of these CVEs, while important to note, is pretty straightforward to replicate with off the shelf, open source tools. I suspect the other is as well but I have not yet done so. MT should definitely address this and add testing of neighbor table issues to the ROS CI/CD process. I would also recommend adding a tunable for limiting the neighbor table size per device for those of us that wish to set hard limits on ND cache to a known quantity.
Unfortunately the "third party" who "discovered" the issue has been very cagey and a bit sensational about the discovery, which seems to have 1. hyped up the talk he is to give and 2. get folks riled up and in knee-jerk mode.
MT has never been an organization that has reacted in flamboyant ways when it comes to fixing bugs or vulnerabilities - they simply assess the situation and act based on their internal assessment. Sure, it would have been nice to see this fixed faster, but they didn't, and we don't know the back channel communication that happened when it was initially reported or how it was presented.
My point is that these aren't cryptic, esoteric issues. They are common IPv6 implementation problems that anyone running v6 at scale has probably seen in years past with other vendors. They have a magnified spotlight on them since ROS has had a year of high profile issues.
Keep calm, we're gonna be ok.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Mar 29, 2019 6:59 pm

/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
2001:db8:3::/64 new-connection-mark=drop passthrough=yes[/code]
Looking at the remaining workaround, usual end-user blocking any incoming traffic not already established / related, isn't impacted, right?
 
User avatar
davidcx
just joined
Posts: 16
Joined: Tue Nov 14, 2017 7:06 pm

Re: UKNOF 43 CVE

Fri Mar 29, 2019 7:04 pm

Looking at the remaining workaround, usual end-user blocking any incoming traffic not already established / related, isn't impacted, right?
You'd be limiting traffic to 2 new flows per second which is not an option except in the tiniest of networks.

Now, there may be a higher value that is acceptable to your conditions (if you are not a transit network where you have no business dictating to your customers how many flows they may have), but whether that mitigates the bug in question - specifically whether the memory leaked is recovered at a faster rate than the rate limit prevents new flows - remains to be seen when the bug details are released.

-davidc
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Mar 29, 2019 7:14 pm

Not what I mean.

Looking at the remaining workaround, usual end-user blocking any incoming traffic not already established / related, isn't impacted, right?

Usual end-user scenario, so not a major network operation, is to block any incoming traffic, unless it is related to already established traffic from inside. So accepting internal risk, users dropping their own internet pipe, nothing extra needs to be done since, it's going to be dropped anyway if a new connection arrives from internet direction internal network.
Last edited by sebastia on Fri Mar 29, 2019 7:43 pm, edited 1 time in total.
 
User avatar
luciano
just joined
Posts: 10
Joined: Fri Nov 25, 2005 12:32 am
Location: Ponta Grossa/PR
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 7:27 pm

What happy friday! Disabling BGP IPv6 Peers on our clients...
Computers are like air-conditioners. When you open "Windows" they stop work.

My bio is here: http://www.about.me/luciano_santos
MTCNA and MTCRE
 
anav
Forum Guru
Forum Guru
Posts: 3091
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: UKNOF 43 CVE

Fri Mar 29, 2019 7:30 pm

They have to fix the code without breaking the rest of it. Appears this is why it took so long. If they are changing something at a deep deep root level, then the ramifications will spread out like spider webs and each strand has to be dealt with. Not surprising that it takes time. All the while they have been dealing with other exploits in quick fashion and as well handling the more mundane improvements.
IF anyone likes extraordinary amounts of stress just apply to work at MT. Its not all pizza and beer and video games. Speaking about thirst, I have some Canadian free beer for any MT employee that comes to Canada. :-)
Or Normis can fly me to Latvia and I will bring some beer (but only after ipv6 is fixed, no distractions needed at the moment).
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
User avatar
leopiri
just joined
Posts: 5
Joined: Fri May 08, 2009 9:25 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:29 pm

Maznu, thank you for showing that you are seeking a solution for the whole community.
Could you inform me if disabling SSH and Winbox service also works the exploit?
Using RoMon only can be a "temporary" solution?

Appreciate.
Leonardo
This version fixes:
1) Soft lockup when IPv6 router is forwarding IPv6 packets;
2) Soft lockup when the router is forwarding packets to a local network (directly connected) due to large IPv6 Neighbor table.

We are still working on improvements for IPv6 Neighbor table processing in userspace which can lead up to OOM condition.
Since CVE detials are not yet published, we did assume that CVE targets software lockup (#2) which we did fix in this release.
I've sent through all the details in a new ticket, Ticket#2019032922005182, which I hope includes all the information you need collated into one place.

I'm very glad you've prioritised this issue with your development team, and I look forward to testing releases that address the problem soon.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Mar 29, 2019 10:52 pm

Maznu, thank you for showing that you are seeking a solution for the whole community.
Could you inform me if disabling SSH and Winbox service also works the exploit?
Using RoMon only can be a "temporary" solution?
How you access the router isn't the major factor here… I'm not sure I understand your question?
Marek
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Fri Mar 29, 2019 11:36 pm

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.
Normis:

That may work for a small end user network. It does not work for a medium sized internet provider.

Running beta releases on production networks ALSO does not work for an internet provider.

I will have to decomission at least 12x 72core and 36core CCRs in the next week and replace them with something from another vendor. Having ignored requests for implementing IPv6-Delegated-Prefix on PPPoE for years !! is one of the reasons we would have done this anyhow.

Another reason is that we will be migrating away from Mikrotik entirely. And we have 1000s of Mikrotik devices in the network. When it takes over a week or more to get a response to a support ticket, then that is not acceptable. If you get a response at all. And that is the norm currently.

But ignoring a bug like this for a year is taking the biscuit. Also .. it was reported to Mikrotik. If Mikrotik were not sure or did have to think about, what and how these exploits were achieved it would have been Mikrotiks RESPONSIBILITY to engage with the person reporting it, to make sure you understood the issue correct. It is not the reporter that has the responsibility for your products and your customers. It is Mikrotik.

Guessing, that you have fixed the issue, is not good enough. If you thought you had fixed it and were not sure, you should have engaged with the person reporting it AFTER the fix was released and made sure you had all bases covered. That is if you care about your business and your customers.

/M




Last edited by marlow on Fri Mar 29, 2019 11:36 pm, edited 1 time in total.
Communication is the beginning of understanding
-- AT&T
 
cmurrayis
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri May 15, 2009 4:31 am

Re: UKNOF 43 CVE

Fri Mar 29, 2019 11:43 pm

@normis while it's great that you plan on fixing the issues by the release date it doesn't help the rest of the world who have to patch their networks before this time also.

We have to plan outages months in advance for some customers and even for our core it's 7 days notice for critical patches which first have to go through stability testing in lab first. This leaves is with not much time to be confident no new issues are introduced.
 
killersoft
Member Candidate
Member Candidate
Posts: 134
Joined: Mon Apr 11, 2011 2:34 pm
Location: Victoria, Australia
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 1:44 am

While many of you are notably upset about the extraordinary amount of time that has gone by on this issue. I note some of you are wanting to move to new product vendors. This is your prerogative to do so. That said, I will point out the BIG VENDORS such as CISCO are smashed by CVE's problems ALL the time. Mikrotik is doing pretty well on the CVE footprint issue comparatively.

Take a look at cisco CVE's just this year I count 130 as of this writing, out of 3100 since 1995.
https://tools.cisco.com/security/center ... nListing.x

Makes you wonder what hackers/etc haven't publicly released with known bugs in those big 'brand' name products...

So even if you PAY the big bucks, your probably getting an 'Average' 'network' product(comes supplied with buggy O/S's) with chrome trim's and a nice help desk support number, that may/may-not be able to assist in a timely manor.

You are all lucky that for the relatively cheap(very cheap!) price of MT products, you get a beta release every ~1-3 weeks, while the one O/S RouterOS in this case is being improved.
Last edited by killersoft on Sat Mar 30, 2019 1:55 am, edited 1 time in total.
MIT, BIT, ITIL, CERT IV Electronics.
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 1:49 am

While many of you are notably upset about the extraordinary amount of time that has gone by on this issue. I note some of you are wanting to move to new product vendors. This is your prerogative to do so. That said, I will point out the BIG VENDORS such as CISCO are smashed by CVE's problems ALL the time. Mikrotik is doing pretty well on the CVE footprint issue comparatively.

They are doing well on the ones that go public, yes. The issue here is not just these issues. It's the whole approach on handling support requests and the time it takes to get a response, changing things in RouterOS effectively breaking working features without even discussing the reasoning for these changes, not implementing fundamental features.

Also, a lot of bugs don't get reported anymore, because we know it doesn't lead anywhere anyhow.

At some point enough, is enough. And yes, other vendors have other issues. Other vendors may also be more costly. But at least other vendors take responsibility for their products, have a clear guideline what a timely response to a ticket is and implement critical features, that customers and the industry needs. Again .. a good example: it's taken Mikrotik over a decade to implement IPV6-Delegated-Prefix, but then they decided only to implement it for DHCP and not for PPPoE. I mean .. where is the logic in that ?

They have also recently (as of 6.43) made it mandatory to use MS-CHAP for user-authentication via radius. So whoever was using CHAP .. which would be an industry standard .. vs MS-CHAP being a proprietary extension ... got their entire setup broken.

These are just example that show how they treat their customers.

The handling of these 2 CVEs reflect the same fundamental problems within Mikrotik. A workaround then gets posted for one of the issues, that works for an end-user network, but essential would cripple the network of an ISP that has an actual IPv6 rollout with thousands of customers using IPv6. Again ... the bigger picture gets missed. Because is Mikrotiks biggest market end-users ? or providers ?

/M
Communication is the beginning of understanding
-- AT&T
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 2:03 am

We believe that we have now recreated the conditions of both CVEs and have been able to cause a memory leak and router crash in both of the conditions listed below using software from a common offensive linux security tool for IPv6.

soft lockup when forwarding IPv6 packets (CVE-2018-19299);
soft lockup when processing large IPv6 Neighbor table (CVE-2018-19298)

Our engineers are working on mitigation options and so far have been able to keep the test routers online by limiting to 5k new connections with 256MB of memory. We still have some testing to do and a number of different options to try.

Will post our rule sets as we further refine the testing but it is a variation on what Normis already posted.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
killersoft
Member Candidate
Member Candidate
Posts: 134
Joined: Mon Apr 11, 2011 2:34 pm
Location: Victoria, Australia
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 2:05 am

At some point enough, is enough. And yes, other vendors have other issues. Other vendors may also be more costly. But at least other vendors take responsibility for their products, have a clear guideline what a timely response to a ticket is and implement critical features, that customers and the industry needs. Again .. a good example: it's taken Mikrotik over a decade to implement IPV6-Delegated-Prefix, but then they decided only to implement it for DHCP and not for PPPoE. I mean .. where is the logic in that ?

OK, fair enough on the timelines, but I ask what are your PAYING $ for MT support at the moment.... ?

Also if your unhappy with RouterOS on your hardware, feel free to flash OPENWRT onto your MT hardware devices, and become pro-active in network code design and development for the community
https://openwrt.org/toh/hwdata/mikrotik/start
MIT, BIT, ITIL, CERT IV Electronics.
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 2:40 am

Also if your unhappy with RouterOS on your hardware, feel free to flash OPENWRT onto your MT hardware devices, and become pro-active in network code design and development for the community

You will find with a little of research, that I'm very active in the OpenSource community. Especially OpenWRT / Freemesh / Freifunk. But that's neither here nor there.

The issue is, that I'm also a high volume customer of Mikrotik products .. or have been so until recent times. And that pays for the development of said features. I buy 100s of Mikrotik units every year and have done so since 2005. Every single year. So don't tell me, I don't pay them for the support and development, that I expect.

The software is not free. It's paid for as part of either the license or the hardware purchase.

/M
Communication is the beginning of understanding
-- AT&T
 
killersoft
Member Candidate
Member Candidate
Posts: 134
Joined: Mon Apr 11, 2011 2:34 pm
Location: Victoria, Australia
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 2:49 am

Maybe its time for MT to consider a parallel "community" like edition version of RouterOS. That open to view /compile "source code" and allows the community to quickly fix issues(CVE's !!!) and add networking functionality as community made plugin's for MT Hardware..
MIT, BIT, ITIL, CERT IV Electronics.
 
User avatar
leopiri
just joined
Posts: 5
Joined: Fri May 08, 2009 9:25 am

Re: UKNOF 43 CVE

Sat Mar 30, 2019 3:13 am

Maznu, do the following:

ip service disable [find]

Verify that even with all Mikrotik access media services the problem occurs?
Maznu, thank you for showing that you are seeking a solution for the whole community.
Could you inform me if disabling SSH and Winbox service also works the exploit?
Using RoMon only can be a "temporary" solution?
How you access the router isn't the major factor here… I'm not sure I understand your question?
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 3:24 am

Maybe its time for MT to consider a parallel "community" like edition version of RouterOS. That open to view /compile "source code" and allows the community to quickly fix issues(CVE's !!!) and add networking functionality as community made plugin's for MT Hardware..

Here is your fundamental issue:
In RouterOS 2.9 and before, the routing suite was still based on Quagga (OSPF, BGP etc.)

From RouterOS 3 it was replaced by their own code introducing all sorts of issue and behaviour changes.

The main reason for the change was because Mikrotik did not want to release the source code of their platform. This has happened to other parts of the RouterOS in a similar fashion.

So you have an inherent issue that is, that Mikrotik is not interested in Open Source. The reason I have stuck around for so long is because they have managed to provide a fully integrated platform, with an easy to manage interface, at a cost point that is affordable. That including optimizing (proprietary) wireless protocols, that exceed standard 802.11 in performance and are much more interference resilient.

So over a little more than a year, we have started to migrate the end user side/last mile of our business to other vendors. The edge has never been Mikrotik. The missing part is now the RAS side of things. And that's where these two CVEs hit extremely hard. Because I've provided IPv6 to end users since 2008... by default. Whatever way we could make it work. And traffic volumes on IPv6 are not marginal anymore. It's 20-30% plus.

/M
Last edited by marlow on Sat Mar 30, 2019 3:28 am, edited 1 time in total.
Communication is the beginning of understanding
-- AT&T
 
User avatar
leopiri
just joined
Posts: 5
Joined: Fri May 08, 2009 9:25 am

Re: UKNOF 43 CVE

Sat Mar 30, 2019 3:26 am

Do not forget the providers that use FastPath, if activating any rule in the equipment can bump the cpu.

I'm considering a CCR1072 with 15Gb + or CCR1036 with 6Gb +.

Is this fix you're testing going to kill FastPath?

We believe that we have now recreated the conditions of both CVEs and have been able to cause a memory leak and router crash in both of the conditions listed below using software from a common offensive linux security tool for IPv6.

soft lockup when forwarding IPv6 packets (CVE-2018-19299);
soft lockup when processing large IPv6 Neighbor table (CVE-2018-19298)

Our engineers are working on mitigation options and so far have been able to keep the test routers online by limiting to 5k new connections with 256MB of memory. We still have some testing to do and a number of different options to try.

Will post our rule sets as we further refine the testing but it is a variation on what Normis already posted.
 
aussiewan
just joined
Posts: 24
Joined: Wed Sep 07, 2011 5:28 am

Re: UKNOF 43 CVE

Sat Mar 30, 2019 4:05 am

The suggested temporary workaround also does not work if you have connection tracking disabled, as we do for some of our higher-throughput devices.
I think that a key takeaway from this issue is that you shouldn't put all your eggs in one basket. Don't rely on any single vendor for your critical infrastructure. Every vendor has issues from time to time, and if a single issue like this can take your business offline, then you need to re-assess your systems and look into adding some other vendors into the mix. In this case, you don't have redundancy if you have all MikroTik devices running your Internet edge.
 
rkj
just joined
Posts: 15
Joined: Sun Jun 11, 2006 7:38 pm

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:22 am

While the response for this security issue has been sub-optimal, Mikrotik attitude towards criticism comes from long time and also includes lack of features, bugs and anything different from using what is available.

So while this is an escalation, it's a foreseeable development of the existing trend. Anyone using MT should know that by now.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 8:33 am

Maznu, do the following:

ip service disable [find]

Verify that even with all Mikrotik access media services the problem occurs?
Yes, that is still vulnerable (my test lab has no services enabled because it has no Internet connectivity - only console access). These IPv6 handling problems are not about accessing the router. They are about causing the router to crash by how it routes IPv6 packets.
Marek
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 10:30 am

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.
Meanwhile CVE-2018-19299 still needs fixing, because even with those performance-crippling firewall rules on 6.45beta22, I'm crashing routers.

In exactly the same lab layout as before, and with exactly the same single command that MikroTik has known for over 11 months, and still attacking 2001:db8:3::/64 (so your firewall rules are just a copy-and-paste): https://youtu.be/vJBUdAMrKJw
Marek
 
mistry7
Forum Guru
Forum Guru
Posts: 1323
Joined: Tue Oct 13, 2009 11:57 am
Location: Germany

Re: UKNOF 43 CVE

Sat Mar 30, 2019 10:37 am

Mikrotik what is need to happen that you are wake up and get ROS 7 with newer Kernel ready?

ROS Kernel is from 2012!

Good for us, we removed all Mikrotik at Core Routers in the last 3 Month, because of bad BGP Speed, and Missing IPv6 PPPoE Features
No i Think that was a very good decision!
 
rockymtndata
just joined
Posts: 5
Joined: Sat Jul 13, 2013 10:00 pm

Re: UKNOF 43 CVE

Sat Mar 30, 2019 11:03 am

Mistry7,

Good call moving off of mikrotik for security reasons. I couldn't agree more. I did too for some things. Today is a security day for us Cisco fans too. There are 17 flaws to fix. Have you seen this.

https://www.networkworld.com/article/33 ... flaws.html

R
 
mistry7
Forum Guru
Forum Guru
Posts: 1323
Joined: Tue Oct 13, 2009 11:57 am
Location: Germany

Re: UKNOF 43 CVE

Sat Mar 30, 2019 11:23 am

Mistry7,

Good call moving off of mikrotik for security reasons. I couldn't agree more. I did too for some things. Today is a security day for us Cisco fans too. There are 17 flaws to fix. Have you seen this.

https://www.networkworld.com/article/33 ... flaws.html

R
Fixes and Sec Holes are all over the world, we did change not from security side, but Mikrotik don't tells when they are fixing there missing IPv6 PPPoE
Features, and we are not willing to wait until we are out of IPv4 Addresses, so we need a working solution until Mikrotik can't do the job.

viewtopic.php?f=1&t=89443&start=150

Needed Features are requested years ago....same story as ROS 7, and Missing Wireless Features......
But there are others on the Market, and they have this features....
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 1:47 pm

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.
Meanwhile CVE-2018-19299 still needs fixing, because even with those performance-crippling firewall rules on 6.45beta22, I'm crashing routers.

In exactly the same lab layout as before, and with exactly the same single command that MikroTik has known for over 11 months, and still attacking 2001:db8:3::/64 (so your firewall rules are just a copy-and-paste): https://youtu.be/vJBUdAMrKJw
We have had some success mitigating CVE-2018-19299 now that we know what command to run in the linux IPv6 exploit tools to crash edge or transit routers. However, we are moving testing over to our dual stack data center on hardware that isn't serving customers to get a better idea of the real world IPv6 results as labs disconnected from the Internet can produce results that don't always translate to the real world in the same way.

The initial hardware we'll be testing will be RB4011s and CCRs. We'll be working on expanding the MikroTik FW rules that we are already working on as well as developing L2 scrubbing policies

While i'm very eager to get a fix from MikroTik, I'm going to keep working on improving our mitigation and will share the results with the group once we've done some testing on actual hardware in a prod IPv6 deployment.
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
pcunite
Forum Veteran
Forum Veteran
Posts: 945
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: UKNOF 43 CVE

Sat Mar 30, 2019 2:43 pm

I want to reiterate what I've stated elsewhere: I believe that all modern software companies need to implement a certain type of business process, known as Lean-Agile. Bugs will happen, no reasonable person is upset about that. Rather it is the release cadence, release channel, and review process that are broken. MikroTik's staff are not incompetent, to suggest that is insulting to them and frankly to those of us who rely on their products.

Nomis, when you get a chance, read that book I linked to. I think it will really help MikroTik. We all want you to win and do well. That is why there is a lot of passion in these comments. The angry outbursts are not about this particular CVE, instead it is the groundswell of a systemic pattern of frustrating end-user experience updates that are causing the hopelessness. It is a confidence issue. If I'm scared to apply your updates, I wonder how others feel? Truth or perception. Which one do we choose?
 
dmayan
just joined
Posts: 22
Joined: Sun Nov 10, 2013 9:28 pm

Re: UKNOF 43 CVE

Sat Mar 30, 2019 3:47 pm

We will never see it. Mikrotik is more concerned at improving Kids Control and other useless features than working on a better platform for IPv6 and BGP multi threading for the CCR series. We have nine CCR1072 that can't receive full tables because they literally froze processing them. Already thinking on changing our routing equipment, and we are one of the biggest ISPs of the south of South America. Now I have to disable IPv6 (We are the FIRST provider implementing this on our region) because of this bold***t.

And not speaking about CCRs rebooting itself with no reason and other things like that....

Keep up the good work Mikrotik
The fix is in v7 guys c'mon
Ros 7 is like the Yeti or Mrs. Colombo. Everyone talks about it, but nobody has ever seen it.
 
neutronlaser
Member Candidate
Member Candidate
Posts: 212
Joined: Thu Jan 18, 2018 5:18 pm

Re: UKNOF 43 CVE

Sat Mar 30, 2019 3:56 pm

You don't have to disable it.
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 4:35 pm

You don't have to disable it.

The fixes posted above will result in useless user experience when implemented by an internet service provider, which is a no go either. So disabling/removing IPv6 from public routers is the only choice here until a fix is in a STABLE release.

So yes... as an internet provider, that has to deliver service to thousands of customers, it's actually the only choice right now.

/M
Communication is the beginning of understanding
-- AT&T
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 4:44 pm

Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure. As a relative outsider that has been involved in this kind of thing in the past, my observation is that this entire process has been bungled beyond belief. I feel like I am watching a slow moving train wreck where the train engineer (@maznu) is doing almost nothing other than complain that nothing is being done to keep the train from crashing whilst promoting his talk that he intends to give to take advantage of the public black eye that MT has from previous high profile issues. Fine, I get it. It's an easy way to self promote (although not the way I would have handled it).
The only person trying to actually mitigate the issue other than Mikrotik (which is expectedly a black box of which we cannot see inside), is @IPANetEngineer who has essentially reverse engineered the bugs independently and has potential temporary mitigations. Given what I have seen this is a solvable - albeit inconvenient - issue, that *can* be mitigated with a little work, but the users seem more interested in throwing hands up and turning off the v6 internet, and the author seems fixated on his talk rather than mitigating the issue.
Even as someone involved in building early broadband and very early IPv6 production networks where bugs were numerous and significant, this is painful to watch.

Also, no one that I have ever seen does multi-threaded BGPd. What you want is a more optimized BGP Daemon, which likely won't happen on a CCR because they the Tilera is optimized for power efficiency and not DFZ sized route table optimization. Lame? Yes. Unexpected? Not really.

nb
We will never see it. Mikrotik is more concerned at improving Kids Control and other useless features than working on a better platform for IPv6 and BGP multi threading for the CCR series. We have nine CCR1072 that can't receive full tables because they literally froze processing them. Already thinking on changing our routing equipment, and we are one of the biggest ISPs of the south of South America. Now I have to disable IPv6 (We are the FIRST provider implementing this on our region) because of this bold***t.

And not speaking about CCRs rebooting itself with no reason and other things like that....

Keep up the good work Mikrotik
The fix is in v7 guys c'mon
Ros 7 is like the Yeti or Mrs. Colombo. Everyone talks about it, but nobody has ever seen it.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
highonsnow
newbie
Posts: 34
Joined: Fri Oct 19, 2007 3:30 am

Re: UKNOF 43 CVE

Sat Mar 30, 2019 4:52 pm

Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure.

Are you suggesting that people apply last minute rushed patches days or hours before disclosure (who knows when it'll be released or stable)? Is that how you operate in your production environments? I'm quite sure any serious ISPs who are affected won't be applying such update policies without having at least tested or proven the firmware before throwing another variable into the mixture. The reaction is preventative and responsible.

Imagine a scenario where they actually took this one seriously and produced a solution for it in good time.. and not waiting until the gun was nearly placed to their head with the looming deadline. None of this would have been happening.
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 4:53 pm

Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure.

It is actually not.

MT having a fix ready before disclosure is not enough. The fix needs to be ready, it needs to be in a stable release tree, it needs to tested and validated and customers need time to implement it.

@maznu has provided a clear timeline of when this was reported to Mikrotik, when the CVEs were issued and Mikrotik has ignored this until an imminent threat of it going public came about. If they had used the time from the report and the creation of the CVEs to mitigate this, there would have been plenty of time.

So in my case, it means IPv6 will be removed from all Mikrotik gear, because the risk of loss of business is too great. And where necessary, the gear will have been replaced with gear from other vendors, that's long term proven. Even if Mikrotik released a fix in time, there would be no time for testing. I would literally have to roll it out across 300+ devices in the core network. We have near to 10k Mikrotik devices in our network, if every one of them needs to be updated. This is not something you do in a few hours .. or days. And automated updates are not an option. The quality control of the release trees supplied by Mikrotik are not good enough. We aren't even on the current long-term release yet, as it has bugs we can't live with.

And the issue is, that it is not the first time, that Mikrotik has handled an issue this way. It is every single time. They only react, when issues become public knowledge.

/M
Communication is the beginning of understanding
-- AT&T
 
anav
Forum Guru
Forum Guru
Posts: 3091
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:13 pm

If your networks are so huge and yet you have failed to scale your infrastructure accordingly so that updating them is not manageable, and yet you have money to change over many of your devices just tells me you have other issues to overcome before changing equipment over. Besides the fact that you actually believe that a proven long term vendor exists............. Tis a Brave New World Indeed.
In other words it is easy to throw stones without a complete picture or understanding of the whole situation. Suggest your continued ranting may make you feel somehow better, but it does nothing for readers. Perhaps you should put on your hat MMGA ( make mt great again ;-) )
Last edited by anav on Sat Mar 30, 2019 8:41 pm, edited 1 time in total.
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:19 pm

If your networks are so huge and yet you have failed to scale your infrastructure accordingly so that updating them is not manageable, and yet you have money to change over many of your devices just tells me you have other issues to overcome before changing equipment over.

You don't have the bigger picture here. This has nothing to do with scaling. We actually regularly update most of our network, semi-automated. But only after long term testing, because what is considered stable by Mikrotik is not considered stable by us. And sitting around on a bug like that for a year without even doing anything about it is negligence by Mikrotik.

Besides the fact that you actually believe that a proven long term vendor exists.............

When i say proven long term, then that refers to equipment we have tested long term in an aim to replace quite a bit of our Mikrotik infrastructure. This didn't just happen yesterday. The bug above just accelerates some of the replacements. Don't take my post out of context. I said, a vendor that is proven long term (by us). Not in general.

You can have a look on the left there .. under my avatar. I didn't just sign up here yesterday. I've been 12 years longer on this forum than you.

/M
Communication is the beginning of understanding
-- AT&T
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:39 pm

The ugly baby is how this has been dealt with over time, it isn't anyones process or workflow. Read my previous post. I stand by my statement - this has been handled poorly on all sides and because of this unnecessarily bad and totally avoidable mis-handling if it, we're forced to treat it like a zero-day. My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power. Therefor, since there has been a compressed timeline, for me, mitigation is the solution. I am in no way suggesting that anyone run untested software, we all know that MT is just as well known for [re-]introducing bugs in "stable" releases as it is for being economical. If you refer to my previous post, nowhere do I suggest doing anything close to that. What is disappointing is that we don't have enough folks working on a mitigation plan.
What I am advocating against is knee-jerk tactics, it's painting over the rust until such time as the metal can be repaired permanently. If disabling v6 is your solution, you do you. I would never suggest that someone has a workflow that is "wrong" if it works for them. Personally, I would rather spend the time necessary to operate in a degraded yet mitigated state than spend the cycles yanking an entire routed protocol out of a network to be re-enabled later, or never. My main point is that people should be working on as sane of a temporary solution as possible as opposed to right away throwing hands up and shutting things off, although if that's the last ditch effort, then that's what to do. My second point is that the author of the bug reports seems far more interested in poking things with a stick and giving his conference talk rather than providing insight into a possible temporary solution, which may impact the weight of his presentation.
By mining previous threads we have basic knowledge of what the issues are despite the author has being less than helpful about providing details. And again, it is implied that @IPANetEngineer has worked out a likely solution that is being tested more thoroughly as we speak.

nb


Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure.

It is actually not.

MT having a fix ready before disclosure is not enough. The fix needs to be ready, it needs to be in a stable release tree, it needs to tested and validated and customers need time to implement it.

@maznu has provided a clear timeline of when this was reported to Mikrotik, when the CVEs were issued and Mikrotik has ignored this until an imminent threat of it going public came about. If they had used the time from the report and the creation of the CVEs to mitigate this, there would have been plenty of time.

So in my case, it means IPv6 will be removed from all Mikrotik gear, because the risk of loss of business is too great. And where necessary, the gear will have been replaced with gear from other vendors, that's long term proven. Even if Mikrotik released a fix in time, there would be no time for testing. I would literally have to roll it out across 300+ devices in the core network. We have near to 10k Mikrotik devices in our network, if every one of them needs to be updated. This is not something you do in a few hours .. or days. And automated updates are not an option. The quality control of the release trees supplied by Mikrotik are not good enough. We aren't even on the current long-term release yet, as it has bugs we can't live with.

And the issue is, that it is not the first time, that Mikrotik has handled an issue this way. It is every single time. They only react, when issues become public knowledge.

/M
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:41 pm

My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power.

But this is where you are getting me wrong:

I'm not shutting IPv6 off on our network. We have been providing IPv6 to endusers since 2008. And even longer on the infrastructure. I'm turning it off on everything Mikrotik, because it is too big a risk.

And a good chunk of CCRs will have to replaced with alternative gear, not only because of this bug, but also because Mikrotik does not implemented features needed by the industry. I have tickets going back to before 2007 for some stuff that still isn't implemented. Can't even get a timeline.

There is plenty of hardware that is not affected.

/M
Communication is the beginning of understanding
-- AT&T
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:47 pm

despite the author has being less than helpful about providing details.
I have provided MikroTik with every detail at every step of the way.

I cannot provide anyone else with any more detail at all as this would literally give them the means to carry out the attack.

I have not shared any mitigation options because I do not believe there are any which do not involve significant expenditure of time and resources — something that many MikroTik users may not be able to afford.

I remain convinced that the fix can only come from MikroTik.
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:50 pm

Fair point, and great job on providing v6 back that far - few ISPs have that foresight. It's hard to infer context from a forum, and this thread runs the line of going "full-on UBNT forum" as it touches on a lot of peoples long held beliefs. Again, people should do what works for them. I'm just disappointed in the handling of this from soup to nuts.

nb
My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power.

But this is where you are getting me wrong:

I'm not shutting IPv6 off on our network. We have been providing IPv6 to endusers since 2008. And even longer on the infrastructure. I'm turning it off on everything Mikrotik, because it is too big a risk.

And a good chunk of CCRs will have to replaced with alternative gear, not only because of this bug, but also because Mikrotik does not implemented features needed by the industry. I have tickets going back to before 2007 for some stuff that still isn't implemented. Can't even get a timeline.

There is plenty of hardware that is not affected.

/M
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 5:58 pm

No, I get it, it's not my first rodeo with zero-day or high priority CVE, or with giving talks on any number of high sensitivity or previously embargoed subjects. I agree that you have said you provided the details to them and that handling has also been poor from their side, I trust that happened in one way or another. As far as the CVE, At least one of them is easily replicated using off the shelf tools, and is a pretty widely traveled technique (which frankly is a low hanging fruit in a pen-tester's toolbox). The other, I believe, is fairly straightforward (albeit egregious) as well, but again it's being pieced together with prior details in old threads. Likely not solvable at scale without a patch, but at least operationally minimized from the outside given the right strategy.

Having spoken my opinion I'm bowing out of this thread. My efforts are better spent on the mitigation plan.

nb
despite the author has being less than helpful about providing details.
I have provided MikroTik with every detail at every step of the way.

I cannot provide anyone else with any more detail at all as this would literally give them the means to carry out the attack.

I have not shared any mitigation options because I do not believe there are any which do not involve significant expenditure of time and resources — something that many MikroTik users may not be able to afford.

I remain convinced that the fix can only come from MikroTik.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
mstead
Member Candidate
Member Candidate
Posts: 113
Joined: Sat Mar 04, 2006 2:41 am

Re: UKNOF 43 CVE

Sat Mar 30, 2019 7:17 pm

I would make a good guess that the technique involves sending lots and lots of oversized neighbor discovery packets with the target IP of the victim. Easy to craft and while I never tested it, most likely could run the victim out of memory space if done at a high enough rate.
 
thantoldo
just joined
Posts: 22
Joined: Tue Apr 10, 2012 10:08 pm

Re: UKNOF 43 CVE

Sat Mar 30, 2019 8:10 pm

Meanwhile, Cisco dealing with its latest security vulnerabilities https://twitter.com/RedTeamPT/status/11 ... 6657238016
It can be much worse, guys..
Attitude to security should never be like ‘The squeaky wheel gets the grease’. Maznu just made it squeak!
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Sat Mar 30, 2019 9:02 pm

I would like to note that Buraglio isn't exactly green. He runs one of the largest capacity networks on the planet. I wouldn't dismiss his statements.
-----
Mike Hammett

The Brothers WISP
 
aussiewan
just joined
Posts: 24
Joined: Wed Sep 07, 2011 5:28 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:03 am

If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:16 am

If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
An alternative that doesn't necessarily mean we need to disclose vulnerabilities to each other or pass around weaponised PoCs is that MikroTik could send us the beta and we all bash it with our labs on Monday morning…?
Marek
 
aussiewan
just joined
Posts: 24
Joined: Wed Sep 07, 2011 5:28 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:36 am

@maznu - While I agree that sending around this potentially dangerous information has risks, I think that you have proven that you're more interested in a fix than having every IPv6-enabled MikroTik device in the world start spontaneously rebooting. Many others here are in the same boat. To clarify, I never intended it to sound like I wanted the information posted here, just shared directly between those who already have the information. I don't want to add to your workload, maznu, but it would probably make sense if those who have found methods sent you a "is this how you did it?" type message, if you're happy to receive them.

But yes, having another beta to stress-test sooner rather than later would also be fantastic!
If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
An alternative that doesn't necessarily mean we need to disclose vulnerabilities to each other or pass around weaponised PoCs is that MikroTik could send us the beta and we all bash it with our labs on Monday morning…?
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 1:42 am

Ideally this is how I would handle this. Again, we're super late in the game.
1. disclosed the environment hardware, in detail, that was used to test and confirm the the issue in.
2. have both validated it with a trusted, embargoed outside source(s). Ideally one is the vendor, clearly that didn't happen. 3rd party validation is pretty important either way as it proves a problem can be repeated independently.
3. If unable to define and at least indicate that there is a remediation, leverage outside trusted source to make sure that’s not possible. This seems to be kept between the person who discovered it and Mikrotik, who we *know* isn’t going to say anything. If getting credit for it is that important, agree upon that beforehand, it's doable and pretty straightforward with the right trusted sources.
A problem like this that is as egregious as described needs to be independently validated and ideally a remediation found. If there isn’t one, fine, Mikrotik has their head on the block, but we don’t know, and that’s the problem. At this point it would be sublime if @manzu could work with someone under embargo to validate this independently.

nb
If you haven't already, I would strongly encourage those of you who discovered and reverse engineered these bugs to compare notes and check that they are in fact the same methods - the last thing we need is for MikroTik to release a fix for the original issue, and then find that those who reverse engineered it discovered a related but different issue that is still unfixed.
An alternative that doesn't necessarily mean we need to disclose vulnerabilities to each other or pass around weaponised PoCs is that MikroTik could send us the beta and we all bash it with our labs on Monday morning…?
ForwardingPlane, LLC
https://www.forwardingplane.net
 
AlexS
Member Candidate
Member Candidate
Posts: 258
Joined: Thu Oct 10, 2013 7:21 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 9:49 am

My opinion is clear: IPv6 is a required service, disabling it is akin to shutting off power.

But this is where you are getting me wrong:

I'm not shutting IPv6 off on our network. We have been providing IPv6 to endusers since 2008. And even longer on the infrastructure. I'm turning it off on everything Mikrotik, because it is too big a risk.

And a good chunk of CCRs will have to replaced with alternative gear, not only because of this bug, but also because Mikrotik does not implemented features needed by the industry. I have tickets going back to before 2007 for some stuff that still isn't implemented. Can't even get a timeline.

There is plenty of hardware that is not affected.

/M
Have to agree with the above. I have been a Mikrotik user for 8+ years - loved the devices at first. still like them now. But I hate being let down by the developers...
So some grips - apologies if these have been addresses - its been a while since i followed the forums
* single threaded BGP ... sigh
* 1G max through put for single stream tcp on ccr.. I believe this is again a cpu (single vs multi) issue on the ccr1072. - again going to be fixed in V7
* ability to share session state between routers - i raised this as feature when they announced it in linux. just add the daemon and it does the work
* basically V7 ... how long.... OMG.

So like above, I am slowly decreasing my usage of mikrotik.. This is just natural progression don't keep up with features others are offering.

But - I do love the interface and the tools...
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:02 am

now that we know what command to run
@IPANetEngineer: do you want to compare notes now that we are probably on the same page? Prompted by something MikroTik told me last thing on Friday about the nature of the underlying problem, and following my own research last night, I've got some good news to share — so it would be great to know if we're now independently testing and trying to mitigate the same things.

If you were leveraging the tool to attack subnet 2001:db8:3::/64 using the interface eth2, there is a command you would type. Let's assume you don't need to "sudo", and that the binary is in your $PATH (i.e. you don't need /usr/whatever/ at the beginning). You know the command that you would type (without any other options) to do this attack.

echo "command you would type" | md5sum

The first four characters of your output should match mine - I get "eaba". And if your first four characters match mine, then you now know that we are almost certainly using the same exploits.

If you want to work together on this, so that I know that we are both "fully weaponised", what are the next four characters of the hash? Drop me a private message, DM, email — something a bit more private so that we can start swapping notes.
Marek
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:03 am

I've done my own investigation and I think I can reproduce the problem.
First it is important to note that the target of the attack needs not to be the Mikrotik itself: if it is forwarding traffic from an attack, its memory can exhaust and eventually it will reboot.

So my lab is similar to this: an attacker connected to a quagga router. This quagga router is connected to a chr and then this chr connected to another quagga that is connected to the target network.

When I launch the attack, the chr reboots but the other routers are not affected by the attack.

Firewall rules seems not to be effective.

But if I increase the chr memory from about 300 MiB to 3000 MiB the router seems to be ok: the free memory goes between 2200 and 2400.

As my lab is made in gns3 maybe it can't be the same as IRL with Mikrotik hardware so I plan to make some tests tomorrow with several ccr1009.

But I would ask @maznu if he can check what happens if he increase the ram. In his Twitter video he has about 300 MiB. And I would ask him if he has tested with routerboards or only chr.

It would be great if this thread keeps constructive and anyone who wants to cry or complain start another thread so we can silent it

Enviado desde mi Redmi 3 mediante Tapatalk

 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:18 am

When I launch the attack, the chr reboots but the other routers are not affected by the attack.

Firewall rules seems not to be effective.

But if I increase the chr memory from about 300 MiB to 3000 MiB the router seems to be ok: the free memory goes between 2200 and 2400.

As my lab is made in gns3 maybe it can't be the same as IRL with Mikrotik hardware so I plan to make some tests tomorrow with several ccr1009.

But I would ask @maznu if he can check what happens if he increase the ram. In his Twitter video he has about 300 MiB. And I would ask him if he has tested with routerboards or only chr.
This sounds almost exactly the same as what MikroTik will be fixing on Monday.

I have tested with non-CHR equipment before, with similar results. e.g. I crashed a TILE router with full transit tables loaded into memory (i.e. it already had a significant part of its memory allocated); and crashed all the mmips, mipsbe, and arm equipment I had available around me when I originally reported.

I will assume you are using the same tools I am, so @ jprietove: can you please perform the same thing as in my post immediately above yours? This way we all know we are talking about the same kind of attack.

What would be characters 9, 10, 11, 12 of the md5sum?
Last edited by maznu on Sun Mar 31, 2019 11:21 am, edited 2 times in total.
Marek
 
MichaelHallager
just joined
Posts: 24
Joined: Sun May 13, 2018 8:12 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:32 am

I can report I had (now disabled) IPv6 connectivity and a few days ago my router rebooted for no obvious reason. A couple of days later I was alerted to this issue.

As a consequence, I am now assuming the exploit is out there in the wild and is being used.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:50 am

As a consequence, I am now assuming the exploit is out there in the wild and is being used.
Thanks for this information, @MichaelHallager. I've saw something similar several times in the first two weeks of March this year, and advised MikroTik on 2019-03-15 about this, asking for urgent action.

At this stage I do not think it is part of a deliberate "attack campaign" — possibly it is incidental, collateral damage in some other activity on the Internet. But I cannot help worrying that it is reconnaissance — my worst fear would be a bad actor is looking to find vulnerable targets to shake down in an extortion campaign.

Do you have logs showing you why the router crashed? You can usually see that immediately after you log in.
Marek
 
MichaelHallager
just joined
Posts: 24
Joined: Sun May 13, 2018 8:12 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:57 am

I have been spreading the word around in other forums.

If it's of any interest / help I am happy to act as a remote test case providing no harm is done.
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 12:01 pm

This sounds almost exactly the same as what MikroTik will be fixing on Monday.

What would be characters 9, 10, 11, 12 of the md5sum?
Sorry @maznu but I don't get the same md5sum you expected. Maybe mine is a different but correlated attack
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 12:03 pm

I have been spreading the word around in other forums.

If it's of any interest / help I am happy to act as a remote test case providing no harm is done.
At this stage, my best advice would be that people monitor the memory usage on their routers and graph it. If your memory usage is stable for many weeks, and then increases by several hundred Mb within a few minutes — possibly causing a crash as a result — that is one of the signs I would expect to see during an attack.
Marek
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 12:07 pm

Sorry @maznu but I don't get the same md5sum you expected. Maybe mine is a different but correlated attack
It is possible we are using different tools to trigger the same issue — there is more than one way to make some IPv6 packets. ;-)

If you're happy to discuss in private anyway, please drop me an email? Or DM me on Twitter (I just followed you).
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 6:56 pm

Can anyone verify in what order uRPF and route cache writes are processed? I suspect this is largely a solved problem, this was an issue in the early days of IPv6.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 8:28 pm

I can replicate both issues. This is very, vey easy to execute using simple, opensource pen-testing tools and is pretty effective at making a box reload, or under smaller load stop transiting traffic until the event stops. I do not consider this a security flaw at all. It's a very unfortunate implementation deficiency. I have not yet been able to replicate it with TCP or UDP, but I am not sure it can't be done as it has much less to do with the traffic type and more to do with the handling of particular caching functions. Mikrotik has called this neither a bug or vuln, and they're not exactly wrong, but it's an issue that needs addressed regardless, for sure, given it's extremely simple and low bar to execute.
A known mitigation strategy that won't work:
1. Netflow based detection and null routing where the flow generation is done on the devices under load. They don't export flow data once the event starts, and even if they did the 5m rollover of most flow data tools is far too long. FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write.

Still testing others.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 8:50 pm

FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write.
EDIT: with only:

* a route back to the attacker
* and only a default null route in my victim router's table
* (besides dynamic routes from link local addresses)

my lab looks like it will still OOM.

(the attack is coming from a single IPv6 address)
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 9:10 pm

If I understand correctly, you null route the offending host /128 (or /64) and the exhaustion still occurs, correct? If you null route the attacker IP address on the device that is transiting the traffic, does the OOM still occur? I am assuming that all routers are a mikrotik? I am working on this now.
FastNetMon may work if the netflow is being generated by an intermediate device in the path (like off of a tap), it's very fast and can potentially mitigate assuming null routing is performed before cache write.
EDIT: with only:

* a route back to the attacker
* and only a default null route in my victim router's table
* (besides dynamic routes from link local addresses)

my lab looks like it will still OOM.

(the attack is coming from a single IPv6 address)
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Sun Mar 31, 2019 9:52 pm

Dumb question, have you validated that this is remotely exploitable outside of a contained lab?
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
davidcx
just joined
Posts: 16
Joined: Tue Nov 14, 2017 7:06 pm

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:40 pm

Mikrotik have publicly disclosed the details of the vulnerability, on a Sunday, in a way that a child could exploit it - before even providing a fixed beta, let alone a stable release version, let along giving us time to test and deploy it.

Truly despicable behaviour there Mikrotik. Do you have no respect for your customers at all?

-davidc
 
mstead
Member Candidate
Member Candidate
Posts: 113
Joined: Sat Mar 04, 2006 2:41 am

Re: UKNOF 43 CVE

Sun Mar 31, 2019 11:52 pm

Mikrotik have publicly disclosed the details of the vulnerability, on a Sunday, in a way that a child could exploit it - before even providing a fixed beta, let alone a stable release version, let along giving us time to test and deploy it.

Truly despicable behaviour there Mikrotik. Do you have no respect for your customers at all?

-davidc
Where is this posted? I did a quick search and didn't find anything.
 
bmann
just joined
Posts: 8
Joined: Sat Jan 05, 2013 2:10 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:17 am

I would say that MikroTik has good products for many people. For consumers it is great product.
You pay for a gear and then you can run it several years with updated RouterOS version and without any additional cost.
And in fact the issues are fixed in time. There is not many vendors doing this.

The reaction of MikroTik team is not probably ideal every time. But
How much does the gear cost with comparison with BIG network vendors?
How much do you pay for support per year?

I think that MikroTik helped to build a business for many ISPs and for many people. You know how they work, the benefits and limitations.
If your business is to big and MikroTik does not fit well anymore then you may choose another vendor with more powerful devices, support services
and of course with some costs for it.
 
MichaelHallager
just joined
Posts: 24
Joined: Sun May 13, 2018 8:12 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:27 am

@bmann has made some very good points which I can relate to.

I come from the Cisco camp and I was amazed when I bought my RB1100AHx4 what I was getting for the money... and it's made in Latvia, not China!

Personally, I think Mikrotik products are possibly a bit too cheap and I would be happy to pay a few extra bucks if it meant they were properly resourced so we don't have these types of issues.
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:30 am

Personally, I think Mikrotik products are possibly a bit too cheap and I would be happy to pay a few extra bucks if it meant they were properly resourced so we don't have these types of issues.
The reason, that here isn't such an option or a service contract is, that they'd have to take responsibility then and also implement features that customers require, instead of choosing themselves what they see fit to fix and implement.

I would have no problem paying a service contract, IF that would give me what I needed.

I have a good mix of Cisco, Mikrotik, Ceragon, Radwin and other gear. Cisco is just too expensive for some stuff and Mikrotik doesn't take things serious enough with other things.

/M
Last edited by marlow on Mon Apr 01, 2019 12:31 am, edited 1 time in total.
Communication is the beginning of understanding
-- AT&T
 
User avatar
davidcx
just joined
Posts: 16
Joined: Tue Nov 14, 2017 7:06 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:30 am

Where is this posted? I did a quick search and didn't find anything.
viewtopic.php?p=724264#p724238
 
proximus
Member Candidate
Member Candidate
Posts: 112
Joined: Tue Oct 04, 2011 1:46 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:49 am

Dumb question, have you validated that this is remotely exploitable outside of a contained lab?
.
That is certainly a critical question that needs to be answered and understood. The extraordinary amount of "pre-show publicity" has lead many to form strong opinions and responses before the full facts are understood.
 
msatter
Forum Guru
Forum Guru
Posts: 1277
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:52 am

@bmann has made some very good points which I can relate to.

I come from the Cisco camp and I was amazed when I bought my RB1100AHx4 what I was getting for the money... and it's made in Latvia, not China!

Personally, I think Mikrotik products are possibly a bit too cheap and I would be happy to pay a few extra bucks if it meant they were properly resourced so we don't have these types of issues.
I think assembled is a more fitting description.

https://www.made-in-china.com/manufactu ... rotik.html
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 6:35 am

I have come pretty close to being able to exhaust the route cache "in the wild" (a controlled real network built for this purpose), meaning on gear attached to a public network. I am sure I can do it, but I want to know who else tried this. There is an old thread that implies some of this was triggered by "real traffic", and I have definitely seen an uptick in ipv6 scanning that impacts major vendor router platforms to the point that they drop traffic and spike CPUs. This feels pretty darned similar, and not the catastrophic event horizon it's being played out to be. It's still bad, it stinks, and it's definitely been handled in a way that is less than optimal. I realize I am being a tad pedantic, but it's important to call a duck a duck in order to handle it in the correct manner. This is not a security vulnerability as I would describe it. There is no remote or local execution of code, no raised privileges, no mechanism of compromise. I don't think this is a software bug, either. It's close, but in reality what this is is a poorly implemented protocol that had what I can only believe was a poverty of testing before release and as such allows for a remotely triggered denial of service.
Denial of service isn't necessarily malicious, it just means exactly what it says: service is denied. As mentioned earlier in the thread, this is like the old ping of death issue: a poorly executed stack.

I'd like to know who has executed this in the wild and how far back this goes in their code train. I don't need to know how - I already have that sorted - I just want to know who and when. I'll bet dollars to donuts this has always been there. I've seen behavior like this before and I have seen denial of services a lot like this before (think exhausting the state table of a stateful packet filtering device). Easy pickings, low hanging fruit.
Dumb question, have you validated that this is remotely exploitable outside of a contained lab?
.
That is certainly a critical question that needs to be answered and understood. The extraordinary amount of "pre-show publicity" has lead many to form strong opinions and responses before the full facts are understood.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
xrobau
just joined
Posts: 5
Joined: Wed Jan 03, 2018 4:18 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 7:39 am

> This is not a security vulnerability as I would describe it.

Then what EXACTLY would you call the ability to stop your router from routing, remotely, without authentication? Because we in the Infosec world call that an Unauthenticated Denial of Service. (Which is a bad thing)

Look, MikroTik have dropped the ball on this. That they haven't gone 'Shit, yes, we made a mistake, we're sorry, and we're making (these changes) to make sure it doesn't happen again' is MORE telling.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Mon Apr 01, 2019 9:13 am

People above you in this thread are saying that there should be separate terminology between denial of service (CPU high, out of memory, crash etc) and something that allows attacker to gain access to devices, steal credentials, install malware and read private data. You are shoving them both under one term. I'm not expert, but at least that is what I mean when I said Denial of Service and Vulnerability are different things.
No answer to your question? How to write posts
 
msatter
Forum Guru
Forum Guru
Posts: 1277
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:15 am

The beta released today, addresses IPv6 route cache using more memory than available.

MAJOR CHANGES IN v6.45:
----------------------
!) ipv6 - fixed soft lockup when forwarding IPv6 packets;
!) ipv6 - fixed soft lockup when processing large IPv6 Neighbor table;
----------------------
Changes in this release:
*) ipsec - properly drop already established tunnel when address change detected;
*) ipv6 - adjust IPv6 route cache max size based on total RAM memory;
*) smb - fixed possible buffer overflow;
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:22 am

I have just tested this beta and I confirm that with 300 Mb RAM the router's memory doesn't fill. A CHR with 300 Mb of RAM with OSPF-v3 has 237 Mb of free-memory and during the attack it keeps on around 200 Mb.

Hopefully this fix will be in long-term and current branches soon.
 
p1p1p1
just joined
Posts: 22
Joined: Sat Feb 23, 2013 1:53 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:46 am

hmm testing beta in my sandbox routers, waiting for LONG TERM ...
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:19 pm

It has to filter to 6.43 long term first.

Too many problems in 6.44.

/M


Communication is the beginning of understanding
-- AT&T
 
cantanko
newbie
Posts: 28
Joined: Mon Apr 05, 2010 12:53 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 12:20 pm

People above you in this thread are saying that there should be separate terminology between denial of service (CPU high, out of memory, crash etc) and something that allows attacker to gain access to devices, steal credentials, install malware and read private data. You are shoving them both under one term. I'm not expert, but at least that is what I mean when I said Denial of Service and Vulnerability are different things.

I prefix this with the fact that I am a complete rank-amateur compared to some here, although I have to concur with Normis - the neighbour cache exhaustion issue is a denial-of-service, plain and simple. What is irksome is that it's a device-level, not protocol-level, DoS. If you're dual-stack on the same device, when you get scanned both of your stacks go away for the amount of time it takes for the router to reboot.

I also consider the fact that this issue has been knocking around for at least the last six years is what has many people frustrated (a nice discussion of it from March 2013 here).

Admittedly, I never bothered checking it either, but you kinda hope you don't have to...
 
enzain
just joined
Posts: 20
Joined: Wed Jan 17, 2018 9:15 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 2:52 pm

This may be the final straw. Time to start researching other products.
You always can use cisco:)
 
CTSsean
just joined
Posts: 23
Joined: Fri Sep 15, 2017 12:56 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:16 pm

We have near to 10k Mikrotik devices in our network, if every one of them needs to be updated. This is not something you do in a few hours .. or days.

...

And the issue is, that it is not the first time, that Mikrotik has handled an issue this way. It is every single time. They only react, when issues become public knowledge.

/M
Um, there are more than a few ways to automate the roll out of a config change or RouterOS package. Unimus or Ansible make this easy. Example PDF: https://mum.mikrotik.com/presentations/ ... 842532.pdf


As for how Mikrotik handled the issue... I suspect the blame is on both sides where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE. The issue is at kernal level and trying to re-write kernal code is not fun. Mikrotik has handled things better than most, but I think this RouterOS v6 is just becoming too long in tooth at this point. I'm a Mikrotik fanatic and everything lately is about fixing RouterOS6 when it needs be left behind and a newer kernal needs to be rolled up, even if they have to abandon Tile for RouterOS7.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:20 pm

It's true, MikroTik should have handled it better. It was mistakenly filed as just another ipv6 bug, which there are more of, until the author complained about lack of action. Then we looked at it again, and started to look for alternative solutions to handle it (because kernel change is not possible in v6). But in the end, the issue is fixed finally.

P.S.: we don't rely on kernel for TILE support, so new kernel doesn't affect it.
No answer to your question? How to write posts
 
BRMateus2
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Thu Oct 26, 2017 11:18 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:46 pm

Linux 5 is there, an stable kernel with tons of fixes from very dangerous bugs of 4.14-4.16 (BFQ with SCSI kernel locks (can corrupt the disk), Ryzen bugs ie suspend locks).
This is an DDoS kind, not something an dedicated blackhole subnet or country couldn't handle.

Good for Mikrotik and Normis telling details, please continue that attitude from now on, would be less of a rage earlier, good handling anyway.
 
enzain
just joined
Posts: 20
Joined: Wed Jan 17, 2018 9:15 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 4:48 pm

We have nine CCR1072 that can't receive full tables because they literally froze processing them.
Sorry, what?

How many tables on 1072?
I use 1036 - and this work with 2 FV tables
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:10 pm

We have nine CCR1072 that can't receive full tables because they literally froze processing them.
Sorry, what?

How many tables on 1072?
I use 1036 - and this work with 2 FV tables
Don't do full tables on CCRs. They are terrible at it.
-----
Mike Hammett

The Brothers WISP
 
wtm
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Tue May 24, 2011 5:27 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:20 pm

So what firmware do we need to install on what routers to prevent this ?
 
enzain
just joined
Posts: 20
Joined: Wed Jan 17, 2018 9:15 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:22 pm

Don't do full tables on CCRs. They are terrible at it.
Why?
Is work fine, i receive FV and default route (while router compute FV's - used default route, when session up, internet work after 2 sec)
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:31 pm

where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE
The full timeline will be available next week. But when I reported this in April 2018, my request to MikroTik was to plead with support to treat this as a serious security vulnerability, and asked that they themselves allocate a CVE to it. After no action for six months, I then raised CVEs myself and communicated this to MikroTik to try and pressure them into taking another look at this.
Marek
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:34 pm

Don't do full tables on CCRs. They are terrible at it.
Why?
Is work fine, i receive FV and default route (while router compute FV's - used default route, when session up, internet work after 2 sec)
Default routes mean you can't use uRPF.

Such high CPU usage during table churn means you'll be at best sub-optimally routing packets and at worst, sending them down blackholes for 10 - 20 minutes at a time.
-----
Mike Hammett

The Brothers WISP
 
cantanko
newbie
Posts: 28
Joined: Mon Apr 05, 2010 12:53 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 7:16 pm

So what firmware do we need to install on what routers to prevent this ?

As of right now, 6.45beta23 AFAIK.
 
mstead
Member Candidate
Member Candidate
Posts: 113
Joined: Sat Mar 04, 2006 2:41 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 7:45 pm

where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE
The full timeline will be available next week. But when I reported this in April 2018, my request to MikroTik was to plead with support to treat this as a serious security vulnerability, and asked that they themselves allocate a CVE to it. After no action for six months, I then raised CVEs myself and communicated this to MikroTik to try and pressure them into taking another look at this.
maznu - can you contact me via Twitter? I sent you a tweet already.
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 8:11 pm

I understand how things work in "the infosec world". I have stated above how I describe this and I stand by my comments. Unauthenticated Denial of Service is just as I described it, and it is not exclusively in the domain of security vulnerability. It can definitely be leveraged that way, but the same way as a car can be used to run over a person, we don't call it a weapon. No one is debating or defending anything, I am indifferent on pretty much all hardware and software. This is really just semantics here, and we can all agree that this is a crummy situation that no one loves, but right or wrong, blame solves precisely zero problems.
If you read my post you'll see that my statements are pretty clear. Surprisingly, I have seen more admittance of a mistake out of a vendor than I have seen in quite some time.
> This is not a security vulnerability as I would describe it.

Then what EXACTLY would you call the ability to stop your router from routing, remotely, without authentication? Because we in the Infosec world call that an Unauthenticated Denial of Service. (Which is a bad thing)

Look, MikroTik have dropped the ball on this. That they haven't gone 'Shit, yes, we made a mistake, we're sorry, and we're making (these changes) to make sure it doesn't happen again' is MORE telling.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 8:13 pm

I tested null routing with ip-filter enabled and it still drives the cache up. What I don't know is if
/ip rp-filter
also covers IPv6. If it doesn't then there appears to be no way to enable ipv6 RPF checking.
Don't do full tables on CCRs. They are terrible at it.
Why?
Is work fine, i receive FV and default route (while router compute FV's - used default route, when session up, internet work after 2 sec)
Default routes mean you can't use uRPF.

Such high CPU usage during table churn means you'll be at best sub-optimally routing packets and at worst, sending them down blackholes for 10 - 20 minutes at a time.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 8:17 pm

maznu - can you contact me via Twitter? I sent you a tweet already.
My timeline exploded a bit, as you might imagine. I'm @maznu on Twitter, DMs are open :)
Marek
 
faraujo88
just joined
Posts: 13
Joined: Fri Feb 15, 2019 2:28 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 9:03 pm

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
Something about this question?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8317
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:05 pm

Something about this question?
Was answered on the first page.
viewtopic.php?p=723912#p723912
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
slimmerwifi
just joined
Posts: 13
Joined: Tue Aug 01, 2017 6:05 pm
Location: Netherlands

Re: UKNOF 43 CVE

Tue Apr 02, 2019 12:19 am

It's true, MikroTik should have handled it better. It was mistakenly filed as just another ipv6 bug, which there are more of, until the author complained about lack of action. Then we looked at it again, and started to look for alternative solutions to handle it (because kernel change is not possible in v6). But in the end, the issue is fixed finally.

P.S.: we don't rely on kernel for TILE support, so new kernel doesn't affect it.
Appreciate this!

Also, send @maznu a present/gift/bounty/4011. He sure as hell earned it.
We manage 50+ corporate wifi networks in the Netherlands using Mikrotik & Cloudcore equipment.
 
cmurrayis
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri May 15, 2009 4:31 am

Re: UKNOF 43 CVE

Tue Apr 02, 2019 12:31 am

@normis

Is it possible to get an honest responce about ROS v7's release timetable? We and many other people have been struggling with BGP performance so much so we've had to reject larger potential clients because we cannot offer MPLS due to large packet loss with routing convergence on the CCR1072.

We are not a high volume network and cannot hold a single full table from any of our Transit providers because of CPU load and extended outages if for some reason one of the sessions drop. We were holding out until MUM Europe as we were told by support there would be an announcement there about it but alas there wasn't one.

We now have to seriously look at switching vendors away from Mikrotik which we've used for 15+ years because we just cannot wait any more with such a loose open ended timeline that everything appears to be "Fixed in ROS v7" and there being no release deadline.

Even if you were to release it today it would be 6 months before we could roll it out with enough confidence and testing in LAB given the extended development timeline.

I hope you can take the time to responce publicly or even privately as I'm under alot of pressure to grow our network but with current hardware not able to without potentially impacting client performance and reliability.

We'd also be happy to pay a few thousand dollars a year for advanced support/updates if required to offset the cheap licensing charges.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 2:03 am

Also, send @maznu a present/gift/bounty/4011. He sure as hell earned it.
That's very kind, but after we've all got the patch in longterm and stable, I want to know how I can mail order a crate of beer to MikroTik's offices to say thank you for getting this fixed.
Marek
 
VipITBE
just joined
Posts: 23
Joined: Tue Apr 02, 2013 10:40 am

Re: UKNOF 43 CVE

Tue Apr 02, 2019 8:49 am

@normis

Is it possible to get an honest responce about ROS v7's release timetable? We and many other people have been struggling with BGP performance so much so we've had to reject larger potential clients because we cannot offer MPLS due to large packet loss with routing convergence on the CCR1072.

We are not a high volume network and cannot hold a single full table from any of our Transit providers because of CPU load and extended outages if for some reason one of the sessions drop. We were holding out until MUM Europe as we were told by support there would be an announcement there about it but alas there wasn't one.

We now have to seriously look at switching vendors away from Mikrotik which we've used for 15+ years because we just cannot wait any more with such a loose open ended timeline that everything appears to be "Fixed in ROS v7" and there being no release deadline.

Even if you were to release it today it would be 6 months before we could roll it out with enough confidence and testing in LAB given the extended development timeline.

I hope you can take the time to responce publicly or even privately as I'm under alot of pressure to grow our network but with current hardware not able to without potentially impacting client performance and reliability.

We'd also be happy to pay a few thousand dollars a year for advanced support/updates if required to offset the cheap licensing charges.
If you have performance issues with the CCR1072, then why don't you get a real router? Real hardware and pay the price instead of banging on Mikrotik for not releasing ROSv7?
Also, there are plenty of WISPs using MPLS and routing on mikrotik without issues...
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Tue Apr 02, 2019 8:55 am

Is it possible to get an honest responce about ROS v7's release timetable?
Maybe other topic ?
No answer to your question? How to write posts
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2291
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: UKNOF 43 CVE

Tue Apr 02, 2019 10:16 am

@maznu - beta23 fixes both vulnerability? Did you test?
LAN, FTTx, Wireless. ISP operator
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 10:42 am

@maznu - beta23 fixes both vulnerability? Did you test?
I emailed MikroTik yesterday, tweeted, and posted about this on the 6.45beta thread - yes!

MikroTik has said that another beta is expected to make the settings on the affected components more "optimal" for devices with low RAM. I hope it lands soon.
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 5:17 pm

More testing has yielded more data. This has not been properly replicated by anyone else that I know of, so take it as plausible hypothesis. I think I found more fallout from the ipv6 flaw: boxes that have their ND cache or their ipv6 route cache run up but not to the point of OOM reload experience a degraded performance over time in fun and unique ways after a scan attempt or resource exhaustion. I have seen two symptoms, all resolved by a reboot.
SNMP responds with wildly inaccurate data, or no data at all, but the snmp daemon continues to run
Both IPv4 and IPv6 packet forwarding experiences odd blips in function that don't seem to correspond to anything I can find (again, no snmp data to validate against)

Again, this is unverified but I suspect very likely. It'd be nice to see someone else replicate this or disprove it.

nb
@maznu - beta23 fixes both vulnerability? Did you test?
I emailed MikroTik yesterday, tweeted, and posted about this on the 6.45beta thread - yes!

MikroTik has said that another beta is expected to make the settings on the affected components more "optimal" for devices with low RAM. I hope it lands soon.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 5:26 pm

More testing has yielded more data. This has not been properly replicated by anyone else that I know of, so take it as plausible hypothesis. I think I found more fallout from the ipv6 flaw: boxes that have their ND cache or their ipv6 route cache run up but not to the point of OOM reload experience a degraded performance over time in fun and unique ways after a scan attempt or resource exhaustion. I have seen two symptoms, all resolved by a reboot.
This gels with something I found when I was doing some testing last night. I suspect the problem is that the various processes in RouterOS struggle to malloc() memory for various tasks, and it makes for a very painful experience. My symptoms included: typing on the console being very laggy, issuing commands was also quite slow, and a reboot fixed it. In extreme memory exhaustion (but again, not quite OOM), the router seems to become unresponsive for a short while (won't answer telnet connections, etc), but sort of seems to be answering ping and forwarding packets… kind of. I hypothesise that this is what MikroTik has been referring to as a "soft lockup" in their 6.45beta22 changelog entries. It "felt" very similar to the feeling of interacting with a Linux server under extreme load, swapping or otherwise thrashing.
Marek
 
cantanko
newbie
Posts: 28
Joined: Mon Apr 05, 2010 12:53 am

Re: UKNOF 43 CVE

Tue Apr 02, 2019 7:13 pm

This gels with something I found when I was doing some testing last night. I suspect the problem is that the various processes in RouterOS struggle to malloc() memory for various tasks, and it makes for a very painful experience.

That description of malloc() after high memory pressure agrees with my experience with completely unrelated-but-linux-based software that was continually malloc(), realloc() and free()ing memory all over the place, causing heavy page-table fragmentation and thus real contention issues for other processes trying to malloc() memory. The enterprise distros I was working with at the time kinda sorta try to deal with it using page table defragmentation / consolidation, but that too has its own associated costs. With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available.

I don't envy the Mikrotik devs on this one if it is something along those lines...
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 7:43 pm

With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available.
The data structure that the Linux kernel used in RouterOS v6 uses for the IPv6 route table (and also route cache) is a sort of radix tree. It's a very different different from how the data structures for IPv4 work (because the IPv6 routing table is very sparse by comparison), and it's different again to how the more modern kernels do this too. I suspect the kernel is allocating the memory for this in fairly large slabs, but it would not at all surprise me if there is a possibility of fragmentation.

All that aside, it is possible to consume a nice big chunk of contiguous memory in RouterOS's user-space — and that ought to have very similar effects. That was how I got a bunch of CHRs behaving in a way that I believe matches what buraglio is describing above. A fairly easy way to try this out might be to inject a bunch of unreachable BGP routes to make the "route" process use more memory. As you get down to a few Mbytes of free RAM, things get icky… but this is completely expected behaviour (even if it would be unfortunate).
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 8:12 pm

Dang, I was hoping I was wrong. Looking like probably not.

nb
With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available.
The data structure that the Linux kernel used in RouterOS v6 uses for the IPv6 route table (and also route cache) is a sort of radix tree. It's a very different different from how the data structures for IPv4 work (because the IPv6 routing table is very sparse by comparison), and it's different again to how the more modern kernels do this too. I suspect the kernel is allocating the memory for this in fairly large slabs, but it would not at all surprise me if there is a possibility of fragmentation.

All that aside, it is possible to consume a nice big chunk of contiguous memory in RouterOS's user-space — and that ought to have very similar effects. That was how I got a bunch of CHRs behaving in a way that I believe matches what buraglio is describing above. A fairly easy way to try this out might be to inject a bunch of unreachable BGP routes to make the "route" process use more memory. As you get down to a few Mbytes of free RAM, things get icky… but this is completely expected behaviour (even if it would be unfortunate).
ForwardingPlane, LLC
https://www.forwardingplane.net
 
BRMateus2
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Thu Oct 26, 2017 11:18 pm

Re: UKNOF 43 CVE

Tue Apr 02, 2019 11:44 pm

Memory fragmentation undefined behaviour bugs happens even in Windows (do a 500MB free test without swap, Windows craps out).
It is very hard to handle such malloc issues, because it is part of malloc itself - any kind of new route or such, will bug out the dynamic buffers, etc.
 
User avatar
Jesse6
just joined
Posts: 6
Joined: Fri Nov 17, 2017 6:35 pm
Location: Brazil

Re: UKNOF 43 CVE

Wed Apr 03, 2019 4:48 pm

It is not new that Mikrotik cares very very little about IPv6 deployment in their products, features such as, 'remote-dhcpv6-pd-pool' for radius, VRF support for IPv6 (or, something I believe it's even better, source based routing), Dude (that is a very excellent product, but don't have IPv6 support) with IPv6 monitoring capability, etc., are in discuss for almost a decade, with no signs of foreseeable implementation. Since at now we are almost 5 years that IPv6 is the new internet standard, I really hope you guys put the IPv6 at the top of the list in your development. And yes, I know that a vulnerability is something that happens to EVERY human developed device/firmware, it is understandable, and I think you guys are working hard to fix that. But this is a good moment to highlight the weakest Mikrotik's point for the upcoming changes: IPv6.
I wish you guys good luck with that fix, and hope you see IPv6 more seriously.
 
User avatar
KeepBurning
just joined
Posts: 1
Joined: Mon Apr 01, 2019 7:51 pm
Location: Brazil

Re: UKNOF 43 CVE

Wed Apr 03, 2019 9:08 pm

So can I assume that version 6.45beta23 is safe to use IPv6?
MAJOR CHANGES IN v6.45:
----------------------
!) ipv6 - fixed soft lockup when forwarding IPv6 packets;
!) ipv6 - fixed soft lockup when processing large IPv6 Neighbor table;
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2291
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: UKNOF 43 CVE

Thu Apr 04, 2019 11:10 am

Fixel also in long-term - 6.43.14
and Current - 6.44.2
LAN, FTTx, Wireless. ISP operator
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 12:18 pm

Fixel also in long-term - 6.43.14
and Current - 6.44.2
Attacked both, and both releases fix CVE-2018-19299. Fantastic news — but now the hard work for all us network operators begins:

1. 🔬 test

2. 🧠 plan

3. 🔨 deploy

4. 🔍 monitor

5. 🍺🍻🎉

6. 😴 🛌
Marek
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Thu Apr 04, 2019 12:29 pm

Just wondering what will happen / be the effect when "under attack" and hitting memory limit?
* on neighbour mem limit
* on routing cache limit

Router will survive, but what with the legit connections?
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 12:44 pm

Just wondering what will happen / be the effect when "under attack" and hitting memory limit?
* on neighbour mem limit
* on routing cache limit
Router will survive, but what with the legit connections?
The tests I did on 6.45beta23 suggested different levels of memory usage would be used for the IPv6 routing table/cache depending on the installed RAM in the router. For example, routers with 2Gb of memory would permit a few hundred thousand entries in the IPv6 routing table/cache.

The challenge on Linux 3.3.5 (and many kernel version after) is that the IPv6 routing cache is stored in the same data structure (and therefore memory) as the routing table, which means that an attack can still fill that allocated memory. Further research is required to find out whether this would then deny real routes being inserted (e.g. learned from upstream providers via BGP), or whether a "least recently used" type of algorithm will allow a route insertion to replace a temporary cache entry.
Marek
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Thu Apr 04, 2019 2:21 pm

What I don't understand: why is it not possible to firewall against it.
When you limit the addresses that are routed, e.g. by dropping traffic in the raw prerouting table, does it still
create entries for the dropped traffic in the route cache or neighbor table? Why?
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24258
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Thu Apr 04, 2019 2:23 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
No answer to your question? How to write posts
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 2:31 pm

What I don't understand: why is it not possible to firewall against it.
When you limit the addresses that are routed, e.g. by dropping traffic in the raw prerouting table, does it still
create entries for the dropped traffic in the route cache or neighbor table? Why?
If you DROP in PREROUTING then it is safe against CVE-2018-19299.

Dropping in chain=forward is not safe, because a routing table lookup has to be done before the firewall rule can be evaluated (otherwise the firewall rule could not know which the in- and out-interfaces are, etc). This means a cache entry will have already been created, even if the chain=forward rule does action=drop.
Marek
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Thu Apr 04, 2019 3:13 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
Of course it would still be possible to exploit it from the inside, but frankly I always worry more about exploiting from outside than from inside.
For some time I have a dynamic address list in IPv6 that contains all internal addresses that have attempted to make outgoing traffic (plus some static servers),
and drops all incoming traffic towards addresses not in that list. This drop is now in the forward chain, I will move it to the raw prerouting chain.

Of course this countermeasure generates a new attack surface, where local users are able to fill that address list with 2^64 entries, but as I wrote
I am not so worried that this will happen.
 
anav
Forum Guru
Forum Guru
Posts: 3091
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: UKNOF 43 CVE

Thu Apr 04, 2019 4:02 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
Congrats to your Team Normis, under what must be an incredibly stressful pressure cooker of a year. Proost, gun bei, sit, cin cin, Na zdrowie and Priekā!
phuckin A batman!
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:11 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5940
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:14 pm

Completely unrelated to the topic.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:14 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
Marek
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5940
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:21 pm

It is an upgrade problem because of no free space on the router, not related to this thread at all.
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 6:13 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.

I fear the only way to get this onto a hAP Lite is to Netinstall.

If I have to send people to clients' homes to netinstall their routerboards, I'd rather just go and buy TPLink 840s and send the techs off to install them. Less hassle for everyone.

As far as I am concerned I do not have an update to install on a hAP and the matter is far from over.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 6:14 pm

It is an upgrade problem because of no free space on the router, not related to this thread at all.
I have 6.43.14 installed on a hAP ac lite (64Mb RAM), and it is still vulnerable. Ticket#2019040222005195 and Ticket#2019032922005182
Marek
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 993
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 7:48 pm

It is an upgrade problem because of no free space on the router, not related to this thread at all.
Question - if there is a problem with not enough free space (on some mikrotiks) …
On a remote wireless client , is it possible to do something like this …
- downgrade the remote client to a smaller ROS (or even a stripped down tiny ROS) that keeps the remote client connected
- then upgrade the remote client to 6.44.2 (or what ever newer version works)
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 8:23 pm

I have done several tests with GNS3 using CHR 6.44.2 (stable) and as long as the router has enough memory, it doesn't crash. In my tests, the attack 'steals' around 180 MiB.

Using a CHR with 256 MB, system resources shows a total memory of 224 MiB and free-memory of 197 MiB before attack. During the attack, only from one computer, the free memory decreases to around 20 MiB and sometimes to 13 MiB. Using two attackers, it seems the results are the same and not worst.

With 200 MB the router reboots because OOM.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Thu Apr 04, 2019 9:15 pm

@pe1chl, question: in your setup externally initiated ipv6 traffic is disallowed right?
 
neutronlaser
Member Candidate
Member Candidate
Posts: 212
Joined: Thu Jan 18, 2018 5:18 pm

Re: UKNOF 43 CVE

Thu Apr 04, 2019 11:17 pm

ipv6 dumb extravagance anyway
 
highonsnow
newbie
Posts: 34
Joined: Fri Oct 19, 2007 3:30 am

Re: UKNOF 43 CVE

Thu Apr 04, 2019 11:35 pm

ipv6 dumb extravagance anyway
Welcome to 2019.. you must have been asleep since DARPA were experimenting with this TCP/IP thing.. that's ok though, we'll help you through it.

It all started with RIPE and global commerce..
 
anav
Forum Guru
Forum Guru
Posts: 3091
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:50 am

ipv6 dumb extravagance anyway
Welcome to 2019.. you must have been asleep since DARPA were experimenting with this TCP/IP thing.. that's ok though, we'll help you through it.

It all started with RIPE and global commerce..
Thank god for that, otherwise I wouldn't had the pure joy of playing doom with a brit and a yank all being served up by one of our pcs all over a 33.6k modem connection for about 10min before crashing LOL. Really rocked with the 56k modem though. ;-)
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 11:56 am

@pe1chl, question: in your setup externally initiated ipv6 traffic is disallowed right?
Yes, externally initiated IPv6 traffic to random addresses is disallowed. I added this when NDP exhaustion attacks were discussed.
Due to the address list, only systems that have initiated outbound traffic (within the last 8 hours) plus a number of addresses of
servers put in the address list as static entries are allowed inbound.

But I had this rule in the /ipv6 firewall filter chain=forward list which should be fine to prevent NDP exhaustion attacks but apparently is not enough
for the route cache table overflow attack, so I moved it to /ipv6 firewall raw chain=prerouting list.
Of course there also is a rule in /ipv6 firewall filter chain=forward that adds the src address to the list (with 8 hour timeout) for new outbound traffic.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Apr 05, 2019 12:48 pm

Thx for info, I guess you do it (the address addition to the list) on connection level, not for each packet?
Last edited by sebastia on Fri Apr 05, 2019 2:28 pm, edited 1 time in total.
 
nostromog
Member Candidate
Member Candidate
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 1:32 pm

@pe1chl, question: in your setup externally initiated ipv6 traffic is disallowed right?
Yes, externally initiated IPv6 traffic to random addresses is disallowed. I added this when NDP exhaustion attacks were discussed.
Due to the address list, only systems that have initiated outbound traffic (within the last 8 hours) plus a number of addresses of
servers put in the address list as static entries are allowed inbound.

But I had this rule in the /ipv6 firewall filter chain=forward list which should be fine to prevent NDP exhaustion attacks but apparently is not enough
for the route cache table overflow attack, so I moved it to /ipv6 firewall raw chain=prerouting list.
Of course there also is a rule in /ipv6 firewall filter chain=forward that adds the src address to the list (with 8 hour timeout) for new outbound traffic.
It looks like a very interesting idea.
Thx for info, I guess you do it on connection level, not for each packet?
If I understood correctly pe1chl, the capturing of outward addresses is done at the connection=new level, but the dropping of packets has to be done before connection tracking to avoid the routing machinery to engage, so the dropping has to happen at the packet level at raw...
Last edited by nostromog on Fri Apr 05, 2019 2:09 pm, edited 1 time in total.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 1:35 pm

I have done several tests with GNS3 using CHR 6.44.2 (stable) and as long as the router has enough memory, it doesn't crash. In my tests, the attack 'steals' around 180 MiB.

Using a CHR with 256 MB, system resources shows a total memory of 224 MiB and free-memory of 197 MiB before attack. During the attack, only from one computer, the free memory decreases to around 20 MiB and sometimes to 13 MiB. Using two attackers, it seems the results are the same and not worst.

With 200 MB the router reboots because OOM.
I had a response from MikroTik earlier saying: "Next beta will have further improvements."

Fingers crossed, everyone…
Marek
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Fri Apr 05, 2019 2:09 pm


This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.
Eben: you are aware that you can pull the "all_packages.zip" file, only upload the modules you need and upgrade ?

Installing everything is not always an advantage and this is not the first time in the lifetime of RouterOS, that this has been an issue (example RB112 and 113c for ROS3 and upwards)

/M


Communication is the beginning of understanding
-- AT&T
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 2:31 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.
Eben: you are aware that you can pull the "all_packages.zip" file, only upload the modules you need and upgrade ?

Installing everything is not always an advantage and this is not the first time in the lifetime of RouterOS, that this has been an issue (example RB112 and 113c for ROS3 and upwards)

/M
We can do package by package, not on a couple of thousand routers in three days.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 2:33 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.
Eben: you are aware that you can pull the "all_packages.zip" file, only upload the modules you need and upgrade ?

Installing everything is not always an advantage and this is not the first time in the lifetime of RouterOS, that this has been an issue (example RB112 and 113c for ROS3 and upwards)

/M
We can do package by package, not on a couple of thousand routers in three days.
https://unimus.net/blog/network-wide-mi ... grade.html
-----
Mike Hammett

The Brothers WISP
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:25 pm

Can it do package by package, or just platform by platform?
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:29 pm

Can it do package by package, or just platform by platform?
On the master ROS install, just have only the packages you do want.
-----
Mike Hammett

The Brothers WISP
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:54 pm

If I understood correctly pe1chl, the capturing of outward addresses is done at the connection=new level, but the dropping of packets has to be done before connection tracking to avoid the routing machinery to engage, so the dropping has to happen at the packet level at raw...
Correct!
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Apr 05, 2019 5:17 pm

In such form, it does introduce another potential issue: connections lasting longer than "timeout" can be impacted as their packets will get dropped for IPv6 implementing privacy features, which change ipv6 after a while.
Something to keep in mind.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 9:10 pm

Well to be more precise I add the entry for outgoing traffic AND src address not in list.
So an existing connection will survive when at least it has some outgoing traffic.
Besides, the type of usage we have usually don't notice an interrupted connection that has been sitting idle.
(mostly phones, tablets and laptops used for websurfing)
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Fri Apr 05, 2019 10:51 pm

We can do package by package, not on a couple of thousand routers in three days.

I don't know, how you get the idea of package by package. You upload the needed packages. The bare minimum. Reboot. It installs all the npk files in one go.

It's no different than uploading the combined package. Just multiple files instead.

/M
Communication is the beginning of understanding
-- AT&T
 
tdw
Member Candidate
Member Candidate
Posts: 196
Joined: Sat May 05, 2018 11:55 am

Re: UKNOF 43 CVE

Sat Apr 06, 2019 12:37 am

Yes, externally initiated IPv6 traffic to random addresses is disallowed. I added this when NDP exhaustion attacks were discussed.
Due to the address list, only systems that have initiated outbound traffic (within the last 8 hours) plus a number of addresses of
servers put in the address list as static entries are allowed inbound.

But I had this rule in the /ipv6 firewall filter chain=forward list which should be fine to prevent NDP exhaustion attacks but apparently is not enough
for the route cache table overflow attack, so I moved it to /ipv6 firewall raw chain=prerouting list.
Of course there also is a rule in /ipv6 firewall filter chain=forward that adds the src address to the list (with 8 hour timeout) for new outbound traffic.
@pe1chl - having moved your rule from chain=forward to chain=prerouting do you add your WAN address to the list too (as chain=prerouting will affect input as well as forward traffic), or does your router not provide any external services?
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Sat Apr 06, 2019 1:27 am

@pe1chl - having moved your rule from chain=forward to chain=prerouting do you add your WAN address to the list too (as chain=prerouting will affect input as well as forward traffic), or does your router not provide any external services?
We have no external IPv6 service on the router but indeed if you have, you should add it as a static entry.
(we do have that for a web- and a mailserver)

Edit: actually the mechanism is a little more complicated than I described which makes listing the router's own address unnecessary in my case, but if implemented
as I described it certainly is required to add the router's address as used on the link to the ISP.
Last edited by pe1chl on Sat Apr 06, 2019 12:28 pm, edited 1 time in total.
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Sat Apr 06, 2019 8:49 am

We can do package by package, not on a couple of thousand routers in three days.

I don't know, how you get the idea of package by package. You upload the needed packages. The bare minimum. Reboot. It installs all the npk files in one go.

It's no different than uploading the combined package. Just multiple files instead.

/M
We tested this on three routers during the night. It works - just, but there's no way we'll be able to finish within the time constraints.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Sat Apr 06, 2019 8:55 am

We can do package by package, not on a couple of thousand routers in three days.

I don't know, how you get the idea of package by package. You upload the needed packages. The bare minimum. Reboot. It installs all the npk files in one go.

It's no different than uploading the combined package. Just multiple files instead.

/M
We tested this on three routers during the night. It works - just, but there's no way we'll be able to finish within the time constraints.
How do you normally update your routers?
-----
Mike Hammett

The Brothers WISP
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Sat Apr 06, 2019 9:03 am

How do you normally update your routers?
We have a set of scripts. All non smips routers are done and dusted.
 
schadom
Member Candidate
Member Candidate
Posts: 139
Joined: Sun Jun 25, 2017 2:47 am
Location: Austria

Re: UKNOF 43 CVE

Wed Apr 10, 2019 1:45 pm

 
User avatar
shaoranrch
Member Candidate
Member Candidate
Posts: 183
Joined: Thu Feb 13, 2014 8:03 pm

Re: UKNOF 43 CVE

Wed Apr 10, 2019 10:51 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
Of course it would still be possible to exploit it from the inside, but frankly I always worry more about exploiting from outside than from inside.
For some time I have a dynamic address list in IPv6 that contains all internal addresses that have attempted to make outgoing traffic (plus some static servers),
and drops all incoming traffic towards addresses not in that list. This drop is now in the forward chain, I will move it to the raw prerouting chain.

Of course this countermeasure generates a new attack surface, where local users are able to fill that address list with 2^64 entries, but as I wrote
I am not so worried that this will happen.
Hi,

I'd like to know about the 2^64 entries. Is this the limit for IPv6 address list entries? I'm guessing the attack you are mentioning is basically by producing a DoS since no more entries can be added to the ADL. If this is the case, where did you get the number? also, do you happen to know which one is for IPv4?

Regards,
Rafael Carvallo
Telecommunications Engineer

Need consultation?
Need a hotspot with facebook integration?
Send a PM!

Hablamos español, atendemos el mercado de latinoamérica visita nuestra página web:
http://www.tuproximosalto.com
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Wed Apr 10, 2019 11:17 pm

In ipv6 usual prefix is /64. So a local attack will not be filtered by the rules proposed and the number of possible hosts is 2^64 because ipv6 addresses are 128 bit numbers.

Enviado desde mi Mi A2 mediante Tapatalk

 
vmiskos
just joined
Posts: 1
Joined: Mon Jan 26, 2015 4:29 pm

Re: UKNOF 43 CVE

Thu Apr 11, 2019 8:09 pm

Since 2 days (9 April) all our mikrotik devices with 64Mb RAM are rebooting continuously after 1 minute and 40-50 seconds. IPv6 package was already disabled since long time. RouterOS versions are not latest but we have very strong security rules. Is this the same DDoS issue? We are not able to reboot or update firmware since reboot command is not responding. Any ideas?
 
Reinis
MikroTik Support
MikroTik Support
Posts: 66
Joined: Wed Jan 02, 2019 12:14 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Fri Apr 12, 2019 2:14 pm

Since 2 days (9 April) all our mikrotik devices with 64Mb RAM are rebooting continuously after 1 minute and 40-50 seconds. IPv6 package was already disabled since long time. RouterOS versions are not latest but we have very strong security rules. Is this the same DDoS issue? We are not able to reboot or update firmware since reboot command is not responding. Any ideas?
If IPv6 was not enabled, then this CVE could not be the reason. Please isolate at least one of the devices which get rebooted, generate a Supout.rif file and send it to support@mikrotik.com, of course, if you have any additional information, provide that too.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5940
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Fri Apr 12, 2019 2:48 pm

Anyone who still had problems with small RAMs -> viewtopic.php?f=21&t=146087&p=726299#p726296
 
User avatar
shaoranrch
Member Candidate
Member Candidate
Posts: 183
Joined: Thu Feb 13, 2014 8:03 pm

Re: UKNOF 43 CVE

Fri Apr 12, 2019 4:13 pm

In ipv6 usual prefix is /64. So a local attack will not be filtered by the rules proposed and the number of possible hosts is 2^64 because ipv6 addresses are 128 bit numbers.

Enviado desde mi Mi A2 mediante Tapatalk
Hey,

I still don't quite get it. I do understand that this vector won't the blocked from inside by the proposed rules. What I don't get is why it'll open a new vector.

If I do get what he's doing correctly this is what is happening:

1.- Host A internally initiates traffic towards the outside
2.- Router adds an address list item with Host A local IP
3.- Return traffic comes in and checks dst-address-list looking for the Host A IP address
4.- If it is, the traffic is allowed if not, it's discarded
5.- Address list entries timeout in 8 hours

What is this attack surface he's talking about? this is what I'm not fully getting

A.- allowing all the /64 ips in that subnet unrestricted inbound access?
B.- DoS the entire the network because it's not possible to have more than 2^64 addresses into the address-list so other subnets trying to initiate traffic can't due to the return traffic getting blocked?

Or am I missing something here?

Regards,
Rafael Carvallo
Telecommunications Engineer

Need consultation?
Need a hotspot with facebook integration?
Send a PM!

Hablamos español, atendemos el mercado de latinoamérica visita nuestra página web:
http://www.tuproximosalto.com
 
pe1chl
Forum Guru
Forum Guru
Posts: 5909
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 12, 2019 7:13 pm

What is this attack surface he's talking about? this is what I'm not fully getting
The issue is that you can set an outgoing address for your device, send a packet to outside, the address will be added
to the address list, then you set another address, send a packet, another address added to the address list, etc.
When you repeat this for all 2^64 possible addresses all the RAM in the router will be used up for this address list.
(of course happens long before that number of addresses is tried)

This requires a malicious user on the inside, which I consider a much lower risk than an outside user who would
scan all the addresses inside your space (which would cause the issue that was described in the CVE).
 
keefe007
Member Candidate
Member Candidate
Posts: 124
Joined: Sun Jun 25, 2006 3:01 am

Re: UKNOF 43 CVE

Fri Jun 07, 2019 5:39 pm

What are the symptoms of this issue?

Who is online

Users browsing this forum: No registered users and 68 guests