Community discussions

 
wtm
Frequent Visitor
Frequent Visitor
Posts: 52
Joined: Tue May 24, 2011 5:27 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:20 pm

So what firmware do we need to install on what routers to prevent this ?
 
enzain
just joined
Posts: 20
Joined: Wed Jan 17, 2018 9:15 pm

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:22 pm

Don't do full tables on CCRs. They are terrible at it.
Why?
Is work fine, i receive FV and default route (while router compute FV's - used default route, when session up, internet work after 2 sec)
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:31 pm

where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE
The full timeline will be available next week. But when I reported this in April 2018, my request to MikroTik was to plead with support to treat this as a serious security vulnerability, and asked that they themselves allocate a CVE to it. After no action for six months, I then raised CVEs myself and communicated this to MikroTik to try and pressure them into taking another look at this.
Marek
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 5:34 pm

Don't do full tables on CCRs. They are terrible at it.
Why?
Is work fine, i receive FV and default route (while router compute FV's - used default route, when session up, internet work after 2 sec)
Default routes mean you can't use uRPF.

Such high CPU usage during table churn means you'll be at best sub-optimally routing packets and at worst, sending them down blackholes for 10 - 20 minutes at a time.
-----
Mike Hammett

The Brothers WISP
 
cantanko
newbie
Posts: 28
Joined: Mon Apr 05, 2010 12:53 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 7:16 pm

So what firmware do we need to install on what routers to prevent this ?

As of right now, 6.45beta23 AFAIK.
 
mstead
Member Candidate
Member Candidate
Posts: 113
Joined: Sat Mar 04, 2006 2:41 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 7:45 pm

where the reporter didn't report it as a security concern and left it for 6 months till he was able to get a CVE
The full timeline will be available next week. But when I reported this in April 2018, my request to MikroTik was to plead with support to treat this as a serious security vulnerability, and asked that they themselves allocate a CVE to it. After no action for six months, I then raised CVEs myself and communicated this to MikroTik to try and pressure them into taking another look at this.
maznu - can you contact me via Twitter? I sent you a tweet already.
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 8:11 pm

I understand how things work in "the infosec world". I have stated above how I describe this and I stand by my comments. Unauthenticated Denial of Service is just as I described it, and it is not exclusively in the domain of security vulnerability. It can definitely be leveraged that way, but the same way as a car can be used to run over a person, we don't call it a weapon. No one is debating or defending anything, I am indifferent on pretty much all hardware and software. This is really just semantics here, and we can all agree that this is a crummy situation that no one loves, but right or wrong, blame solves precisely zero problems.
If you read my post you'll see that my statements are pretty clear. Surprisingly, I have seen more admittance of a mistake out of a vendor than I have seen in quite some time.
> This is not a security vulnerability as I would describe it.

Then what EXACTLY would you call the ability to stop your router from routing, remotely, without authentication? Because we in the Infosec world call that an Unauthenticated Denial of Service. (Which is a bad thing)

Look, MikroTik have dropped the ball on this. That they haven't gone 'Shit, yes, we made a mistake, we're sorry, and we're making (these changes) to make sure it doesn't happen again' is MORE telling.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 8:13 pm

I tested null routing with ip-filter enabled and it still drives the cache up. What I don't know is if
/ip rp-filter
also covers IPv6. If it doesn't then there appears to be no way to enable ipv6 RPF checking.
Don't do full tables on CCRs. They are terrible at it.
Why?
Is work fine, i receive FV and default route (while router compute FV's - used default route, when session up, internet work after 2 sec)
Default routes mean you can't use uRPF.

Such high CPU usage during table churn means you'll be at best sub-optimally routing packets and at worst, sending them down blackholes for 10 - 20 minutes at a time.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 8:17 pm

maznu - can you contact me via Twitter? I sent you a tweet already.
My timeline exploded a bit, as you might imagine. I'm @maznu on Twitter, DMs are open :)
Marek
 
faraujo88
just joined
Posts: 13
Joined: Fri Feb 15, 2019 2:28 am

Re: UKNOF 43 CVE

Mon Apr 01, 2019 9:03 pm

If you just have the package enabled and absolutely no configuration from an IPv6 perspective are you okay?
Something about this question?
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8319
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: UKNOF 43 CVE

Mon Apr 01, 2019 11:05 pm

Something about this question?
Was answered on the first page.
viewtopic.php?p=723912#p723912
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
slimmerwifi
just joined
Posts: 13
Joined: Tue Aug 01, 2017 6:05 pm
Location: Netherlands

Re: UKNOF 43 CVE

Tue Apr 02, 2019 12:19 am

It's true, MikroTik should have handled it better. It was mistakenly filed as just another ipv6 bug, which there are more of, until the author complained about lack of action. Then we looked at it again, and started to look for alternative solutions to handle it (because kernel change is not possible in v6). But in the end, the issue is fixed finally.

P.S.: we don't rely on kernel for TILE support, so new kernel doesn't affect it.
Appreciate this!

Also, send @maznu a present/gift/bounty/4011. He sure as hell earned it.
We manage 50+ corporate wifi networks in the Netherlands using Mikrotik & Cloudcore equipment.
 
cmurrayis
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Fri May 15, 2009 4:31 am

Re: UKNOF 43 CVE

Tue Apr 02, 2019 12:31 am

@normis

Is it possible to get an honest responce about ROS v7's release timetable? We and many other people have been struggling with BGP performance so much so we've had to reject larger potential clients because we cannot offer MPLS due to large packet loss with routing convergence on the CCR1072.

We are not a high volume network and cannot hold a single full table from any of our Transit providers because of CPU load and extended outages if for some reason one of the sessions drop. We were holding out until MUM Europe as we were told by support there would be an announcement there about it but alas there wasn't one.

We now have to seriously look at switching vendors away from Mikrotik which we've used for 15+ years because we just cannot wait any more with such a loose open ended timeline that everything appears to be "Fixed in ROS v7" and there being no release deadline.

Even if you were to release it today it would be 6 months before we could roll it out with enough confidence and testing in LAB given the extended development timeline.

I hope you can take the time to responce publicly or even privately as I'm under alot of pressure to grow our network but with current hardware not able to without potentially impacting client performance and reliability.

We'd also be happy to pay a few thousand dollars a year for advanced support/updates if required to offset the cheap licensing charges.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 2:03 am

Also, send @maznu a present/gift/bounty/4011. He sure as hell earned it.
That's very kind, but after we've all got the patch in longterm and stable, I want to know how I can mail order a crate of beer to MikroTik's offices to say thank you for getting this fixed.
Marek
 
VipITBE
just joined
Posts: 23
Joined: Tue Apr 02, 2013 10:40 am

Re: UKNOF 43 CVE

Tue Apr 02, 2019 8:49 am

@normis

Is it possible to get an honest responce about ROS v7's release timetable? We and many other people have been struggling with BGP performance so much so we've had to reject larger potential clients because we cannot offer MPLS due to large packet loss with routing convergence on the CCR1072.

We are not a high volume network and cannot hold a single full table from any of our Transit providers because of CPU load and extended outages if for some reason one of the sessions drop. We were holding out until MUM Europe as we were told by support there would be an announcement there about it but alas there wasn't one.

We now have to seriously look at switching vendors away from Mikrotik which we've used for 15+ years because we just cannot wait any more with such a loose open ended timeline that everything appears to be "Fixed in ROS v7" and there being no release deadline.

Even if you were to release it today it would be 6 months before we could roll it out with enough confidence and testing in LAB given the extended development timeline.

I hope you can take the time to responce publicly or even privately as I'm under alot of pressure to grow our network but with current hardware not able to without potentially impacting client performance and reliability.

We'd also be happy to pay a few thousand dollars a year for advanced support/updates if required to offset the cheap licensing charges.
If you have performance issues with the CCR1072, then why don't you get a real router? Real hardware and pay the price instead of banging on Mikrotik for not releasing ROSv7?
Also, there are plenty of WISPs using MPLS and routing on mikrotik without issues...
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24268
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Tue Apr 02, 2019 8:55 am

Is it possible to get an honest responce about ROS v7's release timetable?
Maybe other topic ?
No answer to your question? How to write posts
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2292
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: UKNOF 43 CVE

Tue Apr 02, 2019 10:16 am

@maznu - beta23 fixes both vulnerability? Did you test?
LAN, FTTx, Wireless. ISP operator
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 10:42 am

@maznu - beta23 fixes both vulnerability? Did you test?
I emailed MikroTik yesterday, tweeted, and posted about this on the 6.45beta thread - yes!

MikroTik has said that another beta is expected to make the settings on the affected components more "optimal" for devices with low RAM. I hope it lands soon.
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 5:17 pm

More testing has yielded more data. This has not been properly replicated by anyone else that I know of, so take it as plausible hypothesis. I think I found more fallout from the ipv6 flaw: boxes that have their ND cache or their ipv6 route cache run up but not to the point of OOM reload experience a degraded performance over time in fun and unique ways after a scan attempt or resource exhaustion. I have seen two symptoms, all resolved by a reboot.
SNMP responds with wildly inaccurate data, or no data at all, but the snmp daemon continues to run
Both IPv4 and IPv6 packet forwarding experiences odd blips in function that don't seem to correspond to anything I can find (again, no snmp data to validate against)

Again, this is unverified but I suspect very likely. It'd be nice to see someone else replicate this or disprove it.

nb
@maznu - beta23 fixes both vulnerability? Did you test?
I emailed MikroTik yesterday, tweeted, and posted about this on the 6.45beta thread - yes!

MikroTik has said that another beta is expected to make the settings on the affected components more "optimal" for devices with low RAM. I hope it lands soon.
ForwardingPlane, LLC
https://www.forwardingplane.net
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 5:26 pm

More testing has yielded more data. This has not been properly replicated by anyone else that I know of, so take it as plausible hypothesis. I think I found more fallout from the ipv6 flaw: boxes that have their ND cache or their ipv6 route cache run up but not to the point of OOM reload experience a degraded performance over time in fun and unique ways after a scan attempt or resource exhaustion. I have seen two symptoms, all resolved by a reboot.
This gels with something I found when I was doing some testing last night. I suspect the problem is that the various processes in RouterOS struggle to malloc() memory for various tasks, and it makes for a very painful experience. My symptoms included: typing on the console being very laggy, issuing commands was also quite slow, and a reboot fixed it. In extreme memory exhaustion (but again, not quite OOM), the router seems to become unresponsive for a short while (won't answer telnet connections, etc), but sort of seems to be answering ping and forwarding packets… kind of. I hypothesise that this is what MikroTik has been referring to as a "soft lockup" in their 6.45beta22 changelog entries. It "felt" very similar to the feeling of interacting with a Linux server under extreme load, swapping or otherwise thrashing.
Marek
 
cantanko
newbie
Posts: 28
Joined: Mon Apr 05, 2010 12:53 am

Re: UKNOF 43 CVE

Tue Apr 02, 2019 7:13 pm

This gels with something I found when I was doing some testing last night. I suspect the problem is that the various processes in RouterOS struggle to malloc() memory for various tasks, and it makes for a very painful experience.

That description of malloc() after high memory pressure agrees with my experience with completely unrelated-but-linux-based software that was continually malloc(), realloc() and free()ing memory all over the place, causing heavy page-table fragmentation and thus real contention issues for other processes trying to malloc() memory. The enterprise distros I was working with at the time kinda sorta try to deal with it using page table defragmentation / consolidation, but that too has its own associated costs. With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available.

I don't envy the Mikrotik devs on this one if it is something along those lines...
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 7:43 pm

With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available.
The data structure that the Linux kernel used in RouterOS v6 uses for the IPv6 route table (and also route cache) is a sort of radix tree. It's a very different different from how the data structures for IPv4 work (because the IPv6 routing table is very sparse by comparison), and it's different again to how the more modern kernels do this too. I suspect the kernel is allocating the memory for this in fairly large slabs, but it would not at all surprise me if there is a possibility of fragmentation.

All that aside, it is possible to consume a nice big chunk of contiguous memory in RouterOS's user-space — and that ought to have very similar effects. That was how I got a bunch of CHRs behaving in a way that I believe matches what buraglio is describing above. A fairly easy way to try this out might be to inject a bunch of unreachable BGP routes to make the "route" process use more memory. As you get down to a few Mbytes of free RAM, things get icky… but this is completely expected behaviour (even if it would be unfortunate).
Marek
 
buraglio
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Mon Aug 10, 2015 5:59 pm
Location: +1 (217)
Contact:

Re: UKNOF 43 CVE

Tue Apr 02, 2019 8:12 pm

Dang, I was hoping I was wrong. Looking like probably not.

nb
With extreme fragmentation, it can result in no contiguous memory that satisfies the malloc() or realloc() and you either segfault in userland or (I'd imagine) panic in the kernel, hence the reboot even with memory theoretically available.
The data structure that the Linux kernel used in RouterOS v6 uses for the IPv6 route table (and also route cache) is a sort of radix tree. It's a very different different from how the data structures for IPv4 work (because the IPv6 routing table is very sparse by comparison), and it's different again to how the more modern kernels do this too. I suspect the kernel is allocating the memory for this in fairly large slabs, but it would not at all surprise me if there is a possibility of fragmentation.

All that aside, it is possible to consume a nice big chunk of contiguous memory in RouterOS's user-space — and that ought to have very similar effects. That was how I got a bunch of CHRs behaving in a way that I believe matches what buraglio is describing above. A fairly easy way to try this out might be to inject a bunch of unreachable BGP routes to make the "route" process use more memory. As you get down to a few Mbytes of free RAM, things get icky… but this is completely expected behaviour (even if it would be unfortunate).
ForwardingPlane, LLC
https://www.forwardingplane.net
 
BRMateus2
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Thu Oct 26, 2017 11:18 pm

Re: UKNOF 43 CVE

Tue Apr 02, 2019 11:44 pm

Memory fragmentation undefined behaviour bugs happens even in Windows (do a 500MB free test without swap, Windows craps out).
It is very hard to handle such malloc issues, because it is part of malloc itself - any kind of new route or such, will bug out the dynamic buffers, etc.
 
User avatar
Jesse6
just joined
Posts: 6
Joined: Fri Nov 17, 2017 6:35 pm
Location: Brazil

Re: UKNOF 43 CVE

Wed Apr 03, 2019 4:48 pm

It is not new that Mikrotik cares very very little about IPv6 deployment in their products, features such as, 'remote-dhcpv6-pd-pool' for radius, VRF support for IPv6 (or, something I believe it's even better, source based routing), Dude (that is a very excellent product, but don't have IPv6 support) with IPv6 monitoring capability, etc., are in discuss for almost a decade, with no signs of foreseeable implementation. Since at now we are almost 5 years that IPv6 is the new internet standard, I really hope you guys put the IPv6 at the top of the list in your development. And yes, I know that a vulnerability is something that happens to EVERY human developed device/firmware, it is understandable, and I think you guys are working hard to fix that. But this is a good moment to highlight the weakest Mikrotik's point for the upcoming changes: IPv6.
I wish you guys good luck with that fix, and hope you see IPv6 more seriously.
 
User avatar
KeepBurning
just joined
Posts: 1
Joined: Mon Apr 01, 2019 7:51 pm
Location: Brazil

Re: UKNOF 43 CVE

Wed Apr 03, 2019 9:08 pm

So can I assume that version 6.45beta23 is safe to use IPv6?
MAJOR CHANGES IN v6.45:
----------------------
!) ipv6 - fixed soft lockup when forwarding IPv6 packets;
!) ipv6 - fixed soft lockup when processing large IPv6 Neighbor table;
 
User avatar
honzam
Forum Guru
Forum Guru
Posts: 2292
Joined: Wed Feb 27, 2008 10:27 pm
Location: Czech Republic

Re: UKNOF 43 CVE

Thu Apr 04, 2019 11:10 am

Fixel also in long-term - 6.43.14
and Current - 6.44.2
LAN, FTTx, Wireless. ISP operator
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 12:18 pm

Fixel also in long-term - 6.43.14
and Current - 6.44.2
Attacked both, and both releases fix CVE-2018-19299. Fantastic news — but now the hard work for all us network operators begins:

1. 🔬 test

2. 🧠 plan

3. 🔨 deploy

4. 🔍 monitor

5. 🍺🍻🎉

6. 😴 🛌
Marek
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Thu Apr 04, 2019 12:29 pm

Just wondering what will happen / be the effect when "under attack" and hitting memory limit?
* on neighbour mem limit
* on routing cache limit

Router will survive, but what with the legit connections?
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 12:44 pm

Just wondering what will happen / be the effect when "under attack" and hitting memory limit?
* on neighbour mem limit
* on routing cache limit
Router will survive, but what with the legit connections?
The tests I did on 6.45beta23 suggested different levels of memory usage would be used for the IPv6 routing table/cache depending on the installed RAM in the router. For example, routers with 2Gb of memory would permit a few hundred thousand entries in the IPv6 routing table/cache.

The challenge on Linux 3.3.5 (and many kernel version after) is that the IPv6 routing cache is stored in the same data structure (and therefore memory) as the routing table, which means that an attack can still fill that allocated memory. Further research is required to find out whether this would then deny real routes being inserted (e.g. learned from upstream providers via BGP), or whether a "least recently used" type of algorithm will allow a route insertion to replace a temporary cache entry.
Marek
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Thu Apr 04, 2019 2:21 pm

What I don't understand: why is it not possible to firewall against it.
When you limit the addresses that are routed, e.g. by dropping traffic in the raw prerouting table, does it still
create entries for the dropped traffic in the route cache or neighbor table? Why?
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24268
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: UKNOF 43 CVE

Thu Apr 04, 2019 2:23 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
No answer to your question? How to write posts
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 2:31 pm

What I don't understand: why is it not possible to firewall against it.
When you limit the addresses that are routed, e.g. by dropping traffic in the raw prerouting table, does it still
create entries for the dropped traffic in the route cache or neighbor table? Why?
If you DROP in PREROUTING then it is safe against CVE-2018-19299.

Dropping in chain=forward is not safe, because a routing table lookup has to be done before the firewall rule can be evaluated (otherwise the firewall rule could not know which the in- and out-interfaces are, etc). This means a cache entry will have already been created, even if the chain=forward rule does action=drop.
Marek
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Thu Apr 04, 2019 3:13 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
Of course it would still be possible to exploit it from the inside, but frankly I always worry more about exploiting from outside than from inside.
For some time I have a dynamic address list in IPv6 that contains all internal addresses that have attempted to make outgoing traffic (plus some static servers),
and drops all incoming traffic towards addresses not in that list. This drop is now in the forward chain, I will move it to the raw prerouting chain.

Of course this countermeasure generates a new attack surface, where local users are able to fill that address list with 2^64 entries, but as I wrote
I am not so worried that this will happen.
 
anav
Forum Guru
Forum Guru
Posts: 3113
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: UKNOF 43 CVE

Thu Apr 04, 2019 4:02 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
Congrats to your Team Normis, under what must be an incredibly stressful pressure cooker of a year. Proost, gun bei, sit, cin cin, Na zdrowie and Priekā!
phuckin A batman!
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:11 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5942
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:14 pm

Completely unrelated to the topic.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:14 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
Marek
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5942
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 5:21 pm

It is an upgrade problem because of no free space on the router, not related to this thread at all.
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 6:13 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.

I fear the only way to get this onto a hAP Lite is to Netinstall.

If I have to send people to clients' homes to netinstall their routerboards, I'd rather just go and buy TPLink 840s and send the techs off to install them. Less hassle for everyone.

As far as I am concerned I do not have an update to install on a hAP and the matter is far from over.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 6:14 pm

It is an upgrade problem because of no free space on the router, not related to this thread at all.
I have 6.43.14 installed on a hAP ac lite (64Mb RAM), and it is still vulnerable. Ticket#2019040222005195 and Ticket#2019032922005182
Marek
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 998
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 7:48 pm

It is an upgrade problem because of no free space on the router, not related to this thread at all.
Question - if there is a problem with not enough free space (on some mikrotiks) …
On a remote wireless client , is it possible to do something like this …
- downgrade the remote client to a smaller ROS (or even a stripped down tiny ROS) that keeps the remote client connected
- then upgrade the remote client to 6.44.2 (or what ever newer version works)
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Thu Apr 04, 2019 8:23 pm

I have done several tests with GNS3 using CHR 6.44.2 (stable) and as long as the router has enough memory, it doesn't crash. In my tests, the attack 'steals' around 180 MiB.

Using a CHR with 256 MB, system resources shows a total memory of 224 MiB and free-memory of 197 MiB before attack. During the attack, only from one computer, the free memory decreases to around 20 MiB and sometimes to 13 MiB. Using two attackers, it seems the results are the same and not worst.

With 200 MB the router reboots because OOM.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Thu Apr 04, 2019 9:15 pm

@pe1chl, question: in your setup externally initiated ipv6 traffic is disallowed right?
 
neutronlaser
Member Candidate
Member Candidate
Posts: 212
Joined: Thu Jan 18, 2018 5:18 pm

Re: UKNOF 43 CVE

Thu Apr 04, 2019 11:17 pm

ipv6 dumb extravagance anyway
 
highonsnow
newbie
Posts: 34
Joined: Fri Oct 19, 2007 3:30 am

Re: UKNOF 43 CVE

Thu Apr 04, 2019 11:35 pm

ipv6 dumb extravagance anyway
Welcome to 2019.. you must have been asleep since DARPA were experimenting with this TCP/IP thing.. that's ok though, we'll help you through it.

It all started with RIPE and global commerce..
 
anav
Forum Guru
Forum Guru
Posts: 3113
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:50 am

ipv6 dumb extravagance anyway
Welcome to 2019.. you must have been asleep since DARPA were experimenting with this TCP/IP thing.. that's ok though, we'll help you through it.

It all started with RIPE and global commerce..
Thank god for that, otherwise I wouldn't had the pure joy of playing doom with a brit and a yank all being served up by one of our pcs all over a 33.6k modem connection for about 10min before crashing LOL. Really rocked with the 56k modem though. ;-)
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 11:56 am

@pe1chl, question: in your setup externally initiated ipv6 traffic is disallowed right?
Yes, externally initiated IPv6 traffic to random addresses is disallowed. I added this when NDP exhaustion attacks were discussed.
Due to the address list, only systems that have initiated outbound traffic (within the last 8 hours) plus a number of addresses of
servers put in the address list as static entries are allowed inbound.

But I had this rule in the /ipv6 firewall filter chain=forward list which should be fine to prevent NDP exhaustion attacks but apparently is not enough
for the route cache table overflow attack, so I moved it to /ipv6 firewall raw chain=prerouting list.
Of course there also is a rule in /ipv6 firewall filter chain=forward that adds the src address to the list (with 8 hour timeout) for new outbound traffic.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Apr 05, 2019 12:48 pm

Thx for info, I guess you do it (the address addition to the list) on connection level, not for each packet?
Last edited by sebastia on Fri Apr 05, 2019 2:28 pm, edited 1 time in total.
 
nostromog
Member Candidate
Member Candidate
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 1:32 pm

@pe1chl, question: in your setup externally initiated ipv6 traffic is disallowed right?
Yes, externally initiated IPv6 traffic to random addresses is disallowed. I added this when NDP exhaustion attacks were discussed.
Due to the address list, only systems that have initiated outbound traffic (within the last 8 hours) plus a number of addresses of
servers put in the address list as static entries are allowed inbound.

But I had this rule in the /ipv6 firewall filter chain=forward list which should be fine to prevent NDP exhaustion attacks but apparently is not enough
for the route cache table overflow attack, so I moved it to /ipv6 firewall raw chain=prerouting list.
Of course there also is a rule in /ipv6 firewall filter chain=forward that adds the src address to the list (with 8 hour timeout) for new outbound traffic.
It looks like a very interesting idea.
Thx for info, I guess you do it on connection level, not for each packet?
If I understood correctly pe1chl, the capturing of outward addresses is done at the connection=new level, but the dropping of packets has to be done before connection tracking to avoid the routing machinery to engage, so the dropping has to happen at the packet level at raw...
Last edited by nostromog on Fri Apr 05, 2019 2:09 pm, edited 1 time in total.
 
User avatar
maznu
Member Candidate
Member Candidate
Posts: 197
Joined: Tue May 05, 2015 11:12 am
Location: Manchester, UK
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 1:35 pm

I have done several tests with GNS3 using CHR 6.44.2 (stable) and as long as the router has enough memory, it doesn't crash. In my tests, the attack 'steals' around 180 MiB.

Using a CHR with 256 MB, system resources shows a total memory of 224 MiB and free-memory of 197 MiB before attack. During the attack, only from one computer, the free memory decreases to around 20 MiB and sometimes to 13 MiB. Using two attackers, it seems the results are the same and not worst.

With 200 MB the router reboots because OOM.
I had a response from MikroTik earlier saying: "Next beta will have further improvements."

Fingers crossed, everyone…
Marek
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Fri Apr 05, 2019 2:09 pm


This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.
Eben: you are aware that you can pull the "all_packages.zip" file, only upload the modules you need and upgrade ?

Installing everything is not always an advantage and this is not the first time in the lifetime of RouterOS, that this has been an issue (example RB112 and 113c for ROS3 and upwards)

/M


Communication is the beginning of understanding
-- AT&T
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 2:31 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.
Eben: you are aware that you can pull the "all_packages.zip" file, only upload the modules you need and upgrade ?

Installing everything is not always an advantage and this is not the first time in the lifetime of RouterOS, that this has been an issue (example RB112 and 113c for ROS3 and upwards)

/M
We can do package by package, not on a couple of thousand routers in three days.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 2:33 pm

This is far from over.

Please refer to ticket 2019040422005244 and advise.
I'm hearing reports that this isn't fixed on routers with 64Mb or less of RAM. Is your ticket about this, eben? Or something else? :-|
I've tried installing 6.44.2 on about 50 hAP Lites using manual update, Dude Update, Winbox update, Commandline update vir puTTY.

Fail on all fronts.

There isn't enough memory / storage for the update.
Eben: you are aware that you can pull the "all_packages.zip" file, only upload the modules you need and upgrade ?

Installing everything is not always an advantage and this is not the first time in the lifetime of RouterOS, that this has been an issue (example RB112 and 113c for ROS3 and upwards)

/M
We can do package by package, not on a couple of thousand routers in three days.
https://unimus.net/blog/network-wide-mi ... grade.html
-----
Mike Hammett

The Brothers WISP
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:25 pm

Can it do package by package, or just platform by platform?
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:29 pm

Can it do package by package, or just platform by platform?
On the master ROS install, just have only the packages you do want.
-----
Mike Hammett

The Brothers WISP
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 3:54 pm

If I understood correctly pe1chl, the capturing of outward addresses is done at the connection=new level, but the dropping of packets has to be done before connection tracking to avoid the routing machinery to engage, so the dropping has to happen at the packet level at raw...
Correct!
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1790
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: UKNOF 43 CVE

Fri Apr 05, 2019 5:17 pm

In such form, it does introduce another potential issue: connections lasting longer than "timeout" can be impacted as their packets will get dropped for IPv6 implementing privacy features, which change ipv6 after a while.
Something to keep in mind.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 05, 2019 9:10 pm

Well to be more precise I add the entry for outgoing traffic AND src address not in list.
So an existing connection will survive when at least it has some outgoing traffic.
Besides, the type of usage we have usually don't notice an interrupted connection that has been sitting idle.
(mostly phones, tablets and laptops used for websurfing)
 
User avatar
marlow
Member Candidate
Member Candidate
Posts: 159
Joined: Thu Mar 16, 2006 6:59 pm
Location: Ireland

Re: UKNOF 43 CVE

Fri Apr 05, 2019 10:51 pm

We can do package by package, not on a couple of thousand routers in three days.

I don't know, how you get the idea of package by package. You upload the needed packages. The bare minimum. Reboot. It installs all the npk files in one go.

It's no different than uploading the combined package. Just multiple files instead.

/M
Communication is the beginning of understanding
-- AT&T
 
tdw
Member Candidate
Member Candidate
Posts: 196
Joined: Sat May 05, 2018 11:55 am

Re: UKNOF 43 CVE

Sat Apr 06, 2019 12:37 am

Yes, externally initiated IPv6 traffic to random addresses is disallowed. I added this when NDP exhaustion attacks were discussed.
Due to the address list, only systems that have initiated outbound traffic (within the last 8 hours) plus a number of addresses of
servers put in the address list as static entries are allowed inbound.

But I had this rule in the /ipv6 firewall filter chain=forward list which should be fine to prevent NDP exhaustion attacks but apparently is not enough
for the route cache table overflow attack, so I moved it to /ipv6 firewall raw chain=prerouting list.
Of course there also is a rule in /ipv6 firewall filter chain=forward that adds the src address to the list (with 8 hour timeout) for new outbound traffic.
@pe1chl - having moved your rule from chain=forward to chain=prerouting do you add your WAN address to the list too (as chain=prerouting will affect input as well as forward traffic), or does your router not provide any external services?
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Sat Apr 06, 2019 1:27 am

@pe1chl - having moved your rule from chain=forward to chain=prerouting do you add your WAN address to the list too (as chain=prerouting will affect input as well as forward traffic), or does your router not provide any external services?
We have no external IPv6 service on the router but indeed if you have, you should add it as a static entry.
(we do have that for a web- and a mailserver)

Edit: actually the mechanism is a little more complicated than I described which makes listing the router's own address unnecessary in my case, but if implemented
as I described it certainly is required to add the router's address as used on the link to the ISP.
Last edited by pe1chl on Sat Apr 06, 2019 12:28 pm, edited 1 time in total.
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Sat Apr 06, 2019 8:49 am

We can do package by package, not on a couple of thousand routers in three days.

I don't know, how you get the idea of package by package. You upload the needed packages. The bare minimum. Reboot. It installs all the npk files in one go.

It's no different than uploading the combined package. Just multiple files instead.

/M
We tested this on three routers during the night. It works - just, but there's no way we'll be able to finish within the time constraints.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 737
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: UKNOF 43 CVE

Sat Apr 06, 2019 8:55 am

We can do package by package, not on a couple of thousand routers in three days.

I don't know, how you get the idea of package by package. You upload the needed packages. The bare minimum. Reboot. It installs all the npk files in one go.

It's no different than uploading the combined package. Just multiple files instead.

/M
We tested this on three routers during the night. It works - just, but there's no way we'll be able to finish within the time constraints.
How do you normally update your routers?
-----
Mike Hammett

The Brothers WISP
 
User avatar
eben
Member
Member
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: UKNOF 43 CVE

Sat Apr 06, 2019 9:03 am

How do you normally update your routers?
We have a set of scripts. All non smips routers are done and dusted.
 
schadom
Member Candidate
Member Candidate
Posts: 139
Joined: Sun Jun 25, 2017 2:47 am
Location: Austria

Re: UKNOF 43 CVE

Wed Apr 10, 2019 1:45 pm

 
User avatar
shaoranrch
Member Candidate
Member Candidate
Posts: 183
Joined: Thu Feb 13, 2014 8:03 pm

Re: UKNOF 43 CVE

Wed Apr 10, 2019 10:51 pm

It can be firewalled like you say, I posted rules that give you ideas how (and you can tune it to your needs).
But many said that they have legitimate traffic coming from a single source to multiple destinations.
Of course it would still be possible to exploit it from the inside, but frankly I always worry more about exploiting from outside than from inside.
For some time I have a dynamic address list in IPv6 that contains all internal addresses that have attempted to make outgoing traffic (plus some static servers),
and drops all incoming traffic towards addresses not in that list. This drop is now in the forward chain, I will move it to the raw prerouting chain.

Of course this countermeasure generates a new attack surface, where local users are able to fill that address list with 2^64 entries, but as I wrote
I am not so worried that this will happen.
Hi,

I'd like to know about the 2^64 entries. Is this the limit for IPv6 address list entries? I'm guessing the attack you are mentioning is basically by producing a DoS since no more entries can be added to the ADL. If this is the case, where did you get the number? also, do you happen to know which one is for IPv4?

Regards,
Rafael Carvallo
Telecommunications Engineer

Need consultation?
Need a hotspot with facebook integration?
Send a PM!

Hablamos español, atendemos el mercado de latinoamérica visita nuestra página web:
http://www.tuproximosalto.com
 
User avatar
jprietove
Trainer
Trainer
Posts: 96
Joined: Fri Jun 03, 2016 3:00 pm
Location: Cádiz, Spain
Contact:

Re: UKNOF 43 CVE

Wed Apr 10, 2019 11:17 pm

In ipv6 usual prefix is /64. So a local attack will not be filtered by the rules proposed and the number of possible hosts is 2^64 because ipv6 addresses are 128 bit numbers.

Enviado desde mi Mi A2 mediante Tapatalk

 
vmiskos
just joined
Posts: 1
Joined: Mon Jan 26, 2015 4:29 pm

Re: UKNOF 43 CVE

Thu Apr 11, 2019 8:09 pm

Since 2 days (9 April) all our mikrotik devices with 64Mb RAM are rebooting continuously after 1 minute and 40-50 seconds. IPv6 package was already disabled since long time. RouterOS versions are not latest but we have very strong security rules. Is this the same DDoS issue? We are not able to reboot or update firmware since reboot command is not responding. Any ideas?
 
Reinis
MikroTik Support
MikroTik Support
Posts: 67
Joined: Wed Jan 02, 2019 12:14 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Fri Apr 12, 2019 2:14 pm

Since 2 days (9 April) all our mikrotik devices with 64Mb RAM are rebooting continuously after 1 minute and 40-50 seconds. IPv6 package was already disabled since long time. RouterOS versions are not latest but we have very strong security rules. Is this the same DDoS issue? We are not able to reboot or update firmware since reboot command is not responding. Any ideas?
If IPv6 was not enabled, then this CVE could not be the reason. Please isolate at least one of the devices which get rebooted, generate a Supout.rif file and send it to support@mikrotik.com, of course, if you have any additional information, provide that too.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5942
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: UKNOF 43 CVE

Fri Apr 12, 2019 2:48 pm

Anyone who still had problems with small RAMs -> viewtopic.php?f=21&t=146087&p=726299#p726296
 
User avatar
shaoranrch
Member Candidate
Member Candidate
Posts: 183
Joined: Thu Feb 13, 2014 8:03 pm

Re: UKNOF 43 CVE

Fri Apr 12, 2019 4:13 pm

In ipv6 usual prefix is /64. So a local attack will not be filtered by the rules proposed and the number of possible hosts is 2^64 because ipv6 addresses are 128 bit numbers.

Enviado desde mi Mi A2 mediante Tapatalk
Hey,

I still don't quite get it. I do understand that this vector won't the blocked from inside by the proposed rules. What I don't get is why it'll open a new vector.

If I do get what he's doing correctly this is what is happening:

1.- Host A internally initiates traffic towards the outside
2.- Router adds an address list item with Host A local IP
3.- Return traffic comes in and checks dst-address-list looking for the Host A IP address
4.- If it is, the traffic is allowed if not, it's discarded
5.- Address list entries timeout in 8 hours

What is this attack surface he's talking about? this is what I'm not fully getting

A.- allowing all the /64 ips in that subnet unrestricted inbound access?
B.- DoS the entire the network because it's not possible to have more than 2^64 addresses into the address-list so other subnets trying to initiate traffic can't due to the return traffic getting blocked?

Or am I missing something here?

Regards,
Rafael Carvallo
Telecommunications Engineer

Need consultation?
Need a hotspot with facebook integration?
Send a PM!

Hablamos español, atendemos el mercado de latinoamérica visita nuestra página web:
http://www.tuproximosalto.com
 
pe1chl
Forum Guru
Forum Guru
Posts: 5919
Joined: Mon Jun 08, 2015 12:09 pm

Re: UKNOF 43 CVE

Fri Apr 12, 2019 7:13 pm

What is this attack surface he's talking about? this is what I'm not fully getting
The issue is that you can set an outgoing address for your device, send a packet to outside, the address will be added
to the address list, then you set another address, send a packet, another address added to the address list, etc.
When you repeat this for all 2^64 possible addresses all the RAM in the router will be used up for this address list.
(of course happens long before that number of addresses is tried)

This requires a malicious user on the inside, which I consider a much lower risk than an outside user who would
scan all the addresses inside your space (which would cause the issue that was described in the CVE).
 
keefe007
Member Candidate
Member Candidate
Posts: 124
Joined: Sun Jun 25, 2006 3:01 am

Re: UKNOF 43 CVE

Fri Jun 07, 2019 5:39 pm

What are the symptoms of this issue?

Who is online

Users browsing this forum: No registered users and 119 guests