UKNOF 43 CVE

msatter · Fri Mar 29, 2019 3:35 pm

Thanks Maznu for finding this and reporting it to Mikrotik. Good to see that the communications is up-to-speed now so that Mikrotik can handle this correctly and in time for us Mikrotik device owners.

honzam · Fri Mar 29, 2019 4:28 pm

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish.

Normis - "Maznu" wrote the dates and when sent it to you. Ticket numbers... Description... All this a year ago. What do you need more?

Maznu - Thank you for your comments and activity

highonsnow · Fri Mar 29, 2019 4:30 pm

For everyone here, I wanted to clarify, that to my best knowledge, the author of the CVE has not contacted MikroTik and we are in the dark as to what he plans to publish.
"maznu" wrote the dates he sent it to you. Ticket numbers. Description... All this a year ago. What do you need more?

Maznu - Thank you for your comments

It's quite clear that this wasn't given much priority.. only since there was a fire lit under the subject did it get pounded into a beta

Fri Mar 29, 2019 4:43 pm

Normis - "Maznu" wrote the dates and when sent it to you. Ticket numbers... Description... All this a year ago. What do you need more?

No. He did not send proof of concept for all issues, just a generic report about a crash. When he now said that CVE number such and such is not fixed, It was not clear, since we don't know what he will publish in that CVE. There is not a single issue, there are multiple issues, we fixed most, now he has stumbled upon another (memory leak in some other condition). We are fixing the others as well.

Fri Mar 29, 2019 4:46 pm

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:

admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes

Replace 2001:db8:3::/64 with your network.

eben · Fri Mar 29, 2019 4:53 pm

Can I run this on my edge routers and use my /32 or do I need to do it on each router?

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
Code: Select all
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.

Fri Mar 29, 2019 4:56 pm

It should be enough to limit on edge router, since it already limits to 2 new connections every second, unless routers further along the path have less than 100MB free ram, then probably you will need to limit even more on that specific router.

TomjNorthIdaho · Fri Mar 29, 2019 5:01 pm

I think we should listen to what he has to say …

many many many many many years ago, I had both Microsoft and Sun Microsystems on the phone telling both of them I could lockup their Internet connected computers. They did not believe me so with both of them online in a conference call to both, I locked up the computers on the IP addresses they told me to try. Well - that little easy lockup trick is called "Ping of Death".
Yup - that was me that discovered it and worked with both Sun and Microsoft verify they later fixed their TCP/IP stack buffer-over-run problem.

Another one with Microsoft that would totally lockup a Windows computer was the left over PIP "ConCon" ( a carry over from the original CPM operating system ).

The point I am trying to make is that if somebody says there is a vulnerability, I wish they would get together in a live on-line conference and show their stuff and prove their point - prior to releasing the vulnerability to the general public.

North Idaho Tom Jones

TomjNorthIdaho · Fri Mar 29, 2019 5:10 pm

I think we should listen to what he has to say …

many many many many many years ago, I had both Microsoft and Sun Microsystems on the phone telling both of them I could lockup their Internet connected computers. They did not believe me so with both of them online in a conference call to both, I locked up the computers on the IP addresses they told me to try. Well - that little easy lockup trick is called "Ping of Death".
Yup - that was me that discovered it and worked with both Sun and Microsoft verify they later fixed their TCP/IP stack buffer-over-run problem.

Another one with Microsoft that would totally lockup a Windows computer was the left over PIP "ConCon" ( a carry over from the original CPM operating system ).

The point I am trying to make is that if somebody says there is a vulnerability, I wish they would get together in a live on-line conference and show their stuff and prove their point - prior to releasing the vulnerability to the general public.

Question - Am I correct in guessing this is something like "Packet of death in IPv6" , pretty much the same issue in IPv6 that is called Ping of Death in IPv4 ?

North Idaho Tom Jones

maznu · Fri Mar 29, 2019 5:12 pm

No. He did not send proof of concept for all issues, just a generic report about a crash. When he now said that CVE number such and such is not fixed, It was not clear, since we don't know what he will publish in that CVE. There is not a single issue, there are multiple issues, we fixed most, now he has stumbled upon another (memory leak in some other condition). We are fixing the others as well.

I sent support@mikrotik.com two tickets in April 2018 about two different issues. Later, in November, I advised MikroTik that these had now been allocated two different CVE numbers.

Both tickets in April were acknowledged and accepted as deficiencies by support@mikrotik.com, and I had given you the exact commands I had used so that you could reproduce them. Your team wrote back to my saying:

We have tested the same on our local network and managed to reproduce the same problem. We will see what we can do about this problem and improve IPv6 functionality in future RouterOS releases.

Your report and input into reproducing this problem is higly appreciated.

In those initial emails, almost a year ago, I even gave your support team my ideas about what I thought the problems were and — guess what! — your support team just told me about an hour ago that they believe the they've now found some problems to be — and they are exactly what I said in April 2018.

dottxt · Fri Mar 29, 2019 5:20 pm

For CVE-2018-19299, Are systems that do not have IPv6 connection tracking enabled affected?

maznu · Fri Mar 29, 2019 5:23 pm

For CVE-2018-19299, Are systems that do not have IPv6 connection tracking enabled affected?

Yes.

IPANetEngineer · Fri Mar 29, 2019 6:15 pm

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
Code: Select all
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.

I'm repeating the test several times just to make sure we have it right. Should this mitigate both CVEs?

maznu · Fri Mar 29, 2019 6:23 pm

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.

May I please add "discovered independently by a third party" to the timeline and credit you during my UKNOF 43 talk, then?

eles · Fri Mar 29, 2019 6:43 pm

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.
May I please add "discovered independently by a third party" to the timeline and credit you during my UKNOF 43 talk, then?

The post @IPANetEngineer was responding to, was Normis's.

buraglio · Fri Mar 29, 2019 6:45 pm

I've been watching this quietly since it started. One of these CVEs, while important to note, is pretty straightforward to replicate with off the shelf, open source tools. I suspect the other is as well but I have not yet done so. MT should definitely address this and add testing of neighbor table issues to the ROS CI/CD process. I would also recommend adding a tunable for limiting the neighbor table size per device for those of us that wish to set hard limits on ND cache to a known quantity.
Unfortunately the "third party" who "discovered" the issue has been very cagey and a bit sensational about the discovery, which seems to have 1. hyped up the talk he is to give and 2. get folks riled up and in knee-jerk mode.
MT has never been an organization that has reacted in flamboyant ways when it comes to fixing bugs or vulnerabilities - they simply assess the situation and act based on their internal assessment. Sure, it would have been nice to see this fixed faster, but they didn't, and we don't know the back channel communication that happened when it was initially reported or how it was presented.
My point is that these aren't cryptic, esoteric issues. They are common IPv6 implementation problems that anyone running v6 at scale has probably seen in years past with other vendors. They have a magnified spotlight on them since ROS has had a year of high profile issues.
Keep calm, we're gonna be ok.

sebastia · Fri Mar 29, 2019 6:59 pm

/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
2001:db8:3::/64 new-connection-mark=drop passthrough=yes[/code]

Looking at the remaining workaround, usual end-user blocking any incoming traffic not already established / related, isn't impacted, right?

davidcx · Fri Mar 29, 2019 7:04 pm

Looking at the remaining workaround, usual end-user blocking any incoming traffic not already established / related, isn't impacted, right?

You'd be limiting traffic to 2 new flows per second which is not an option except in the tiniest of networks.

Now, there may be a higher value that is acceptable to your conditions (if you are not a transit network where you have no business dictating to your customers how many flows they may have), but whether that mitigates the bug in question - specifically whether the memory leaked is recovered at a faster rate than the rate limit prevents new flows - remains to be seen when the bug details are released.

-davidc

sebastia · Fri Mar 29, 2019 7:14 pm

Not what I mean.

Looking at the remaining workaround, usual end-user blocking any incoming traffic not already established / related, isn't impacted, right?

Usual end-user scenario, so not a major network operation, is to block any incoming traffic, unless it is related to already established traffic from inside. So accepting internal risk, users dropping their own internet pipe, nothing extra needs to be done since, it's going to be dropped anyway if a new connection arrives from internet direction internal network.

luciano · Fri Mar 29, 2019 7:27 pm

What happy friday! Disabling BGP IPv6 Peers on our clients...

anav · Fri Mar 29, 2019 7:30 pm

They have to fix the code without breaking the rest of it. Appears this is why it took so long. If they are changing something at a deep deep root level, then the ramifications will spread out like spider webs and each strand has to be dealt with. Not surprising that it takes time. All the while they have been dealing with other exploits in quick fashion and as well handling the more mundane improvements.
IF anyone likes extraordinary amounts of stress just apply to work at MT. Its not all pizza and beer and video games. Speaking about thirst, I have some Canadian free beer for any MT employee that comes to Canada.

Or Normis can fly me to Latvia and I will bring some beer (but only after ipv6 is fixed, no distractions needed at the moment).

leopiri · Fri Mar 29, 2019 10:29 pm

Maznu, thank you for showing that you are seeking a solution for the whole community.
Could you inform me if disabling SSH and Winbox service also works the exploit?
Using RoMon only can be a "temporary" solution?

Appreciate.
Leonardo

This version fixes:
1) Soft lockup when IPv6 router is forwarding IPv6 packets;
2) Soft lockup when the router is forwarding packets to a local network (directly connected) due to large IPv6 Neighbor table.

We are still working on improvements for IPv6 Neighbor table processing in userspace which can lead up to OOM condition.
Since CVE detials are not yet published, we did assume that CVE targets software lockup (#2) which we did fix in this release.
I've sent through all the details in a new ticket, Ticket#2019032922005182, which I hope includes all the information you need collated into one place.

I'm very glad you've prioritised this issue with your development team, and I look forward to testing releases that address the problem soon.

maznu · Fri Mar 29, 2019 10:52 pm

Maznu, thank you for showing that you are seeking a solution for the whole community.
Could you inform me if disabling SSH and Winbox service also works the exploit?
Using RoMon only can be a "temporary" solution?

How you access the router isn't the major factor here… I'm not sure I understand your question?

marlow · Fri Mar 29, 2019 11:36 pm

Until we release the next beta with memory exhaustion fix, this firewall config should stop any attack even with small amount of RAM:
Code: Select all
admin@MikroTik] /ipv6 firewall> export 
/ipv6 firewall filter
add action=drop chain=forward connection-mark=drop connection-state=new
/ipv6 firewall mangle
add action=accept chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 limit=2,5:packet
add action=mark-connection chain=prerouting connection-state=new dst-address=\
    2001:db8:3::/64 new-connection-mark=drop passthrough=yes
Replace 2001:db8:3::/64 with your network.

Normis:

That may work for a small end user network. It does not work for a medium sized internet provider.

Running beta releases on production networks ALSO does not work for an internet provider.

I will have to decomission at least 12x 72core and 36core CCRs in the next week and replace them with something from another vendor. Having ignored requests for implementing IPv6-Delegated-Prefix on PPPoE for years !! is one of the reasons we would have done this anyhow.

Another reason is that we will be migrating away from Mikrotik entirely. And we have 1000s of Mikrotik devices in the network. When it takes over a week or more to get a response to a support ticket, then that is not acceptable. If you get a response at all. And that is the norm currently.

But ignoring a bug like this for a year is taking the biscuit. Also .. it was reported to Mikrotik. If Mikrotik were not sure or did have to think about, what and how these exploits were achieved it would have been Mikrotiks RESPONSIBILITY to engage with the person reporting it, to make sure you understood the issue correct. It is not the reporter that has the responsibility for your products and your customers. It is Mikrotik.

Guessing, that you have fixed the issue, is not good enough. If you thought you had fixed it and were not sure, you should have engaged with the person reporting it AFTER the fix was released and made sure you had all bases covered. That is if you care about your business and your customers.

/M

cmurrayis · Fri Mar 29, 2019 11:43 pm

@normis while it's great that you plan on fixing the issues by the release date it doesn't help the rest of the world who have to patch their networks before this time also.

We have to plan outages months in advance for some customers and even for our core it's 7 days notice for critical patches which first have to go through stability testing in lab first. This leaves is with not much time to be confident no new issues are introduced.

killersoft · Sat Mar 30, 2019 1:44 am

While many of you are notably upset about the extraordinary amount of time that has gone by on this issue. I note some of you are wanting to move to new product vendors. This is your prerogative to do so. That said, I will point out the BIG VENDORS such as CISCO are smashed by CVE's problems ALL the time. Mikrotik is doing pretty well on the CVE footprint issue comparatively.

Take a look at cisco CVE's just this year I count 130 as of this writing, out of 3100 since 1995.
https://tools.cisco.com/security/center ... nListing.x

Makes you wonder what hackers/etc haven't publicly released with known bugs in those big 'brand' name products...

So even if you PAY the big bucks, your probably getting an 'Average' 'network' product(comes supplied with buggy O/S's) with chrome trim's and a nice help desk support number, that may/may-not be able to assist in a timely manor.

You are all lucky that for the relatively cheap(very cheap!) price of MT products, you get a beta release every ~1-3 weeks, while the one O/S RouterOS in this case is being improved.

marlow · Sat Mar 30, 2019 1:49 am

While many of you are notably upset about the extraordinary amount of time that has gone by on this issue. I note some of you are wanting to move to new product vendors. This is your prerogative to do so. That said, I will point out the BIG VENDORS such as CISCO are smashed by CVE's problems ALL the time. Mikrotik is doing pretty well on the CVE footprint issue comparatively.

They are doing well on the ones that go public, yes. The issue here is not just these issues. It's the whole approach on handling support requests and the time it takes to get a response, changing things in RouterOS effectively breaking working features without even discussing the reasoning for these changes, not implementing fundamental features.

Also, a lot of bugs don't get reported anymore, because we know it doesn't lead anywhere anyhow.

At some point enough, is enough. And yes, other vendors have other issues. Other vendors may also be more costly. But at least other vendors take responsibility for their products, have a clear guideline what a timely response to a ticket is and implement critical features, that customers and the industry needs. Again .. a good example: it's taken Mikrotik over a decade to implement IPV6-Delegated-Prefix, but then they decided only to implement it for DHCP and not for PPPoE. I mean .. where is the logic in that ?

They have also recently (as of 6.43) made it mandatory to use MS-CHAP for user-authentication via radius. So whoever was using CHAP .. which would be an industry standard .. vs MS-CHAP being a proprietary extension ... got their entire setup broken.

These are just example that show how they treat their customers.

The handling of these 2 CVEs reflect the same fundamental problems within Mikrotik. A workaround then gets posted for one of the issues, that works for an end-user network, but essential would cripple the network of an ISP that has an actual IPv6 rollout with thousands of customers using IPv6. Again ... the bigger picture gets missed. Because is Mikrotiks biggest market end-users ? or providers ?

/M

IPANetEngineer · Sat Mar 30, 2019 2:03 am

We believe that we have now recreated the conditions of both CVEs and have been able to cause a memory leak and router crash in both of the conditions listed below using software from a common offensive linux security tool for IPv6.

soft lockup when forwarding IPv6 packets (CVE-2018-19299);
soft lockup when processing large IPv6 Neighbor table (CVE-2018-19298)

Our engineers are working on mitigation options and so far have been able to keep the test routers online by limiting to 5k new connections with 256MB of memory. We still have some testing to do and a number of different options to try.

Will post our rule sets as we further refine the testing but it is a variation on what Normis already posted.

killersoft · Sat Mar 30, 2019 2:05 am

At some point enough, is enough. And yes, other vendors have other issues. Other vendors may also be more costly. But at least other vendors take responsibility for their products, have a clear guideline what a timely response to a ticket is and implement critical features, that customers and the industry needs. Again .. a good example: it's taken Mikrotik over a decade to implement IPV6-Delegated-Prefix, but then they decided only to implement it for DHCP and not for PPPoE. I mean .. where is the logic in that ?

OK, fair enough on the timelines, but I ask what are your PAYING $ for MT support at the moment.... ?

Also if your unhappy with RouterOS on your hardware, feel free to flash OPENWRT onto your MT hardware devices, and become pro-active in network code design and development for the community

https://openwrt.org/toh/hwdata/mikrotik/start

marlow · Sat Mar 30, 2019 2:40 am

Also if your unhappy with RouterOS on your hardware, feel free to flash OPENWRT onto your MT hardware devices, and become pro-active in network code design and development for the community

You will find with a little of research, that I'm very active in the OpenSource community. Especially OpenWRT / Freemesh / Freifunk. But that's neither here nor there.

The issue is, that I'm also a high volume customer of Mikrotik products .. or have been so until recent times. And that pays for the development of said features. I buy 100s of Mikrotik units every year and have done so since 2005. Every single year. So don't tell me, I don't pay them for the support and development, that I expect.

The software is not free. It's paid for as part of either the license or the hardware purchase.

/M

killersoft · Sat Mar 30, 2019 2:49 am

Maybe its time for MT to consider a parallel "community" like edition version of RouterOS. That open to view /compile "source code" and allows the community to quickly fix issues(CVE's !!!) and add networking functionality as community made plugin's for MT Hardware..

leopiri · Sat Mar 30, 2019 3:13 am

Maznu, do the following:

ip service disable [find]

Verify that even with all Mikrotik access media services the problem occurs?

Maznu, thank you for showing that you are seeking a solution for the whole community.
Could you inform me if disabling SSH and Winbox service also works the exploit?
Using RoMon only can be a "temporary" solution?
How you access the router isn't the major factor here… I'm not sure I understand your question?

marlow · Sat Mar 30, 2019 3:24 am

Maybe its time for MT to consider a parallel "community" like edition version of RouterOS. That open to view /compile "source code" and allows the community to quickly fix issues(CVE's !!!) and add networking functionality as community made plugin's for MT Hardware..

Here is your fundamental issue:
In RouterOS 2.9 and before, the routing suite was still based on Quagga (OSPF, BGP etc.)

From RouterOS 3 it was replaced by their own code introducing all sorts of issue and behaviour changes.

The main reason for the change was because Mikrotik did not want to release the source code of their platform. This has happened to other parts of the RouterOS in a similar fashion.

So you have an inherent issue that is, that Mikrotik is not interested in Open Source. The reason I have stuck around for so long is because they have managed to provide a fully integrated platform, with an easy to manage interface, at a cost point that is affordable. That including optimizing (proprietary) wireless protocols, that exceed standard 802.11 in performance and are much more interference resilient.

So over a little more than a year, we have started to migrate the end user side/last mile of our business to other vendors. The edge has never been Mikrotik. The missing part is now the RAS side of things. And that's where these two CVEs hit extremely hard. Because I've provided IPv6 to end users since 2008... by default. Whatever way we could make it work. And traffic volumes on IPv6 are not marginal anymore. It's 20-30% plus.

/M

leopiri · Sat Mar 30, 2019 3:26 am

Do not forget the providers that use FastPath, if activating any rule in the equipment can bump the cpu.

I'm considering a CCR1072 with 15Gb + or CCR1036 with 6Gb +.

Is this fix you're testing going to kill FastPath?

We believe that we have now recreated the conditions of both CVEs and have been able to cause a memory leak and router crash in both of the conditions listed below using software from a common offensive linux security tool for IPv6.

soft lockup when forwarding IPv6 packets (CVE-2018-19299);
soft lockup when processing large IPv6 Neighbor table (CVE-2018-19298)

Our engineers are working on mitigation options and so far have been able to keep the test routers online by limiting to 5k new connections with 256MB of memory. We still have some testing to do and a number of different options to try.

Will post our rule sets as we further refine the testing but it is a variation on what Normis already posted.

aussiewan · Sat Mar 30, 2019 4:05 am

The suggested temporary workaround also does not work if you have connection tracking disabled, as we do for some of our higher-throughput devices.
I think that a key takeaway from this issue is that you shouldn't put all your eggs in one basket. Don't rely on any single vendor for your critical infrastructure. Every vendor has issues from time to time, and if a single issue like this can take your business offline, then you need to re-assess your systems and look into adding some other vendors into the mix. In this case, you don't have redundancy if you have all MikroTik devices running your Internet edge.

rkj · Sat Mar 30, 2019 5:22 am

While the response for this security issue has been sub-optimal, Mikrotik attitude towards criticism comes from long time and also includes lack of features, bugs and anything different from using what is available.

So while this is an escalation, it's a foreseeable development of the existing trend. Anyone using MT should know that by now.

maznu · Sat Mar 30, 2019 8:33 am

Maznu, do the following:

ip service disable [find]

Verify that even with all Mikrotik access media services the problem occurs?

Yes, that is still vulnerable (my test lab has no services enabled because it has no Internet connectivity - only console access). These IPv6 handling problems are not about accessing the router. They are about causing the router to crash by how it routes IPv6 packets.

maznu · Sat Mar 30, 2019 10:30 am

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.

Meanwhile CVE-2018-19299 still needs fixing, because even with those performance-crippling firewall rules on 6.45beta22, I'm crashing routers.

In exactly the same lab layout as before, and with exactly the same single command that MikroTik has known for over 11 months, and still attacking 2001:db8:3::/64 (so your firewall rules are just a copy-and-paste): https://youtu.be/vJBUdAMrKJw

mistry7 · Sat Mar 30, 2019 10:37 am

Mikrotik what is need to happen that you are wake up and get ROS 7 with newer Kernel ready?

ROS Kernel is from 2012!

Good for us, we removed all Mikrotik at Core Routers in the last 3 Month, because of bad BGP Speed, and Missing IPv6 PPPoE Features
No i Think that was a very good decision!

rockymtndata · Sat Mar 30, 2019 11:03 am

Mistry7,

Good call moving off of mikrotik for security reasons. I couldn't agree more. I did too for some things. Today is a security day for us Cisco fans too. There are 17 flaws to fix. Have you seen this.

https://www.networkworld.com/article/33 ... flaws.html

R

mistry7 · Sat Mar 30, 2019 11:23 am

Mistry7,

Good call moving off of mikrotik for security reasons. I couldn't agree more. I did too for some things. Today is a security day for us Cisco fans too. There are 17 flaws to fix. Have you seen this.

https://www.networkworld.com/article/33 ... flaws.html

R

Fixes and Sec Holes are all over the world, we did change not from security side, but Mikrotik don't tells when they are fixing there missing IPv6 PPPoE
Features, and we are not willing to wait until we are out of IPv4 Addresses, so we need a working solution until Mikrotik can't do the job.

viewtopic.php?f=1&t=89443&start=150

Needed Features are requested years ago....same story as ROS 7, and Missing Wireless Features......
But there are others on the Market, and they have this features....

IPANetEngineer · Sat Mar 30, 2019 1:47 pm

Normis...i'm pretty confident we have replicated the conditions of one of the CVEs from doing some digging on our own for this issue. Without the rules, the router crashed. When we added the rules the router stayed online.
Meanwhile CVE-2018-19299 still needs fixing, because even with those performance-crippling firewall rules on 6.45beta22, I'm crashing routers.

In exactly the same lab layout as before, and with exactly the same single command that MikroTik has known for over 11 months, and still attacking 2001:db8:3::/64 (so your firewall rules are just a copy-and-paste): https://youtu.be/vJBUdAMrKJw

We have had some success mitigating CVE-2018-19299 now that we know what command to run in the linux IPv6 exploit tools to crash edge or transit routers. However, we are moving testing over to our dual stack data center on hardware that isn't serving customers to get a better idea of the real world IPv6 results as labs disconnected from the Internet can produce results that don't always translate to the real world in the same way.

The initial hardware we'll be testing will be RB4011s and CCRs. We'll be working on expanding the MikroTik FW rules that we are already working on as well as developing L2 scrubbing policies

While i'm very eager to get a fix from MikroTik, I'm going to keep working on improving our mitigation and will share the results with the group once we've done some testing on actual hardware in a prod IPv6 deployment.

pcunite · Sat Mar 30, 2019 2:43 pm

I want to reiterate what I've stated elsewhere: I believe that all modern software companies need to implement a certain type of business process, known as Lean-Agile. Bugs will happen, no reasonable person is upset about that. Rather it is the release cadence, release channel, and review process that are broken. MikroTik's staff are not incompetent, to suggest that is insulting to them and frankly to those of us who rely on their products.

Nomis, when you get a chance, read that book I linked to. I think it will really help MikroTik. We all want you to win and do well. That is why there is a lot of passion in these comments. The angry outbursts are not about this particular CVE, instead it is the groundswell of a systemic pattern of frustrating end-user experience updates that are causing the hopelessness. It is a confidence issue. If I'm scared to apply your updates, I wonder how others feel? Truth or perception. Which one do we choose?

dmayan · Sat Mar 30, 2019 3:47 pm

We will never see it. Mikrotik is more concerned at improving Kids Control and other useless features than working on a better platform for IPv6 and BGP multi threading for the CCR series. We have nine CCR1072 that can't receive full tables because they literally froze processing them. Already thinking on changing our routing equipment, and we are one of the biggest ISPs of the south of South America. Now I have to disable IPv6 (We are the FIRST provider implementing this on our region) because of this bold***t.

And not speaking about CCRs rebooting itself with no reason and other things like that....

Keep up the good work Mikrotik

The fix is in v7 guys c'mon
Ros 7 is like the Yeti or Mrs. Colombo. Everyone talks about it, but nobody has ever seen it.

neutronlaser · Sat Mar 30, 2019 3:56 pm

You don't have to disable it.

marlow · Sat Mar 30, 2019 4:35 pm

You don't have to disable it.

The fixes posted above will result in useless user experience when implemented by an internet service provider, which is a no go either. So disabling/removing IPv6 from public routers is the only choice here until a fix is in a STABLE release.

So yes... as an internet provider, that has to deliver service to thousands of customers, it's actually the only choice right now.

/M

buraglio · Sat Mar 30, 2019 4:44 pm

Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure. As a relative outsider that has been involved in this kind of thing in the past, my observation is that this entire process has been bungled beyond belief. I feel like I am watching a slow moving train wreck where the train engineer (@maznu) is doing almost nothing other than complain that nothing is being done to keep the train from crashing whilst promoting his talk that he intends to give to take advantage of the public black eye that MT has from previous high profile issues. Fine, I get it. It's an easy way to self promote (although not the way I would have handled it).
The only person trying to actually mitigate the issue other than Mikrotik (which is expectedly a black box of which we cannot see inside), is @IPANetEngineer who has essentially reverse engineered the bugs independently and has potential temporary mitigations. Given what I have seen this is a solvable - albeit inconvenient - issue, that *can* be mitigated with a little work, but the users seem more interested in throwing hands up and turning off the v6 internet, and the author seems fixated on his talk rather than mitigating the issue.
Even as someone involved in building early broadband and very early IPv6 production networks where bugs were numerous and significant, this is painful to watch.

Also, no one that I have ever seen does multi-threaded BGPd. What you want is a more optimized BGP Daemon, which likely won't happen on a CCR because they the Tilera is optimized for power efficiency and not DFZ sized route table optimization. Lame? Yes. Unexpected? Not really.

nb

We will never see it. Mikrotik is more concerned at improving Kids Control and other useless features than working on a better platform for IPv6 and BGP multi threading for the CCR series. We have nine CCR1072 that can't receive full tables because they literally froze processing them. Already thinking on changing our routing equipment, and we are one of the biggest ISPs of the south of South America. Now I have to disable IPv6 (We are the FIRST provider implementing this on our region) because of this bold***t.

And not speaking about CCRs rebooting itself with no reason and other things like that....

Keep up the good work Mikrotik

The fix is in v7 guys c'mon
Ros 7 is like the Yeti or Mrs. Colombo. Everyone talks about it, but nobody has ever seen it.

highonsnow · Sat Mar 30, 2019 4:52 pm

Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure.

Are you suggesting that people apply last minute rushed patches days or hours before disclosure (who knows when it'll be released or stable)? Is that how you operate in your production environments? I'm quite sure any serious ISPs who are affected won't be applying such update policies without having at least tested or proven the firmware before throwing another variable into the mixture. The reaction is preventative and responsible.

Imagine a scenario where they actually took this one seriously and produced a solution for it in good time.. and not waiting until the gun was nearly placed to their head with the looming deadline. None of this would have been happening.

marlow · Sat Mar 30, 2019 4:53 pm

Totally disabling IPv6 before the details of the bug as well as how to exploit it are even public is over-reactionary and knee-jerk extreme, especially since MT has said that a fix will be available before the disclosure.

It is actually not.

MT having a fix ready before disclosure is not enough. The fix needs to be ready, it needs to be in a stable release tree, it needs to tested and validated and customers need time to implement it.

@maznu has provided a clear timeline of when this was reported to Mikrotik, when the CVEs were issued and Mikrotik has ignored this until an imminent threat of it going public came about. If they had used the time from the report and the creation of the CVEs to mitigate this, there would have been plenty of time.

So in my case, it means IPv6 will be removed from all Mikrotik gear, because the risk of loss of business is too great. And where necessary, the gear will have been replaced with gear from other vendors, that's long term proven. Even if Mikrotik released a fix in time, there would be no time for testing. I would literally have to roll it out across 300+ devices in the core network. We have near to 10k Mikrotik devices in our network, if every one of them needs to be updated. This is not something you do in a few hours .. or days. And automated updates are not an option. The quality control of the release trees supplied by Mikrotik are not good enough. We aren't even on the current long-term release yet, as it has bugs we can't live with.

And the issue is, that it is not the first time, that Mikrotik has handled an issue this way. It is every single time. They only react, when issues become public knowledge.

/M

anav · Sat Mar 30, 2019 5:13 pm

If your networks are so huge and yet you have failed to scale your infrastructure accordingly so that updating them is not manageable, and yet you have money to change over many of your devices just tells me you have other issues to overcome before changing equipment over. Besides the fact that you actually believe that a proven long term vendor exists............. Tis a Brave New World Indeed.
In other words it is easy to throw stones without a complete picture or understanding of the whole situation. Suggest your continued ranting may make you feel somehow better, but it does nothing for readers. Perhaps you should put on your hat MMGA ( make mt great again

)

Who is online