Remote Host Scanning our IPv6 Network

maznu · Wed Sep 20, 2017 6:26 pm

A few days ago one of our routers was hitting IPv6 neighbor cache exhaustion. The symptoms were occasional unreachability via IPv6. I pulled up Torch and found someone was actually scanning our network, probing consecutive addresses in a /64 to see if anything responded!

Dropping this traffic is easy, but remember that because this is a neighbor discovery cache exhaustion attack, you can't protect yourself with /ipv6 firewall filter to drop on forward. You need something like:

/ipv6 firewall raw add action=drop chain=prerouting src-address-list=shitpit
/ipv6 firewall address-list add list=shitpit address=2001:0db8:85a3:0000:0000:8a2e:0370:7334

Hope that helps someone who encounters this rather… pointless scanning.

Wed Sep 20, 2017 11:31 pm

I wish a pack of wild dogs would devour anyone who does this crap on purpose.

And kudos for using the doc prefix in your example.

maznu · Thu Sep 21, 2017 12:38 am

And kudos for using the doc prefix in your example. :)

Thanks.

Currently I'm unsure whether this scan (coming from a university network) is "legitimate research" or "pwned box" — as yet, no response from their abuse contact — otherwise my example might have named-and-shamed the device in question ;-)

Modestas · Thu Sep 21, 2017 9:21 am

Hope that helps someone who encounters this rather… pointless scanning.

Hi
I think you have found risk which calls for mitigation techniques in RouterOS.
Default IPv6 max-neighbor-entries table size is 8K, entries with status="failed" stay there nearly 30 secs. Quite easy to keep such table busy.
Unless we have some method to detect next candidate and update "shitpit" ACL automagically, more ideas would be welcome.
Should it be configurable ND cache expiration time? Dropping icmpv6 responses to attacker IP in firewall when certain threshold is reached? Clearing oldest entries in failed neighbor list when next one is probed?

Thu Sep 21, 2017 5:39 pm

Default IPv6 max-neighbor-entries table size is 8K, entries with status="failed" stay there nearly 30 secs. Quite easy to keep such table busy.
Unless we have some method to detect next candidate and update "shitpit" ACL automagically, more ideas would be welcome.
Should it be configurable ND cache expiration time? Dropping icmpv6 responses to attacker IP in firewall when certain threshold is reached? Clearing oldest entries in failed neighbor list when next one is probed?

This last suggestion sounds pretty awesome to me.

nicktc · Tue Sep 26, 2017 10:30 pm

Any suggestions? We are currently filtering target by doing !IPv6/112 instead of allowing the entire /64.. luckily we did static addresses or this would not be possible.

idlemind · Thu Sep 28, 2017 6:41 pm

I wonder if a firewall filter with limits would pick this scanning up effectively without blocking real traffic. Setting the packet count to just below the maximum you can set the "MaxNeigbhorEntries" value to without experiencing exhaustion from a ND cache perspective but not high enough to cause resource exhaustion on the device.

Possibly a rule that matches on connection-state=new and by per packet limiting?

A 3 pronged approach:

Double the cache to 16384 (/ipv6 settings set max-neighbor-entries=16384)
Apply firewall rules that identify the number of connections and on violation adds them to a source address list that can be dropped in raw for more performance
Use /126 addressing on PtP links and only use /64 on shared connections (where you need SLAAC)

/ipv6 firewall filter add connection-limit=6000,128 action=add-src-to-address-list ...

A real fix would be able to control IPv6 ND traffic or at least control the cache like we do with ARP from a timing perspective. Some previous documentation and testing indicates this is difficult to exploit when the cache time is drastically lowered.

[admin@rtr1] > ipv6 settings set max-neighbor-entries=

MaxNeighborEntries ::= 0..4294967295    (integer number)

pe1chl · Thu Sep 28, 2017 7:11 pm

When you really need an IPv6 network to be reachable from outside without stateful filtering, it is better to limit
its size e.g. to /120 or /112 as mentioned.
"usually" most people would have a stateful firewall or allow incoming traffic only to a few addresses and this issue
would not occur.
When the network is simply open, most routers will have this problem.

Thu Sep 28, 2017 7:51 pm

Or use link-local addressing for router-to-router links and /128 on loopback interfaces.
That'd clear it right up.

I'm mostly kidding because I understand the drawbacks to using link-local-only addressing within non-trivial topologies.

idlemind · Thu Sep 28, 2017 7:59 pm

Derp, now able to replicate it on my 750Gr3. Oddly, I see the number of entries in /ipv6 neighbor climbing well past my current setting of maxneighbors. That said it hasn't seemed to hit my level of free-memory yet. Thankfully the 750Gr3 has 256 MB. It is slowly creeping downwards though. I'm definitely able to populate the table faster than it is purged by default. I suspect I'll be able to fill all free-memory. We'll see if the 750Gr3 locks up or what.

I'll have to see if my connection-limiting would protect from this. I'm not sure if it'd be a stretch to perform connection tracking on a "per-tower" basis where you want to use a /64 for the benefit of SLAAC from a hardware requirements perspective.

I know this has been mitigated by default in the latest Cisco routers for a while now with cache limits per interface.

At the time of this post ...

[admin@rtr1] > ipv6 neighbor print count-only where interface=br1-vlan41 
46883

[admin@rtr1] > system resource print 
                   uptime: 1w1d2h33m40s
                  version: 6.41rc31 (testing)
               build-time: Sep/20/2017 06:56:52
         factory-software: 6.36.1
              free-memory: 144.7MiB
             total-memory: 256.0MiB
                      cpu: MIPS 1004Kc V2.15
                cpu-count: 4
            cpu-frequency: 880MHz
                 cpu-load: 2%
           free-hdd-space: 6.0MiB
          total-hdd-space: 16.3MiB

[admin@rtr1] > ipv6 settings print 
                       forward: yes
              accept-redirects: yes-if-forwarding-disabled
  accept-router-advertisements: yes-if-forwarding-disabled
          max-neighbor-entries: 8192

pe1chl · Thu Sep 28, 2017 8:37 pm

I know this has been mitigated by default in the latest Cisco routers for a while now with cache limits per interface.

Cache limits only limit the problem of complete resource (memory) exhaustion in the router, not the "denial of service" problem on the network itself.
They were introduced when it was shown that routers could be completely DOS'ed (for IPv6 AND IPv4) using a quite slow scan.
They fix that problem, but not the problem this topic started with.

acruhl · Sat Sep 30, 2017 5:38 am

A few days ago one of our routers was hitting IPv6 neighbor cache exhaustion. The symptoms were occasional unreachability via IPv6. I pulled up Torch and found someone was actually scanning our network, probing consecutive addresses in a /64 to see if anything responded!
....
Hope that helps someone who encounters this rather… pointless scanning.

That type of scanning technique is indeed pointless or dimwitted, however it may have served it's purpose... That might be the real context.

Turns out IPv6 scanning isn't "that" hard, although it's still much harder than IPv4 with zmap:

https://tools.ietf.org/html/rfc7707

(Gives me the heebie jeebies a little bit.)

Anyone scanning sequential addresses is either an idiot or they are doing exactly what happened to you. Glad to see some discussion on it.

Thinking out loud: What if your router could reply from every non allocated address in the PD? Would be interesting to "see what happens". Chaos, obviously. Fun.

acruhl · Sat Sep 30, 2017 3:30 pm

So I thought about what I said above, and it wasn't well thought out. I guess what I'm suggesting is that anyone on an IPv6 endpoint, that is trying to run actual IPv6 services to the internet (web, streaming, whatever), might be well served by a honeypot that responds to "everything" on the PD subnets not meant for a legitimate server. Or maybe even just respond to random addresses. You'd tie up anyone trying to discover real hosts basically "forever" and it would be pointless. Would be easy to DDoS but things are these days anyway. Maybe what I'm saying isn't a new idea in IPv6 land.

maznu · Sun Oct 01, 2017 4:17 pm

That type of scanning technique is indeed pointless or dimwitted, however it may have served it's purpose... That might be the real context.

It was still >2million more attempts (many hours) before this "attack" ceased. Maybe neighbour cache exhaustion was the purpose… never heard back from the university in question so probably not "real" research.

maznu · Sat Mar 31, 2018 12:47 pm

Another one for our IPv6 address space scanning shitlist:

add address=2607:f140:4800::/48 list=shitpit

It's likely being done by one of the authors of this paper (or someone working with them) at Berkeley University: https://conferences.sigcomm.org/imc/201 ... nal245.pdf

They're doing a sort of enumerative scan of IPv6 address space, depth-first, which results in about 100pps of IPv6 traffic from them. That soon fills up a neighbour cache on a smaller device (even if you've set it to 100k+ entries), and pretty soon afterwards your little device loses its connectivity (cue lots of Nagios alerts).

I've contacted the abuse address and registered address in WHOIS, and reached out to one of the researchers on Twitter, and suggested they add some "ethical considerations" to section 4 of their paper — and think about potential impact upon targeted networks.

Meanwhile is there anything MikroTik could do in RouterOS to make it easier to flush out "failed" or "noarp" entries in the IPv6 neighbour cache? Or at least let us adjust the timeout for the failed entries to be much quicker than the successful ones?

maznu · Sat Mar 31, 2018 12:48 pm

They're doing a sort of enumerative scan of IPv6 address space, depth-first, which results in about 100pps of IPv6 traffic from them. That soon fills up a neighbour cache on a smaller device (even if you've set it to 100k+ entries), and pretty soon afterwards your little device loses its connectivity (cue lots of Nagios alerts).

In the time it's taken me to email their network abuse contact, and shitpost about them on Twitter, they've probed ~150k addresses. Fun times.

pe1chl · Sat Mar 31, 2018 6:50 pm

Those "researchers" are the worst. With a hacker or prober (those guys that keep a list of available addresses/ports for hackers)
you at least know they do it to cause (indirect) damage. The researchers however are claiming they are doing it for a good
purpose, yet they are causing the same damage.

I think starting this kind of research immediately disqualifies them as a credible scientist. One that would first investigate the situation,
define a research project, and find out about possible unwanted effects before even starting the actual operation.

idlemind · Sat Mar 31, 2018 7:13 pm

Seems like MikroTik needs a ND policer like everyone else implemented in 2012 or earlier. That said it would constitute IPv6 feature work and we know how unlikely that is at MikroTik.

Maybe a high severity CVE is needed to get MikroTik's attention to effectively mitigate this.

Or, take my approach which is growing I'm the community, stop purchasing new MikroTik equipment until they begin to take IPv6 seriously.

maznu · Sat Mar 31, 2018 7:39 pm

In the time it's taken me to email their network abuse contact, and shitpost about them on Twitter, they've probed ~150k addresses. Fun times.

I received a reply just now:

Hi Marek,

We've added your prefix […] to our blacklist.

We spread probes as evenly as possible across routed prefixes, and shuffle targets within each prefix.

Regards,
6Gen Team

If they're *actually* spreading things out evenly across routed prefixes, then to get the quantity of packets we were receiving they are scanning the Internet at somewhere around 20-40Mpps.

Seems like MikroTik needs a ND policer like everyone else implemented in 2012 or earlier.
Maybe a high severity CVE is needed to get MikroTik's attention to effectively mitigate this.

"mikrotik.com has IPv6 address 2a02:610:7501:1000::2"

It's only a matter of time before the researchers hit that…

pe1chl · Sat Mar 31, 2018 7:47 pm

Seems like MikroTik needs a ND policer like everyone else implemented in 2012 or earlier.
Maybe a high severity CVE is needed to get MikroTik's attention to effectively mitigate this.
"mikrotik.com has IPv6 address 2a02:610:7501:1000::2"

It's only a matter of time before the researchers hit that…

But of course that is not likely to cause a problem. Especially with an address like that.
The issue only occurs when there are large (/64) subnets behind a router that does not do any filtering.
Hosting usually is on small subnets (/112) and often there will be a firewall on the router that only passes
traffic to a few wanted ports on some specific addresses. When you see addresses like ::2 it points
to a situation where the admin actually considered this.

idlemind · Sat Mar 31, 2018 8:15 pm

Seems like MikroTik needs a ND policer like everyone else implemented in 2012 or earlier.
Maybe a high severity CVE is needed to get MikroTik's attention to effectively mitigate this.
"mikrotik.com has IPv6 address 2a02:610:7501:1000::2"

It's only a matter of time before the researchers hit that…
But of course that is not likely to cause a problem. Especially with an address like that.
The issue only occurs when there are large (/64) subnets behind a router that does not do any filtering.
Hosting usually is on small subnets (/112) and often there will be a firewall on the router that only passes
traffic to a few wanted ports on some specific addresses. When you see addresses like ::2 it points
to a situation where the admin actually considered this.

MikroTik needs to support the usage of global unicast IPv6 at /64 hands down. That means they need a ND policer to begin to control this. Any recent Cisco IOS does this by not allowing the number of unresolved entries beyond a certain point. This does of course limit the discobery of a legitimate host that comes only using ND sourced from the router. That said it's far more likely to be added to the ND cache as a reachable entry from actual traffic sourced by the new client.

pe1chl · Sat Mar 31, 2018 9:11 pm

There already is a limit like that. It is not like in the original Cisco IOS where the ND cache simply allocated
out of the entire free memory pool and all memory was used (and thus the entire router got into trouble) as a
result of such a scan.
In RouterOS, like now in the Cisco routers, the size of the ND cache is limited. The only improvement could
be in the handling of extra requests beyond the limit.
Aside from that, there is a lot of work to be done on the IPv6 facilities in RouterOS, let's not distract from that
by focussing on the least important issues like this one.

idlemind · Sat Mar 31, 2018 10:38 pm

There already is a limit like that. It is not like in the original Cisco IOS where the ND cache simply allocated
out of the entire free memory pool and all memory was used (and thus the entire router got into trouble) as a
result of such a scan.
In RouterOS, like now in the Cisco routers, the size of the ND cache is limited. The only improvement could
be in the handling of extra requests beyond the limit.
Aside from that, there is a lot of work to be done on the IPv6 facilities in RouterOS, let's not distract from that
by focussing on the least important issues like this one.

As of 6.41rc it does not appear to work. Look at my post early on. I had set the cache limit and it easily exceeded that while slowly chewing up memory which would eventually OOM the router. I didn't actually get my HEX to crash though. So op, can you provide the models and RouterOS versions you're seeing that are effected?

Linux has this in the form of a kernel option:

neigh/default/unres_qlen - INTEGER
The maximum number of packets which may be queued for each
unresolved address by other network layers.
(deprecated in linux 3.3) : use unres_qlen_bytes instead.
Prior to linux 3.3, the default value is 3 which may cause
unexpected packet loss. The current default value is calculated
according to default value of unres_qlen_bytes and true size of
packet.
Default: 101

sysctl -a | grep ipv6.neigh.

You'll see a value you can set as a default or per interface. Being that MikroTik is Linux underneath it's possible this is already exposed. Not sure how to verify if it's present in the kernel MikroTik is using.

pe1chl · Sun Apr 01, 2018 11:48 am

Linux has this in the form of a kernel option:

neigh/default/unres_qlen - INTEGER
The maximum number of packets which may be queued for each
unresolved address by other network layers.
(deprecated in linux 3.3) : use unres_qlen_bytes instead.
Prior to linux 3.3, the default value is 3 which may cause
unexpected packet loss. The current default value is calculated
according to default value of unres_qlen_bytes and true size of
packet.
Default: 101
Code: Select all
sysctl -a | grep ipv6.neigh.
You'll see a value you can set as a default or per interface. Being that MikroTik is Linux underneath it's possible this is already exposed. Not sure how to verify if it's present in the kernel MikroTik is using.

That is actually something different. There are two different issues here:

1. the neighbor table itself expanding too much because there are so many addresses in a /64 subnet, most of them unused, and the scan is causing Neighbor Solicitations being sent and temporary neighbor table entries being created waiting for the response or a time-out. when the neighbor table is dynamically allocated and has no limit, this threatens to use up all available memory.

2. the router may hold the packet that it has received and cannot forward because it has no neighbor table entry. Cisco routers do not do that, this is why they always lose the first packet sent to a new host. however, Linux does do that: it keeps some of those packets on the unres_q and the above parameter is for setting that. the router would still work when this is zero, it can be set a bit higher to improve the quality of service, but it should not be set so high that this queue threatens to use all memory

The fact that you see increasing memory use is not alarming, the mentioned router types have an abundance of memory and can easily spend some tens of MB on things like this without running out of memory during normal operation. So you first have to present proof that it can actually run out of memory due to incorrect settings of that unres_qlen.
I think it is more likely the actual neighbor table was filled with unresponsive entries and hence there was no more room for actual hosts.
As mentioned before, the allocation algorithm within that fixed length table could maybe be improved.

idlemind · Sun Apr 01, 2018 3:27 pm

xfrm6_gc_thresh - INTEGER
The threshold at which we will start garbage collecting for IPv6
destination cache entries. At twice this value the system will
refuse new allocations.

^^ probably a more correct Linux kernel option for what we're looking for the. One I posted earlier is not correct as you pointed out pe1chi

Now, in Linux I wonder what entries the garbage collector purges. The op has stated he loses connectivity. Not sure if it's from the host failing because of an OOM state or because the router loses the neighbor table data for downstream devices while being scanned. If it's the latter I'd suspect the garbage collector is dropping valid neighbors to make room for more scans. I'll have to boot up a few Linux VMs and check out how they handle it. In the Cisco IOS case the valid entries stay as they explicitly garbe collect and restrict only new neighbor discovery attempts (default 512).

Op, any updates on your situation? Any contact from MikroTik support?

idlemind · Sun Apr 01, 2018 3:28 pm

xfrm6_gc_thresh - INTEGER
The threshold at which we will start garbage collecting for IPv6
destination cache entries. At twice this value the system will
refuse new allocations.

^^ probably a more correct Linux kernel option for what we're looking for the. One I posted earlier is not correct as you pointed out pe1chi

Now, in Linux I wonder what entries the garbage collector purges. The op has stated he loses connectivity. Not sure if it's from the host failing because of an OOM state or because the router loses the neighbor table data for downstream devices while being scanned. If it's the latter I'd suspect the garbage collector is dropping valid neighbors to make room for more scans. I'll have to boot up a few Linux VMs and check out how they handle it. In the Cisco IOS case the valid entries stay as they explicitly garbe collect and restrict only new neighbor discovery attempts (default 512).

Op, any updates on your situation? Any contact from MikroTik support?

maznu · Sun Apr 01, 2018 11:08 pm

Now, in Linux I wonder what entries the garbage collector purges. The op has stated he loses connectivity. Not sure if it's from the host failing because of an OOM state or because the router loses the neighbor table data for downstream devices while being scanned. If it's the latter I'd suspect the garbage collector is dropping valid neighbors to make room for more scans. I'll have to boot up a few Linux VMs and check out how they handle it. In the Cisco IOS case the valid entries stay as they explicitly garbe collect and restrict only new neighbor discovery attempts (default 512).

It's not an OOM, at least I don't think so. I'd expect there to be more significant impact were it an OOM (e.g. routing protocols might get OOM-killed and restart, ssh/webfig would become problematic when viewing large datasets, etc). I think it's the neighbour table getting completely full of unresolved destinations — when I looked, on a device with a ~128k limit, there were about 128k entries. Only about 20 of those entries were shown as "valid", and that's for a router with many more actual physical neighbours than just 20. So I would agree with your supposition that it could be the garbage collector naïvely reaping the oldest entries, rather than reaping oldest non-valid neighbours.

maznu · Sun Apr 01, 2018 11:10 pm

Not sure if it's from the host failing because of an OOM state

Oh, yes, and the other reason I don't think it's an OOM:

Increasing the IPv6 neighbour cache size (e.g. doubling it) is an instant fix. At least till the tables have filled up again. But that bought me enough time to figure out where the scan was coming from and raw-table-drop it.

idlemind · Mon Apr 02, 2018 5:30 pm

I wonder if we can poll the ND cache with something like SNMP so it could be automatically monitored or at least dumped effectively for a post-mortem to see how the garbage collection happens to determine if the cache fills with discovery in progress entries and deletes older but valid entries.

If that's the cache a ND policer like mechanism is definitively needed. The IOS one simply constrains the number of addresses that can be tried for resolution at one time allow the cache to never fill up and generic garbage collection to never kick in. Any IPv6 enabled network even on a LAN could be DoS'd without it. Now that I'm home today if I have time I'll tinker with it myself some.

idlemind · Sat Apr 07, 2018 4:50 am

https://tools.ietf.org/html/rfc6583#section-6.4

A good read from 6.4 down and definitely should be read, documented and best practice recommendations made by MikroTik staff.

pe1chl · Sat Apr 07, 2018 2:06 pm

I did the following as an experiment:
- source addresses of new outgoing traffic are added to an addresslist "inuse" with 8-hour timeout
- new incoming traffic to addresses not in this "inuse" list gets the source address added to a "scanners" addresslist with 8-hour timeout
- new incoming traffic from addresses in this "scanners" list is dropped

It appears to work well. Of course it relies on systems making regular outgoing traffic. "servers" could be
permanently added to the "inuse" list without timeout.

idlemind · Sun Apr 08, 2018 2:21 am

So ... Grab a Linux box, put it on your LAN and prepare to LULz to the point of tears.

My normal desktop is a Fedora machine plugged into a Meraki L2 switch and then into a MikroTik HEX w6.42rc52.

Pick an IPv6 /64 that the HEX has to route locally (another network / VLAN in your environment where it would perform discovery).

sudo dnf -y install thc-ipv6
sudo ndpexhaust26 -p -r enp3s0 2605:a000:a8ca:2741::/64

Start laughing or crying at that point because your HEX is rebooting. Yes, rebooting. Target a Cisco 1841 w/15.0 code and default settings ... nothing happens.

Output from the 1841 when viewing it's ND Cache:

rtr1841-top#sh ipv6 neighbors statistics 
IPv6 ND Statistics
 Entries 516, High-water 516, Gleaned 3, Scavenged 2, Static 0
 Entry States
   INCMP 512  REACH 2  STALE 1  GLEAN 0  DELAY 0  PROBE 1
 Resolutions
   Requested 1404, timeouts 2936, resolved 6, failed 886
   In-progress 512, High-water 512, Throttled 12882, Data discards 1
 NUD
   Requested 45, timeouts 9, resolved 41, failed 3
   in-progress 1, high-water 3, throttled 0, current queue 0, queue high-water 0

^^ You can see it is keeping the concurrent requests in check. A continuous ping also shows that connected hosts with REACHABLE entries keeps them reachable.

At this time, this is now another IPv6 deficiency that will prevent me from recommending MikroTik products for anything other than home routing. The fact the router can be REBOOTED simply by an end-user running ndpexhaust26 is scary enough not to mention the ISP side implications of using this hardware and exposing yourself to continue ND related issues.

maznu · Sun Apr 08, 2018 12:29 pm

Start laughing or crying at that point because your HEX is rebooting. Yes, rebooting. Target a Cisco 1841 w/15.0 code and default settings ... nothing happens.

For what it's worth, I've raised this with MikroTik support, Ticket#2018040822000592.

pe1chl · Sun Apr 08, 2018 1:09 pm

Target a Cisco 1841 w/15.0 code and default settings ... nothing happens.

A Cisco 1841 cost 30 times what a HEX cost, and for that price you don't even get software updates.
(you need to get a support contract for that in addition, which will set you back for the same amount of money over the lifetime)

maznu · Fri Apr 13, 2018 1:18 pm

Start laughing or crying at that point because your HEX is rebooting. Yes, rebooting. Target a Cisco 1841 w/15.0 code and default settings ... nothing happens.

Confirmed. It'll OOM a CCR with 2GB RAM and 256k IPv6 neighbour entries. It doesn't even need to be to a subnet which is directly connected to the CCR, from what I can tell.

As requested by MikroTik support, I've sent them supout files. Hopefully they'll take this seriously, because it's looking like a very exploitable problem right now.

maznu · Fri Apr 13, 2018 1:19 pm

Another one for our IPv6 address space scanning shitlist:

add address=2607:f140:4800::/48 list=shitpit

It's likely being done by one of the authors of this paper (or someone working with them) at Berkeley University: https://conferences.sigcomm.org/imc/201 ... nal245.pdf

I've been contacted by the researchers, and we've got a bit of a dialogue going. Hello, Austin, when you click through to this thread. Thanks for getting in touch :-)

maznu · Fri Apr 13, 2018 2:58 pm

So ... Grab a Linux box, put it on your LAN and prepare to LULz to the point of tears.

At this time, this is now another IPv6 deficiency that will prevent me from recommending MikroTik products for anything other than home routing. The fact the router can be REBOOTED simply by an end-user running ndpexhaust26 is scary enough not to mention the ISP side implications of using this hardware and exposing yourself to continue ND related issues.

I've done some tests, and the problem looks to be worse than it appears…

Am in discussions with Martins S. at MikroTik support — hopefully we will start to make some progress on this soon.

pe1chl · Fri Apr 13, 2018 3:12 pm

I'm a bit confused about what is your problem. When it is in the ND it should only affect reachable networks.
Are you getting a static prefix from your ISP? (e.g. a /48)
Did you make sure to have an "unreachable" route for your entire prefix in the route table? (with only /64 routes to your reachable networks in place, and the default)

maznu · Fri Apr 13, 2018 3:36 pm

I'm a bit confused about what is your problem. When it is in the ND it should only affect reachable networks.
Are you getting a static prefix from your ISP? (e.g. a /48)
Did you make sure to have an "unreachable" route for your entire prefix in the route table? (with only /64 routes to your reachable networks in place, and the default)

We've got our own /29 and AS number, full BGP tables, and we do null-route the prefix as a whole.

Obviously if there is a /64 routed to a customer CPE device (for example), and the CPE is running RouterOS, then it is easy to crash the CPE (OOM) or cause unreachability problems (ND table exhaustion).

However, I think it is also possible to cause problems for transit routers which are not directly connected to /64 being attacked. This might be because of memory exhaustion in the IPv6 routing cache… not sure yet. I need to do some more experiments over the weekend with some test lab equipment.

idlemind · Fri Apr 13, 2018 5:40 pm

However, I think it is also possible to cause problems for transit routers which are not directly connected to /64 being attacked. This might be because of memory exhaustion in the IPv6 routing cache… not sure yet. I need to do some more experiments over the weekend with some test lab equipment.

In addition to everything else this is what I'm seeing. Just transiting a high volume of ICMPv6 traffic is enough to cause the router to reboot. I updated to 6.42rc56 this morning and no stealth fixes were contained in that release. It's trivial to reboot spam any MikroTik router w/ndpexhaust26 on the segment.

R1CH · Fri Apr 13, 2018 6:46 pm

This doesn't seem to affect Linux itself, wonder what crazy stuff Mikrotik are doing with IPv6 to introduce a vulnerability like this?

maznu · Fri Apr 13, 2018 6:49 pm

In addition to everything else this is what I'm seeing. Just transiting a high volume of ICMPv6 traffic is enough to cause the router to reboot. I updated to 6.42rc56 this morning and no stealth fixes were contained in that release. It's trivial to reboot spam any MikroTik router w/ndpexhaust26 on the segment.

I didn't think I was imagining it… but this definitely puts this into "very scary" territory. Especially if type 1, 2, 3, or 4 can trigger this.

maznu · Fri Apr 13, 2018 6:55 pm

This doesn't seem to affect Linux itself, wonder what crazy stuff Mikrotik are doing with IPv6 to introduce a vulnerability like this?

Maybe this is just a case of bad values for some IPv6 sysctl parameters — e.g. memory exhaustion by the kernel because of a value set too large. It will be interesting to hear back from MikroTik support — next week, I guess.

pe1chl · Fri Apr 13, 2018 7:00 pm

Maybe it is connection tracking. That would explain why it also effects unreachable networks.
However, it does not really explain why it would be triggered by low volume traffic. Connection tracking
should survive moderate traffic. Of course pumping a gigabit of ICMP probes like those friendly programs
do will kill the router.

maznu · Fri Apr 13, 2018 7:09 pm

Maybe it is connection tracking.

I added a "notrack" rule to ipv6 raw prerouting, and RouterOS still crashes.

Of course pumping a gigabit of ICMP probes like those friendly programs do will kill the router.

I could understand if the router dropped packets because it does not have the CPU power to push them. But crashing is not a good engineering solution.

idlemind · Fri Apr 13, 2018 9:02 pm

Maybe it is connection tracking. That would explain why it also effects unreachable networks.
However, it does not really explain why it would be triggered by low volume traffic. Connection tracking
should survive moderate traffic. Of course pumping a gigabit of ICMP probes like those friendly programs
do will kill the router.

In the case of excessive traffic the device shouldn't crash and reboot (verify with uptime). It's acceptable and understable for any excess traffic to be dropped by the router when it's reached it's limits either in software or hardware.

maznu · Mon Apr 16, 2018 3:05 pm

[Ticket#2018041622003823] opened for the ICMPv6 transit crash.

maznu · Tue Apr 17, 2018 3:49 pm

Response from Maris:

You can exhaust resources of any device by simply sending large amount of data.
Set up ipv6 firewall to protect your router.
Best way is to limit amount of accepted icmpv6 packets in IPv6 RAW firewall.

I have asked them to reconsider, as this isn't traffic destined for the router that crashes. Nor do I believe it affects ICMPv6 only.

maznu · Tue Apr 17, 2018 5:44 pm

I think all the emails back and forth were testing Maris' patience, but I am so incredibly grateful that they have listened and we have understood each other.

"We will test this scenario."

Excellent news - and good luck, MikroTik team!

maznu · Fri Jun 29, 2018 6:49 pm

Unfortunately problem is not resolved yet. I also can not give you any ETA for such fixes.
When problem will be resolved, then RouterOS release notes will include such fix description.

I guess we keep on waiting, and hoping...

idlemind · Sun Jul 01, 2018 4:29 am

Unfortunately problem is not resolved yet. I also can not give you any ETA for such fixes.
When problem will be resolved, then RouterOS release notes will include such fix description.
I guess we keep on waiting, and hoping...

Yup, I don't have any plan to use MikroTik equipment in net new projects until I see more active IPv6 feature development. Speaking with the wallet is the most effective agent of change I have available.

Even UniFi has IPv6 support that is being actively brought up to snuff. For everything else I suppose I'll have to give my money to Cisco.

Who is online