A few days ago one of our routers was hitting IPv6 neighbor cache exhaustion. The symptoms were occasional unreachability via IPv6. I pulled up Torch and found someone was actually scanning our network, probing consecutive addresses in a /64 to see if anything responded!
Dropping this traffic is easy, but remember that because this is a neighbor discovery cache exhaustion attack, you can’t protect yourself with /ipv6 firewall filter to drop on forward. You need something like:
Currently I’m unsure whether this scan (coming from a university network) is “legitimate research” or “pwned box” — as yet, no response from their abuse contact — otherwise my example might have named-and-shamed the device in question
Hi
I think you have found risk which calls for mitigation techniques in RouterOS.
Default IPv6 max-neighbor-entries table size is 8K, entries with status=“failed” stay there nearly 30 secs. Quite easy to keep such table busy.
Unless we have some method to detect next candidate and update “shitpit” ACL automagically, more ideas would be welcome.
Should it be configurable ND cache expiration time? Dropping icmpv6 responses to attacker IP in firewall when certain threshold is reached? Clearing oldest entries in failed neighbor list when next one is probed?
Any suggestions? We are currently filtering target by doing !IPv6/112 instead of allowing the entire /64.. luckily we did static addresses or this would not be possible.
I wonder if a firewall filter with limits would pick this scanning up effectively without blocking real traffic. Setting the packet count to just below the maximum you can set the “MaxNeigbhorEntries” value to without experiencing exhaustion from a ND cache perspective but not high enough to cause resource exhaustion on the device.
Possibly a rule that matches on connection-state=new and by per packet limiting?
A 3 pronged approach:
Double the cache to 16384 (/ipv6 settings set max-neighbor-entries=16384)
Apply firewall rules that identify the number of connections and on violation adds them to a source address list that can be dropped in raw for more performance
Use /126 addressing on PtP links and only use /64 on shared connections (where you need SLAAC)
A real fix would be able to control IPv6 ND traffic or at least control the cache like we do with ARP from a timing perspective. Some previous documentation and testing indicates this is difficult to exploit when the cache time is drastically lowered.
When you really need an IPv6 network to be reachable from outside without stateful filtering, it is better to limit
its size e.g. to /120 or /112 as mentioned.
“usually” most people would have a stateful firewall or allow incoming traffic only to a few addresses and this issue
would not occur.
When the network is simply open, most routers will have this problem.
Derp, now able to replicate it on my 750Gr3. Oddly, I see the number of entries in /ipv6 neighbor climbing well past my current setting of maxneighbors. That said it hasn’t seemed to hit my level of free-memory yet. Thankfully the 750Gr3 has 256 MB. It is slowly creeping downwards though. I’m definitely able to populate the table faster than it is purged by default. I suspect I’ll be able to fill all free-memory. We’ll see if the 750Gr3 locks up or what.
I’ll have to see if my connection-limiting would protect from this. I’m not sure if it’d be a stretch to perform connection tracking on a “per-tower” basis where you want to use a /64 for the benefit of SLAAC from a hardware requirements perspective.
I know this has been mitigated by default in the latest Cisco routers for a while now with cache limits per interface.
Cache limits only limit the problem of complete resource (memory) exhaustion in the router, not the “denial of service” problem on the network itself.
They were introduced when it was shown that routers could be completely DOS’ed (for IPv6 AND IPv4) using a quite slow scan.
They fix that problem, but not the problem this topic started with.
Anyone scanning sequential addresses is either an idiot or they are doing exactly what happened to you. Glad to see some discussion on it.
Thinking out loud: What if your router could reply from every non allocated address in the PD? Would be interesting to “see what happens”. Chaos, obviously. Fun.
So I thought about what I said above, and it wasn’t well thought out. I guess what I’m suggesting is that anyone on an IPv6 endpoint, that is trying to run actual IPv6 services to the internet (web, streaming, whatever), might be well served by a honeypot that responds to “everything” on the PD subnets not meant for a legitimate server. Or maybe even just respond to random addresses. You’d tie up anyone trying to discover real hosts basically “forever” and it would be pointless. Would be easy to DDoS but things are these days anyway. Maybe what I’m saying isn’t a new idea in IPv6 land.
It was still >2million more attempts (many hours) before this “attack” ceased. Maybe neighbour cache exhaustion was the purpose… never heard back from the university in question so probably not “real” research.
They’re doing a sort of enumerative scan of IPv6 address space, depth-first, which results in about 100pps of IPv6 traffic from them. That soon fills up a neighbour cache on a smaller device (even if you’ve set it to 100k+ entries), and pretty soon afterwards your little device loses its connectivity (cue lots of Nagios alerts).
I’ve contacted the abuse address and registered address in WHOIS, and reached out to one of the researchers on Twitter, and suggested they add some “ethical considerations” to section 4 of their paper — and think about potential impact upon targeted networks.
Meanwhile is there anything MikroTik could do in RouterOS to make it easier to flush out “failed” or “noarp” entries in the IPv6 neighbour cache? Or at least let us adjust the timeout for the failed entries to be much quicker than the successful ones?
Those “researchers” are the worst. With a hacker or prober (those guys that keep a list of available addresses/ports for hackers)
you at least know they do it to cause (indirect) damage. The researchers however are claiming they are doing it for a good
purpose, yet they are causing the same damage.
I think starting this kind of research immediately disqualifies them as a credible scientist. One that would first investigate the situation,
define a research project, and find out about possible unwanted effects before even starting the actual operation.
Seems like MikroTik needs a ND policer like everyone else implemented in 2012 or earlier. That said it would constitute IPv6 feature work and we know how unlikely that is at MikroTik.
Maybe a high severity CVE is needed to get MikroTik’s attention to effectively mitigate this.
Or, take my approach which is growing I’m the community, stop purchasing new MikroTik equipment until they begin to take IPv6 seriously.
We spread probes as evenly as possible across routed prefixes, and shuffle targets within each prefix.
Regards,
6Gen Team
If they’re actually spreading things out evenly across routed prefixes, then to get the quantity of packets we were receiving they are scanning the Internet at somewhere around 20-40Mpps.
“mikrotik.com has IPv6 address 2a02:610:7501:1000::2”
It’s only a matter of time before the researchers hit that…
But of course that is not likely to cause a problem. Especially with an address like that.
The issue only occurs when there are large (/64) subnets behind a router that does not do any filtering.
Hosting usually is on small subnets (/112) and often there will be a firewall on the router that only passes
traffic to a few wanted ports on some specific addresses. When you see addresses like ::2 it points
to a situation where the admin actually considered this.
