Why source-based blackhole instead of firewall drop

RouterOS General
1

EDIT : There was an initial flaw in my testing, jump to this post https://forum.mikrotik.com/viewtopic.php?f=2&t=114664#p599689 which shows that using the firewall RAW DROP has the same efficiency as source-based blackhole + route cache disabled - thanks mrz & cha0s


Disclaimer:
I am by no means a network expert and this post is intended for those with smaller public networks and / or beginners.
I welcome any experts recommendation if any of my information below is incorrect

Background
I decided to post this after a post I did in this topic - http://forum.mikrotik.com/t/72-core-ccr/88532/1

Firewalling can be bad mmmkay
Firewalling is great and everyone should use it, but try to do this as close to the edge (as close to the customer / server) of your network as possible.
Do not firewall on your border routers (your routers between your network and the internet) unless you absolutely have to (eg: NAT).
When making a decision to firewall, expect there to be a throughput impact in high packet-per-second (pps) attacks.
The more firewall rules you have, the more you affect the throughput.This is because every packet has to be matched against the firewall rules until one condition is met.

This is shown clearly in Mikrotik’s own performance tests, here is an example from the CCR-1072 router tests:

Without any firewalling, 44Gbits and 86 million pps are possible through the CCR router (64byte).
Simply add 25 firewall rules and your throughput drops dramatically to 3Gbits and 5.8 million pps.
That is a 93% drop in throughput by adding firewall rules to your router.

Using the CCR-1072 as an example, If you have a 5Gb attack on your network and you are firewalling on your border router, the moment 3Gbits of traffic is exceeded in the attack, everything behind your border router is affected. Regardless of having a 10Gb link.

If however you have no firewall on your border router and rather firewall close to the edge, the CCR will withstand this attack and the only part of your network that will be affected is the area behind the firewall on the edge of your network. This means the rest of your network behind the border router is still accessible. This will limit the impact of what is affected by the attack on your network.

The opposite also applies, if an attack is initiated from an edge device on your network, you can stop it as close to the device as possible to prevent it affecting the rest of your network.

Cool story bro, but how do I now block traffic at the border of the network?
There are a number of ways like Remote Trigger Blackhole with your ISP, but this blackholes the target IP on the network upstream.
You essentially giving an attacker what he wants, a host that’s down. You are saving the rest of your network at the expense of the target IP being inaccessible.
Sometimes this is absolutely necessary, but sometimes you might want to just block the source addresses attacking the target to keep the target up to legitimate traffic.
In this case, a source-based blackhole routing can be used to achieve blocking, instead of a firewall rules.
Keep in mind that routes do not slow down the performance of throughput on your router as much as firewall rules do.

Here is an example I did to illistrate how effective source-based blackholes can be over a Firewall Drop rule.(a youtube video):
https://youtu.be/SHjBbbF8Ek4

I setup a test server running hping3 in flood mode with 64b packets to simulate a DDoS attack.
The test server sends packets to the mikrotik router, which then routes the traffic to the target IP.

The initial attack puts the CHR at 40% CPU usage, when the firewall DROP rule is activated, the CPU averages out around 18% usage, but you can still see the traffic being received on the interface (both in RX bandwidth and in RX packets-per-second.) Yes the drop rule prevents the traffic from reaching the target IP, but doesn’t solve the load on your mikrotik or the traffic on the interface.

I then disable the firewall rule, which allows the DDoS traffic to flow freely again, except this time I enable the source-based blackhole route for the DDoS source IP address.
Almost immediately you can see it perform the exact same action as the firewall DROP rule, except after a number of seconds you will see the traffic drop off entirely from the interface and the mikrotik CPU return to 0% usage.

One thing to note when doing the above is having RP Filter set to loose (or strict, but strict can cause some routing issues in asymmetric routing).
Loose RP Filter checks the packet and forwards it if there is a route entry for the source IP of the incoming packet in the router FIB. If the router
does not have an FIB entry for the source IP address, or if the entry points to blackhole, the Reverse Path Forwarding (RPF) check fails, and the
packet is dropped

A source-based blackhole achieves the same result as a firewall drop, but more efficiently and without impacting the throughput of your device.
You can also use source-based blackholes in conjunction with your normal firewall rules as a means of dropping IP traffic.

There are many tools out there (wanguard, fastnetmon, etc) which automate the detection of a DDoS and can call the mikrotik API (or through BGP) to automatically create these source-based blackhole rules.

Regards,
Dave

2

Nice. What about to drop in raw table according to the blacklist while an attacking ip address was identified by firewall rules and put on the list?

3

I think what this post really says is DON’T USE A MIKROTIK FOR FIREWALLING.

4

By this I assume you mean what is referred to in this post - http://forum.mikrotik.com/viewtopic.php?f=2&t=54607&start=50#p463532 which refer’s to this code:

/ip firewall filter
add action=jump chain=forward connection-state=new jump-target=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddosed address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddoser address-list-timeout=10m chain=detect-ddos
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=ddosed new-routing-mark=ddoser-route-mark passthrough=no src-address-list=ddoser
/ip route
add distance=1 routing-mark=ddoser-route-mark type=blackhole

There is still firewall interaction as the packets need to be marked here and it yields the same results as a firewall drop policy.
Here’s the test - https://youtu.be/9xoO_tUBYEA

5

No. I do not refer to mangling in prerouting and dropping connection by marks but to direct packet drop in raw firewall table without any other interactions.

6

While I agree that having lots of “/ip firewall filter” entries will slow your router down (CPU has to go through each rule in sequence), using an address-list should consolidate a lot of blocked IPs into one firewall rule. I believe address-list is implemented as the Linux kernel’s IPSET feature for iptables, and so it will work in pretty much the same amount of time even if there are thousands of entries in the list (it’s more like a hash-table than a linear list).

We have no problem blocking traffic at our provider edge using this technique. I gave a talk about this at the UK MUM yesterday, actually. We use an /ip route rule, packet marking in /ip firewall mangle, and an /ip firewall address-list to move abusive traffic away from our customers and to special VMs which are honeypots. However, if you just want to block traffic, it’d be fine to use a filter instead of /ip firewall mangle.

PDF of Presentation: https://faelix.link/mum16

MUM Video: https://faelix.link/mum16video

7

100% correct. I’m more illistrating that the moment firewalling is involved in anyway with the border router, it will cause CPU load under attack as all the packets still need to be filtered.
The source-based blackhole seems more efficient at achieving the same result.

Ah yes I see what you mean, I did another test with running the hping3 on a 3 minute test and took a graph from the network switch.
One for a normal firewall drop rule, one for source-based blackhole and one with a firewall RAW table drop rule.

The firewall raw table drop rule worked very slightly better than a normal firewall drop rule, with the CPU sitting at 14% instead of 18%.


We have no problem blocking traffic at our provider edge using this technique. I gave a talk about this at the UK MUM yesterday, actually. We use an /ip route rule, packet marking in /ip firewall mangle, and an /ip firewall address-list to move abusive traffic away from our customers and to special VMs which are honeypots. However, if you just want to block traffic, it’d be fine to use a filter instead of /ip firewall mangle.

Do you not lose fastpath by doing this though?

8

Hard data prooves… I have expected better results. Good job.

9

Just to be sure I’ve tested a few different attack variations with the initial tests being syn flood attacks.
I’ve now tested:

  • udp flood attacks
  • syn attack with random source and spoofed source
  • udp attack with random source and spoofed source
  • TCP Connect attack with 200 000 pps directed to an open port

All show the exact same great result with using RP Filter (loose) and source-based blackhole.
It’s basically tricking the RP Filter into dropping the source address.

10

Yes, but in our use case that isn’t something I’m worried about. I need to use an address-list (so lose fastpath) so that I can redirect abusive traffic:

  1. to display a “stop typing your password wrong!” message to customers failing to log themselves in to websites or whatever
  2. to direct hacking traffic away from customer servers
  3. to take the attack traffic to our honeypot for further analysis so that we can see what things they’re scanning for (to make more rules to detect them, etc)

For us it’s better that our edge CCRs took the CPU hit and used ~30W more power than, say, a rack of servers hit a loadavg of 60+ due to attack traffic. With only one mangle rule in the firewall, we’re not going to choke the CPU as much as the “25 filter rules” statistics quoted on the routerboard.com website. And so long as we’ve chosen CCRs that will cope until our upstream links are saturated, that’s fine - after that, it’s not the CCRs’ fault :wink:

11

We had a similar use case on our side, but a slightly different approach (as we don’t want to lose fastpath).

  1. In our code of our portals we give the “stop trying your password is wrong” messages and after X brute-force attempts the code on the portal calls the mikrotik API to blackhole the IP for XX period of time. It starts with a few minutes and then removes the blackhole. Each time it picked up subsequent brute-force attempts it increases the time they blackholed.

2 & 3 - We do this using Wanguard. It detects the hacking / attack traffic and informs the filter (honeypot in your case) to inject a route into the mikrotik to redirect this traffic to itself where it can be captured for analysis. This is also filtered and the clean traffic can then be injected back into the network.

12

Ah interesting. How does the Wanguard get the traffic in the first place to determine what is an attack? Is it working off NetFlow/IPFIX data? Or is it connected to a mirror port or something?

In our case, our main abuse detection - besides fastnetmon etc - is because the end servers are running some inspection (e.g. mod_security, fail2ban, etc).

13

It supports both Flow Sensors or Packet Sensor. We use the Packet Sensor (create a mirror port on your switch) as it can pick up an attack in less than 1 second.
A flow sensor can detect the an attack in “flow export time + 5 seconds”.

This explains it better than I can : https://www.andrisoft.com/software/wanguard/ddos-mitigation-protection

In our case, our main abuse detection - besides fastnetmon etc - is because the end servers are running some inspection (e.g. mod_security, fail2ban, etc).

You can still use these, just have them trigger a script that speaks to the mikrotik API, for example:

/usr/bin/blackhole.php

It calls the mikrotik api and then stores the block time in a db / flat file.
then have a cron that runs every minute that looks in the db / flat file for items which are “expired based on time” and calls the CCR to remove the blackhole.

14

Oh, our fail2ban/etc boxes don’t have access to the edge routers’ API. They send attack information to a process we run which stores IP reputation data, and that decides whether to block addresses at the edge. It does that with an address-list entry, with an appropriate timer. Again, like yours, we block for longer periods of time for longer attacks. And our code expands the blocked range from a /32 gradually up towards a /24 if neighbouring IPs are implicated or are attacking other hosts in our network.

More details in the PDF of the talk I linked.

15

Can you please share the code for this source based blackhole? From above I can see that I have to packet mark so filter is still involved?!

16

Two different people debating different approaches.

Murmaider’s using a loose RP filter and route injection to trick the router into dropping packets.

I’m using a combination of /ip route rule with /ip firewall mangle and an /ip firewall address-list to redirect traffic from our border routers (i.e. as soon as it comes in from our upstream providers) straight to a honeypot.

17

For the mikrotik, enable RP Filter under IP => Settings (set to loose)
Then to add a blackhole use:

/ip route
add distance=1 dst-address=<source address> type=blackhole

Source address can be an individual ip or a network range.

18

I think the last question I have about this for Murmaider is:

Will your RP-based blackhole approach work where there are multiple valid routes (learned by BGP) to external addresses, but only one is “active” in the RouterOS routing table? Because we see traffic come in on interfaces which aren’t necessarily the best path that we would have chosen for sending traffic out. Presumably this is why you’re “firewalling” as close to the destination as possible, whereas we’re redirecting attack traffic as close as possible to its ingress point to our network (which means on routers where there could be multiple possible routes available).

19

With RP Filter set to loose it should. Loose RP Filter checks the packet and forwards it if there is a route entry for the source IP of the incoming packet in the router FIB. This includes a default route or any route you have learned via BGP. If however an entry for the source ip points to blackhole, the Reverse Path Forwarding (RPF) check fails, and the packet is dropped.

This approach has been around for a while:
http://packetlife.net/blog/2010/aug/23/source-based-rtbh/
http://packetlife.net/blog/2009/jul/20/urpf-access-layer-deter-ddos-attacks/
https://anetworkerblog.com/2009/08/25/rtbf/

20

Ahhhh, that’s the bit I didn’t grok. I’d only ever encountered reverse path filtering in a strict sense as part of a poor-man’s BCP38. Ok, that’s interesting.

Thanks - I learned something today.