Blacklist Filter update script

RouterOS Scripting
1

I’ve started development of the replacement service. Please read the Development topic here:
https://forum.mikrotik.com/viewtopic.php?f=9&t=136666

Here is a form to fill out if you want to be notified when the new service goes live.
https://goo.gl/forms/UQMYqKJ54E0iV35l2








New blacklist system! (7-July-2017)

RouterOS version 6.36 or higher is now required.

Okay guys, I’m posting my first RC for the new system. To simplify things, I’m only posting an Installer / Updater script.
This will install the new blacklist update script, the config script, and the schedulers. You will end up with the following:

  • Scripts
  • blacklistUpdate - the primary script for checking for the list and installing it
  • blacklistUpdate.conf - Configuration for the script. the auto-script-update will not touch this.
  • blacklistScriptUpdater - this is the auto-upgrade script. I recommend calling it once a day to make sure you are current.


  • Scheduler
  • blacklistUpdate - this will run hourly, checking to see if a new list is available. Updating ONLY if the list is new
  • blacklistUpdateOnBoot - This is for loading the current list when the router boots

the list name has changed You will need to update your rules to use “intrusBL” instead of “dynamicBlacklist”.

Updates are now done in place. Old entries have their expiration lowered to 30 minutes so that they expire soon. This replaces the remove process and lets them expire naturally. Current retirees are updated to 25 hours. New entries are added and set to 25 hours.

Checking for updates is done via DNS. a quick lookup to my DNS server (checking 127.0.0.3) returns the current serial number of the list. If the serial matches what is currently installed, no update is done. If the serial is higher, the new list is downloaded and installed.

I look forward to your feedback

# Intrus Technologies Blacklist Installer
# © 2017 David Joyce, Intrus Technologies
# 
# Version 2.0.5
#
# This is used to install and update the blacklist importer script
# as well as the scheduler tasks used to update the address-lists
# and the scheduler task used to update the scripts
#
# These are offered free of charge to the MikroTik community. No warranty is expressed or implied.
# I am not responsible for any loss of data, time, money, access, or anything else.  Use at your own risk.
#
# P.S. Changing the script names will break things. Badly.

:do {
    :local currentScriptVersion [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.2 ]
    :put "Installing blacklistUpdate script version: $currentScriptVersion";
    :local sourceServer "https://mikrotikfilters.com/";
    :local sourceServerPort "6501";
    :local scriptName "blInstaller.rsc";
    :put "Downloading update script...";
    :do {
        /tool fetch url="$sourceServer$scriptName" mode=https port=$sourceServerPort dst-path="/$scriptName";
    } on-error={
        :put "Error. Download failed";
    }
   :put "Importing update script...";
    :do {
        /import "$scriptName";
    } on-error={
        :put "import failed. unknown error.";
    }
    :put "Removing update script...";
    :do {
        /file remove "$scriptName";
    } on-error={}
    :put "Update Complete.";
}



  • Version History
  • 2.0.5 Released
  • Script and server changes to allow blacklisted IP’s to still access the list
  • 2.0.4 Released
  • auto-update for script is default disabled, can be enabled in the config
  • added global “blScriptUpdate” to the config to enable/disable script updating
  • 2.0.3 Released
  • Script Updater cleanup
  • Installer now have full permissions (ros bug)
  • 2.0.2.1 and 2.0.2.2 Released
  • minor typos fixed
  • new Auto-Update script is now installed
    • Auto-updater can be run manually, or on a daily schedule
  • new cleaner installer, can now be copy/pasted to the console
  • 2.0.2 Released
  • Fixed a logging typo
  • changed the auto-updater to stop removing the config if run twice
  • Better cleanup of globals used as functions
  • started framework for checking available disk space before downloading
  • 2.0.1 Released
  • improved URL Encoding function
  • much simpler CHR system ID detection
  • changed Script Version to global variable (prep for auto-script update)

Please remember to give a positive rating is you like and use this service

2

This is an archived post. Please refer to the post above.

I’ve gone ahead and started publishing my dynamic filter list for RouterOS 6.x. My server generates the list each night after collecting data on all known botnets, C&C server, and spammers. Currently the list runs about 3k entries, so it may not work well on low end routers. Here is the script to update the list, as well as my personal firewall rules. As always, adjust them to fit your needs.

Client Statistics can now be found here: https://mikrotikfilters.com/blstats.php

Feedback and suggestions are always welcome.



The list is updated every 6 hours. 00:00, 06:00, 12:00, 18:00. PLEASE DO NOT RUN EVERY MINUTE. Running the script every minute is a waste of bandwidth and puts undue strain on the NAND. I recommend updating every 12 to 24 hours.

The address-list entries are now Dynamic with a 25 hour timeout. This will cut the number of writes to NAND down dramatically.

The script only needs Read, Write, & Test permissions. Name the script “updateBlacklist”. Removing the variables sent will prevent the server from sending the updates. I use them for accounting so I can keep track of the number of requests and the amount of bandwidth used.


Don’t forget the schedule:



And, if you are interested, here are my filter rules:

/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2

/ip firewall address-list
add address=172.16.0.0/12 list=PrivateIPs
add address=10.0.0.0/8 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs

If you are using my script and Scheduler names, you can use this script to auto-update:

/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
/system script remove updateBlacklist
/system scheduler remove updateBlacklist
/system scheduler remove updateBlacklistOnBoot
/import updateBlacklist.rsc;
/file remove updateBlacklist.rsc;

Change Log:

  • Version 2017-07-05a
  • Changed logging - Now only mutes the “firewall”, no longer mutes all “info”
  • Changed default path to / instead of /disk1/ - current issue with CCR using microSD
  • More accurate logging of what is happening
  • minor text formatting changes
  • Now sets two globals - blSerial and blVersion (for future auto-update script)
3

update #2
added model specific support. smaller list for models that can’t handle the full list. larger lists for 1GB+ models.

4

update #3
better model / memory detection

5

Update 4
server side - prevent generating duplicate “add” lines in the script.

6

Thank you for providing this to the community!

7

It’s the least I could do in return for all the help that the comunity has given me over the last two years.

8

First of all, thanks for this script!
It was something I wanted to implement for ages now, but never had the time to do so :slight_smile:

At the moment it works by using your server.

Could you post your server-side code to be able to run this without needing to access your server?

Thanks :slight_smile:

9

Unfortunately, I can not. Much of the list is generated by my own routers. Just under 50 of them currently. As my servers and routers around the world get attacked, probed and spammed, they block the addresses and then send the addresses back to my server to be compiled into one list, when is then sent back out to the routers.

10

I see. I though you were using some pre-made lists and just convert them to mikrotik commands.

Do you know any public blacklists I could use?

Btw, how often do you run this script?
If I am not mistaken this method keeps writing on the NAND storage on each fetch?
Any ideas how we could make this run completely in RAM so the NAND doesn’t wear out?

11

you can check out dshield.org for a premade list.
My server collects the banned IP’s 24/7 and publish the list at 3am PST.
As for not writing to NAND - I don’t know of any way to prevent that. For a lower end unit with little storage, maybe only run the script once a month. For my own routers, I understand and accept that it will ware down the NAND - just the cost of security I suppose. Even replacing the routers every other year, they are still cheaper than a Cisco.

12

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.

13

Updated the script, moved to https.

14

I’m added limits on the server to stop routers from requesting the list too often. Once router was downloading the list every minute. Over 1400 times a day - just over 320M (the list averages 250kb). Once every 24 hours is enough.

15

added scheduler code.

16

This is awesome! Thanks!!

17

/tool fetch url=“https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory” mode=http

Care to elaborate why those variables has any significance to you?

18

Ah well.

403 forbidden errors in any case…

Just FYI - but it will be MUCH better (both on you, routers, management, and resources) to simply distribute lists of IPs using private ASN numbers and multi-hop BGP…

People peering with your BGP feed can then just get the updates as you push them, and blackhole the routes.

Much, much more efficient than hammering routers with 3K firewall rules :laughing:

Just a thought…

19

On the server side, it allows the server to select a list optimized for your unit.
Model lets it know how much CPU power you have, memory for how big of a list, and version for quirks in the script.

20

I don’t use BGP because most users that will want this don’t know how to setup BGP to start with.
And yes, server gives a 403 is you try to access it directly, instead of a RouterOS device pulling it.