Community discussions

MikroTik App
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Blacklist Filter update script

Wed Jul 22, 2015 9:44 am



I've started development of the replacement service. Please read the Development topic here:
viewtopic.php?f=9&t=136666

Here is a form to fill out if you want to be notified when the new service goes live.
https://goo.gl/forms/UQMYqKJ54E0iV35l2








New blacklist system! (7-July-2017)

RouterOS version 6.36 or higher is now required.

Okay guys, I'm posting my first RC for the new system. To simplify things, I'm only posting an Installer / Updater script.
This will install the new blacklist update script, the config script, and the schedulers. You will end up with the following:
  • Scripts
  • blacklistUpdate - the primary script for checking for the list and installing it
  • blacklistUpdate.conf - Configuration for the script. the auto-script-update will not touch this.
  • blacklistScriptUpdater - this is the auto-upgrade script. I recommend calling it once a day to make sure you are current.
  • Scheduler
  • blacklistUpdate - this will run hourly, checking to see if a new list is available. Updating ONLY if the list is new
  • blacklistUpdateOnBoot - This is for loading the current list when the router boots
the list name has changed You will need to update your rules to use "intrusBL" instead of "dynamicBlacklist".

Updates are now done in place. Old entries have their expiration lowered to 30 minutes so that they expire soon. This replaces the remove process and lets them expire naturally. Current retirees are updated to 25 hours. New entries are added and set to 25 hours.

Checking for updates is done via DNS. a quick lookup to my DNS server (checking 127.0.0.3) returns the current serial number of the list. If the serial matches what is currently installed, no update is done. If the serial is higher, the new list is downloaded and installed.

I look forward to your feedback
# Intrus Technologies Blacklist Installer
# © 2017 David Joyce, Intrus Technologies
# 
# Version 2.0.5
#
# This is used to install and update the blacklist importer script
# as well as the scheduler tasks used to update the address-lists
# and the scheduler task used to update the scripts
#
# These are offered free of charge to the MikroTik community. No warranty is expressed or implied.
# I am not responsible for any loss of data, time, money, access, or anything else.  Use at your own risk.
#
# P.S. Changing the script names will break things. Badly.

:do {
    :local currentScriptVersion [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.2 ]
    :put "Installing blacklistUpdate script version: $currentScriptVersion";
    :local sourceServer "https://mikrotikfilters.com/";
    :local sourceServerPort "6501";
    :local scriptName "blInstaller.rsc";
    :put "Downloading update script...";
    :do {
        /tool fetch url="$sourceServer$scriptName" mode=https port=$sourceServerPort dst-path="/$scriptName";
    } on-error={
        :put "Error. Download failed";
    }
   :put "Importing update script...";
    :do {
        /import "$scriptName";
    } on-error={
        :put "import failed. unknown error.";
    }
    :put "Removing update script...";
    :do {
        /file remove "$scriptName";
    } on-error={}
    :put "Update Complete.";
}

  • Version History
  • 2.0.5 Released
  • Script and server changes to allow blacklisted IP's to still access the list
  • 2.0.4 Released
  • auto-update for script is default disabled, can be enabled in the config
  • added global "blScriptUpdate" to the config to enable/disable script updating
  • 2.0.3 Released
  • Script Updater cleanup
  • Installer now have full permissions (ros bug)
  • 2.0.2.1 and 2.0.2.2 Released
  • minor typos fixed
  • new Auto-Update script is now installed
  • - Auto-updater can be run manually, or on a daily schedule
  • new cleaner installer, can now be copy/pasted to the console
  • 2.0.2 Released
  • Fixed a logging typo
  • changed the auto-updater to stop removing the config if run twice
  • Better cleanup of globals used as functions
  • started framework for checking available disk space before downloading
  • 2.0.1 Released
  • improved URL Encoding function
  • much simpler CHR system ID detection
  • changed Script Version to global variable (prep for auto-script update)
Please remember to give a positive rating is you like and use this service
Last edited by IntrusDave on Sat Aug 04, 2018 11:22 pm, edited 54 times in total.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 23, 2015 3:35 am

This is an archived post. Please refer to the post above.

I've gone ahead and started publishing my dynamic filter list for RouterOS 6.x. My server generates the list each night after collecting data on all known botnets, C&C server, and spammers. Currently the list runs about 3k entries, so it may not work well on low end routers. Here is the script to update the list, as well as my personal firewall rules. As always, adjust them to fit your needs.  

Client Statistics can now be found here: https://mikrotikfilters.com/blstats.php

Feedback and suggestions are always welcome.



The list is updated every 6 hours. 00:00, 06:00, 12:00, 18:00. PLEASE DO NOT RUN EVERY MINUTE. Running the script every minute is a waste of bandwidth and puts undue strain on the NAND. I recommend updating every 12 to 24 hours.

The address-list entries are now Dynamic with a 25 hour timeout. This will cut the number of writes to NAND down dramatically.

The script only needs Read, Write, & Test permissions. Name the script "updateBlacklist". Removing the variables sent will prevent the server from sending the updates. I use them for accounting so I can keep track of the number of requests and the amount of bandwidth used.


Don't forget the schedule:



And, if you are interested, here are my filter rules:
/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2

/ip firewall address-list
add address=172.16.0.0/12 list=PrivateIPs
add address=10.0.0.0/8 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs

If you are using my script and Scheduler names, you can use this script to auto-update:
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
/system script remove updateBlacklist
/system scheduler remove updateBlacklist
/system scheduler remove updateBlacklistOnBoot
/import updateBlacklist.rsc;
/file remove updateBlacklist.rsc;
Change Log:
  • Version 2017-07-05a
  • Changed logging - Now only mutes the "firewall", no longer mutes all "info"
  • Changed default path to / instead of /disk1/ - current issue with CCR using microSD
  • More accurate logging of what is happening
  • minor text formatting changes
  • Now sets two globals - blSerial and blVersion (for future auto-update script)
Last edited by IntrusDave on Wed Jul 19, 2017 5:42 am, edited 2 times in total.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 23, 2015 5:35 am

update #2
added model specific support. smaller list for models that can't handle the full list. larger lists for 1GB+ models.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 24, 2015 5:04 am

update #3
better model / memory detection
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 25, 2015 8:16 pm

Update 4
server side - prevent generating duplicate "add" lines in the script.
 
RackKing
Member
Member
Posts: 380
Joined: Wed Oct 09, 2013 1:59 pm

Re: Blacklist Filter update script

Sun Jul 26, 2015 4:22 pm

Thank you for providing this to the community!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 26, 2015 10:05 pm

It's the least I could do in return for all the help that the comunity has given me over the last two years.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:03 pm

First of all, thanks for this script!
It was something I wanted to implement for ages now, but never had the time to do so :)

At the moment it works by using your server.

Could you post your server-side code to be able to run this without needing to access your server?

Thanks :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:09 pm

Unfortunately, I can not. Much of the list is generated by my own routers. Just under 50 of them currently. As my servers and routers around the world get attacked, probed and spammed, they block the addresses and then send the addresses back to my server to be compiled into one list, when is then sent back out to the routers.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:12 pm

I see. I though you were using some pre-made lists and just convert them to mikrotik commands.

Do you know any public blacklists I could use?

Btw, how often do you run this script?
If I am not mistaken this method keeps writing on the NAND storage on each fetch?
Any ideas how we could make this run completely in RAM so the NAND doesn't wear out?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:17 pm

you can check out dshield.org for a premade list.
My server collects the banned IP's 24/7 and publish the list at 3am PST.
As for not writing to NAND - I don't know of any way to prevent that. For a lower end unit with little storage, maybe only run the script once a month. For my own routers, I understand and accept that it will ware down the NAND - just the cost of security I suppose. Even replacing the routers every other year, they are still cheaper than a Cisco.
 
User avatar
mmv
Trainer
Trainer
Posts: 65
Joined: Wed Feb 24, 2010 5:03 pm
Location: Moscow, Russia
Contact:

Re: Blacklist Filter update script

Thu Aug 13, 2015 8:38 pm

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Aug 14, 2015 6:26 pm

Updated the script, moved to https.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 17, 2015 6:09 am

I'm added limits on the server to stop routers from requesting the list too often. Once router was downloading the list every minute. Over 1400 times a day - just over 320M (the list averages 250kb). Once every 24 hours is enough.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Sep 03, 2015 8:26 pm

added scheduler code.
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Fri Oct 09, 2015 8:18 am

This is awesome! Thanks!!
 
savage
Forum Guru
Forum Guru
Posts: 1262
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: Blacklist Filter update script

Fri Oct 09, 2015 2:23 pm

/tool fetch url="https://mikrotik.intrustech.com/downloa ... ry=$memory" mode=http

Care to elaborate why those variables has any significance to you?
 
savage
Forum Guru
Forum Guru
Posts: 1262
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: Blacklist Filter update script

Fri Oct 09, 2015 2:30 pm

Ah well.

403 forbidden errors in any case...

Just FYI - but it will be MUCH better (both on you, routers, management, and resources) to simply distribute lists of IPs using private ASN numbers and multi-hop BGP...

People peering with your BGP feed can then just get the updates as you push them, and blackhole the routes.

Much, much more efficient than hammering routers with 3K firewall rules :lol:

Just a thought...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Oct 09, 2015 5:01 pm

/tool fetch url="https://mikrotik.intrustech.com/downloa ... ry=$memory" mode=http

Care to elaborate why those variables has any significance to you?
On the server side, it allows the server to select a list optimized for your unit.
Model lets it know how much CPU power you have, memory for how big of a list, and version for quirks in the script.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Oct 09, 2015 5:03 pm

Ah well.

403 forbidden errors in any case...

Just FYI - but it will be MUCH better (both on you, routers, management, and resources) to simply distribute lists of IPs using private ASN numbers and multi-hop BGP...

People peering with your BGP feed can then just get the updates as you push them, and blackhole the routes.

Much, much more efficient than hammering routers with 3K firewall rules :lol:

Just a thought...
I don't use BGP because most users that will want this don't know how to setup BGP to start with.
And yes, server gives a 403 is you try to access it directly, instead of a RouterOS device pulling it.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 19, 2015 7:01 am

Updated the server side to create dynamic entries so that they are not written to NAND. This has a downside, the list will need to be reloaded on reboot. The entries now also expire after 48 hours.
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Sun Feb 07, 2016 7:00 pm

:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "intrusBlacklist" ) do={ /ip firewall address-list remove $i } }
Can be simplified to
/ip firewall address-list remove [/ip firewall address-list find comment = "intrusBlacklist"]
Should actually increase the efficiency.
 
sachmonz
just joined
Posts: 11
Joined: Mon Feb 22, 2016 9:44 am

Re: Blacklist Filter update script

Mon Apr 18, 2016 11:08 am

Many thanks for this.

On the latest Router OS I can see it's created the entries under Address Lists, but on the filter rules I don't see any new filter rules showing up that reference the address list called blacklist as created by the script.

Model is MikroTik RB951G-2HND

Cheers
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Blacklist Filter update script

Mon Apr 18, 2016 11:47 am

/ip firewall filter
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
Where I have already see part of that rules on exact order? :lol:
Ah:
http://forum.mikrotik.com/viewtopic.php?f=9&t=83387

Sort the rules for efficency (simply drop if coming from blocked list, not first check malformed packet then drop)
/ip firewall filter
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 18, 2016 6:11 pm

good set of rules.
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Thu May 21, 2015 11:14 am

Re: Blacklist Filter update script

Wed Apr 20, 2016 12:20 pm

Where I have already see part of that rules on exact order? :lol:
Ah:
http://forum.mikrotik.com/viewtopic.php?f=9&t=83387

Sort the rules for efficency (simply drop if coming from blocked list, not first check malformed packet then drop)
/ip firewall filter
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
Thanks for this ruleset.
Is it useful to use to put the input and forward chain into this same set of rules?

What i am also interested, is it possible to say how much performance influance has a rule?
I have a home use setup with an rb2011 and so i think i will never reach the limit (2-5 devices running @ same time)
It only interests me ;)

PS: Cant give karma, dont know why
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Blacklist Filter update script

Wed Apr 20, 2016 10:24 pm

Thanks for this ruleset.
Is it useful to use to put the input and forward chain into this same set of rules?

What i am also interested, is it possible to say how much performance influance has a rule?
I have a home use setup with an rb2011 and so i think i will never reach the limit (2-5 devices running @ same time)
It only interests me ;)
on s.o.h.o. or at home as really no impact, except if layer-7 are used for filter traffic
on big amount of traffic count the power of the RouterBOARD
In my office (I'm a WISP) one 2011-RM is sufficient for 1Gbit internal traffic for 12 PC, 1 Server, 1 NAS, Public Local HotSpot (one Metal 2 out of the door) printers, VoIP systems (20 phones / 4 lines), failover between main line (20Mbps/20Mbps) and one ADSL line (7Mbps up / 380kbps down) and the firewall have near 80 rules on filter, 40 on NAT and 20 on mangle, hotspot service active withe relative rules, EoIP tunnels, DNS and address-list filter and web-proxy.

(DNS, address-list filter and web-proxy are used for remove all the ADs on web pages, https or not...)

I NOT use NAND,r USB or microSD for user-manager storage, I use one separate PC with RouterOS
Last edited by rextended on Mon Mar 20, 2023 6:33 pm, edited 1 time in total.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Apr 20, 2016 11:34 pm

These are the same filters I use, along with the blacklist that ranges from 2000~5000 IP's and subnets.
If you are interested, You can look at the stats for the routers that I monitor directly. The demo key only allows access to 15 of the routers. I will leave it open for a day or so.

https://mikrotik.intrustech.com/status.php?key=demo
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Thu May 21, 2015 11:14 am

Re: Blacklist Filter update script

Thu Apr 21, 2016 10:52 am

Thanks for this information!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat May 28, 2016 9:21 am

I've switched all my servers over to SSL only. Here is an updated script with https enabled.
I have new filter rules coming soon using the raw prerouting , once 6.36 is released.
:log warning "Downloading current Blacklist for this model";
:local model [/system resource get board-name]
:local version  [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
/tool fetch url="https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=https dst-path="/currentBlacklist.rsc";

# Disable Logging so each add and remove isn't in the system log, we turn it back on at the end
:log warning "Disabling system Logging";
/system logging disable 0

# Find and Remove the old filters
:log warning "Removing previous Blacklist Address-List entries";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "intrusBlacklist" ) do={ /ip firewall address-list remove $i } }

# Import the new filters
:log warning "Importing current Blacklist";
/import file-name=/currentBlacklist.rsc

# Delete the import file
:log warning "Removing temp files";
/file remove currentBlacklist.rsc

# Enable the Logging
:log warning "Enabling system logging, all done.";
/system logging enable 0
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26287
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Blacklist Filter update script

Fri Jun 17, 2016 5:32 pm

Great work! 
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Jul 01, 2016 12:20 pm

I've switched all my servers over to SSL only. Here is an updated script with https enabled.
I have new filter rules coming soon using the raw prerouting , once 6.36 is released.
:log warning "Downloading current Blacklist for this model";
:local model [/system resource get board-name]
:local version  [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
/tool fetch url="https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=https dst-path="/currentBlacklist.rsc";

# Disable Logging so each add and remove isn't in the system log, we turn it back on at the end
:log warning "Disabling system Logging";
/system logging disable 0

# Find and Remove the old filters
:log warning "Removing previous Blacklist Address-List entries";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "intrusBlacklist" ) do={ /ip firewall address-list remove $i } }

# Import the new filters
:log warning "Importing current Blacklist";
/import file-name=/currentBlacklist.rsc

# Delete the import file
:log warning "Removing temp files";
/file remove currentBlacklist.rsc

# Enable the Logging
:log warning "Enabling system logging, all done.";
/system logging enable 0
is this lastest working version ?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:47 pm

It does. However, it appears that if you are on older versions of RouterOS, you will need to disable https. the current RC fixed the issue.

Personally, I would recommend upgrading to the RC so you can take advantage of the RAW filter.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:48 pm

Im using V6.35.4 and I will try this. also I found this while searching http://joshaven.com/resources/tricks/mi ... ress-list/ and its seems working good.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:52 pm

Replace this: 
/tool fetch url="https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=https dst-path="/currentBlacklist.rsc";
with this:
/tool fetch url="http://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=http dst-path="/currentBlacklist.rsc";
For some reason, anything before 6.36RC will not connect to the server via https. 
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:54 pm

Thanks A lot... I will try this...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 01, 2016 5:07 pm

let me know if it doesn't work. I've been meaning to give this scrip a rewrite to match the new servers.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sat Jul 02, 2016 8:24 am

gives an error;

failure: closing connection: <301 Moved Permanently> 104.27.182.151:80 (4)

but i found this. http://joshaven.com/resources/tricks/mi ... ress-list/ and its working... i dont know which one is better
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sat Jul 02, 2016 10:16 am

Which kind of mineral water is "better": sparkling or non-sparkling ? :-)

You can combine both lists if you feel that each one is not enough for you.
If not, just check counters when you use each of them and decide which blocks more attacks and use it. Remeber that lists are dynamic so the their goodness could change each time you reload it.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sat Jul 02, 2016 10:35 am

Which kind of mineral water is "better": sparkling or non-sparkling ? :-)
choose the non-sparkling one.. it will better.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 02, 2016 6:18 pm

New script - I found that RouterOS doesn't like my new CDN (CloudFlare) so I've setup a separate server for the lists.
:log warning "Downloading current Blacklist for this model";
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) do={ /ip firewall address-list remove $i } }
:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Please note that the address-list name has been changed.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sun Jul 03, 2016 12:47 am

New script - I found that RouterOS doesn't like my new CDN (CloudFlare) so I've setup a separate server for the lists.
:log warning "Downloading current Blacklist for this model";
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) do={ /ip firewall address-list remove $i } }
:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Please note that the address-list name has been changed.


Thats worked... Thank you very much... and i schedule it to work every 24h.. can i ask something for about listed ip addresses ? are these include Spamhaus DROP List and OpenBl List ?

Thanks A lot
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 03, 2016 2:46 am


Thats worked... Thank you very much... and i schedule it to work every 24h.. can i ask something for about listed ip addresses ? are these include Spamhaus DROP List and OpenBl List ?

Thanks A lot
My server pulls the lists from Spamhaus, OpenBL, malc0de, and emergingthreats. In addition to those, I have just over 40 servers and routers that report in and add to the list. Currently the server builds a new list every 24 hours. I'm working on a new system that will be updated continuously. 
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sun Jul 03, 2016 2:05 pm


Thats worked... Thank you very much... and i schedule it to work every 24h.. can i ask something for about listed ip addresses ? are these include Spamhaus DROP List and OpenBl List ?

Thanks A lot
My server pulls the lists from Spamhaus, OpenBL, malc0de, and emergingthreats. In addition to those, I have just over 40 servers and routers that report in and add to the list. Currently the server builds a new list every 24 hours. I'm working on a new system that will be updated continuously. 
Thank you very much... I start to use it. and i add some firewall rule for this can you check it for me also
/ip firewall filter
add action=drop chain=input comment=dynamicBlacklist in-interface=wan src-address-list=dynamicBlacklist
add action=drop chain=input comment=dynamicBlacklist dst-address-list=dynamicBlacklist in-interface=wan
add action=drop chain=forward comment=dynamicBlacklist in-interface=wan src-address-list=dynamicBlacklist
add action=drop chain=forward comment=dynamicBlacklist dst-address-list=dynamicBlacklist in-interface=wan
Thanks
 
Zorro
Long time Member
Long time Member
Posts: 675
Joined: Wed Apr 16, 2014 2:43 pm

Re: Blacklist Filter update script

Fri Jul 15, 2016 9:05 pm

how about adding fetching ads lists from popular lists(like one from say ublock 3rd subscriptions. like this one https://pgl.yoyo.org/adservers/) malware from malwaredomains.com and similar resources(known to be several comunities for) ?
then you can proces/convert then and then safely block them in dns static override (in worst case by adress lists).
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 15, 2016 9:33 pm

it's very possible to do that, but I would need to see what the impact on the routers would be. I'm not a big fan of the built-in DNS as it is and I'm not sure how well it would hold up with several thousand hostnames added to it.
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:16 am

I copied and pasted your recent code but I can not get it to work.... Tried it on two different CCR routers. One running 6.36 and the other running 6.35.2. It does not give an error it just doesn't do anything.....
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:30 am

:local model 	[/system resource get board-name]
:local version  [/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	 [/system identity get name]
:local scriptVer 2016.7.4a

:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Try this.. Same thing, just different formatting. Maybe it was a copy/paste issue?
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:37 am

Still nothing.Shows in the log file that it is going but that is it, no errors or anything just nothing. Strange.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:42 am

strange. any messages if you run it from the terminal?
you should see this...
[djoyce@Intrus_AltaLoma] > /system script run updateBlacklist 
      status: finished
  downloaded: 231KiB
       total: 231KiB
    duration: 1s

Script file loaded and executed successfully
[djoyce@Intrus_AltaLoma] > 
Then you should have have 2000~3000 items in the "dynamicBlacklist" address-list.
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:49 am

                                                                                                                                                                                                                                                            
 










































  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.36 (c) 1999-2016       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
  
[mike@Knittel Home CCR] > :local model    [/system resource get board-name]
[mike@Knittel Home CCR] > :local version  [/system resource get version]
[mike@Knittel Home CCR] > :local memory   [/system resource get total-memory]
[mike@Knittel Home CCR] > :local uname    [/system identity get name]
[mike@Knittel Home CCR] > :local scriptVer 2016.7.4a
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Downloading current Blacklist for this model";
[mike@Knittel Home CCR] > /tool fetch mode=https dst-path="/dynamic.rsc" \
\...    url="http://mikrotikfilters.com/download.php                      

<url> -- 
  status: failed

failure: closing connection: <301 Moved Permanently> 172.102.241.58:80 (4)
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
[mike@Knittel Home CCR] > ynamic.rsc
bad command name ynamic.rsc (line 1 column 1)
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Removing temp file...";
[mike@Knittel Home CCR] > /file remove dynamic.rsc
no such item
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Blacklist Update Complete.";
[mike@Knittel Home CCR] > /system logging enable 0
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:53 am

I can tell right off, that's not a current script.
you need to use the current (in the first post), and put it into a script, not the console. 
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:57 am

LOL okay went to the very first post you started this thread and copied and pasted it all.... Still get an error.... EDIT: I also tried this in the script file, no dice

 MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.36 (c) 1999-2016       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
  
[mike@Knittel Home CCR] > # Import Intrus Managed Filter Lists
[mike@Knittel Home CCR] > # (C)2016 David Joyce, Intrus Technologies
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :local model    [/system resource get board-name]
[mike@Knittel Home CCR] > :local version   [/system resource get version]
[mike@Knittel Home CCR] > :local memory   [/system resource get total-memory]
[mike@Knittel Home CCR] > :local uname   [/system identity get name]
[mike@Knittel Home CCR] > :local scriptVer   2016.7.4a
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Downloading current Blacklist for this model";
[mike@Knittel Home CCR] > /tool fetch mode=https dst-path="/dynamic.rsc" \
\...    url="https://mikrotikfilters.com/download.php                     
  status: failed

failure: closing connection: <404 Not Found> 172.102.241.58:443 (4)
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
[mike@Knittel Home CCR] > e]
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:01 am

It won't run from the terminal at all. It needs to be a script.

what is your output from this?
/system script run updateBlacklist
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:04 am

Bah! Been using your stuff for over a year and love it. Problems lately now.... 
  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.36 (c) 1999-2016       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
  
[mike@Knittel Home CCR] > /system script run updateBlacklist
  status: failed

failure: closing connection: <400 Bad Request> 172.102.241.58:443 (4)
[mike@Knittel Home CCR] > 
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:15 am

400 errors mean that the request was formatted wrong. Something is wrong with the copy/paste.

If you want, we can do a remote support session and I can take a look.  I use TeamViewer with my clients...

https://898.tv/intrus
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 208
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:21 am

Would love to but have to get to bed. Why is there no private messaging anymore? Or is it just my account?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:22 am

Not sure. I looked for that too...  Messaging and rep are both gone now.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:31 am

Try this - Delete the current script, then run these commands one at a time. (do not paste them all at once)
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https
/import updateBlacklist.rsc
/file remove updateBlacklist.rsc
/system script run updateBlacklist
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Wed Aug 10, 2016 7:55 pm

Nice. Done any benchmark comparing RAW to Filter?
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Wed Aug 10, 2016 8:45 pm

Not sure. I looked for that too...  Messaging and rep are both gone now.
Rep's still there - users can only give rep directly to someone on their profile nowadays.
PMs are definitely disabled though - I wonder why they did it.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 12:18 am

Isn't it forum layout scheme dependent?
There is an option to try different layouts so it might start to work for you. At least it works for me, I guess. There are voting buttons at each post I see. But using tapatalk mostly, not the Web..
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Re: RE: Re: Blacklist Filter update script

Thu Aug 11, 2016 12:26 am

Try this - Delete the current script, then run these commands one at a time. (do not paste them all at once)
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https
/import updateBlacklist.rsc
/file remove updateBlacklist.rsc
/system script run updateBlacklist
You can maybe start to distribute the blacklist via dns records to 6.36. An user can just put one domain name that would load whole list of ip addresses and keep it updating according the ttl. No need for scripts and files further.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:16 pm

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:19 pm

There are voting buttons at each post I see.
No, it's because you're an admin.
Admins get to vote up/down on individual posts, but standard users do not.
I guess they disabled it because some people got into downvote wars sometimes.
(my one and only downvote was from a time I jumped in on one such brawl, knowing full well that I was going to get a minus from the angry tantrum thrower)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:43 pm

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
I do, but the vast majority of routers pulling the list are still running 6.35 and lower. 6.32.4 makes up about 85% of the total. Once the majority are running a RouterOS that supports the RAW table, then I will move to that.

As it is now, you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:55 pm

you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
And here I was expecting your server and installer to just take the appropriate action.
Quit slacking, man! ;)
 
kivimart
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Thu Oct 10, 2013 3:06 pm

Re: RE: Re: RE: Re: Blacklist Filter update script

Thu Aug 11, 2016 10:29 pm

You can maybe start to distribute the blacklist via dns records to 6.36. An user can just put one domain name that would load whole list of ip addresses and keep it updating according the ttl. No need for scripts and files further.
One vote for dns distribution



Skickat från min Nexus 6P via Tapatalk
 
Zorro
Long time Member
Long time Member
Posts: 675
Joined: Wed Apr 16, 2014 2:43 pm

Re: Blacklist Filter update script

Fri Aug 12, 2016 1:16 am

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
I do, but the vast majority of routers pulling the list are still running 6.35 and lower. 6.32.4 makes up about 85% of the total. Once the majority are running a RouterOS that supports the RAW table, then I will move to that.

As it is now, you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
once "bugfix" was moved to 6.36 branch, which eventually happen later - it will, perhaps.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Aug 12, 2016 8:12 am

you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
And here I was expecting your server and installer to just take the appropriate action.
Quit slacking, man! ;)
It only puts in the script to load the address list, the user needs to make their own rules. There is just no realistic way to automate the rules, as every setup is different.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: RE: Re: Blacklist Filter update script

Fri Aug 12, 2016 8:17 am

You can maybe start to distribute the blacklist via dns records to 6.36. An user can just put one domain name that would load whole list of ip addresses and keep it updating according the ttl. No need for scripts and files further.
I can't think of a way to do that. A few issues - Server side, it needs to know what version, CPU and how much memory the router has. Their are several times a month that the list can ballon up to 8~10k addresses, so the server needs to server a smaller list to the low memory and low cpu routers. Also, while you can resolve an address, you can't push 3~5k IP's and subnets through one query. It would work if RouterOS had a DNSBL function.

You have other thoughts on how you would do it?
 
User avatar
hknet
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Jul 17, 2016 6:05 pm
Location: Vienna, Austria
Contact:

Re: Blacklist Filter update script

Fri Aug 12, 2016 1:18 pm

This initiative by IntrusDave makes for an interesting read!

I'd ask IntrusDave to consider delivering this blacklist in another format if possible, this would allow different use-cases, especially multiple 10G+ uplinks make it hard to handle stuff using firewall policies and blackhole-routes would make things much easier performance-wise imho.

In case you'd take contributions for the blacklist I'd be willing to host a honeypot as datasource.

Regards
hk
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Fri Aug 12, 2016 6:43 pm

There is just no realistic way to automate the rules, as every setup is different.
Yeah, very true.

If I were to vote for another delivery method of this list, I would choose BGP feed, which could easily be used as a means to blackhole route the offending addresses.
You could even get super fancy with it by using communities in your feed if you wanted - communities that specify what activites an IP was banned for, or how threatening an address is considered, etc. If Mikrotik adds a routing filter action of "add to address list" then BGP would be quite an awesome means to keep the list updated in real-time w/o the need for fetching/parsing lists over http.
 
User avatar
hknet
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Jul 17, 2016 6:05 pm
Location: Vienna, Austria
Contact:

Re: Blacklist Filter update script

Fri Aug 12, 2016 9:37 pm

Hi
currently testing your script on four small RBs.
2 x RB750UP - installed and works fine.
1 x hEX PoE lite - installed and works fine.
another hEX PoE lite yet fails:
/sys scr run updateBlacklist
status: failed

failure: closing connection: <400 Bad Request> 172.102.241.58:443 (4)

Therefore I'd ask for a bit more error-reporting ;)

Regards,
hk
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Aug 13, 2016 2:52 am

the 400 bad request means that something is wrong with your copy/paste of the script.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Aug 13, 2016 9:47 am

If I were to vote for another delivery method of this list, I would choose BGP feed, which could easily be used as a means to blackhole route the offending addresses.
You could even get super fancy with it by using communities in your feed if you wanted - communities that specify what activites an IP was banned for, or how threatening an address is considered, etc. If Mikrotik adds a routing filter action of "add to address list" then BGP would be quite an awesome means to keep the list updated in real-time w/o the need for fetching/parsing lists over http.

To be honest, BGP is my big weakness. I haven't needed, so I never learned about it. I'm not even sure where I should start.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 5:11 pm

BGP feeds for address lists are a bit different than routing with BGP because they're just using BGP as a vector to transmit the list, since its behavior is very well suited to the task - send the current list in full whenever a connection forms, and then send only deltas thereafter. It's quite efficient for this.

The way you could use this as a filter list right now would be to set all routes in the list as type=blackhole (via an in-filter on the client router) and enabling strict RPF on the client as well.
This will block traffic from going TO a blacklisted destination because it null-routes the destination.
The RPF causes the blacklisted addresses to get blocked because real packets won't arrive via the black hole interface. Since the reverse path doesn't match the routing table, RPF will discard packets from the blacklisted sources.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 7:58 pm

can you give me a sample export of what I would need to get started?
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 8:55 pm

...just brainstorming here...

One way to do it is by using ExaBGP

When you create the rules for the address-list with your (I presume) daily script, you can also push the prefixes to ExaBGP with simple scripting (Python, PHP, bash, etc).
The only difference is that you will probably need to keep track of what you have sent to ExaBGP so in the next update of the list, you only send the differences (new advertisements, and withdrawals). This way you keep the bgp traffic to minimum, and the remote routers will update their blacklists far more efficient (as ZeroByte pointed out, only on first connection it will fetch the entire prefix list, and then it will keep the received prefixes updated by adding new prefixes, or withdrawing removed prefixes from the routing table).

Then from there you can either have everyone connect to ExaBGP (BGP Session) and send them the prefixes or peer ExaBGP with a BGP router of your choice and everyone establish their BGP sessions with that.

RouterOS is not ideal (I think) for this task since you would have to configure separate BGP peers for each and everyone that wants to use this service.
AFAIK routeros doesn't allow for non-configured peers to connect to BGP.
I haven't researched it recently, but if I recall correctly, quagga can allow incoming bgp sessions without being explicitly configured.
Probably ExaBGP does this as well.
Or if you prefer routeros, you could make a simple web page where an interested party can register their routers and using the routeros API, automatically create/update/delete the BGP peers.
I am not sure what's the peer limit on routeros (if any).

Also I would suggest running two BGP instances (ideally in separate datacenters/locations) so that if you need to restart the instance, or the BGP sessions terminate due to a network problem, the second BGP instance will keep everything running on all the remote peers without having their blacklists withdraw all of a sudden.

On the client's side, they would need some routing filters to handle all prefixes received by you adding them as blackholes to the routing table.
Just like with the firewall filters, it's pretty much the same with routing filters, in the sense that it's not easy to provide an automated way of configuring them since everyone has its own filters.

By the way Team cymru provides a similar service like this.

Also I just checked the blacklist and I noticed that you include many /16 prefixes in there. Eg: 138.200.0.0/16
That includes a large range belonging to Hetzner, a large server provider in Germany. Sure they might have some 'bad apples' in there, but they also run many many legit servers/services.
What are your criteria for adding a prefix in your list? This and many other /16s are bound to contain many legit stuff since they are kind huge ranges. Are you sure you should be blocking such large ranges?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 9:40 pm

My lists is regenerated every 4 hours. When /16's are added, It's almost always because the ISP has been notified of a BOTNET being run on their network, AND they have refused to look into it. They are also added the honeypots see attacks / spam from more than 50% of the IP's in that range. The networks are removed form the list as soon as the ISP responds to the issue, or the honeypots see that the issue has been resolved.

Thanks for the BGP info. I will look into it. I certainly don't want to setup BGP peers for every site that wants to use my list. As it is now, I have about 2700 routers that pull the list every 24 hours. (and 5 that insist on pulling it every 60 seconds)
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 9:48 pm

Thanks for the info.

Actually I was wrong about 138.200.0.0/16 belonging to Hetzner. :oops:
138.201.0.0/16 belongs to Hetzner.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 9:56 pm

138.200.0.0/16 is currently listed because of high-volume bulk email (spam)
Just to my active SMTP server and my honeypots, they have attempted 3.2M emails from 14,000 IP's in the subnet over the last 7 days.

Stammhaus also has that subnet listed in their DNSBL and BGP.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 10:27 pm

BGP peering would at least let people get real-time updates w/o having to download the list every 60 seconds.

I think that using the API for a web-based sign-up might be the best idea if you want to use ROS as the BGP source.
I would recommend that the master set of addresses be kept separate from the hosts that subscribers actually peer with.
Furthermore, I'm not sure how this translates into RouterOS, but in Cisco, grouping peers into peer-groups has a marked improvement on the performance because the BGP process makes announce/withdraw decisions once for the group and then sends them out. Each un-grouped peer must be computed separately - causing much more CPU load for hosts w/ large number of peers.
(I am not certain if ROS even has a similar construct to peer groups - never deployed BGP on a production Mikrotik router)

If I were setting this up for BGP distribution, I would probably do the following:
Private ASN on my side
Bogus next-hop IP (e.g. 127.0.0.2 or 169.254.0.1 or something like that), so that if the subscriber forgets to blackhole the route properly, it won't try to actually route the address to a real next-hop.
EBGP + multihop = 256
in-filter=drop-all filter

out-filter = this is where interesting things could happen. If you wanted to allow your subscribers have the ability to specify certain filter types (akin to the list size limits you do for smaller client routers) then you could make similar filters and let the customer choose which one to apply to their session. Of course, a customer with a small router could still opt to take the full feed and just filter it themselves by whatever criteria they like.

Here would be a great getting-started configuration in ROS:
/routing bgp instance
set default AS=64567 router=id=64.5.6.7 redistribute-static=yes out-filter=bgp-static-filter
/routing filter
add action=accept bgp-origin=igp chain=bgp-static-filter protocol=static routing-mark=blacklist set-out-nexthop=127.0.0.2
add action=discard chain=bgp-static-filter
add action=discard chain=discard-all

Then for each route that you want to publish in the blacklist, add a static route:
/ip route add dst-address=169.254.1.2/32 routing-mark=blacklist type=blackhole
(The routing-mark is just a way to make certain that only these routes should be exported to BGP peers - the system-wide BGP-out filter.)

Each subscriber would be added as follows:
/routing bgp peer
add name=peerName remote-address=x.x.x.x remote-as=XXX in-filter=discard-all multihop=yes

You may want to play with the keepalive timers and/or specify an MD5 password for each peer. (agreed upon in the web-based signup)

The peer should just set next hop to blackhole / null / some null prefix in their network, depending on the type of router they're using.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 11:12 pm

ZeroByte's approach seems a lot more straightforward than mine!
Upon a little research, so far I see only the big vendors (cisco, juniper, etc) have implemented 'Dynamic Neighbors' support into BGP.
Quagga, BIRD, ExaBGP - as far as I can tell - they don't support it yet. So I guess the creation of BGP peers (via API, or some other way) seems the (only) way to go.
But that brings problems of its own. You may end up with tons of 'dead' peers over time of users that stopped using the service, so there should be a periodic check for long-dead peers to delete them.

I tried to test ZeroByte's approach but I noticed two issues.

First, the BGP instance must be configured for the 'blacklist' routing table otherwise it only redistributes static routes that are on the main routing table.
/routing bgp instance
set default AS=64567 router=id=64.5.6.7 redistribute-static=yes out-filter=bgp-static-filter routing-table=blacklist
Incidentally, trying this I noticed that the BGP Advertisements does not show anything even though it actually advertises stuff. Probably a bug?..
[admin@MikroTik] > /routing bgp advertisements print 
PEER     PREFIX               NEXTHOP          AS-PATH      ORIGIN     LOCAL-PREF
Of course the BGP instance could run on a dedicated mikrotik installation (CHR?) on the main routing table without interfering with any real/backbone traffic.

Second, RouterOS' BGP does not seem to accept any prefixes with a non reachable/bogus next-hop (or smth similar, I am not sure yet). Or at least I couldn't find a way to do it.
If I don't set the out next-hop at all, then the prefix is added to the routing table as inactive with gateway the (multihop) IP of the bgp peer. So its pretty much useless at this point.
But, simply setting an incoming filter on the 'client' side to change the type of the incoming prefixes to blackhole, the prefixes are installed as active-blackhole to the main routing table properly.
IIRC quagga accepts prefixes with bogus next-hop (I haven't tried in a while so I maybe wrong about this).

All in all, I find ZeroByte's solution much easier to implement (taking into account that you haven't worked with BGP before).
I avoid doing any kind of redistributions on BGP so naturally the first idea that came into mind was not the simplicity of static routes redistribution :P
Also routeros does not advertise more than 200 networks (/routing bgp network) per instance. But it should work perfectly fine with thousands of redistributed prefixes.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 11:33 pm

Sounding more and more like the script is a much simpler way to go :)
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 1:28 am

All in all, I find ZeroByte's solution much easier to implement (taking into account that you haven't worked with BGP before).
I avoid doing any kind of redistributions on BGP so naturally the first idea that came into mind was not the simplicity of static routes redistribution :P
Also routeros does not advertise more than 200 networks (/routing bgp network) per instance. But it should work perfectly fine with thousands of redistributed prefixes.
Glad you liked the idea - and in general, you'll see that I'm quite the opponent of redistributing routes, ESPECIALLY into BGP, but this is a special case where all of that best practice stuff for network engineering goes right out the window. The easiest thing to do is just redistribute routes into BGP on a box that is not otherwise doing any routing.

Setting BGP into its own routing table at the process level makes things even simpler - I didn't actually try to lab this up, but had I done so, I certainly would've caught that requirement.
Sounding more and more like the script is a much simpler way to go
Funny thing is - to me, a script is always less desirable than leveraging the built-in behaviors of a system. People a lot smarter than me had a conference to make these standard protocols as robust as possible, so using them is like using a wheel as opposed to inventing my own wheel. But BGP distribution is not without its drawbacks - the biggest one right now is that null routing + RPF enforcement is the only thing you can do with it, and while it's effective as a blacklist, it is nowhere near as flexible as an IP address list. (which is why I'm hoping they do implement "add target to address list" as an action for routing filters).

I will say that the BGP method would be simpler to manage over a large distribution, and the implementation on the client side is brain-dead simple:
enable BGP (if not already using BGP) with any private ASN other than 64567. (or just use their real ASN if they're already running BGP).
in-filter=accept all -> action=set route type=blackhole
out-filter=discard all
enable strict RPF in IP options.

The nice thing about BGP is that the subscriber can put whatever kind of filters they like against the feed - they can specify no prefixes shorter than /22 for instance, if they hate the idea of blacklisting entire /16 or /8 prefixes. They can specify IP blocks to ignore. If running real BGP, they can set local_pref to 1 (very very bad) on the blacklist peer, so that no publicly-routed prefix can be blacklisted in its entirety.... it gives the paranoid administrator much more control than simply importing a list carte blanche and black-holing everything in it.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 3:01 am

I started a very basic bgp blacklisting service as a proof of concept of what we are talking about.

Anyone that would like to try it can use this page to register their bgp peers:
https://bgp-register.cha0s.gr

You give the IP & AS of your bgp instance and it will automatically create the BGP peer on my side and provide the user with basic copy/paste routing filters + bgp peer configuration.

It automatically generates a random 10 digit MD5 key for the BGP peer.

If someone wants to delete their peer from the system they can do so by providing the IP + AS + MD5 key.

Also the peers on my side are configured in passive mode so that they don't try again and again to connect to remote peers (in case someone stops using the service for example and forgets to delete their peers)

Since this totally a proof of concept, the design and usability of the page (or lack thereof :P) is obviously bad.
Also I haven't put any effort in input sanitization so it's quite probable that someone may find a way to break it :P :lol:

I wrote a simple script that downloads the blacklist from IntrusDave's service, converts the commands to static blackhole routes and adds them to my BGP router which in turn advertises them to every BGP peer.
I haven't implemented a way to keep updating the static routes upon changes of the list.

If there's interest I could develop the service a little more to be more complete and stable (ie: 2 bgp instances on different datacenters, proper registration UI, etc).


I completely agree with ZeroByte regarding BGP. It's way more efficient and flexible.
To be honest I've known about this blacklist since its beginnings but the reason I haven't used yet is because I don't like the idea of receiving a number of IPs for blocking without any way of filtering stuff that I might not want blocked.
Using BGP filters anyone can accept only the blacklists that he/she wants.
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Tue Aug 16, 2016 8:37 am

How would bgp work for dynamic clients?


Sent from my iPad using Tapatalk
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 2:12 pm

At the moment it wouldn't. Not without somekind of VPN I guess...

That's where dynamic neighbors would help. You would (in theory - I haven't used it) establish a BGP session regardless of your IP.
I believe there was a feature request about this a while back. I can't find it at the moment.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 3:22 pm

I haven't implemented a way to keep updating the static routes upon changes of the list.
I extended the conversion script to check all the current static routes on the router and remove or add any changes that occur from IntrusDave's blacklist.

I've set it up to run every 24 hours at 00:00 GMT+3 DST.

So it's pretty much all automatic now. It will keep all the static routes (blackholes) up to date with minimum effort and maximum efficiency (only changes are propagated to all the bgp peers instead of the whole list every day).

Another advantage of using BGP is that you can push changes almost in real time instead of periodically checking for an updated list.
Once you update the list the BGP can push the changes right away to everyone.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 4:12 pm

I am not sure what's the peer limit on routeros (if any).
I just tried adding ~18.000 peers on a single BGP instance and it worked fine. So there doesn't seem to be any (low) limit on how many peers you can have on ROS BGP. :)
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 4:51 pm

Nice favicon on the BGP registration page, Cha0s. ;)
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 5:09 pm

Yeah, that's probably the only pretty thing about the page :lol:
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 5:32 pm

Coming back to the efficiency argument, I just peered a couple of mAP-Lite with the blacklist and they loaded ~2900 prefixes into the routing table without breaking a sweat. Upon connection establishment it took ~2 seconds of high cpu usage (no more than 80%) and that was it.

Subsequent updates to the list will take virtually no resources (assuming that the updates are a few dozen prefixes +- every day).
Only on peer re-connection will have a cpu spike to re-add all prefixes to the routing table.

Prefix matching on the routing table level is much more efficient than matching an address list with iptables.
So even small devices can handle many addresses in the blacklist without any compromise in speeds.

One caveat is that BGP will need more ram the more prefixes it handles. Though anything above 64MB should work with many thousands of prefixes without problem.
http://wiki.mikrotik.com/wiki/Manual:BG ... e_table.3F
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 7:21 pm

Have you done any forwarding performance stress tests with the BGP feed in place and fully synchronized?
How about with strict RPF checks enabled?
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 8:21 pm

I set up a little lab to try this.

RB3011 <-> RB450 <-> hAP Lite

I did bandwidth tests from RB3011 to hAP Lite and measured the forward performance on RB450 (since it's old and with such small address list the results are more notable)

Bandwidth tests were done at 512 Bytes (UDP) and were limited at 100mbit due to fast-ethernet ports on the RB450 & hAP Lite.

The results are the following

PLAIN TEST (no BGP, no firewall)
100mbit traffic @ 22kpps
Route cache on: 55-60% CPU (top process: ethernet 25-30%, second top process: networking 15-20%, third: unclassified 9%)
Route cache off: 90-95% CPU (top process: networking 25-30%, second top process: ethernet 25-30%, third: routing 15%)
Total blacklist entries: 0
Total blackhole routes: 0
Free Memory: 12.2MB

RP Filter did not make any difference


BGP TEST
100mbit traffic @ 22kpps
Route cache on: 55-60% CPU (top process: ethernet 25-30%, second top process: networking 15-20%, third: unclassified 9%)
Route cache off: 90-95% CPU (top process: networking 30-35%, second top process: ethernet 25-30%, third: routing 15-20%)
Total blackhole routes: 2966
Free Memory: 10.4MB
Time to load all prefixes: ~2seconds

RP Filter did not make any difference


FIREWALL TEST (single drop rule- NOTHING else at all)
100mbit traffic @ 22kpps
Route cache on: 95-100% CPU (top process: ethernet 30-35%, second top process: firewall 25-30%, third: networking ~20%)
Route cache off: 100% CPU (top process: networking 30-35%, second top process: firewall 25-30%, third: ethernet 20-25%)
Total blacklist entries: 2966
Free Memory: 6.9MB
Time to load address-list: ~7seconds

RP Filter did not make any difference


So it's rather obvious that making decisions using the routing table instead of the address-list is more efficient. Especially with route cache on (by default it's on already).
It's also less memory hungry.

Disclaimer: The above tests were done very quick without many repeats so they are not 100% accurate or reflect real world performance (multiple connections, multiple firewall rules, etc).
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Tue Aug 16, 2016 9:39 pm

BGP seems like the way to go...

Sent from my XT1575 using Tapatalk
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 18, 2016 8:32 am

Very cool. I guess I need to figure out how to do this on my end too. I have a CHR sitting idle in my datacenter.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:16 pm

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
I do, but the vast majority of routers pulling the list are still running 6.35 and lower. 6.32.4 makes up about 85% of the total. Once the majority are running a RouterOS that supports the RAW table, then I will move to that.

As it is now, you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.

hi,

can you share your raw rules ? Im using 6.36.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:24 pm

You can download the list with curl
curl -A "Mikrotik/6.x Fetch" "https://mikrotikfilters.com/download.php?get=dynamic&model=RB3011UiAS&version=6.36 (stable)&memory=1011.3 MiB&id=MikroTik&ver=2016.7.4a"
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:27 pm

can you share your raw rules ? Im using 6.36.
I would just say put two rules in raw table:
chain=prerouting src-address-list=blacklist action=drop
chain=prerouting dst-address-list=blacklist action=drop
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:48 pm

Hi ZeroByte,

thank you verymuch. I did it like that;
0 chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist
1 chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist
 
savage
Forum Guru
Forum Guru
Posts: 1262
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: Blacklist Filter update script

Sat Sep 17, 2016 9:03 pm

BGP seems like the way to go...

Sent from my XT1575 using Tapatalk
Said that in post #18 already - but then it was thought better to give me negative karma for the post :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Sep 18, 2016 8:25 am

You can download the list with curl
curl -A "Mikrotik/6.x Fetch" "https://mikrotikfilters.com/download.php?get=dynamic&model=RB3011UiAS&version=6.36 (stable)&memory=1011.3 MiB&id=MikroTik&ver=2016.7.4a"

I would prefer that people don't do this. I already have one site that is mirroring my list and claiming it as his own. Very annoying.

As for BGP - I simply don't care to put the time in to building a system to setup the peers. Yes, I know it may ultimately be a better way to do this, but the current way is VERY easy for me, and I don't need to do any extra work for the number of little RB951's (hAP's) that I deploy. I really don't care to setup BGP on them, and given the limited memory, having the server only serve them the "3 day" list keeps things small and simple.

Maybe someday I'll BGP.. just not now. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Sep 18, 2016 8:31 am

BGP seems like the way to go...

Sent from my XT1575 using Tapatalk
Said that in post #18 already - but then it was thought better to give me negative karma for the post :)
Withdrawn. Still don't think it the best way to go :)
 
brianlewis
Member Candidate
Member Candidate
Posts: 134
Joined: Tue Jul 20, 2004 10:54 am
Location: Irvine, CA

Re: Blacklist Filter update script

Wed Oct 12, 2016 12:41 am

Dave,

I've started using your list and I'm getting reports of legitimate sites being blocked dynamically from the filter list.

salesforce.com not coming up (72.21.81.200 which isn't their primary ip, maybe an image cache server?)
ssl.cdn-redfin.com (72.21.91.8) which prevents redfin.com, trulia, hilton.com and many other sites from working)

Any way we can look at why 72.21.x.x is blocked and consider adjusting that blacklisting?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 12, 2016 1:48 am

the ip (72.21.81.200) was flagged because it is currently serving malware in the form of infected images.
As the blacklist is free for use and was designed to keep my clients safe from infection - I will not be removing the IP.
If you really need it, I would recommend creating a separate whitelist for IPs you do not want blocked.

For me, I will not allow my clients to access sites that are currently serving malware.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 12, 2016 1:54 am

Correction, it's serving Ransomware via Javascript.
Once they have fixed the issue, it will automatically be removed.
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Mon Oct 17, 2016 10:07 pm

Any reason the below suggestions never made it to the updateBlacklist script?
(note that I've already updated the first code fragment below from `intrusBlacklist` to `dynamicBlacklist`)

Same for the ordering of the firewall rules further below: any reason why?
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "dynamicBlacklist" ) do={ /ip firewall address-list remove $i } }
Can be simplified to
/ip firewall address-list remove [/ip firewall address-list find comment = "dynamicBlacklist"]
Should actually increase the efficiency.
Firewall rules order:
/ip firewall filter
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
Where I have already see part of that rules on exact order? :lol:
Ah:
http://forum.mikrotik.com/viewtopic.php?f=9&t=83387

Sort the rules for efficency (simply drop if coming from blocked list, not first check malformed packet then drop)
/ip firewall filter
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
--jeroen
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Oct 18, 2016 5:03 am

You are welcome to change the script and rules as much as you like.
Script is written as it is because it works without fail on all 6.x versions. I don't normally change things if they are working.
The rules in the first post do have the blacklist drops at the top.
However, most by this point should be using raw drops instead of filter drops.
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Tue Oct 18, 2016 11:14 pm

You are welcome to change the script and rules as much as you like.
Script is written as it is because it works without fail on all 6.x versions. I don't normally change things if they are working.
The rules in the first post do have the blacklist drops at the top.
However, most by this point should be using raw drops instead of filter drops.
Thanks. Just wanted to know the reasoning while wading through this thread just in case I missed something.

With `raw` drops, you mean rules like in http://forum.mikrotik.com/posting.php?m ... 9&p=553094 right?

--jeroen
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 19, 2016 1:28 am

Filter rules are more of a personal thing. There is no one-size-fits-all solution. Mine evolve all the time and I don't go back to the first post and update all the time. They are just an example. That said, here are my current "starter set" for most new routers I deploy.

/ip firewall filter
add action=reject chain=Filter dst-address-list=dynamicBlacklist reject-with=icmp-admin-prohibited
add action=accept chain=Filter connection-state=established,related
add action=drop chain=Filter comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Filter comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Filter comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Filter comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Filter comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=return chain=Filter comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Filter\" chain" jump-target=Filter
add action=accept chain=input dst-port=8291,22,443 protocol=tcp src-address-list=trustedHosts
add action=accept chain=input in-interface=lanBridge
add action=drop chain=input in-interface=internet
add action=jump chain=forward comment="Check for bad stuff in \"Filter\" chain" jump-target=Filter
add action=accept chain=forward in-interface=lanBridge out-interface=internet
add action=drop chain=forward in-interface=internet
/ip firewall raw
add action=drop chain=prerouting src-address-list=dynamicBlacklist
Notice that the inbound blacklist drops are in the raw table, while the outbound are in the filter table. Also inbound are drops, outbound are ICMP Admin Prohibited.
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Wed Oct 19, 2016 6:29 pm

... here are my current "starter set" for most new routers I deploy.

...

Notice that the inbound blacklist drops are in the raw table, while the outbound are in the filter table. Also inbound are drops, outbound are ICMP Admin Prohibited.
Thanks again. There is still a lot of stuff to be learned for me.

How did you find about the meaning of the various `reject-with` values? They are not documented any more and in the past were never explained in the documentation.

--jeroen
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 19, 2016 8:00 pm

That's more of an intro to TCP/IP thing. Note a Mikrotik thing.
 
Bipe
just joined
Posts: 1
Joined: Fri Jul 15, 2016 6:20 pm

Re: Blacklist Filter update script

Thu Nov 10, 2016 9:58 pm

Hi IntrusDave,

I've been having issues with your blacklist and bit.ly
looks like 67.199.248.10 and 67.199.248.11 are used by bit.ly and with firewall rule drop connections to blacklisted I can't access any short url's generated by it (had to exclude them).
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 10, 2016 11:41 pm

That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.

I will not manually remove addresses.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8709
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Thu Nov 17, 2016 9:35 pm

This topic looks interesting for many people. Let's make it sticky :)
 
dadaniel
Member Candidate
Member Candidate
Posts: 220
Joined: Fri May 14, 2010 11:51 pm

Re: Blacklist Filter update script

Mon Nov 21, 2016 4:24 pm

That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.

I will not manually remove addresses.
bit.ly is a referer-website (like shorturl), it never serves anything from its own IP address. Could you please have a look again?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Nov 21, 2016 7:31 pm

No, I'm sorry. As I said, the processes is automated. It receives and processes nearly 100,000 IP's each day. If an IP makes it on the list, then it has been directly or indirectly responsible for malware. The whole thing was designed to keep my personal clients safe. If it's not working for you, then you have a few options.
1) ask for a refund and don't use the list.
2) use the list as an incoming only filter
3) use the list as a raw in and out list, and whitelist the addresses you feel are wrongly blocked.

Personally, I use option 3 for businesses, and I use option 2 for home users.
 
User avatar
bajodel
Long time Member
Long time Member
Posts: 551
Joined: Sun Nov 24, 2013 8:30 am
Location: Italy

Re: Blacklist Filter update script

Wed Nov 23, 2016 3:57 pm

.. [CUT] ..
3) use the list as a raw in and out list, and whitelist the addresses you feel are wrongly blocked.
Personally, I use option 3 for businesses .. [CUT] ..
Testing (mode 3) now on a new hEX and works like a charm. Thanks.. (rep+)
I'm wondering if I can consider "reliable" your service (not in terms of false positive or alike.. but) in terms of availability of updates ; I'm considering to put this in production but I'm evaluating if create blacklists by myself or (pay and) pretend to have reliable external service. I think you can understand
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Nov 23, 2016 5:44 pm

I feel it's reliable. I have all of my clients using it. (24 regional hospitals and medical centers). In addition, I have 1.830 other routers that use it. My only wish would be that all of these routers were able to send back addresses that they are attacked by. Unfortunately, there is no good way without putting their privacy at risk.

After all is said and done, The back end system I've built is fully self contained and is running on hardware that was built to last the decade.
 
nickperkins
just joined
Posts: 8
Joined: Mon Nov 24, 2014 6:04 am
Location: Masterton, New Zealand
Contact:

Re: Blacklist Filter update script

Thu Nov 24, 2016 11:39 pm

I had to make a slight change to the script for a couple of my clients that had spaces in their router identity. The spaces were causing the fetch to fail, have added this in to resolve the issue:
:local uname	[/system identity get name]
:for i from=0 to=( [ :len $name ]-1 ) do={ 
:local tmp [ :pick $name $i ]
:if ($tmp != " ") do={ :set $newname "$newname$tmp" }
}
:set $uname $newname
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 24, 2016 11:43 pm

Odd, the server normally deals with that. Can you give me the name the fails? I'd like to try and reproduce the error.
 
nickperkins
just joined
Posts: 8
Joined: Mon Nov 24, 2014 6:04 am
Location: Masterton, New Zealand
Contact:

Re: Blacklist Filter update script

Fri Nov 25, 2016 12:36 am

Odd, the server normally deals with that. Can you give me the name the fails? I'd like to try and reproduce the error.
Hi IntrusDave, the first case I found it the identity was 'Nick Home AP', I see there is a warning in the log when running that there is a new version at updater.php, had a look at that and I see that version doesn't read the identity. Perhaps that's the better option?
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Wed Nov 30, 2016 11:50 pm

My only wish would be that all of these routers were able to send back addresses that they are attacked by. Unfortunately, there is no good way without putting their privacy at risk.
Let's discuss (in a new thread if needed) on how to make this possible.

I'm keeping dynamic login failure and unknown-port usage blacklists currently having 10k+ entries (a huge increase after the TR-069 issues at German Telekom) with expiration of 14 days.

It would be cool if I could get them at your place somehow.

--jeroen
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Dec 01, 2016 1:56 am

I am using the list and service by IntrusDave since a few day and it works very well and I had many hits on the rule. I use it more selective by filtering obvious illegal request out in advance. I run three services and that is mail, web en secure web.

Now those botnets have that many bots and that will result in very long lists of IP and results decreases the efficiency in using those lists if you look at filtering time. Last weekend it was very busy and many many bots tried to get in so I looked at what they were doing. There were not hat many that were caught by the list and 99% were filtered by the following rule that just leaves a window for the services I serve, and filter out any thing else on TCP that is obvious illegal.
/ip firewall raw
add action=drop chain=prerouting comment="Pre-filter TCP" dst-port=!25,80,443 in-interface=pppoe-out log-prefix=\
    "New drop" protocol=tcp tcp-flags=syn,!fin,!rst,!psh,!ack,!urg,!ece,!cwr
In this way all filters behind this do only have focus on the services that are running and check if there is everything ok before conveying the requests to the servers.
 
User avatar
HiltonT
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Mon Feb 07, 2011 4:24 am
Location: 'Srayamate
Contact:

Re: Blacklist Filter update script

Fri Dec 02, 2016 12:42 am

I started a very basic bgp blacklisting service as a proof of concept of what we are talking about.

Anyone that would like to try it can use this page to register their bgp peers:
https://bgp-register.cha0s.gr

Since this totally a proof of concept, the design and usability of the page (or lack thereof :P) is obviously bad.
Also I haven't put any effort in input sanitization so it's quite probable that someone may find a way to break it :P :lol:

If there's interest I could develop the service a little more to be more complete and stable (ie: 2 bgp instances on different datacenters, proper registration UI, etc)...
I gather this page/service didn't garner enough interest to keep it online (or has it just temporarily gone offline)? That's a shame - I like the idea of a centrally maintained set of blacklists that I can subscribe various devices to, however I'm not a massive fan of the lag a massive address-list puts on the traversal time of packets through the firewall, nor the CPU load this requires.

I know as much about BGP as the next person who has never used it, but I'm definitely interested in learning more about it (with the use of Private AS Numbers as the cost of a public one isn't going to be justifiable for many of our clients) for use in blocking unwanted traffic - I really don't see it being useful for SMBs with one primary and a backup Internet connection when there's an AU$500/year fee for the public AS Number.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Dec 02, 2016 12:50 am

Configured correctly, you should not see much, if any increased CPU load or lag. You should be filtering the initial connection, not the established.
 
telepro
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Sun Apr 03, 2011 7:50 pm

Re: Blacklist Filter update script

Mon Dec 12, 2016 5:03 pm

We have been using the blacklist provided by IntrusDave with success. However, we have on infrequent occasions found the list contains an IP address that is the source of Microsoft download web sites and files linked to by their corporate web pages. Momentarily turning off the use of the blacklist provides successful web access to these Microsoft sites. Has anyone else see this issue?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Dec 12, 2016 5:34 pm

it is not uncommon. The blacklist is an automated system that flags any IP that has served malware in the last 7 days. Just because a CDN is used/owned by Microsoft doesn't mean that it is impervious to malware.

Again, as I have stated before, This system was designed by me to keep my paid clients as safe as possible. I use this for all 24 of my hospitals and clinics. It works well to help stop the attacks of botnets and helps to prevent infection. That said, I will not whitelist any IPs just because they are used by a large company. Any website or CDN can be infected, no one is exempt from being filtered for it.
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Mon Dec 12, 2016 6:56 pm

Love the work you did on this script. It's fantastic! however I too am finding a few services are blocked. Microsoft for one, Periscope (A Twitter company) is blocked. I'm currently deciding if its worse the hassle :)

But you did do a fantastic job, great service, I thank you much for it!
 
telepro
Frequent Visitor
Frequent Visitor
Posts: 68
Joined: Sun Apr 03, 2011 7:50 pm

Re: Blacklist Filter update script

Mon Dec 12, 2016 7:04 pm

I concur. The provided programming and database has proved to be very useful; when employed, it filters out a significant amount of unwanted and problematic traffic. Thus our question regarding whether anyone else was seeing traffic from Microsoft corporate sites blocked. Thanks for the info.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Dec 12, 2016 9:09 pm

These companies use CDNs. So what you see as blocked, I may not see blocked. When something is added to the block list, it is because that IP was found to have some form of malware.

The filters can be used in many ways.
The list can be used in the RAW or the standard filters. Both incoming and outgoing.

If you are not able to access a website because of the list, that means that you are using it either in RAW, or on an outbound rule.
You should be using it in the INPUT chain, with the New Connection flag. You do not need to be filtering established connections.
You also don't need to filter destination IPs, unless you want more malware protection.
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Mon Dec 12, 2016 10:02 pm

I was using the new RAW rules and blocking destination etc I'll just use an input/forward rule then :)
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Wed Dec 14, 2016 7:53 am

I was using the new RAW rules and blocking destination etc I'll just use an input/forward rule then :)
share your rules for us :)
 
brianlewis
Member Candidate
Member Candidate
Posts: 134
Joined: Tue Jul 20, 2004 10:54 am
Location: Irvine, CA

Re: Blacklist Filter update script

Wed Dec 21, 2016 5:07 pm

Any particular reason Microsoft’s Ajax CDN (72.21.81.200) is being blacklisted?
 
zhup
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Dec 03, 2015 10:10 pm

Re: Blacklist Filter update script

Thu Dec 22, 2016 8:23 pm

Hello IntrusDave,
Great work!

Could you please change the script? It would be good to write the lists to the pendrive instead of the nand.
Thank you in advance.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Dec 22, 2016 8:31 pm

You are welcome to change it as you like. I don't use flash drives in my routers.
 
zhup
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Dec 03, 2015 10:10 pm

Re: Blacklist Filter update script

Thu Dec 22, 2016 9:56 pm

You are welcome to change it as you like. I don't use flash drives in my routers.
Could you please check if I made all necessary changes for using the pendrive?
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
:delay 30

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2016.7.4a

:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/disk1/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/disk1/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Thank you in advance.

Do you use the blacklist from OpenBL.org?
 
jo2jo
Forum Guru
Forum Guru
Posts: 1003
Joined: Fri May 26, 2006 1:25 am

Re: Blacklist Filter update script

Sun Jan 01, 2017 1:27 pm

Just wanted to say THANK YOU for all your hard work on this list! its really excellent! and your decision to use dynamic address list entries is really sharp.

(dynamic address entries simply means that on the address-list rules, he sets a timeout value, so that the mikrotik stores the address list in RAM memory until it times out, versus it being a normal address list entry with no time out, and thus the mikrotik stores the entry on its "disk" or nand drive, so that it will persist through reboots). However ALL nand based memory has a "limited" lifetime which is slightly reduced by each write (think write endurance on SSD drives) - the better way is to store this type of data in RAM memory as it does not have this degradation through writes issue.)

ie:
non "dynamic" address list entry (will be stored on the routers "disk" such that it will persist/remain through reboots or power failures, but the nand memory *does* degrade with each write):
/ip firewall address-list add address=x.y.z.z

vs a "dynamic" address list entry (will be stored on the routers RAM , which has *no* degradation with each write):
/ip firewall address-list add address=x.y.z.z timeout=2d

Thanks again! and for any future messages from users with problems with a specific IP on his list (ie x.y.z.a IP address is blocked, but really belongs to google.com which my network needs to access), Please, PLEASE read back through this thread, the author has addressed his reasons (for ip inclusion, and why he also will not remove specific IPs from his list) and he also has provided a clear way for you to "fix" or override specific IPs which you "feel" your network must access
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 10:39 am

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
May I please be so bold, whats the commands to change the temporary file storage location?

I use a RB750Gr3 and has a microSD card installed. Currently its only really been used for backup configs and some logs as I haven't found away to switch more to use it.
 
flazzarini
just joined
Posts: 19
Joined: Thu Jun 13, 2013 11:05 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 11:42 am

Hi there,

I would be interested to know which source you are using to get this list of IP addresses to block? Would you care to share this? I would be interested to integrate the list you are serving into Blocklister (Github).

Thanks for your help and keep up the good work!
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 11:45 am

Hi there,

I would be interested to know which source you are using to get this list of IP addresses to block? Would you care to share this? I would be interested to integrate the list you are serving into Blocklister (Github).

Thanks for your help and keep up the good work!
The OP said in a previous post that he compiles the list himself from his 50 or so routers which gets attacked.
 
flazzarini
just joined
Posts: 19
Joined: Thu Jun 13, 2013 11:05 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 12:36 pm

Hi there,

I would be interested to know which source you are using to get this list of IP addresses to block? Would you care to share this? I would be interested to integrate the list you are serving into Blocklister (Github).

Thanks for your help and keep up the good work!
The OP said in a previous post that he compiles the list himself from his 50 or so routers which gets attacked.
Thanks for the answer to that!
 
nwa
newbie
Posts: 29
Joined: Sun Aug 17, 2014 3:02 pm

Re: Blacklist Filter update script

Wed Jan 11, 2017 11:14 pm

I want only to say.... thanks !!!!

I hope the list works good for german locatet routerboards and this project never ends ;)
 
ignore
just joined
Posts: 1
Joined: Sat Feb 11, 2012 7:31 pm

Re: Blacklist Filter update script

Thu Jan 12, 2017 10:28 pm

can you share your raw rules ? Im using 6.36.
I would just say put two rules in raw table:
chain=prerouting src-address-list=blacklist action=drop
chain=prerouting dst-address-list=blacklist action=drop
 
proximus
Member Candidate
Member Candidate
Posts: 119
Joined: Tue Oct 04, 2011 1:46 pm

Re: Blacklist Filter update script

Thu Jan 12, 2017 11:11 pm

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
May I please be so bold, whats the commands to change the temporary file storage location?

I use a RB750Gr3 and has a microSD card installed. Currently its only really been used for backup configs and some logs as I haven't found away to switch more to use it.
In the script, edit it use disk1. So, the relevant parts would be:
/tool fetch mode=https dst-path="/disk1/dynamic.rsc" 
/import file-name=/disk1/dynamic.rsc
/file remove disk1/dynamic.rsc
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Sun Jan 15, 2017 2:56 am

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
May I please be so bold, whats the commands to change the temporary file storage location?

I use a RB750Gr3 and has a microSD card installed. Currently its only really been used for backup configs and some logs as I haven't found away to switch more to use it.
In the script, edit it use disk1. So, the relevant parts would be:
/tool fetch mode=https dst-path="/disk1/dynamic.rsc" 
/import file-name=/disk1/dynamic.rsc
/file remove disk1/dynamic.rsc
Thanks very much. I did figure it out in the end and forgot to post here that I had found the solution. Thanks anyway for letting me know.
 
chippers
newbie
Posts: 26
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Mon Jan 30, 2017 12:12 pm

great script, I am seeing lots of hits against the listed IP's.

On reboot, is there a way to load the script automatically or do we have to wait for the scheduled update time?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jan 30, 2017 4:31 pm

you can add a second schedule to run at startup.
 
chippers
newbie
Posts: 26
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Mon Jan 30, 2017 10:20 pm

Yes, I have that but doesnt seem to work :(

i'll try to troubleshoot, thanks

/system scheduler
add interval=1d name=updateBlacklist on-event="/system script run updateBlacklist" policy=read,write,test start-time=startup
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jan 30, 2017 11:03 pm

This works and you have to be patient because the script waits for 3+30 seconds giving the interfaces time to start completely because you need access to the internet.
add name="Startup updateBlacklist" on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test start-time=startup
 
chippers
newbie
Posts: 26
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Tue Jan 31, 2017 2:30 pm

ok, turns out I wasnt being impatient :)

I copied the schedules from the start of this thread and there are a couple of issues.
1. The schedule names are the same and this causes the import of the second schedule to fail, solution is to rename the second schedule
2. The run command differed in both schedules (run updateBlacklist) VS (run blacklistUpdate)

Here is what I ended up with and works as expected. List reloads about 30 seconds after reboot.
/system scheduler
add interval=1d name=UpdateBlackListDaily on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
add name=UpdateBlackListOnReboot on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=startup
Adding these simple rules as mentioned elsewhere in this thread and the IP list is working great!
/ip firewall raw
add action=drop chain=prerouting comment=\
    "Drop connections from Blacklisted addresses" src-address-list=\
    dynamicBlacklist
add action=drop chain=prerouting comment=\
    "Drop connections to Blacklisted addresses" dst-address-list=\
    dynamicBlacklist
    
Thanks for a great contribution...
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jan 31, 2017 3:44 pm


2. The run command differed in both schedules (run updateBlacklist) VS (run blacklistUpdate)

Here is what I ended up with and works as expected. List reloads about 30 seconds after reboot.
/system scheduler
add interval=1d name=UpdateBlackListDaily on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
add name=UpdateBlackListOnReboot on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=startup
Are you sure because I see twice the same script run command.

I will check if the script can't be started in sequence. I remember that this was not a problem but you never know. ;-)

Update: I have now checked it and the list was updated automatically this afternoon. I have different names for the script and I think you also wanted you communicatited.

It works really great now and maybe a default "startup" can be added to the installation script.

I have good results and in the log I see hits on the blacklist every day.
 
mhyll
just joined
Posts: 12
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Feb 01, 2017 4:09 pm

If you want, you can use my blacklists. Blacklists updated every hour.

TOR Exit Nodes
OpenBL
SpamHaus DROP list
DShield
malc0de

RSC will create address-list named "Blacklist", IP's will be commented. Duplicate IP's will be skipped, if exists.

And of course, don't forget to schedule it and make corresponding filter rules. ;)

Script:
# Script will now download IP blacklists
/tool fetch url="http://www.securelan.eu/mikrotik/torexitnodes.rsc" mode=http;
:log info "Downloaded torexitnodes.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/openbl.rsc" mode=http;
:log info "Downloaded openbl.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/spamhaus.rsc" mode=http;
:log info "Downloaded spamhaus.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/dshield.rsc" mode=http;
:log info "Downloaded dshield.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/malc0de.rsc" mode=http;
:log info "Downloaded malc0de.rsc from SecureLAN.eu";
# Script will now replace old blacklists with the new ones
/ip firewall address-list remove [find where comment="TorExitNodes"]
/import file-name=torexitnodes.rsc;
:log info "TorExitNodes records updated successfully.";
/ip firewall address-list remove [find where comment="DShield"]
/import file-name=dshield.rsc;
:log info "DShield records updated successfully.";
/ip firewall address-list remove [find where comment="SpamHaus"]
/import file-name=spamhaus.rsc;
:log info "SpamHaus records updated successfully.";
/ip firewall address-list remove [find where comment="OpenBL"]
/import file-name=openbl.rsc;
:log info "OpenBL records updated successfully.";
/ip firewall address-list remove [find where comment="malc0de"]
/import file-name=malc0de.rsc;
:log info "Malc0de records updated successfully.";
:log info "All blacklist records were updated successfully.";
 
mhyll
just joined
Posts: 12
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Feb 01, 2017 10:03 pm

I've gone ahead and started publishing my dynamic filter list for RouterOS 6.x. My server generates the list each night after collecting data on all known botnets, C&C server, and spammers. Currently the list runs about 3k entries, so it may not work well on low end routers. Here is the script to update the list, as well as my personal firewall rules. As always, adjust them to fit your needs.  
Does your list contain also TOR network exit nodes? If not, you can probably add it. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Feb 01, 2017 10:22 pm

No it doesn't. That is not something that I am interested in blocking. I am a big privacy advocate and I don't want to take away that option
 
mhyll
just joined
Posts: 12
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Feb 01, 2017 10:31 pm

Yeah, but privacy is not always secure.... in Tor there is lot of ransomware servers hidden. No connection to TOR, no encrypted disk. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Feb 02, 2017 12:36 am

If a user is using TOR, then they are on their own for security. At this time I have no interest in blocking TOR.
 
mhyll
just joined
Posts: 12
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Thu Feb 02, 2017 12:59 am

yeah...that's true...but.. :) for me, in enterprise environment, tor should not be allowed.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Feb 02, 2017 1:05 am

Then you should filter it. However, nearly impossible to track the ever changing exit nodes, and impossible to detect.
 
mhyll
just joined
Posts: 12
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Thu Feb 02, 2017 1:07 am

That's why I am generating TOR exit nodes list every hour. :) Check my post earlier. :) Could you compare my lists with yours? Probably there's something to make better...on both. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Feb 06, 2017 8:21 pm

Sorry, not going to block TOR nodes. I am an active donor to the TOR project. It would be hypocritical of me to block it. But thank you for the input.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Feb 11, 2017 12:08 am

I wanted to give a status update on my blacklist.

As of this morning, the Blacklist has 3,500 routers downloading the list everyday. They are pulling 1.7GB of data every 24 hours. Just about 52GB per month. I have moved the handling of the blacklist to a dedicated server. I currently use 4 high-profile blacklist services, in addition to the 215 honeypots that I collect data from all over the USA.

I have watching the FCC rulings very closely, and I will not hesitate to move the servers outside of the USA if I feel the list is at risk. I am currently looking into ways of having RouterOS check a SHA256sum to verify the validity of the list.

Again, this list was started for my own use on the MikroTik routers that I manage. I do not charge for this list, and I have never asked for donations. That said, I have always been open to suggestions to make it better, but please remember that my primary concern is the safety of the medical groups and hospitals that I manage.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Feb 11, 2017 3:37 pm

I have an idea how to bring back the traffic generated by the Blacklist.

When I lookup sites I get sometimes a list of IP addresses back:
Name: microsoft.com
Addresses: 23.100.122.175
23.96.52.53
191.239.213.197
104.40.211.35
104.43.195.251
So if you can convert the list and put it in a DNS, then one record/domain name will supply all IP addresses in one go.

You could to make weekday's list like monday.blacklist.xxx / tuesday.blacklist.xxx....sunday.blacklist.xxx
Give the DNS-record a lifetime of 24+1 hour and remove the that day when it is the next day is generated and is uploaded. In this way the your are sure that the cache DNS servers up the stream are cleaned to read in that weekday.blacklist.xxx when there is an request for it in the Internet.

When a weekday*7.backlist.xxx is in the cache of the DNS in the Mikrotik, you only need one line in the address list to be able to filter. I think that a script is useful to make a hard delete of the outdated weekday to make room for the new weekday list.

The DNS of the provider/supplier which the Mikrotik owner is using is handling the traffic now. You have each day a one time upload and the the DNS structure is distributing your list for you. Delays are common and because the used weekday was not present for the last 5 days so there should be direct request to the DNS.

This way of working I already use myself and I put the extra IP addresses in the host file on the machine where my DMSmasq is running. DNSmasq reads the host file and returns the list of IP addresses when the domainname is requested. In doing so I have only need one line to be able to filter more addresses in one go.

I don't know if this is possible or even legal to use the DNS in that way.....

updated: 12 February 2017
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Mar 06, 2017 6:16 pm

Just hit 4000 active routers using the BlackList.
Notable users are T-Mobile, using it on there Fixed LTE deployments. And even more so, several US Government sites have begun pulling the list.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Mar 07, 2017 12:45 pm

Just hit 4000 active routers using the BlackList.
Notable users are T-Mobile, using it on there Fixed LTE deployments. And even more so, several US Government sites have begun pulling the list.
Good to see the grow from 2700 to 4000 clients in the last seven months.
I made a suggestion to use DNS to distribute the list and now I read again the start page of this posting and BGP also seems a solution.
The blacklist get many hits on my connection and I am pleased that those connections tries are terminated!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Mar 07, 2017 6:45 pm

DNS and BGP both complicate things dramatically. The current distribution method is very simple, stable and requires very little to setup.
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Tue Mar 07, 2017 8:47 pm

I have an idea how to bring back the traffic generated by the Blacklist.

When I lookup sites I get sometimes a list of IP addresses back:
Name: microsoft.com
Addresses: 23.100.122.175
23.96.52.53
191.239.213.197
104.40.211.35
104.43.195.251
So if you can convert the list and put it in a DNS, then one record/domain name will supply all IP addresses in one go.

You could to make weekday's list like monday.blacklist.xxx / tuesday.blacklist.xxx....sunday.blacklist.xxx
Give the DNS-record a lifetime of 24+1 hour and remove the that day when it is the next day is generated and is uploaded. In this way the your are sure that the cache DNS servers up the stream are cleaned to read in that weekday.blacklist.xxx when there is an request for it in the Internet.

When a weekday*7.backlist.xxx is in the cache of the DNS in the Mikrotik, you only need one line in the address list to be able to filter. I think that a script is useful to make a hard delete of the outdated weekday to make room for the new weekday list.

The DNS of the provider/supplier which the Mikrotik owner is using is handling the traffic now. You have each day a one time upload and the the DNS structure is distributing your list for you. Delays are common and because the used weekday was not present for the last 5 days so there should be direct request to the DNS.

This way of working I already use myself and I put the extra IP addresses in the host file on the machine where my DMSmasq is running. DNSmasq reads the host file and returns the list of IP addresses when the domainname is requested. In doing so I have only need one line to be able to filter more addresses in one go.

I don't know if this is possible or even legal to use the DNS in that way.....

updated: 12 February 2017
If you insist in doing it via DNS then look into rbldnsd which is designed for exactly this purpose. You can feed it a list of IP's/hostnames and it can respond with whatever you want. RBL's used for mail etc commonly uses this method for their black/white or rep lists.

You can do more then just this, for example this guy here http://countries.nerd.dk/more.html uses it to make a countries lookup via dns which can then be used for things like mail/web etc black/white lists etc.

Anyway, personally, the way the list is right now is best as it can easily be adapted to whatever method/way you like.

Just my two cents.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Mar 08, 2017 1:47 am

Using RBL's crossed my mind but then the amount of traffiic would be the same like it is with BGP.

When using DNS you will also have some traffic but the main part is distributed by external DNS severs as I see it.
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Wed Mar 08, 2017 1:54 am

Using RBL's crossed my mind but then the amount of traffiic would be the same like it is with BGP.

When using DNS you will also have some traffic but the main part is distributed by external DNS severs as I see it.
Distributed & cached which the cache will lower the amount of traffic needed.

However whether DNS is less then BGP traffic wise taking into effect of caching etc, i'm not sure. I think if there was enough devices pulling the data, BGP probably total up to more but thats an educated guess more then fact.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 2:35 pm

Looks like a very interesting system you got here.
I know it has been running fine for soon 2 years I guess, but I do have a few small suggestions for your update script.
  • You should escape the "?" in the URL ("\?")
  • Add brackets around the script ("{ }")
  • Add a ":put" with the script version for debugging (":put "Script version: $scriptVer"")
These changes would make you able to run the script in the terminal. Or did you intentionally write it so that it didn't work in the terminal?
Example:
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies
{
    :log warning "Blacklist download will start in 30 seconds..."
    :delay 30
    
    :local model    [/system resource get board-name]
    :local version   [/system resource get version]
    :local memory   [/system resource get total-memory]
    :local uname   [/system identity get name]
    :local scriptVer   "2016.7.4a (Deantwo)"
    :put "Script version: $scriptVer"
    
    :log warning "Downloading current Blacklist for this model"
    /tool fetch mode=https dst-path="/dynamic.rsc" \
       url="https://mikrotikfilters.com/download.php\?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer"
    
    :log warning "Disabling info logging..."
    /system logging disable 0
    
    :log warning "Removing expiring address-list entries..."
    /ip firewall address-list remove [find list="dynamicBlacklist"]
    
    :log warning "Importing current Blacklist..."
    /import file-name=/dynamic.rsc
    
    :log warning "Removing temp file..."
    /file remove dynamic.rsc
    
    :log warning "Blacklist Update Complete."
    /system logging enable 0
}
When testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?

By the way, what is the reason for wanting the router's identity? Wouldn't it be more reliable to just use the serial number?
Your script doesn't require the identity to be sent, right? I can omit it?
Last edited by Deantwo on Mon May 01, 2017 11:59 am, edited 2 times in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Mar 09, 2017 3:36 pm

When testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?
Quotes from IntrusDave
My server collects the banned IP's 24/7 and publish the list at 3am PST.
That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.
I will not manually remove addresses.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:33 pm

When testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?
Quotes from IntrusDave
My server collects the banned IP's 24/7 and publish the list at 3am PST.
That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.
I will not manually remove addresses.
Doesn't explain why the timeout of the dynamic address-list entries is only 24 hours when it is stated in the opening post that the timeout is 48 hours.
From the opening post:
The address-list entries are now Dynamic with a 48 hour timeout. This will cut the number of writes to NAND down dramatically.
If nothing else the opening post just needs to be updated.

There is a small chance that the dynamic address-list manages to timeout before the new dynamic address-list is downloaded and applied. This could leave the system vulnerable for at least a couple of seconds each day when the update script is running. It could easily be fixed by extending the timeout by an hour or less. But maybe I am just overreacting at that.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:34 pm

At one point the list was updated every 48 hours, but as malware has spread faster and responses are faster, the list now expires after 24 hours. Maybe upping that to 26 hours will help some. My routers update themselves every 23 hours. The script does run from the terminal as a whole...
/system script run UpdateBlacklist
It's not meant to be run line by line.
I use the identity to group the routers for stats and troubleshooting. Example; all of my routers ID's start with "Intrus :: " this allows me to sort them and quickly track down problems. While it's not currently required, it really is the only method that I have to keep track of how many routers are active daily. I do not use the serial number because I feel that is too invasive to request. I can not go by IP, because many are behind the same proxies. I could use the WAN MAC address, but I was betting that some would object to that too.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:45 pm

I use the identity to group the routers for stats and troubleshooting. Example; all of my routers ID's start with "Intrus :: " this allows me to sort them and quickly track down problems. While it's not currently required, it really is the only method that I have to keep track of how many routers are active daily. I do not use the serial number because I feel that is too invasive to request. I can not go by IP, because many are behind the same proxies. I could use the WAN MAC address, but I was betting that some would object to that too.
I was mostly asking because we have customer numbers and names as router identity, so I may be forced to not send you those if we start using your service.

On another note. The second scheduler in the opening post, isn't it meant to be on startup?
I use my startup scheduler scripts like this:
/system scheduler
add name="MyScheduler1" \
    start-time=startup \
    policy=read,write,test \
    on-event=":delay 120\r\
    \n/system script run \"MyScript1\""
Don't know if "start-date=jan/01/1970 start-time=00:00:0 interval=00:00:00" translate to "start-time=startup" somehow.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:56 pm

Updated the first post and the timeout to 25 hours.

The identity is never seen by anyone but me. I do have DOD clearance, so nothing to worry about.. Well, I guess that doesn't mean much now days. You are welcome to set a static name for each router in the script. The database is stored on a separate server, with no direct internet connection. As for the schedule, you will have to play with it. It was originally setup back when the routers didn't store the date and time over a reboot, so on first boot the date and time was "1970-01-01 00:00:00". RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 5:13 pm

As for the schedule, you will have to play with it. It was originally setup back when the routers didn't store the date and time over a reboot, so on first boot the date and time was "1970-01-01 00:00:00". RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.
Not 100% sure rather or not to add the "start-date=jan/01/1970" to the scheduler, since I haven't messed with them for a while. But the scheduler I posted does work, and I use a two minute delay before calling my scripts because I need to be sure that VPN tunnels are up.
 
cashwu
just joined
Posts: 4
Joined: Mon Sep 12, 2016 5:42 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 7:47 pm


RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.

Problem why scheduler cannot execute script is because script as more permissions than scheduler.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 09, 2017 7:53 pm

The startup is not a permissions issue. It has to do with the interval. When the interval is 24 hours, the first run doesn't occur until 24 hours after the boot.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue Mar 14, 2017 10:14 am

And, if you are interested, here are my filter rules:
/ip firewall address-list
add address=172.16.0.0/16 list=PrivateIPs
add address=10.0.0.0/8 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs
Found a little error in your provided example firewall.

Incorrect netmask for the 172 private range, it should be /12.
Like this:
/ip firewall address-list
add address=10.0.0.0/8 list=PrivateIPs
add address=172.16.0.0/12 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs
See: https://en.wikipedia.org/wiki/Private_network
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Mar 15, 2017 9:34 pm

You are correct. I will fix this.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue Mar 21, 2017 9:44 am

Something seems to have gone wrong.
I am receiving incomplete "dynamic.rsc" files, ending in the middle of an address-list entry add command.

Example:
# Generated on Mon Mar 20 04:00:54 PDT 2017 by Intrus Technologies
/ip firewall address-list

add list=dynamicBlacklist address=1.10.16.0/20 timeout="1d 01:00:00" comment=Blacklisted
# Omited 5226 lines.
add list=dynamicBlacklist address=42.62.51.27 timeout="1d 01:00:00" comment=Blacklisted
add list=dynamicBlacklist address=42.83.80.0/22 timeout="1d 01
Log shows that it is not always the same places that these files fail, for example:
mar/21 05:21:57 script,error script error: failure: already have such entry
mar/20 05:21:44 script,error script error: expected end of command (line 5586 column 70)
mar/19 05:21:56 script,error script error: expected end of command (line 5770 column 27)
mar/17 05:22:08 script,error script error: value of address expects range of ip addresses
mar/16 05:22:09 script,error script error: invalid time value for argument timeout
There is however a patten to how they fail.
For example the "already have such entry" error seems to be because it has created a non-dynamic address-list entry with the address 0.0.0.0 on the list "dynamicBlacklis", but the error indicate that it has done it more than once the exact same way.

My guess is that you are assuming the length of each line? But the length of the lines has changed. Maybe because it was changed from "timeout=1d" to "timeout="1d 01:00:00"" on each line, adding a total of 11 characters per line.

If you need to make the lines shorter, you could remove the comment, since it is kind of redundenet when you have the list name. Don't know if anyone rely on the comment though.

An annoying consequence of all this is that, if the import fails, info logging is never re-enabled.
/system logging enable 0
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Mar 21, 2017 4:57 pm

You issue is that the router simply didn't complete the download. Today's download is 603k. If it's getting out off, you may want to see if your ISP is trying to proxy ssl connections.
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 205
Joined: Wed May 20, 2015 10:14 pm
Location: Lake Grove, NY

Re: Blacklist Filter update script

Tue Mar 21, 2017 7:32 pm

Hi! I'm trying to put this rules in a CCR1072, this router has direct connection to the internet without any restriction, but when i tried to fetch the first file got this message:

/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https
status: failed

failure: connection timeout

It seems to be connecting, but before a few minutes it stops everything, i tried to dowload this file via browser and it works, but when i run the script in the CCR1072 i got the same error. Do you have any suggestion to fix this issue?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Mar 22, 2017 6:44 am

Unfortunately, I don't know how to help you with this. I don't see any errors in my server logs. I can only assume that you are getting ssl errors. You should be able to manually install the scripts from the first post.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Wed Mar 22, 2017 10:46 am

You issue is that the router simply didn't complete the download. Today's download is 603k. If it's getting out off, you may want to see if your ISP is trying to proxy ssl connections.
Ok yeah, maybe I was a little hasty to my conclusion.
I am able to download the file just fine from the company network, but this one customer router seems to have the issue.
[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 496KiB
       total: 603KiB
    duration: 3s

[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 336KiB
       total: 603KiB
    duration: 2s

[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 510KiB
       total: 603KiB
    duration: 2s

[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 460KiB
       total: 603KiB
    duration: 3s
Looks like it isn't downloading the full file, and MikroTik does nothing to check this it seems.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Mar 22, 2017 3:26 pm

I don't even know where to start with that. Maybe MTU? running pppoe? ssl proxy? wrong MTU? anything different about this router over others?
 
blackzero
newbie
Posts: 25
Joined: Tue Aug 09, 2011 3:40 pm

Re: Blacklist Filter update script

Thu Mar 23, 2017 11:46 am

Your two schedulers don't seem to work as the name for either is conflicting each others. Renaming it will work. Maybe you need to mention this in your first post.

Thanks for the good work.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Mar 24, 2017 2:20 am

Schedules are allowed to have the same name.

--

The server side was updated today. I was forced to make the server require the identity. The public IP and Identity are used for accounting so I can track the bandwidth and number of requires. I understand that some will object to this, and I will provide a full refund to those. (ha ha..)

The list hit 4500 active users this afternoon.

Anyone have thoughts on using the WAN MAC address instead of the identity?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Mar 24, 2017 10:29 am

Indeed the identity name is very common and besides the MAC also the ID in the VPN nanme in the quickset screen which is quite unique.

f7c4250638xxxxxx.sn.mynetname.net which contains the serial of the box reversed.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 30, 2017 8:50 pm

Some interesting stats...
+-----+--------------------+
| QTY | model              |
+-----+--------------------+
| 721 | RB951G-2HnD        |
| 548 | RB2011UiAS-2HnD    |
| 374 | RB2011UiAS         |
| 309 | hAP+ac             |
| 298 | RB951Ui-2HnD       |
| 182 | RB751G-2HnD        |
| 178 | CCR1016-12G        |
| 174 | SXT+Lite5          |
| 166 | CCR1009-8G-1S-1S+  |
| 159 | RB3011UiAS         |
| 148 | hAP+lite           |
| 114 | RB850Gx2           |
| 112 | RB450G             |
| 102 | RB750GL            |
|  94 | RB750              |
|  82 | hEX                |
|  81 | CCR1036-12G-4S     |
|  78 | RB1100AHx2         |
|  68 | hAP+ac+lite        |
|  65 | RB2011UAS          |
|  64 | SXT+LTE            |
|  54 | CRS109-8G-1S-2HnD  |
|  53 | CHR                |
|  52 | x86                |
|  47 | RB493G             |
|  45 | hEX+lite           |
|  40 | mAP                |
|  40 | hAP                |
|  30 | CCR1009-8G-1S      |
|  30 | RB912UAG-2HPnD     |
|  28 | RB912UAG-5HPnD     |
|  25 | RB+Groove+5Hn      |
|  22 | mAP+lite           |
|  21 | CCR1036-8G-2S+     |
|  20 | CRS125-24G-1S      |
|  18 | RB2011UAS-2HnD     |
|  17 | RB751U-2HnD        |
|  16 | RB2011L            |
|  15 | RB2011iL           |
|  12 | RB750UP            |
|   8 | CCR1016-12S-1S+    |
|   6 | RB1100             |
|   6 | RB1200             |
|   6 | RB951-2n           |
|   5 | CRS125-24G-1S-2HnD |
|   4 | RB1100AH           |
|   4 | RB750G             |
|   4 | RB2011iLS          |
|   4 | RB433              |
|   2 | OmniTIK+5+ac       |
|   2 | CRS226-24G-2S+     |
|   2 | RB1100Hx2          |
|   2 | hEX+PoE            |
|   2 | hEX+PoE+lite       |
|   2 | %24model           |
|   2 | CCR1009-7G-1C      |
|   2 | CCR1009-7G-1C-1S+  |
|   2 | RB2011LS           |
|   1 | RB+SXT+5HnD        |
|   1 | RB433AH            |
|   1 | RB800              |
|   1 | GrooveA+52         |
|   1 | CCR1072-1G-8S+     |
|   1 | PowerBOX           |
|   1 | RB750r2            |
|   1 | SXT+Lite5+ac       |
|   1 | RB333              |
|   1 | 911+Lite5+dual     |
|   1 | RB1100AH2X         |
|   1 | RB1000             |
|   1 | RB911G-5HPnD       |
|   1 | RB+OmniTIK+U-5HnD  |
|   1 | RB493              |
|   1 | RB450              |
|   1 | BaseBox+5          |
|   1 | wAP+ac             |
|   1 | RB600              |
|   1 |                    |
+-----+--------------------+
 
User avatar
Rhoos
just joined
Posts: 22
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Mon Apr 03, 2017 3:27 am

I am a beginner at Mikrotik and my knowledge of networking is limited, for that reason my biggest thanks to people like "IntrusDave" and all who have collaborated with this magnificent work to keep our home networks safe. Thank you very much!
 
toxicfusion
Member Candidate
Member Candidate
Posts: 267
Joined: Mon Jan 14, 2013 6:02 pm

Re: Blacklist Filter update script

Wed Apr 05, 2017 6:41 pm

I just went ahead and downloaded your script and applied to one of my MikroTiks for testing. So far so good! I'll roll this out to my client devices very soon, added security is always welcomed.

Thanks for a great contribution!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Apr 06, 2017 6:54 pm

Glad it's working out for you.
List usage jumped from 4800 to 5100 in the last two days.
 
mhyll
just joined
Posts: 12
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Mon Apr 10, 2017 10:39 pm

Your firewall rules are great. Only DST-NAT is not working....

Last two filter rules needs to be modded like this:
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1 connection-nat-state=!dstnat
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2 connection-nat-state=!dstnat
By the way...do you know what's happened with OpenBL?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 10, 2017 10:49 pm

The rules are just examples, and should always be adjusted to suit the needs of the network.

I don't know what's going on with OpenBL. I can only assume they have either shut down, or are under DDoS.
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 205
Joined: Wed May 20, 2015 10:14 pm
Location: Lake Grove, NY

Re: Blacklist Filter update script

Tue Apr 11, 2017 6:37 pm

Unfortunately, I don't know how to help you with this. I don't see any errors in my server logs. I can only assume that you are getting ssl errors. You should be able to manually install the scripts from the first post.
I dont know why, but it finnally works in my CCR1072, thanks for your help!
 
xlighting
just joined
Posts: 6
Joined: Wed Apr 02, 2014 6:08 pm

Re: Blacklist Filter update script

Wed Apr 12, 2017 7:33 am

Hello, Dave:
I have noticed that the rule file is now less than 100kb(<1000 filter entries), but you said it was 600kb+ in March 2017, so I'm wondering if my downloading is incomplete;
I've try downloading via different Internet connection( China/HK) and via different Routerboards(RB951G and RB750Gr3) but same result..
I've also try downloading via Chrome, but seems you've restricted downloading via Routerboard only;
(I was able to download a >5000 filter file in March with same device and same Internet connection)
I've checked the .rsc file downloaded, and not seen any “broken/ending in the middle of line”

so, is there anything I can do to further investigate where the problem is?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Apr 12, 2017 4:35 pm

OpenBL is currently offline. So right now the filters are limited to my internal sources.
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Wed Apr 19, 2017 9:55 am

Thanks for the update script, it is working perfectly on my RB2011UIAS-2HnD-IN.

However, I have some trouble to get it working on my RB3011UIAS-RM. When I execute:
/system script run updateBlacklist;
I get a failure: closing connection: <400 Bad Request> 172.102.241.58:443 (4).
The script on both routers is exactly the same...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Apr 19, 2017 5:39 pm

Every time that I have seen a 400 Error, it is because the Copy/Paste didn't work. Something is the script is wrong... Maybe it has extra formatting, or maybe invalid characters. Make sure the OS that you are using supports UTF-8. Try copying and pasting the script to Notepad, and then copying and pasting into WinBox.
 
rioven
just joined
Posts: 6
Joined: Mon Dec 15, 2014 5:19 am

Re: Blacklist Filter update script

Thu Apr 20, 2017 2:00 pm

Unfortunately, OpenBL gonna stop its update by end of this month
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:23 am

I don't blame them. Over the last 3 months my block list has gone from 5k entries to 30k. With most of the attacks coming from Russia and China. I'm starting to consider blocking all of Russia's IP ranges. I know that isn't good for most of the world, but my networks here in the USA are under constant attack from them.
 
brianlewis
Member Candidate
Member Candidate
Posts: 134
Joined: Tue Jul 20, 2004 10:54 am
Location: Irvine, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:26 am

As great as this resource has been, in the last week it has started to block huge /16 blocks including most of Vietnam, Shopify, and many other networks that shouldn't be just added in huge /16, /19, and /24 blankets. Obviously this resource allows us to control what we want to do about these ip ranges, ie just block for specific ports or block entirely. Since we were blocking entirely the phone has been ringing off the huge by very upset customers not able to route to many areas of the world. Maybe its time to split this filter into different lists based on aggressive huge /16,/24 ranges being blocked or conservative where only specific ips or smaller /24 ranges are blocked based on their danger.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:32 am

The filters are intended to be used as incoming filters, not outgoing. If you change your rules to only block new connections coming in on the WAN interface, all should be good. I don't recommend using the list with the RAW filters.

By blocking incoming on the WAN and new connections, you prevent the attacks, but you do not block new outbound connections.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:35 am

On that note - what is really pissing me off is that big hosts like AWS and Google aren't doing anything about shutting down the attacks coming from their networks. Much of the spam is coming from AWS servers that change IP's every hour. So the only way to stop them is to block the whole subnet.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:37 am

Oh, and I ran some tests today. Filtering based on IP *ONLY* and not subnet.. the download was 112M and had over 2M entries.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:50 pm

The filters are intended to be used as incoming filters, not outgoing. If you change your rules to only block new connections coming in on the WAN interface, all should be good. I don't recommend using the list with the RAW filters.

By blocking incoming on the WAN and new connections, you prevent the attacks, but you do not block new outbound connections.
I am confused by this about using RAW. Using the filters for incoming traffic in the RAW part not as efficient?

For outgoing I use a DNS filter and out of band port filtering for new connections in Mangle.

It is really bad out there and have lots of connections wanting to deliver mail which I don't want. It in waves since a few months and sometimes there is it quiet for days and then it starts again.
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon Apr 24, 2017 9:46 am

Every time that I have seen a 400 Error, it is because the Copy/Paste didn't work. Something is the script is wrong... Maybe it has extra formatting, or maybe invalid characters. Make sure the OS that you are using supports UTF-8. Try copying and pasting the script to Notepad, and then copying and pasting into WinBox.
Thanks for your reply, I will try pasting it via Notepad later this week.

Regarding to the shutdown of OpenBL, is there any other alternative for an updated blacklist?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 24, 2017 4:35 pm

I don't know. I stopped using OpenBL a while back.
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon Apr 24, 2017 10:13 pm

I tried to paste the code in Notepad first, but still I get the 400 bad request error...
Are you sure the RB3011UIAS-RM is supported?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 24, 2017 10:28 pm

That is the same unit I use for writing my scripts. I have just over 500 of them pulling the list every morning. The error you posted is almost always a simple format or encoding error.
Last edited by IntrusDave on Wed Apr 26, 2017 12:00 am, edited 1 time in total.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue Apr 25, 2017 4:54 pm

hi,
Im using this rule for dnymic blacklist
chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 
chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 
chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 
chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist 
is there any way to keep some ip's without block I mean some Exceptions?

Thanks
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Apr 25, 2017 6:53 pm

Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Wed Apr 26, 2017 8:43 am

Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
should it like this ? and what about order of the rules ? is that correct ?
8 chain=prerouting action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

 9 chain=output action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

10;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 

11 ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 

12 ;;; BlackList
      chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 

13;;; BlackList
      chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist 
Thanks
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Wed Apr 26, 2017 9:41 pm

That is the same unit I use for writing my scripts. I have just over 500 of them pulling the list every morning. The error you posted is almost always a simple format or encoding error.
Can you send me the script you are using?
I even get the error when I use your automated installer script...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Apr 27, 2017 3:00 am

Try downloading directly from here: https://mikrotikfilters.com/updateBlacklist.rsc
Unfortunately, I don't have a router that gets this error, so I really can't troubleshoot it.

If one of you want to give me access to a router that is having a problem with the script, I can try and figure out what the problem is.
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon May 01, 2017 3:01 pm

Dave,

Can you give me an update URL without or with preset variables?
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local scriptVer   2016.7.4a
"https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";
I think it is going wrong with the URL containing (maybe unknown) variables.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue May 02, 2017 11:14 am

Can you give me an update URL without or with preset variables?
...
I think it is going wrong with the URL containing (maybe unknown) variables.
Like this?
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=750&version=6.39&memory=33554432&id=mk13139&ver=DeanHelp";
Last edited by Deantwo on Fri Aug 10, 2018 3:23 pm, edited 5 times in total.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue May 02, 2017 11:46 am

Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
should it like this ? and what about order of the rules ? is that correct ?
8 chain=prerouting action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

 9 chain=output action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

10;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 

11 ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 

12 ;;; BlackList
      chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 

13;;; BlackList
      chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist 
Thanks
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Tue May 02, 2017 11:46 am

Can you give me an update URL without or with preset variables?
...
I think it is going wrong with the URL containing (maybe unknown) variables.
/tool fetch mode=https dst-path="/dynamic.rsc" \
   url="https://mikrotikfilters.com/download.php\?get=dynamic&model=750&version=6.39&memory=33554432&id=mk13139&ver=DeanHelp";
Like that?
Yes exactly!
I got it working last night using the variables of my RB2011UAS-2HnD, hardcoding them into the update URL of the RB3011UIAS-RM:
/tool fetch mode=https dst-path="/dynamic.rsc" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=RB2011UAS-2HnD&version=6.38.5+(stable)&memory=128.0MiB&id=MikroTik+router&ver=2016.7.4a";
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue May 02, 2017 5:06 pm

I think it is going wrong with the URL containing (maybe unknown) variables.
Out of curiosity, what does your router say to the following if you paste it in the terminal?
:put [/system resource get board-name]
:put [/system resource get version]
:put [/system resource get total-memory]
:put [/system identity get name]
Last edited by Deantwo on Fri Aug 10, 2018 3:20 pm, edited 1 time in total.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 03, 2017 12:13 am

Give this a try...
# Import Intrus Managed Filter Lists
# ©2016-2017 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
# :delay 10

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2017.5.2b

:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}


:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
 
User avatar
Rhoos
just joined
Posts: 22
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Thu May 04, 2017 8:05 am

[quote="mhyll"]Your firewall rules are great. Only DST-NAT is not working....

Last two filter rules needs to be modded like this:
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1 connection-nat-state=!dstnat
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2 connection-nat-state=!dstnat

Thanks to your observation, I was able to make my security cameras visible from outside my house; However for the "raw" rule in prerouting the "connection-nat-state =! Dstnat" is not possible, and I have it disabled.
You know how I could make this rule work without blocking the cameras, Thanks!

Resolved!!!!

I had to put the rules of accepting the list of white IPs from first into "RAW", and everything was fine now. Thanks!!!!!
You do not have the required permissions to view the files attached to this post.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Mon May 08, 2017 9:08 am

Hi,

I need help for this script. Im using this script but some times my wan adress comes to the list OR my ipblock. i create new rule to accept my ip but traffic down when i try to use. here is my rules, can some one help me to solve this issue ?

Thanks.
8  ;;; Exceptions
      chain=prerouting action=accept log=no log-prefix="" src-address-list=exceptions 

 9  ;;; Exceptions
      chain=prerouting action=accept log=no log-prefix="" dst-address-list=exceptions 

10  ;;; Exceptions
      chain=output action=accept log=no log-prefix="" src-address-list=exceptions 

11  ;;; Exceptions
      chain=output action=accept log=no log-prefix="" dst-address-list=exceptions 

12    ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 

13    ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 

14    ;;; BlackList
      chain=output action=drop log=no log-prefix="blcklist src" src-address-list=dynamicBlacklist 

15    ;;; BlackList
      chain=output action=drop log=no log-prefix="blcklist dst" dst-address-list=dynamicBlacklist 

16    chain=prerouting action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=tcp src-address-list=!secure 

17    chain=prerouting action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=udp src-address-list=!secure 

18    chain=output action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=tcp src-address-list=!secure 

19    chain=output action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=udp src-address-list=!secure 

20    chain=prerouting action=drop in-interface=wan dst-port=53 log=no log-prefix="" protocol=tcp 

21    chain=prerouting action=drop in-interface=wan dst-port=53 log=no log-prefix="" protocol=udp 
Last edited by amt on Mon May 08, 2017 3:49 pm, edited 1 time in total.
 
mk13139
just joined
Posts: 11
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon May 08, 2017 2:36 pm

Give this a try...
# Import Intrus Managed Filter Lists
# ©2016-2017 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
# :delay 10

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2017.5.2b

:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}


:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Thanks! I will check it out when I'm on location.
 
User avatar
Squidblacklist
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Wed Jun 26, 2013 11:06 am

Re: Blacklist Filter update script

Mon May 15, 2017 1:47 pm

it's very possible to do that, but I would need to see what the impact on the routers would be. I'm not a big fan of the built-in DNS as it is and I'm not sure how well it would hold up with several thousand hostnames added to it.
Actually, Im glad to inform you today that the current release has added a new patch for greatly improved import speed for the importing of static dns entries, one thing you will notice is that, the cpu usage is no longer at 100% during import and the import process is much faster. I will be doing some benchmarks of RouterOS before and after the patch to demonstrate the difference, and it is a remarkable improvement indeed.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon May 15, 2017 6:24 pm

My list will not be moving to DNS. It over complicates the process and provides little if any advantages.
 
plisken
Forum Guru
Forum Guru
Posts: 2509
Joined: Sun May 15, 2011 12:24 am
Location: Belgium
Contact:

Re: Blacklist Filter update script

Fri May 19, 2017 9:45 am

What is the command to write the blacklist to a usb-stick?
Thanks
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri May 19, 2017 6:38 pm

The list is stored in memory while active.
If you need to use a flash drive for the update, just add the path of the usb drive to the path of the fetch and import lines.
 
Jacka
Member Candidate
Member Candidate
Posts: 125
Joined: Thu Jan 13, 2011 11:34 am

Re: Blacklist Filter update script

Mon May 22, 2017 10:43 am

Hi,

First of all thank you for this great script. I have a few questions:
1. Why there are 2 schedules? And if there are 2 it can't have the same name as in your example.
 /system scheduler
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=05:00:0
/system scheduler
add interval=00:00:00 name=UpdateBlackList on-event="/system script run blacklistUpdate" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=00:00:0
2. What kind of chain is this "Attacks" ? It should be input or forward chain, am I right ?
/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Mon May 22, 2017 1:44 pm

1. Why there are 2 schedules? And if there are 2 it can't have the same name as in your example.
 /system scheduler
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=05:00:0
/system scheduler
add interval=00:00:00 name=UpdateBlackList on-event="/system script run blacklistUpdate" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=00:00:0
Two schedulers can have the same name, it is weird to have though.

Also not sure the second scheduler is totally correct. At least it can be written better.
See: viewtopic.php?f=9&t=98804&start=150#p587752
2. What kind of chain is this "Attacks" ? It should be input or forward chain, am I right ?
/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
The "Attacks" chain is a custom chain, take a look at the jump rules further down.
/ip firewall filter
#...
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
#...
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
You can read more here: https://wiki.mikrotik.com/wiki/Manual:I ... all/Filter

The firewall shown in the first post is just an example, you might want you edit it for your own use or make your own.
Last edited by Deantwo on Fri Aug 10, 2018 3:24 pm, edited 1 time in total.
 
Jacka
Member Candidate
Member Candidate
Posts: 125
Joined: Thu Jan 13, 2011 11:34 am

Re: Blacklist Filter update script

Mon May 22, 2017 3:16 pm

Thanks for explanation.
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue May 23, 2017 8:06 am

can some one share firewall rules for this script ? which is correct ? using raw or filter for this script. also sometimes my puplic ip's come to the list. how can i remove them ? and how can i create expection rule for them ?

Thanks.
 
dmcosta
just joined
Posts: 8
Joined: Wed Oct 30, 2013 3:47 pm

Re: Blacklist Filter update script

Wed May 24, 2017 7:58 pm

Hello Dave!

Works perfectly on hAp AC lite , RB951G-2Hnd.

Thank you very much for this! Great work!!

cheers
 
b3h3m07h
newbie
Posts: 40
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Mon May 29, 2017 6:43 am

Here is the script i use to save and execute the blacklist (dynamic.rsc) from a usb drive (named usb1)

/system script add name=updateBlacklist-usb1 owner=admin policy=read,write,test source="# Import Intrus Managed Filter Lists\r\n# (C)2016 David Joyce, Intrus Technologies\r\n\r\n:log warning \"Blacklist update in 30 seconds\";\r\n# :delay 10\r\n\r\n:local model \t[/system resource get board-name]\r\n:local version\t[/system resource get version]\r\n:local memory\t[/system resource get total-memory]\r\n:local uname\t[/system identity get name]\r\n:local scriptVer 2017.5.2a\r\n\r\n:local name \"\"\r\n:local ver \"\"\r\n\r\n:for i from=0 to=([:len \$uname] - 1) do={ \r\n :local char [:pick \$uname \$i]\r\n :if (\$char = \" \") do={ :set \$char \"%20\" }\r\n :set name (\$name . \$char)\r\n}\r\n\r\n:for i from=0 to=([:len \$version] - 1) do={ \r\n :local char [:pick \$version \$i]\r\n :if (\$char = \" \") do={\r\n :set \$char \"%20\"\r\n }\r\n :set ver (\$ver . \$char)\r\n}\r\n\r\n\r\n:log warning \"Downloading current Blacklist for this model\";\r\n/tool fetch mode=https dst-path=\"/usb1/dynamic.rsc\" \\\r\n\turl=\"https://mikrotikfilters.com/download.ph ... \n\r\n:log warning \"Disabling info logging...\";\r\n/system logging disable 0\r\n\r\n:log warning \"Removing expiring address-list entries...\";\r\n:foreach i in=[/ip firewall address-list find ] \\\r\n\tdo={ :if ( [/ip firewall address-list get \$i list] = \"dynamicBlacklist\" ) \\\r\n do={ /ip firewall address-list remove \$i } }\r\n\r\n:log warning \"Importing current Blacklist...\";\r\n/import file-name=/usb1/dynamic.rsc\r\n\r\n:log warning \"Removing temp file...\";\r\n/file remove usb1/dynamic.rsc\r\n\r\n:log warning \"Blacklist Update Complete.\";\r\n/system logging enable 0"

so far, so good.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Mon May 29, 2017 7:47 am

Could you please edit your post and use "Code" tag to paste script content once again. It is hard to read it now.
 
b3h3m07h
newbie
Posts: 40
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Mon May 29, 2017 8:58 am

here you go :-)
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
# :delay 10

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2017.5.2a

:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}


:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/usb1/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/usb1/dynamic.rsc

:log warning "Removing temp file...";
/file remove usb1/dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 12:36 am

I've updated the script with support for USB Flash as well as the new RB1100AHx4 with internal storage.
I has also reworked the backend and script for more accurate accounting. Please update your scripts.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk1/dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30c

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
Also - The server is now blocking routers that excessively download the list. (I have several people that are trying to update every minute.)
Last edited by IntrusDave on Wed May 31, 2017 8:03 am, edited 1 time in total.
 
b3h3m07h
newbie
Posts: 40
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Wed May 31, 2017 5:03 am

Just made a few changes to the script as it didn't seem to delete the blacklist at the end
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "usb1/"
:global datafile "dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30b

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/$datapath$datafile" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="/$datapath$datafile"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove "$datapath$datafile"

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
 
cashwu
just joined
Posts: 4
Joined: Mon Sep 12, 2016 5:42 pm

Re: Blacklist Filter update script

Wed May 31, 2017 6:22 am

Just tried it, the results stopped here ...
Set:
:global datapath "disk-8G/"
:global datafile "dynamic.rsc"
未命名.png
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 6:37 am

Try this
:global datapath "/disk-8G/"
 
b3h3m07h
newbie
Posts: 40
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Wed May 31, 2017 7:19 am

try this, worked fine on my rb2011 and usb drive
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk-8G/"
:global datafile "dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30b

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/$datapath$datafile" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="/$datapath$datafile"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove "$datapath$datafile"

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
These lines were also changed

/tool fetch mode=https dst-path="/$datapath$datafile" \
url="https://mikrotikfilters.com/download.ph ... id=$softid";

/import file-name="/$datapath$datafile"

/file remove "$datapath$datafile"
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 7:41 am

Yup, clearly a problem with the remove. I can't seem to get it to accept a variable
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 8:05 am

Okay, I've updated the script again. It didn't like having the path and filename separate.

# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk1/dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30c

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
 
cashwu
just joined
Posts: 4
Joined: Mon Sep 12, 2016 5:42 pm

Re: Blacklist Filter update script

Wed May 31, 2017 8:16 am

Hello Dave & b3h3m07h, thank you for your reply.

The latest version works fine.

And then your reply, to understand the difference between the two ways.

Once again thank you.
 
b3h3m07h
newbie
Posts: 40
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Wed May 31, 2017 8:19 am

Nice work. All good here.
 
dmcosta
just joined
Posts: 8
Joined: Wed Oct 30, 2013 3:47 pm

Re: Blacklist Filter update script

Wed May 31, 2017 4:24 pm

Thanks Dave for the update! Great work!
 
aboiles
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 5:09 pm

Hello Dave,
I am now getting an error when I run the script-

url="https://mikrotikfilters.com/download.ph ... model&vers
ion=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
status: failed

failure: closing connection: <404 Not Found> 172.102.241.58:443 (4)

The script worked fine last night, but is now failing with the same error on the four routers it's running on.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 5:12 pm

Your URL is wrong.
Note the ? between "download.php" and "get"
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid"
 
aboiles
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 5:36 pm

Hello Dave,
The script has the ?, when pasted in terminal it disappears.
The log only has an entry of-
script error: expected command name (line 1 column 1)
The downloaded dynamic.rsc only has one line-
All fields are required. Please update your script.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 5:39 pm

That would mean that you need the current script. It's available in the first post.
 
aboiles
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 6:16 pm

tried both the auto installer - script ver 2017.5.2a
and the code ver 2017.5.30c.
Am still getting the All fields are required. Please update your script. dynamic.rsc
 
aboiles
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 8:14 pm

slightly modified the script by removing the extra spaces in the local info section and now have it running on a 2011UiAS-2HnD, 951G-2HnD and a CRS125-24G-1S-2HnD.
still fails on a CHR with the same error-All fields are required. Please update your script. dynamic.rsc
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 8:27 pm

Sorry man. More than 500 routers already updated and working with the new script. You are having copy/paste issues. I can't fix that for you.
 
aboiles
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 8:47 pm

Don't know Dave,
I think it may have something to do with the software-id.
I'm getting a blank for software-id from the chr's.

[admin@router] > :put [/system resource get board-name]
CHR
[admin@router] > :put [/system resource get version]
6.40rc15 (testing)
[admin@router] > :put [/system resource get total-memory]
2071535616
[admin@router] > :put [/system identity get name]
router
[admin@router] > :put [/system license get software-id]

[admin@router] >
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Jun 01, 2017 5:31 am

I'm having issues too, started out with the autoinstaller script in your first post, it always leads with
[admin@TaylorMikrotik] >> /import updateBlacklist.rsc;                                                 
syntax error (line 62 column 11)
I edited it and put start-time=startup on the same line as the last add for scheduler.

That worked but then it complains about that name already existing, which is from the double schedules? not sure why there are two named the same.

After editing this tho, I get what others have gotten.
All fields are required. Please update your script.
i have not copy or pasted anything outside of downloading the script from winbox to my pc, open in notepad++ with it in UNIX lineending mode, and put that part on the right line, then re uploaded to the tik.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 6:48 am

I'm guessing that everyone with issues are running CHR. I've found the problem and I'm working on a fix right now. I'll post the update in about an hour.
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Jun 01, 2017 6:56 am

I am on a RB951Ui-2HnD
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 6:59 am

I am on a RB951Ui-2HnD
can you post the /system license print ?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 7:40 am

I've updated the script to deal with the CHR using system-id instead of software-id. Annoying that they are different...

I've tested on the following units with no failures.
CCR1009-7G-1C-1S+
CCR1009-8G-1S-1S+
CCR1016-12G
CCR1036-12G-4S
CHR
CRS109-8G-1S-2HnD
CRS125-24G-1S
CRS125-24G-1S-2HnD
hAP+ac
hAP+ac+lite
hEX
RB2011UAS-2HnD
RB2011UiAS
RB2011UiAS-2HnD
RB3011UiAS
RB450G
RB951G-2HnD
RB951Ui-2HnD
x86
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 7:42 am


syntax error (line 62 column 11)[/code]
I found the line 62 error and corrected it. delete the items you have, and reinstall. it should be good to go.
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Jun 01, 2017 8:03 am

Seems the CHR changes, fixed mine too... fyi the output was
[admin@TaylorMikrotik] >> /system license print     
  software-id: 15LP-6RVD
       nlevel: 4
     features: 
 
aboiles
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Thu Jun 01, 2017 8:27 am

Thanks Dave,
Script works great on the CHR now!
 
User avatar
amt
Long time Member
Long time Member
Posts: 529
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Thu Jun 01, 2017 10:06 am

Hi,
I had this erorr All fields are required. Please update your script
here log;
10:05:19 script,warning Downloading current Blacklist for this model
10:05:20 info fetch: file "dynamic.rsc" downloaded
10:05:20 script,warning Disabling info logging...
10:05:20 script,warning Removing expiring address-list entries...
10:05:20 script,warning Importing current Blacklist...
10:05:20 script,error script error: expected command name (line 1 column 1)

I update script with this;
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk1/dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30c

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
and now working.
thanks
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:02 am

Glad it's working for everyone now. Stats are MUCH more accurate now. The server was starting to block devices behind NAT routers because it thought some were downloading hundreds of times per hour. Now it sees each as a separate device.
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:10 am

Awesome! Thanks for still doing this. Now that you got more stats, you should create some public pages cause i love me some random statistics!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:14 am

Awesome! Thanks for still doing this. Now that you got more stats, you should create some public pages cause i love me some random statistics!
I was just starting on a page that shows each type and number of routers that pulls the list.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:58 am

I've cleared all my starts and started fresh. Here is a quick and dirty stats page on the hardware accessing the list.

https://mikrotikfilters.com/blstats.php
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jun 02, 2017 4:18 pm

Data taken from 28 days of router uptime

blackmail is my list composed from addresses dropped with greylisting

98 395 - total started SMTP sessions
8362 - unique smtp src addresses
7 515 - dropped by my blackmail list
70 596 - dropped by Dave's list
bm.PNG
You do not have the required permissions to view the files attached to this post.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jun 03, 2017 11:10 am

Many thanks for all your work and I was following this thread with great interest and checked this morning if I needed to update my script. That was the case and since three days I had a error on line one.

So I downloaded the latest script and imported it after removing the running script. The are some things I had to change: user djoyce --> admin to get the line back in my log where the dynamic.rsc has been downloaded memory info fetch: file "dynamic.rsc" downloaded. I set the start delay time to 30 seconds because I have a PPPoE connection that takes a bit longer to come up after reboot.

Default the location of the dynamic.rsc is now disk1 and that is ok by me because I have an SD card in my RB750Gr3 but I can change that to flash (mirrored in RAM) again if I like.

I also noticed that on importing updateBlacklist.rsc to the script that I got: /import updateBlacklist.rsc; failure: item with this name already exists despite it did not exist until after the import.
 
pkrexer
just joined
Posts: 20
Joined: Sat May 21, 2016 4:39 pm

Re: Blacklist Filter update script

Mon Jun 05, 2017 3:56 am

Not sure why its not working all of a sudden. I updated the script a few days ago and was working as of yesterday... Now when the script runs, it says its downloading the blacklist but nothing else happens.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 05, 2017 3:58 am

Not sure why its not working all of a sudden. I updated the script a few days ago and was working as of yesterday... Now when the script runs, it says its downloading the blacklist but nothing else happens.
What are the last two octets of the public IP?
 
pkrexer
just joined
Posts: 20
Joined: Sat May 21, 2016 4:39 pm

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:05 am

11.204
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:12 am

Fixed. Sorry about that. typo in the code.
 
pkrexer
just joined
Posts: 20
Joined: Sat May 21, 2016 4:39 pm

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:16 am

np! Thanks for the quick fix... Appreciate all the work you do!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:17 am

No problem at all. I enjoy it.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 07, 2017 1:30 pm

I noticed today when I started Firefox that I were getting hits on the blacklist. I followed the IP and found that it lead to hackademix.net and secure.informaction.com and looking on the site it was probably an plug-in was generating the hits and that was No-script. I use this plug-in for years and I allow or disallow the default running of scripts filtered on the domain the are served by.

Plugin homepage: https://noscript.net/

Name: secure.informaction.com
Addresses: 69.195.158.194
69.195.158.198
69.195.158.197
69.195.158.195
69.195.158.196

I understand how the blacklist is build and that it based on bad traffic and if there is a problem of a domain being misused then I can contact them to ask to look if they are hacked in any way?

Found it and I don't know why I did not see it before: the block is: 69.195.158.0/24 in the dynamicblacklist
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 08, 2017 3:17 am

I've updated the statistics page today. It now normalizes the memory and shows the percentage of each category
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 08, 2017 3:21 am

I noticed today when I started Firefox that I were getting hits on the blacklist. I followed the IP and found that it lead to hackademix.net and secure.informaction.com and looking on the site it was probably an plug-in was generating the hits and that was No-script. I use this plug-in for years and I allow or disallow the default running of scripts filtered on the domain the are served by.

Plugin homepage: https://noscript.net/

Name: secure.informaction.com
Addresses: 69.195.158.194
69.195.158.198
69.195.158.197
69.195.158.195
69.195.158.196

I understand how the blacklist is build and that it based on bad traffic and if there is a problem of a domain being misused then I can contact them to ask to look if they are hacked in any way?

Found it and I don't know why I did not see it before: the block is: 69.195.158.0/24 in the dynamicblacklist
I'm not sure what you are asking here. You are always welcome to contact a site and ask them to fix any issues. The subnet will be removed from the list automatically once whatever issue they were having is fixed. Many times it's that they are hosting a botnet that they do not even know about. Other times it may be that they are serving viruses in ads. AWS and Google Compute have both been blocked several times because they refuse to take down a virtual host that is being used to attack other networks.
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Thu Jun 08, 2017 8:37 pm

Hi Dave,

I use your dynamic blacklist and it suits well.
For some reason a subnet from which I really need to use 1 address from appeared in one of the lists loaded into dynamicBlacklist ...
It would be a great help if I could whitelist a subnet inside the script somehow ...

Keep up the great work

Eddie
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 08, 2017 11:00 pm

Whitelisting is accomplished by creating a new address-list and a new filter rule.

1) Create an address list - say.. "Whitelist" and add the IP addresses that you need never be blocked.
2) create a new filter "Accept" rule, using the src-address-list you created.
3) place the new Whitelist Accept rule ABOVE the blacklist Drop rule.

There is no need to modify the script, and this can not be done on the server side.
Please keep in mind that it's always better to understand why the IP/Subnet ended up on the blacklist and attempt to get that corrected first. I have seen several networks penetrated because an admin whitelisted an address that was serving malware, instead of contacting that site/service and getting the issue resolved.
 
eddieb
Member
Member
Posts: 305
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Fri Jun 09, 2017 9:11 am

Hi Dave,

thanks for your response. It was a bit complex but I did manage to add the subnet in a whitelist and that works for now...
I am aware of the problems that might cause such a whitelist. In this case the subnet is from a local provider with many customers fighting spam ...
sometimes one of them gets blacklisted for that reason and sometimes the entire subnet is.
To do make sure that 2 of the servers within my responsibility are not causing troubles I need to have access to them.
For now I whitelisted those 2 and that does the job.

regards,

Eddie
 
User avatar
leemans
Frequent Visitor
Frequent Visitor
Posts: 70
Joined: Thu Apr 07, 2005 12:55 am
Location: Belgium
Contact:

Re: Blacklist Filter update script

Fri Jun 09, 2017 5:51 pm

Hi Dave,

Dear,
It's not working on my RB600.
It used to work for a long time...
Any idea how come?

Thanks,
Patrick
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 09, 2017 6:01 pm

The script was updated last week to work with the new backend servers. You can find the update in the first post of this thread
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jun 10, 2017 5:16 pm

Thanks to someone setting up 50 routers to download every 2 minutes, the server is now blocking any router that downloads more than 4 times in a 24 hour period.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jun 10, 2017 9:33 pm

Is the blockage permanent or will it be lifted after a certain time? I can imagine that one that is testing has to reboot a few times in a few hours. So when testing one should switch off the startup reload until all is stable again.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jun 10, 2017 9:43 pm

SHouldn't be an issue for most. The server will flag routers that get excessive and throttle them to 4 download in a 24 hour period.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 11, 2017 12:55 am

And if the devil was at work with my Mikrotik and made it crashed and luckily I managed to switch off the startupscript. I had already had three strikes so I also should disable the normal update, for the time being.

Maybe it is possible to keep the file on the disk (when not using flash) and delete it on the next regular reload of 24 hours. After the first import it would have to be renamed with the time, of the first import in the name.

The script looks on a restart or regular reload if the file is older dan 23 hours and then it would get a new one. If the file is younger than 23hours the script will reload the file form the disk.

You can then still throttle addresses that reload more than three times with a blank router. If they have to setup routers then they also should copy the current file to the disk on each router.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 11, 2017 10:55 am

AAAARGH lost some sleep by trying to find out the way how to convert date+time so that I could subtract those and have the difference in time. No I did not manage but manage to go to sleep after staying up much to long.

In the morning my mind started to seek for a solution and I had different ideas but none of the would solve this. Then I got a great idea to just make a different script just for only the start-up. All pieces fell in place and no calculation and string delidding needed and just let nature do its work and follow the natural flow.

The main script updateBlacklist is changed so that dynamic.rsc file is not deleted after importing. The new startup startupBlacklist is the current updateBlacklist stripped of all download and statistical parts.



In updateBlacklist I commented out the removal of the dynamic.rsc file after importing and it will overwritten by the new dynamic.rsc file when the daily update is run. This is the changed code part form updateBlacklist



Hope that you like this adaptation and so also give your server bit of rest because restarted Mikrotik devices will not bother it when just restarting and only knock on the door for the real daily update.

So I am going now to eat my breakfast and enjoy my Sunday which is also today a sunny day.
Last edited by msatter on Fri Jun 16, 2017 12:50 am, edited 3 times in total.
 
nico599
just joined
Posts: 2
Joined: Mon Jun 12, 2017 11:42 am

Re: Blacklist Filter update script

Mon Jun 12, 2017 12:12 pm

Hi DAVE
yesterday , i change new script ,
it's working in my RB450G & RB750GL,
but in my CCR1009-8G-1S-1S+ ......,
just show Message "Downloading current Blacklist for this model",
but can't download anything...
i use the same script,.......

How can i to deal with this problem?

Sorry,English not my mother Language...
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Mon Jun 12, 2017 12:49 pm

Hello Dave,
The script has the ?, when pasted in terminal it disappears.
The log only has an entry of-
script error: expected command name (line 1 column 1)
The downloaded dynamic.rsc only has one line-
All fields are required. Please update your script.
That would mean that you need the current script. It's available in the first post.
Dave you could just escape the "?". That would allow it to be run in the terminal without issue, and it will make no difference for non-terminal running.
I mentioned it before, here: viewtopic.php?f=9&t=98804&start=150#p587708
"...\?..."
Last edited by Deantwo on Fri Aug 10, 2018 3:25 pm, edited 2 times in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jun 12, 2017 1:10 pm

A bit of saving on traffic you could save about 20% of traffic by not adding the "comment" part on every dynamicblacklist line.

I don't know if RouterOS can handle deflated traffic when downloading. However there is xz/LZMA used when a firmware update is applied. So if on saving a file with the .gz extension then it could be automatically be extracted and then your dynamic.rsc.gz would 20 times smaller and just 50KB instead of 970KB.

Update: all the .NPK files are zipped and extracted in some way when installed. Looking into the system.npk I find the program "unexpak" but I can't see what it is doing. When I look in \lib\ I see the library libz.so and if I am not wrong that is a compress/decompress code.

The only thing I found Mikrotik talking about compression stated that due to limitations of the size of the flash not allowing a compression tool to the users.

So maybe Mikrotik could give us the option to export compressed and then to normal extension like .RSC and .BACKUP add .GZ and automatically decompress files with .GZ when read.
 
reb00t
just joined
Posts: 2
Joined: Wed Jul 22, 2015 10:24 pm

Re: Blacklist Filter update script

Wed Jun 14, 2017 6:51 pm

My updates stopped working so I went to investigate and found when running the update script (v2017.5.31f) from the command line:
[admin@redacted_name.com] > /system script run updateBlacklist;                                                  
  status: failed

failure: cannot open file
I've installed the script via the download method described in the first post so I don't believe it's a copy/paste issue. Maybe I accessed the download more than four times during testing one day? Are there any other reasons to get that failure message?

Here's my basic info:
[admin@redacted_name.com] /system resource> print
             uptime: 4d4h2m55s
            version: 6.39.2 (stable)
         build-time: Mar/09/2017 11:32:49
        free-memory: 1712.6MiB
       total-memory: 1956.2MiB
                cpu: tilegx
          cpu-count: 9
      cpu-frequency: 1200MHz
           cpu-load: 0%
     free-hdd-space: 78.5MiB
    total-hdd-space: 128.0MiB
  architecture-name: tile
         board-name: CCR1009-8G-1S-1S+
           platform: MikroTik
[admin@redacted_name.com] /system resource> /system license print            
  software-id: 8RW2-IFMS
       nlevel: 6
     features: 
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 14, 2017 10:15 pm

try a copy/paste from the first post. Not sure what the issue is, the server isn't reporting any issues.
 
reb00t
just joined
Posts: 2
Joined: Wed Jul 22, 2015 10:24 pm

Re: Blacklist Filter update script

Thu Jun 15, 2017 3:35 pm

Doing the copy and paste from post #1 worked. Still not sure why it stopped working. Thank you!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1286
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 15, 2017 11:57 pm

Doing the copy and paste from post #1 worked. Still not sure why it stopped working. Thank you!
Sweet, glad it fixed it for you.
 
jgro
newbie
Posts: 49
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Wed Jun 21, 2017 10:18 am

Thank you for this, David!

Curious why you use a loop:
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }
instead of
/ip firewall address-list remove [find list="dynamicBlacklist"]
On my system, the loop takes 88 seconds while the 1-liner takes 32 seconds
How does your blacklist compare with the one being provided by squidblacklist.org (which just combines spamhaus drop, edrop, dshield, malc0de, blocklist.de )?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 21, 2017 10:36 am

That is indeed an easier and faster way. Dave wrote about that earlier about using those lists: viewtopic.php?f=9&t=98804&p=545381&hilit=Drop#p545381

I tried the change in code and the removal of the addresses went from 37 seconds down to 20 seconds and the total time is now 48 seconds and before it was 66 seconds. So the save time when using almost 20.000 IP addresses is around 17 seconds.

Who is online

Users browsing this forum: No registered users and 15 guests