Community discussions

  • 1
  • 2
  • 3
  • 4
  • 5
  • 13
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Blacklist Filter update script

Wed Jul 22, 2015 9:44 am



I've started development of the replacement service. Please read the Development topic here:
viewtopic.php?f=9&t=136666

Here is a form to fill out if you want to be notified when the new service goes live.
https://goo.gl/forms/UQMYqKJ54E0iV35l2








New blacklist system! (7-July-2017)

RouterOS version 6.36 or higher is now required.

Okay guys, I'm posting my first RC for the new system. To simplify things, I'm only posting an Installer / Updater script.
This will install the new blacklist update script, the config script, and the schedulers. You will end up with the following:
  • Scripts
  • blacklistUpdate - the primary script for checking for the list and installing it
  • blacklistUpdate.conf - Configuration for the script. the auto-script-update will not touch this.
  • blacklistScriptUpdater - this is the auto-upgrade script. I recommend calling it once a day to make sure you are current.
  • Scheduler
  • blacklistUpdate - this will run hourly, checking to see if a new list is available. Updating ONLY if the list is new
  • blacklistUpdateOnBoot - This is for loading the current list when the router boots
the list name has changed You will need to update your rules to use "intrusBL" instead of "dynamicBlacklist".

Updates are now done in place. Old entries have their expiration lowered to 30 minutes so that they expire soon. This replaces the remove process and lets them expire naturally. Current retirees are updated to 25 hours. New entries are added and set to 25 hours.

Checking for updates is done via DNS. a quick lookup to my DNS server (checking 127.0.0.3) returns the current serial number of the list. If the serial matches what is currently installed, no update is done. If the serial is higher, the new list is downloaded and installed.

I look forward to your feedback
# Intrus Technologies Blacklist Installer
# © 2017 David Joyce, Intrus Technologies
# 
# Version 2.0.5
#
# This is used to install and update the blacklist importer script
# as well as the scheduler tasks used to update the address-lists
# and the scheduler task used to update the scripts
#
# These are offered free of charge to the MikroTik community. No warranty is expressed or implied.
# I am not responsible for any loss of data, time, money, access, or anything else.  Use at your own risk.
#
# P.S. Changing the script names will break things. Badly.

:do {
    :local currentScriptVersion [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.2 ]
    :put "Installing blacklistUpdate script version: $currentScriptVersion";
    :local sourceServer "https://mikrotikfilters.com/";
    :local sourceServerPort "6501";
    :local scriptName "blInstaller.rsc";
    :put "Downloading update script...";
    :do {
        /tool fetch url="$sourceServer$scriptName" mode=https port=$sourceServerPort dst-path="/$scriptName";
    } on-error={
        :put "Error. Download failed";
    }
   :put "Importing update script...";
    :do {
        /import "$scriptName";
    } on-error={
        :put "import failed. unknown error.";
    }
    :put "Removing update script...";
    :do {
        /file remove "$scriptName";
    } on-error={}
    :put "Update Complete.";
}

  • Version History
  • 2.0.5 Released
  • Script and server changes to allow blacklisted IP's to still access the list
  • 2.0.4 Released
  • auto-update for script is default disabled, can be enabled in the config
  • added global "blScriptUpdate" to the config to enable/disable script updating
  • 2.0.3 Released
  • Script Updater cleanup
  • Installer now have full permissions (ros bug)
  • 2.0.2.1 and 2.0.2.2 Released
  • minor typos fixed
  • new Auto-Update script is now installed
  • - Auto-updater can be run manually, or on a daily schedule
  • new cleaner installer, can now be copy/pasted to the console
  • 2.0.2 Released
  • Fixed a logging typo
  • changed the auto-updater to stop removing the config if run twice
  • Better cleanup of globals used as functions
  • started framework for checking available disk space before downloading
  • 2.0.1 Released
  • improved URL Encoding function
  • much simpler CHR system ID detection
  • changed Script Version to global variable (prep for auto-script update)
Please remember to give a positive rating is you like and use this service
Last edited by IntrusDave on Sat Aug 04, 2018 11:22 pm, edited 54 times in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 23, 2015 3:35 am

This is an archived post. Please refer to the post above.

I've gone ahead and started publishing my dynamic filter list for RouterOS 6.x. My server generates the list each night after collecting data on all known botnets, C&C server, and spammers. Currently the list runs about 3k entries, so it may not work well on low end routers. Here is the script to update the list, as well as my personal firewall rules. As always, adjust them to fit your needs.  

Client Statistics can now be found here: https://mikrotikfilters.com/blstats.php

Feedback and suggestions are always welcome.



The list is updated every 6 hours. 00:00, 06:00, 12:00, 18:00. PLEASE DO NOT RUN EVERY MINUTE. Running the script every minute is a waste of bandwidth and puts undue strain on the NAND. I recommend updating every 12 to 24 hours.

The address-list entries are now Dynamic with a 25 hour timeout. This will cut the number of writes to NAND down dramatically.

The script only needs Read, Write, & Test permissions. Name the script "updateBlacklist". Removing the variables sent will prevent the server from sending the updates. I use them for accounting so I can keep track of the number of requests and the amount of bandwidth used.


Don't forget the schedule:



And, if you are interested, here are my filter rules:
/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2

/ip firewall address-list
add address=172.16.0.0/12 list=PrivateIPs
add address=10.0.0.0/8 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs

If you are using my script and Scheduler names, you can use this script to auto-update:
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
/system script remove updateBlacklist
/system scheduler remove updateBlacklist
/system scheduler remove updateBlacklistOnBoot
/import updateBlacklist.rsc;
/file remove updateBlacklist.rsc;
Change Log:
  • Version 2017-07-05a
  • Changed logging - Now only mutes the "firewall", no longer mutes all "info"
  • Changed default path to / instead of /disk1/ - current issue with CCR using microSD
  • More accurate logging of what is happening
  • minor text formatting changes
  • Now sets two globals - blSerial and blVersion (for future auto-update script)
Last edited by IntrusDave on Wed Jul 19, 2017 5:42 am, edited 2 times in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 23, 2015 5:35 am

update #2
added model specific support. smaller list for models that can't handle the full list. larger lists for 1GB+ models.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 24, 2015 5:04 am

update #3
better model / memory detection
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 25, 2015 8:16 pm

Update 4
server side - prevent generating duplicate "add" lines in the script.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
RackKing
Member Candidate
Member Candidate
Posts: 257
Joined: Wed Oct 09, 2013 1:59 pm

Re: Blacklist Filter update script

Sun Jul 26, 2015 4:22 pm

Thank you for providing this to the community!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 26, 2015 10:05 pm

It's the least I could do in return for all the help that the comunity has given me over the last two years.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 882
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:03 pm

First of all, thanks for this script!
It was something I wanted to implement for ages now, but never had the time to do so :)

At the moment it works by using your server.

Could you post your server-side code to be able to run this without needing to access your server?

Thanks :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:09 pm

Unfortunately, I can not. Much of the list is generated by my own routers. Just under 50 of them currently. As my servers and routers around the world get attacked, probed and spammed, they block the addresses and then send the addresses back to my server to be compiled into one list, when is then sent back out to the routers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 882
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:12 pm

I see. I though you were using some pre-made lists and just convert them to mikrotik commands.

Do you know any public blacklists I could use?

Btw, how often do you run this script?
If I am not mistaken this method keeps writing on the NAND storage on each fetch?
Any ideas how we could make this run completely in RAM so the NAND doesn't wear out?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 13, 2015 7:17 pm

you can check out dshield.org for a premade list.
My server collects the banned IP's 24/7 and publish the list at 3am PST.
As for not writing to NAND - I don't know of any way to prevent that. For a lower end unit with little storage, maybe only run the script once a month. For my own routers, I understand and accept that it will ware down the NAND - just the cost of security I suppose. Even replacing the routers every other year, they are still cheaper than a Cisco.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mmv
Trainer
Trainer
Posts: 63
Joined: Wed Feb 24, 2010 5:03 pm
Location: Moscow, Russia
Contact:

Re: Blacklist Filter update script

Thu Aug 13, 2015 8:38 pm

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
Mikhail Moskalev
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Aug 14, 2015 6:26 pm

Updated the script, moved to https.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 17, 2015 6:09 am

I'm added limits on the server to stop routers from requesting the list too often. Once router was downloading the list every minute. Over 1400 times a day - just over 320M (the list averages 250kb). Once every 24 hours is enough.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Sep 03, 2015 8:26 pm

added scheduler code.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 194
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Fri Oct 09, 2015 8:18 am

This is awesome! Thanks!!
 
savage
Forum Guru
Forum Guru
Posts: 1191
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: Blacklist Filter update script

Fri Oct 09, 2015 2:23 pm

/tool fetch url="https://mikrotik.intrustech.com/downloa ... ry=$memory" mode=http

Care to elaborate why those variables has any significance to you?
Regards,
Chris
 
savage
Forum Guru
Forum Guru
Posts: 1191
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: Blacklist Filter update script

Fri Oct 09, 2015 2:30 pm

Ah well.

403 forbidden errors in any case...

Just FYI - but it will be MUCH better (both on you, routers, management, and resources) to simply distribute lists of IPs using private ASN numbers and multi-hop BGP...

People peering with your BGP feed can then just get the updates as you push them, and blackhole the routes.

Much, much more efficient than hammering routers with 3K firewall rules :lol:

Just a thought...
Regards,
Chris
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Oct 09, 2015 5:01 pm

/tool fetch url="https://mikrotik.intrustech.com/downloa ... ry=$memory" mode=http

Care to elaborate why those variables has any significance to you?
On the server side, it allows the server to select a list optimized for your unit.
Model lets it know how much CPU power you have, memory for how big of a list, and version for quirks in the script.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Oct 09, 2015 5:03 pm

Ah well.

403 forbidden errors in any case...

Just FYI - but it will be MUCH better (both on you, routers, management, and resources) to simply distribute lists of IPs using private ASN numbers and multi-hop BGP...

People peering with your BGP feed can then just get the updates as you push them, and blackhole the routes.

Much, much more efficient than hammering routers with 3K firewall rules :lol:

Just a thought...
I don't use BGP because most users that will want this don't know how to setup BGP to start with.
And yes, server gives a 403 is you try to access it directly, instead of a RouterOS device pulling it.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 19, 2015 7:01 am

Updated the server side to create dynamic entries so that they are not written to NAND. This has a downside, the list will need to be reloaded on reboot. The entries now also expire after 48 hours.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
efaden
Forum Guru
Forum Guru
Posts: 1711
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Sun Feb 07, 2016 7:00 pm

:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "intrusBlacklist" ) do={ /ip firewall address-list remove $i } }
Can be simplified to
/ip firewall address-list remove [/ip firewall address-list find comment = "intrusBlacklist"]
Should actually increase the efficiency.
 
sachmonz
just joined
Posts: 11
Joined: Mon Feb 22, 2016 9:44 am

Re: Blacklist Filter update script

Mon Apr 18, 2016 11:08 am

Many thanks for this.

On the latest Router OS I can see it's created the entries under Address Lists, but on the filter rules I don't see any new filter rules showing up that reference the address list called blacklist as created by the script.

Model is MikroTik RB951G-2HND

Cheers
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2946
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Blacklist Filter update script

Mon Apr 18, 2016 11:47 am

/ip firewall filter
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
Where I have already see part of that rules on exact order? :lol:
Ah:
http://forum.mikrotik.com/viewtopic.php?f=9&t=83387

Sort the rules for efficency (simply drop if coming from blocked list, not first check malformed packet then drop)
/ip firewall filter
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
I'm Italian, not English. Sorry for my imperfect grammar.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 18, 2016 6:11 pm

good set of rules.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu May 21, 2015 11:14 am

Re: Blacklist Filter update script

Wed Apr 20, 2016 12:20 pm

Where I have already see part of that rules on exact order? :lol:
Ah:
http://forum.mikrotik.com/viewtopic.php?f=9&t=83387

Sort the rules for efficency (simply drop if coming from blocked list, not first check malformed packet then drop)
/ip firewall filter
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
Thanks for this ruleset.
Is it useful to use to put the input and forward chain into this same set of rules?

What i am also interested, is it possible to say how much performance influance has a rule?
I have a home use setup with an rb2011 and so i think i will never reach the limit (2-5 devices running @ same time)
It only interests me ;)

PS: Cant give karma, dont know why
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 2946
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Blacklist Filter update script

Wed Apr 20, 2016 10:24 pm

Thanks for this ruleset.
Is it useful to use to put the input and forward chain into this same set of rules?

What i am also interested, is it possible to say how much performance influance has a rule?
I have a home use setup with an rb2011 and so i think i will never reach the limit (2-5 devices running @ same time)
It only interests me ;)

PS: Cant give karma, dont know why
The rating are actually disabled...

on s.o.h.o. or at home as really no impact, except if layer-7 are used for filter traffic
on big amount of traffic count the power of the RouterBOARD
In my office (I'm a WISP) one 2011-RM is sufficient for 1Gbit internal traffic for 12 PC, 1 Server, 1 NAS, Public Local HotSpot (one Metal 2 out of the door) printers, VoIP systems (20 phones / 4 lines), failover between main line (20Mbps/20Mbps) and one ADSL line (7Mbps up / 380kbps down) and the firewall have near 80 rules on filter, 40 on NAT and 20 on mangle, hotspot service active withe relative rules, EoIP tunnels, DNS and address-list filter and web-proxy.

(DNS, address-list filter and web-proxy are used for remove all the ADs on web pages, https or not...)

I NOT use NAND,r USB or microSD for user-manager storage, I use one separate PC with RouterOS
I'm Italian, not English. Sorry for my imperfect grammar.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Apr 20, 2016 11:34 pm

These are the same filters I use, along with the blacklist that ranges from 2000~5000 IP's and subnets.
If you are interested, You can look at the stats for the routers that I monitor directly. The demo key only allows access to 15 of the routers. I will leave it open for a day or so.

https://mikrotik.intrustech.com/status.php?key=demo
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu May 21, 2015 11:14 am

Re: Blacklist Filter update script

Thu Apr 21, 2016 10:52 am

Thanks for this information!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat May 28, 2016 9:21 am

I've switched all my servers over to SSL only. Here is an updated script with https enabled.
I have new filter rules coming soon using the raw prerouting , once 6.36 is released.
:log warning "Downloading current Blacklist for this model";
:local model [/system resource get board-name]
:local version  [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
/tool fetch url="https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=https dst-path="/currentBlacklist.rsc";

# Disable Logging so each add and remove isn't in the system log, we turn it back on at the end
:log warning "Disabling system Logging";
/system logging disable 0

# Find and Remove the old filters
:log warning "Removing previous Blacklist Address-List entries";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "intrusBlacklist" ) do={ /ip firewall address-list remove $i } }

# Import the new filters
:log warning "Importing current Blacklist";
/import file-name=/currentBlacklist.rsc

# Delete the import file
:log warning "Removing temp files";
/file remove currentBlacklist.rsc

# Enable the Logging
:log warning "Enabling system logging, all done.";
/system logging enable 0
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24077
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Blacklist Filter update script

Fri Jun 17, 2016 5:32 pm

Great work! 
No answer to your question? How to write posts
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Jul 01, 2016 12:20 pm

I've switched all my servers over to SSL only. Here is an updated script with https enabled.
I have new filter rules coming soon using the raw prerouting , once 6.36 is released.
:log warning "Downloading current Blacklist for this model";
:local model [/system resource get board-name]
:local version  [/system resource get version]
:local memory [/system resource get total-memory]
:local uname [/system identity get name]
/tool fetch url="https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=https dst-path="/currentBlacklist.rsc";

# Disable Logging so each add and remove isn't in the system log, we turn it back on at the end
:log warning "Disabling system Logging";
/system logging disable 0

# Find and Remove the old filters
:log warning "Removing previous Blacklist Address-List entries";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "intrusBlacklist" ) do={ /ip firewall address-list remove $i } }

# Import the new filters
:log warning "Importing current Blacklist";
/import file-name=/currentBlacklist.rsc

# Delete the import file
:log warning "Removing temp files";
/file remove currentBlacklist.rsc

# Enable the Logging
:log warning "Enabling system logging, all done.";
/system logging enable 0
is this lastest working version ?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:47 pm

It does. However, it appears that if you are on older versions of RouterOS, you will need to disable https. the current RC fixed the issue.

Personally, I would recommend upgrading to the RC so you can take advantage of the RAW filter.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:48 pm

Im using V6.35.4 and I will try this. also I found this while searching http://joshaven.com/resources/tricks/mi ... ress-list/ and its seems working good.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:52 pm

Replace this: 
/tool fetch url="https://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=https dst-path="/currentBlacklist.rsc";
with this:
/tool fetch url="http://mikrotik.intrustech.com/download.php?get=complete&model=$model&version=$version&memory=$memory&id=$uname" mode=http dst-path="/currentBlacklist.rsc";
For some reason, anything before 6.36RC will not connect to the server via https. 
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Jul 01, 2016 4:54 pm

Thanks A lot... I will try this...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 01, 2016 5:07 pm

let me know if it doesn't work. I've been meaning to give this scrip a rewrite to match the new servers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sat Jul 02, 2016 8:24 am

gives an error;

failure: closing connection: <301 Moved Permanently> 104.27.182.151:80 (4)

but i found this. http://joshaven.com/resources/tricks/mi ... ress-list/ and its working... i dont know which one is better
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1702
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sat Jul 02, 2016 10:16 am

Which kind of mineral water is "better": sparkling or non-sparkling ? :-)

You can combine both lists if you feel that each one is not enough for you.
If not, just check counters when you use each of them and decide which blocks more attacks and use it. Remeber that lists are dynamic so the their goodness could change each time you reload it.
Real admins use real keyboards.
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sat Jul 02, 2016 10:35 am

Which kind of mineral water is "better": sparkling or non-sparkling ? :-)
choose the non-sparkling one.. it will better.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 02, 2016 6:18 pm

New script - I found that RouterOS doesn't like my new CDN (CloudFlare) so I've setup a separate server for the lists.
:log warning "Downloading current Blacklist for this model";
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) do={ /ip firewall address-list remove $i } }
:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Please note that the address-list name has been changed.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sun Jul 03, 2016 12:47 am

New script - I found that RouterOS doesn't like my new CDN (CloudFlare) so I've setup a separate server for the lists.
:log warning "Downloading current Blacklist for this model";
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) do={ /ip firewall address-list remove $i } }
:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Please note that the address-list name has been changed.


Thats worked... Thank you very much... and i schedule it to work every 24h.. can i ask something for about listed ip addresses ? are these include Spamhaus DROP List and OpenBl List ?

Thanks A lot
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 03, 2016 2:46 am


Thats worked... Thank you very much... and i schedule it to work every 24h.. can i ask something for about listed ip addresses ? are these include Spamhaus DROP List and OpenBl List ?

Thanks A lot
My server pulls the lists from Spamhaus, OpenBL, malc0de, and emergingthreats. In addition to those, I have just over 40 servers and routers that report in and add to the list. Currently the server builds a new list every 24 hours. I'm working on a new system that will be updated continuously. 
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 525
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Sun Jul 03, 2016 2:05 pm


Thats worked... Thank you very much... and i schedule it to work every 24h.. can i ask something for about listed ip addresses ? are these include Spamhaus DROP List and OpenBl List ?

Thanks A lot
My server pulls the lists from Spamhaus, OpenBL, malc0de, and emergingthreats. In addition to those, I have just over 40 servers and routers that report in and add to the list. Currently the server builds a new list every 24 hours. I'm working on a new system that will be updated continuously. 
Thank you very much... I start to use it. and i add some firewall rule for this can you check it for me also
/ip firewall filter
add action=drop chain=input comment=dynamicBlacklist in-interface=wan src-address-list=dynamicBlacklist
add action=drop chain=input comment=dynamicBlacklist dst-address-list=dynamicBlacklist in-interface=wan
add action=drop chain=forward comment=dynamicBlacklist in-interface=wan src-address-list=dynamicBlacklist
add action=drop chain=forward comment=dynamicBlacklist dst-address-list=dynamicBlacklist in-interface=wan
Thanks
 
Zorro
Long time Member
Long time Member
Posts: 676
Joined: Wed Apr 16, 2014 2:43 pm

Re: Blacklist Filter update script

Fri Jul 15, 2016 9:05 pm

how about adding fetching ads lists from popular lists(like one from say ublock 3rd subscriptions. like this one https://pgl.yoyo.org/adservers/) malware from malwaredomains.com and similar resources(known to be several comunities for) ?
then you can proces/convert then and then safely block them in dns static override (in worst case by adress lists).
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 15, 2016 9:33 pm

it's very possible to do that, but I would need to see what the impact on the routers would be. I'm not a big fan of the built-in DNS as it is and I'm not sure how well it would hold up with several thousand hostnames added to it.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 194
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:16 am

I copied and pasted your recent code but I can not get it to work.... Tried it on two different CCR routers. One running 6.36 and the other running 6.35.2. It does not give an error it just doesn't do anything.....
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:30 am

:local model 	[/system resource get board-name]
:local version  [/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	 [/system identity get name]
:local scriptVer 2016.7.4a

:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Try this.. Same thing, just different formatting. Maybe it was a copy/paste issue?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 194
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:37 am

Still nothing.Shows in the log file that it is going but that is it, no errors or anything just nothing. Strange.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1282
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:42 am

strange. any messages if you run it from the terminal?
you should see this...
[djoyce@Intrus_AltaLoma] > /system script run updateBlacklist 
      status: finished
  downloaded: 231KiB
       total: 231KiB
    duration: 1s

Script file loaded and executed successfully
[djoyce@Intrus_AltaLoma] > 
Then you should have have 2000~3000 items in the "dynamicBlacklist" address-list.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
  • 1
  • 2
  • 3
  • 4
  • 5
  • 13

Who is online

Users browsing this forum: No registered users and 8 guests