Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

RouterOS Scripting
41

ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->“data”)
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data “\n”]]~“[1]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}”) do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:if ([:pick $data 0 [:find $data “\n”]]~“[a-z0-9]+([\-\.]{1}[a-z0-9]+)\.[a-z]{2,5}(:[0-9]{1,5})?(\/.)?”) do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data “\n”]+1) [:len $data]]
}
} on-error={:log warning “Address list <$description> update failed”}
}
$update url=> http://feeds.dshield.org/block.txt > description=DShield delimiter=(“\t”) cidr=/24
$update url=> http://www.spamhaus.org/drop/drop.txt > description=“Spamhaus DROP” delimiter=(“_”)
$update url=> http://www.spamhaus.org/drop/edrop.txt > description=“Spamhaus EDROP” delimiter=(“_”)
$update url=> http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt > description=“Bambenek High-Confidence C2” delimiter=(“\2C”)
$update url=> https://sslbl.abuse.ch/blacklist/sslipblacklist.txt > description=“Abuse.ch SSLBL” delimiter=(“\r”)
$update url=> http://malc0de.com/bl/IP_Blacklist.txt > description=“malc0de” delimiter=(“\n”)
$update url=> https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset > description=“FireHOL Level2” delimiter=(“\n”)
$update url=> https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset > description=“FireHOL Level1” delimiter=(“\n”)
$update url=> https://raw.githubusercontent.com/hectorm/hmirror/master/data/adaway.org/list.txt > description=“hectorm adaway.org” delimiter=(“\n”)

Trying to add support for address lists containing the URL as apposed to IP, such as hectorm’s lists for PiHole:
https://discourse.pi-hole.net/t/update-the-best-blocking-lists-for-the-pi-hole-alternative-dns-servers-2019/13620

However, some of the comments come out as the actual URL entry and the timeouts aren’t set.

  1. 0-9 ↩︎

42

Try:
$update url=https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone description=“IPdeny cn-aggregated” delimiter=(“\n”)

43

How do we get around this 63KiB limit? can we ask mikrotik about this

We are trying to automate the download and add of

https://www.ipdeny.com/ipblocks/data/countries/gb.zone which is 124KiB

44

Don’t forget to add

{:delay 20};

at the beginning of the script to give time if running after reboot is needed.

45

Perhaps the only way is to have some really smart script parse this list further into large(r) CIDR-blocks. So take several /24 “lines” and aggregate them further where there are adjacencies.
I’ve seen some sort of script here somewhere (used in another context) but it might be doable to gain a certain % of reduction.

46

The problem is thst first have to read whole list before you can start reducing.

If Miktotik implement resume download then we could chop up the file in little parts.

47

According to the following Manual:Scripting-examples – file size limitation has been removed

Read and write large files

Many users requested ability to work with files. Now you can do it without limitations

Create and write to file:

:global newContent “new file content\r\nanother line\r\n”;
[/lua “local f=assert(io.open(‘/test.txt’, ‘w+’)); f:write(newContent); f:close()” ];
Read file content to variable:

:global cnt “”
[/lua “local f=assert(io.open(‘/test.txt’, ‘r’)); cnt=f:read(‘*all’); f:close()” ];
:put $cnt

I just found this wiki entry but I have not tried to adapt to blacklists … if this code actually works that would be excellent.

48

Is /lua back then?

49

Don’t think so. That Wiki page states : This page was last edited on 18 October 2017, at 10:37.
As it says on the page : After RouterOS v4.0beta4, Lua support is removed until further notice

50

My sincere apologies – I did not see the part that After RouterOS v4.0beta4, Lua support is removed until further notice

What a shame, all that Lua stuff should be removed IMO BUT if certainly would be nice if MikroTik brought back LUA support or provided another means to work with any file size.

51

Is there a reasonable way of bypassing Mk’s limit or another approach?
I’m a Mk hardcore user but I’m considering other vendors if they do not apply a better concept to protect our equipments.

BTW, is there a way to have these working?

https://github.com/firehol/blocklist-ipsets/blob/master/firehol_level1.netset			40.8 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_wannacry.ipset	         6.15 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_bruteforce.ipset.     	4.64 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield_30d.netset					2.17 KB
https://github.com/firehol/blocklist-ipsets/blob/master/spamhaus_edrop.netset				1.98 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield_7d.netset					1.5 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_webscan.ipset	1.42 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield.netset						1.04 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_wormscan.ipset	0.97 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_dnsscan.ipset	0.86 KB
52

Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So…


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly “prepare” the file before download to the device, but that cannot be the show stopper I guess.

53

I see. And then, how do people do with servers with an open port? Let’s say people need to access my server in a specific port?

54

It is important to take all considerations into account when you make the design. If “people” are in fact scattered across the world coming from virtually anyplace then perhaps you need to provide this service at another level. Eg. use some form of authentication with your users (possibly combined with VPN-application).
What is sitting behind this specific port ? Is this something that understand the concept of user-authentication ?
If you run a business and you know your users (eg. employees) are located in country X then filter strict and only allow country X IP which will reduce the surface already A LOT.

55

Lv1 was working fine and now it is not. Probably it does not fit anymore.

56

You [everyone] should be aware that:

level1 check frequency = 1 minute and average update frequency = 2 hours and 27 minutes
level2 check frequency = 1 minute and average update frequency = 17 minutes
level3 check frequency = 1 minute and average update frequency = 45 minutes
level4 check frequency = 1 minute and average update frequency = 44 minutes
webclient check frequency = 1 minute and average update frequency = 8 hours and 36 minutes
webserver check frequency = 1 minute and average update frequency = 23 hours and 16 minutes

So why is this important to note?
Because changes [adds/deletions] are frequent and that can have a dramatic change in file size.
Also of importance to note is that many duplicates reside when lists are combined - so your processing engine needs to remove duplicates and then reorders them for faster processing.

Depending on which MikroTik Router model being used MOAB combines some of these lists or ALL of these lists 3 times each day spaced 8 hours apart.

57

What is the recommended way to find out why an update failed?

Address list update failed

Is great to see in the logs, but where do I look to try and figure out why it failed?

Spamhaus DROP and EDROP are not over 63 kb, so that isn’t the reason..

At the moment I am focusing on the IPs used for email SPAM, but it doesn’t really matter.. I have the Spamhaus and Bambenek lists failing but I don’t know why.

58

This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?

59

They don’t load always.

60

@kevinds
You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
So it is critical that duplicate IP be avoided via a pre-process that first checks for duplicates, removes the duplicates, reorders [sorts] the list for faster processing then proceeds with the load.