Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

msatter · November 7, 2019, 12:41pm

I tried it endless to find that and this great. I knew the “total” part but did not thought op putting that in the variable.

if (([:tool fetch url=$url output=user as-value]->"total")<64) do={:local data ([:tool fetch url={$url output=user as-value]->"data")} else= {tool e-mail send ...}

It did not work for me.

Shumkov · November 8, 2019, 7:30am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?

Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html

RackKing · November 8, 2019, 8:49am

Many thanks.

msatter · November 8, 2019, 12:10pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.

# Written by Shumkov
# Adapted by blacklister
# 20201025
{
/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
}

The first is loaded and the second is not because of the size, being over 63KiB

I use separate blacklists and not one blacklist with different comments.

Update: also using now:

~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}"

RackKing · November 8, 2019, 2:21pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.

# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n")

The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.

I gave this a shot - but it did not run. No message in the log and no address list.

msatter · November 8, 2019, 2:37pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.

# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n")

The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.

I gave this a shot - but it did not run. No message in the log and no address list.

Remove one of the “(” in the line beginning with

:if (([:pick

hci · February 27, 2020, 10:53pm

I imagine the 63k limit is due to a variable size limit in Mikrotik scripting? It would be nice to be able to download larger blacklists.

sjafka · February 28, 2020, 10:43am

This is beautifull mate! Thanks for your work! If you have a site with paypal donation, i would like to get you a beer!
PS.: I used before the squid blacklist, but the guy, who created it died (RIP m8 and thank you for your work!) last year, but it had like 30k entries, this has “only around 1500”, but i see a lot of /24 subnets, so this is a huge list too!

inteq · March 1, 2020, 12:30pm

PSA
Make sure you have whitelisted your private IPs if using https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset

xenuc · March 2, 2020, 7:07am

The script is great, thanks. Now we just wait another 10 years to bypass the 65k limit.

HZsolt · March 2, 2020, 7:46pm

63 → 8192 and I downloaded the larger blacklist. But all lines did not load properly to the address-list.

https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset

How can I merge to one address-list the above addess-lists? I would like to use one address-list in the firewall of MikroTik instead of four address-lists. Fewer line in the firewall, faster processing and fewer load.

Shumkov · March 3, 2020, 3:06pm

Variant 2:

ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=https://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=https://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=https://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")

the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It’s harder and slower, but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an “array”?
Because the default “find” function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default “find” function.

HZsolt · March 3, 2020, 5:57pm

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,value,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find key=$ip in=$array]-1)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It’s harder and slower , but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an “array”?
Because the default “find” function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default “find” function.

With the above script can I properly (full lists) download the below lists?
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset

Shumkov · March 3, 2020, 6:15pm

Download full lists - you can’t. 63KiB is a limitation of RouterOS, here scripts are powerless.

HZsolt · March 3, 2020, 6:20pm

What is 63 KiB limitation of RouterOS?

msatter · March 3, 2020, 7:50pm

My version checks for list larger than 63KiB and logs then if the list is loaded or not.

There no way to import a list bigger than that through an array.

Bigger lists can be used but that is an other story.

Krusty · March 11, 2020, 9:23am

LifeSaver, thank you guys you are awesome

Shumkov · March 12, 2020, 12:05pm

Bugfix:

correct regexp is “^[1]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}”

The bug is not critical, it’s just that in some cases the script could process strings containing not only IP addresses, but simply numerical combinations similar in format.

0-9 ↩︎

Xtreme512 · March 25, 2020, 2:14am

Nice, very nice working script, thank you!

64 KB limit, on the other hand, is so annoying though… Gotta find a workaround, like maybe splitting files on-the-fly?

frantacech · April 18, 2020, 8:14pm

Hello!
how can i import it?

for example aggregated ip from china?
https://www.ipdeny.com/ipblocks/
https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone

I try, but it doesn’t work