Community discussions

 
User avatar
Shumkov
just joined
Topic Author
Posts: 5
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Oct 01, 2019 11:00 pm

Hello!
The new parameter "output=user" provided new scripting capabilities that I decided to take full advantage of.

- the script does not need third-party servers, since address lists are downloaded directly from the source and processed directly on the router.

- the script does NOT save the downloaded files to the disk (thereby preventing premature wear and failure of the disk).

- the script can be adapted to download and process any number of address lists of a similar format (the maximum file size is 63 KiB (64512 bytes). It is better than 4 KiB :)).

At the moment the script can download and update next lists:
- DShield
- Spamhaus DROP
- Spamhaus EDROP
- Bambenek High-Confidence C2
- Abuse.ch SSLBL
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
And yes, the script size is very small.
Required policy: read, write, test.
Perhaps this script will be useful to someone :)

P.S. Sorry for my English :oops:
Last edited by Shumkov on Wed Oct 30, 2019 9:14 am, edited 7 times in total.
RB951G-2HnD / RouterOS 6.44.6 (Long-term)
 
Zebble
newbie
Posts: 45
Joined: Mon Oct 17, 2011 4:07 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 18, 2019 12:12 am

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
 
liuyao
just joined
Posts: 1
Joined: Wed Sep 04, 2019 9:14 am
Location: China

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 18, 2019 4:29 pm

Hello:

Thank you for sharing。 But the way you write functions is hard to understand. If any boss is rewritten, the written statement is perfect like the official example. Thank you
小白充大神
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:20 pm

Hi - This looks great. I will give it a try.

Update -
I just run this and it works great - no errors and works perfectly

What is general recommendation on how often to grab new lists - daily?

Am I correct it removes or ignores duplicate entries?

It would be great to keep this updated with additional!

Thank you so much for this!!!
Last edited by RackKing on Sun Nov 03, 2019 5:40 pm, edited 1 time in total.
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:37 pm

How does it handle 1.2.3.0/24 addresses and as far I could it enters 1.2.3.0 in the addresslist without the /24?

Update: I ran the script and it does handles the range (cidr) correctly. Going to look if I can add some more lists.

Update 2: excellent script and I have added the option to filter on a specific label in file and that also can be used to remove a list that is not used anymore, from the current blacklist in the addresslist.
Last edited by msatter on Sun Nov 03, 2019 7:57 pm, edited 4 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:42 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 7:50 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
This is the correct syntax
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 7:51 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
It works if poster zeb put it as code here:
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")

REALLY PLEASED with the script from Shumkov and the added option by Mikrotik and it is now very easy to import lists without having to use other computers to prepare the lists up front
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 8:00 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
It works if poster zeb put it as code here:
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")

REALLY PLEASED with the script from Shumkov and the added option by Mikrotik and it is now very easy to import lists without having to use other computers to prepare the lists up front
That Level2 list is huge.... trying to sort the different levels they have. Any thoughts? Also, would you fun this daily?
 
User avatar
Shumkov
just joined
Topic Author
Posts: 5
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 8:39 pm

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
RB951G-2HnD / RouterOS 6.44.6 (Long-term)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 10:53 pm

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
Ah - that makes sense. You are quite correct. Thanks for the explanation on the removal.

Are there any other lists you would consider or a good source?
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 1:58 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
:local data ([/tool fetch url=$url output=user as-value~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}"]->"data");
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:42 am

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
It looks like FireHOL Level1 may be a better choice and is under the file size limit.... barely. Any reason no to use this? That large of a list would probably have a pretty big performance hit on the router?

@Shumkov what was your goal/strategy based on the lists you choose? I am trying to sort what lists should be used and what is a happy medium.

Edit - after taking a closer look it appears the individual sources you are using is very similar to firehol_level1. With a goal of having no false positives this is a great place to start. I guess whether you grab them individually or through firehol is personal preference.

What a great script - thank you very much.
Last edited by RackKing on Mon Nov 04, 2019 3:34 am, edited 2 times in total.
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:56 am

malc0de

$update url=http://malc0de.com/bl/IP_Blacklist.txt description="Malc0de" delimiter=("\n")
 
User avatar
Shumkov
just joined
Topic Author
Posts: 5
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 9:51 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
This does not work :)
"data" is an element of the array, and is accepted for processing only in its entirety - you cannot process only part of the element.
@Shumkov what was your goal/strategy based on the lists you choose? I am trying to sort what lists should be used and what is a happy medium.

Edit - after taking a closer look it appears the individual sources you are using is very similar to firehol_level1.
That's right, I took FireHOL Level1 as the basis.
I removed “Feodo Tracker” and “Ransomware Tracker”, replaced “Bambenek C2” with “Bambenek High-Confidence C2” (as Bambenek recommended it myself), and also removed “Fullbogons” - I get them using BGP.
RB951G-2HnD / RouterOS 6.44.6 (Long-term)
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 10:38 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
This does not work :)
"data" is an element of the array, and is accepted for processing only in its entirety - you cannot process only part of the element.
I agree and my angle is to filter traffic (stream) on the way to the data array.

Like this in scripting:
wget -q -O - $url | gawk --posix --field-separator=, '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/ { print "$i a=" $1;}'  > $saveTo/$filename
This is something only Mikrotik can create to intercepting the stream.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:29 pm

That's right, I took FireHOL Level1 as the basis.
I removed “Feodo Tracker” and “Ransomware Tracker”, replaced “Bambenek C2” with “Bambenek High-Confidence C2” (as Bambenek recommended it myself), and also removed “Fullbogons” - I get them using BGP.
Makes perfect sense. Thank you again so much for this.
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 4:00 pm

Is there a way to check the file size and have it trigger the email tool if it gets beyond the max file size?
 
User avatar
Shumkov
just joined
Topic Author
Posts: 5
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 12:41 pm

Is there a way to check the file size and have it trigger the email tool if it gets beyond the max file size?
You can try this:
if (([tool fetch url=<url> output=user as-value]->"total")>63) do={tool e-mail send ...}
RB951G-2HnD / RouterOS 6.44.6 (Long-term)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 2:13 pm

Thanks you for that.

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 2:41 pm

I tried it endless to find that and this great. I knew the "total" part but did not thought op putting that in the variable.
if (([:tool fetch url=$url output=user as-value]->"total")<64) do={:local data ([:tool fetch url={$url output=user as-value]->"data")} else= {tool e-mail send ...}
It did not work for me.
Last edited by msatter on Fri Nov 08, 2019 10:52 am, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Shumkov
just joined
Topic Author
Posts: 5
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 9:30 am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html
RB951G-2HnD / RouterOS 6.44.6 (Long-term)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 10:49 am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html
Many thanks.
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 2:10 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
Last edited by msatter on Fri Nov 08, 2019 4:38 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
RackKing
Member Candidate
Member Candidate
Posts: 281
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 4:21 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
I gave this a shot - but it did not run. No message in the log and no address list.
 
msatter
Forum Guru
Forum Guru
Posts: 1281
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 4:37 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
I gave this a shot - but it did not run. No message in the log and no address list.
Remove one of the "(" in the line beginning with
:if (([:pick
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)

Who is online

Users browsing this forum: No registered users and 13 guests