Community discussions

MikroTik App
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Oct 01, 2019 11:00 pm

Hello!
The new parameter "output=user" provided new scripting capabilities that I decided to take full advantage of.

- the script does not need third-party servers, since address lists are downloaded directly from the source and processed directly on the router.

- the script does NOT save the downloaded files to the disk (thereby preventing premature wear and failure of the disk).

- the script can be adapted to download and process any number of address lists of a similar format (the maximum file size is 63 KiB (64512 bytes). It is better than 4 KiB).

At the moment the script can download and update next lists:
- DShield
- Spamhaus DROP
- Spamhaus EDROP
- Abuse.ch SSLBL

Variant 1:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=https://www.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=https://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=https://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script deletes all addresses matching the condition "list=blacklist comment=$description", after which it fills out address lists from scratch. It's easier and faster.

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=https://www.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=https://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=https://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It's harder and slower, but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an "array"?
Because the default "find" function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default "find" function.

Required policy: read, write, test.
Perhaps this script will be useful to someone.

P.S. Sorry for my English.
Last edited by Shumkov on Sat Dec 12, 2020 6:22 pm, edited 15 times in total.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
Zebble
newbie
Posts: 46
Joined: Mon Oct 17, 2011 4:07 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 18, 2019 12:12 am

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
 
liuyao
just joined
Posts: 2
Joined: Wed Sep 04, 2019 9:14 am
Location: China

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 18, 2019 4:29 pm

Hello:

Thank you for sharing。 But the way you write functions is hard to understand. If any boss is rewritten, the written statement is perfect like the official example. Thank you
小白充大神
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:20 pm

Hi - This looks great. I will give it a try.

Update -
I just run this and it works great - no errors and works perfectly

What is general recommendation on how often to grab new lists - daily?

Am I correct it removes or ignores duplicate entries?

It would be great to keep this updated with additional!

Thank you so much for this!!!
Last edited by RackKing on Sun Nov 03, 2019 5:40 pm, edited 1 time in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:37 pm

How does it handle 1.2.3.0/24 addresses and as far I could it enters 1.2.3.0 in the addresslist without the /24?

Update: I ran the script and it does handles the range (cidr) correctly. Going to look if I can add some more lists.

Update 2: excellent script and I have added the option to filter on a specific label in file and that also can be used to remove a list that is not used anymore, from the current blacklist in the addresslist.
Last edited by msatter on Sun Nov 03, 2019 7:57 pm, edited 4 times in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:42 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 7:50 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
This is the correct syntax
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 7:51 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
It works if poster zeb put it as code here:
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")

REALLY PLEASED with the script from Shumkov and the added option by Mikrotik and it is now very easy to import lists without having to use other computers to prepare the lists up front
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 8:00 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
It works if poster zeb put it as code here:
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")

REALLY PLEASED with the script from Shumkov and the added option by Mikrotik and it is now very easy to import lists without having to use other computers to prepare the lists up front
That Level2 list is huge.... trying to sort the different levels they have. Any thoughts? Also, would you fun this daily?
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 8:39 pm

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 10:53 pm

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
Ah - that makes sense. You are quite correct. Thanks for the explanation on the removal.

Are there any other lists you would consider or a good source?
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 1:58 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
:local data ([/tool fetch url=$url output=user as-value~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}"]->"data");
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:42 am

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
It looks like FireHOL Level1 may be a better choice and is under the file size limit.... barely. Any reason no to use this? That large of a list would probably have a pretty big performance hit on the router?

@Shumkov what was your goal/strategy based on the lists you choose? I am trying to sort what lists should be used and what is a happy medium.

Edit - after taking a closer look it appears the individual sources you are using is very similar to firehol_level1. With a goal of having no false positives this is a great place to start. I guess whether you grab them individually or through firehol is personal preference.

What a great script - thank you very much.
Last edited by RackKing on Mon Nov 04, 2019 3:34 am, edited 2 times in total.
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:56 am

malc0de

$update url=http://malc0de.com/bl/IP_Blacklist.txt description="Malc0de" delimiter=("\n")
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 9:51 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
This does not work :)
"data" is an element of the array, and is accepted for processing only in its entirety - you cannot process only part of the element.
@Shumkov what was your goal/strategy based on the lists you choose? I am trying to sort what lists should be used and what is a happy medium.

Edit - after taking a closer look it appears the individual sources you are using is very similar to firehol_level1.
That's right, I took FireHOL Level1 as the basis.
I removed “Feodo Tracker” and “Ransomware Tracker”, replaced “Bambenek C2” with “Bambenek High-Confidence C2” (as Bambenek recommended it myself), and also removed “Fullbogons” - I get them using BGP.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 10:38 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
This does not work :)
"data" is an element of the array, and is accepted for processing only in its entirety - you cannot process only part of the element.
I agree and my angle is to filter traffic (stream) on the way to the data array.

Like this in scripting:
wget -q -O - $url | gawk --posix --field-separator=, '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/ { print "$i a=" $1;}'  > $saveTo/$filename
This is something only Mikrotik can create to intercepting the stream.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:29 pm

That's right, I took FireHOL Level1 as the basis.
I removed “Feodo Tracker” and “Ransomware Tracker”, replaced “Bambenek C2” with “Bambenek High-Confidence C2” (as Bambenek recommended it myself), and also removed “Fullbogons” - I get them using BGP.
Makes perfect sense. Thank you again so much for this.
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 4:00 pm

Is there a way to check the file size and have it trigger the email tool if it gets beyond the max file size?
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 12:41 pm

Is there a way to check the file size and have it trigger the email tool if it gets beyond the max file size?
You can try this:
if (([tool fetch url=<url> output=user as-value]->"total")>63) do={tool e-mail send ...}
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 2:13 pm

Thanks you for that.

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 2:41 pm

I tried it endless to find that and this great. I knew the "total" part but did not thought op putting that in the variable.
if (([:tool fetch url=$url output=user as-value]->"total")<64) do={:local data ([:tool fetch url={$url output=user as-value]->"data")} else= {tool e-mail send ...}
It did not work for me.
Last edited by msatter on Fri Nov 08, 2019 10:52 am, edited 1 time in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 9:30 am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 10:49 am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html
Many thanks.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 2:10 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20201025
{
/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
}
The first is loaded and the second is not because of the size, being over 63KiB

I use separate blacklists and not one blacklist with different comments.

Update: also using now:
~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}"
Last edited by msatter on Sun Oct 25, 2020 10:06 am, edited 6 times in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
RackKing
Member
Member
Posts: 381
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 4:21 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
I gave this a shot - but it did not run. No message in the log and no address list.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 4:37 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
I gave this a shot - but it did not run. No message in the log and no address list.
Remove one of the "(" in the line beginning with
:if (([:pick
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
hci
Long time Member
Long time Member
Posts: 645
Joined: Fri May 28, 2004 5:10 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Feb 28, 2020 12:53 am

I imagine the 63k limit is due to a variable size limit in Mikrotik scripting? It would be nice to be able to download larger blacklists.
 
User avatar
sjafka
Frequent Visitor
Frequent Visitor
Posts: 89
Joined: Wed Jan 03, 2018 5:45 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Feb 28, 2020 12:43 pm

This is beautifull mate! Thanks for your work! If you have a site with paypal donation, i would like to get you a beer! :D
PS.: I used before the squid blacklist, but the guy, who created it died (RIP m8 and thank you for your work!) last year, but it had like 30k entries, this has "only around 1500", but i see a lot of /24 subnets, so this is a huge list too!
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 286
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Mar 01, 2020 2:30 pm

PSA
Make sure you have whitelisted your private IPs if using https://raw.githubusercontent.com/fireh ... el1.netset
 
xenuc
just joined
Posts: 1
Joined: Mon Mar 02, 2020 8:28 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Mar 02, 2020 9:07 am

The script is great, thanks. Now we just wait another 10 years to bypass the 65k limit.
 
HZsolt
newbie
Posts: 31
Joined: Tue Apr 24, 2018 7:31 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Mar 02, 2020 9:46 pm

63 --> 8192 and I downloaded the larger blacklist. But all lines did not load properly to the address-list.

https://raw.githubusercontent.com/fireh ... el1.netset
https://raw.githubusercontent.com/fireh ... el2.netset
https://raw.githubusercontent.com/fireh ... el3.netset
https://raw.githubusercontent.com/fireh ... el4.netset

How can I merge to one address-list the above addess-lists? I would like to use one address-list in the firewall of MikroTik instead of four address-lists. Fewer line in the firewall, faster processing and fewer load.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 5:06 pm

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=https://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=https://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=https://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It's harder and slower, but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an "array"?
Because the default "find" function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default "find" function.
Last edited by Shumkov on Sun Oct 25, 2020 7:52 am, edited 4 times in total.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
HZsolt
newbie
Posts: 31
Joined: Tue Apr 24, 2018 7:31 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 7:57 pm

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,value,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find key=$ip in=$array]-1)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It's harder and slower :), but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an "array"?
Because the default "find" function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default "find" function.
With the above script can I properly (full lists) download the below lists?
https://raw.githubusercontent.com/fireh ... el1.netset
https://raw.githubusercontent.com/fireh ... el2.netset
https://raw.githubusercontent.com/fireh ... el3.netset
https://raw.githubusercontent.com/fireh ... el4.netset
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 8:15 pm

With the above script can I properly (full lists) download the below lists?
Download full lists - you can’t. 63KiB is a limitation of RouterOS, here scripts are powerless.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
HZsolt
newbie
Posts: 31
Joined: Tue Apr 24, 2018 7:31 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 8:20 pm

With the above script can I properly (full lists) download the below lists?
Download full lists - you can’t. 63KiB is a limitation of RouterOS, here scripts are powerless.
What is 63 KiB limitation of RouterOS?
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 9:50 pm

My version checks for list larger than 63KiB and logs then if the list is loaded or not.

There no way to import a list bigger than that through an array.

Bigger lists can be used but that is an other story.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
Krusty
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Fri May 02, 2008 11:14 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Wed Mar 11, 2020 11:23 am

LifeSaver, thank you guys you are awesome
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Mar 12, 2020 2:05 pm

Bugfix:
- correct regexp is "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}"

The bug is not critical, it’s just that in some cases the script could process strings containing not only IP addresses, but simply numerical combinations similar in format.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
User avatar
Xtreme512
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Sun Jun 08, 2014 2:43 pm
Location: Nicosia, CY
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Wed Mar 25, 2020 4:14 am

Nice, very nice working script, thank you!

64 KB limit, on the other hand, is so annoying though... Gotta find a workaround, like maybe splitting files on-the-fly?
I Walk Alone
 
frantacech
just joined
Posts: 3
Joined: Tue Jul 25, 2017 6:55 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Apr 18, 2020 11:14 pm

Hello!
how can i import it?

for example aggregated ip from china?
https://www.ipdeny.com/ipblocks/
https://www.ipdeny.com/ipblocks/data/ag ... gated.zone

I try, but it doesn't work
 
shed909
just joined
Posts: 2
Joined: Sat Apr 25, 2020 5:59 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Apr 25, 2020 6:02 am

ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:if ([:pick $data 0 [:find $data "\n"]]~"[a-z0-9]+([\\-\\.]{1}[a-z0-9]+)*\\.[a-z]{2,5}(:[0-9]{1,5})?(\\/.*)?") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/fee ... t-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
$update url=http://malc0de.com/bl/IP_Blacklist.txt description="malc0de" delimiter=("\n")
$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")
$update url=https://raw.githubusercontent.com/fireh ... el1.netset description="FireHOL Level1" delimiter=("\n")
$update url=https://raw.githubusercontent.com/hecto ... g/list.txt description="hectorm adaway.org" delimiter=("\n")
Trying to add support for address lists containing the URL as apposed to IP, such as hectorm's lists for PiHole:
https://discourse.pi-hole.net/t/update- ... 2019/13620

However, some of the comments come out as the actual URL entry and the timeouts aren't set.
Last edited by shed909 on Sat Apr 25, 2020 7:13 am, edited 3 times in total.
 
shed909
just joined
Posts: 2
Joined: Sat Apr 25, 2020 5:59 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Apr 25, 2020 6:04 am

Hello!
how can i import it?

for example aggregated ip from china?
https://www.ipdeny.com/ipblocks/
https://www.ipdeny.com/ipblocks/data/ag ... gated.zone

I try, but it doesn't work
Try:
$update url=https://www.ipdeny.com/ipblocks/data/ag ... gated.zone description="IPdeny cn-aggregated" delimiter=("\n")
 
pukka
just joined
Posts: 13
Joined: Sun Jun 26, 2011 4:05 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 01, 2020 2:22 pm

How do we get around this 63KiB limit? can we ask mikrotik about this

We are trying to automate the download and add of

https://www.ipdeny.com/ipblocks/data/countries/gb.zone which is 124KiB
 
Lebzul
Member Candidate
Member Candidate
Posts: 110
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 16, 2020 11:38 pm

Don't forget to add
{:delay 20};
at the beginning of the script to give time if running after reboot is needed.
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 17, 2020 8:55 am

How do we get around this 63KiB limit? can we ask mikrotik about this

We are trying to automate the download and add of

https://www.ipdeny.com/ipblocks/data/countries/gb.zone which is 124KiB
Perhaps the only way is to have some really smart script parse this list further into large(r) CIDR-blocks. So take several /24 "lines" and aggregate them further where there are adjacencies.
I've seen some sort of script here somewhere (used in another context) but it might be doable to gain a certain % of reduction.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 17, 2020 11:20 am

The problem is thst first have to read whole list before you can start reducing.

If Miktotik implement resume download then we could chop up the file in little parts.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 2:02 pm

According to the following Manual:Scripting-examples -- file size limitation has been removed
Read and write large files

Many users requested ability to work with files. Now you can do it without limitations

Create and write to file:

:global newContent "new file content\r\nanother line\r\n";
[/lua "local f=assert(io.open('/test.txt', 'w+')); f:write(newContent); f:close()" ];
Read file content to variable:

:global cnt ""
[/lua "local f=assert(io.open('/test.txt', 'r')); cnt=f:read('*all'); f:close()" ];
:put $cnt
I just found this wiki entry but I have not tried to adapt to blacklists ..... if this code actually works that would be excellent.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 3:34 pm

According to the following Manual:Scripting-examples -- file size limitation has been removed
Read and write large files

Many users requested ability to work with files. Now you can do it without limitations

Create and write to file:

:global newContent "new file content\r\nanother line\r\n";
[/lua "local f=assert(io.open('/test.txt', 'w+')); f:write(newContent); f:close()" ];
Read file content to variable:

:global cnt ""
[/lua "local f=assert(io.open('/test.txt', 'r')); cnt=f:read('*all'); f:close()" ];
:put $cnt
I just found this wiki entry but I have not tried to adapt to blacklists ..... if this code actually works that would be excellent.
Is /lua back then?
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 3:39 pm

Don't think so. That Wiki page states : This page was last edited on 18 October 2017, at 10:37.
As it says on the page : After RouterOS v4.0beta4, Lua support is removed until further notice
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 4:38 pm

Don't think so. That Wiki page states : This page was last edited on 18 October 2017, at 10:37.
As it says on the page : After RouterOS v4.0beta4, Lua support is removed until further notice
My sincere apologies -- I did not see the part that After RouterOS v4.0beta4, Lua support is removed until further notice

What a shame, all that Lua stuff should be removed IMO BUT if certainly would be nice if MikroTik brought back LUA support or provided another means to work with any file size.
 
Lebzul
Member Candidate
Member Candidate
Posts: 110
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 23, 2020 8:09 pm

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.

BTW, is there a way to have these working?
https://github.com/firehol/blocklist-ipsets/blob/master/firehol_level1.netset			40.8 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_wannacry.ipset	         6.15 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_bruteforce.ipset.     	4.64 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield_30d.netset					2.17 KB
https://github.com/firehol/blocklist-ipsets/blob/master/spamhaus_edrop.netset				1.98 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield_7d.netset					1.5 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_webscan.ipset	1.42 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield.netset						1.04 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_wormscan.ipset	0.97 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_dnsscan.ipset	0.86 KB
Last edited by Lebzul on Sat May 23, 2020 10:25 pm, edited 2 times in total.
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 23, 2020 8:38 pm

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.
Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So......


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly "prepare" the file before download to the device, but that cannot be the show stopper I guess.
 
Lebzul
Member Candidate
Member Candidate
Posts: 110
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 23, 2020 10:27 pm

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.
Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So......


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly "prepare" the file before download to the device, but that cannot be the show stopper I guess.
I see. And then, how do people do with servers with an open port? Let's say people need to access my server in a specific port?
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 24, 2020 12:05 am

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.
Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So......


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly "prepare" the file before download to the device, but that cannot be the show stopper I guess.
I see. And then, how do people do with servers with an open port? Let's say people need to access my server in a specific port?
It is important to take all considerations into account when you make the design. If "people" are in fact scattered across the world coming from virtually anyplace then perhaps you need to provide this service at another level. Eg. use some form of authentication with your users (possibly combined with VPN-application).
What is sitting behind this specific port ? Is this something that understand the concept of user-authentication ?
If you run a business and you know your users (eg. employees) are located in country X then filter strict and only allow country X IP which will reduce the surface already A LOT.
 
Lebzul
Member Candidate
Member Candidate
Posts: 110
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu May 28, 2020 6:08 am

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
Lv1 was working fine and now it is not. Probably it does not fit anymore.
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu May 28, 2020 1:15 pm

Lv1 was working fine and now it is not. Probably it does not fit anymore.
You [everyone] should be aware that:

level1 check frequency = 1 minute and average update frequency = 2 hours and 27 minutes
level2 check frequency = 1 minute and average update frequency = 17 minutes
level3 check frequency = 1 minute and average update frequency = 45 minutes
level4 check frequency = 1 minute and average update frequency = 44 minutes
webclient check frequency = 1 minute and average update frequency = 8 hours and 36 minutes
webserver check frequency = 1 minute and average update frequency = 23 hours and 16 minutes

So why is this important to note?
Because changes [adds/deletions] are frequent and that can have a dramatic change in file size.
Also of importance to note is that many duplicates reside when lists are combined - so your processing engine needs to remove duplicates and then reorders them for faster processing.

Depending on which MikroTik Router model being used MOAB combines some of these lists or ALL of these lists 3 times each day spaced 8 hours apart.
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 29, 2020 3:18 pm

What is the recommended way to find out *why* an update failed?

Address list <Spamhaus DROP> update failed

Is great to see in the logs, but where do I look to try and figure out why it failed?

Spamhaus DROP and EDROP are not over 63 kb, so that isn't the reason..

At the moment I am focusing on the IPs used for email SPAM, but it doesn't really matter.. I have the Spamhaus and Bambenek lists failing but I don't know why.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 29, 2020 9:52 pm

What is the recommended way to find out *why* an update failed?

Address list <Spamhaus DROP> update failed

Is great to see in the logs, but where do I look to try and figure out why it failed?
This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 29, 2020 10:42 pm


This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?
They don't load always.
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 3:57 pm


This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?
They don't load always.
@kevinds
You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
So it is critical that duplicate IP be avoided via a pre-process that first checks for duplicates, removes the duplicates, reorders [sorts] the list for faster processing then proceeds with the load.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 15
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 7:25 pm

You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
Script ignores duplicates via on-error={}, processing is not interrupted.
RB951G-2HnD / RouterOS 6.48.5 (Long-term)
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 8:37 pm

You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
Script ignores duplicates via on-error={}, processing is not interrupted.
Do you mean this line: on-error={:log warning "Address list <$description> update failed"} ?

Where in your script do you check for duplicate ip?
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 8:44 pm


Do you mean this line: on-error={:log warning "Address list <$description> update failed"} ?
comment=$description timeout=1d} on-error={}
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 31, 2020 3:40 pm

comment=$description timeout=1d} on-error={}
Thanks .... I just tested @Shumkov code and it works very nicely .... excellent work.
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 31, 2020 4:38 pm

Thanks .... I just tested @Shumkov code and it works very nicely .... excellent work.
Yeah, I have a couple honeypot IPs that when hit, adds the IP to a drop rule, then a script that runs that expands the /32 to a larger block.. I needed something similar to handle multiple IPs from the same larger block.. For when asshats decide to use an entire /16 to do a port-scan of every port.. lol

That was interesting to watch.. haha

But yeah, I still have lots to learn, but I'm not sure how to get a better log for why both variations are failing.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Jun 01, 2020 11:55 pm

My version:
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
Collecting ranges of IP addresses that are knocking at the door: viewtopic.php?f=2&t=152953&p=758068&hil ... os#p758068
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Jun 02, 2020 12:05 am

My version:
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
Collecting ranges of IP addresses that are knocking at the door: viewtopic.php?f=2&t=152953&p=758068&hil ... os#p758068
Ok.. Reading this...

It downloads the list.

It tries to import it

If successful, gives a successful message,
If import fails it says too big..

So if import fails for any reason, it says too big, what if it fails for another reason?

I don't see your version checking it's size beforehand, so the error message could say 'Failed because a butterfly flapped it's wings..' and would still be a more useful error message (because it wouldn't be stating an incorrect reason for it to fail). ;)

I hope I am wrong reading this.. If I am, I am very sorry.
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Jun 02, 2020 12:14 am

Collecting ranges of IP addresses that are knocking at the door: viewtopic.php?f=2&t=152953&p=758068&hil ... os#p758068
I do something very similar to the linked thread..

I have honey-pot IP addresses, anything that attempts to connect to them, gets their IP added to the block list, these addresses have never been used, so nothing legitimate would have any reason to try and connect.

Then another script runs and turns them into a /24, with a 7 day timeout..

Usually, the router has 60-75k addresses in the list at any time. After a reboot the list is reset, takes 6-12 to get back up there.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 2342
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Jun 02, 2020 8:28 am

I have honey-pot IP addresses, anything that attempts to connect to them, gets their IP added to the block list, these addresses have never been used, so nothing legitimate would have any reason to try and connect.
I do nearly the same. Since I do not have an extra public IP, I have and access rule that if any tries to connect to a port that is not open, they get blocked to all ports (65535-6 ports) , also normally open port (6 ports) for 24 hour.
 
Try Splunk> to monitor your MikroTik Router(s). How to set it up. :mrgreen:

MikroTik->Splunk
 
 
Sigma

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Jun 07, 2020 10:01 pm

Have to say thank you OP for the Script works great out-of-the-box. :D

Sincerely
Sigma
 
User avatar
mac86
Member Candidate
Member Candidate
Posts: 122
Joined: Sat Nov 25, 2006 12:52 am
Location: bahia blanca - argentina
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Jun 20, 2020 12:38 am

Hello!
...........

EXCELENT POST.
THANK YOU!!!
https://www.netpro-ar.com
MTINE - MTCWE - MTCRE - MTCTCE - MTCNA
Patagonia Argentina IT Consultant

Mikrotik user from V2.7.7 - Aug/2003
 
Lebzul
Member Candidate
Member Candidate
Posts: 110
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Jul 18, 2020 3:38 pm

comment=$description timeout=1d} on-error={}
Thanks .... I just tested @Shumkov code and it works very nicely .... excellent work.
If this aforementioned line is like that, then the OP has a typo.
 
faxxe
newbie
Posts: 36
Joined: Wed Dec 12, 2018 1:46 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Oct 24, 2020 6:48 pm

Hi
i tryed the different scripts but get on all lists "Address list <name of the list> update failed"
CCR1009 v6.46.7

What could be wrong?

-faxxe
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Oct 24, 2020 9:29 pm

Hi
i tryed the different scripts but get on all lists "Address list <name of the list> update failed"
CCR1009 v6.46.7
What could be wrong?
-faxxe
Do you have by any chance spaces or special characters in the names of the lists?
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
faxxe
newbie
Posts: 36
Joined: Wed Dec 12, 2018 1:46 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Oct 24, 2020 11:09 pm


Do you have by any chance spaces or special characters in the names of the lists?
In which lists? I have to define them before running the scripts? :/ Maybe i use it wrong ...

-faxxe
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Oct 25, 2020 12:08 am

It is indeed a bit confusing. Original there was one address-list named blacklist and the desciption/comment separated the different imported address-list.

Please post the scipt you use then can have a look at it.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
faxxe
newbie
Posts: 36
Joined: Wed Dec 12, 2018 1:46 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Oct 25, 2020 12:40 am

Please post the scipt you use then can have a look at it.
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
Thank you, msatter :)

-faxxe
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Oct 25, 2020 1:33 am

Quick check. The first line you are changing to /ip firewall address-list but you not copied the needed a "/" in front when already being already in a menu.

Update: I see that all omit this "/" and this works if you are already in the root of the menus. I always put a "/" in front to be sure I land where I need, every time, where ever I am.

This list contains no IP addresses anymore and should be removed from your list.
http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
faxxe
newbie
Posts: 36
Joined: Wed Dec 12, 2018 1:46 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Oct 25, 2020 9:02 am

Update: I see that all omit this "/" and this works if you are already in the root of the menus. I always put a "/" in front to be sure I land where I need, every time, where ever I am.
Thank you, i add the "/" to the first line but with the same result. All lists cant get updated

-faxxe
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Oct 25, 2020 10:05 am

Update: I see that all omit this "/" and this works if you are already in the root of the menus. I always put a "/" in front to be sure I land where I need, every time, where ever I am.
Thank you, i add the "/" to the first line but with the same result. All lists cant get updated
-faxxe
Are you running the code directly in terminal or do you as intented put it in a script box and run then the script?

To test I have put "{" at the beginning and at the end a "}" in my version and check if you can run that directly in Terminal. This will also work in a script box (/system script)

viewtopic.php?f=9&t=152632&p=824755#p759427

Remark: on the moment both files are smaller than 64KB so they load with not problem.

It could also be the case that your firewall settings don't allow to download directly from the router. Test this with this in Terminal:
 /tool fetch url=http://feeds.dshield.org/block.txt  as-value output=user 
If you get: failure: connection timeout then your firewall is blocking.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
faxxe
newbie
Posts: 36
Joined: Wed Dec 12, 2018 1:46 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Oct 25, 2020 11:06 am

It could also be the case that your firewall settings don't allow to download directly from the router. Test this with this in Terminal:
 /tool fetch url=http://feeds.dshield.org/block.txt  as-value output=user 
If you get: failure: connection timeout then your firewall is blocking.
Sir, you are entitled to a beer at my expense :) That's the problem. Connection timeout .....
I can use the ping command but I cannot download anything to the router. I have to solve this now.
Many thanks for your patience and helpfulness....
-faxxe
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Oct 27, 2020 6:45 am

Alright, I made some tweaks to allow logins to the download servers, in case the IP list you want to download is password protected..

Variant #1:
/ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url user=$user password=$password output=user as-value]->"data")
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=https://feeds.dshield.org/block.txt user="anonymous" password="anonymous" description=DShield delimiter=("\t") cidr=/24
$update url=https://www.spamhaus.org/drop/drop.txt user="anonymous" password="anonymous" description="Spamhaus DROP" delimiter=("\_")
$update url=https://www.spamhaus.org/drop/edrop.txt user="anonymous" password="anonymous" description="Spamhaus EDROP" delimiter=("\_")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt user="anonymous" password="anonymous" description="Abuse.ch SSLBL" delimiter=("\r")

Variant #2:
/ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url user=$user password=$password output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt user="anonymous" password="anonymous" description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt user="anonymous" password="anonymous" description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt user="anonymous" password="anonymous" description="Spamhaus EDROP" delimiter=("\_")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt user="anonymous" password="anonymous" description="Abuse.ch SSLBL" delimiter=("\r")
I don't know if the "user" and "password" are required to be set to "anonymous" on each update line, I did in case it sends the value from a previous line, to know/predict exactly what it is doing.

I figured out why the two variants were failing for me above.. It is/was a common complaint/issue I have with RouterOS, it was using the wrong Source-IP address.
 
sachlj
just joined
Posts: 1
Joined: Fri Oct 30, 2020 10:11 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 30, 2020 3:05 pm

$update url=http://feeds.dshield.org/block.txt description="DShield" delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=https://osint.bambenekconsulting.com/fe ... t-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
$update url=http://malc0de.com/bl/IP_Blacklist.txt description="malc0de" delimiter=("\n")
$update url=https://iplists.firehol.org/files/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")
$update url=https://iplists.firehol.org/files/firehol_level1.netset description="FireHOL Level1" delimiter=("\n")
$update url=https://raw.githubusercontent.com/hecto ... g/list.txt description="hectorm adaway.org" delimiter=("\n")
..........................
$update url=https://raw.githubusercontent.com/hecto ... g/list.txt description="hectorm adaway.org" delimiter=("\n")
https://gist.github.com/sathwikv143/d2a ... 38342ef455
............................
 
kevinds
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 02, 2020 1:19 pm

$update url=http://feeds.dshield.org/block.txt description="DShield" delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=https://osint.bambenekconsulting.com/fe ... t-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
$update url=http://malc0de.com/bl/IP_Blacklist.txt description="malc0de" delimiter=("\n")
$update url=https://iplists.firehol.org/files/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")
$update url=https://iplists.firehol.org/files/firehol_level1.netset description="FireHOL Level1" delimiter=("\n")
$update url=https://raw.githubusercontent.com/hecto ... g/list.txt description="hectorm adaway.org" delimiter=("\n")
..........................
$update url=https://raw.githubusercontent.com/hecto ... g/list.txt description="hectorm adaway.org" delimiter=("\n")
https://gist.github.com/sathwikv143/d2a ... 38342ef455
............................
Why all of these? Did you check what any of them offer? More than one of these has been taken down/offline. At least one of them is too big for RouterOS as well, at least one is a list of IP list URLs, not an IP list that can be imported.
 
nickcarr
just joined
Posts: 5
Joined: Tue Jul 13, 2021 6:43 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Wed Jul 14, 2021 5:04 pm

Hello!
The new parameter "output=user" provided new scripting capabilities that I decided to take full advantage of.
....

P.S. Sorry for my English.
Thanks for posting it. And also to other Ppl.
Great job
 
elstiv73
just joined
Posts: 9
Joined: Wed Jun 10, 2020 9:34 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Aug 05, 2021 2:51 pm

Many thanks for this brilliant script. Should this be run via the scheduler (interval) every day ? I am asking because under the column 'Timeout' there is a 24 hour countdown timer. I am not sure whether the script refreshes itself automatically every 24 hours or whether we should refresh it via scheduler interval. I am using the first variant of the code. Thanks
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Aug 05, 2021 3:08 pm

I figured out why the two variants were failing for me above.. It is/was a common complaint/issue I have with RouterOS, it was using the wrong Source-IP address.

/tool fetch src-address=
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Aug 05, 2021 5:24 pm

search tag # rextended definitive ip posix regex

remember than this are wroten for be put directly on script,
if tested on terminal you must add \ before the ?
if tested on regex101 must be removed \ before \. and \/


The used regexp
[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}
match from 0.0.0.0 to 999.999.999.999

My POSIX regex is better, also match subnet mask, if present:
([0-2]{0,1}[0-9]{1,2}\\.){3}[0-2]{0,1}[0-9]{1,2}(\\/[0-3]{0,1}[0-9]{1,1}){0,1}
0.0.0.0 to 299.299.299.299
000.000.000.000 to 299.299.299.299
xxxx/0 to /39
xxxx/00 to /39


Correct regex to match exactly from 0.0.0.0/0 (or 000.000.000.000/00) to 255.255.255.255/32
is too much complicated and the CPU go 100% until all lists are parsed...
with mandatory subnet mask
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[0-2]?[0-9])

with optional subnet mask
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])(\\/(3[0-2]|[0-2]?[0-9])){0,1}

without subnet mask
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])
can be checked using :toip, if the result checked from :typeof is not "ip" or "ip-prefix", is not a valid IP or IP/Prefix

also must be skipped some IPs for not self-block all own networks if for errors (or not) own LAN IP or WAN IP go on blacklist...
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 3:51 am

A Frankenstein script using HTTP chunking. It is not perfect because it can't predict where the splits are made so you will miss some data or data is wrong. But is a start and certainly it can be improved. This ofcourse only usable if the HTTP server supports chunking.
USE LATER VERSION PUBLISHED BELOW
Inspiration: viewtopic.php?f=9&t=177530#p873931

Update: Chunking problem solved by using a negative overlap of 512 bytes for each part. I first reduced the maxsize with 512 bytes so they are in sync.
Update 2: removing the first array line of each chunk to avoid importing incomplete lines.
Last edited by msatter on Tue Aug 24, 2021 10:26 pm, edited 8 times in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 10:00 am

Thanks both @msatter and @rextended for this revamp/update of the "generic lists downloader" able to pass the 64K boundary !
I'll give it a try for sure!
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 10:28 am

I've test against the Project Turris list and it obtained 9058 entries ? Did you get similar values ?
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 12:05 pm

I've test against the Project Turris list and it obtained 9058 entries ? Did you get similar values ?
Yes, the first line of the file is ignored and the last 6 are IPv6 addresses. The script is still in the workings, however the first results are promising.

Update: Script is updated to avoid importing the incomplete first and last lines of a chunk that might be corrupt because of chunking.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
profinet
just joined
Posts: 5
Joined: Mon Apr 23, 2018 1:17 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 12:46 pm

Thank you @msatter. I posted this two scripts in another post, but i didn't know how combinated it.

It is possible added a delimiter option for import spamhaus database or another in one script?
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 1:48 pm

Thank you @msatter. I posted this two scripts in another post, but i didn't know how combinated it.

It is possible added a delimiter option for import spamhaus database or another in one script?
Those use a range (example: /22) and then the RegEX has to be adapted/extended.

On second thought, I think it will work as is now.

viewtopic.php?f=9&t=152632&p=873984#p826157
Last edited by msatter on Sun Aug 22, 2021 2:20 pm, edited 1 time in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 2:09 pm

@msatter
See, you hope they throw me out of the forum and then you use my scripts...
Really pathetic...
You have to "wait" for someone to quote them to see them, eh???

The original post is here, and is indicated also how to manage the "chunk"
viewtopic.php?f=9&t=177530#p872372
Obviously some line are splitted on two between two parts like
\n69.100.54.120/19\n\
splitted on two, first read end with:
..............\n69.100.5
second read start with:
4.120/19\n\..............

The solution is simple, instead of read exactly the max 64KB (64512B) make part slightly less like 63K (63488B) to have room on variable to put previous "reminder" on top,
like if "reminder" is "69.100.5" and file part is "4.120/19\n\.............." with
:set varcontent "$reminder$filepart"
is obtained inside varcontent "69.100.54.120/19\n\.............."
and the IP is readable correctly.
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 8:16 pm

I've also changed/replaced the regex inside the script with the one @rextended explained earlier, this seems the most complete one.
I've managed to load/import all US-cidr onto a list, 61259 entries long just for testing. It took a loooong time on my RB3011 utilizing 1-core, but that is nothing new and already discussed earlier.
Doesn't really matter with these kinds of list that do not change often.

The more entries that added to the list, the slower it gets. I had the impression once you go above 30.000-40.000 adding few thousand entries get really, really slow.
So for reasonable small amount of entries (eg. 5000-15000) is looked pretty fast.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 8:45 pm

The RegEX is only there to detect if there is a valid IP adress on the line.

This version only allows one list and there are different versions that can do more. Even a version that allows you to provide login credentials if needed.

I made this Frankenstein version to see if it worked.

About huge lists. Importing those huge list should be not slow as long you first delete the old entries. To optimize stuff, a second array named array containing the IP adresses in the current list. We don't look in the address-list but instead in the array. This was made for lists up to 63KB but chunking that also now, to could optimize it. However not all lists are uniform and that makes chunking there flaky. Beter is to just not compare and always delete all dynamic entries before adding what alway the fasted method.

However it is already a long time ago I worked with others on this so I am I bit of touch on the actual working.

Update: for HUGE lists, variant 1 in OP
That was easy, three snips and the compare array was disabled, importing Turris was much much faster this way. Try with this version on a HUGE list and let me know?
Update 2:
I replaced the code with a more streamlined one and added some comments to make it easier to understand the workings. Remark, today Turris did it's weekly update and the number of lines is now 8327. I am taking this now in production on my own router and can do deactivate the cron which created the to imported list on a Linux system.
Update 3: added the option to use extra filtering like keywords in a RegEX like dns|sip (Hei rules) to only have those lines accepted out of the whole list. This is checked every line.
Update 3.5: small optimizations by not using variables if you only that one once and use a variable if that one is used several times.
Example:
$update url=https://project.turris.cz/greylist-data/greylist-latest.csv delimiter=, listname=turris timeout=8d heirule=dns|sip

{
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :local filesize ([/tool fetch url=$url keep-result=no as-value]->"total")
 :local start 0
 :local maxsize 64000;		# reqeusted chunk size
 :local end ($maxsize - 1);	# because start is zero the maxsize has to be reduced by one
 :local partnumber	($filesize / ($maxsize / 1024)); # how many chunk are maxsize
 :local remainder	($filesize % ($maxsize / 1024)); # the last partly chunk 
 :if ($remainder > 0) do={ :set partnumber ($partnumber + 1) }; # total number of chunks
  :put "Deleting all Dynamic enties in address-list: $listname"
  :if (heirule != null) do={:put "Using as extra filtering: $heirule"}
  :if ($heirule = null) do={:set $heirule "."}
 # remove the current list completely
 :do { /ip firewall address-list remove [find where list=$listname dynamic]} on-error={};
 :for x from=1 to=$partnumber step=1 do={ :put "Reading Part: $x $start - $end"
   # fetching the chunks from the webserver
   :local data ([:tool fetch url=$url http-header-field="Range: bytes=$start-$end" output=user as-value]->"data")
   # Only remove the first line only if you are not at the start of list
   :if ($start > 0) do={:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
     :while ([:len $data]!=0) do={
       :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
       :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={    
        :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$description timeout=$timeout} on-error={}; # on error avoids any panics        
       }; # if IP address && extra filter if present
      :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
      # Cut of the end of the chunks by removing the last lines...very dirty but it works
      :if ([:len $data] < 256) do={:set data [:toarray ""]}    
     }; # while

  :set start (($start-512) + $maxsize); # shifts the subquential start back by 512
  :set end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the 
 }; #do for x
 :put "Completed importing $listname."
}; # do

$update url=https://project.turris.cz/greylist-data/greylist-latest.csv delimiter=, listname=turris timeout=8d heirule=dns|sip
}
Last edited by msatter on Mon Aug 23, 2021 10:05 pm, edited 8 times in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 9:44 pm

A rerun of this script on my (already existing) [test] access-list "us-blacklist" (61250 entries) only updates the dynamic timers (reset to 1d / 24:00:00) -> This whole operation whent very fast, less than 1 minute.

A complete erase + re-run of this version of the script is much,much faster. All +- 62k entries inserted into the list a "a few minutes" (< 5, didn't take out the stopwatch)
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Aug 22, 2021 11:21 pm

Ping
"Give me a ping, Vasili. One ping only, please."
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
mozerd
Long time Member
Long time Member
Posts: 550
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 6:19 pm

Try with this version on a HUGE list and let me know?
Update 2:
I replaced the code with a more streamlined one and added some comments to make it easier to understand the workings. Remark, today Turris did it's weekly update and the number of lines is now 8327. I am taking this now in production on my own router and can do deactivate the cron which created the to imported list on a Linux system.
@msatter, excellent code ..... very very fast !!!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 6:24 pm

@msatter, excellent code ..... very very fast !!!
If it wasn't for me, for the methods of download big files (> 64K) @msatter would have nothing to work on...
viewtopic.php?f=9&t=177530

It's very comic, I'm on foe list of @msatter and he use my code inside his router!!! ahahahahaha!!! :lol: :lol: :lol:
Last edited by rextended on Mon Aug 23, 2021 7:02 pm, edited 2 times in total.
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 6:47 pm

@msatter, excellent code ..... very very fast !!!
If it wasn't for me, for the methods of download big files (> 64K) @msatter would have nothing to work on...
viewtopic.php?f=9&t=177530

It's very comic, I'm on foe list of @msatter and he use my code inside his router!!! ahahahahaha!!! :lol: :lol: :lol:
Nah don't worry about that. It's very obvious you have contributed the pivotal aspect of this approach/solution.
Never has anybody came up with this concept before to my knowledge, I've never seen it in any posting over the past years.
That was some clever problem solving! and you deserve full credit for this one.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 7:01 pm

Never has anybody came up with this concept before to my knowledge, I've never seen it in any posting over the past years.
That was some clever problem solving! and you deserve full credit for this one.
Remember, you and @jotne were my inspiration for that!
viewtopic.php?f=9&t=166293&p=872435#p872376
Last edited by rextended on Mon Aug 23, 2021 7:04 pm, edited 3 times in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 7:02 pm

I still do not like this method of importing list on address-list without any sanitization first, and the use of on-error also do not have any sense, like on delete.
Sooner or later or for some "on-error", or on purpose from website where the list is,
on address-list go 0.0.0.0/0 and block all, or a wrong prefix like 151.99.125.9/2 (instead of /24) block all from 128.x.x.x to 191.x.x.x
because 151.99.125.9/2 is imported on routeros like 128.0.0.0/2


viewtopic.php?f=9&t=166293#p872049
I'm already working on a method than use lists >64K and sanitizing what are imported, like:
[...]
4) Create whitelist, before add the IP / IP prefix check if it is on whitelist, then if is it, no add
5) Check on add if the ip-prefix is already present inside other IP-prefix already on address-list
6) Check on add if the ip-prefix is comprehensive of one or more IP-prefix on address-list, remove old(s) and add new bigger.
7) for security accept only from /12 to /32 prefix. /11 or less on IPv4 is too much big for be true...
8 ) Set an option for put the IP on the address-list but on temporary way (Dynamic) for specified time (from 1 second to near 35 weeks),
this do not export this type of IP on address-list on export or backup
whith this option set, if the address is found again on the imported list, instead to delete it and re-import, have time resetted again (from 1 second to near 35 weeks)
[...]
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 7:25 pm

Talking about sanitizing, I did stumble today on the fact that my "Turris" list was suddenly down to only +- 3000 entries!
This happened at the time the list performs its update in the morning. Very weird.
In the afternoon I completely flushed/erased the list and started the script manually and now its up with +- 8K entries.

Not too sure what happened there, and I'll be manually testing some more because such thing needs to be rock-solid and handle weirdness as it is ingested & processed, even it takes significantly longer.

Image


Hmm, I've start the script and did see 2 entries in the log fly by : Address list <> update failed
The list seems to maintain the amount of items then before I started the script, so nothing was flushed nor is the timestamp updated.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 8:48 pm

It is a weekly list so just update it at 06H30 am and polling it will only create mor load on their side. No wonder there is now a proxy in front. ;-)

Also nice is that you can select certain kinds of list. This could be a second selector beside the IP address presence.
Legend for current Hei rules
----------------------------

amplifiers           Easily exploitable services for amplification
broken_http          Broken inbound HTTP (known services)
cryptocoin           Cryptocoin miners
databases            Database servers
dns                  Incoming DNS queries
http_scan            HTTP/S scans
low_ports            Low ports (<1024)
netbios              NetBIOS
netis                Netis router exploit
ntp                  NTP
proxy_scan           Scans for HTTP/S and SOCKS proxies
remote_access        Remote access services (RDP, VNC, etc.)
samba                Samba (Windows shares)
sip                  SIP ports
ssdp                 SSDP
ssh                  SSH
synology             Synology NAS
telnet               Telnet
torrent              Common Torrent ports
Update to the script above: added the option to use extra filtering like keywords in a RegEX like dns|sip (Hei rules) to only have those lines accepted out of the whole list. This is checked every line. If this option is omitted then every line with an valid IP address is imported.

Example:
$update url=https://project.turris.cz/greylist-data/greylist-latest.csv delimiter=, listname=turris timeout=8d heirule=dns|sip
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Aug 23, 2021 9:47 pm

Not really relevant I did pick up something weird. So I have the Turris list loaded and I'm also using this in the forward chain preventing any communication from inside my LAN towards any outside IP on that list.
Strangely enough I get some hits on this ;-(
The IP address 47.94.96.203 seems to go back to an IP on the list below, belonging to Alibaba Advertising Holding or something.

https://github.com/firehol/blocklist-ip ... ous.netset

The problem is that it seems to originate from my NAS, on which +15 docker containers are running, 3 VM's etc,etc.
At the moment its not very clear who initiates.
All my containers that are running are from trusted repo's etc. No funny stuff to my knowledge. (more things like influxdb,grafana,telegraf,watchtower,mosquitto etc)

The packet towards the Alibaba IP was "ICMP" , only a single instance.
The packet towards another Turris marked IP is also found in the abuseIP database.
This packet was dropped trying to creep out of my LAN coming from "something" on my NAS, source-port 6800 > dst-port tcp/48881

I've tried on my Synology using tcpdump on the "docker0" bridge instance so I see a lot of action of 172.17.x.x (internal) container traffic, but I could not capture anything trying to reach the above IP's...
Interesting .... a home network ;-)

https://www.abuseipdb.com/check/45.146. ... WjcnBszQil


Interesting observations ;-)
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 24, 2021 2:09 am

The packet towards the Alibaba IP was "ICMP" , only a single instance.
The packet towards another Turris marked IP is also found in the abuseIP database.
This packet was dropped trying to creep out of my LAN coming from "something" on my NAS, source-port 6800 > dst-port tcp/48881

I've tried on my Synology using tcpdump on the "docker0" bridge instance so I see a lot of action of 172.17.x.x (internal) container traffic, but I could not capture anything trying to reach the above IP's...
Interesting .... a home network ;-)

Interesting observations ;-)
Do you have Solar-panels & inverter or/and using ModBus to access them? That port range is other used to discover inverters or other ModBus stuff, however normally UDP is used as broadcast.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 24, 2021 9:13 pm

NOTE : The mystery on the ACL-hit got solved, it turned out to be old port-forwarding for torrent-traffic that hitted the NAS (torrent client not running) but the NAS effectively replied back that IP address in stead of playing silent. No worries.

However again today the update of the list did not go well,

At 2 PM, the script starts, suddenly loosing quite some entries.
The weird thing : it remains stable for a few hours until around 5PM and I notice the list was completely emptied and was also removed. Hence no more data after 6PM
So does the latest version of the script UPDATES the dynamic timer ? Or does it create from scratch all entries that are downloaded ?
Why the heck would it start throwing out entries hours after the download...

Image
Last edited by jvanhambelgium on Wed Aug 25, 2021 12:06 am, edited 1 time in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 24, 2021 10:13 pm

I have trouble following you on this.

If you read about the Greylist by Turris you will see that is updated once a week. They know that. If you keep hammering the proxy they might put you in SRC-IP adress jail. ;-)

BTW you are using an old verdion of the scipt that keeps the list active for one day. I use eigth days. Seven days is the refresh and one day spare. After seven days scheduler read the new list that not refreshed by Turris for an other seven days.

Update: you're using a version that does not remove old list and you could end up blocking addresses that are not on the list any more.
Last edited by msatter on Tue Aug 24, 2021 10:24 pm, edited 1 time in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 24, 2021 10:23 pm

Ah, yeah you mentioned something like this earlier. I'll adapt the timeout to 1w of each of the entries and schedule it to run 1x / week
But nevertheless, I wonder why it behaves like I see ; the "drop" in entries I can understand if I receive only partial info from their end, but why the sudden drop to "0" entries several HOURS later.
That is the part that is not clear to me, as if the dynamic entries had a short lifespan....but the script issues 1d lifetime.

hmm, yeah, lets start issuing only 1x / week and we'll see...

Thx for pointing it out.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 24, 2021 10:32 pm

I have removed the first version of the script to avoid this happening to others. The first version was more a prove of concept that the Frankenstein, two parts joined, script worked.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 31, 2021 12:25 pm

I have added support for domain names beside IP addresses. Not tested yet but it should work.

In bold the changes an I hope the '+' is supported. Else it could be replaced by a '*'.
:if (( $line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" || $line~"^.+\\.[a-z.]{2,7}" ) && $line~heirule) do={
This version is replaced by the version below. That one finds out the delimiter on it's own.
Last edited by msatter on Thu Sep 09, 2021 4:46 pm, edited 1 time in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Aug 31, 2021 1:32 pm

search tag # rextended DNS RegEx

POSIX syntax:
^(([a-zA-Z0-9][a-zA-Z0-9-]{0,61}){0,1}[a-zA-Z]\.){1,9}[a-zA-Z][a-zA-Z0-9-]{0,28}[a-zA-Z]$

for MikroTik:
$line~"^(([a-zA-Z0-9][a-zA-Z0-9-]{0,61}){0,1}[a-zA-Z]\\.){1,9}[a-zA-Z][a-zA-Z0-9-]{0,28}[a-zA-Z]\$"

limited plausibly to 9 levels label+domain x9x.x8x.x7x.x6x.x5x.x4x.x3x.x2x.x1x.domain
and limited 30 characters for top domain (the longest actually existant is 24 characters XN--VERMGENSBERATUNG-PWB )

Rule for DNS names:
the format is label.domain or label2x.label1x.domain or label3x.label2x.label1x.domain etc. (ignoring never present on address list fqdn label.domain. )
max length for label and domain is 63 characters, but the longest domain today is 24 characters (XN--VERMGENSBERATUNG-PWB)
min length for label and domain are formerly 1 characters for label and 2 for domain
allowed characters for label and domain are case-insensitive a-z A-Z number 0-9 and the minus - (the _ are used for special cases, not for full domain name)
the first or the last (or the unique) character of label or domain can't be -
the last (or the unique) character of label or domain can't be a number
the first character of domain can't be a number
the first character of label can be a number but must be followed by at least one letter
the max lengt of the string must be 253 characters
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

[Auto find delimiter] Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Sep 09, 2021 4:44 pm

I have implemented the determination of the delimiter. This should make the import script more flexible and no need anymore to specify the different delimiters for each list.

A version that also recognizes what kind of format is used, IPv4/IPv4 with range/domain names/IPv6. A list can then only contain one type. Having to check every line will be doable but I assume the whole script would become very slow.

Update: added support for different kinds of lists. Supported are plain IPv4, IPv4 with range and domain names. I use a simple regEX for domain names.
Update 2: allows mixed lists when delimiter is set in the config line and it shows what the delimiter is. Output shows the kind of list is recognized.
:local R "[0-9]{1,3}"; # storing RegEX part in variable to have shorter strings in the code
:if ($sline ~ "^$R\\.$R\\.$R\\.$R")		do={:set $posix "^$R\\.$R\\.$R\\.$R";}
:if ($sline ~ "^$R\\.$R\\.$R\\.$R/[0-9]{1,2}")	do={:set $posix "^$R\\.$R\\.$R\\.$R/[0-9]{1,2}"}
:if ($sline ~ "^.+\\.[a-z.]{2,7}")		do={:set $posix "^.+\\.[a-z.]{2,7}"}
A strange thing was that I could not use :local in code above and had to resort to :set

When using a defined delimiter should allow to import mixed lists. This has still be implemented and can be done by using a regEX.
New version can be found in a later posting.
Last edited by msatter on Sat Sep 11, 2021 11:01 am, edited 7 times in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Sep 09, 2021 5:08 pm

Well done and nice idea...

If you want, you can use those regex for determine what type of items the file containing: DNS, IP-Prefix or only IP
search for valid DNS
(([a-zA-Z0-9][a-zA-Z0-9-]{0,61}){0,1}[a-zA-Z]\\.){1,9}[a-zA-Z][a-zA-Z0-9-]{0,28}[a-zA-Z]

IP-Prefix: IP with mandatory subnet mask
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\/(3[0-2]|[0-2]?[0-9])

IP or IP-Prefix if present optional subnet mask
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])(\\/(3[0-2]|[0-2]?[0-9])){0,1}

IP without prefix
((25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]?[0-9]?)[0-9])
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Sep 11, 2021 11:00 am

A new version and I think this is more or less complete now. I have also added a short explanation at the end what all the parameters do.

Update: During import is checked if the source file has changed in size and if so the import is retried after a 2 minutes wait. If no successfull import was possible you get now a specific message.
Next update: If an import failed in the end the list would be erased on forehand. Deleting is only now done on a successful import and this is possible because a all the current entries are renamed to a backup address-list. That backup address-list is removed on a successful import. I don't change the timeout time, so the entries could timeout before the next import. So keep an eye on the log if used.
{
/ip firewall address-list
:local update do={
 :put "Starting import of address-list: $listname"
 :if ($nolog = null) do={:log warning "Starting import of address-list: $listname"}

 :local maxretry 3
 :local retrywaitingtime 120s
 :local retryflag true
 :for retry from=1 to=$maxretry step=1 do={
  :if (retryflag) do={ :set $retryflag false; :set $sounter 0
  :if (retry > 1) do={
   :put "Source file changed. Retring after a $retrywaitingtime wait..."
   :if ($nolog = null) do={:log warning "Source file changed. Retring after a $retrywaitingtime wait..."}
   :delay $retrywaitingtime  }
  
 :local filesize ([/tool fetch url=$url keep-result=no as-value]->"total")
 :local start 0
 :local maxsize 64000;	        # reqeusted chunk size
 :local end ($maxsize - 1);	# because start is zero the maxsize has to be reduced by one
 :local partnumber	 ($filesize / ($maxsize / 1024)); # how many chunk are maxsize
 :local remainder	 ($filesize % ($maxsize / 1024)); # the last partly chunk 
 :if ($remainder > 0)    do={ :set $partnumber ($partnumber + 1) }; # total number of chunks
 :if ($heirule != null) do={:put "Using as extra filtering: $heirule"} else={:set $heirule "."}
 # remove the current list completely if "erase" is not present (default setting)
  :if ($noerase = null) do={  
   :if ($timeout = null) do={:set $timeout 00:00:00; :do {:foreach i in=[/ip firewall address-list find list=$listname] do={/ip firewall address-list set list=("backup".$listname) $i }} on-error={} } else={
   :do {:foreach i in=[/ip firewall address-list find list=$listname dynamic] do={/ip firewall address-list set list=("backup".$listname) $i }} on-error={} };                
   :put ("Conditional deleting all".$dynamic." entries in address-list: $listname")
   :if ($nolog = null) do={:log warning ("Conditional deleting all".$dynamic." entries in address-list: $listname")}
  } else={:put "Entries not conditional deleted in address-list: $listname"}; # ENDIF ERASE
 :for x from=1 to=$partnumber step=1 do={
   # get filesize to be compared to the orignal one and if changed then retry
   :local comparesize ([/tool fetch url=$url keep-result=no as-value]->"total")
   
#:set $comparesize 5 

   # fetching the chunks from the webserver when the size of the source file has not changed
   # empty array when the source file changed. No processing is done till the next complete retry
   :if ($comparesize = $filesize) do={:set $data ([:tool fetch url=$url http-header-field="Range: bytes=$start-$end" output=user as-value]->"data")} else={:set $data [:toarray ""]; :set $retryflag true}
     #:if ($ownposix = null) do={
  # determining the used delimiter in the list if not provided in the config
   # this only run once and so the impact on the import time is low
    :local ipv4Posix	  "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}"
    :local ipv4rangePosix "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}"
    :local domainPosix	  "^.+\\.[a-z.]{2,7}"
    :local sdata $data;     
    :while ([:len $sdata]!=0 && $delimiter = null) do={ # The check on length of $sdata is for if no delimiter is found.
       	:local sline [:pick $sdata 0 [:find $sdata "\n"]]; :local slen [:len $sline];
       	# set posix depending of type of data used in the list
       	:if ($sline ~ $ipv4Posix)	do={:set $posix $ipv4Posix;	 :set $iden "List identified as a IPv4 list"}
       	:if ($sline ~ $ipv4rangePosix)	do={:set $posix $ipv4rangePosix; :set $iden "List identified as a IPv4 with ranges list"}
       	:if ($sline ~ $domainPosix)	do={:set $posix $domainPosix;	 :set $iden "List identified as a domain list"}
       	:if ($sline ~ $posix) do={:put $iden}
      	:if ($sline ~ $posix) do={ # only explore the line if there is match at the start of the line.
	 :do {:if ([:pick $sline 0 ($slen-$send)] ~ ($posix."\$")|| $send > $slen) do={:set $delimiter [:pick $sline ($slen-$send) ($slen-($send-1))]; :set $result true} else={:set $send ($send+1);} } while (!$result);
	}; #IF posix
	:set $sdata [:pick $sdata ([:find $sdata "\n"]+1) [:len $sdata]];
	:if ($delimiter != null) do={:local sdata [:toarray ""]}; #Clear array sdata and it is not needed anymore and triggering so the While to end
    }; #WHILE END $sdata
    :local sdata [:toarray ""] 
   #} else={:put "User defind Posix: $ownposix"; :set $posix $ownposix } ; # ENDIF ownposix = null   
   :if ($posix = null && $delimiter != null) do={:set $posix "."; :put "Using config-line defined delimiter: \"$delimiter\""}; # delimter provided by config line
   :if (!retryflag) do={:put "Reading Part: $x $start - $end"}   
   :if ($timeout = null) do={:local timeout 00:00:00}; # if no timeout is defined make it a static entry.    
   # Only remove the first line only if you are not at the start of list
   
   :if ($start > 0) do={:set $data [:pick $data ([:find $data "\n"]+1) [:len $data]]}
     :while ([:len $data]!=0) do={
       :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable
       :if ( $line ~ $posix && $line~heirule) do={    
        :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$comment timeout=$timeout; :set $counter ($counter + 1)} on-error={}; # on error avoids any panics        
       }; # if IP address && extra filter if present
      :set $data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array
      # Cut of the end of the chunks by removing the last lines...very dirty but it works
      :if ([:len $data] < 256) do={:set $data [:toarray ""]}    
     }; # while

  :set $start (($start-512) + $maxsize); # shifts the subquential start back by 512  
  :set $end (($end-512) + $maxsize); # shift the subquential ends back by 512 to keep the 
  }; # if retryflag
 }; #do for x
 
}; # for retry
 :if ($counter < 1) do={:set $resultline "Import was NOT successfull! Check if the list $listname is still being maintained."} else={:set $resultline "Completed reading $counter items into address-list $listname." } 
 :put $resultline
 :if ($nolog = null) do={:log warning $resultline }
 :if ($counter > 0) do={:do {/ip firewall address-list remove [find where list=("backup".$listname)]} on-error={} } else={
 :do {:foreach i in=[/ip firewall address-list find list=("backup".$listname)] do={/ip firewall address-list set list=$listname $i }} on-error={}
 :put "Restoring backup list: $listname" 
 :if ($nolog = null) do={:log warning "Restoring backup list: $listname"}
 }; # if counter restore on failure and remove on success
}; # do
$update url=https://project.turris.cz/greylist-data/greylist-latest.csv listname=turris timeout=8d heirule=http nolog=1
}

# To be used configline settings:
# url=	        https://name.of.the.list
# listname=	name of address-list

# Optinal settings
# timeout=	the time the entry should be active. If omited then static entries are created.
# comment=	puts this comment on every line in the choosen address-list (default: no comment)
# heirule=	this will select on a word on each line if to import or not (default: no heirule)
# noerase=	any value, then the current list is not erased (default: erase)
# ownPosix=	allow to enter a onw regEX posix to be used (not ative at this moment)
# nolog=        any value, then don't write to the log (default: writing to log)
Last edited by msatter on Sun Sep 12, 2021 11:09 pm, edited 7 times in total.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
jvanhambelgium
Long time Member
Long time Member
Posts: 587
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Sep 11, 2021 11:05 am

A big "Thank you!" towards all contributors!
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Sep 11, 2021 2:43 pm

A big "Thank you!" towards all contributors!
Especially to profinet who thought of combining the two scripts to create this "Frankenstein".
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 5896
Joined: Tue Feb 25, 2014 12:49 pm
Location: Capalbio, Tuscany, Italy

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Sep 11, 2021 2:52 pm

I slow my version because I want also manage fetch errors
(thanks for msatter for the idea of identify inside the type of list)
(I never see a msatter thanks to me for the method for download a file only one piece at time)

viewtopic.php?f=2&t=178355&p=878643#p878643

This is a work in progress code for manage the fetch errors, is ready to paste on terminal for testing purpose.
/file remove [find where name="testfetch.txt"]
{
    :local jobid [:execute file=testfetch.txt script="/tool fetch url=http://mikrotik.com"]
    :put "Waiting the end of process for file testfetch.txt to be ready, max 20 seconds..."
    :global Gltesec 0
    :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
        :set Gltesec ($Gltesec + 1)
        :delay 1s
        :put "waiting... $Gltesec"
    }
    :put "Done. Elapsed Seconds: $Gltesec\r\n"
    :if ([:len [/file find where name="testfetch.txt"]] = 1) do={
        :local filecontent [/file get [/file find where name="testfetch.txt"] contents]
        :put "Result of Fetch:\r\n****************************\r\n$filecontent\r\n****************************"
    } else={
        :put "File not created."
    }
}

on this case we obtain at the end "closing connection: <302 Found "https://mikrotik.com/"> 159.148.147.196:80 (4)"
because "http ://mikrotik.com" redirect to "https ://mikrotik.com/" (and redirect again to "https ://www.mikrotik.com/") (added spaces on purpose)

Who is online

Users browsing this forum: No registered users and 11 guests