Community discussions

MikroTik App
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Oct 01, 2019 11:00 pm

Hello!
The new parameter "output=user" provided new scripting capabilities that I decided to take full advantage of.

- the script does not need third-party servers, since address lists are downloaded directly from the source and processed directly on the router.

- the script does NOT save the downloaded files to the disk (thereby preventing premature wear and failure of the disk).

- the script can be adapted to download and process any number of address lists of a similar format (the maximum file size is 63 KiB (64512 bytes). It is better than 4 KiB :)).

At the moment the script can download and update next lists:
- DShield
- Spamhaus DROP
- Spamhaus EDROP
- Bambenek High-Confidence C2
- Abuse.ch SSLBL

Variant 1:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script deletes all addresses matching the condition "list=blacklist comment=$description", after which it fills out address lists from scratch. It's easier and faster :)

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It's harder and slower :), but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an "array"?
Because the default "find" function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default "find" function.

Required policy: read, write, test.
Perhaps this script will be useful to someone :)

P.S. Sorry for my English :oops:
Last edited by Shumkov on Mon Mar 16, 2020 4:10 pm, edited 12 times in total.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
Zebble
newbie
Posts: 46
Joined: Mon Oct 17, 2011 4:07 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 18, 2019 12:12 am

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
 
liuyao
just joined
Posts: 1
Joined: Wed Sep 04, 2019 9:14 am
Location: China

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Oct 18, 2019 4:29 pm

Hello:

Thank you for sharing。 But the way you write functions is hard to understand. If any boss is rewritten, the written statement is perfect like the official example. Thank you
小白充大神
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:20 pm

Hi - This looks great. I will give it a try.

Update -
I just run this and it works great - no errors and works perfectly

What is general recommendation on how often to grab new lists - daily?

Am I correct it removes or ignores duplicate entries?

It would be great to keep this updated with additional!

Thank you so much for this!!!
Last edited by RackKing on Sun Nov 03, 2019 5:40 pm, edited 1 time in total.
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:37 pm

How does it handle 1.2.3.0/24 addresses and as far I could it enters 1.2.3.0 in the addresslist without the /24?

Update: I ran the script and it does handles the range (cidr) correctly. Going to look if I can add some more lists.

Update 2: excellent script and I have added the option to filter on a specific label in file and that also can be used to remove a list that is not used anymore, from the current blacklist in the addresslist.
Last edited by msatter on Sun Nov 03, 2019 7:57 pm, edited 4 times in total.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 5:42 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 7:50 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
This is the correct syntax
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 7:51 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
It works if poster zeb put it as code here:
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")

REALLY PLEASED with the script from Shumkov and the added option by Mikrotik and it is now very easy to import lists without having to use other computers to prepare the lists up front
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 8:00 pm

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
This appears to fail for me.
It works if poster zeb put it as code here:
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset description="FireHOL Level2" delimiter=("\n")

REALLY PLEASED with the script from Shumkov and the added option by Mikrotik and it is now very easy to import lists without having to use other computers to prepare the lists up front
That Level2 list is huge.... trying to sort the different levels they have. Any thoughts? Also, would you fun this daily?
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 8:39 pm

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Nov 03, 2019 10:53 pm

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
Ah - that makes sense. You are quite correct. Thanks for the explanation on the removal.

Are there any other lists you would consider or a good source?
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 1:58 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
:local data ([/tool fetch url=$url output=user as-value~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}"]->"data");
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:42 am

Do not forget about file size - maximum 63 KiB.
If the file size is larger than the maximum, only part of the file will be processed (the first 63 KiB), and the rest of the file will be discarded.
FireHOL Level2 is bigger than 63 KiB :)
What is general recommendation on how often to grab new lists - daily?
I set the scheduler interval to 8 hours.
In general, the interval depends on the specific list and the frequency of updating this list by its provider.
it removes or ignores duplicate entries?
The script removes only addresses that are in the "blacklist" list and have a comment=description.
It looks like FireHOL Level1 may be a better choice and is under the file size limit.... barely. Any reason no to use this? That large of a list would probably have a pretty big performance hit on the router?

@Shumkov what was your goal/strategy based on the lists you choose? I am trying to sort what lists should be used and what is a happy medium.

Edit - after taking a closer look it appears the individual sources you are using is very similar to firehol_level1. With a goal of having no false positives this is a great place to start. I guess whether you grab them individually or through firehol is personal preference.

What a great script - thank you very much.
Last edited by RackKing on Mon Nov 04, 2019 3:34 am, edited 2 times in total.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:56 am

malc0de

$update url=http://malc0de.com/bl/IP_Blacklist.txt description="Malc0de" delimiter=("\n")
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 9:51 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
This does not work :)
"data" is an element of the array, and is accepted for processing only in its entirety - you cannot process only part of the element.
@Shumkov what was your goal/strategy based on the lists you choose? I am trying to sort what lists should be used and what is a happy medium.

Edit - after taking a closer look it appears the individual sources you are using is very similar to firehol_level1.
That's right, I took FireHOL Level1 as the basis.
I removed “Feodo Tracker” and “Ransomware Tracker”, replaced “Bambenek C2” with “Bambenek High-Confidence C2” (as Bambenek recommended it myself), and also removed “Fullbogons” - I get them using BGP.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 10:38 am

It would be nice if this would be possible using a filter to have only the needed data in the variable. So there would be a lot more space in the variable
This does not work :)
"data" is an element of the array, and is accepted for processing only in its entirety - you cannot process only part of the element.
I agree and my angle is to filter traffic (stream) on the way to the data array.

Like this in scripting:
wget -q -O - $url | gawk --posix --field-separator=, '/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/ { print "$i a=" $1;}'  > $saveTo/$filename
This is something only Mikrotik can create to intercepting the stream.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 2:29 pm

That's right, I took FireHOL Level1 as the basis.
I removed “Feodo Tracker” and “Ransomware Tracker”, replaced “Bambenek C2” with “Bambenek High-Confidence C2” (as Bambenek recommended it myself), and also removed “Fullbogons” - I get them using BGP.
Makes perfect sense. Thank you again so much for this.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Nov 04, 2019 4:00 pm

Is there a way to check the file size and have it trigger the email tool if it gets beyond the max file size?
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 12:41 pm

Is there a way to check the file size and have it trigger the email tool if it gets beyond the max file size?
You can try this:
if (([tool fetch url=<url> output=user as-value]->"total")>63) do={tool e-mail send ...}
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 2:13 pm

Thanks you for that.

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Nov 07, 2019 2:41 pm

I tried it endless to find that and this great. I knew the "total" part but did not thought op putting that in the variable.
if (([:tool fetch url=$url output=user as-value]->"total")<64) do={:local data ([:tool fetch url={$url output=user as-value]->"data")} else= {tool e-mail send ...}
It did not work for me.
Last edited by msatter on Fri Nov 08, 2019 10:52 am, edited 1 time in total.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 9:30 am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 10:49 am

Do you have a dedicated link the fullbogons piece? I cannot seem to fined a direct url for it?
Fullbogons_IPv4: http://www.team-cymru.org/Services/Bogo ... s-ipv4.txt
All bogon lists: https://www.team-cymru.com/bogon-reference-http.html
Bogons via BGP: https://www.team-cymru.com/bogon-reference-bgp.html
Many thanks.
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 2:10 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
Last edited by msatter on Fri Nov 08, 2019 4:38 pm, edited 1 time in total.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
RackKing
Member
Member
Posts: 339
Joined: Wed Oct 09, 2013 1:59 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 4:21 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
I gave this a shot - but it did not run. No message in the log and no address list.
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Nov 08, 2019 4:37 pm

Do not insert lists that are bigger than 63KiB, those would only will be loaded incomplete.
# Written by Shumkov
# Adapted by blacklister
# 20191108

/ip firewall address-list
:local update do={
 :do {
 :local result [/tool fetch url=$url as-value output=user]; :if ($result->"downloaded" != "63") do={ :local data ($result->"data")
  :do { remove [find list=$blacklist] } on-error={}
   :while ([:len $data]!=0) do={
      :if (([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
      :do {add list=$blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) timeout=7d} on-error={}
      }
   :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
}

$update url=https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset blacklist="firehole-1" delimiter=("\n") 
$update url=https://raw.githubusercontent.com/ktsaou/blocklist-ipsets/master/firehol_level2.netset blacklist="firehole-2" delimiter=("\n") 
The first is loaded and the second is not because of the size being over 63KiB

I use separate blacklists and not one blacklist with different comments.
I gave this a shot - but it did not run. No message in the log and no address list.
Remove one of the "(" in the line beginning with
:if (([:pick
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
hci
Long time Member
Long time Member
Posts: 612
Joined: Fri May 28, 2004 5:10 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Feb 28, 2020 12:53 am

I imagine the 63k limit is due to a variable size limit in Mikrotik scripting? It would be nice to be able to download larger blacklists.
 
User avatar
sjafka
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Jan 03, 2018 5:45 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri Feb 28, 2020 12:43 pm

This is beautifull mate! Thanks for your work! If you have a site with paypal donation, i would like to get you a beer! :D
PS.: I used before the squid blacklist, but the guy, who created it died (RIP m8 and thank you for your work!) last year, but it had like 30k entries, this has "only around 1500", but i see a lot of /24 subnets, so this is a huge list too!
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 185
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun Mar 01, 2020 2:30 pm

PSA
Make sure you have whitelisted your private IPs if using https://raw.githubusercontent.com/fireh ... el1.netset
 
xenuc
just joined
Posts: 1
Joined: Mon Mar 02, 2020 8:28 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Mar 02, 2020 9:07 am

The script is great, thanks. Now we just wait another 10 years to bypass the 65k limit.
 
HZsolt
newbie
Posts: 31
Joined: Tue Apr 24, 2018 7:31 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Mar 02, 2020 9:46 pm

63 --> 8192 and I downloaded the larger blacklist. But all lines did not load properly to the address-list.

https://raw.githubusercontent.com/fireh ... el1.netset
https://raw.githubusercontent.com/fireh ... el2.netset
https://raw.githubusercontent.com/fireh ... el3.netset
https://raw.githubusercontent.com/fireh ... el4.netset

How can I merge to one address-list the above addess-lists? I would like to use one address-list in the firewall of MikroTik instead of four address-lists. Fewer line in the firewall, faster processing and fewer load.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 5:06 pm

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find $array $ip]-[:len $array]/2)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It's harder and slower :), but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an "array"?
Because the default "find" function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default "find" function.
Last edited by Shumkov on Mon Mar 16, 2020 4:10 pm, edited 3 times in total.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
HZsolt
newbie
Posts: 31
Joined: Tue Apr 24, 2018 7:31 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 7:57 pm

Variant 2:
ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
:local array [find dynamic list=blacklist]
:foreach value in=$array do={:set array (array,value,[get $value address])}
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}") do={
:local ip ([:pick $data 0 [:find $data $delimiter]].$cidr)
:do {add list=blacklist address=$ip comment=$description timeout=1d} on-error={
:do {set ($array->([:find key=$ip in=$array]-1)) timeout=1d} on-error={}
}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/feeds/c2-ipmasterlist-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
- the script does NOT delete actual addresses, but prolongs their timeout. Addresses that are not in the downloadable list are deleted by the system automatically after their timeout. It's harder and slower :), but it makes it possible to track the date/time of addresses added to the blacklist.
Why is the script using an "array"?
Because the default "find" function is VERY slow. Using an additional array allows to speed up the script several times, since operations are performed directly with the indexes, bypassing the default "find" function.
With the above script can I properly (full lists) download the below lists?
https://raw.githubusercontent.com/fireh ... el1.netset
https://raw.githubusercontent.com/fireh ... el2.netset
https://raw.githubusercontent.com/fireh ... el3.netset
https://raw.githubusercontent.com/fireh ... el4.netset
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 8:15 pm

With the above script can I properly (full lists) download the below lists?
Download full lists - you can’t. 63KiB is a limitation of RouterOS, here scripts are powerless.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
HZsolt
newbie
Posts: 31
Joined: Tue Apr 24, 2018 7:31 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 8:20 pm

With the above script can I properly (full lists) download the below lists?
Download full lists - you can’t. 63KiB is a limitation of RouterOS, here scripts are powerless.
What is 63 KiB limitation of RouterOS?
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Mar 03, 2020 9:50 pm

My version checks for list larger than 63KiB and logs then if the list is loaded or not.

There no way to import a list bigger than that through an array.

Bigger lists can be used but that is an other story.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
Krusty
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Fri May 02, 2008 11:14 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Wed Mar 11, 2020 11:23 am

LifeSaver, thank you guys you are awesome
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu Mar 12, 2020 2:05 pm

Bugfix:
- correct regexp is "^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}"

The bug is not critical, it’s just that in some cases the script could process strings containing not only IP addresses, but simply numerical combinations similar in format.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
User avatar
Xtreme512
Frequent Visitor
Frequent Visitor
Posts: 75
Joined: Sun Jun 08, 2014 2:43 pm
Location: Nicosia, CY
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Wed Mar 25, 2020 4:14 am

Nice, very nice working script, thank you!

64 KB limit, on the other hand, is so annoying though... Gotta find a workaround, like maybe splitting files on-the-fly?
I Walk Alone
 
frantacech
just joined
Posts: 2
Joined: Tue Jul 25, 2017 6:55 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Apr 18, 2020 11:14 pm

Hello!
how can i import it?

for example aggregated ip from china?
https://www.ipdeny.com/ipblocks/
https://www.ipdeny.com/ipblocks/data/ag ... gated.zone

I try, but it doesn't work
 
shed909
just joined
Posts: 2
Joined: Sat Apr 25, 2020 5:59 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Apr 25, 2020 6:02 am

ip firewall address-list
:local update do={
:do {
:local data ([:tool fetch url=$url output=user as-value]->"data")
remove [find list=blacklist comment=$description]
:while ([:len $data]!=0) do={
:if ([:pick $data 0 [:find $data "\n"]]~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:if ([:pick $data 0 [:find $data "\n"]]~"[a-z0-9]+([\\-\\.]{1}[a-z0-9]+)*\\.[a-z]{2,5}(:[0-9]{1,5})?(\\/.*)?") do={
:do {add list=blacklist address=([:pick $data 0 [:find $data $delimiter]].$cidr) comment=$description timeout=1d} on-error={}
}
:set data [:pick $data ([:find $data "\n"]+1) [:len $data]]
}
} on-error={:log warning "Address list <$description> update failed"}
}
$update url=http://feeds.dshield.org/block.txt description=DShield delimiter=("\t") cidr=/24
$update url=http://www.spamhaus.org/drop/drop.txt description="Spamhaus DROP" delimiter=("\_")
$update url=http://www.spamhaus.org/drop/edrop.txt description="Spamhaus EDROP" delimiter=("\_")
$update url=http://osint.bambenekconsulting.com/fee ... t-high.txt description="Bambenek High-Confidence C2" delimiter=("\2C")
$update url=https://sslbl.abuse.ch/blacklist/sslipblacklist.txt description="Abuse.ch SSLBL" delimiter=("\r")
$update url=http://malc0de.com/bl/IP_Blacklist.txt description="malc0de" delimiter=("\n")
$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")
$update url=https://raw.githubusercontent.com/fireh ... el1.netset description="FireHOL Level1" delimiter=("\n")
$update url=https://raw.githubusercontent.com/hecto ... g/list.txt description="hectorm adaway.org" delimiter=("\n")
Trying to add support for address lists containing the URL as apposed to IP, such as hectorm's lists for PiHole:
https://discourse.pi-hole.net/t/update- ... 2019/13620

However, some of the comments come out as the actual URL entry and the timeouts aren't set.
Last edited by shed909 on Sat Apr 25, 2020 7:13 am, edited 3 times in total.
 
shed909
just joined
Posts: 2
Joined: Sat Apr 25, 2020 5:59 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat Apr 25, 2020 6:04 am

Hello!
how can i import it?

for example aggregated ip from china?
https://www.ipdeny.com/ipblocks/
https://www.ipdeny.com/ipblocks/data/ag ... gated.zone

I try, but it doesn't work
Try:
$update url=https://www.ipdeny.com/ipblocks/data/ag ... gated.zone description="IPdeny cn-aggregated" delimiter=("\n")
 
pukka
just joined
Posts: 10
Joined: Sun Jun 26, 2011 4:05 pm

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 01, 2020 2:22 pm

How do we get around this 63KiB limit? can we ask mikrotik about this

We are trying to automate the download and add of

https://www.ipdeny.com/ipblocks/data/countries/gb.zone which is 124KiB
 
Lebzul
Member Candidate
Member Candidate
Posts: 100
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 16, 2020 11:38 pm

Don't forget to add
{:delay 20};
at the beginning of the script to give time if running after reboot is needed.
 
User avatar
jvanhambelgium
Member Candidate
Member Candidate
Posts: 230
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 17, 2020 8:55 am

How do we get around this 63KiB limit? can we ask mikrotik about this

We are trying to automate the download and add of

https://www.ipdeny.com/ipblocks/data/countries/gb.zone which is 124KiB
Perhaps the only way is to have some really smart script parse this list further into large(r) CIDR-blocks. So take several /24 "lines" and aggregate them further where there are adjacencies.
I've seen some sort of script here somewhere (used in another context) but it might be doable to gain a certain % of reduction.
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 17, 2020 11:20 am

The problem is thst first have to read whole list before you can start reducing.

If Miktotik implement resume download then we could chop up the file in little parts.
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
User avatar
mozerd
Member
Member
Posts: 379
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 2:02 pm

According to the following Manual:Scripting-examples -- file size limitation has been removed
Read and write large files

Many users requested ability to work with files. Now you can do it without limitations

Create and write to file:

:global newContent "new file content\r\nanother line\r\n";
[/lua "local f=assert(io.open('/test.txt', 'w+')); f:write(newContent); f:close()" ];
Read file content to variable:

:global cnt ""
[/lua "local f=assert(io.open('/test.txt', 'r')); cnt=f:read('*all'); f:close()" ];
:put $cnt
I just found this wiki entry but I have not tried to adapt to blacklists ..... if this code actually works that would be excellent.
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 3:34 pm

According to the following Manual:Scripting-examples -- file size limitation has been removed
Read and write large files

Many users requested ability to work with files. Now you can do it without limitations

Create and write to file:

:global newContent "new file content\r\nanother line\r\n";
[/lua "local f=assert(io.open('/test.txt', 'w+')); f:write(newContent); f:close()" ];
Read file content to variable:

:global cnt ""
[/lua "local f=assert(io.open('/test.txt', 'r')); cnt=f:read('*all'); f:close()" ];
:put $cnt
I just found this wiki entry but I have not tried to adapt to blacklists ..... if this code actually works that would be excellent.
Is /lua back then?
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
User avatar
jvanhambelgium
Member Candidate
Member Candidate
Posts: 230
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 3:39 pm

Don't think so. That Wiki page states : This page was last edited on 18 October 2017, at 10:37.
As it says on the page : After RouterOS v4.0beta4, Lua support is removed until further notice
 
User avatar
mozerd
Member
Member
Posts: 379
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon May 18, 2020 4:38 pm

Don't think so. That Wiki page states : This page was last edited on 18 October 2017, at 10:37.
As it says on the page : After RouterOS v4.0beta4, Lua support is removed until further notice
My sincere apologies -- I did not see the part that After RouterOS v4.0beta4, Lua support is removed until further notice

What a shame, all that Lua stuff should be removed IMO BUT if certainly would be nice if MikroTik brought back LUA support or provided another means to work with any file size.
 
Lebzul
Member Candidate
Member Candidate
Posts: 100
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 23, 2020 8:09 pm

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.

BTW, is there a way to have these working?
https://github.com/firehol/blocklist-ipsets/blob/master/firehol_level1.netset			40.8 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_wannacry.ipset	         6.15 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_bruteforce.ipset.     	4.64 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield_30d.netset					2.17 KB
https://github.com/firehol/blocklist-ipsets/blob/master/spamhaus_edrop.netset				1.98 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield_7d.netset					1.5 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_webscan.ipset	1.42 KB
https://github.com/firehol/blocklist-ipsets/blob/master/dshield.netset						1.04 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_wormscan.ipset	0.97 KB
https://github.com/firehol/blocklist-ipsets/blob/master/normshield_all_dnsscan.ipset	0.86 KB
Last edited by Lebzul on Sat May 23, 2020 10:25 pm, edited 2 times in total.
 
User avatar
jvanhambelgium
Member Candidate
Member Candidate
Posts: 230
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 23, 2020 8:38 pm

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.
Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So......


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly "prepare" the file before download to the device, but that cannot be the show stopper I guess.
 
Lebzul
Member Candidate
Member Candidate
Posts: 100
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 23, 2020 10:27 pm

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.
Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So......


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly "prepare" the file before download to the device, but that cannot be the show stopper I guess.
I see. And then, how do people do with servers with an open port? Let's say people need to access my server in a specific port?
 
User avatar
jvanhambelgium
Member Candidate
Member Candidate
Posts: 230
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 24, 2020 12:05 am

Is there a reasonable way of bypassing Mk's limit or another approach?
I'm a Mk hardcore user but I'm considering other vendors if they do not apply a better concept to protect our equipments.
Nothing wrong with the concept I think. The idea of deploying such huge massive IP-lists and filter against them is something not infinitely possible also with other vendors.
Eg. Palo Alto networks.

A maximum of 10 External Block Lists (PanOS 7.x) on a PA-200
A maximum of 50000 IPs in all external lists combined. (1 list with 50000 IPs or 10 Lists with 5000 IPs both are supported)
If you use more than 10 EBLs in a device you will see the following error during commit:
Exceeding max number of supported external block lists (10)

In terms of harware limit

Hardware Maximum Address Entries
PA-220 : 2500
PA-820 : 2500
PA-850 : 3500
PA-3020 : 5000
PA-5020 :10000
PA-5220 : 40000
PA-7050 : 80000

So......


The only option is multiple cascaded lists that each remain within the boundary of 65K processing.
But indeed, you need some intermediate processing thing to properly "prepare" the file before download to the device, but that cannot be the show stopper I guess.
I see. And then, how do people do with servers with an open port? Let's say people need to access my server in a specific port?
It is important to take all considerations into account when you make the design. If "people" are in fact scattered across the world coming from virtually anyplace then perhaps you need to provide this service at another level. Eg. use some form of authentication with your users (possibly combined with VPN-application).
What is sitting behind this specific port ? Is this something that understand the concept of user-authentication ?
If you run a business and you know your users (eg. employees) are located in country X then filter strict and only allow country X IP which will reduce the surface already A LOT.
 
Lebzul
Member Candidate
Member Candidate
Posts: 100
Joined: Wed Feb 21, 2018 12:54 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu May 28, 2020 6:08 am

Nice Work!

I added FireHOL Level2 to the script as well, in case you're interested. Just added this line:

$update url=https://raw.githubusercontent.com/ktsao ... el2.netset description="FireHOL Level2" delimiter=("\n")

-zeb
Lv1 was working fine and now it is not. Probably it does not fit anymore.
 
User avatar
mozerd
Member
Member
Posts: 379
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Thu May 28, 2020 1:15 pm

Lv1 was working fine and now it is not. Probably it does not fit anymore.
You [everyone] should be aware that:

level1 check frequency = 1 minute and average update frequency = 2 hours and 27 minutes
level2 check frequency = 1 minute and average update frequency = 17 minutes
level3 check frequency = 1 minute and average update frequency = 45 minutes
level4 check frequency = 1 minute and average update frequency = 44 minutes
webclient check frequency = 1 minute and average update frequency = 8 hours and 36 minutes
webserver check frequency = 1 minute and average update frequency = 23 hours and 16 minutes

So why is this important to note?
Because changes [adds/deletions] are frequent and that can have a dramatic change in file size.
Also of importance to note is that many duplicates reside when lists are combined - so your processing engine needs to remove duplicates and then reorders them for faster processing.

Depending on which MikroTik Router model being used MOAB combines some of these lists or ALL of these lists 3 times each day spaced 8 hours apart.
 
kevinds
Member Candidate
Member Candidate
Posts: 114
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 29, 2020 3:18 pm

What is the recommended way to find out *why* an update failed?

Address list <Spamhaus DROP> update failed

Is great to see in the logs, but where do I look to try and figure out why it failed?

Spamhaus DROP and EDROP are not over 63 kb, so that isn't the reason..

At the moment I am focusing on the IPs used for email SPAM, but it doesn't really matter.. I have the Spamhaus and Bambenek lists failing but I don't know why.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 29, 2020 9:52 pm

What is the recommended way to find out *why* an update failed?

Address list <Spamhaus DROP> update failed

Is great to see in the logs, but where do I look to try and figure out why it failed?
This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
kevinds
Member Candidate
Member Candidate
Posts: 114
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Fri May 29, 2020 10:42 pm


This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?
They don't load always.
 
User avatar
mozerd
Member
Member
Posts: 379
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 3:57 pm


This error occurs if the file is for some reason not available for download. The address list does not load SOMETIMES? Or always?
They don't load always.
@kevinds
You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
So it is critical that duplicate IP be avoided via a pre-process that first checks for duplicates, removes the duplicates, reorders [sorts] the list for faster processing then proceeds with the load.
 
User avatar
Shumkov
just joined
Topic Author
Posts: 11
Joined: Tue Oct 01, 2019 9:08 pm
Location: Russian Federation

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 7:25 pm

You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
Script ignores duplicates via on-error={}, processing is not interrupted.
RB951G-2HnD / RouterOS 6.45.9 (Long-term)
 
User avatar
mozerd
Member
Member
Posts: 379
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 8:37 pm

You should be aware that when loading lists IF a duplicate IP is present the list will not load and processing stops.
Script ignores duplicates via on-error={}, processing is not interrupted.
Do you mean this line: on-error={:log warning "Address list <$description> update failed"} ?

Where in your script do you check for duplicate ip?
 
kevinds
Member Candidate
Member Candidate
Posts: 114
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sat May 30, 2020 8:44 pm


Do you mean this line: on-error={:log warning "Address list <$description> update failed"} ?
comment=$description timeout=1d} on-error={}
 
User avatar
mozerd
Member
Member
Posts: 379
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 31, 2020 3:40 pm

comment=$description timeout=1d} on-error={}
Thanks .... I just tested @Shumkov code and it works very nicely .... excellent work.
 
kevinds
Member Candidate
Member Candidate
Posts: 114
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Sun May 31, 2020 4:38 pm

Thanks .... I just tested @Shumkov code and it works very nicely .... excellent work.
Yeah, I have a couple honeypot IPs that when hit, adds the IP to a drop rule, then a script that runs that expands the /32 to a larger block.. I needed something similar to handle multiple IPs from the same larger block.. For when asshats decide to use an entire /16 to do a port-scan of every port.. lol

That was interesting to watch.. haha

But yeah, I still have lots to learn, but I'm not sure how to get a better log for why both variations are failing.
 
msatter
Forum Guru
Forum Guru
Posts: 1645
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Mon Jun 01, 2020 11:55 pm

My version:
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
Collecting ranges of IP addresses that are knocking at the door: viewtopic.php?f=2&t=152953&p=758068&hil ... os#p758068
One RB4011 (cooled) and a RB760iGS (hEX S) in series. 4011 Does PPPoE/IKEv2.
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.12
NordVPN viewtopic.php?f=2&t=158439&p=781009 for multiple connections.
 
kevinds
Member Candidate
Member Candidate
Posts: 114
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Jun 02, 2020 12:05 am

My version:
   } ;  :log warning "Imported address list < $blacklist> from file: $url"
   } else={:log warning "Address list: <$blacklist>, downloaded file to big: $url" }
 } on-error={:log warning "Address list <$blacklist> update failed"}
Collecting ranges of IP addresses that are knocking at the door: viewtopic.php?f=2&t=152953&p=758068&hil ... os#p758068
Ok.. Reading this...

It downloads the list.

It tries to import it

If successful, gives a successful message,
If import fails it says too big..

So if import fails for any reason, it says too big, what if it fails for another reason?

I don't see your version checking it's size beforehand, so the error message could say 'Failed because a butterfly flapped it's wings..' and would still be a more useful error message (because it wouldn't be stating an incorrect reason for it to fail). ;)

I hope I am wrong reading this.. If I am, I am very sorry.
 
kevinds
Member Candidate
Member Candidate
Posts: 114
Joined: Wed Jan 14, 2015 8:41 am

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Jun 02, 2020 12:14 am

Collecting ranges of IP addresses that are knocking at the door: viewtopic.php?f=2&t=152953&p=758068&hil ... os#p758068
I do something very similar to the linked thread..

I have honey-pot IP addresses, anything that attempts to connect to them, gets their IP added to the block list, these addresses have never been used, so nothing legitimate would have any reason to try and connect.

Then another script runs and turns them into a /24, with a 7 day timeout..

Usually, the router has 60-75k addresses in the list at any time. After a reboot the list is reset, takes 6-12 to get back up there.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 1641
Joined: Sat Dec 24, 2016 11:17 am
Location: jo.overland at gmail.com

Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

Tue Jun 02, 2020 8:28 am

I have honey-pot IP addresses, anything that attempts to connect to them, gets their IP added to the block list, these addresses have never been used, so nothing legitimate would have any reason to try and connect.
I do nearly the same. Since I do not have an extra public IP, I have and access rule that if any tries to connect to a port that is not open, they get blocked to all ports (65535-6 ports) , also normally open port (6 ports) for 24 hour.
 
How to use Splunk to monitor your MikroTik Router(s)

MikroTik->Splunk
 

Who is online

Users browsing this forum: johnmes and 16 guests