Community discussions

 
flazzarini
just joined
Topic Author
Posts: 18
Joined: Thu Jun 13, 2013 11:05 am

Blocklister WebAPI to get precompiled IP Addresslists

Wed Jul 15, 2015 11:31 am

Hi there,

https://blocklister.gefoo.org

Blocklister is a simple API for generating MikroTik RouterOS compatible addresslists. The ipaddresses included in these lists are regularly updated using source sites such as iblocklist.com, dshield.org etc. Importing the resulting scripts into your routers is done by using a simple script which can later be scheduled to run every x days for example. There is no need to clean you addresslists manually as the resulting scripts includes a cleanup based on the comment of the addresslists, just be careful that you don't change the comments on the go. Name and comment of the resulting addresslists scripts can be customized using query parameters.

This solution has been heavily inspired by http://joshaven.com/resources/tricks/mi ... ress-list/.

Please note that each source ip address is limited to 10 requests per day!


Usage Example

Following a simplified example output. Each response's content-type is in plain/text. In this example I customize the addresslist name to "foo" and the comment to "bar".
$ curl https://blocklister.gefoo.org/ads?listname=foo&comment=bar
Result (please note that the result has been striped of here and the ips have been replaced by bogus ips)
:foreach i in=[/ip firewall address-list find ] do={
  :if ( [/ip firewall address-list get $i comment] = "bar") do={
      /ip firewall address-list remove $i
  }
}

# Now let's add the new ones
/ip firewall address-list
add address="1.1.1.1-1.1.1.255" list="foo" comment="bar"
add address="2.2.2.1-2.2.2.10" list="foo" comment="bar"
As you can see the resulting script also takes care of cleaning your addresslists first by removing everything in your addresslist which had the comment "bar" in it.

RouterOS Script Example

To automatically download and import the addresslists in your router you could use a script similar to this one. You will find the same script in the documentation link of the API https://blocklister.gefoo.org .
/system script add name="Blocklister_download_Ads" source="/tool fetch url=\"https://blocklister.gefoo.org/ads?listname=foo&comment=bar\" dst-path=\"ads.rsc\"; /import file-name=\"ads.rsc\";"

API Links
Query parameters

With these query parameters you can customized the name and comment which will be used to create the address-list on your router.
  • listname - The name of the resulting addresslist
  • comment - Use this query parameter to adjust the comment which will be used for this addresslist (Please make sure that this comment is unique among your addresslists)

WARNING

Please use this with care as you could possibly delete all of your addresslists entries.
Be careful with lists such as Level1, Badpeers they are extremely big and could crash your router.

Suggestions

Any suggestions are welcome. If you have some other ip lists that you would like to have added please let me know.
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu May 21, 2015 11:14 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Wed Oct 14, 2015 4:30 pm

Hi,

would it be possible to split the level1 list? The other lists are working fine.
I tried to apply the list but the router crashes. It rans out of Memory.
I have an rb2011

Regards
 
kivimart
newbie
Posts: 40
Joined: Thu Oct 10, 2013 3:06 pm

Re: Blocklister WebAPI to get precompiled IP Addresslists

Wed Oct 14, 2015 9:17 pm

This is great thanks.
 
flazzarini
just joined
Topic Author
Posts: 18
Joined: Thu Jun 13, 2013 11:05 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Thu Oct 22, 2015 10:27 am

Hi,

would it be possible to split the level1 list? The other lists are working fine.
I tried to apply the list but the router crashes. It rans out of Memory.
I have an rb2011

Regards
This could be done and idea that pops into mind is to introduce a pagination of some sort. So the idea would be that you would have to define the page_nr and page_size in your request. So in final you would say http://blocklister.domain.com/level1?pa ... ge_size=10 this would return you the 10 first firewall entries of the level1 list.

What do you think of this idea?
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu May 21, 2015 11:14 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Thu Oct 22, 2015 8:10 pm

Hi,

this would help me and is an fine solution.
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu May 21, 2015 11:14 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Sat Oct 24, 2015 11:43 am

I dont know its a problem of your script but when i update badpeers i get the error in the log
script error: expected end of command (line 47713 column68)
And this error i got one time:
sricpt error: missing values(s) of arguments list
 
flazzarini
just joined
Topic Author
Posts: 18
Joined: Thu Jun 13, 2013 11:05 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Fri Feb 19, 2016 10:50 pm

I will need to control this.

I want to point out a general update on this, I am quite ready to push this project to github now, so you could open up an issue for this.
 
flazzarini
just joined
Topic Author
Posts: 18
Joined: Thu Jun 13, 2013 11:05 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Mon Mar 28, 2016 7:07 pm

Hi guys. I am happy to announce that blocklister is available on https://github.com/flazzarini/blocklister, now. On top of that I've released and pushed a new version of blocklister version 1.3.0. No major changes only to the code structure and dependencies.

If you have any requests please feel free to post them over at https://github.com/flazzarini/blocklister.


If I find the time I will prepare as well a Virtual Machine having blocklister already installed, so you could deploy blocklister conveniently at home.
 
Zebble
newbie
Posts: 41
Joined: Mon Oct 17, 2011 4:07 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Mon Apr 04, 2016 12:21 am

Based on experience, you may want to use blackhole routes instead of an Address List. For larger lists, blackhole routes take up less CPU and RAM so they tend to perform much better and on lesser hardware. The only thing you lose is the ability to count the number of "hits" to the blacklists through firewall rules, as blackhole routes are processed far earlier in the packet flow than firewall rules (part of the reason they're so efficient).

Example blackhole route:

/ip route add dst-address=x.x.x.x/x type=blackhole comment="blacklist"

Example to clear out the blacklist before importing a new one:

/ip route remove [find comment="blacklist"]


This could also all be done with BGP routes, which would allow you to dynamically add/remove individual entries without having to clear out and re-import the entire list every time.

Just my $0.02!

-wade
 
flazzarini
just joined
Topic Author
Posts: 18
Joined: Thu Jun 13, 2013 11:05 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Mon Apr 04, 2016 9:25 am

Based on experience, you may want to use blackhole routes instead of an Address List. For larger lists, blackhole routes take up less CPU and RAM so they tend to perform much better and on lesser hardware. The only thing you lose is the ability to count the number of "hits" to the blacklists through firewall rules, as blackhole routes are processed far earlier in the packet flow than firewall rules (part of the reason they're so efficient).

Thanks @Zebble for pointing that out. We could introduce a new feature to blocklister where you would have the choice between addresslist and blackholing. Not sure yet the route on the api would look like but it could be as simple as.

http://domain.com/ads?render_as=blackhole


Let me know what you think. Might also be worth it to open up an issue over at https://github.com/flazzarini/blocklister/issues
 
flazzarini
just joined
Topic Author
Posts: 18
Joined: Thu Jun 13, 2013 11:05 am

Re: Blocklister WebAPI to get precompiled IP Addresslists

Wed Jan 04, 2017 12:33 pm

Hi there,

happy to announce that version 1.6.0 of blocklister is now available at https://blocklister.gefoo.org and the code has been published at https://github.com/flazzarini/blocklister. For the changelog please visit https://blocklister.gefoo.org/changelog, major new feature is to return ip ranges as CIDR notation, by passing a query parameter for instance https://blocklister.gefoo.org/ads?cidr=True

Hope this helps

Who is online

Users browsing this forum: No registered users and 5 guests