For years I have provided a dynamic blacklist as a free service for MikroTik users who would like to subscribe to well maintained blacklists. Due to its popularity (over 6000 daily requests) I have recently upgraded my server environment to increase availability & reduce latency for users anywhere around the world.

My lists are 100% free and 100% transparent in that I fully disclose all info and creation sources.

http://joshaven.com/resources/tricks/mi ... ress-list/

If you have any questions, concerns, or other comments please feel free to reach out to me.

You should contact another user here - IntrusDave who is also doing a similar thing. Perhaps you can pool your resources and have an even larger detection surface.

For the single-address blacklist openbl, you could consider setting up a DNS name like openbl.joshaven.com
with the A records for that list. Then a user of 6.36 or higher can just create an address list item with
that domain name to receive the uptodate version of the list without any scripting and scheduling.
Unfortunately this is not (yet) possible for address lists that hold subnets.

For the single-address blacklist openbl, you could consider setting up a DNS name like openbl.joshaven.com
with the A records for that list. Then a user of 6.36 or higher can just create an address list item with
that domain name to receive the uptodate version of the list without any scripting and scheduling.
Unfortunately this is not (yet) possible for address lists that hold subnets.

That isn't really good practice because DNS isn't set up to send hundreds/thousands of records in a single reply.

I think some servers/clients can do it, but it's not a guarantee. The reason there are only 13 root server addresses is that 13 hosts was the most that could be fit into a single query response with the size limitation in place.

I have it in use with >1850 addresses on a single DNS name and it works OK.
This test is done with CHR version of RouterOS and Bind9 as server.
The lookup is first done with UDP, reply is that the result is too large for UDP, then it is re-tried in TCP
by RouterOS and the result is returned and put in the address list.

The only problem I am facing is not related to DNS and applies to lists loaded via the service mentioned
in this topic just as well: when the IP->Firewall->Address list page is visited in WebFig with the filter set
to all or to this list, the browser has a very hard time to format the page and spends more CPU time than
the interval between updates. This makes the UI almost freeze. But the same thing happens in other
pages, like the NAT Connections table. I have suggested an improvement to set max number of displayed
rows as a system parameter and end the table with a "not all rows displayed" warning. Again, this is
not related to the method the list is populated, it is just a problem with the WebFig UI.

The 13 address limit applies only to UDP replies and it is not really valid anymore due to extended UDP
reply size, DNSSEC, etc.
It would be nice when RouterOS did TXT queries in addition to A queries for address lists tied to DNS,
then accept TXT records of the form \d+.\d+.\d+.\d+/\d+ (like 12.34.56.78/12) to add subnets to an address
list.

For the single-address blacklist openbl, you could consider setting up a DNS name like openbl.joshaven.com
with the A records for that list. Then a user of 6.36 or higher can just create an address list item with
that domain name to receive the uptodate version of the list without any scripting and scheduling.
Unfortunately this is not (yet) possible for address lists that hold subnets.

This is a very interesting idea... Thanks!!!
At the moment I don't have the flexibility to update the dns records easily enough but I am sure I can just use a SQL based DNS service to solve this issue.

You can write the zone to a textfile on the server after you have retrieved it and then send a reload command to Bind to activate this zone.
I do it by simply taking a fixed header, then catting the list of addresses via a sed command that transforms the addresses
in <tab>IN<tab>A<tab>ipaddress lines.

For the single-address blacklist openbl, you could consider setting up a DNS name like openbl.joshaven.com
with the A records for that list. Then a user of 6.36 or higher can just create an address list item with
that domain name to receive the uptodate version of the list without any scripting and scheduling.
Unfortunately this is not (yet) possible for address lists that hold subnets.

That isn't really good practice because DNS isn't set up to send hundreds/thousands of records in a single reply.

I think some servers/clients can do it, but it's not a guarantee. The reason there are only 13 root server addresses is that 13 hosts was the most that could be fit into a single query response with the size limitation in place.

The answer size would be a problem if a single request was supposed to return a huge list... I'm feeling an answer just around the corner though... Thinking along these lines may yet yield fruit. I'm gonna give good consideration to solutions using DNS and routing based principals.

Great stuff guys!

You can write the zone to a textfile on the server after you have retrieved it and then send a reload command to Bind to activate this zone.
I do it by simply taking a fixed header, then catting the list of addresses via a sed command that transforms the addresses
in <tab>IN<tab>A<tab>ipaddress lines.

I hear ya on that... one problem is that my users are all around the world and I don't want 300+ ms of latency on replies if I am going to offer a DNS based solution so the solution needs to scale well. I some pretty high peak request spikes as it is. (I've altered my list writeup to base the example schedule based on the page load time to try to spread the load out more randomly).

I have very seriously considered creating a subscription based service that allows you to selectively choose sources and output formats which could easily include DNS answers. In this case I would match the requesting party and allow unique answers based upon source IP, IP range, URL query string, etc...

I hear ya on that... one problem is that my users are all around the world and I don't want 300+ ms of latency on replies if I am going to offer a DNS based solution so the solution needs to scale well.

To clarify what DNS based address lists do on RouterOS: it does NOT send a DNS query for every packet or connection, it sends a DNS query to load the entire list (same as what /tool fetch would do, but in a more compact format), and puts all the entries in the local address list.
The request is re-run at the frequency specified by the TTL of the DNS name. So you can control that completely. You can set it at 1 day, 1 hour, whatever you like and what your servers can handle, and the users will follow the interval at every query. (so you could even vary it dynamically based on the loading of your link)

I've looked into it for my service and it's just not a path that I want to go down. DNS can not deliver subnets, and breaking out a single class A into separate IP's would make the overall bandwidth skyrocket.

I've looked into it for my service and it's just not a path that I want to go down. DNS can not deliver subnets, and breaking out a single class A into separate IP's would make the overall bandwidth skyrocket.

The interesting thing about the DNS solution is that the router can be instructed from a DNS query which I've not given much thought to previously. BGP community strings are just a 32-bit appendage to a route advertisement however this is utilized to halt ddos attacks all the time. Even if a DNS query isn't useful for blacklists it may still be useful for whitelists or publishing short term events to your other routers like ssh hacking detections. I've just not given it enough thought to be sure that I've considered all possibilities but it is an interesting new feature for sure.

You should contact another user here - IntrusDave who is also doing a similar thing. Perhaps you can pool your resources and have an even larger detection surface.

I think we are already pooling our resources without knowing it. Upon a spot-checking of his list, it looks like he is using Spamhaus and OpenBL but not dshield.

i'm using spamhaus, shield, openbl, malc0de, emerging threats and 8 honeypots in 8 different datacenter around the US. Also, the 200+ Mikrotik routers i manage report back what is hitting them.

i'm using spamhaus, shield, openbl, malc0de, emerging threats and 8 honeypots in 8 different datacenter around the US. Also, the 200+ Mikrotik routers i manage report back what is hitting them.

Are you interested in working together on this? I've been itching to expand my sources for a while now and just haven't gotten around to aggregating the data I from various routers and servers under my administration or trusted others. Even more so I've been kicking around the idea of creating a more dynamic system and it sounds like you are working on the same problem.

If you are interested please email me at josh@g2wireless.co as this conversation is obviously going beyond the scope of a user forum. I would have sent this via email but I am not seeing a good alternative way of contacting you.

CloudFlare has been working out very well and I am glad to have implemented it. My lists are now much closer to the end users due to the CDN. When testing from locations around the US and internationally it has been at least doubling the speed of accessing my lists and the caching has been very effective as I have seen a 66% reduction in server bandwidth. Attached is screenshot from CloudFlare that shows some of it's effectiveness. If anyone is considering a CDN then I recommend taking a look at CloudFlare!

Special thanks to CloudFlare for being so generous as they are the reason for the performance improvements and they are providing it as a free service.