Blacklist Filter (Development Topic)

RouterOS Scripting
1

Hey guys, I wanted to have this a little more open of a discussion, so I made a new thread.

So I’m starting to plan out the new system and I’m going SQL based. The old system used a boat load of regex, awk, grep, etc. It was pretty dirty, but it worked. The lists generated were stored in a flat file. The new system is going to be way more flexible.

My thoughts are this…

  • SQL based realtime list generation
  • Subscriber managed private black/white lists (configured per device)
  • Subscriber selectable list size (for 32M, 64M, 256M, 512M, 1G+ device)
  • Subscriber selectable country blocking (for devices that have enough memory)
    Moving to SQL will give this functionality, it will also allow the server to update the blacklists in realtime without blocking downloads. I haven’t yet found a way to do non-blocking updates on the client side. (Sorry, no BGP - too complicated to manage, this needs to be fully automated)

So, this is all still only on paper, so if anyone has more ideas, lets hear it.

Here is a form to fill out if you are interested in being notified:
https://goo.gl/forms/UQMYqKJ54E0iV35l2

2

I was charmed by your previous implementation of using DNS to determine which version of the list and partial update (add’s) should be provided.

3

How would this translate into update frequency for the clients?

4

I do plan on continuing to use DNS for versioning. Ultimate goal will be to have the client send the last update date and time, then request the just the changes from that point.

The effect on the client side would be that the client determines it’s own update schedule.

5

I am looking forward to it and will definitely be a paying customer!!!

6

So, I JUST started coding this week, so this is really rough.. like pre-alpha. This is going to take some time.. I can code in python, php, perl, and C. but just because I can, doesn’t mean I enjoy it. I really hate coding, it’s boring and frustrating. Anyway..

I have the database being populated IPs from all the big blacklist sources, I haven’t written the module for pulling in the honeypot data. I’m holding off on that because I also want to either rewrite the honeypot code, or move to an open source honeypot with an API.
I’m also pulling in some whitelists. The .rsc will import dynamic address-lists named “IntrusBL” and “IntrusWL”. I simply added two RAW rules, 1 to accept the WL one to drop the BL.
I also haven’t built any of the accounting, or config system.

If you would like to use the pre-alpha, this script will pull it for you. just change the “priority” to 1, 2, or 3. 1 being the smallest list (about 2k), 2 being middle (18k), and 3 being the whole thing (over 135,000)
I don’t recommend the priority 3 list unless you are running servers open to the world with a router that has at least 1GB RAM.
The script doesn’t collect anything from your end. yet. As it nears beta, the accounting system will be in place that will require at minimum, the software ID, ether1 MAC address, and maybe CPU type. I will need these to positively identify the router so that the server can generate the router’s customized list. I will also be including an opt-in option to provide some “router demographics” so I can generate stats on models/ram/etc.

So, here it is. I make NO PROMISES that it works all the time. my personal router is updating itself every hour, and my development network router is updating every time I make a commit to the source code.

please note that enabling Cloud DDNS is required, no exceptions.

:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/secureFetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;
7

Dave,

Thank you for your job.

Testing:

[admin@RBTEST] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="> https://bl.mikrotikfilters.com/fetch.php?priority=1> ";
status: failed

failure: closing connection: <500 Internal Server Error> 35.236.78.203:443 (4)
[admin@RBTEST] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="> https://bl.mikrotikfilters.com/fetch.php?priority=1> ";
status: failed

failure: closing connection: <500 Internal Server Error> 35.236.78.203:443 (4)

8

fixed. having issues with mysql terminating

9

Fixed.

10

[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url=“https://bl
.mikrotikfilters.com/fetch.phppriority=3”;
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url=“https://bl
.mikrotikfilters.com/fetch.phppriority=3”;
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)

11

you are missing the “?” in the url.

12

i can’t put “?” in terminal
ctrv^v reset “?” in url-address

13

Use ?

The script in post #9 was wrong.

14

Thanks Sid

15

Just incorporated the script to one router, 12h interval. For now, it seems to works well. We’ll see.

16

The script is not wrong, it’s intended to be a script, NOT command line.

17

I loved your service and used it, im def. willing to pay once you are live. Do you have anywhere we can sign up for an email alert or some info once the paid service is done/live? (subbing to this thread or the other main/closed thread will produce alot of “noise”) tks

18

Hi, since I’m interested about the blacklist service and in order to evaluate whether it’s useful to me, I’d like to know, what exactly is blacklisted?
Who/what created such list of IPs? Thanks in advance.

19

I hear that. Thank you Dave! I will be trying this out tonight to give you some more stress-testing data. Cheers!

20

the dev of this script/list uses both publicly available lists of “bad” ips (spamhaus , malcode ect), as well as his own “honeypot” devices which look for public IPs that are doing suspicious activities (then adds those IPs to his own “private” list, for distribution to ppl running his script , before he closed the service). Def was a great service + script, and one i plan on paying for once he re-launches.

I think in the main forum thread (ie not this new paid/development thread), the dev lists some of these sources he uses.