Community discussions

 
User avatar
43north
Member Candidate
Member Candidate
Posts: 197
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:49 am

                                                                                                                                                                                                                                                            
 










































  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.36 (c) 1999-2016       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
  
[mike@Knittel Home CCR] > :local model    [/system resource get board-name]
[mike@Knittel Home CCR] > :local version  [/system resource get version]
[mike@Knittel Home CCR] > :local memory   [/system resource get total-memory]
[mike@Knittel Home CCR] > :local uname    [/system identity get name]
[mike@Knittel Home CCR] > :local scriptVer 2016.7.4a
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Downloading current Blacklist for this model";
[mike@Knittel Home CCR] > /tool fetch mode=https dst-path="/dynamic.rsc" \
\...    url="http://mikrotikfilters.com/download.php                      

<url> -- 
  status: failed

failure: closing connection: <301 Moved Permanently> 172.102.241.58:80 (4)
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
[mike@Knittel Home CCR] > ynamic.rsc
bad command name ynamic.rsc (line 1 column 1)
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Removing temp file...";
[mike@Knittel Home CCR] > /file remove dynamic.rsc
no such item
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Blacklist Update Complete.";
[mike@Knittel Home CCR] > /system logging enable 0
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:53 am

I can tell right off, that's not a current script.
you need to use the current (in the first post), and put it into a script, not the console. 
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 197
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 8:57 am

LOL okay went to the very first post you started this thread and copied and pasted it all.... Still get an error.... EDIT: I also tried this in the script file, no dice

 MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.36 (c) 1999-2016       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
  
[mike@Knittel Home CCR] > # Import Intrus Managed Filter Lists
[mike@Knittel Home CCR] > # (C)2016 David Joyce, Intrus Technologies
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :local model    [/system resource get board-name]
[mike@Knittel Home CCR] > :local version   [/system resource get version]
[mike@Knittel Home CCR] > :local memory   [/system resource get total-memory]
[mike@Knittel Home CCR] > :local uname   [/system identity get name]
[mike@Knittel Home CCR] > :local scriptVer   2016.7.4a
[mike@Knittel Home CCR] > 
[mike@Knittel Home CCR] > :log warning "Downloading current Blacklist for this model";
[mike@Knittel Home CCR] > /tool fetch mode=https dst-path="/dynamic.rsc" \
\...    url="https://mikrotikfilters.com/download.php                     
  status: failed

failure: closing connection: <404 Not Found> 172.102.241.58:443 (4)
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
-- [Q quit|D dump|C-z pause]
[mike@Knittel Home CCR] > e]
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:01 am

It won't run from the terminal at all. It needs to be a script.

what is your output from this?
/system script run updateBlacklist
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 197
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:04 am

Bah! Been using your stuff for over a year and love it. Problems lately now.... 
  MMM      MMM       KKK                          TTTTTTTTTTT      KKK
  MMMM    MMMM       KKK                          TTTTTTTTTTT      KKK
  MMM MMMM MMM  III  KKK  KKK  RRRRRR     OOOOOO      TTT     III  KKK  KKK
  MMM  MM  MMM  III  KKKKK     RRR  RRR  OOO  OOO     TTT     III  KKKKK
  MMM      MMM  III  KKK KKK   RRRRRR    OOO  OOO     TTT     III  KKK KKK
  MMM      MMM  III  KKK  KKK  RRR  RRR   OOOOOO      TTT     III  KKK  KKK

  MikroTik RouterOS 6.36 (c) 1999-2016       http://www.mikrotik.com/

[?]             Gives the list of available commands
command [?]     Gives help on the command and list of arguments

[Tab]           Completes the command/word. If the input is ambiguous,
                a second [Tab] gives possible options

/               Move up to base level
..              Move up one level
/command        Use command at the base level
  
[mike@Knittel Home CCR] > /system script run updateBlacklist
  status: failed

failure: closing connection: <400 Bad Request> 172.102.241.58:443 (4)
[mike@Knittel Home CCR] > 
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:15 am

400 errors mean that the request was formatted wrong. Something is wrong with the copy/paste.

If you want, we can do a remote support session and I can take a look.  I use TeamViewer with my clients...

https://898.tv/intrus
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 197
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:21 am

Would love to but have to get to bed. Why is there no private messaging anymore? Or is it just my account?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:22 am

Not sure. I looked for that too...  Messaging and rep are both gone now.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 26, 2016 9:31 am

Try this - Delete the current script, then run these commands one at a time. (do not paste them all at once)
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https
/import updateBlacklist.rsc
/file remove updateBlacklist.rsc
/system script run updateBlacklist
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
efaden
Forum Guru
Forum Guru
Posts: 1711
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Wed Aug 10, 2016 7:55 pm

Nice. Done any benchmark comparing RAW to Filter?
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Wed Aug 10, 2016 8:45 pm

Not sure. I looked for that too...  Messaging and rep are both gone now.
Rep's still there - users can only give rep directly to someone on their profile nowadays.
PMs are definitely disabled though - I wonder why they did it.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
jarda
Forum Guru
Forum Guru
Posts: 7604
Joined: Mon Oct 22, 2012 4:46 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 12:18 am

Isn't it forum layout scheme dependent?
There is an option to try different layouts so it might start to work for you. At least it works for me, I guess. There are voting buttons at each post I see. But using tapatalk mostly, not the Web..
 
jarda
Forum Guru
Forum Guru
Posts: 7604
Joined: Mon Oct 22, 2012 4:46 pm

Re: RE: Re: Blacklist Filter update script

Thu Aug 11, 2016 12:26 am

Try this - Delete the current script, then run these commands one at a time. (do not paste them all at once)
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https
/import updateBlacklist.rsc
/file remove updateBlacklist.rsc
/system script run updateBlacklist
You can maybe start to distribute the blacklist via dns records to 6.36. An user can just put one domain name that would load whole list of ip addresses and keep it updating according the ttl. No need for scripts and files further.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:16 pm

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:19 pm

There are voting buttons at each post I see.
No, it's because you're an admin.
Admins get to vote up/down on individual posts, but standard users do not.
I guess they disabled it because some people got into downvote wars sometimes.
(my one and only downvote was from a time I jumped in on one such brawl, knowing full well that I was going to get a minus from the angry tantrum thrower)
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:43 pm

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
I do, but the vast majority of routers pulling the list are still running 6.35 and lower. 6.32.4 makes up about 85% of the total. Once the majority are running a RouterOS that supports the RAW table, then I will move to that.

As it is now, you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Thu Aug 11, 2016 9:55 pm

you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
And here I was expecting your server and installer to just take the appropriate action.
Quit slacking, man! ;)
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
kivimart
newbie
Posts: 40
Joined: Thu Oct 10, 2013 3:06 pm

Re: RE: Re: RE: Re: Blacklist Filter update script

Thu Aug 11, 2016 10:29 pm

You can maybe start to distribute the blacklist via dns records to 6.36. An user can just put one domain name that would load whole list of ip addresses and keep it updating according the ttl. No need for scripts and files further.
One vote for dns distribution



Skickat från min Nexus 6P via Tapatalk
 
Zorro
Long time Member
Long time Member
Posts: 676
Joined: Wed Apr 16, 2014 2:43 pm

Re: Blacklist Filter update script

Fri Aug 12, 2016 1:16 am

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
I do, but the vast majority of routers pulling the list are still running 6.35 and lower. 6.32.4 makes up about 85% of the total. Once the majority are running a RouterOS that supports the RAW table, then I will move to that.

As it is now, you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
once "bugfix" was moved to 6.36 branch, which eventually happen later - it will, perhaps.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Aug 12, 2016 8:12 am

you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.
And here I was expecting your server and installer to just take the appropriate action.
Quit slacking, man! ;)
It only puts in the script to load the address list, the user needs to make their own rules. There is just no realistic way to automate the rules, as every setup is different.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: RE: Re: Blacklist Filter update script

Fri Aug 12, 2016 8:17 am

You can maybe start to distribute the blacklist via dns records to 6.36. An user can just put one domain name that would load whole list of ip addresses and keep it updating according the ttl. No need for scripts and files further.
I can't think of a way to do that. A few issues - Server side, it needs to know what version, CPU and how much memory the router has. Their are several times a month that the list can ballon up to 8~10k addresses, so the server needs to server a smaller list to the low memory and low cpu routers. Also, while you can resolve an address, you can't push 3~5k IP's and subnets through one query. It would work if RouterOS had a DNSBL function.

You have other thoughts on how you would do it?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
hknet
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Sun Jul 17, 2016 6:05 pm
Location: Vienna, Austria
Contact:

Re: Blacklist Filter update script

Fri Aug 12, 2016 1:18 pm

This initiative by IntrusDave makes for an interesting read!

I'd ask IntrusDave to consider delivering this blacklist in another format if possible, this would allow different use-cases, especially multiple 10G+ uplinks make it hard to handle stuff using firewall policies and blackhole-routes would make things much easier performance-wise imho.

In case you'd take contributions for the blacklist I'd be willing to host a honeypot as datasource.

Regards
hk
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Fri Aug 12, 2016 6:43 pm

There is just no realistic way to automate the rules, as every setup is different.
Yeah, very true.

If I were to vote for another delivery method of this list, I would choose BGP feed, which could easily be used as a means to blackhole route the offending addresses.
You could even get super fancy with it by using communities in your feed if you wanted - communities that specify what activites an IP was banned for, or how threatening an address is considered, etc. If Mikrotik adds a routing filter action of "add to address list" then BGP would be quite an awesome means to keep the list updated in real-time w/o the need for fetching/parsing lists over http.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
hknet
Frequent Visitor
Frequent Visitor
Posts: 88
Joined: Sun Jul 17, 2016 6:05 pm
Location: Vienna, Austria
Contact:

Re: Blacklist Filter update script

Fri Aug 12, 2016 9:37 pm

Hi
currently testing your script on four small RBs.
2 x RB750UP - installed and works fine.
1 x hEX PoE lite - installed and works fine.
another hEX PoE lite yet fails:
/sys scr run updateBlacklist
status: failed

failure: closing connection: <400 Bad Request> 172.102.241.58:443 (4)

Therefore I'd ask for a bit more error-reporting ;)

Regards,
hk
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Aug 13, 2016 2:52 am

the 400 bad request means that something is wrong with your copy/paste of the script.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Aug 13, 2016 9:47 am

If I were to vote for another delivery method of this list, I would choose BGP feed, which could easily be used as a means to blackhole route the offending addresses.
You could even get super fancy with it by using communities in your feed if you wanted - communities that specify what activites an IP was banned for, or how threatening an address is considered, etc. If Mikrotik adds a routing filter action of "add to address list" then BGP would be quite an awesome means to keep the list updated in real-time w/o the need for fetching/parsing lists over http.

To be honest, BGP is my big weakness. I haven't needed, so I never learned about it. I'm not even sure where I should start.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 5:11 pm

BGP feeds for address lists are a bit different than routing with BGP because they're just using BGP as a vector to transmit the list, since its behavior is very well suited to the task - send the current list in full whenever a connection forms, and then send only deltas thereafter. It's quite efficient for this.

The way you could use this as a filter list right now would be to set all routes in the list as type=blackhole (via an in-filter on the client router) and enabling strict RPF on the client as well.
This will block traffic from going TO a blacklisted destination because it null-routes the destination.
The RPF causes the blacklisted addresses to get blocked because real packets won't arrive via the black hole interface. Since the reverse path doesn't match the routing table, RPF will discard packets from the blacklisted sources.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 7:58 pm

can you give me a sample export of what I would need to get started?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 8:55 pm

...just brainstorming here...

One way to do it is by using ExaBGP

When you create the rules for the address-list with your (I presume) daily script, you can also push the prefixes to ExaBGP with simple scripting (Python, PHP, bash, etc).
The only difference is that you will probably need to keep track of what you have sent to ExaBGP so in the next update of the list, you only send the differences (new advertisements, and withdrawals). This way you keep the bgp traffic to minimum, and the remote routers will update their blacklists far more efficient (as ZeroByte pointed out, only on first connection it will fetch the entire prefix list, and then it will keep the received prefixes updated by adding new prefixes, or withdrawing removed prefixes from the routing table).

Then from there you can either have everyone connect to ExaBGP (BGP Session) and send them the prefixes or peer ExaBGP with a BGP router of your choice and everyone establish their BGP sessions with that.

RouterOS is not ideal (I think) for this task since you would have to configure separate BGP peers for each and everyone that wants to use this service.
AFAIK routeros doesn't allow for non-configured peers to connect to BGP.
I haven't researched it recently, but if I recall correctly, quagga can allow incoming bgp sessions without being explicitly configured.
Probably ExaBGP does this as well.
Or if you prefer routeros, you could make a simple web page where an interested party can register their routers and using the routeros API, automatically create/update/delete the BGP peers.
I am not sure what's the peer limit on routeros (if any).

Also I would suggest running two BGP instances (ideally in separate datacenters/locations) so that if you need to restart the instance, or the BGP sessions terminate due to a network problem, the second BGP instance will keep everything running on all the remote peers without having their blacklists withdraw all of a sudden.

On the client's side, they would need some routing filters to handle all prefixes received by you adding them as blackholes to the routing table.
Just like with the firewall filters, it's pretty much the same with routing filters, in the sense that it's not easy to provide an automated way of configuring them since everyone has its own filters.

By the way Team cymru provides a similar service like this.

Also I just checked the blacklist and I noticed that you include many /16 prefixes in there. Eg: 138.200.0.0/16
That includes a large range belonging to Hetzner, a large server provider in Germany. Sure they might have some 'bad apples' in there, but they also run many many legit servers/services.
What are your criteria for adding a prefix in your list? This and many other /16s are bound to contain many legit stuff since they are kind huge ranges. Are you sure you should be blocking such large ranges?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 9:40 pm

My lists is regenerated every 4 hours. When /16's are added, It's almost always because the ISP has been notified of a BOTNET being run on their network, AND they have refused to look into it. They are also added the honeypots see attacks / spam from more than 50% of the IP's in that range. The networks are removed form the list as soon as the ISP responds to the issue, or the honeypots see that the issue has been resolved.

Thanks for the BGP info. I will look into it. I certainly don't want to setup BGP peers for every site that wants to use my list. As it is now, I have about 2700 routers that pull the list every 24 hours. (and 5 that insist on pulling it every 60 seconds)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 9:48 pm

Thanks for the info.

Actually I was wrong about 138.200.0.0/16 belonging to Hetzner. :oops:
138.201.0.0/16 belongs to Hetzner.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 9:56 pm

138.200.0.0/16 is currently listed because of high-volume bulk email (spam)
Just to my active SMTP server and my honeypots, they have attempted 3.2M emails from 14,000 IP's in the subnet over the last 7 days.

Stammhaus also has that subnet listed in their DNSBL and BGP.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 10:27 pm

BGP peering would at least let people get real-time updates w/o having to download the list every 60 seconds.

I think that using the API for a web-based sign-up might be the best idea if you want to use ROS as the BGP source.
I would recommend that the master set of addresses be kept separate from the hosts that subscribers actually peer with.
Furthermore, I'm not sure how this translates into RouterOS, but in Cisco, grouping peers into peer-groups has a marked improvement on the performance because the BGP process makes announce/withdraw decisions once for the group and then sends them out. Each un-grouped peer must be computed separately - causing much more CPU load for hosts w/ large number of peers.
(I am not certain if ROS even has a similar construct to peer groups - never deployed BGP on a production Mikrotik router)

If I were setting this up for BGP distribution, I would probably do the following:
Private ASN on my side
Bogus next-hop IP (e.g. 127.0.0.2 or 169.254.0.1 or something like that), so that if the subscriber forgets to blackhole the route properly, it won't try to actually route the address to a real next-hop.
EBGP + multihop = 256
in-filter=drop-all filter

out-filter = this is where interesting things could happen. If you wanted to allow your subscribers have the ability to specify certain filter types (akin to the list size limits you do for smaller client routers) then you could make similar filters and let the customer choose which one to apply to their session. Of course, a customer with a small router could still opt to take the full feed and just filter it themselves by whatever criteria they like.

Here would be a great getting-started configuration in ROS:
/routing bgp instance
set default AS=64567 router=id=64.5.6.7 redistribute-static=yes out-filter=bgp-static-filter
/routing filter
add action=accept bgp-origin=igp chain=bgp-static-filter protocol=static routing-mark=blacklist set-out-nexthop=127.0.0.2
add action=discard chain=bgp-static-filter
add action=discard chain=discard-all

Then for each route that you want to publish in the blacklist, add a static route:
/ip route add dst-address=169.254.1.2/32 routing-mark=blacklist type=blackhole
(The routing-mark is just a way to make certain that only these routes should be exported to BGP peers - the system-wide BGP-out filter.)

Each subscriber would be added as follows:
/routing bgp peer
add name=peerName remote-address=x.x.x.x remote-as=XXX in-filter=discard-all multihop=yes

You may want to play with the keepalive timers and/or specify an MD5 password for each peer. (agreed upon in the web-based signup)

The peer should just set next hop to blackhole / null / some null prefix in their network, depending on the type of router they're using.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Mon Aug 15, 2016 11:12 pm

ZeroByte's approach seems a lot more straightforward than mine!
Upon a little research, so far I see only the big vendors (cisco, juniper, etc) have implemented 'Dynamic Neighbors' support into BGP.
Quagga, BIRD, ExaBGP - as far as I can tell - they don't support it yet. So I guess the creation of BGP peers (via API, or some other way) seems the (only) way to go.
But that brings problems of its own. You may end up with tons of 'dead' peers over time of users that stopped using the service, so there should be a periodic check for long-dead peers to delete them.

I tried to test ZeroByte's approach but I noticed two issues.

First, the BGP instance must be configured for the 'blacklist' routing table otherwise it only redistributes static routes that are on the main routing table.
/routing bgp instance
set default AS=64567 router=id=64.5.6.7 redistribute-static=yes out-filter=bgp-static-filter routing-table=blacklist
Incidentally, trying this I noticed that the BGP Advertisements does not show anything even though it actually advertises stuff. Probably a bug?..
[admin@MikroTik] > /routing bgp advertisements print 
PEER     PREFIX               NEXTHOP          AS-PATH      ORIGIN     LOCAL-PREF
Of course the BGP instance could run on a dedicated mikrotik installation (CHR?) on the main routing table without interfering with any real/backbone traffic.

Second, RouterOS' BGP does not seem to accept any prefixes with a non reachable/bogus next-hop (or smth similar, I am not sure yet). Or at least I couldn't find a way to do it.
If I don't set the out next-hop at all, then the prefix is added to the routing table as inactive with gateway the (multihop) IP of the bgp peer. So its pretty much useless at this point.
But, simply setting an incoming filter on the 'client' side to change the type of the incoming prefixes to blackhole, the prefixes are installed as active-blackhole to the main routing table properly.
IIRC quagga accepts prefixes with bogus next-hop (I haven't tried in a while so I maybe wrong about this).

All in all, I find ZeroByte's solution much easier to implement (taking into account that you haven't worked with BGP before).
I avoid doing any kind of redistributions on BGP so naturally the first idea that came into mind was not the simplicity of static routes redistribution :P
Also routeros does not advertise more than 200 networks (/routing bgp network) per instance. But it should work perfectly fine with thousands of redistributed prefixes.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Aug 15, 2016 11:33 pm

Sounding more and more like the script is a much simpler way to go :)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 1:28 am

All in all, I find ZeroByte's solution much easier to implement (taking into account that you haven't worked with BGP before).
I avoid doing any kind of redistributions on BGP so naturally the first idea that came into mind was not the simplicity of static routes redistribution :P
Also routeros does not advertise more than 200 networks (/routing bgp network) per instance. But it should work perfectly fine with thousands of redistributed prefixes.
Glad you liked the idea - and in general, you'll see that I'm quite the opponent of redistributing routes, ESPECIALLY into BGP, but this is a special case where all of that best practice stuff for network engineering goes right out the window. The easiest thing to do is just redistribute routes into BGP on a box that is not otherwise doing any routing.

Setting BGP into its own routing table at the process level makes things even simpler - I didn't actually try to lab this up, but had I done so, I certainly would've caught that requirement.
Sounding more and more like the script is a much simpler way to go
Funny thing is - to me, a script is always less desirable than leveraging the built-in behaviors of a system. People a lot smarter than me had a conference to make these standard protocols as robust as possible, so using them is like using a wheel as opposed to inventing my own wheel. But BGP distribution is not without its drawbacks - the biggest one right now is that null routing + RPF enforcement is the only thing you can do with it, and while it's effective as a blacklist, it is nowhere near as flexible as an IP address list. (which is why I'm hoping they do implement "add target to address list" as an action for routing filters).

I will say that the BGP method would be simpler to manage over a large distribution, and the implementation on the client side is brain-dead simple:
enable BGP (if not already using BGP) with any private ASN other than 64567. (or just use their real ASN if they're already running BGP).
in-filter=accept all -> action=set route type=blackhole
out-filter=discard all
enable strict RPF in IP options.

The nice thing about BGP is that the subscriber can put whatever kind of filters they like against the feed - they can specify no prefixes shorter than /22 for instance, if they hate the idea of blacklisting entire /16 or /8 prefixes. They can specify IP blocks to ignore. If running real BGP, they can set local_pref to 1 (very very bad) on the blacklist peer, so that no publicly-routed prefix can be blacklisted in its entirety.... it gives the paranoid administrator much more control than simply importing a list carte blanche and black-holing everything in it.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 3:01 am

I started a very basic bgp blacklisting service as a proof of concept of what we are talking about.

Anyone that would like to try it can use this page to register their bgp peers:
https://bgp-register.cha0s.gr

You give the IP & AS of your bgp instance and it will automatically create the BGP peer on my side and provide the user with basic copy/paste routing filters + bgp peer configuration.

It automatically generates a random 10 digit MD5 key for the BGP peer.

If someone wants to delete their peer from the system they can do so by providing the IP + AS + MD5 key.

Also the peers on my side are configured in passive mode so that they don't try again and again to connect to remote peers (in case someone stops using the service for example and forgets to delete their peers)

Since this totally a proof of concept, the design and usability of the page (or lack thereof :P) is obviously bad.
Also I haven't put any effort in input sanitization so it's quite probable that someone may find a way to break it :P :lol:

I wrote a simple script that downloads the blacklist from IntrusDave's service, converts the commands to static blackhole routes and adds them to my BGP router which in turn advertises them to every BGP peer.
I haven't implemented a way to keep updating the static routes upon changes of the list.

If there's interest I could develop the service a little more to be more complete and stable (ie: 2 bgp instances on different datacenters, proper registration UI, etc).


I completely agree with ZeroByte regarding BGP. It's way more efficient and flexible.
To be honest I've known about this blacklist since its beginnings but the reason I haven't used yet is because I don't like the idea of receiving a number of IPs for blocking without any way of filtering stuff that I might not want blocked.
Using BGP filters anyone can accept only the blacklists that he/she wants.
 
efaden
Forum Guru
Forum Guru
Posts: 1711
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Tue Aug 16, 2016 8:37 am

How would bgp work for dynamic clients?


Sent from my iPad using Tapatalk
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 2:12 pm

At the moment it wouldn't. Not without somekind of VPN I guess...

That's where dynamic neighbors would help. You would (in theory - I haven't used it) establish a BGP session regardless of your IP.
I believe there was a feature request about this a while back. I can't find it at the moment.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 3:22 pm

I haven't implemented a way to keep updating the static routes upon changes of the list.
I extended the conversion script to check all the current static routes on the router and remove or add any changes that occur from IntrusDave's blacklist.

I've set it up to run every 24 hours at 00:00 GMT+3 DST.

So it's pretty much all automatic now. It will keep all the static routes (blackholes) up to date with minimum effort and maximum efficiency (only changes are propagated to all the bgp peers instead of the whole list every day).

Another advantage of using BGP is that you can push changes almost in real time instead of periodically checking for an updated list.
Once you update the list the BGP can push the changes right away to everyone.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 4:12 pm

I am not sure what's the peer limit on routeros (if any).
I just tried adding ~18.000 peers on a single BGP instance and it worked fine. So there doesn't seem to be any (low) limit on how many peers you can have on ROS BGP. :)
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 4:51 pm

Nice favicon on the BGP registration page, Cha0s. ;)
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 5:09 pm

Yeah, that's probably the only pretty thing about the page :lol:
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 5:32 pm

Coming back to the efficiency argument, I just peered a couple of mAP-Lite with the blacklist and they loaded ~2900 prefixes into the routing table without breaking a sweat. Upon connection establishment it took ~2 seconds of high cpu usage (no more than 80%) and that was it.

Subsequent updates to the list will take virtually no resources (assuming that the updates are a few dozen prefixes +- every day).
Only on peer re-connection will have a cpu spike to re-add all prefixes to the routing table.

Prefix matching on the routing table level is much more efficient than matching an address list with iptables.
So even small devices can handle many addresses in the blacklist without any compromise in speeds.

One caveat is that BGP will need more ram the more prefixes it handles. Though anything above 64MB should work with many thousands of prefixes without problem.
http://wiki.mikrotik.com/wiki/Manual:BG ... e_table.3F
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 7:21 pm

Have you done any forwarding performance stress tests with the BGP feed in place and fully synchronized?
How about with strict RPF checks enabled?
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Tue Aug 16, 2016 8:21 pm

I set up a little lab to try this.

RB3011 <-> RB450 <-> hAP Lite

I did bandwidth tests from RB3011 to hAP Lite and measured the forward performance on RB450 (since it's old and with such small address list the results are more notable)

Bandwidth tests were done at 512 Bytes (UDP) and were limited at 100mbit due to fast-ethernet ports on the RB450 & hAP Lite.

The results are the following

PLAIN TEST (no BGP, no firewall)
100mbit traffic @ 22kpps
Route cache on: 55-60% CPU (top process: ethernet 25-30%, second top process: networking 15-20%, third: unclassified 9%)
Route cache off: 90-95% CPU (top process: networking 25-30%, second top process: ethernet 25-30%, third: routing 15%)
Total blacklist entries: 0
Total blackhole routes: 0
Free Memory: 12.2MB

RP Filter did not make any difference


BGP TEST
100mbit traffic @ 22kpps
Route cache on: 55-60% CPU (top process: ethernet 25-30%, second top process: networking 15-20%, third: unclassified 9%)
Route cache off: 90-95% CPU (top process: networking 30-35%, second top process: ethernet 25-30%, third: routing 15-20%)
Total blackhole routes: 2966
Free Memory: 10.4MB
Time to load all prefixes: ~2seconds

RP Filter did not make any difference


FIREWALL TEST (single drop rule- NOTHING else at all)
100mbit traffic @ 22kpps
Route cache on: 95-100% CPU (top process: ethernet 30-35%, second top process: firewall 25-30%, third: networking ~20%)
Route cache off: 100% CPU (top process: networking 30-35%, second top process: firewall 25-30%, third: ethernet 20-25%)
Total blacklist entries: 2966
Free Memory: 6.9MB
Time to load address-list: ~7seconds

RP Filter did not make any difference


So it's rather obvious that making decisions using the routing table instead of the address-list is more efficient. Especially with route cache on (by default it's on already).
It's also less memory hungry.

Disclaimer: The above tests were done very quick without many repeats so they are not 100% accurate or reflect real world performance (multiple connections, multiple firewall rules, etc).
 
efaden
Forum Guru
Forum Guru
Posts: 1711
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Blacklist Filter update script

Tue Aug 16, 2016 9:39 pm

BGP seems like the way to go...

Sent from my XT1575 using Tapatalk
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 18, 2016 8:32 am

Very cool. I guess I need to figure out how to do this on my end too. I have a CHR sitting idle in my datacenter.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:16 pm

I haven't looked through the rules / etc on your list, Dave, but I was wondering if you plan to use the Raw table for the rule to drop blacklisted source/destination packets so that they don't create entries in the connection tracking table.
I do, but the vast majority of routers pulling the list are still running 6.35 and lower. 6.32.4 makes up about 85% of the total. Once the majority are running a RouterOS that supports the RAW table, then I will move to that.

As it is now, you can simple move the drop rule from the firewall to RAW and it works nicely. That's what I do, personally.

hi,

can you share your raw rules ? Im using 6.36.
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 907
Joined: Tue Oct 11, 2005 4:53 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:24 pm

You can download the list with curl
curl -A "Mikrotik/6.x Fetch" "https://mikrotikfilters.com/download.php?get=dynamic&model=RB3011UiAS&version=6.36 (stable)&memory=1011.3 MiB&id=MikroTik&ver=2016.7.4a"
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:27 pm

can you share your raw rules ? Im using 6.36.
I would just say put two rules in raw table:
chain=prerouting src-address-list=blacklist action=drop
chain=prerouting dst-address-list=blacklist action=drop
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Fri Aug 19, 2016 4:48 pm

Hi ZeroByte,

thank you verymuch. I did it like that;
0 chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist
1 chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist
 
savage
Forum Guru
Forum Guru
Posts: 1211
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: Blacklist Filter update script

Sat Sep 17, 2016 9:03 pm

BGP seems like the way to go...

Sent from my XT1575 using Tapatalk
Said that in post #18 already - but then it was thought better to give me negative karma for the post :)
Regards,
Chris
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Sep 18, 2016 8:25 am

You can download the list with curl
curl -A "Mikrotik/6.x Fetch" "https://mikrotikfilters.com/download.php?get=dynamic&model=RB3011UiAS&version=6.36 (stable)&memory=1011.3 MiB&id=MikroTik&ver=2016.7.4a"

I would prefer that people don't do this. I already have one site that is mirroring my list and claiming it as his own. Very annoying.

As for BGP - I simply don't care to put the time in to building a system to setup the peers. Yes, I know it may ultimately be a better way to do this, but the current way is VERY easy for me, and I don't need to do any extra work for the number of little RB951's (hAP's) that I deploy. I really don't care to setup BGP on them, and given the limited memory, having the server only serve them the "3 day" list keeps things small and simple.

Maybe someday I'll BGP.. just not now. :)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Sep 18, 2016 8:31 am

BGP seems like the way to go...

Sent from my XT1575 using Tapatalk
Said that in post #18 already - but then it was thought better to give me negative karma for the post :)
Withdrawn. Still don't think it the best way to go :)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
brianlewis
Member Candidate
Member Candidate
Posts: 131
Joined: Tue Jul 20, 2004 10:54 am
Location: Irvine, CA

Re: Blacklist Filter update script

Wed Oct 12, 2016 12:41 am

Dave,

I've started using your list and I'm getting reports of legitimate sites being blocked dynamically from the filter list.

salesforce.com not coming up (72.21.81.200 which isn't their primary ip, maybe an image cache server?)
ssl.cdn-redfin.com (72.21.91.8) which prevents redfin.com, trulia, hilton.com and many other sites from working)

Any way we can look at why 72.21.x.x is blocked and consider adjusting that blacklisting?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 12, 2016 1:48 am

the ip (72.21.81.200) was flagged because it is currently serving malware in the form of infected images.
As the blacklist is free for use and was designed to keep my clients safe from infection - I will not be removing the IP.
If you really need it, I would recommend creating a separate whitelist for IPs you do not want blocked.

For me, I will not allow my clients to access sites that are currently serving malware.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 12, 2016 1:54 am

Correction, it's serving Ransomware via Javascript.
Once they have fixed the issue, it will automatically be removed.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Mon Oct 17, 2016 10:07 pm

Any reason the below suggestions never made it to the updateBlacklist script?
(note that I've already updated the first code fragment below from `intrusBlacklist` to `dynamicBlacklist`)

Same for the ordering of the firewall rules further below: any reason why?
:foreach i in=[/ip firewall address-list find ] do={ :if ( [/ip firewall address-list get $i comment] = "dynamicBlacklist" ) do={ /ip firewall address-list remove $i } }
Can be simplified to
/ip firewall address-list remove [/ip firewall address-list find comment = "dynamicBlacklist"]
Should actually increase the efficiency.
Firewall rules order:
/ip firewall filter
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
Where I have already see part of that rules on exact order? :lol:
Ah:
http://forum.mikrotik.com/viewtopic.php?f=9&t=83387

Sort the rules for efficency (simply drop if coming from blocked list, not first check malformed packet then drop)
/ip firewall filter
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=input comment="Allow ICMP Response" icmp-options=8:0 protocol=icmp
add chain=input comment="Allow any packets from our trusted \"IPSec\" partners" connection-state=new src-address-list=ipSec
add chain=input comment="Allow the Private IP ranges to access the router" connection-state=new src-address-list=PrivateIPs
add chain=input comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add action=drop chain=input comment="Drop everything else by default"
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
add chain=forward comment="Allow current valid connections as well as valid related packets" connection-state=established,related
add chain=forward comment="Allow the Private IP ranges to be forwarded by the router" connection-state=new src-address-list=PrivateIPs
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=blacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=blacklist
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Attacks comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Attacks comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Attacks comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Attacks comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Attacks comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=drop chain=Attacks comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=return chain=Attacks comment="Return to the chain that jumped"
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2
--jeroen
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Oct 18, 2016 5:03 am

You are welcome to change the script and rules as much as you like.
Script is written as it is because it works without fail on all 6.x versions. I don't normally change things if they are working.
The rules in the first post do have the blacklist drops at the top.
However, most by this point should be using raw drops instead of filter drops.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Tue Oct 18, 2016 11:14 pm

You are welcome to change the script and rules as much as you like.
Script is written as it is because it works without fail on all 6.x versions. I don't normally change things if they are working.
The rules in the first post do have the blacklist drops at the top.
However, most by this point should be using raw drops instead of filter drops.
Thanks. Just wanted to know the reasoning while wading through this thread just in case I missed something.

With `raw` drops, you mean rules like in http://forum.mikrotik.com/posting.php?m ... 9&p=553094 right?

--jeroen
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 19, 2016 1:28 am

Filter rules are more of a personal thing. There is no one-size-fits-all solution. Mine evolve all the time and I don't go back to the first post and update all the time. They are just an example. That said, here are my current "starter set" for most new routers I deploy.

/ip firewall filter
add action=reject chain=Filter dst-address-list=dynamicBlacklist reject-with=icmp-admin-prohibited
add action=accept chain=Filter connection-state=established,related
add action=drop chain=Filter comment="Invalid packets (No valid current connection)" connection-state=invalid
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=!fin,!syn,!rst,!ack
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,syn
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,rst
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,!ack
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=fin,urg
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=syn,rst
add action=drop chain=Filter comment="Invalid TCP flag combo" protocol=tcp tcp-flags=rst,urg
add action=drop chain=Filter comment="Invalid TCP source port (0)" protocol=tcp src-port=0
add action=drop chain=Filter comment="Invalid TCP destination port (0)" dst-port=0 protocol=tcp
add action=drop chain=Filter comment="Invalid UDP source port (0)" protocol=udp src-port=0
add action=drop chain=Filter comment="Invalid UDP destination port (0)" dst-port=0 protocol=udp
add action=return chain=Filter comment="Return to the chain that jumped"
add action=jump chain=input comment="Check for bad stuff in \"Filter\" chain" jump-target=Filter
add action=accept chain=input dst-port=8291,22,443 protocol=tcp src-address-list=trustedHosts
add action=accept chain=input in-interface=lanBridge
add action=drop chain=input in-interface=internet
add action=jump chain=forward comment="Check for bad stuff in \"Filter\" chain" jump-target=Filter
add action=accept chain=forward in-interface=lanBridge out-interface=internet
add action=drop chain=forward in-interface=internet
/ip firewall raw
add action=drop chain=prerouting src-address-list=dynamicBlacklist
Notice that the inbound blacklist drops are in the raw table, while the outbound are in the filter table. Also inbound are drops, outbound are ICMP Admin Prohibited.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Wed Oct 19, 2016 6:29 pm

... here are my current "starter set" for most new routers I deploy.

...

Notice that the inbound blacklist drops are in the raw table, while the outbound are in the filter table. Also inbound are drops, outbound are ICMP Admin Prohibited.
Thanks again. There is still a lot of stuff to be learned for me.

How did you find about the meaning of the various `reject-with` values? They are not documented any more and in the past were never explained in the documentation.

--jeroen
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 19, 2016 8:00 pm

That's more of an intro to TCP/IP thing. Note a Mikrotik thing.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Bipe
just joined
Posts: 1
Joined: Fri Jul 15, 2016 6:20 pm

Re: Blacklist Filter update script

Thu Nov 10, 2016 9:58 pm

Hi IntrusDave,

I've been having issues with your blacklist and bit.ly
looks like 67.199.248.10 and 67.199.248.11 are used by bit.ly and with firewall rule drop connections to blacklisted I can't access any short url's generated by it (had to exclude them).
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 10, 2016 11:41 pm

That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.

I will not manually remove addresses.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8318
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Thu Nov 17, 2016 9:35 pm

This topic looks interesting for many people. Let's make it sticky :)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
dadaniel
Member Candidate
Member Candidate
Posts: 158
Joined: Fri May 14, 2010 11:51 pm

Re: Blacklist Filter update script

Mon Nov 21, 2016 4:24 pm

That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.

I will not manually remove addresses.
bit.ly is a referer-website (like shorturl), it never serves anything from its own IP address. Could you please have a look again?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Nov 21, 2016 7:31 pm

No, I'm sorry. As I said, the processes is automated. It receives and processes nearly 100,000 IP's each day. If an IP makes it on the list, then it has been directly or indirectly responsible for malware. The whole thing was designed to keep my personal clients safe. If it's not working for you, then you have a few options.
1) ask for a refund and don't use the list.
2) use the list as an incoming only filter
3) use the list as a raw in and out list, and whitelist the addresses you feel are wrongly blocked.

Personally, I use option 3 for businesses, and I use option 2 for home users.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
bajodel
Long time Member
Long time Member
Posts: 545
Joined: Sun Nov 24, 2013 8:30 am
Location: Italy

Re: Blacklist Filter update script

Wed Nov 23, 2016 3:57 pm

.. [CUT] ..
3) use the list as a raw in and out list, and whitelist the addresses you feel are wrongly blocked.
Personally, I use option 3 for businesses .. [CUT] ..
Testing (mode 3) now on a new hEX and works like a charm. Thanks.. (rep+)
I'm wondering if I can consider "reliable" your service (not in terms of false positive or alike.. but) in terms of availability of updates ; I'm considering to put this in production but I'm evaluating if create blacklists by myself or (pay and) pretend to have reliable external service. I think you can understand
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Nov 23, 2016 5:44 pm

I feel it's reliable. I have all of my clients using it. (24 regional hospitals and medical centers). In addition, I have 1.830 other routers that use it. My only wish would be that all of these routers were able to send back addresses that they are attacked by. Unfortunately, there is no good way without putting their privacy at risk.

After all is said and done, The back end system I've built is fully self contained and is running on hardware that was built to last the decade.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
nickperkins
just joined
Posts: 8
Joined: Mon Nov 24, 2014 6:04 am
Location: Masterton, New Zealand
Contact:

Re: Blacklist Filter update script

Thu Nov 24, 2016 11:39 pm

I had to make a slight change to the script for a couple of my clients that had spaces in their router identity. The spaces were causing the fetch to fail, have added this in to resolve the issue:
:local uname	[/system identity get name]
:for i from=0 to=( [ :len $name ]-1 ) do={ 
:local tmp [ :pick $name $i ]
:if ($tmp != " ") do={ :set $newname "$newname$tmp" }
}
:set $uname $newname
MTCNA - MTCWE
Nick Perkins
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 24, 2016 11:43 pm

Odd, the server normally deals with that. Can you give me the name the fails? I'd like to try and reproduce the error.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
nickperkins
just joined
Posts: 8
Joined: Mon Nov 24, 2014 6:04 am
Location: Masterton, New Zealand
Contact:

Re: Blacklist Filter update script

Fri Nov 25, 2016 12:36 am

Odd, the server normally deals with that. Can you give me the name the fails? I'd like to try and reproduce the error.
Hi IntrusDave, the first case I found it the identity was 'Nick Home AP', I see there is a warning in the log when running that there is a new version at updater.php, had a look at that and I see that version doesn't read the identity. Perhaps that's the better option?
MTCNA - MTCWE
Nick Perkins
 
jeroenp
Member Candidate
Member Candidate
Posts: 159
Joined: Mon Mar 17, 2014 11:30 am
Location: Amsterdam
Contact:

Re: Blacklist Filter update script

Wed Nov 30, 2016 11:50 pm

My only wish would be that all of these routers were able to send back addresses that they are attacked by. Unfortunately, there is no good way without putting their privacy at risk.
Let's discuss (in a new thread if needed) on how to make this possible.

I'm keeping dynamic login failure and unknown-port usage blacklists currently having 10k+ entries (a huge increase after the TR-069 issues at German Telekom) with expiration of 14 days.

It would be cool if I could get them at your place somehow.

--jeroen
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Dec 01, 2016 1:56 am

I am using the list and service by IntrusDave since a few day and it works very well and I had many hits on the rule. I use it more selective by filtering obvious illegal request out in advance. I run three services and that is mail, web en secure web.

Now those botnets have that many bots and that will result in very long lists of IP and results decreases the efficiency in using those lists if you look at filtering time. Last weekend it was very busy and many many bots tried to get in so I looked at what they were doing. There were not hat many that were caught by the list and 99% were filtered by the following rule that just leaves a window for the services I serve, and filter out any thing else on TCP that is obvious illegal.
/ip firewall raw
add action=drop chain=prerouting comment="Pre-filter TCP" dst-port=!25,80,443 in-interface=pppoe-out log-prefix=\
    "New drop" protocol=tcp tcp-flags=syn,!fin,!rst,!psh,!ack,!urg,!ece,!cwr
In this way all filters behind this do only have focus on the services that are running and check if there is everything ok before conveying the requests to the servers.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
HiltonT
Frequent Visitor
Frequent Visitor
Posts: 75
Joined: Mon Feb 07, 2011 4:24 am
Location: 'Srayamate
Contact:

Re: Blacklist Filter update script

Fri Dec 02, 2016 12:42 am

I started a very basic bgp blacklisting service as a proof of concept of what we are talking about.

Anyone that would like to try it can use this page to register their bgp peers:
https://bgp-register.cha0s.gr

Since this totally a proof of concept, the design and usability of the page (or lack thereof :P) is obviously bad.
Also I haven't put any effort in input sanitization so it's quite probable that someone may find a way to break it :P :lol:

If there's interest I could develop the service a little more to be more complete and stable (ie: 2 bgp instances on different datacenters, proper registration UI, etc)...
I gather this page/service didn't garner enough interest to keep it online (or has it just temporarily gone offline)? That's a shame - I like the idea of a centrally maintained set of blacklists that I can subscribe various devices to, however I'm not a massive fan of the lag a massive address-list puts on the traversal time of packets through the firewall, nor the CPU load this requires.

I know as much about BGP as the next person who has never used it, but I'm definitely interested in learning more about it (with the use of Private AS Numbers as the cost of a public one isn't going to be justifiable for many of our clients) for use in blocking unwanted traffic - I really don't see it being useful for SMBs with one primary and a backup Internet connection when there's an AU$500/year fee for the public AS Number.
Regards,
Hilton Travis
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Dec 02, 2016 12:50 am

Configured correctly, you should not see much, if any increased CPU load or lag. You should be filtering the initial connection, not the established.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
telepro
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Sun Apr 03, 2011 7:50 pm

Re: Blacklist Filter update script

Mon Dec 12, 2016 5:03 pm

We have been using the blacklist provided by IntrusDave with success. However, we have on infrequent occasions found the list contains an IP address that is the source of Microsoft download web sites and files linked to by their corporate web pages. Momentarily turning off the use of the blacklist provides successful web access to these Microsoft sites. Has anyone else see this issue?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Dec 12, 2016 5:34 pm

it is not uncommon. The blacklist is an automated system that flags any IP that has served malware in the last 7 days. Just because a CDN is used/owned by Microsoft doesn't mean that it is impervious to malware.

Again, as I have stated before, This system was designed by me to keep my paid clients as safe as possible. I use this for all 24 of my hospitals and clinics. It works well to help stop the attacks of botnets and helps to prevent infection. That said, I will not whitelist any IPs just because they are used by a large company. Any website or CDN can be infected, no one is exempt from being filtered for it.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Mon Dec 12, 2016 6:56 pm

Love the work you did on this script. It's fantastic! however I too am finding a few services are blocked. Microsoft for one, Periscope (A Twitter company) is blocked. I'm currently deciding if its worse the hassle :)

But you did do a fantastic job, great service, I thank you much for it!
 
telepro
Frequent Visitor
Frequent Visitor
Posts: 63
Joined: Sun Apr 03, 2011 7:50 pm

Re: Blacklist Filter update script

Mon Dec 12, 2016 7:04 pm

I concur. The provided programming and database has proved to be very useful; when employed, it filters out a significant amount of unwanted and problematic traffic. Thus our question regarding whether anyone else was seeing traffic from Microsoft corporate sites blocked. Thanks for the info.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Dec 12, 2016 9:09 pm

These companies use CDNs. So what you see as blocked, I may not see blocked. When something is added to the block list, it is because that IP was found to have some form of malware.

The filters can be used in many ways.
The list can be used in the RAW or the standard filters. Both incoming and outgoing.

If you are not able to access a website because of the list, that means that you are using it either in RAW, or on an outbound rule.
You should be using it in the INPUT chain, with the New Connection flag. You do not need to be filtering established connections.
You also don't need to filter destination IPs, unless you want more malware protection.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Mon Dec 12, 2016 10:02 pm

I was using the new RAW rules and blocking destination etc I'll just use an input/forward rule then :)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Wed Dec 14, 2016 7:53 am

I was using the new RAW rules and blocking destination etc I'll just use an input/forward rule then :)
share your rules for us :)
 
brianlewis
Member Candidate
Member Candidate
Posts: 131
Joined: Tue Jul 20, 2004 10:54 am
Location: Irvine, CA

Re: Blacklist Filter update script

Wed Dec 21, 2016 5:07 pm

Any particular reason Microsoft’s Ajax CDN (72.21.81.200) is being blacklisted?
 
zhup
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Thu Dec 03, 2015 10:10 pm

Re: Blacklist Filter update script

Thu Dec 22, 2016 8:23 pm

Hello IntrusDave,
Great work!

Could you please change the script? It would be good to write the lists to the pendrive instead of the nand.
Thank you in advance.
zhup
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Dec 22, 2016 8:31 pm

You are welcome to change it as you like. I don't use flash drives in my routers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
zhup
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Thu Dec 03, 2015 10:10 pm

Re: Blacklist Filter update script

Thu Dec 22, 2016 9:56 pm

You are welcome to change it as you like. I don't use flash drives in my routers.
Could you please check if I made all necessary changes for using the pendrive?
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
:delay 30

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2016.7.4a

:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/disk1/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/disk1/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Thank you in advance.

Do you use the blacklist from OpenBL.org?
zhup
 
jo2jo
Forum Veteran
Forum Veteran
Posts: 958
Joined: Fri May 26, 2006 1:25 am

Re: Blacklist Filter update script

Sun Jan 01, 2017 1:27 pm

Just wanted to say THANK YOU for all your hard work on this list! its really excellent! and your decision to use dynamic address list entries is really sharp.

(dynamic address entries simply means that on the address-list rules, he sets a timeout value, so that the mikrotik stores the address list in RAM memory until it times out, versus it being a normal address list entry with no time out, and thus the mikrotik stores the entry on its "disk" or nand drive, so that it will persist through reboots). However ALL nand based memory has a "limited" lifetime which is slightly reduced by each write (think write endurance on SSD drives) - the better way is to store this type of data in RAM memory as it does not have this degradation through writes issue.)

ie:
non "dynamic" address list entry (will be stored on the routers "disk" such that it will persist/remain through reboots or power failures, but the nand memory *does* degrade with each write):
/ip firewall address-list add address=x.y.z.z

vs a "dynamic" address list entry (will be stored on the routers RAM , which has *no* degradation with each write):
/ip firewall address-list add address=x.y.z.z timeout=2d

Thanks again! and for any future messages from users with problems with a specific IP on his list (ie x.y.z.a IP address is blocked, but really belongs to google.com which my network needs to access), Please, PLEASE read back through this thread, the author has addressed his reasons (for ip inclusion, and why he also will not remove specific IPs from his list) and he also has provided a clear way for you to "fix" or override specific IPs which you "feel" your network must access
:beep :beep :beep
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 10:39 am

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
May I please be so bold, whats the commands to change the temporary file storage location?

I use a RB750Gr3 and has a microSD card installed. Currently its only really been used for backup configs and some logs as I haven't found away to switch more to use it.
 
flazzarini
just joined
Posts: 19
Joined: Thu Jun 13, 2013 11:05 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 11:42 am

Hi there,

I would be interested to know which source you are using to get this list of IP addresses to block? Would you care to share this? I would be interested to integrate the list you are serving into Blocklister (Github).

Thanks for your help and keep up the good work!
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 11:45 am

Hi there,

I would be interested to know which source you are using to get this list of IP addresses to block? Would you care to share this? I would be interested to integrate the list you are serving into Blocklister (Github).

Thanks for your help and keep up the good work!
The OP said in a previous post that he compiles the list himself from his 50 or so routers which gets attacked.
 
flazzarini
just joined
Posts: 19
Joined: Thu Jun 13, 2013 11:05 am

Re: Blacklist Filter update script

Thu Jan 05, 2017 12:36 pm

Hi there,

I would be interested to know which source you are using to get this list of IP addresses to block? Would you care to share this? I would be interested to integrate the list you are serving into Blocklister (Github).

Thanks for your help and keep up the good work!
The OP said in a previous post that he compiles the list himself from his 50 or so routers which gets attacked.
Thanks for the answer to that!
 
nwa
just joined
Posts: 23
Joined: Sun Aug 17, 2014 3:02 pm

Re: Blacklist Filter update script

Wed Jan 11, 2017 11:14 pm

I want only to say.... thanks !!!!

I hope the list works good for german locatet routerboards and this project never ends ;)
 
ignore
just joined
Posts: 1
Joined: Sat Feb 11, 2012 7:31 pm

Re: Blacklist Filter update script

Thu Jan 12, 2017 10:28 pm

can you share your raw rules ? Im using 6.36.
I would just say put two rules in raw table:
chain=prerouting src-address-list=blacklist action=drop
chain=prerouting dst-address-list=blacklist action=drop
 
proximus
Member Candidate
Member Candidate
Posts: 112
Joined: Tue Oct 04, 2011 1:46 pm

Re: Blacklist Filter update script

Thu Jan 12, 2017 11:11 pm

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
May I please be so bold, whats the commands to change the temporary file storage location?

I use a RB750Gr3 and has a microSD card installed. Currently its only really been used for backup configs and some logs as I haven't found away to switch more to use it.
In the script, edit it use disk1. So, the relevant parts would be:
/tool fetch mode=https dst-path="/disk1/dynamic.rsc" 
/import file-name=/disk1/dynamic.rsc
/file remove disk1/dynamic.rsc
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Sun Jan 15, 2017 2:56 am

If external USB or SD disk available, NAND wearing can be avoided by write temporary files to them.

PS. Downloading and executing rsc from not own server and/or by insecure channel look dangerous.
May I please be so bold, whats the commands to change the temporary file storage location?

I use a RB750Gr3 and has a microSD card installed. Currently its only really been used for backup configs and some logs as I haven't found away to switch more to use it.
In the script, edit it use disk1. So, the relevant parts would be:
/tool fetch mode=https dst-path="/disk1/dynamic.rsc" 
/import file-name=/disk1/dynamic.rsc
/file remove disk1/dynamic.rsc
Thanks very much. I did figure it out in the end and forgot to post here that I had found the solution. Thanks anyway for letting me know.
 
chippers
newbie
Posts: 25
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Mon Jan 30, 2017 12:12 pm

great script, I am seeing lots of hits against the listed IP's.

On reboot, is there a way to load the script automatically or do we have to wait for the scheduled update time?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jan 30, 2017 4:31 pm

you can add a second schedule to run at startup.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
chippers
newbie
Posts: 25
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Mon Jan 30, 2017 10:20 pm

Yes, I have that but doesnt seem to work :(

i'll try to troubleshoot, thanks

/system scheduler
add interval=1d name=updateBlacklist on-event="/system script run updateBlacklist" policy=read,write,test start-time=startup
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jan 30, 2017 11:03 pm

This works and you have to be patient because the script waits for 3+30 seconds giving the interfaces time to start completely because you need access to the internet.
add name="Startup updateBlacklist" on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test start-time=startup
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
chippers
newbie
Posts: 25
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Tue Jan 31, 2017 2:30 pm

ok, turns out I wasnt being impatient :)

I copied the schedules from the start of this thread and there are a couple of issues.
1. The schedule names are the same and this causes the import of the second schedule to fail, solution is to rename the second schedule
2. The run command differed in both schedules (run updateBlacklist) VS (run blacklistUpdate)

Here is what I ended up with and works as expected. List reloads about 30 seconds after reboot.
/system scheduler
add interval=1d name=UpdateBlackListDaily on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
add name=UpdateBlackListOnReboot on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=startup
Adding these simple rules as mentioned elsewhere in this thread and the IP list is working great!
/ip firewall raw
add action=drop chain=prerouting comment=\
    "Drop connections from Blacklisted addresses" src-address-list=\
    dynamicBlacklist
add action=drop chain=prerouting comment=\
    "Drop connections to Blacklisted addresses" dst-address-list=\
    dynamicBlacklist
    
Thanks for a great contribution...
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jan 31, 2017 3:44 pm


2. The run command differed in both schedules (run updateBlacklist) VS (run blacklistUpdate)

Here is what I ended up with and works as expected. List reloads about 30 seconds after reboot.
/system scheduler
add interval=1d name=UpdateBlackListDaily on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=jan/01/1970 start-time=05:00:00
add name=UpdateBlackListOnReboot on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=startup
Are you sure because I see twice the same script run command.

I will check if the script can't be started in sequence. I remember that this was not a problem but you never know. ;-)

Update: I have now checked it and the list was updated automatically this afternoon. I have different names for the script and I think you also wanted you communicatited.

It works really great now and maybe a default "startup" can be added to the installation script.

I have good results and in the log I see hits on the blacklist every day.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Feb 01, 2017 4:09 pm

If you want, you can use my blacklists. Blacklists updated every hour.

TOR Exit Nodes
OpenBL
SpamHaus DROP list
DShield
malc0de

RSC will create address-list named "Blacklist", IP's will be commented. Duplicate IP's will be skipped, if exists.

And of course, don't forget to schedule it and make corresponding filter rules. ;)

Script:
# Script will now download IP blacklists
/tool fetch url="http://www.securelan.eu/mikrotik/torexitnodes.rsc" mode=http;
:log info "Downloaded torexitnodes.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/openbl.rsc" mode=http;
:log info "Downloaded openbl.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/spamhaus.rsc" mode=http;
:log info "Downloaded spamhaus.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/dshield.rsc" mode=http;
:log info "Downloaded dshield.rsc from SecureLAN.eu";
/tool fetch url="http://www.securelan.eu/mikrotik/malc0de.rsc" mode=http;
:log info "Downloaded malc0de.rsc from SecureLAN.eu";
# Script will now replace old blacklists with the new ones
/ip firewall address-list remove [find where comment="TorExitNodes"]
/import file-name=torexitnodes.rsc;
:log info "TorExitNodes records updated successfully.";
/ip firewall address-list remove [find where comment="DShield"]
/import file-name=dshield.rsc;
:log info "DShield records updated successfully.";
/ip firewall address-list remove [find where comment="SpamHaus"]
/import file-name=spamhaus.rsc;
:log info "SpamHaus records updated successfully.";
/ip firewall address-list remove [find where comment="OpenBL"]
/import file-name=openbl.rsc;
:log info "OpenBL records updated successfully.";
/ip firewall address-list remove [find where comment="malc0de"]
/import file-name=malc0de.rsc;
:log info "Malc0de records updated successfully.";
:log info "All blacklist records were updated successfully.";
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Feb 01, 2017 10:03 pm

I've gone ahead and started publishing my dynamic filter list for RouterOS 6.x. My server generates the list each night after collecting data on all known botnets, C&C server, and spammers. Currently the list runs about 3k entries, so it may not work well on low end routers. Here is the script to update the list, as well as my personal firewall rules. As always, adjust them to fit your needs.  
Does your list contain also TOR network exit nodes? If not, you can probably add it. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Feb 01, 2017 10:22 pm

No it doesn't. That is not something that I am interested in blocking. I am a big privacy advocate and I don't want to take away that option
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Feb 01, 2017 10:31 pm

Yeah, but privacy is not always secure.... in Tor there is lot of ransomware servers hidden. No connection to TOR, no encrypted disk. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Feb 02, 2017 12:36 am

If a user is using TOR, then they are on their own for security. At this time I have no interest in blocking TOR.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Thu Feb 02, 2017 12:59 am

yeah...that's true...but.. :) for me, in enterprise environment, tor should not be allowed.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Feb 02, 2017 1:05 am

Then you should filter it. However, nearly impossible to track the ever changing exit nodes, and impossible to detect.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Thu Feb 02, 2017 1:07 am

That's why I am generating TOR exit nodes list every hour. :) Check my post earlier. :) Could you compare my lists with yours? Probably there's something to make better...on both. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Feb 06, 2017 8:21 pm

Sorry, not going to block TOR nodes. I am an active donor to the TOR project. It would be hypocritical of me to block it. But thank you for the input.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Feb 11, 2017 12:08 am

I wanted to give a status update on my blacklist.

As of this morning, the Blacklist has 3,500 routers downloading the list everyday. They are pulling 1.7GB of data every 24 hours. Just about 52GB per month. I have moved the handling of the blacklist to a dedicated server. I currently use 4 high-profile blacklist services, in addition to the 215 honeypots that I collect data from all over the USA.

I have watching the FCC rulings very closely, and I will not hesitate to move the servers outside of the USA if I feel the list is at risk. I am currently looking into ways of having RouterOS check a SHA256sum to verify the validity of the list.

Again, this list was started for my own use on the MikroTik routers that I manage. I do not charge for this list, and I have never asked for donations. That said, I have always been open to suggestions to make it better, but please remember that my primary concern is the safety of the medical groups and hospitals that I manage.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Feb 11, 2017 3:37 pm

I have an idea how to bring back the traffic generated by the Blacklist.

When I lookup sites I get sometimes a list of IP addresses back:
Name: microsoft.com
Addresses: 23.100.122.175
23.96.52.53
191.239.213.197
104.40.211.35
104.43.195.251
So if you can convert the list and put it in a DNS, then one record/domain name will supply all IP addresses in one go.

You could to make weekday's list like monday.blacklist.xxx / tuesday.blacklist.xxx....sunday.blacklist.xxx
Give the DNS-record a lifetime of 24+1 hour and remove the that day when it is the next day is generated and is uploaded. In this way the your are sure that the cache DNS servers up the stream are cleaned to read in that weekday.blacklist.xxx when there is an request for it in the Internet.

When a weekday*7.backlist.xxx is in the cache of the DNS in the Mikrotik, you only need one line in the address list to be able to filter. I think that a script is useful to make a hard delete of the outdated weekday to make room for the new weekday list.

The DNS of the provider/supplier which the Mikrotik owner is using is handling the traffic now. You have each day a one time upload and the the DNS structure is distributing your list for you. Delays are common and because the used weekday was not present for the last 5 days so there should be direct request to the DNS.

This way of working I already use myself and I put the extra IP addresses in the host file on the machine where my DMSmasq is running. DNSmasq reads the host file and returns the list of IP addresses when the domainname is requested. In doing so I have only need one line to be able to filter more addresses in one go.

I don't know if this is possible or even legal to use the DNS in that way.....

updated: 12 February 2017
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Mar 06, 2017 6:16 pm

Just hit 4000 active routers using the BlackList.
Notable users are T-Mobile, using it on there Fixed LTE deployments. And even more so, several US Government sites have begun pulling the list.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Mar 07, 2017 12:45 pm

Just hit 4000 active routers using the BlackList.
Notable users are T-Mobile, using it on there Fixed LTE deployments. And even more so, several US Government sites have begun pulling the list.
Good to see the grow from 2700 to 4000 clients in the last seven months.
I made a suggestion to use DNS to distribute the list and now I read again the start page of this posting and BGP also seems a solution.
The blacklist get many hits on my connection and I am pleased that those connections tries are terminated!
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Mar 07, 2017 6:45 pm

DNS and BGP both complicate things dramatically. The current distribution method is very simple, stable and requires very little to setup.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Tue Mar 07, 2017 8:47 pm

I have an idea how to bring back the traffic generated by the Blacklist.

When I lookup sites I get sometimes a list of IP addresses back:
Name: microsoft.com
Addresses: 23.100.122.175
23.96.52.53
191.239.213.197
104.40.211.35
104.43.195.251
So if you can convert the list and put it in a DNS, then one record/domain name will supply all IP addresses in one go.

You could to make weekday's list like monday.blacklist.xxx / tuesday.blacklist.xxx....sunday.blacklist.xxx
Give the DNS-record a lifetime of 24+1 hour and remove the that day when it is the next day is generated and is uploaded. In this way the your are sure that the cache DNS servers up the stream are cleaned to read in that weekday.blacklist.xxx when there is an request for it in the Internet.

When a weekday*7.backlist.xxx is in the cache of the DNS in the Mikrotik, you only need one line in the address list to be able to filter. I think that a script is useful to make a hard delete of the outdated weekday to make room for the new weekday list.

The DNS of the provider/supplier which the Mikrotik owner is using is handling the traffic now. You have each day a one time upload and the the DNS structure is distributing your list for you. Delays are common and because the used weekday was not present for the last 5 days so there should be direct request to the DNS.

This way of working I already use myself and I put the extra IP addresses in the host file on the machine where my DMSmasq is running. DNSmasq reads the host file and returns the list of IP addresses when the domainname is requested. In doing so I have only need one line to be able to filter more addresses in one go.

I don't know if this is possible or even legal to use the DNS in that way.....

updated: 12 February 2017
If you insist in doing it via DNS then look into rbldnsd which is designed for exactly this purpose. You can feed it a list of IP's/hostnames and it can respond with whatever you want. RBL's used for mail etc commonly uses this method for their black/white or rep lists.

You can do more then just this, for example this guy here http://countries.nerd.dk/more.html uses it to make a countries lookup via dns which can then be used for things like mail/web etc black/white lists etc.

Anyway, personally, the way the list is right now is best as it can easily be adapted to whatever method/way you like.

Just my two cents.
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Mar 08, 2017 1:47 am

Using RBL's crossed my mind but then the amount of traffiic would be the same like it is with BGP.

When using DNS you will also have some traffic but the main part is distributed by external DNS severs as I see it.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
majestic
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Mon Dec 05, 2016 11:19 am

Re: Blacklist Filter update script

Wed Mar 08, 2017 1:54 am

Using RBL's crossed my mind but then the amount of traffiic would be the same like it is with BGP.

When using DNS you will also have some traffic but the main part is distributed by external DNS severs as I see it.
Distributed & cached which the cache will lower the amount of traffic needed.

However whether DNS is less then BGP traffic wise taking into effect of caching etc, i'm not sure. I think if there was enough devices pulling the data, BGP probably total up to more but thats an educated guess more then fact.
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 2:35 pm

Looks like a very interesting system you got here.
I know it has been running fine for soon 2 years I guess, but I do have a few small suggestions for your update script.
  • You should escape the "?" in the URL ("\?")
  • Add brackets around the script ("{ }")
  • Add a ":put" with the script version for debugging (":put "Script version: $scriptVer"")
These changes would make you able to run the script in the terminal. Or did you intentionally write it so that it didn't work in the terminal?
Example:
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies
{
    :log warning "Blacklist download will start in 30 seconds..."
    :delay 30
    
    :local model    [/system resource get board-name]
    :local version   [/system resource get version]
    :local memory   [/system resource get total-memory]
    :local uname   [/system identity get name]
    :local scriptVer   "2016.7.4a (Deantwo)"
    :put "Script version: $scriptVer"
    
    :log warning "Downloading current Blacklist for this model"
    /tool fetch mode=https dst-path="/dynamic.rsc" \
       url="https://mikrotikfilters.com/download.php\?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer"
    
    :log warning "Disabling info logging..."
    /system logging disable 0
    
    :log warning "Removing expiring address-list entries..."
    /ip firewall address-list remove [find list="dynamicBlacklist"]
    
    :log warning "Importing current Blacklist..."
    /import file-name=/dynamic.rsc
    
    :log warning "Removing temp file..."
    /file remove dynamic.rsc
    
    :log warning "Blacklist Update Complete."
    /system logging enable 0
}
When testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?

By the way, what is the reason for wanting the router's identity? Wouldn't it be more reliable to just use the serial number?
Your script doesn't require the identity to be sent, right? I can omit it?
Last edited by Deantwo on Mon May 01, 2017 11:59 am, edited 2 times in total.
I wish my FTP was FTL.
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Mar 09, 2017 3:36 pm

When testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?
Quotes from IntrusDave
My server collects the banned IP's 24/7 and publish the list at 3am PST.
That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.
I will not manually remove addresses.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:33 pm

When testing it I also found that the address-list entry timeout don't quite match up with what you say in the opening post.
It states that the address-list entries are dynamic with a 48 hour timeout, but the file I am getting shows them having a 24 hour timeout. Shouldn't this at least be 25 hours to patch the possible hole between updates, or be changed back to 48 hours?
Quotes from IntrusDave
My server collects the banned IP's 24/7 and publish the list at 3am PST.
That means that the ip/ subnet is or has been serving malware for at least 12 hours. The list is automated and will remove the address once it has been clean for 24 hours.
I will not manually remove addresses.
Doesn't explain why the timeout of the dynamic address-list entries is only 24 hours when it is stated in the opening post that the timeout is 48 hours.
From the opening post:
The address-list entries are now Dynamic with a 48 hour timeout. This will cut the number of writes to NAND down dramatically.
If nothing else the opening post just needs to be updated.

There is a small chance that the dynamic address-list manages to timeout before the new dynamic address-list is downloaded and applied. This could leave the system vulnerable for at least a couple of seconds each day when the update script is running. It could easily be fixed by extending the timeout by an hour or less. But maybe I am just overreacting at that.
I wish my FTP was FTL.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:34 pm

At one point the list was updated every 48 hours, but as malware has spread faster and responses are faster, the list now expires after 24 hours. Maybe upping that to 26 hours will help some. My routers update themselves every 23 hours. The script does run from the terminal as a whole...
/system script run UpdateBlacklist
It's not meant to be run line by line.
I use the identity to group the routers for stats and troubleshooting. Example; all of my routers ID's start with "Intrus :: " this allows me to sort them and quickly track down problems. While it's not currently required, it really is the only method that I have to keep track of how many routers are active daily. I do not use the serial number because I feel that is too invasive to request. I can not go by IP, because many are behind the same proxies. I could use the WAN MAC address, but I was betting that some would object to that too.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:45 pm

I use the identity to group the routers for stats and troubleshooting. Example; all of my routers ID's start with "Intrus :: " this allows me to sort them and quickly track down problems. While it's not currently required, it really is the only method that I have to keep track of how many routers are active daily. I do not use the serial number because I feel that is too invasive to request. I can not go by IP, because many are behind the same proxies. I could use the WAN MAC address, but I was betting that some would object to that too.
I was mostly asking because we have customer numbers and names as router identity, so I may be forced to not send you those if we start using your service.

On another note. The second scheduler in the opening post, isn't it meant to be on startup?
I use my startup scheduler scripts like this:
/system scheduler
add name="MyScheduler1" \
    start-time=startup \
    policy=read,write,test \
    on-event=":delay 120\r\
    \n/system script run \"MyScript1\""
Don't know if "start-date=jan/01/1970 start-time=00:00:0 interval=00:00:00" translate to "start-time=startup" somehow.
I wish my FTP was FTL.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 09, 2017 4:56 pm

Updated the first post and the timeout to 25 hours.

The identity is never seen by anyone but me. I do have DOD clearance, so nothing to worry about.. Well, I guess that doesn't mean much now days. You are welcome to set a static name for each router in the script. The database is stored on a separate server, with no direct internet connection. As for the schedule, you will have to play with it. It was originally setup back when the routers didn't store the date and time over a reboot, so on first boot the date and time was "1970-01-01 00:00:00". RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 5:13 pm

As for the schedule, you will have to play with it. It was originally setup back when the routers didn't store the date and time over a reboot, so on first boot the date and time was "1970-01-01 00:00:00". RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.
Not 100% sure rather or not to add the "start-date=jan/01/1970" to the scheduler, since I haven't messed with them for a while. But the scheduler I posted does work, and I use a two minute delay before calling my scripts because I need to be sure that VPN tunnels are up.
I wish my FTP was FTL.
 
cashwu
just joined
Posts: 4
Joined: Mon Sep 12, 2016 5:42 pm

Re: Blacklist Filter update script

Thu Mar 09, 2017 7:47 pm


RouterOS seems to have some issues with startup scripts, and I haven't had time to work out what needs to be changed.

Problem why scheduler cannot execute script is because script as more permissions than scheduler.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 09, 2017 7:53 pm

The startup is not a permissions issue. It has to do with the interval. When the interval is 24 hours, the first run doesn't occur until 24 hours after the boot.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue Mar 14, 2017 10:14 am

And, if you are interested, here are my filter rules:
/ip firewall address-list
add address=172.16.0.0/16 list=PrivateIPs
add address=10.0.0.0/8 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs
Found a little error in your provided example firewall.

Incorrect netmask for the 172 private range, it should be /12.
Like this:
/ip firewall address-list
add address=10.0.0.0/8 list=PrivateIPs
add address=172.16.0.0/12 list=PrivateIPs
add address=192.168.0.0/16 list=PrivateIPs
See: https://en.wikipedia.org/wiki/Private_network
I wish my FTP was FTL.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Mar 15, 2017 9:34 pm

You are correct. I will fix this.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue Mar 21, 2017 9:44 am

Something seems to have gone wrong.
I am receiving incomplete "dynamic.rsc" files, ending in the middle of an address-list entry add command.

Example:
# Generated on Mon Mar 20 04:00:54 PDT 2017 by Intrus Technologies
/ip firewall address-list

add list=dynamicBlacklist address=1.10.16.0/20 timeout="1d 01:00:00" comment=Blacklisted
# Omited 5226 lines.
add list=dynamicBlacklist address=42.62.51.27 timeout="1d 01:00:00" comment=Blacklisted
add list=dynamicBlacklist address=42.83.80.0/22 timeout="1d 01
Log shows that it is not always the same places that these files fail, for example:
mar/21 05:21:57 script,error script error: failure: already have such entry
mar/20 05:21:44 script,error script error: expected end of command (line 5586 column 70)
mar/19 05:21:56 script,error script error: expected end of command (line 5770 column 27)
mar/17 05:22:08 script,error script error: value of address expects range of ip addresses
mar/16 05:22:09 script,error script error: invalid time value for argument timeout
There is however a patten to how they fail.
For example the "already have such entry" error seems to be because it has created a non-dynamic address-list entry with the address 0.0.0.0 on the list "dynamicBlacklis", but the error indicate that it has done it more than once the exact same way.

My guess is that you are assuming the length of each line? But the length of the lines has changed. Maybe because it was changed from "timeout=1d" to "timeout="1d 01:00:00"" on each line, adding a total of 11 characters per line.

If you need to make the lines shorter, you could remove the comment, since it is kind of redundenet when you have the list name. Don't know if anyone rely on the comment though.

An annoying consequence of all this is that, if the import fails, info logging is never re-enabled.
/system logging enable 0
I wish my FTP was FTL.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Mar 21, 2017 4:57 pm

You issue is that the router simply didn't complete the download. Today's download is 603k. If it's getting out off, you may want to see if your ISP is trying to proxy ssl connections.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 191
Joined: Wed May 20, 2015 10:14 pm
Location: Quito

Re: Blacklist Filter update script

Tue Mar 21, 2017 7:32 pm

Hi! I'm trying to put this rules in a CCR1072, this router has direct connection to the internet without any restriction, but when i tried to fetch the first file got this message:

/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https
status: failed

failure: connection timeout

It seems to be connecting, but before a few minutes it stops everything, i tried to dowload this file via browser and it works, but when i run the script in the CCR1072 i got the same error. Do you have any suggestion to fix this issue?
MikroTik Soporte y Consultoría - Español / English +593 98 709 3502
https://www.safenet.ec/consultoria.html/ soporte@safenet.ec
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Mar 22, 2017 6:44 am

Unfortunately, I don't know how to help you with this. I don't see any errors in my server logs. I can only assume that you are getting ssl errors. You should be able to manually install the scripts from the first post.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Wed Mar 22, 2017 10:46 am

You issue is that the router simply didn't complete the download. Today's download is 603k. If it's getting out off, you may want to see if your ISP is trying to proxy ssl connections.
Ok yeah, maybe I was a little hasty to my conclusion.
I am able to download the file just fine from the company network, but this one customer router seems to have the issue.
[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 496KiB
       total: 603KiB
    duration: 3s

[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 336KiB
       total: 603KiB
    duration: 2s

[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 510KiB
       total: 603KiB
    duration: 2s

[deantwo@router] > /tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=RB493&version=6.35 (stable)&memory=67108864&id=A-----:############&ver=2016.7.4a (Deantwo)"
      status: finished
  downloaded: 460KiB
       total: 603KiB
    duration: 3s
Looks like it isn't downloading the full file, and MikroTik does nothing to check this it seems.
I wish my FTP was FTL.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Mar 22, 2017 3:26 pm

I don't even know where to start with that. Maybe MTU? running pppoe? ssl proxy? wrong MTU? anything different about this router over others?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
blackzero
just joined
Posts: 21
Joined: Tue Aug 09, 2011 3:40 pm

Re: Blacklist Filter update script

Thu Mar 23, 2017 11:46 am

Your two schedulers don't seem to work as the name for either is conflicting each others. Renaming it will work. Maybe you need to mention this in your first post.

Thanks for the good work.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Mar 24, 2017 2:20 am

Schedules are allowed to have the same name.

--

The server side was updated today. I was forced to make the server require the identity. The public IP and Identity are used for accounting so I can track the bandwidth and number of requires. I understand that some will object to this, and I will provide a full refund to those. (ha ha..)

The list hit 4500 active users this afternoon.

Anyone have thoughts on using the WAN MAC address instead of the identity?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Mar 24, 2017 10:29 am

Indeed the identity name is very common and besides the MAC also the ID in the VPN nanme in the quickset screen which is quite unique.

f7c4250638xxxxxx.sn.mynetname.net which contains the serial of the box reversed.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Mar 30, 2017 8:50 pm

Some interesting stats...
+-----+--------------------+
| QTY | model              |
+-----+--------------------+
| 721 | RB951G-2HnD        |
| 548 | RB2011UiAS-2HnD    |
| 374 | RB2011UiAS         |
| 309 | hAP+ac             |
| 298 | RB951Ui-2HnD       |
| 182 | RB751G-2HnD        |
| 178 | CCR1016-12G        |
| 174 | SXT+Lite5          |
| 166 | CCR1009-8G-1S-1S+  |
| 159 | RB3011UiAS         |
| 148 | hAP+lite           |
| 114 | RB850Gx2           |
| 112 | RB450G             |
| 102 | RB750GL            |
|  94 | RB750              |
|  82 | hEX                |
|  81 | CCR1036-12G-4S     |
|  78 | RB1100AHx2         |
|  68 | hAP+ac+lite        |
|  65 | RB2011UAS          |
|  64 | SXT+LTE            |
|  54 | CRS109-8G-1S-2HnD  |
|  53 | CHR                |
|  52 | x86                |
|  47 | RB493G             |
|  45 | hEX+lite           |
|  40 | mAP                |
|  40 | hAP                |
|  30 | CCR1009-8G-1S      |
|  30 | RB912UAG-2HPnD     |
|  28 | RB912UAG-5HPnD     |
|  25 | RB+Groove+5Hn      |
|  22 | mAP+lite           |
|  21 | CCR1036-8G-2S+     |
|  20 | CRS125-24G-1S      |
|  18 | RB2011UAS-2HnD     |
|  17 | RB751U-2HnD        |
|  16 | RB2011L            |
|  15 | RB2011iL           |
|  12 | RB750UP            |
|   8 | CCR1016-12S-1S+    |
|   6 | RB1100             |
|   6 | RB1200             |
|   6 | RB951-2n           |
|   5 | CRS125-24G-1S-2HnD |
|   4 | RB1100AH           |
|   4 | RB750G             |
|   4 | RB2011iLS          |
|   4 | RB433              |
|   2 | OmniTIK+5+ac       |
|   2 | CRS226-24G-2S+     |
|   2 | RB1100Hx2          |
|   2 | hEX+PoE            |
|   2 | hEX+PoE+lite       |
|   2 | %24model           |
|   2 | CCR1009-7G-1C      |
|   2 | CCR1009-7G-1C-1S+  |
|   2 | RB2011LS           |
|   1 | RB+SXT+5HnD        |
|   1 | RB433AH            |
|   1 | RB800              |
|   1 | GrooveA+52         |
|   1 | CCR1072-1G-8S+     |
|   1 | PowerBOX           |
|   1 | RB750r2            |
|   1 | SXT+Lite5+ac       |
|   1 | RB333              |
|   1 | 911+Lite5+dual     |
|   1 | RB1100AH2X         |
|   1 | RB1000             |
|   1 | RB911G-5HPnD       |
|   1 | RB+OmniTIK+U-5HnD  |
|   1 | RB493              |
|   1 | RB450              |
|   1 | BaseBox+5          |
|   1 | wAP+ac             |
|   1 | RB600              |
|   1 |                    |
+-----+--------------------+
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Mon Apr 03, 2017 3:27 am

I am a beginner at Mikrotik and my knowledge of networking is limited, for that reason my biggest thanks to people like "IntrusDave" and all who have collaborated with this magnificent work to keep our home networks safe. Thank you very much!
RB3011 UiAS (arm)
Best regards
Ricardo
 
toxicfusion
Member Candidate
Member Candidate
Posts: 138
Joined: Mon Jan 14, 2013 6:02 pm

Re: Blacklist Filter update script

Wed Apr 05, 2017 6:41 pm

I just went ahead and downloaded your script and applied to one of my MikroTiks for testing. So far so good! I'll roll this out to my client devices very soon, added security is always welcomed.

Thanks for a great contribution!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Apr 06, 2017 6:54 pm

Glad it's working out for you.
List usage jumped from 4800 to 5100 in the last two days.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Mon Apr 10, 2017 10:39 pm

Your firewall rules are great. Only DST-NAT is not working....

Last two filter rules needs to be modded like this:
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1 connection-nat-state=!dstnat
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2 connection-nat-state=!dstnat
By the way...do you know what's happened with OpenBL?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 10, 2017 10:49 pm

The rules are just examples, and should always be adjusted to suit the needs of the network.

I don't know what's going on with OpenBL. I can only assume they have either shut down, or are under DDoS.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 191
Joined: Wed May 20, 2015 10:14 pm
Location: Quito

Re: Blacklist Filter update script

Tue Apr 11, 2017 6:37 pm

Unfortunately, I don't know how to help you with this. I don't see any errors in my server logs. I can only assume that you are getting ssl errors. You should be able to manually install the scripts from the first post.
I dont know why, but it finnally works in my CCR1072, thanks for your help!
MikroTik Soporte y Consultoría - Español / English +593 98 709 3502
https://www.safenet.ec/consultoria.html/ soporte@safenet.ec
 
xlighting
just joined
Posts: 6
Joined: Wed Apr 02, 2014 6:08 pm

Re: Blacklist Filter update script

Wed Apr 12, 2017 7:33 am

Hello, Dave:
I have noticed that the rule file is now less than 100kb(<1000 filter entries), but you said it was 600kb+ in March 2017, so I'm wondering if my downloading is incomplete;
I've try downloading via different Internet connection( China/HK) and via different Routerboards(RB951G and RB750Gr3) but same result..
I've also try downloading via Chrome, but seems you've restricted downloading via Routerboard only;
(I was able to download a >5000 filter file in March with same device and same Internet connection)
I've checked the .rsc file downloaded, and not seen any “broken/ending in the middle of line”

so, is there anything I can do to further investigate where the problem is?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Apr 12, 2017 4:35 pm

OpenBL is currently offline. So right now the filters are limited to my internal sources.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Wed Apr 19, 2017 9:55 am

Thanks for the update script, it is working perfectly on my RB2011UIAS-2HnD-IN.

However, I have some trouble to get it working on my RB3011UIAS-RM. When I execute:
/system script run updateBlacklist;
I get a failure: closing connection: <400 Bad Request> 172.102.241.58:443 (4).
The script on both routers is exactly the same...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Apr 19, 2017 5:39 pm

Every time that I have seen a 400 Error, it is because the Copy/Paste didn't work. Something is the script is wrong... Maybe it has extra formatting, or maybe invalid characters. Make sure the OS that you are using supports UTF-8. Try copying and pasting the script to Notepad, and then copying and pasting into WinBox.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
rioven
just joined
Posts: 5
Joined: Mon Dec 15, 2014 5:19 am

Re: Blacklist Filter update script

Thu Apr 20, 2017 2:00 pm

Unfortunately, OpenBL gonna stop its update by end of this month
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:23 am

I don't blame them. Over the last 3 months my block list has gone from 5k entries to 30k. With most of the attacks coming from Russia and China. I'm starting to consider blocking all of Russia's IP ranges. I know that isn't good for most of the world, but my networks here in the USA are under constant attack from them.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
brianlewis
Member Candidate
Member Candidate
Posts: 131
Joined: Tue Jul 20, 2004 10:54 am
Location: Irvine, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:26 am

As great as this resource has been, in the last week it has started to block huge /16 blocks including most of Vietnam, Shopify, and many other networks that shouldn't be just added in huge /16, /19, and /24 blankets. Obviously this resource allows us to control what we want to do about these ip ranges, ie just block for specific ports or block entirely. Since we were blocking entirely the phone has been ringing off the huge by very upset customers not able to route to many areas of the world. Maybe its time to split this filter into different lists based on aggressive huge /16,/24 ranges being blocked or conservative where only specific ips or smaller /24 ranges are blocked based on their danger.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:32 am

The filters are intended to be used as incoming filters, not outgoing. If you change your rules to only block new connections coming in on the WAN interface, all should be good. I don't recommend using the list with the RAW filters.

By blocking incoming on the WAN and new connections, you prevent the attacks, but you do not block new outbound connections.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:35 am

On that note - what is really pissing me off is that big hosts like AWS and Google aren't doing anything about shutting down the attacks coming from their networks. Much of the spam is coming from AWS servers that change IP's every hour. So the only way to stop them is to block the whole subnet.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:37 am

Oh, and I ran some tests today. Filtering based on IP *ONLY* and not subnet.. the download was 112M and had over 2M entries.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Apr 22, 2017 1:50 pm

The filters are intended to be used as incoming filters, not outgoing. If you change your rules to only block new connections coming in on the WAN interface, all should be good. I don't recommend using the list with the RAW filters.

By blocking incoming on the WAN and new connections, you prevent the attacks, but you do not block new outbound connections.
I am confused by this about using RAW. Using the filters for incoming traffic in the RAW part not as efficient?

For outgoing I use a DNS filter and out of band port filtering for new connections in Mangle.

It is really bad out there and have lots of connections wanting to deliver mail which I don't want. It in waves since a few months and sometimes there is it quiet for days and then it starts again.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon Apr 24, 2017 9:46 am

Every time that I have seen a 400 Error, it is because the Copy/Paste didn't work. Something is the script is wrong... Maybe it has extra formatting, or maybe invalid characters. Make sure the OS that you are using supports UTF-8. Try copying and pasting the script to Notepad, and then copying and pasting into WinBox.
Thanks for your reply, I will try pasting it via Notepad later this week.

Regarding to the shutdown of OpenBL, is there any other alternative for an updated blacklist?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 24, 2017 4:35 pm

I don't know. I stopped using OpenBL a while back.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon Apr 24, 2017 10:13 pm

I tried to paste the code in Notepad first, but still I get the 400 bad request error...
Are you sure the RB3011UIAS-RM is supported?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Apr 24, 2017 10:28 pm

That is the same unit I use for writing my scripts. I have just over 500 of them pulling the list every morning. The error you posted is almost always a simple format or encoding error.
Last edited by IntrusDave on Wed Apr 26, 2017 12:00 am, edited 1 time in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue Apr 25, 2017 4:54 pm

hi,
Im using this rule for dnymic blacklist
chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 
chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 
chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 
chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist 
is there any way to keep some ip's without block I mean some Exceptions?

Thanks
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Apr 25, 2017 6:53 pm

Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Wed Apr 26, 2017 8:43 am

Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
should it like this ? and what about order of the rules ? is that correct ?
8 chain=prerouting action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

 9 chain=output action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

10;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 

11 ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 

12 ;;; BlackList
      chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 

13;;; BlackList
      chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist 
Thanks
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Wed Apr 26, 2017 9:41 pm

That is the same unit I use for writing my scripts. I have just over 500 of them pulling the list every morning. The error you posted is almost always a simple format or encoding error.
Can you send me the script you are using?
I even get the error when I use your automated installer script...
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Apr 27, 2017 3:00 am

Try downloading directly from here: https://mikrotikfilters.com/updateBlacklist.rsc
Unfortunately, I don't have a router that gets this error, so I really can't troubleshoot it.

If one of you want to give me access to a router that is having a problem with the script, I can try and figure out what the problem is.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon May 01, 2017 3:01 pm

Dave,

Can you give me an update URL without or with preset variables?
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local scriptVer   2016.7.4a
"https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$version&memory=$memory&id=$uname&ver=$scriptVer";
I think it is going wrong with the URL containing (maybe unknown) variables.
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue May 02, 2017 11:14 am

Can you give me an update URL without or with preset variables?
...
I think it is going wrong with the URL containing (maybe unknown) variables.
Like this?
/tool fetch mode=https dst-path="/dynamic.rsc" url="https://mikrotikfilters.com/download.php\?get=dynamic&model=750&version=6.39&memory=33554432&id=mk13139&ver=DeanHelp";
Last edited by Deantwo on Fri Aug 10, 2018 3:23 pm, edited 5 times in total.
I wish my FTP was FTL.
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue May 02, 2017 11:46 am

Yes, You can create an address list with addresses that you never want blocked, then add an accept rule above the drop rules.
should it like this ? and what about order of the rules ? is that correct ?
8 chain=prerouting action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

 9 chain=output action=accept log=no log-prefix="" src-address-list=Exceptions dst-address-list=Exceptions 

10;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 

11 ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 

12 ;;; BlackList
      chain=output action=drop log=yes log-prefix="" src-address-list=dynamicBlacklist 

13;;; BlackList
      chain=output action=drop log=yes log-prefix="" dst-address-list=dynamicBlacklist 
Thanks
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Tue May 02, 2017 11:46 am

Can you give me an update URL without or with preset variables?
...
I think it is going wrong with the URL containing (maybe unknown) variables.
/tool fetch mode=https dst-path="/dynamic.rsc" \
   url="https://mikrotikfilters.com/download.php\?get=dynamic&model=750&version=6.39&memory=33554432&id=mk13139&ver=DeanHelp";
Like that?
Yes exactly!
I got it working last night using the variables of my RB2011UAS-2HnD, hardcoding them into the update URL of the RB3011UIAS-RM:
/tool fetch mode=https dst-path="/dynamic.rsc" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=RB2011UAS-2HnD&version=6.38.5+(stable)&memory=128.0MiB&id=MikroTik+router&ver=2016.7.4a";
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Tue May 02, 2017 5:06 pm

I think it is going wrong with the URL containing (maybe unknown) variables.
Out of curiosity, what does your router say to the following if you paste it in the terminal?
:put [/system resource get board-name]
:put [/system resource get version]
:put [/system resource get total-memory]
:put [/system identity get name]
Last edited by Deantwo on Fri Aug 10, 2018 3:20 pm, edited 1 time in total.
I wish my FTP was FTL.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 03, 2017 12:13 am

Give this a try...
# Import Intrus Managed Filter Lists
# ©2016-2017 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
# :delay 10

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2017.5.2b

:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}


:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Thu May 04, 2017 8:05 am

[quote="mhyll"]Your firewall rules are great. Only DST-NAT is not working....

Last two filter rules needs to be modded like this:
add action=drop chain=forward comment="Drop everything else on WAN1" in-interface=wan1 connection-nat-state=!dstnat
add action=drop chain=forward comment="Drop everything else on WAN2" in-interface=wan2 connection-nat-state=!dstnat

Thanks to your observation, I was able to make my security cameras visible from outside my house; However for the "raw" rule in prerouting the "connection-nat-state =! Dstnat" is not possible, and I have it disabled.
You know how I could make this rule work without blocking the cameras, Thanks!

Resolved!!!!

I had to put the rules of accepting the list of white IPs from first into "RAW", and everything was fine now. Thanks!!!!!
You do not have the required permissions to view the files attached to this post.
RB3011 UiAS (arm)
Best regards
Ricardo
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Mon May 08, 2017 9:08 am

Hi,

I need help for this script. Im using this script but some times my wan adress comes to the list OR my ipblock. i create new rule to accept my ip but traffic down when i try to use. here is my rules, can some one help me to solve this issue ?

Thanks.
8  ;;; Exceptions
      chain=prerouting action=accept log=no log-prefix="" src-address-list=exceptions 

 9  ;;; Exceptions
      chain=prerouting action=accept log=no log-prefix="" dst-address-list=exceptions 

10  ;;; Exceptions
      chain=output action=accept log=no log-prefix="" src-address-list=exceptions 

11  ;;; Exceptions
      chain=output action=accept log=no log-prefix="" dst-address-list=exceptions 

12    ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" src-address-list=dynamicBlacklist 

13    ;;; BlackList
      chain=prerouting action=drop log=no log-prefix="" dst-address-list=dynamicBlacklist 

14    ;;; BlackList
      chain=output action=drop log=no log-prefix="blcklist src" src-address-list=dynamicBlacklist 

15    ;;; BlackList
      chain=output action=drop log=no log-prefix="blcklist dst" dst-address-list=dynamicBlacklist 

16    chain=prerouting action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=tcp src-address-list=!secure 

17    chain=prerouting action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=udp src-address-list=!secure 

18    chain=output action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=tcp src-address-list=!secure 

19    chain=output action=drop dst-port=22,23,8290,8291 log=no log-prefix="" protocol=udp src-address-list=!secure 

20    chain=prerouting action=drop in-interface=wan dst-port=53 log=no log-prefix="" protocol=tcp 

21    chain=prerouting action=drop in-interface=wan dst-port=53 log=no log-prefix="" protocol=udp 
Last edited by amt on Mon May 08, 2017 3:49 pm, edited 1 time in total.
 
mk13139
just joined
Posts: 8
Joined: Mon Dec 30, 2013 3:32 am

Re: Blacklist Filter update script

Mon May 08, 2017 2:36 pm

Give this a try...
# Import Intrus Managed Filter Lists
# ©2016-2017 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
# :delay 10

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2017.5.2b

:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}


:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/dynamic.rsc

:log warning "Removing temp file...";
/file remove dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
Thanks! I will check it out when I'm on location.
 
User avatar
Squidblacklist
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Wed Jun 26, 2013 11:06 am

Re: Blacklist Filter update script

Mon May 15, 2017 1:47 pm

it's very possible to do that, but I would need to see what the impact on the routers would be. I'm not a big fan of the built-in DNS as it is and I'm not sure how well it would hold up with several thousand hostnames added to it.
Actually, Im glad to inform you today that the current release has added a new patch for greatly improved import speed for the importing of static dns entries, one thing you will notice is that, the cpu usage is no longer at 100% during import and the import process is much faster. I will be doing some benchmarks of RouterOS before and after the patch to demonstrate the difference, and it is a remarkable improvement indeed.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon May 15, 2017 6:24 pm

My list will not be moving to DNS. It over complicates the process and provides little if any advantages.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
plisken
Forum Guru
Forum Guru
Posts: 2425
Joined: Sun May 15, 2011 12:24 am
Location: Belgium
Contact:

Re: Blacklist Filter update script

Fri May 19, 2017 9:45 am

What is the command to write the blacklist to a usb-stick?
Thanks
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri May 19, 2017 6:38 pm

The list is stored in memory while active.
If you need to use a flash drive for the update, just add the path of the usb drive to the path of the fetch and import lines.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Jacka
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Jan 13, 2011 11:34 am

Re: Blacklist Filter update script

Mon May 22, 2017 10:43 am

Hi,

First of all thank you for this great script. I have a few questions:
1. Why there are 2 schedules? And if there are 2 it can't have the same name as in your example.
 /system scheduler
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=05:00:0
/system scheduler
add interval=00:00:00 name=UpdateBlackList on-event="/system script run blacklistUpdate" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=00:00:0
2. What kind of chain is this "Attacks" ? It should be input or forward chain, am I right ?
/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Mon May 22, 2017 1:44 pm

1. Why there are 2 schedules? And if there are 2 it can't have the same name as in your example.
 /system scheduler
add interval=1d name=UpdateBlackList on-event="/system script run updateBlacklist" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=05:00:0
/system scheduler
add interval=00:00:00 name=UpdateBlackList on-event="/system script run blacklistUpdate" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive \
    start-date=jan/01/1970 start-time=00:00:0
Two schedulers can have the same name, it is weird to have though.

Also not sure the second scheduler is totally correct. At least it can be written better.
See: viewtopic.php?f=9&t=98804&start=150#p587752
2. What kind of chain is this "Attacks" ? It should be input or forward chain, am I right ?
/ip firewall filter
add action=drop chain=Attacks comment="Drop connections FROM blacklisted hosts" src-address-list=dynamicBlacklist
add action=drop chain=Attacks comment="Drop connections TO blacklisted hosts" dst-address-list=dynamicBlacklist
The "Attacks" chain is a custom chain, take a look at the jump rules further down.
/ip firewall filter
#...
add action=jump chain=input comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
#...
add action=jump chain=forward comment="Check for bad stuff in \"Attack\" chain" jump-target=Attacks
You can read more here: https://wiki.mikrotik.com/wiki/Manual:I ... all/Filter

The firewall shown in the first post is just an example, you might want you edit it for your own use or make your own.
Last edited by Deantwo on Fri Aug 10, 2018 3:24 pm, edited 1 time in total.
I wish my FTP was FTL.
 
Jacka
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Jan 13, 2011 11:34 am

Re: Blacklist Filter update script

Mon May 22, 2017 3:16 pm

Thanks for explanation.
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue May 23, 2017 8:06 am

can some one share firewall rules for this script ? which is correct ? using raw or filter for this script. also sometimes my puplic ip's come to the list. how can i remove them ? and how can i create expection rule for them ?

Thanks.
 
dmcosta
just joined
Posts: 8
Joined: Wed Oct 30, 2013 3:47 pm

Re: Blacklist Filter update script

Wed May 24, 2017 7:58 pm

Hello Dave!

Works perfectly on hAp AC lite , RB951G-2Hnd.

Thank you very much for this! Great work!!

cheers
 
b3h3m07h
newbie
Posts: 30
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Mon May 29, 2017 6:43 am

Here is the script i use to save and execute the blacklist (dynamic.rsc) from a usb drive (named usb1)

/system script add name=updateBlacklist-usb1 owner=admin policy=read,write,test source="# Import Intrus Managed Filter Lists\r\n# (C)2016 David Joyce, Intrus Technologies\r\n\r\n:log warning \"Blacklist update in 30 seconds\";\r\n# :delay 10\r\n\r\n:local model \t[/system resource get board-name]\r\n:local version\t[/system resource get version]\r\n:local memory\t[/system resource get total-memory]\r\n:local uname\t[/system identity get name]\r\n:local scriptVer 2017.5.2a\r\n\r\n:local name \"\"\r\n:local ver \"\"\r\n\r\n:for i from=0 to=([:len \$uname] - 1) do={ \r\n :local char [:pick \$uname \$i]\r\n :if (\$char = \" \") do={ :set \$char \"%20\" }\r\n :set name (\$name . \$char)\r\n}\r\n\r\n:for i from=0 to=([:len \$version] - 1) do={ \r\n :local char [:pick \$version \$i]\r\n :if (\$char = \" \") do={\r\n :set \$char \"%20\"\r\n }\r\n :set ver (\$ver . \$char)\r\n}\r\n\r\n\r\n:log warning \"Downloading current Blacklist for this model\";\r\n/tool fetch mode=https dst-path=\"/usb1/dynamic.rsc\" \\\r\n\turl=\"https://mikrotikfilters.com/download.ph ... \n\r\n:log warning \"Disabling info logging...\";\r\n/system logging disable 0\r\n\r\n:log warning \"Removing expiring address-list entries...\";\r\n:foreach i in=[/ip firewall address-list find ] \\\r\n\tdo={ :if ( [/ip firewall address-list get \$i list] = \"dynamicBlacklist\" ) \\\r\n do={ /ip firewall address-list remove \$i } }\r\n\r\n:log warning \"Importing current Blacklist...\";\r\n/import file-name=/usb1/dynamic.rsc\r\n\r\n:log warning \"Removing temp file...\";\r\n/file remove usb1/dynamic.rsc\r\n\r\n:log warning \"Blacklist Update Complete.\";\r\n/system logging enable 0"

so far, so good.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1717
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Mon May 29, 2017 7:47 am

Could you please edit your post and use "Code" tag to paste script content once again. It is hard to read it now.
Real admins use real keyboards.
 
b3h3m07h
newbie
Posts: 30
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Mon May 29, 2017 8:58 am

here you go :-)
# Import Intrus Managed Filter Lists
# (C)2016 David Joyce, Intrus Technologies

:log warning "Blacklist update in 30 seconds";
# :delay 10

:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local scriptVer   2017.5.2a

:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}


:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/usb1/dynamic.rsc" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer";

:log warning "Disabling info logging...";
/system logging disable 0

:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

:log warning "Importing current Blacklist...";
/import file-name=/usb1/dynamic.rsc

:log warning "Removing temp file...";
/file remove usb1/dynamic.rsc

:log warning "Blacklist Update Complete.";
/system logging enable 0
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 12:36 am

I've updated the script with support for USB Flash as well as the new RB1100AHx4 with internal storage.
I has also reworked the backend and script for more accurate accounting. Please update your scripts.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk1/dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30c

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
Also - The server is now blocking routers that excessively download the list. (I have several people that are trying to update every minute.)
Last edited by IntrusDave on Wed May 31, 2017 8:03 am, edited 1 time in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
b3h3m07h
newbie
Posts: 30
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Wed May 31, 2017 5:03 am

Just made a few changes to the script as it didn't seem to delete the blacklist at the end
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "usb1/"
:global datafile "dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30b

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/$datapath$datafile" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="/$datapath$datafile"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove "$datapath$datafile"

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
 
cashwu
just joined
Posts: 4
Joined: Mon Sep 12, 2016 5:42 pm

Re: Blacklist Filter update script

Wed May 31, 2017 6:22 am

Just tried it, the results stopped here ...
Set:
:global datapath "disk-8G/"
:global datafile "dynamic.rsc"
未命名.png
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 6:37 am

Try this
:global datapath "/disk-8G/"
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
b3h3m07h
newbie
Posts: 30
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Wed May 31, 2017 7:19 am

try this, worked fine on my rb2011 and usb drive
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk-8G/"
:global datafile "dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30b

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="/$datapath$datafile" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="/$datapath$datafile"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove "$datapath$datafile"

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
These lines were also changed

/tool fetch mode=https dst-path="/$datapath$datafile" \
url="https://mikrotikfilters.com/download.ph ... id=$softid";

/import file-name="/$datapath$datafile"

/file remove "$datapath$datafile"
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 7:41 am

Yup, clearly a problem with the remove. I can't seem to get it to accept a variable
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 8:05 am

Okay, I've updated the script again. It didn't like having the path and filename separate.

# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk1/dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model 	[/system resource get board-name]
:local version	[/system resource get version]
:local memory	[/system resource get total-memory]
:local uname	[/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30c

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
	url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
	do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
cashwu
just joined
Posts: 4
Joined: Mon Sep 12, 2016 5:42 pm

Re: Blacklist Filter update script

Wed May 31, 2017 8:16 am

Hello Dave & b3h3m07h, thank you for your reply.

The latest version works fine.

And then your reply, to understand the difference between the two ways.

Once again thank you.
 
b3h3m07h
newbie
Posts: 30
Joined: Sat Dec 28, 2013 3:06 am

Re: Blacklist Filter update script

Wed May 31, 2017 8:19 am

Nice work. All good here.
 
dmcosta
just joined
Posts: 8
Joined: Wed Oct 30, 2013 3:47 pm

Re: Blacklist Filter update script

Wed May 31, 2017 4:24 pm

Thanks Dave for the update! Great work!
 
aboiles
newbie
Posts: 47
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 5:09 pm

Hello Dave,
I am now getting an error when I run the script-

url="https://mikrotikfilters.com/download.ph ... model&vers
ion=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";
status: failed

failure: closing connection: <404 Not Found> 172.102.241.58:443 (4)

The script worked fine last night, but is now failing with the same error on the four routers it's running on.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 5:12 pm

Your URL is wrong.
Note the ? between "download.php" and "get"
url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid"
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
aboiles
newbie
Posts: 47
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 5:36 pm

Hello Dave,
The script has the ?, when pasted in terminal it disappears.
The log only has an entry of-
script error: expected command name (line 1 column 1)
The downloaded dynamic.rsc only has one line-
All fields are required. Please update your script.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 5:39 pm

That would mean that you need the current script. It's available in the first post.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
aboiles
newbie
Posts: 47
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 6:16 pm

tried both the auto installer - script ver 2017.5.2a
and the code ver 2017.5.30c.
Am still getting the All fields are required. Please update your script. dynamic.rsc
 
aboiles
newbie
Posts: 47
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 8:14 pm

slightly modified the script by removing the extra spaces in the local info section and now have it running on a 2011UiAS-2HnD, 951G-2HnD and a CRS125-24G-1S-2HnD.
still fails on a CHR with the same error-All fields are required. Please update your script. dynamic.rsc
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed May 31, 2017 8:27 pm

Sorry man. More than 500 routers already updated and working with the new script. You are having copy/paste issues. I can't fix that for you.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
aboiles
newbie
Posts: 47
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Wed May 31, 2017 8:47 pm

Don't know Dave,
I think it may have something to do with the software-id.
I'm getting a blank for software-id from the chr's.

[admin@router] > :put [/system resource get board-name]
CHR
[admin@router] > :put [/system resource get version]
6.40rc15 (testing)
[admin@router] > :put [/system resource get total-memory]
2071535616
[admin@router] > :put [/system identity get name]
router
[admin@router] > :put [/system license get software-id]

[admin@router] >
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Jun 01, 2017 5:31 am

I'm having issues too, started out with the autoinstaller script in your first post, it always leads with
[admin@TaylorMikrotik] >> /import updateBlacklist.rsc;                                                 
syntax error (line 62 column 11)
I edited it and put start-time=startup on the same line as the last add for scheduler.

That worked but then it complains about that name already existing, which is from the double schedules? not sure why there are two named the same.

After editing this tho, I get what others have gotten.
All fields are required. Please update your script.
i have not copy or pasted anything outside of downloading the script from winbox to my pc, open in notepad++ with it in UNIX lineending mode, and put that part on the right line, then re uploaded to the tik.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 6:48 am

I'm guessing that everyone with issues are running CHR. I've found the problem and I'm working on a fix right now. I'll post the update in about an hour.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Jun 01, 2017 6:56 am

I am on a RB951Ui-2HnD
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 6:59 am

I am on a RB951Ui-2HnD
can you post the /system license print ?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 7:40 am

I've updated the script to deal with the CHR using system-id instead of software-id. Annoying that they are different...

I've tested on the following units with no failures.
CCR1009-7G-1C-1S+
CCR1009-8G-1S-1S+
CCR1016-12G
CCR1036-12G-4S
CHR
CRS109-8G-1S-2HnD
CRS125-24G-1S
CRS125-24G-1S-2HnD
hAP+ac
hAP+ac+lite
hEX
RB2011UAS-2HnD
RB2011UiAS
RB2011UiAS-2HnD
RB3011UiAS
RB450G
RB951G-2HnD
RB951Ui-2HnD
x86
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 01, 2017 7:42 am


syntax error (line 62 column 11)[/code]
I found the line 62 error and corrected it. delete the items you have, and reinstall. it should be good to go.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Jun 01, 2017 8:03 am

Seems the CHR changes, fixed mine too... fyi the output was
[admin@TaylorMikrotik] >> /system license print     
  software-id: 15LP-6RVD
       nlevel: 4
     features: 
 
aboiles
newbie
Posts: 47
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter update script

Thu Jun 01, 2017 8:27 am

Thanks Dave,
Script works great on the CHR now!
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Thu Jun 01, 2017 10:06 am

Hi,
I had this erorr All fields are required. Please update your script
here log;
10:05:19 script,warning Downloading current Blacklist for this model
10:05:20 info fetch: file "dynamic.rsc" downloaded
10:05:20 script,warning Disabling info logging...
10:05:20 script,warning Removing expiring address-list entries...
10:05:20 script,warning Importing current Blacklist...
10:05:20 script,error script error: expected command name (line 1 column 1)

I update script with this;
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage

:global datapath "disk1/dynamic.rsc"

###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:log warning "Blacklist update in 10 seconds";
:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]
:local scriptVer   2017.5.30c

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }

##### Import the downloaded blacklist
:log warning "Importing current Blacklist...";
/import file-name="$datapath"

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:log warning "Blacklist Update Complete.";
/system logging enable 0
and now working.
thanks
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:02 am

Glad it's working for everyone now. Stats are MUCH more accurate now. The server was starting to block devices behind NAT routers because it thought some were downloading hundreds of times per hour. Now it sees each as a separate device.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:10 am

Awesome! Thanks for still doing this. Now that you got more stats, you should create some public pages cause i love me some random statistics!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:14 am

Awesome! Thanks for still doing this. Now that you got more stats, you should create some public pages cause i love me some random statistics!
I was just starting on a page that shows each type and number of routers that pulls the list.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 02, 2017 8:58 am

I've cleared all my starts and started fresh. Here is a quick and dirty stats page on the hardware accessing the list.

https://mikrotikfilters.com/blstats.php
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1717
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jun 02, 2017 4:18 pm

Data taken from 28 days of router uptime

blackmail is my list composed from addresses dropped with greylisting

98 395 - total started SMTP sessions
8362 - unique smtp src addresses
7 515 - dropped by my blackmail list
70 596 - dropped by Dave's list
bm.PNG
You do not have the required permissions to view the files attached to this post.
Real admins use real keyboards.
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jun 03, 2017 11:10 am

Many thanks for all your work and I was following this thread with great interest and checked this morning if I needed to update my script. That was the case and since three days I had a error on line one.

So I downloaded the latest script and imported it after removing the running script. The are some things I had to change: user djoyce --> admin to get the line back in my log where the dynamic.rsc has been downloaded memory info fetch: file "dynamic.rsc" downloaded. I set the start delay time to 30 seconds because I have a PPPoE connection that takes a bit longer to come up after reboot.

Default the location of the dynamic.rsc is now disk1 and that is ok by me because I have an SD card in my RB750Gr3 but I can change that to flash (mirrored in RAM) again if I like.

I also noticed that on importing updateBlacklist.rsc to the script that I got: /import updateBlacklist.rsc; failure: item with this name already exists despite it did not exist until after the import.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
pkrexer
just joined
Posts: 19
Joined: Sat May 21, 2016 4:39 pm

Re: Blacklist Filter update script

Mon Jun 05, 2017 3:56 am

Not sure why its not working all of a sudden. I updated the script a few days ago and was working as of yesterday... Now when the script runs, it says its downloading the blacklist but nothing else happens.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 05, 2017 3:58 am

Not sure why its not working all of a sudden. I updated the script a few days ago and was working as of yesterday... Now when the script runs, it says its downloading the blacklist but nothing else happens.
What are the last two octets of the public IP?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
pkrexer
just joined
Posts: 19
Joined: Sat May 21, 2016 4:39 pm

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:05 am

11.204
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:12 am

Fixed. Sorry about that. typo in the code.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
pkrexer
just joined
Posts: 19
Joined: Sat May 21, 2016 4:39 pm

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:16 am

np! Thanks for the quick fix... Appreciate all the work you do!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 05, 2017 4:17 am

No problem at all. I enjoy it.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 07, 2017 1:30 pm

I noticed today when I started Firefox that I were getting hits on the blacklist. I followed the IP and found that it lead to hackademix.net and secure.informaction.com and looking on the site it was probably an plug-in was generating the hits and that was No-script. I use this plug-in for years and I allow or disallow the default running of scripts filtered on the domain the are served by.

Plugin homepage: https://noscript.net/

Name: secure.informaction.com
Addresses: 69.195.158.194
69.195.158.198
69.195.158.197
69.195.158.195
69.195.158.196

I understand how the blacklist is build and that it based on bad traffic and if there is a problem of a domain being misused then I can contact them to ask to look if they are hacked in any way?

Found it and I don't know why I did not see it before: the block is: 69.195.158.0/24 in the dynamicblacklist
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 08, 2017 3:17 am

I've updated the statistics page today. It now normalizes the memory and shows the percentage of each category
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 08, 2017 3:21 am

I noticed today when I started Firefox that I were getting hits on the blacklist. I followed the IP and found that it lead to hackademix.net and secure.informaction.com and looking on the site it was probably an plug-in was generating the hits and that was No-script. I use this plug-in for years and I allow or disallow the default running of scripts filtered on the domain the are served by.

Plugin homepage: https://noscript.net/

Name: secure.informaction.com
Addresses: 69.195.158.194
69.195.158.198
69.195.158.197
69.195.158.195
69.195.158.196

I understand how the blacklist is build and that it based on bad traffic and if there is a problem of a domain being misused then I can contact them to ask to look if they are hacked in any way?

Found it and I don't know why I did not see it before: the block is: 69.195.158.0/24 in the dynamicblacklist
I'm not sure what you are asking here. You are always welcome to contact a site and ask them to fix any issues. The subnet will be removed from the list automatically once whatever issue they were having is fixed. Many times it's that they are hosting a botnet that they do not even know about. Other times it may be that they are serving viruses in ads. AWS and Google Compute have both been blocked several times because they refuse to take down a virtual host that is being used to attack other networks.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
eddieb
Member Candidate
Member Candidate
Posts: 141
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Thu Jun 08, 2017 8:37 pm

Hi Dave,

I use your dynamic blacklist and it suits well.
For some reason a subnet from which I really need to use 1 address from appeared in one of the lists loaded into dynamicBlacklist ...
It would be a great help if I could whitelist a subnet inside the script somehow ...

Keep up the great work

Eddie
Running 6.45.7 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 08, 2017 11:00 pm

Whitelisting is accomplished by creating a new address-list and a new filter rule.

1) Create an address list - say.. "Whitelist" and add the IP addresses that you need never be blocked.
2) create a new filter "Accept" rule, using the src-address-list you created.
3) place the new Whitelist Accept rule ABOVE the blacklist Drop rule.

There is no need to modify the script, and this can not be done on the server side.
Please keep in mind that it's always better to understand why the IP/Subnet ended up on the blacklist and attempt to get that corrected first. I have seen several networks penetrated because an admin whitelisted an address that was serving malware, instead of contacting that site/service and getting the issue resolved.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
eddieb
Member Candidate
Member Candidate
Posts: 141
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Fri Jun 09, 2017 9:11 am

Hi Dave,

thanks for your response. It was a bit complex but I did manage to add the subnet in a whitelist and that works for now...
I am aware of the problems that might cause such a whitelist. In this case the subnet is from a local provider with many customers fighting spam ...
sometimes one of them gets blacklisted for that reason and sometimes the entire subnet is.
To do make sure that 2 of the servers within my responsibility are not causing troubles I need to have access to them.
For now I whitelisted those 2 and that does the job.

regards,

Eddie
Running 6.45.7 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
User avatar
leemans
Frequent Visitor
Frequent Visitor
Posts: 60
Joined: Thu Apr 07, 2005 12:55 am
Location: Belgium
Contact:

Re: Blacklist Filter update script

Fri Jun 09, 2017 5:51 pm

Hi Dave,

Dear,
It's not working on my RB600.
It used to work for a long time...
Any idea how come?

Thanks,
Patrick
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 09, 2017 6:01 pm

The script was updated last week to work with the new backend servers. You can find the update in the first post of this thread
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jun 10, 2017 5:16 pm

Thanks to someone setting up 50 routers to download every 2 minutes, the server is now blocking any router that downloads more than 4 times in a 24 hour period.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jun 10, 2017 9:33 pm

Is the blockage permanent or will it be lifted after a certain time? I can imagine that one that is testing has to reboot a few times in a few hours. So when testing one should switch off the startup reload until all is stable again.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jun 10, 2017 9:43 pm

SHouldn't be an issue for most. The server will flag routers that get excessive and throttle them to 4 download in a 24 hour period.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 11, 2017 12:55 am

And if the devil was at work with my Mikrotik and made it crashed and luckily I managed to switch off the startupscript. I had already had three strikes so I also should disable the normal update, for the time being.

Maybe it is possible to keep the file on the disk (when not using flash) and delete it on the next regular reload of 24 hours. After the first import it would have to be renamed with the time, of the first import in the name.

The script looks on a restart or regular reload if the file is older dan 23 hours and then it would get a new one. If the file is younger than 23hours the script will reload the file form the disk.

You can then still throttle addresses that reload more than three times with a blank router. If they have to setup routers then they also should copy the current file to the disk on each router.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 11, 2017 10:55 am

AAAARGH lost some sleep by trying to find out the way how to convert date+time so that I could subtract those and have the difference in time. No I did not manage but manage to go to sleep after staying up much to long.

In the morning my mind started to seek for a solution and I had different ideas but none of the would solve this. Then I got a great idea to just make a different script just for only the start-up. All pieces fell in place and no calculation and string delidding needed and just let nature do its work and follow the natural flow.

The main script updateBlacklist is changed so that dynamic.rsc file is not deleted after importing. The new startup startupBlacklist is the current updateBlacklist stripped of all download and statistical parts.



In updateBlacklist I commented out the removal of the dynamic.rsc file after importing and it will overwritten by the new dynamic.rsc file when the daily update is run. This is the changed code part form updateBlacklist



Hope that you like this adaptation and so also give your server bit of rest because restarted Mikrotik devices will not bother it when just restarting and only knock on the door for the real daily update.

So I am going now to eat my breakfast and enjoy my Sunday which is also today a sunny day.
Last edited by msatter on Fri Jun 16, 2017 12:50 am, edited 3 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
nico599
just joined
Posts: 2
Joined: Mon Jun 12, 2017 11:42 am

Re: Blacklist Filter update script

Mon Jun 12, 2017 12:12 pm

Hi DAVE
yesterday , i change new script ,
it's working in my RB450G & RB750GL,
but in my CCR1009-8G-1S-1S+ ......,
just show Message "Downloading current Blacklist for this model",
but can't download anything...
i use the same script,.......

How can i to deal with this problem?

Sorry,English not my mother Language...
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Mon Jun 12, 2017 12:49 pm

Hello Dave,
The script has the ?, when pasted in terminal it disappears.
The log only has an entry of-
script error: expected command name (line 1 column 1)
The downloaded dynamic.rsc only has one line-
All fields are required. Please update your script.
That would mean that you need the current script. It's available in the first post.
Dave you could just escape the "?". That would allow it to be run in the terminal without issue, and it will make no difference for non-terminal running.
I mentioned it before, here: viewtopic.php?f=9&t=98804&start=150#p587708
"...\?..."
Last edited by Deantwo on Fri Aug 10, 2018 3:25 pm, edited 2 times in total.
I wish my FTP was FTL.
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jun 12, 2017 1:10 pm

A bit of saving on traffic you could save about 20% of traffic by not adding the "comment" part on every dynamicblacklist line.

I don't know if RouterOS can handle deflated traffic when downloading. However there is xz/LZMA used when a firmware update is applied. So if on saving a file with the .gz extension then it could be automatically be extracted and then your dynamic.rsc.gz would 20 times smaller and just 50KB instead of 970KB.

Update: all the .NPK files are zipped and extracted in some way when installed. Looking into the system.npk I find the program "unexpak" but I can't see what it is doing. When I look in \lib\ I see the library libz.so and if I am not wrong that is a compress/decompress code.

The only thing I found Mikrotik talking about compression stated that due to limitations of the size of the flash not allowing a compression tool to the users.

So maybe Mikrotik could give us the option to export compressed and then to normal extension like .RSC and .BACKUP add .GZ and automatically decompress files with .GZ when read.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
reb00t
just joined
Posts: 2
Joined: Wed Jul 22, 2015 10:24 pm

Re: Blacklist Filter update script

Wed Jun 14, 2017 6:51 pm

My updates stopped working so I went to investigate and found when running the update script (v2017.5.31f) from the command line:
[admin@redacted_name.com] > /system script run updateBlacklist;                                                  
  status: failed

failure: cannot open file
I've installed the script via the download method described in the first post so I don't believe it's a copy/paste issue. Maybe I accessed the download more than four times during testing one day? Are there any other reasons to get that failure message?

Here's my basic info:
[admin@redacted_name.com] /system resource> print
             uptime: 4d4h2m55s
            version: 6.39.2 (stable)
         build-time: Mar/09/2017 11:32:49
        free-memory: 1712.6MiB
       total-memory: 1956.2MiB
                cpu: tilegx
          cpu-count: 9
      cpu-frequency: 1200MHz
           cpu-load: 0%
     free-hdd-space: 78.5MiB
    total-hdd-space: 128.0MiB
  architecture-name: tile
         board-name: CCR1009-8G-1S-1S+
           platform: MikroTik
[admin@redacted_name.com] /system resource> /system license print            
  software-id: 8RW2-IFMS
       nlevel: 6
     features: 
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 14, 2017 10:15 pm

try a copy/paste from the first post. Not sure what the issue is, the server isn't reporting any issues.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
reb00t
just joined
Posts: 2
Joined: Wed Jul 22, 2015 10:24 pm

Re: Blacklist Filter update script

Thu Jun 15, 2017 3:35 pm

Doing the copy and paste from post #1 worked. Still not sure why it stopped working. Thank you!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 15, 2017 11:57 pm

Doing the copy and paste from post #1 worked. Still not sure why it stopped working. Thank you!
Sweet, glad it fixed it for you.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Wed Jun 21, 2017 10:18 am

Thank you for this, David!

Curious why you use a loop:
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }
instead of
/ip firewall address-list remove [find list="dynamicBlacklist"]
On my system, the loop takes 88 seconds while the 1-liner takes 32 seconds
How does your blacklist compare with the one being provided by squidblacklist.org (which just combines spamhaus drop, edrop, dshield, malc0de, blocklist.de )?
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 21, 2017 10:36 am

That is indeed an easier and faster way. Dave wrote about that earlier about using those lists: viewtopic.php?f=9&t=98804&p=545381&hilit=Drop#p545381

I tried the change in code and the removal of the addresses went from 37 seconds down to 20 seconds and the total time is now 48 seconds and before it was 66 seconds. So the save time when using almost 20.000 IP addresses is around 17 seconds.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 21, 2017 11:18 am

And some more info on how to reduce the traffic if RouterOS is supporting gzip/deflate: https://www.scalescale.com/tips/nginx/h ... mpression/

When I now use your site I get no get gzip on the application/octet-stream:
root@search:~# curl --header "Accept-Encoding: gzip,deflate,sdch" -I https://mikrotikfilters.com/updateBlacklist.rsc
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 21 Jun 2017 08:11:29 GMT
Content-Type: application/octet-stream
Content-Length: 4141
Last-Modified: Thu, 01 Jun 2017 04:22:22 GMT
Connection: keep-alive
Keep-Alive: timeout=2
Accept-Ranges: bytes
When compression is active then the saving would be 95.7% and your transfer goes from 1.8MB to 78KB:
................./dynamic.txt is Compressed

Uncompressed Page Size: 1817.7 KB
Compressed Page Size: 77.8 KB
Savings: 95.7%
I see different data when downloading html or the dynamic.rsc when I test it on my own server:

Darn the whole bit below is obsolete because the things I though I could deduct, is bases on not cleared characters by RouterOS. The result is written on the same line as the line shown during transfer "-- [Q quit|D dump|C-z pause]" so I was mislead by what it seemed to state and I was looking for........GRRRRRRRRRRRRRRRR

[admin@MikroTik] > /tool fetch mode=http url=https://xxxx.xx/index.html
       status: finished
  downloaded: 0KiBC-z pause]
       total: 0KiB
    duration: 1s
[admin@MikroTik] > /tool fetch mode=http url=https://xxxx.xx/dynamic.rsc
      status: finished
  downloaded: 1817KiB pause]
       total: 1817KiB
    duration: 1s
I think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC

And a PDF also get -z but nu C:
[admin@MikroTik] > /tool fetch mode=http url=https://xxxxx.xx/files/xxxxxxx.pdf
      status: finished
  downloaded: 71KiB-z pause]
       total: 71KiB
    duration: 1s
[/i]
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 21, 2017 5:57 pm

The loop is to deal with older versions of RouterOS that would only remove the first item it found when using Find.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 21, 2017 7:04 pm

The server does compress the content.... As seen by this compression test.
You do not have the required permissions to view the files attached to this post.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 22, 2017 8:29 am

I think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC
the C-z means "Control-Z to Pause", not compressed-zip
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jun 22, 2017 8:45 am

David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
 #    CHAIN                                             ACTION                            BYTES         PACKETS
 0  D ;;; special dummy rule to show fasttrack counters
      prerouting                                        passthrough                 205 064 681         238 851
 1    ;;; Attack from Intrus blocklist
      prerouting                                        drop                              8 846             206
 2    ;;; Attack from sbl malc0de
      prerouting                                        drop                                  0               0
 3    ;;; Attack from sbl dshield
      prerouting                                        drop                                 52               1
 4    ;;; Attack from sbl blocklist.de
      prerouting                                        drop                              3 309              42
 5    ;;; Attack from sbl spamhaus
      prerouting                                        drop                                  0               0
    
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 22, 2017 9:33 am

I think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC
the C-z means "Control-Z to Pause", not compressed-zip
Hahahaha I know and on the moment I noticed that it was not funny because a lot of time went in. This is the part of my posting about it and what I put above it:
I see different data when downloading html or the dynamic.rsc when I test it on my own server:

Darn the whole bit below is obsolete because the things I though I could deduct, is bases on not cleared characters by RouterOS. The result is written on the same line as the line shown during transfer "-- [Q quit|D dump|C-z pause]" so I was mislead by what it seemed to state and I was looking for........GRRRRRRRRRRRRRRRR

Code: Select all
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8318
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Thu Jun 22, 2017 10:02 am

The loop is to deal with older versions of RouterOS that would only remove the first item it found when using Find.
then consider using both =) first quickly remove for recent versions, then slow cleanup for older ones if necessary
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 22, 2017 4:40 pm

then consider using both =) first quickly remove for recent versions, then slow cleanup for older ones if necessary
I'll do that for the next release.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
ilivlad
just joined
Posts: 14
Joined: Tue Mar 12, 2013 2:02 pm

Re: Blacklist Filter update script

Thu Jun 22, 2017 6:19 pm

Hello!
Funny thing, when I run the script manually, it works, downloads the file and installs address entries but when scheduler runs it, it increases the run count but the script wont start.
I have other scripts running off scheduler without problems.

I have RB2011UiAS-2HnD, 6.39.2 (stable).
Screenshot from 2017-06-22 17-17-32.png
Screenshot from 2017-06-22 17-22-04.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Deantwo
Member
Member
Posts: 308
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Fri Jun 23, 2017 12:06 pm

Minor typo in the 4th line.
##### Update your path, is you are using a USB Flash or other storage
I am thinking you meant to say "if you are using"

By the way, why is the default path "disk1/dynamic.rsc"?

Anyway, fun fun. I hadn't tried this before:
jun/23/2017 10:50:44 system,error,critical router was rebooted without proper shutdown
jun/23/2017 10:50:44 system,error,critical kernel failure in previous boot
jun/23/2017 10:50:44 system,error,critical out of memory condition was detected
My poor little RB750 doesn't seem to like it either way.
jun/23/2017 11:29:13 system,error,critical router was rebooted without proper shut
down by watchdog timer
jun/23/2017 11:42:31 system,error,critical router was rebooted without proper shut
down by watchdog timer
Last edited by Deantwo on Fri Aug 10, 2018 3:26 pm, edited 1 time in total.
I wish my FTP was FTL.
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 25, 2017 1:36 am

Hi Dave, I have now completed the changed script after start-up/reboot of the router. As the dynamic address are all lost during reboot they don't have to be deleted.

In the updateBlacklist script I don't delete the dynamic.rsc file after importing so that they are still available after a new start-up/reboot. If the file does not exist then the normal updateBlacklist script is run so that the router is never without your dynamicBlacklist.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### DO NOT EDIT THE LINES BELOW ######
:local path "";
:local filename "dynamic.rsc"

##### Update your path, to where you have your storage
##### Examples: "disk1/"  or  "usb/" and the default is the temporary storage
#:local path "usb/"
:local path "disk1/"
:global datapath "$path$filename";
:delay 5;

##### Disable the log (We don't need 20k lines of adds and removes in the log
/system logging disable 0

##### Import the downloaded blacklist
:log warning "Importing saved file $datapath as dynamicBlacklist...";
:if ([:len [/file find name="$datapath"]] > 0) do={/import file-name="$datapath"};
:if ([:len [/file find name="$datapath"]] = 0) do={/system script run updateBlacklist};

##### Turn the logging back on
/system logging enable 0
:log warning "dynamicBlacklist $datapath imported.";
Update: reinserted
/system logging enable 0
so that logging is enabled again.

The :delay 5 is there because the router needs more time before reading the dynamic.rsc file.
Last edited by msatter on Sun Jun 25, 2017 6:55 pm, edited 3 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
vitorcsp
just joined
Posts: 4
Joined: Sat May 20, 2017 2:56 am
Location: Rio de Janeiro - RJ
Contact:

Re: Blacklist Filter update script

Sun Jun 25, 2017 3:16 am

Thanks!! Very good ...! i'll test in my RB450G
 
ronix
Member Candidate
Member Candidate
Posts: 152
Joined: Thu Nov 17, 2011 6:51 pm

Re: Blacklist Filter update script

Sun Jun 25, 2017 10:30 am

it didn't work for me (CCR1016-12G)
error :
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
status: failed

failure: connection timeout
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 25, 2017 2:37 pm

Thanks!! Very good ...! i'll test in my RB450G
Thanks , however this has first have to be agreed, because Dave has also to change the original updateBlaclist so that dynamic.rsc is not erased after import. There can be a problem when the file is always President on devices with not much free space.

This version is safe as it looks if the quick start is available and then use that. If the quick start is not possible then it downloads the dynamic.rsc file and imports it.

I can't send Dave any kind of messages through the forum except by making posts. There is a button when I look at his profile but nothing happens when I click it.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 6:32 pm

it didn't work for me (CCR1016-12G)
error :
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
status: failed

failure: connection timeout
Connection Timout on that would imply that your IP may be blocked to start with.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 6:39 pm

By the way, why is the default path "disk1/dynamic.rsc"?
because that is the default path of a USB or SATA drive. If the driver does not exist, it simply creates that path. This way the USB is used if it's there.

Anyway, fun fun. I hadn't tried this before:
jun/23/2017 10:50:44 system,error,critical router was rebooted without proper shutdown
jun/23/2017 10:50:44 system,error,critical kernel failure in previous boot
jun/23/2017 10:50:44 system,error,critical out of memory condition was detected
My poor little RB750 doesn't seem to like it either way.
jun/23/2017 11:29:13 system,error,critical router was rebooted without proper shut
down by watchdog timer
jun/23/2017 11:42:31 system,error,critical router was rebooted without proper shut
down by watchdog timer
I don't have any 32M units myself, but the blacklist stats show that 8 of them are currently pulling the list. It looks like it was a bad weekend for botnets as the list grew to 21,000 items. it may simply be too much for the smallest of routers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 7:05 pm

David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
Done.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 9:03 pm

I rewrote the backend this morning. It now takes all of the sources and purges the /32's into the their corresponding subnet, if it is listed. it cut the size by 50%. it was in the 42,000 range, now back down to 21,000.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jun 27, 2017 8:56 pm

Updated the script with the recommended remove code. It appears to speed the update process by 38~75 seconds on most routers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 12:42 am

Thanks Dave for the update and looking at the code and that you told that older equipment only removed one line when using the modern method. I thought why not use that as an advantage and combine the old and new method into this:
##### Find the "dynamicBlacklist" entries and remove them
:while ([/ip firewall address-list find list="dynamicBlacklist"] != "") do={ /ip firewall address-list remove [find list="dynamicBlacklist"]};
The modern equipment only execute the command once and the older quipment would repeat it until there are no more dynamicBlacklist entries.

Replaces this:
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
/ip firewall address-list remove [find list="dynamicBlacklist"]

##### Remove again - Some older RouterOS versions wont catch them all with the above line.
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }
I can't test it on old equipment so I don't know if is even slower than the :foreach or that it does even work that way on the old stuff.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 1:06 am

That looks like a nice clean solution. I'll test it out on the gear I have and then update the code. Thanks!
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 2:12 am

So far so good. Doesn't help the low end units much.
a quick test...

RB2011 - 123 seconds
CCR1016 - 25 seconds
RB1100AHx4 - 20 seconds
RB3011 - 33 seconds

....WOW! The new RB1100AHx4 is faster than a 16 core CCR.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 3:09 am

I expected a little improvement on the lower units because there is less code to execute. It excellent news that the older units can work with the code combined to one. Makes it all simpler and it fits in one line.

Lets hope it will work in all units and the list is growing fast lately and the list is over 25000 entries tonight.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Wed Jun 28, 2017 8:35 am

thanks dave,
I updated code and working good..
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 12:31 pm

I am testing a more flexible way to update however it seems that I am now throttled by the server. This is no problem however it does not throttle but gives a modified dynamic.rsc which reads:
:log error "Blacklist is updated at 10:00:00 UTC. Please update only once per day."
:log error "You have updated 7 times is the last 24 hours."
:log error "You will be able to update again in 24 hours."
 :for i from=1 to=3 step=1 do={
 :beep frequency=550 length=494ms;
   :delay 494ms;
   :beep frequency=400 length=494ms;
   :delay 494ms;
 }

The lines above is not show in the log and the present dynamicblacklist is removed. This leaves the router without the protection of your list.

Update: to avoid removing the present dynamicBlacklist if there is a throttle file downloaded:
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

#### Get size of the downloaded file
:local fileSize [/file get [ find where name=$datapath] value-name=size];

##### Find the "dynamicBlacklist" entries and remove them
:if ($fileSize > 1000) do={:log warning "Removing expiring address-list entries..."} else={:log error "Using the old Blacklist. Look for info about this error in the log underneath."};
:if ($fileSize > 1000) do={:while ([/ip firewall address-list find list="dynamicBlacklist"] != "") do={ /ip firewall address-list remove [find list="dynamicBlacklist"]}};

##### Import the downloaded blacklist
:if ($fileSize > 1000) do={:log warning "Importing current Blacklist..."};
/import file-name="$datapath";

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:if ($fileSize > 1000) do={:log warning "Blacklist Update Complete."};
/system logging enable 0
I have taken the liberty to include the promising new remove code of the current dynamicBlacklist
Last edited by msatter on Wed Jun 28, 2017 3:19 pm, edited 5 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Wed Jun 28, 2017 2:40 pm

David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
Done.
Thank you!

Unfortunately, it seems like you didn't get the same list as SBL (squidblacklist.org) uses, or you didn't merge the lists correctly. I've been tracking dropped packets by list, and I'm still seeing about 1 dropped packet from SBL's "blocklist.de" list for every 4 from your dynamicBlacklist. (I'm also seeing more hits from dshield, but that may just be a coincidence.) Please look into it when you have a chance.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 7:49 pm

That could be just the update timing. Currently, my list collects the data a 5am PST and rebuilds then. several of the sources also rate limit, but I may be able to push it and rebuild it ever 6 hours. that may keep them more in sync.

Okay, I changed the cron job to run every 6 hours.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 8:44 pm

I updated both the server and script to correct for the notification not displaying. I also changed the script so that the previous entries are not removed if the throttling kicks in. I would love to NOT have to throttle, but several people have set up their units to update every 5 minutes. at 2M each download multiplied by 40ish routers, every 5 minutes... Those routers were pulling 23G every day.

List is still dynamic and expires after 25 hours. This is to prevent false positives from hurting things for more than a day. (Some people were updating one a week, and complaining that false positives were not being removed quick enough)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 9:34 pm

I just ran to often the update so I got throttled and that is not a problem for me. I had to see what is happening and I adapted the script on my side to it and posted it here for you to see.
The messages in the error dynamic.rsc worked later and I incorporated that in my posting so that a clear message was left behind in the log and that not the blacklist was wiped before expiration time.

I now see why you are hesitant to keep the dynamic.rsc for a fast import on reboot despite it will be replaced by the next scheduled import. I wanted to combine the start-up schedule and the normal refresh schedule so that less administration is needed to setup and maybe the administration part can be automated depending on what kind of storage device is used.

Update: There goes the plan to have only one schedule: If interval is set to value other than 0 scheduler will not run at startup.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 9:58 pm

That's why I have always had two scheduled tasks. One for Startup and one every 24 hours.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 11:13 pm

Hi Dave,

So I have updated the start-up schedule so that dynamic.rsc files older than one day are not imported from flash/disk1/usb and the normal updateBlacklist script is run.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### DO NOT EDIT THE LINES BELOW ######
:local path "";
:local filename "dynamic.rsc"

##### Update your path, to where you have your storage
##### Examples: "disk1/"  or  "usb/" and the default is the temporary storage
#:local path "usb/"
:local path "disk1/"
:global datapath "$path$filename";

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:delay 10;

##### Disable the log (We don't need 20k lines of adds and removes in the log
/system logging disable 0

# Declaring and filling the date1 and date2 variable for calculating the time difference
:global globalDaysDiff
:local time [/system clock get time];
:local date [/system clock get date];
:global date2 ("$date" . " " . "$time");
:global date1 [/file get [ find where name=$datapath] value-name=creation-time];

# This script calculates difference between two dates
/system script run diffDate

##### Import the downloaded blacklist
:log warning "Importing saved file $datapath as dynamicBlacklist...";
 :if ([:len [/file find name="$datapath"]] > 0) do={:if ($globalDaysDiff != 0) do={:log error "dynamicBlacklist $datapath to old for fast import."} else={/import file-name="$datapath"}};

# Download Blacklist if there is no dynamic.rsc present 
:if ([:len [/file find name="$datapath"]] = 0) do={/system script run updateBlacklist};

##### Turn the logging back on
/system logging enable 0
:if ([:len [/file find name="$datapath"]] != 0) do={:log warning "dynamicBlacklist $datapath imported."} else={:log error "Nothing happened and no protection by dynamicBlacklist provided!"};
Next the script diffDate that calculates the needed difference between the creation date of dynamic.rsc and the current time:
       ### calculate diff between two dates - yoan tanguy 2017

# format: :global date1 "jan/05/2017 10:00:00";:global date2 "may/15/2018 12:30:00";/system script run diffDate

       
       # expected date format : month/day/year hours:minutes:seconds (ex: mar/14/2017 09:13:54)
       :global date1
       :global date2
       
       
       # date to array format :
       # m a r / 1 4 / 2 0 1 7     0  9  :  1  3  :  5  4
       # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
       :local date1month [:pick $date1 0 3]
       :local date1day [:pick $date1 4 6]
       :local date1year [:pick $date1 7 11]
       :local date1hours [:pick $date1 12 14]
       :local date1minutes [:pick $date1 15 17]
       :local date1seconds [:pick $date1 18 20]
       
       :local date2month [:pick $date2 0 3]
       :local date2day [:pick $date2 4 6]
       :local date2year [:pick $date2 7 11]
       :local date2hours [:pick $date2 12 14]
       :local date2minutes [:pick $date2 15 17]
       :local date2seconds [:pick $date2 18 20]
       
       
       # month to decimal converter - https://forum.mikrotik.com/viewtopic.php?t=58674
       :local months ("jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec");
       :set date1month ([:find $months $date1month -1 ] + 1)
       :set date2month ([:find $months $date2month -1 ] + 1)
       
       
       :global globalDiff 
       :local yearDiff ($date2year - $date1year)
       :local monthDiff ($date2month - $date1month)
       :local dayDiff ($date2day - $date1day) 
       :local hoursDiff ($date2hours - $date1hours)
       :local minutesDiff ($date2minutes - $date1minutes)
       :local secondsDiff ($date2seconds - $date1seconds)
       
       
       # handle diff by converting in seconds, avoid negative hours/minutes/seconds (ex: jan/01/1970 09:00:00, jan/02/1970 08:00:00 must give 0 days 23:00:00 and not 1 days 0-1:00:00)
       # 1 days 23:30:10
       # 1*24*60*60 + 23*60*60 + 30*60 + 10
       # ($dayDiff * 24*60*60) + ($hoursDiff * 60*60) + ($minutesDiff *60) + $secondsDiff
       # ($dayDiff * 86400) + ($hoursDiff * 3600) + ($minutesDiff *60) + $secondsDiff
       :local secondsGlobalDiff
       :set secondsGlobalDiff (($dayDiff * 86400) + ($hoursDiff * 3600) + ($minutesDiff *60) + $secondsDiff)
       :set dayDiff ($secondsGlobalDiff / 86400)
       :set secondsGlobalDiff ($secondsGlobalDiff - ($dayDiff * 86400))
       :set hoursDiff ($secondsGlobalDiff / 3600)
       :set secondsGlobalDiff ($secondsGlobalDiff - ($hoursDiff * 3600))
       :set minutesDiff ($secondsGlobalDiff / 60)
       :set secondsGlobalDiff ($secondsGlobalDiff - ($minutesDiff * 60))
       :set secondsDiff $secondsGlobalDiff
       
       
       # check if date1 is older than date2 to avoid errors in calculation
       if ($yearDiff < 0) do={
           :return "error : date1 should be older that date2 (year check), exiting"
       } else={
           if ($yearDiff = 0) do={
               if ($monthDiff <0) do={
                   :return "error : date1 should be older that date2 (month check), exiting"
               } else={
                   if ($monthDiff = 0) do={
                       if ($dayDiff < 0) do={
                           :return "error : date1 should be older that date2 (day check), exiting"
                       } else={
                           if ($dayDiff = 0) do={
                               if ($hoursDiff < 0) do={
                                   :return "error : date1 should be older that date2 (hours check), exiting"
                               } else={
                                   if ($hoursDiff = 0) do={
                                       if ($minutesDiff < 0) do={
                                           :return "error : date1 should be older that date2 (minutes check), exiting"
                                       } else={
                                           if ($minutesDiff = 0) do={
                                               if ($secondsDiff < 0) do={
                                                   :return "error : date1 should be older that date2 (seconds check), exiting"
                                               }
                                           }
                                       }
                                   }
                               }
                           }
                       }
                   }
               }
           }
       }          
       
       
       # check if leap years - https://wiki.mikrotik.com/wiki/AutomatedBilling/MonthEndScript
       :local isYear1Leap 0
       :local isYear2Leap 0
       if ((($date1year / 4) * 4) = $date1year) do={
           :set isYear1Leap 1
       }
       if ((($date2year / 4) * 4) = $date2year) do={
           :set isYear2Leap 1
       }
       
       
       # find the right amount of days between 2 months
       :local daysInEachMonth ("31","28","31","30","31","30","31","31","30","31","30","31");
       :local daysInEachMonthLeapYear ("31","29","31","30","31","30","31","31","30","31","30","31");
       :local totalDaysBetweenMonths
       
       # same year; yearDiff = 0 so year1 = year2
       if ($yearDiff = 0 and $monthDiff >= 1) do={
           if ($isYear1Leap = 0) do={         
               for month from=($date1month - 1) to=($date2month - 1) step=1 do={
                   :set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonth $month])
               }
           }
           if ($isYear1Leap = 1) do={
               for month from=($date1month - 1) to=(($date2month - 1) - 1) step=1 do={
                   :set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonthLeapYear $month])
               }
           }
       }
       
       # different year, make concatenation of daysInEachMonth arrays first
       :local daysInEachMonthConcatenatedYears
       if ($yearDiff >= 1) do={
       
           for year from=$date1year to=$date2year step=1 do={
               # if leap year, concatenate the right daysInEachMonth array
               if ((($year / 4) * 4) = $year) do={
                   :set daysInEachMonthConcatenatedYears ($daysInEachMonthConcatenatedYears, $daysInEachMonthLeapYear)
               } else={
                   :set daysInEachMonthConcatenatedYears ($daysInEachMonthConcatenatedYears, $daysInEachMonth)
               }
           }
           
           # must add years count 
           for month from=($date1month - 1) to=(($date2month - 1)  + (($yearDiff * 12) - 1)) step=1 do={
               :set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonthConcatenatedYears $month])
           }
       }
       
       :global globalDaysDiff ($totalDaysBetweenMonths + $dayDiff)
       
       
       # add leading zeros if necessary
       :if ($hoursDiff < 10) do={
           :set hoursDiff ("0" . $hoursDiff)
       }
       :if ($minutesDiff < 10) do={
           :set minutesDiff ("0" . $minutesDiff)
       }
       :if ($secondsDiff < 10) do={
           :set secondsDiff ("0" . $secondsDiff)
       } 
       :local d "d"
       :set globalDiff "$globalDaysDiff$d$hoursDiff:$minutesDiff:$secondsDiff"
       :put $globalDiff
So now maybe you can consider to keep the dynamic.rsc between updates and so avoid traffic by rebooting devices and people that run the update script every 5 minutes. The update script would than be updated with the same code and will warn people that they are wearing out their memory by those obsolete updates.

For other users of the script please wait until Dave had his say about this and wait for his updates and do not use this code unless you know what you are doing!!
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jun 29, 2017 3:05 am

Hi Dave,

So I have updated the start-up schedule so that dynamic.rsc files older than one day are not imported from flash/disk1/usb and the normal updateBlacklist script is run.
I think you need to check that you have a reliable date in the first place. It can be a while between boot up and acquiring the current date and time. I would not count on a simple delay being enough, I would sanity check the date.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 29, 2017 7:30 am

I think you need to check that you have a reliable date in the first place. It can be a while between boot up and acquiring the current date and time. I would not count on a simple delay being enough, I would sanity check the date.
I second that. If I've learned anything about RouterOS, it's that you can NOT trust the date and time at boot. I have several routers that take up to 20 minutes before the time is synced correctly.


As for bandwidth, it's not an issue for me. I have a gigabit connection with no metering. The router throttles each incoming IP to 100mbps. Also, the server compresses the list when it sends it, so it's typically only a few hundred kb. Also, I don't want to store the 2~4mb list on the flash because some of the units out there only have 16M and even then, those only normally have about 5M free. This leaves no room for updates. BUT - you are welcome to change the script in anyway you like, I just ask that the fetch isn't changed.

Actually, I was thinking of collecting Total and Free disk space - but I'm not sure how people will feel about that. I wonder if I can make a poll on the forum...
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 29, 2017 11:52 am

I am taking the time of the file that is downloaded by daily updateBlacklist so that is constant. Indeed the current time is a problem if it is not current.

About the compression by the server. I tested it by my own server and I did not see the device using compression and I have to look with Wireshark if that is also the case with your server.

The flash I had already a routine for to not keep the dynamic.rsc for those flash devices and it can be overruled by and variable set by user to ignore that and keep the dynamic.rsc anyway. I did not put that in this version.

I am going to look if the code can be more streamlined because I have the impressing I am doing thing twice.
Last edited by msatter on Thu Jun 29, 2017 12:45 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 29, 2017 12:34 pm

Mikrotik thought of the problem and came up with a solution:
Since v6.16 the current time is saved in the system configuration on reboot and on clock adjustment and is used to set the initial time after reboot.
Benefits:
Router doesn't need direct access to internet and public NTP servers
Allow control of a primary source of clock for your router on only two main routers (primary and secondary)
It can reduce traffic and the load of some public NTP servers by local time caching
Source: https://wiki.mikrotik.com/wiki/Setup_local_NTP_servers

We are thinking here in days not minutes and seconds to decide if a file should be declared outdated. We catching reboots and but also devices set to a higher scheduled update than a day.
Starts of a device can lead to false positives but that will be corrected on the next scheduled run of updateBlacklist.

Still to do flash only devices and automatic recognize flash (default), disk1 or USB. Check if scheduled can be imported set to time of 10 UTC + random time for spreading the load the server.

I have sent support a mail for clarification, on if fetch support deflate/compress and do use that advantage?

Update: I have tested it again and despite the site that checks if the connection is compressed gives an OK on the file the Mikrotik does not use it. I have forced the file to be transmitted compressed by Apache but the Mikrotik did not decompress it.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 29, 2017 4:37 pm

To automatic select the location for dynamic.rsc can be archived with the following code:
:if ([:len [/file find name="flash"]] != 0) do={set datapath "dynamic.rsc"};
:if ([:len [/file find name="disk1"]] != 0) do={set datapath "disk1/dynamic.rsc"};
:if ([:len [/file find name="disk2"]] != 0) do={set datapath "disk2/dynamic.rsc"};
:if ([:len [/file find name="disk3"]] != 0) do={set datapath "disk3/dynamic.rsc"};
:if ([:len [/file find name="usb"]] != 0) do={set datapath "usb/dynamic.rsc"};
:log info "Default location for Blacklist is: $datapath";
Extended with a check on free space and the minimal free space is 3MB to be selected.
:if ([:len [/file find name="flash"]] != 0)  do={:if ([/system resource get free-hdd-space] > 3000000)  do={set datapath "dynamic.rsc"}};
:if ([:len [/file find name="disk1"]] != 0) do={:if ([/disk get [ find where name="disk1"] value-name=free] > 3000000) do={set datapath "disk1/dynamic.rsc"}};
:if ([:len [/file find name="disk2"]] != 0) do={:if ([/disk get [ find where name="disk2"] value-name=free] > 3000000) do={set datapath "disk2/dynamic.rsc"}};
:if ([:len [/file find name="disk3"]] != 0) do={:if ([/disk get [ find where name="disk3"] value-name=free] > 3000000) do={set datapath "disk3/dynamic.rsc"}};
:if ([:len [/file find name="usb"]] != 0) do={:if ([/disk get [ find where name="usb"] value-name=free] > 3000000) do={set datapath "usb/dynamic.rsc"}};
:log info "Default save locationwith 3MB free  for Blacklist is: $datapath";
The Blacklist has become very long but it works and can say that every minute at least one or more block are made by the list on my Mikrotik.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jun 30, 2017 2:00 am

Result from today's Blacklist is 1808 packets caught by the list and above that one I filter connection made to services that I don't have and that were 1474 packets. So in total almost 3300 unwanted connections in one day and four hours. Most of the Blacklist packages came for port 25 to deliver unwanted stuff, so Spamassin is having now a kind of vacation. :-)
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jun 30, 2017 11:14 am

Collecting how many packets are blocked by the Blacklist:
#### Share how many packets are blocked by the Blacklist on your device
:local filterdownBlacklist "0";
:local rawdownBlacklist "0";
:local filterupBlacklist "0";
:local rawupBlacklist "0";

##### downstream
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterdownBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterdownBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawdownBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawdownBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

##### upstream
:if ([:len [/ip firewall filter find dst-address-list="dynamicBlacklist"]] != 0)  do={set filterupBlacklist [/ip firewall filter get [ find dst-address-list="dynamicBlacklist"] packets]}  else={set filterupBlacklist "0"};
:if ([:len [/ip firewall filter find dst-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find dst-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find dst-address-list="dynamicBlacklist"]] != 0)  do={set rawupBlacklist [/ip firewall raw get [ find dst-address-list="dynamicBlacklist"] packets]} else={set rawupBlacklist "0"};
:if ([:len [/ip firewall raw find dst-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find dst-address-list="dynamicBlacklist"]};

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for model $model $ver";
/tool fetch mode=https dst-path="$datapath" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid&filterdown=$filterdownBlacklist&rawdown=$rawdownBlacklist&filterup=$filterupBlacklist&rawup=$rawupBlacklist";
After collecting the numbers, each packets counter in Filters and RAW are reset to zero. In this way you won't get double countings on the next update of the Blacklist.

.....done enough for now and going to do other things. :-) ....added later the upstream so that is also counted..............
Last edited by msatter on Sat Jul 01, 2017 10:44 am, edited 3 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 30, 2017 5:53 pm

Today’s update is going to be huge. Not sure when I will push it it out though. I am rewriting the backend that builds the list. I will be pushing out 3 lists soon.

Small - about 750kb - intended for home users
Standard - about 2M - intended for businesses
Full - about 14M - intended for internet servers

Admins will need to choose wisely as the full list will fill the drive on many units and will cause out of memory panics on the small units.

The full list is currently about 114,000 entries. It pulls from many more sources and i would recommend building a whitelist for use with it as you may end up locked out or remote management if you are on a home IP.

The standard is what we have been using.

The small will average about 7000 to 8000 subnets and ips. Primarily C&C and botnets.

The new script will allow you to select the list you want.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8318
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Fri Jun 30, 2017 6:01 pm

were there thoughts about BGP feed?..
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 30, 2017 7:04 pm

were there thoughts about BGP feed?..
Too much work :)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 01, 2017 3:25 am

The new backend and script are live. Make sure you read the comments and select the correct script for your router.
*** DO NOT SELECT THE LARGE LIST FOR ROUTERS WITH LESS THAN 20M FREE DISK OR LESS THAN 256M Memory! ***

Recommendation:

Routers with 32M~128M memory - "small" list
Routers with 256M~512M memory - "medium" list
Routers with 1G memory and up - "large" list
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 01, 2017 10:32 am

The new backend and script are live. Make sure you read the comments and select the correct script for your router.
*** DO NOT SELECT THE LARGE LIST FOR ROUTERS WITH LESS THAN 20M FREE DISK OR LESS THAN 256M Memory! ***

Recommendation:

Routers with 32M~128M memory - "small" list
Routers with 256M~512M memory - "medium" list
Routers with 1G memory and up - "large" list
Thanks for your great work! I had to make a minor correction to version 2017.7.1d, and propose a modification to give more info to the person who is checking the log.
#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### medium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports
#### Begin download of current blacklist
:log warning "Downloading current $listSize sized Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.ph ... id=$softid";
:local fileSize [/file get [ find where name=$datapath] value-name=size];
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
eddieb
Member Candidate
Member Candidate
Posts: 141
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Sat Jul 01, 2017 10:37 am

Collecting how many packets are blocked by the Blacklist:
#### Share how many packets are blocked by the Blacklist on your device
:local filterBlacklist "0";
:local rawBlacklist "0";

:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for model $model $ver";
/tool fetch mode=https dst-path="$datapath" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid&filter=$filterBlacklist&raw=$rawBlacklist";
After collecting the numbers, each packets counter in Filters and RAW are reset to zero. In this way you won't get double countings on the next update of the Blacklist.
Hi, interesting scripting ...
I tried it as a separate script in the following way :
#### Share how many packets are blocked by the Blacklist on your device
:local filterBlacklist "0";
:local rawBlacklist "0";

:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

:log warning "Count filterBlacklist=$filterBlacklist rawBlacklist=$rawBlacklist";
BUT the counters are NOT reset and the log displays zeroes ...

any suggestions ?
Running 6.45.7 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 01, 2017 10:55 am

Hi, interesting scripting ...
I tried it as a separate script in the following way :
:log warning "Count filterBlacklist=$filterBlacklist rawBlacklist=$rawBlacklist";
BUT the counters are NOT reset and the log displays zeroes ...

any suggestions ?
Try:
:log warning "Count filterBlacklist= $filterBlacklist rawBlacklist= $rawBlacklist";
yes, scripting in the Mikrotik is a PITA. I have that experienced that enough in the last week. ;-)

I have also updated the script to catch the upstream blocks: viewtopic.php?f=9&t=98804&p=605898#p605796 and the variable names changed accordingly.
Last edited by msatter on Sat Jul 01, 2017 11:00 am, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
eddieb
Member Candidate
Member Candidate
Posts: 141
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Sat Jul 01, 2017 11:00 am

tnx,

scripting can be a pain, sometimes it just does not work ...
Count filterBlacklist=0 rawBlacklist=30
it works ;-)
Running 6.45.7 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 01, 2017 11:14 am

tnx,

scripting can be a pain, sometimes it just does not work ...
Count filterBlacklist=0 rawBlacklist=30
it works ;-)
This only for private use on the moment and if you only want to know the score remove the reset lines. When Dave is ready for more statistics then he can implement it.

I am still thinking about how to extrapolate the data when a there was a router reset in that period.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 02, 2017 11:49 am

So thinking on collection more information about the effectiveness of the Blacklist you can also collect the gmt-offset from /system clock so that you can see in which time zone the data is collected.

To get a idea how the effectiveness of Blacklist is between the previous download and the new download packets numbers could be collected every hour. Your can then see how the degradation is of the Blacklist and if there is a significant degradation decide to increase or decrease the updates. These should be only the downstream (incoming) figures and not the more private sensitive info of the upstream (outgoing). This can also, be a consideration with collecting the 24 hour data were I wrote about earlier.
:local timeOffset  [/system clock get value-name=gmt-offset];
The output is 7200 seconds so that is +2 hours in my case.

To get the only one or two variable(s) for the 48 (filter+raw) numbers to be transferred separately you can concatenate them in one or two strings so that you can transfer it when you collecting technical data of the router.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1284
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 02, 2017 6:12 pm

To collect and save the data so that it can survive a reboot an hourly script can be scheduled that executes the following script:
##### Read the save statistics
/import file blacklist.rsc
:global statsFilterBlacklist;
:global statsRAWBlacklist

##### Get current time and set filename to keep statistics
:local date [/system clock get date];
:local time [/system clock get time];
:local filename "blacklist.rsc";

##### Collect and reset packet counters
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterdownBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterdownBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawdownBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawdownBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

#### Build new stats string
:local newStatsFilterBlacklist "$statsFilterBlacklist" . " " . "$filterdownBlacklist";
:local newStatsRAWBlacklist "$statsRAWBlacklist" . " " . "$rawdownBlacklist";
:local newStatDate ("$date" . " " . "$time");

:local writeString ":global $lastStatDate;" . ":global statsFilterBlacklist $newStatsFilterBlacklist;" . " " . ":global statsRAWBlacklist "$StatsRAWBlacklist";

/file set $filename content=$writeString;
Some thoughts. This script can possible collide with the updateBlacklist script and to notice that the blacklist.rsc can be deleted on reading for sending. This script should not execute on that instance and a new blacklist.rsc should be recreated with time plus the two strings without any numbers in it.

Example of the blacklist.rsc statiscs file:
:global lastStatDate "jul/02/2017 15:49:19"; :global statsFilterBlacklist "1 2 3 4 5 6 7 8 9"; :global statsRAWBlacklist "0 9 8 7 6 5 4 3 2 1";
The file supplies the last sample date and time and maybe a the gmt-offset can sync the data with other available data already in the database.

I have not tested the code so please check on syntax and typing errors.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta59 / Winbox 3.20 / MikroTik APP 1.3.7
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Mon Jul 03, 2017 5:24 am

I have modified the scripts in a few ways and am publishing the modified scripts here for whoever wants them. @IntrusDave is welcome to incorporate them into his script or not.
  • Renamed globals so as not to interfere with other scripts
  • Added lots of error handling and corresponding error logging
  • Keep downloaded list for reinstall after reboot
  • Split script into 2 scripts, a download script and an install script, so I can just run the install script at boot time
  • Formatted for 1 statement per line, 2 space indent per block
Note that because the scripts use globals to communicate, they need policy permission in addition to the read, write, and test permissions that IntrusDave's script needs.

The update script downloads the list and calls the install script if successful:
# https://forum.mikrotik.com/viewtopic.php?f=9&t=98804

# Import Intrus Managed Filter Lists
# CUSTOMIZED by jgro, different globals, do not simply replace with update from Intrus
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage
##### Examples:
##### "disk1/dynamic.rsc"  or  "usb/dynamic.rsc"  or  "dynamic.rsc"

:global intrusPath  "disk1/dl/dynamic.rsc"

#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### medium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports

:local listSize "medium"


###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
#:log warning "Blacklist update in 10 seconds";
#:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]

:if ($model = "CHR") do={
  :local temp [/system license get system-id]
  :for i from=0 to=([:len $temp] - 1) do={ 
     :local char [:pick $temp $i]
     :if ($char = "/") do={ :set $char "-" }
     :set softid ($softid . $char)
   }
}
:if ($model !="CHR") do={
  :global softid [/system license get software-id]
}

:local scriptVer   2017.7.1d

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:local fileSize
:log warning "Downloading current Intrus dynamicBlacklist for this model";
:do {
  :do { 
    /tool fetch mode=https dst-path="$intrusPath" \
     url="https://mikrotikfilters.com/download.php?get=$listSize&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

    :set fileSize [/file get [ find where name=$intrusPath] value-name=size];
    :if ($fileSize < 500) do={
      :log error "IntrusBL download is too small"
      :error "IntrusBL download is too small"
    }
  } on-error={
    :log error  "FAILED to download Intrus dynamicBlacklist"
    /system script run "play-alert-sound"
  }

  :if ($fileSize > 500) do={
    /system script run import-intrus-block-list  
  }
} on-error { 
 :log error "FAILED to update Intrus dynamicBlacklist";
}
The import script does the import, and can be run at boot time (if you have saved the list somewhere) before the network even comes up:
##### Update your path, is you are using a USB Flash or other storage
##### Examples:
##### "disk1/dynamic.rsc"  or  "usb/dynamic.rsc"  or  "dynamic.rsc"

:global intrusPath 

:log warning "Starting import of Intrus dynamicBlocklist"

# intrusPath  set by code that does the fetch
# set fallback in case it is unset
:if ("x$intrusPath " = "x") do={
  :set intrusPath  "disk1/dl/dynamic.rsc"
  :log warning "Importing dynamicBlacklist from fallback location: $intrusPath"
}

:if ([/file find name=$intrusPath ] = "") do= {
  :error "FAILED: Importing dynamicBlacklist: file not found: $intrusPath "
}

##### Disable the log (We don't need 20k lines of adds and removes in the log)
:log warning "Disabling info logging while loading dynamicBlacklist...";
:log info "Disabling info logging while loading dynamicBlacklist...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:local status "failed"
:local fileSize [/file get [ find where name=$intrusPath] value-name=size];
:if ($fileSize > 500) do={
  :log warning "Removing expiring address-list entries...";
  /ip firewall address-list remove [find list="dynamicBlacklist"]

  ##### Import the downloaded blacklist
  :log warning "Importing downloaded dynamicBlacklist from $intrusPath ";

  do { 
    /import $intrusPath
    :set status "success"
  } on-error { 
    :log warning "FAILED to import $intrusPath "
  }

####### Find and remove the downloaded file
###:log warning "Removing dynamicBlacklist temp file...";
###/file remove [find name=$intrusPath ]

} else= { :log warning "Intrus blacklist file $intrusPath too small ($fileSize), aborting" }

##### Turn the logging back on
/system logging enable 0
:log warning "info logging enabled"
:log info "info logging enabled";

:if ($status = "success") do={ 
  :log warning "Intrus dynamicBlacklist Update Complete.";
} else={
  :error "FAILED to update Intrus dynamicBlacklist"
}
The script also calls a "play-alert-sound" script for a big problem. You can make an empty one or use this one stolen from Dave:
:log warning "Playing alert sound"
 :for i from=1 to=3 step=1 do={
 :beep frequency=550 length=494ms;
   :delay 494ms;
   :beep frequency=400 length=494ms;
   :delay 494ms;
 }

Who is online

Users browsing this forum: No registered users and 12 guests