Community discussions

 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 21, 2017 11:18 am

And some more info on how to reduce the traffic if RouterOS is supporting gzip/deflate: https://www.scalescale.com/tips/nginx/h ... mpression/

When I now use your site I get no get gzip on the application/octet-stream:
root@search:~# curl --header "Accept-Encoding: gzip,deflate,sdch" -I https://mikrotikfilters.com/updateBlacklist.rsc
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 21 Jun 2017 08:11:29 GMT
Content-Type: application/octet-stream
Content-Length: 4141
Last-Modified: Thu, 01 Jun 2017 04:22:22 GMT
Connection: keep-alive
Keep-Alive: timeout=2
Accept-Ranges: bytes
When compression is active then the saving would be 95.7% and your transfer goes from 1.8MB to 78KB:
................./dynamic.txt is Compressed

Uncompressed Page Size: 1817.7 KB
Compressed Page Size: 77.8 KB
Savings: 95.7%
I see different data when downloading html or the dynamic.rsc when I test it on my own server:

Darn the whole bit below is obsolete because the things I though I could deduct, is bases on not cleared characters by RouterOS. The result is written on the same line as the line shown during transfer "-- [Q quit|D dump|C-z pause]" so I was mislead by what it seemed to state and I was looking for........GRRRRRRRRRRRRRRRR

[admin@MikroTik] > /tool fetch mode=http url=https://xxxx.xx/index.html
       status: finished
  downloaded: 0KiBC-z pause]
       total: 0KiB
    duration: 1s
[admin@MikroTik] > /tool fetch mode=http url=https://xxxx.xx/dynamic.rsc
      status: finished
  downloaded: 1817KiB pause]
       total: 1817KiB
    duration: 1s
I think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC

And a PDF also get -z but nu C:
[admin@MikroTik] > /tool fetch mode=http url=https://xxxxx.xx/files/xxxxxxx.pdf
      status: finished
  downloaded: 71KiB-z pause]
       total: 71KiB
    duration: 1s
[/i]
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 21, 2017 5:57 pm

The loop is to deal with older versions of RouterOS that would only remove the first item it found when using Find.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 21, 2017 7:04 pm

The server does compress the content.... As seen by this compression test.
You do not have the required permissions to view the files attached to this post.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 22, 2017 8:29 am

I think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC
the C-z means "Control-Z to Pause", not compressed-zip
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jun 22, 2017 8:45 am

David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
 #    CHAIN                                             ACTION                            BYTES         PACKETS
 0  D ;;; special dummy rule to show fasttrack counters
      prerouting                                        passthrough                 205 064 681         238 851
 1    ;;; Attack from Intrus blocklist
      prerouting                                        drop                              8 846             206
 2    ;;; Attack from sbl malc0de
      prerouting                                        drop                                  0               0
 3    ;;; Attack from sbl dshield
      prerouting                                        drop                                 52               1
 4    ;;; Attack from sbl blocklist.de
      prerouting                                        drop                              3 309              42
 5    ;;; Attack from sbl spamhaus
      prerouting                                        drop                                  0               0
    
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 22, 2017 9:33 am

I think that C-z in "0KiBC-z" stands for Compression gzip so it is there and now it is how to get that working for the .RSC
the C-z means "Control-Z to Pause", not compressed-zip
Hahahaha I know and on the moment I noticed that it was not funny because a lot of time went in. This is the part of my posting about it and what I put above it:
I see different data when downloading html or the dynamic.rsc when I test it on my own server:

Darn the whole bit below is obsolete because the things I though I could deduct, is bases on not cleared characters by RouterOS. The result is written on the same line as the line shown during transfer "-- [Q quit|D dump|C-z pause]" so I was mislead by what it seemed to state and I was looking for........GRRRRRRRRRRRRRRRR

Code: Select all
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Thu Jun 22, 2017 10:02 am

The loop is to deal with older versions of RouterOS that would only remove the first item it found when using Find.
then consider using both =) first quickly remove for recent versions, then slow cleanup for older ones if necessary
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 22, 2017 4:40 pm

then consider using both =) first quickly remove for recent versions, then slow cleanup for older ones if necessary
I'll do that for the next release.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
ilivlad
just joined
Posts: 14
Joined: Tue Mar 12, 2013 2:02 pm

Re: Blacklist Filter update script

Thu Jun 22, 2017 6:19 pm

Hello!
Funny thing, when I run the script manually, it works, downloads the file and installs address entries but when scheduler runs it, it increases the run count but the script wont start.
I have other scripts running off scheduler without problems.

I have RB2011UiAS-2HnD, 6.39.2 (stable).
Screenshot from 2017-06-22 17-17-32.png
Screenshot from 2017-06-22 17-22-04.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
Deantwo
Member Candidate
Member Candidate
Posts: 295
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Fri Jun 23, 2017 12:06 pm

Minor typo in the 4th line.
##### Update your path, is you are using a USB Flash or other storage
I am thinking you meant to say "if you are using"

By the way, why is the default path "disk1/dynamic.rsc"?

Anyway, fun fun. I hadn't tried this before:
jun/23/2017 10:50:44 system,error,critical router was rebooted without proper shutdown
jun/23/2017 10:50:44 system,error,critical kernel failure in previous boot
jun/23/2017 10:50:44 system,error,critical out of memory condition was detected
My poor little RB750 doesn't seem to like it either way.
jun/23/2017 11:29:13 system,error,critical router was rebooted without proper shut
down by watchdog timer
jun/23/2017 11:42:31 system,error,critical router was rebooted without proper shut
down by watchdog timer
Last edited by Deantwo on Fri Aug 10, 2018 3:26 pm, edited 1 time in total.
I wish my FTP was FTL.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 25, 2017 1:36 am

Hi Dave, I have now completed the changed script after start-up/reboot of the router. As the dynamic address are all lost during reboot they don't have to be deleted.

In the updateBlacklist script I don't delete the dynamic.rsc file after importing so that they are still available after a new start-up/reboot. If the file does not exist then the normal updateBlacklist script is run so that the router is never without your dynamicBlacklist.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### DO NOT EDIT THE LINES BELOW ######
:local path "";
:local filename "dynamic.rsc"

##### Update your path, to where you have your storage
##### Examples: "disk1/"  or  "usb/" and the default is the temporary storage
#:local path "usb/"
:local path "disk1/"
:global datapath "$path$filename";
:delay 5;

##### Disable the log (We don't need 20k lines of adds and removes in the log
/system logging disable 0

##### Import the downloaded blacklist
:log warning "Importing saved file $datapath as dynamicBlacklist...";
:if ([:len [/file find name="$datapath"]] > 0) do={/import file-name="$datapath"};
:if ([:len [/file find name="$datapath"]] = 0) do={/system script run updateBlacklist};

##### Turn the logging back on
/system logging enable 0
:log warning "dynamicBlacklist $datapath imported.";
Update: reinserted
/system logging enable 0
so that logging is enabled again.

The :delay 5 is there because the router needs more time before reading the dynamic.rsc file.
Last edited by msatter on Sun Jun 25, 2017 6:55 pm, edited 3 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
vitorcsp
just joined
Posts: 4
Joined: Sat May 20, 2017 2:56 am
Location: Rio de Janeiro - RJ
Contact:

Re: Blacklist Filter update script

Sun Jun 25, 2017 3:16 am

Thanks!! Very good ...! i'll test in my RB450G
 
ronix
Member Candidate
Member Candidate
Posts: 152
Joined: Thu Nov 17, 2011 6:51 pm

Re: Blacklist Filter update script

Sun Jun 25, 2017 10:30 am

it didn't work for me (CCR1016-12G)
error :
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
status: failed

failure: connection timeout
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jun 25, 2017 2:37 pm

Thanks!! Very good ...! i'll test in my RB450G
Thanks , however this has first have to be agreed, because Dave has also to change the original updateBlaclist so that dynamic.rsc is not erased after import. There can be a problem when the file is always President on devices with not much free space.

This version is safe as it looks if the quick start is available and then use that. If the quick start is not possible then it downloads the dynamic.rsc file and imports it.

I can't send Dave any kind of messages through the forum except by making posts. There is a button when I look at his profile but nothing happens when I click it.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 6:32 pm

it didn't work for me (CCR1016-12G)
error :
/tool fetch url="https://mikrotikfilters.com/updateBlacklist.rsc" mode=https;
status: failed

failure: connection timeout
Connection Timout on that would imply that your IP may be blocked to start with.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 6:39 pm

By the way, why is the default path "disk1/dynamic.rsc"?
because that is the default path of a USB or SATA drive. If the driver does not exist, it simply creates that path. This way the USB is used if it's there.

Anyway, fun fun. I hadn't tried this before:
jun/23/2017 10:50:44 system,error,critical router was rebooted without proper shutdown
jun/23/2017 10:50:44 system,error,critical kernel failure in previous boot
jun/23/2017 10:50:44 system,error,critical out of memory condition was detected
My poor little RB750 doesn't seem to like it either way.
jun/23/2017 11:29:13 system,error,critical router was rebooted without proper shut
down by watchdog timer
jun/23/2017 11:42:31 system,error,critical router was rebooted without proper shut
down by watchdog timer
I don't have any 32M units myself, but the blacklist stats show that 8 of them are currently pulling the list. It looks like it was a bad weekend for botnets as the list grew to 21,000 items. it may simply be too much for the smallest of routers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 7:05 pm

David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
Done.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jun 26, 2017 9:03 pm

I rewrote the backend this morning. It now takes all of the sources and purges the /32's into the their corresponding subnet, if it is listed. it cut the size by 50%. it was in the 42,000 range, now back down to 21,000.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jun 27, 2017 8:56 pm

Updated the script with the recommended remove code. It appears to speed the update process by 38~75 seconds on most routers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 12:42 am

Thanks Dave for the update and looking at the code and that you told that older equipment only removed one line when using the modern method. I thought why not use that as an advantage and combine the old and new method into this:
##### Find the "dynamicBlacklist" entries and remove them
:while ([/ip firewall address-list find list="dynamicBlacklist"] != "") do={ /ip firewall address-list remove [find list="dynamicBlacklist"]};
The modern equipment only execute the command once and the older quipment would repeat it until there are no more dynamicBlacklist entries.

Replaces this:
##### Find the "dynamicBlacklist" entries and remove them
:log warning "Removing expiring address-list entries...";
/ip firewall address-list remove [find list="dynamicBlacklist"]

##### Remove again - Some older RouterOS versions wont catch them all with the above line.
:foreach i in=[/ip firewall address-list find ] \
   do={ :if ( [/ip firewall address-list get $i list] = "dynamicBlacklist" ) \
    do={ /ip firewall address-list remove $i } }
I can't test it on old equipment so I don't know if is even slower than the :foreach or that it does even work that way on the old stuff.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 1:06 am

That looks like a nice clean solution. I'll test it out on the gear I have and then update the code. Thanks!
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 2:12 am

So far so good. Doesn't help the low end units much.
a quick test...

RB2011 - 123 seconds
CCR1016 - 25 seconds
RB1100AHx4 - 20 seconds
RB3011 - 33 seconds

....WOW! The new RB1100AHx4 is faster than a 16 core CCR.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 3:09 am

I expected a little improvement on the lower units because there is less code to execute. It excellent news that the older units can work with the code combined to one. Makes it all simpler and it fits in one line.

Lets hope it will work in all units and the list is growing fast lately and the list is over 25000 entries tonight.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Wed Jun 28, 2017 8:35 am

thanks dave,
I updated code and working good..
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 12:31 pm

I am testing a more flexible way to update however it seems that I am now throttled by the server. This is no problem however it does not throttle but gives a modified dynamic.rsc which reads:
:log error "Blacklist is updated at 10:00:00 UTC. Please update only once per day."
:log error "You have updated 7 times is the last 24 hours."
:log error "You will be able to update again in 24 hours."
 :for i from=1 to=3 step=1 do={
 :beep frequency=550 length=494ms;
   :delay 494ms;
   :beep frequency=400 length=494ms;
   :delay 494ms;
 }

The lines above is not show in the log and the present dynamicblacklist is removed. This leaves the router without the protection of your list.

Update: to avoid removing the present dynamicBlacklist if there is a throttle file downloaded:
##### Disable the log (We don't need 20k lines of adds and removes in the log
:log warning "Disabling info logging...";
/system logging disable 0

#### Get size of the downloaded file
:local fileSize [/file get [ find where name=$datapath] value-name=size];

##### Find the "dynamicBlacklist" entries and remove them
:if ($fileSize > 1000) do={:log warning "Removing expiring address-list entries..."} else={:log error "Using the old Blacklist. Look for info about this error in the log underneath."};
:if ($fileSize > 1000) do={:while ([/ip firewall address-list find list="dynamicBlacklist"] != "") do={ /ip firewall address-list remove [find list="dynamicBlacklist"]}};

##### Import the downloaded blacklist
:if ($fileSize > 1000) do={:log warning "Importing current Blacklist..."};
/import file-name="$datapath";

##### Find and remove the downloaded file
:log warning "Removing temp file...";
/file remove [find name=$datapath]

##### Turn the logging back on
:if ($fileSize > 1000) do={:log warning "Blacklist Update Complete."};
/system logging enable 0
I have taken the liberty to include the promising new remove code of the current dynamicBlacklist
Last edited by msatter on Wed Jun 28, 2017 3:19 pm, edited 5 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Wed Jun 28, 2017 2:40 pm

David, please consider including blocklist.de's block list. I've been using both your blocklist and the one from squidblacklist.org for a little bit and so far the only major difference is from blocklist.de. If you add that then I can drop squidblacklist.org.
Done.
Thank you!

Unfortunately, it seems like you didn't get the same list as SBL (squidblacklist.org) uses, or you didn't merge the lists correctly. I've been tracking dropped packets by list, and I'm still seeing about 1 dropped packet from SBL's "blocklist.de" list for every 4 from your dynamicBlacklist. (I'm also seeing more hits from dshield, but that may just be a coincidence.) Please look into it when you have a chance.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 7:49 pm

That could be just the update timing. Currently, my list collects the data a 5am PST and rebuilds then. several of the sources also rate limit, but I may be able to push it and rebuild it ever 6 hours. that may keep them more in sync.

Okay, I changed the cron job to run every 6 hours.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 8:44 pm

I updated both the server and script to correct for the notification not displaying. I also changed the script so that the previous entries are not removed if the throttling kicks in. I would love to NOT have to throttle, but several people have set up their units to update every 5 minutes. at 2M each download multiplied by 40ish routers, every 5 minutes... Those routers were pulling 23G every day.

List is still dynamic and expires after 25 hours. This is to prevent false positives from hurting things for more than a day. (Some people were updating one a week, and complaining that false positives were not being removed quick enough)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 9:34 pm

I just ran to often the update so I got throttled and that is not a problem for me. I had to see what is happening and I adapted the script on my side to it and posted it here for you to see.
The messages in the error dynamic.rsc worked later and I incorporated that in my posting so that a clear message was left behind in the log and that not the blacklist was wiped before expiration time.

I now see why you are hesitant to keep the dynamic.rsc for a fast import on reboot despite it will be replaced by the next scheduled import. I wanted to combine the start-up schedule and the normal refresh schedule so that less administration is needed to setup and maybe the administration part can be automated depending on what kind of storage device is used.

Update: There goes the plan to have only one schedule: If interval is set to value other than 0 scheduler will not run at startup.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jun 28, 2017 9:58 pm

That's why I have always had two scheduled tasks. One for Startup and one every 24 hours.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jun 28, 2017 11:13 pm

Hi Dave,

So I have updated the start-up schedule so that dynamic.rsc files older than one day are not imported from flash/disk1/usb and the normal updateBlacklist script is run.
# Import Intrus Managed Filter Lists
# © 2016-2017 David Joyce, Intrus Technologies

##### DO NOT EDIT THE LINES BELOW ######
:local path "";
:local filename "dynamic.rsc"

##### Update your path, to where you have your storage
##### Examples: "disk1/"  or  "usb/" and the default is the temporary storage
#:local path "usb/"
:local path "disk1/"
:global datapath "$path$filename";

##### Delay for 10 seconds to allow the WAN to come online after a reboot
:delay 10;

##### Disable the log (We don't need 20k lines of adds and removes in the log
/system logging disable 0

# Declaring and filling the date1 and date2 variable for calculating the time difference
:global globalDaysDiff
:local time [/system clock get time];
:local date [/system clock get date];
:global date2 ("$date" . " " . "$time");
:global date1 [/file get [ find where name=$datapath] value-name=creation-time];

# This script calculates difference between two dates
/system script run diffDate

##### Import the downloaded blacklist
:log warning "Importing saved file $datapath as dynamicBlacklist...";
 :if ([:len [/file find name="$datapath"]] > 0) do={:if ($globalDaysDiff != 0) do={:log error "dynamicBlacklist $datapath to old for fast import."} else={/import file-name="$datapath"}};

# Download Blacklist if there is no dynamic.rsc present 
:if ([:len [/file find name="$datapath"]] = 0) do={/system script run updateBlacklist};

##### Turn the logging back on
/system logging enable 0
:if ([:len [/file find name="$datapath"]] != 0) do={:log warning "dynamicBlacklist $datapath imported."} else={:log error "Nothing happened and no protection by dynamicBlacklist provided!"};
Next the script diffDate that calculates the needed difference between the creation date of dynamic.rsc and the current time:
       ### calculate diff between two dates - yoan tanguy 2017

# format: :global date1 "jan/05/2017 10:00:00";:global date2 "may/15/2018 12:30:00";/system script run diffDate

       
       # expected date format : month/day/year hours:minutes:seconds (ex: mar/14/2017 09:13:54)
       :global date1
       :global date2
       
       
       # date to array format :
       # m a r / 1 4 / 2 0 1 7     0  9  :  1  3  :  5  4
       # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
       :local date1month [:pick $date1 0 3]
       :local date1day [:pick $date1 4 6]
       :local date1year [:pick $date1 7 11]
       :local date1hours [:pick $date1 12 14]
       :local date1minutes [:pick $date1 15 17]
       :local date1seconds [:pick $date1 18 20]
       
       :local date2month [:pick $date2 0 3]
       :local date2day [:pick $date2 4 6]
       :local date2year [:pick $date2 7 11]
       :local date2hours [:pick $date2 12 14]
       :local date2minutes [:pick $date2 15 17]
       :local date2seconds [:pick $date2 18 20]
       
       
       # month to decimal converter - https://forum.mikrotik.com/viewtopic.php?t=58674
       :local months ("jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec");
       :set date1month ([:find $months $date1month -1 ] + 1)
       :set date2month ([:find $months $date2month -1 ] + 1)
       
       
       :global globalDiff 
       :local yearDiff ($date2year - $date1year)
       :local monthDiff ($date2month - $date1month)
       :local dayDiff ($date2day - $date1day) 
       :local hoursDiff ($date2hours - $date1hours)
       :local minutesDiff ($date2minutes - $date1minutes)
       :local secondsDiff ($date2seconds - $date1seconds)
       
       
       # handle diff by converting in seconds, avoid negative hours/minutes/seconds (ex: jan/01/1970 09:00:00, jan/02/1970 08:00:00 must give 0 days 23:00:00 and not 1 days 0-1:00:00)
       # 1 days 23:30:10
       # 1*24*60*60 + 23*60*60 + 30*60 + 10
       # ($dayDiff * 24*60*60) + ($hoursDiff * 60*60) + ($minutesDiff *60) + $secondsDiff
       # ($dayDiff * 86400) + ($hoursDiff * 3600) + ($minutesDiff *60) + $secondsDiff
       :local secondsGlobalDiff
       :set secondsGlobalDiff (($dayDiff * 86400) + ($hoursDiff * 3600) + ($minutesDiff *60) + $secondsDiff)
       :set dayDiff ($secondsGlobalDiff / 86400)
       :set secondsGlobalDiff ($secondsGlobalDiff - ($dayDiff * 86400))
       :set hoursDiff ($secondsGlobalDiff / 3600)
       :set secondsGlobalDiff ($secondsGlobalDiff - ($hoursDiff * 3600))
       :set minutesDiff ($secondsGlobalDiff / 60)
       :set secondsGlobalDiff ($secondsGlobalDiff - ($minutesDiff * 60))
       :set secondsDiff $secondsGlobalDiff
       
       
       # check if date1 is older than date2 to avoid errors in calculation
       if ($yearDiff < 0) do={
           :return "error : date1 should be older that date2 (year check), exiting"
       } else={
           if ($yearDiff = 0) do={
               if ($monthDiff <0) do={
                   :return "error : date1 should be older that date2 (month check), exiting"
               } else={
                   if ($monthDiff = 0) do={
                       if ($dayDiff < 0) do={
                           :return "error : date1 should be older that date2 (day check), exiting"
                       } else={
                           if ($dayDiff = 0) do={
                               if ($hoursDiff < 0) do={
                                   :return "error : date1 should be older that date2 (hours check), exiting"
                               } else={
                                   if ($hoursDiff = 0) do={
                                       if ($minutesDiff < 0) do={
                                           :return "error : date1 should be older that date2 (minutes check), exiting"
                                       } else={
                                           if ($minutesDiff = 0) do={
                                               if ($secondsDiff < 0) do={
                                                   :return "error : date1 should be older that date2 (seconds check), exiting"
                                               }
                                           }
                                       }
                                   }
                               }
                           }
                       }
                   }
               }
           }
       }          
       
       
       # check if leap years - https://wiki.mikrotik.com/wiki/AutomatedBilling/MonthEndScript
       :local isYear1Leap 0
       :local isYear2Leap 0
       if ((($date1year / 4) * 4) = $date1year) do={
           :set isYear1Leap 1
       }
       if ((($date2year / 4) * 4) = $date2year) do={
           :set isYear2Leap 1
       }
       
       
       # find the right amount of days between 2 months
       :local daysInEachMonth ("31","28","31","30","31","30","31","31","30","31","30","31");
       :local daysInEachMonthLeapYear ("31","29","31","30","31","30","31","31","30","31","30","31");
       :local totalDaysBetweenMonths
       
       # same year; yearDiff = 0 so year1 = year2
       if ($yearDiff = 0 and $monthDiff >= 1) do={
           if ($isYear1Leap = 0) do={         
               for month from=($date1month - 1) to=($date2month - 1) step=1 do={
                   :set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonth $month])
               }
           }
           if ($isYear1Leap = 1) do={
               for month from=($date1month - 1) to=(($date2month - 1) - 1) step=1 do={
                   :set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonthLeapYear $month])
               }
           }
       }
       
       # different year, make concatenation of daysInEachMonth arrays first
       :local daysInEachMonthConcatenatedYears
       if ($yearDiff >= 1) do={
       
           for year from=$date1year to=$date2year step=1 do={
               # if leap year, concatenate the right daysInEachMonth array
               if ((($year / 4) * 4) = $year) do={
                   :set daysInEachMonthConcatenatedYears ($daysInEachMonthConcatenatedYears, $daysInEachMonthLeapYear)
               } else={
                   :set daysInEachMonthConcatenatedYears ($daysInEachMonthConcatenatedYears, $daysInEachMonth)
               }
           }
           
           # must add years count 
           for month from=($date1month - 1) to=(($date2month - 1)  + (($yearDiff * 12) - 1)) step=1 do={
               :set totalDaysBetweenMonths ($totalDaysBetweenMonths + [:pick $daysInEachMonthConcatenatedYears $month])
           }
       }
       
       :global globalDaysDiff ($totalDaysBetweenMonths + $dayDiff)
       
       
       # add leading zeros if necessary
       :if ($hoursDiff < 10) do={
           :set hoursDiff ("0" . $hoursDiff)
       }
       :if ($minutesDiff < 10) do={
           :set minutesDiff ("0" . $minutesDiff)
       }
       :if ($secondsDiff < 10) do={
           :set secondsDiff ("0" . $secondsDiff)
       } 
       :local d "d"
       :set globalDiff "$globalDaysDiff$d$hoursDiff:$minutesDiff:$secondsDiff"
       :put $globalDiff
So now maybe you can consider to keep the dynamic.rsc between updates and so avoid traffic by rebooting devices and people that run the update script every 5 minutes. The update script would than be updated with the same code and will warn people that they are wearing out their memory by those obsolete updates.

For other users of the script please wait until Dave had his say about this and wait for his updates and do not use this code unless you know what you are doing!!
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jun 29, 2017 3:05 am

Hi Dave,

So I have updated the start-up schedule so that dynamic.rsc files older than one day are not imported from flash/disk1/usb and the normal updateBlacklist script is run.
I think you need to check that you have a reliable date in the first place. It can be a while between boot up and acquiring the current date and time. I would not count on a simple delay being enough, I would sanity check the date.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jun 29, 2017 7:30 am

I think you need to check that you have a reliable date in the first place. It can be a while between boot up and acquiring the current date and time. I would not count on a simple delay being enough, I would sanity check the date.
I second that. If I've learned anything about RouterOS, it's that you can NOT trust the date and time at boot. I have several routers that take up to 20 minutes before the time is synced correctly.


As for bandwidth, it's not an issue for me. I have a gigabit connection with no metering. The router throttles each incoming IP to 100mbps. Also, the server compresses the list when it sends it, so it's typically only a few hundred kb. Also, I don't want to store the 2~4mb list on the flash because some of the units out there only have 16M and even then, those only normally have about 5M free. This leaves no room for updates. BUT - you are welcome to change the script in anyway you like, I just ask that the fetch isn't changed.

Actually, I was thinking of collecting Total and Free disk space - but I'm not sure how people will feel about that. I wonder if I can make a poll on the forum...
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 29, 2017 11:52 am

I am taking the time of the file that is downloaded by daily updateBlacklist so that is constant. Indeed the current time is a problem if it is not current.

About the compression by the server. I tested it by my own server and I did not see the device using compression and I have to look with Wireshark if that is also the case with your server.

The flash I had already a routine for to not keep the dynamic.rsc for those flash devices and it can be overruled by and variable set by user to ignore that and keep the dynamic.rsc anyway. I did not put that in this version.

I am going to look if the code can be more streamlined because I have the impressing I am doing thing twice.
Last edited by msatter on Thu Jun 29, 2017 12:45 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 29, 2017 12:34 pm

Mikrotik thought of the problem and came up with a solution:
Since v6.16 the current time is saved in the system configuration on reboot and on clock adjustment and is used to set the initial time after reboot.
Benefits:
Router doesn't need direct access to internet and public NTP servers
Allow control of a primary source of clock for your router on only two main routers (primary and secondary)
It can reduce traffic and the load of some public NTP servers by local time caching
Source: https://wiki.mikrotik.com/wiki/Setup_local_NTP_servers

We are thinking here in days not minutes and seconds to decide if a file should be declared outdated. We catching reboots and but also devices set to a higher scheduled update than a day.
Starts of a device can lead to false positives but that will be corrected on the next scheduled run of updateBlacklist.

Still to do flash only devices and automatic recognize flash (default), disk1 or USB. Check if scheduled can be imported set to time of 10 UTC + random time for spreading the load the server.

I have sent support a mail for clarification, on if fetch support deflate/compress and do use that advantage?

Update: I have tested it again and despite the site that checks if the connection is compressed gives an OK on the file the Mikrotik does not use it. I have forced the file to be transmitted compressed by Apache but the Mikrotik did not decompress it.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jun 29, 2017 4:37 pm

To automatic select the location for dynamic.rsc can be archived with the following code:
:if ([:len [/file find name="flash"]] != 0) do={set datapath "dynamic.rsc"};
:if ([:len [/file find name="disk1"]] != 0) do={set datapath "disk1/dynamic.rsc"};
:if ([:len [/file find name="disk2"]] != 0) do={set datapath "disk2/dynamic.rsc"};
:if ([:len [/file find name="disk3"]] != 0) do={set datapath "disk3/dynamic.rsc"};
:if ([:len [/file find name="usb"]] != 0) do={set datapath "usb/dynamic.rsc"};
:log info "Default location for Blacklist is: $datapath";
Extended with a check on free space and the minimal free space is 3MB to be selected.
:if ([:len [/file find name="flash"]] != 0)  do={:if ([/system resource get free-hdd-space] > 3000000)  do={set datapath "dynamic.rsc"}};
:if ([:len [/file find name="disk1"]] != 0) do={:if ([/disk get [ find where name="disk1"] value-name=free] > 3000000) do={set datapath "disk1/dynamic.rsc"}};
:if ([:len [/file find name="disk2"]] != 0) do={:if ([/disk get [ find where name="disk2"] value-name=free] > 3000000) do={set datapath "disk2/dynamic.rsc"}};
:if ([:len [/file find name="disk3"]] != 0) do={:if ([/disk get [ find where name="disk3"] value-name=free] > 3000000) do={set datapath "disk3/dynamic.rsc"}};
:if ([:len [/file find name="usb"]] != 0) do={:if ([/disk get [ find where name="usb"] value-name=free] > 3000000) do={set datapath "usb/dynamic.rsc"}};
:log info "Default save locationwith 3MB free  for Blacklist is: $datapath";
The Blacklist has become very long but it works and can say that every minute at least one or more block are made by the list on my Mikrotik.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jun 30, 2017 2:00 am

Result from today's Blacklist is 1808 packets caught by the list and above that one I filter connection made to services that I don't have and that were 1474 packets. So in total almost 3300 unwanted connections in one day and four hours. Most of the Blacklist packages came for port 25 to deliver unwanted stuff, so Spamassin is having now a kind of vacation. :-)
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jun 30, 2017 11:14 am

Collecting how many packets are blocked by the Blacklist:
#### Share how many packets are blocked by the Blacklist on your device
:local filterdownBlacklist "0";
:local rawdownBlacklist "0";
:local filterupBlacklist "0";
:local rawupBlacklist "0";

##### downstream
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterdownBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterdownBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawdownBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawdownBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

##### upstream
:if ([:len [/ip firewall filter find dst-address-list="dynamicBlacklist"]] != 0)  do={set filterupBlacklist [/ip firewall filter get [ find dst-address-list="dynamicBlacklist"] packets]}  else={set filterupBlacklist "0"};
:if ([:len [/ip firewall filter find dst-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find dst-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find dst-address-list="dynamicBlacklist"]] != 0)  do={set rawupBlacklist [/ip firewall raw get [ find dst-address-list="dynamicBlacklist"] packets]} else={set rawupBlacklist "0"};
:if ([:len [/ip firewall raw find dst-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find dst-address-list="dynamicBlacklist"]};

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for model $model $ver";
/tool fetch mode=https dst-path="$datapath" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid&filterdown=$filterdownBlacklist&rawdown=$rawdownBlacklist&filterup=$filterupBlacklist&rawup=$rawupBlacklist";
After collecting the numbers, each packets counter in Filters and RAW are reset to zero. In this way you won't get double countings on the next update of the Blacklist.

.....done enough for now and going to do other things. :-) ....added later the upstream so that is also counted..............
Last edited by msatter on Sat Jul 01, 2017 10:44 am, edited 3 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 30, 2017 5:53 pm

Today’s update is going to be huge. Not sure when I will push it it out though. I am rewriting the backend that builds the list. I will be pushing out 3 lists soon.

Small - about 750kb - intended for home users
Standard - about 2M - intended for businesses
Full - about 14M - intended for internet servers

Admins will need to choose wisely as the full list will fill the drive on many units and will cause out of memory panics on the small units.

The full list is currently about 114,000 entries. It pulls from many more sources and i would recommend building a whitelist for use with it as you may end up locked out or remote management if you are on a home IP.

The standard is what we have been using.

The small will average about 7000 to 8000 subnets and ips. Primarily C&C and botnets.

The new script will allow you to select the list you want.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Fri Jun 30, 2017 6:01 pm

were there thoughts about BGP feed?..
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jun 30, 2017 7:04 pm

were there thoughts about BGP feed?..
Too much work :)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 01, 2017 3:25 am

The new backend and script are live. Make sure you read the comments and select the correct script for your router.
*** DO NOT SELECT THE LARGE LIST FOR ROUTERS WITH LESS THAN 20M FREE DISK OR LESS THAN 256M Memory! ***

Recommendation:

Routers with 32M~128M memory - "small" list
Routers with 256M~512M memory - "medium" list
Routers with 1G memory and up - "large" list
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 01, 2017 10:32 am

The new backend and script are live. Make sure you read the comments and select the correct script for your router.
*** DO NOT SELECT THE LARGE LIST FOR ROUTERS WITH LESS THAN 20M FREE DISK OR LESS THAN 256M Memory! ***

Recommendation:

Routers with 32M~128M memory - "small" list
Routers with 256M~512M memory - "medium" list
Routers with 1G memory and up - "large" list
Thanks for your great work! I had to make a minor correction to version 2017.7.1d, and propose a modification to give more info to the person who is checking the log.
#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### medium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports
#### Begin download of current blacklist
:log warning "Downloading current $listSize sized Blacklist for this model";
/tool fetch mode=https dst-path="$datapath" \
url="https://mikrotikfilters.com/download.ph ... id=$softid";
:local fileSize [/file get [ find where name=$datapath] value-name=size];
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Sat Jul 01, 2017 10:37 am

Collecting how many packets are blocked by the Blacklist:
#### Share how many packets are blocked by the Blacklist on your device
:local filterBlacklist "0";
:local rawBlacklist "0";

:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

#### Begin download of current blacklist
:log warning "Downloading current Blacklist for model $model $ver";
/tool fetch mode=https dst-path="$datapath" \
   url="https://mikrotikfilters.com/download.php?get=dynamic&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid&filter=$filterBlacklist&raw=$rawBlacklist";
After collecting the numbers, each packets counter in Filters and RAW are reset to zero. In this way you won't get double countings on the next update of the Blacklist.
Hi, interesting scripting ...
I tried it as a separate script in the following way :
#### Share how many packets are blocked by the Blacklist on your device
:local filterBlacklist "0";
:local rawBlacklist "0";

:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

:log warning "Count filterBlacklist=$filterBlacklist rawBlacklist=$rawBlacklist";
BUT the counters are NOT reset and the log displays zeroes ...

any suggestions ?
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 01, 2017 10:55 am

Hi, interesting scripting ...
I tried it as a separate script in the following way :
:log warning "Count filterBlacklist=$filterBlacklist rawBlacklist=$rawBlacklist";
BUT the counters are NOT reset and the log displays zeroes ...

any suggestions ?
Try:
:log warning "Count filterBlacklist= $filterBlacklist rawBlacklist= $rawBlacklist";
yes, scripting in the Mikrotik is a PITA. I have that experienced that enough in the last week. ;-)

I have also updated the script to catch the upstream blocks: viewtopic.php?f=9&t=98804&p=605898#p605796 and the variable names changed accordingly.
Last edited by msatter on Sat Jul 01, 2017 11:00 am, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Sat Jul 01, 2017 11:00 am

tnx,

scripting can be a pain, sometimes it just does not work ...
Count filterBlacklist=0 rawBlacklist=30
it works ;-)
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 01, 2017 11:14 am

tnx,

scripting can be a pain, sometimes it just does not work ...
Count filterBlacklist=0 rawBlacklist=30
it works ;-)
This only for private use on the moment and if you only want to know the score remove the reset lines. When Dave is ready for more statistics then he can implement it.

I am still thinking about how to extrapolate the data when a there was a router reset in that period.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 02, 2017 11:49 am

So thinking on collection more information about the effectiveness of the Blacklist you can also collect the gmt-offset from /system clock so that you can see in which time zone the data is collected.

To get a idea how the effectiveness of Blacklist is between the previous download and the new download packets numbers could be collected every hour. Your can then see how the degradation is of the Blacklist and if there is a significant degradation decide to increase or decrease the updates. These should be only the downstream (incoming) figures and not the more private sensitive info of the upstream (outgoing). This can also, be a consideration with collecting the 24 hour data were I wrote about earlier.
:local timeOffset  [/system clock get value-name=gmt-offset];
The output is 7200 seconds so that is +2 hours in my case.

To get the only one or two variable(s) for the 48 (filter+raw) numbers to be transferred separately you can concatenate them in one or two strings so that you can transfer it when you collecting technical data of the router.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 02, 2017 6:12 pm

To collect and save the data so that it can survive a reboot an hourly script can be scheduled that executes the following script:
##### Read the save statistics
/import file blacklist.rsc
:global statsFilterBlacklist;
:global statsRAWBlacklist

##### Get current time and set filename to keep statistics
:local date [/system clock get date];
:local time [/system clock get time];
:local filename "blacklist.rsc";

##### Collect and reset packet counters
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={set filterdownBlacklist [/ip firewall filter get [ find src-address-list="dynamicBlacklist"] packets]}  else={set filterdownBlacklist "0"};
:if ([:len [/ip firewall filter find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall filter reset-counters numbers=[find src-address-list="dynamicBlacklist"]};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={set rawdownBlacklist [/ip firewall raw get [ find src-address-list="dynamicBlacklist"] packets]} else={set rawdownBlacklist "0"};
:if ([:len [/ip firewall raw find src-address-list="dynamicBlacklist"]] != 0)  do={/ip firewall raw reset-counters numbers=[find src-address-list="dynamicBlacklist"]};

#### Build new stats string
:local newStatsFilterBlacklist "$statsFilterBlacklist" . " " . "$filterdownBlacklist";
:local newStatsRAWBlacklist "$statsRAWBlacklist" . " " . "$rawdownBlacklist";
:local newStatDate ("$date" . " " . "$time");

:local writeString ":global $lastStatDate;" . ":global statsFilterBlacklist $newStatsFilterBlacklist;" . " " . ":global statsRAWBlacklist "$StatsRAWBlacklist";

/file set $filename content=$writeString;
Some thoughts. This script can possible collide with the updateBlacklist script and to notice that the blacklist.rsc can be deleted on reading for sending. This script should not execute on that instance and a new blacklist.rsc should be recreated with time plus the two strings without any numbers in it.

Example of the blacklist.rsc statiscs file:
:global lastStatDate "jul/02/2017 15:49:19"; :global statsFilterBlacklist "1 2 3 4 5 6 7 8 9"; :global statsRAWBlacklist "0 9 8 7 6 5 4 3 2 1";
The file supplies the last sample date and time and maybe a the gmt-offset can sync the data with other available data already in the database.

I have not tested the code so please check on syntax and typing errors.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Mon Jul 03, 2017 5:24 am

I have modified the scripts in a few ways and am publishing the modified scripts here for whoever wants them. @IntrusDave is welcome to incorporate them into his script or not.
  • Renamed globals so as not to interfere with other scripts
  • Added lots of error handling and corresponding error logging
  • Keep downloaded list for reinstall after reboot
  • Split script into 2 scripts, a download script and an install script, so I can just run the install script at boot time
  • Formatted for 1 statement per line, 2 space indent per block
Note that because the scripts use globals to communicate, they need policy permission in addition to the read, write, and test permissions that IntrusDave's script needs.

The update script downloads the list and calls the install script if successful:
# https://forum.mikrotik.com/viewtopic.php?f=9&t=98804

# Import Intrus Managed Filter Lists
# CUSTOMIZED by jgro, different globals, do not simply replace with update from Intrus
# © 2016-2017 David Joyce, Intrus Technologies

##### Update your path, is you are using a USB Flash or other storage
##### Examples:
##### "disk1/dynamic.rsc"  or  "usb/dynamic.rsc"  or  "dynamic.rsc"

:global intrusPath  "disk1/dl/dynamic.rsc"

#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### medium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports

:local listSize "medium"


###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 10 seconds to allow the WAN to come online after a reboot
#:log warning "Blacklist update in 10 seconds";
#:delay 10

##### Pull info to report to the server, used to download the correct list as well as stats collecting
##### software ID is used as the unique ID on the server side, this allows us to identify different 
##### routers behind a NAT router. Please do not remove it.
:local model    [/system resource get board-name]
:local version   [/system resource get version]
:local memory   [/system resource get total-memory]
:local uname   [/system identity get name]
:local softid        [/system license get software-id]

:if ($model = "CHR") do={
  :local temp [/system license get system-id]
  :for i from=0 to=([:len $temp] - 1) do={ 
     :local char [:pick $temp $i]
     :if ($char = "/") do={ :set $char "-" }
     :set softid ($softid . $char)
   }
}
:if ($model !="CHR") do={
  :global softid [/system license get software-id]
}

:local scriptVer   2017.7.1d

##### Scrub the device name and version to prevent http errors
:local name ""
:local ver ""

:for i from=0 to=([:len $uname] - 1) do={ 
  :local char [:pick $uname $i]
  :if ($char = " ") do={ :set $char "%20" }
  :set name ($name . $char)
}

:for i from=0 to=([:len $version] - 1) do={ 
  :local char [:pick $version $i]
  :if ($char = " ") do={
    :set $char "%20"
  }
  :set ver ($ver . $char)
}

#### Begin download of current blacklist
:local fileSize
:log warning "Downloading current Intrus dynamicBlacklist for this model";
:do {
  :do { 
    /tool fetch mode=https dst-path="$intrusPath" \
     url="https://mikrotikfilters.com/download.php?get=$listSize&model=$model&version=$ver&memory=$memory&id=$name&ver=$scriptVer&softid=$softid";

    :set fileSize [/file get [ find where name=$intrusPath] value-name=size];
    :if ($fileSize < 500) do={
      :log error "IntrusBL download is too small"
      :error "IntrusBL download is too small"
    }
  } on-error={
    :log error  "FAILED to download Intrus dynamicBlacklist"
    /system script run "play-alert-sound"
  }

  :if ($fileSize > 500) do={
    /system script run import-intrus-block-list  
  }
} on-error { 
 :log error "FAILED to update Intrus dynamicBlacklist";
}
The import script does the import, and can be run at boot time (if you have saved the list somewhere) before the network even comes up:
##### Update your path, is you are using a USB Flash or other storage
##### Examples:
##### "disk1/dynamic.rsc"  or  "usb/dynamic.rsc"  or  "dynamic.rsc"

:global intrusPath 

:log warning "Starting import of Intrus dynamicBlocklist"

# intrusPath  set by code that does the fetch
# set fallback in case it is unset
:if ("x$intrusPath " = "x") do={
  :set intrusPath  "disk1/dl/dynamic.rsc"
  :log warning "Importing dynamicBlacklist from fallback location: $intrusPath"
}

:if ([/file find name=$intrusPath ] = "") do= {
  :error "FAILED: Importing dynamicBlacklist: file not found: $intrusPath "
}

##### Disable the log (We don't need 20k lines of adds and removes in the log)
:log warning "Disabling info logging while loading dynamicBlacklist...";
:log info "Disabling info logging while loading dynamicBlacklist...";
/system logging disable 0

##### Find the "dynamicBlacklist" entries and remove them
:local status "failed"
:local fileSize [/file get [ find where name=$intrusPath] value-name=size];
:if ($fileSize > 500) do={
  :log warning "Removing expiring address-list entries...";
  /ip firewall address-list remove [find list="dynamicBlacklist"]

  ##### Import the downloaded blacklist
  :log warning "Importing downloaded dynamicBlacklist from $intrusPath ";

  do { 
    /import $intrusPath
    :set status "success"
  } on-error { 
    :log warning "FAILED to import $intrusPath "
  }

####### Find and remove the downloaded file
###:log warning "Removing dynamicBlacklist temp file...";
###/file remove [find name=$intrusPath ]

} else= { :log warning "Intrus blacklist file $intrusPath too small ($fileSize), aborting" }

##### Turn the logging back on
/system logging enable 0
:log warning "info logging enabled"
:log info "info logging enabled";

:if ($status = "success") do={ 
  :log warning "Intrus dynamicBlacklist Update Complete.";
} else={
  :error "FAILED to update Intrus dynamicBlacklist"
}
The script also calls a "play-alert-sound" script for a big problem. You can make an empty one or use this one stolen from Dave:
:log warning "Playing alert sound"
 :for i from=1 to=3 step=1 do={
 :beep frequency=550 length=494ms;
   :delay 494ms;
   :beep frequency=400 length=494ms;
   :delay 494ms;
 }
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jul 03, 2017 12:35 pm

Thank jgro for the scripts!

Some remarks on the startup scheduler script: if the router restarts or first starts there are no dynamic addresses present in the addresslist so cleaning them out is not necessary, save a bit of code so. If you do not have the dynamic.rsc present on disk1 then you will have no protection until the next scheduled run updateBlacklist script and you can catch that and call the updateBlacklist script directly.

Now Dave has fixed the path to dynamic.rsc in 2017.7.1d, the global variable is not needed any more and can be defined as local in every script.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
ilivlad
just joined
Posts: 14
Joined: Tue Mar 12, 2013 2:02 pm

Re: Blacklist Filter update script

Mon Jul 03, 2017 2:38 pm

#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### megium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports

:local listSize "small"

Small typo, megium ...
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jul 03, 2017 3:14 pm

I had a reply from Mikrotik about RouterOS being able to deflates/compressed traffic when fetch download a files. Sadly it will not be compressed so the whole size of the file is traffic.

I went through different options how to reduce traffic and the quick and easy one is removing the comment in the medium and large file and that gives a reduction in traffic of over 20% assuming that the users of the medium and large file know what that addresslist is named dynamicBlacklist stands for...you can shorten also that name "dynamicBlacklist" and saves an other 5 to 10 percent.

Next thought is to only supply the addresses itself and that would shrink the size of the medium file from 4.1MB to 729KB but then we have to split it up in more than 177 files due to 4096 bytes String limit present in RouterOS.

Then we can go for a more complicated mutation file that contains the to remove and to add addresses but to keep that in sync is really complicated. To easy that you can put the mutations for 48 or 36 hours in the file and so you can avoid becoming out-of-sync.

quick setup how it can work.

On start or restart:
- no dynamic.rsc present on disk1/ then do FULL update with download list and apply
- when dynamic.rsc is present on disk1/ and older than a day then do Full update
- when dynamic.rsc is present on disk1/ and not older than a days import it and get mutation file and import that also

Scheduled update:
- no dynamic.rsc present on disk1/ then do FULL update with download list and import but before erase the old address-list in memory
- if dynamic.rsc is present (age not important) download mutation file and apply it.

The mutation file can be mutBlack.rsc and has to be downloaded always and is erased after download and application.

This also means that there is no time out given on each address...or you can set a timeout of one or two weeks and then have a forced full update on that set time.
Last edited by msatter on Mon Jul 03, 2017 3:31 pm, edited 10 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Mon Jul 03, 2017 3:16 pm

#### Select your list size ####
#### large - 10 to 20 Megabyte download - 100k+ entries - intended for firewalls protecting internet servers
#### megium - 2 to 5 Megabyte download - 40k+ entries - intended for corporate networks
#### small - 200 to 700 Kilobyte download - 2k+ entries - intended for networks with no open ports

:local listSize "small"

Small typo, megium ...
Small mistake.. its normal :)

:local listSize "medium" is work.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 03, 2017 6:51 pm

please keep in mind that with all the chaos in the world now, the list is regenerated every 4 hours. I don't recommend holding on to an older list for more than 8 hours. Also, I have no bandwidth caps so I have no issue with people downloading several times a day - But I don't want it abused and pulled every 5 minutes. My router does limit the connection speed to 100mbps, so no one can saturate the full gigabit WAN.

I corrected my typo. Also changed the one global to local. (It was global on my dev unit because another script was using it too)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 03, 2017 6:58 pm

I went through different options how to reduce traffic and the quick and easy one is removing the comment in the medium and large file and that gives a reduction in traffic of over 20% assuming that the users of the medium and large file know what that addresslist is named dynamicBlacklist stands for...you can shorten also that name "dynamicBlacklist" and saves an other 5 to 10 percent.
I can not change the format, as there are still several hundred units that have never (and likely will never) update the script. The first version of the script removed the entries based on the comment (RouterOS was unable to remove by list name at the time) So removing the comments would stop them from working. Versions over the last year remove based on the list name. Again, many have never and will never update.

I prefer to not leave them out to dry. Once I have at least 80% updated, then I can start making changes.

I have also had the thought of pushing out script updates in the .rsc, but I feel that is overstepping and many admin would be VERY upset that the list file had ANYTHING other than just address-list entries.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 03, 2017 7:05 pm

Next thought is to only supply the addresses itself and that would shrink the size of the medium file from 4.1MB to 729KB but then we have to split it up in more than 177 files due to 4096 bytes String limit present in RouterOS.
With more than 80% of the routers pulling the list only having a MIPS CPU, passing only the IPs in CIDR format would cause 100% for more than 10 minutes. (up to 30 minutes in some of my testing). During this time, the router would experience dramatic pocket loss. It also complicates the script. Same reason I won't do BGP - it's just far too complicated for most to setup.

I believe the best solution is to have MikroTik update the fetch so it supports compression.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 03, 2017 10:04 pm

Updated the script with minor bug fixes, speed ups, and more detail when run from the console.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jul 03, 2017 11:23 pm

I had a quick peek at 2017.7.3f and I have to admit that I am a bit lost on it.

Update: Before the v [ScriptVer] would undergo a cleaning of spaces which are replaced by %20 for use in the URL which is not not more done. I have still the word (testing) in my version string with a space in front.
Last edited by msatter on Tue Jul 04, 2017 1:06 am, edited 5 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jul 04, 2017 12:14 am

On a other note. Thanks for the correction of the type and I still only download the new dynamic.rsc every 24 hours.

The format change is clear and you can change the format of the large download because the "stuck" routers don't know of the existence of that huge file. So the small or medium file can't be changed and I assume you deliver the medium file to those routers. And yes, changing the script without the acknowledgement by the administrators is a bad idea.

Mikrotik changes fetch so that it supports compression is something that is not going to happen. They can transmit the already compressed .npk files so need to change anything. Lets see if the famous 7 or 8 version of RouterOS is going to support compression. There is really no need on their side...... :twisted:

So my my last option was to work with mutations files instead of complete updates. This will leave the backbone of the updates intact, and you still produce besides the dynamic.rsc for all the routers, besides the small and large version, that contains the complete list. That way of working is not changed in any way and the "stuck" Mikrotiks will not notice anything because for them noting is changed in fact.

What is now extra are files that contain the removals and updates for the last 48 hours. By this you can reduce traffic and up the frequency.

It is not the only the traffic you generate that I am thinking of but also my traffic and also important that I have on my little but fast RB750Gr3 a windows of 20 seconds that I am not protected by the list because it is busy removing the list and adding the updated list. So remove and add mutations will do away with that exposure. And I am using the medium list and how long is exposure when using the large list.

Yes it is more work for you to also generate the mutation files but all the way is it a win win situation.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 04, 2017 2:11 am

I had a quick peek at 2017.7.3f and I have to admit that I am a bit lost on it.

Update: Before the v [ScriptVer] would undergo a cleaning of spaces which are replaced by %20 for use in the URL which is not not more done. I have still the word (testing) in my version string with a space in front.
It's not a problem. The only issue is with the CHR. The CHR license often has a "/" in it, which needs to be replaced or encoded.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Tue Jul 04, 2017 3:55 am

please keep in mind that with all the chaos in the world now, the list is regenerated every 4 hours. I don't recommend holding on to an older list for more than 8 hours. Also, I have no bandwidth caps so I have no issue with people downloading several times a day - But I don't want it abused and pulled every 5 minutes. My router does limit the connection speed to 100mbps, so no one can saturate the full gigabit WAN.

I corrected my typo. Also changed the one global to local. (It was global on my dev unit because another script was using it too)
@IntrusDave, Your primary post still says your list is updated only once a day and I was still under the impression pulling it more than 4 times a day will result in being banned. Please update your recommendation and limit if needed.

Would it break the old scripts for you to include in the comments your source of the block (e.g. spamhous, malc0de, blocklist.de, your internal network monitoring, etc.)? It would be helpful to know that. If it would break the old scripts, now that you have added a new parameter for size, perhaps you could implement new formats based on the requested size or script version. You have added small, medium, and large, but the old scripts are just getting a default list because they have not specified a size, so you could keep the old format as default but use a new format for scripts that specify a list size.

Right now I have a problem in that a shared server I am using is on your blocklist and so I cannot connect to it. Of course I can whitelist the server, but still it would be super helpful to know why it is on the list and who to talk to about getting it removed. I would PM you but it seems this forum does not allow that.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jul 04, 2017 10:50 am

Second on that to have the initial posting, listing used lists to built dynamicBlacklist. If you want to know which lists are blocking a specific address then there are public pages on which you can search.

Example: http://whatismyipaddress.com/blacklist-check
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 04, 2017 7:46 pm

Unfortunately it's not possible to tell the source of the block. The lists are generated from 12 different high profile blocklists, as well as a network of over 200 routers. Once the server has all of the sources, the IP addresses are extracted and then aggregated into a new list that has the subnets merged. Example, my mikrotiks log port scans, and they often use different source IPs from the same subnet. Each router may record ¼ of the sources. If IP's on one router are logged as 10.10.10.1 through 10.10.10.127 and then another router logs 10.10.10.128 though 10.10.10.254, then the server will merge them into 10.10.10.0/24. This cuts the total list from 800,000 IPs down to 200,000 IPs. Also, all sources may contain many duplications. Once they are in CIDR format, sorted and merged, then there is no way to tell where the address came come.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Tue Jul 04, 2017 10:48 pm

Unfortunately it's not possible to tell the source of the block. The lists are generated from 12 different high profile blocklists, as well as a network of over 200 routers.
Fair enough. So I can do my own investigation, would you please post (and keep updated) the block lists you are including? Of course, you do not need to disclose anything proprietary, but where you are using public lists, it would help to know.

Also, keep in mind for future reference my point that you now really have 4 versions of the list, small, medium, large, and "", with all the un-updated routers getting "", so you have a way to support the old scripts while making changes for the new scripts.

Thank you again for providing this service!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 04, 2017 11:35 pm

Older client scripts requested "dynamic" (the "get=dynamic" in the URL) requests for the old "dynamic" are currently being redirected to "medium", and will soon be switched to an automatic selection based on the CPU and memory.

I'll be honest, I have no interest in maintaining an up to date list of sources. Many of the existing blacklists aren't even used because my router network takes priority and often end up having the same contents. I've stated here many time before - this list and the script are built for my own routers that I manage. I provide it as a *free* service to the MikroTik community to repay the help they have given me in the past. Again - free. I have never asked for a donation or subscription fees. That said, it is what it is. If a recommendation helps my clients or myself, I will likely implement it. If I see no benefit, it will not likely be added.

(not that I haven't thought about charging for it. with upwards of 9000 routers pulling the list every 24 hours, the servers, rack space, bandwidth, and time cost quite a bit now)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jul 05, 2017 10:15 am

First of all thanks for removing the comment string and that will save you 20% in traffic and RAM in the devices. And second but more important, many thanks for keeping us safer with your work and it is much appreciated.

I did spend quite a few hours if not days lately thinking how to improve the list and finding out how to script stuff. That was all new and learned a lot. We have gotten a lot of tools from Mikrotik but like to have some more so importing large files is more efficient.

If I get up with something new that could improve your great work I will put it here and I hope you will manage to get the traffic down and the import/updates even more efficient. I saw that the "stuck" ones are getting the message now to update their script sooner than later.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jul 06, 2017 1:16 am

I've stated here many time before - this list and the script are built for my own routers that I manage.... If a recommendation helps my clients or myself, I will likely implement it. If I see no benefit, it will not likely be added.
That is completely fair and understandable and I thank you again for providing this free service. I have tried to contribute to your effort with more efficient code and better error handling in the same spirit.

Having a list of the sources you draw on would help me identify the source of false positives which then might help you and your clients eliminate them. Not only did I get a (probably) false positive for one of our shared servers I also got a false positive for one of Speedtest.net's servers. The harm from these self-inflicted denials-of-service is probably going to outweigh (for me, at least) the protection provided if it continues like this.
Last edited by jgro on Thu Jul 06, 2017 1:46 am, edited 1 time in total.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 1:32 am

I doubt they are false positives. Speedtest.net servers are NOT controlled by them. They are 3rd parties that are often shared hosts. If they get blocked, it's because they have allowed a host, shared host, or infected host to remain online. Even Amazon's AWS gets blocked because spammers will "rent" a VM, send a ton of spam and then switch IP's. Amazon does not do anything about it, unless you report it AND they see it in progress. Microsoft has been blocked because the Windows Update CDN was hosting malware in the form of rogue ads. Some ISP's get blocked because they are complacent with their networks being used for attacks.

The best corse of action is for you to create your own whitelist, and give that list an Accept rule before the blacklist drop rules.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 06, 2017 1:52 am

I have a request. I was testing with a more informative disabling and enabling from the log entries and when I did not disable and enable again as normal is done on an import I did not get only the normal logging but not the huge numbers of the removals and adds to the list in the log. I was very nice to see that other services were still logging during import.
#/system logging enable [find topics="info"];  and disable logging:  #/system logging disable [find topics="info"];
I had defined ! firewall on info, removed that again and applied but still no removal or add logging by script.
I see still the blocking by the dynamicBlaclist so the firewall is making log entries.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 2:05 am

I have a request. I was testing with a more informative disabling and enabling from the log entries and when I did not disable and enable again as normal is done on an import I did not get only the normal logging but not the huge numbers of the removals and adds to the list in the log. I was very nice to see that other services were still logging during import.
#/system logging enable [find topics="info"];  and disable logging:  #/system logging disable [find topics="info"];
I had defined ! firewall on info, removed that again and applied but still no removal or add logging by script.
I see still the blocking by the dynamicBlaclist so the firewall is making log entries.
I'll include that in the next update.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 6:06 am

New script updated. I'm now including a change log on in the first post.
Last edited by IntrusDave on Thu Jul 06, 2017 9:05 am, edited 1 time in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jul 06, 2017 9:01 am

New script updated. I'm not including a change log on in the first post.
I'm not sure if that was directed at me, but in case it was, I want to say I was never asking you to include a change log. What I wanted was for you to keep up-to-date whatever is true about the current system, things like when it is generated and how often people can and cannot download it, etc.

Thanks.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 9:07 am

ROFL oops. I fixed it. Should have been NOW not NOT

s/not/now/
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 06, 2017 9:36 am

I have a request. I was testing with a more informative disabling and enabling from the log entries and when I did not disable and enable again as normal is done on an import I did not get only the normal logging but not the huge numbers of the removals and adds to the list in the log. I was very nice to see that other services were still logging during import.
#/system logging enable [find topics="info"];  and disable logging:  #/system logging disable [find topics="info"];
I had defined ! firewall on info, removed that again and applied but still no removal or add logging by script.
I see still the blocking by the dynamicBlaclist so the firewall is making log entries.
I'll include that in the next update.
Thanks for the changelog and that save al lot of scrolling the the code in posting one to see what has changed.

The improved disabling and enabling from logging the removal and reading of the dynamicBlacklist. However I don't disable and enable the logging anymore and have not the 45000 plus log entries. I am on 6.40rc32 and you could test it with older RouterOS versions if the mengling with the log is still needed on those?

I go a reasonable fast quadcore router but I need a delay of 10 seconds before reading the saved file on reboot. I see the address-list being filled in memory after their deflation on normal update. After being readed all in they are displayed in the box. I am going to test if the reading can be done in the background while the deletion is running also and see if those are going to bite each other.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jul 06, 2017 10:46 am

ROFL oops. I fixed it. Should have been NOW not NOT
Glad we are all laughing. :D

Here is a "best practices" tweak: save and restore log state rather than reset it
#instead of: /system logging set numbers=0 topics=info;
:local logTopics [/system logging get number=0 value-name=topics] 
/system logging set number=0 topics=info,!firewall,!system

#...

/system logging set number=0 topics=$logTopic
For my setup, I had to include "!system" to the first setting because sometimes the adds and removes show up there instead of under firewall.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 06, 2017 11:06 am

So that works and you seek the items counter going up and down when the removal of the dynamicBlacklist is running and at the same time delayed by 10 seconds (in my case) the import. The big advantage is that the address-list is filled with the blacklist during removal and import. It might to be a to heavy load for not that powerful Mikrotiks. An other thing that I saw thet the reading of the file is only done by one core.

The total import takes longer, but you have no gap any more between removal and import of the blacklist.

Magic line that replaces the removal and import lines from updateBlacklist script if there is a dynamic.rsc file that is bigger than 400 bytes:
:execute "sub-script-remove"; :delay 10; :execute "sub-script-import"
sub-scipt-remove script:
/ip firewall address-list remove [find list="dynamicBlacklist"];
I can't use :while any more as speed up, removal for and maybe the old fashion :for will work also for older RouterOS versions.
sub-script-add script:
/import file-name="disk1/dynamic.rsc";
Going to have breakfast now....I must be working this all out in my sleep I think.

Update: the first fully automated update with the magic line worked as expected and the blacklist was renewed.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 06, 2017 1:38 pm

ROFL oops. I fixed it. Should have been NOW not NOT
Glad we are all laughing. :D

Here is a "best practices" tweak: save and restore log state rather than reset it

For my setup, I had to include "!system" to the first setting because sometimes the adds and removes show up there instead of under firewall.
I am really puzzled why I don't any logging of the adding of the addresses in my Mikrotik while I don't disable/enable logging in any way.
 /system logging> print
Flags: X - disabled, I - invalid, * - default 
 #    TOPICS                                                                                  ACTION
 0  * info                                                                                    memory
 1  * error                                                                                   memory
 2  * warning                                                                                 memory
 3  * critical                                                                                echo
And to combine both of best practice and clarity:
:local logTopics [/system logging get [find topics="info"] value-name=topics]
 /system logging set number=0 topics=info,!firewall,!system
 .
 .
 /system logging set [find topics="info"] topics=$logTopic;


Position 0 is always the case for info in RouterOS but stating the name also makes it clearer when reading the script.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
w177f
newbie
Posts: 27
Joined: Fri Jun 30, 2017 2:21 pm
Location: Dublin, Ire
Contact:

Re: Blacklist Filter update script

Thu Jul 06, 2017 4:58 pm

In your posted code, you have the delay set to 0. It's fine in the hosted code at https://mikrotikfilters.com/updateBlacklist.rsc
###### DO NOT EDIT BELOW THIS POINT ######

##### Delay for 5 seconds to allow the WAN to come online after a reboot
##### You can change this if you need more or less time. Loading the list
##### on reboot will not work without this delay.

:local d 0;
:put "Delaying $d seconds to allow WAN to stabilize.";
:log warning "Blacklist update in $d seconds";
:delay $d;
Cheers,

Will
MTCNA | MTCRE
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 5:53 pm

In your posted code, you have the delay set to 0. It's fine in the hosted code at https://mikrotikfilters.com/updateBlacklist.rsc
Thank you. Corrected.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 6:02 pm

I've started work on "2.0.0". I will no longer be updating this branch.
The new branch (going with more normal version numbers) will be more modular and, if installed with the included installer script, it will keep itself updated with the current version and will only update the blacklist if it has changed. It won't matter if you run the update once a minute or once a day, if the list hasn't changed, it will not update.

I will be using a custom DNS server to inform the script about the current available script version as well as the current version and number of changes of the blacklist.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 06, 2017 6:17 pm

Great new that you are going to take the next step, to have better control of and more flexible way of initiating updates by means of DNS.

I have managed this morning to not have any need any more for smaller files now I can remove and import the dynamicBlacklist at same moment. This reduces the exposure during renewing the Blacklist. This may work for me and similar devices but older equipment can have problems with doing two things at the same time.

I want to share what I noticed today. After erasing the Blacklist the memory is still reserved in RouterOS and not given back to the pool. Importing the Blacklist will reuse that reserved memory so no loss there of space. After the next start the pool will be back to it original size.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 06, 2017 6:24 pm

Great new that you are going to take the next step, to have better control of and more flexible way of initiating updates by means of DNS.

I have managed this morning to not have any need any more for smaller files now I can remove and import the dynamicBlacklist at same moment. This reduces the exposure during renewing the Blacklist. This may work for me and similar devices but older equipment can have problems with doing two things at the same time.

I want to share what I noticed today. After erasing the Blacklist the memory is still reserved in RouterOS and not given back to the pool. Importing the Blacklist will reuse that reserved memory so no loss there of space. After the next start the pool will be back to it original size.
I'm going to test with the various devices I have. I may just include code to make a choice between one-at-a-time and both-at-once.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 07, 2017 4:49 am

Bad news. Removing in the background and importing in foreground doesn't work. The background removal is executed on the same CPU core, so overall speed is only a few seconds difference. The issue I am seeing on all of the multicore routers is that the delay needed before starting the import is 10~20 seconds, depending on the model. RouterOS is able to import much faster than it is able to find and remove the old items. From that point, the delay needed ends up leaving you just as unprotected as just removing first and then importing.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:29 am

To eat the donut and to have the donut :) I propose to prepare import script as follows:
/ip firewall address-list 
:do { add address=X.X.X.X list=blackmail timeout=25h } on-error={set [find where address=X.X.X.X] timeout=25h}
:do { add address=Y.Y.Y.Y list=blackmail timeout=25h } on-error={set [find where address=Y.Y.Y.Y] timeout=25h}
I know, I know ... it makes it bigger and import is slower but with one step we will have added new IPs, old ones included in the current update will stay in place with updated timeout and all adresses from old list not included in the new update will disappear soon naturally with their counting down timeout so the list will be self-cleaning.

We are ALL THE TIME protected.

EDIT:
Code should be:
/ip firewall address-list 
:do { add address=X.X.X.X list=blackmail timeout=25h } on-error={set [find where address=X.X.X.X list=blackmail ] timeout=25h}
:do { add address=Y.Y.Y.Y list=blackmail timeout=25h } on-error={set [find where address=Y.Y.Y.Y list=blackmail ] timeout=25h}
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:35 am

I'm sure that would work, but with 200,000 entries in the "large" list, that would make the file size almost 40M.

I suppose I can generate two sets of lists, one the other way and one this way..?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:42 am

So make two scripts: "safe&bigger" and "smaller&unsafe". User could decide what to import.
With your new mechanism to download update only if it is changed it should be no problem with bandwidth.
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:45 am

My concern isn't so much size, but time.
With the majority of routers pulling the list being single core, my tests have shown that an import / update like that causes dropped packets.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:51 am

So therefore ... as you know the old list and the new one:

A. Prepare current address list on server
B. Make diff of old one and current and then prepare such script according to the result
/ip firewall address-list
# Update timeouts of addresses from old list as they are on the current so they stay and just need new timeout
set [find where address=X.X.X.X list=blackmail ] timeout=25h
set [find where address=Y.Y.Y.Y list=blackmail ] timeout=25h
....
# add new address
add address=Z.Z.Z.Z list=blackmail timeout=25h
...
Real admins use real keyboards.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:55 am

Thanks BartosP and I had already a look at on-error and the way you use it there will be no removals unless they are included in the update script also.

Update: I see that now that use the time out to automaticity remove them if not updated.

I am not yet ready with the magic line as I call it. A removal takes 36 seconds of which the first 10 are to clear memory and the 26 seconds are used to clear the visible list or what I hope also effective addresses. Now the import takes in total 62 seconds of which the first 19 are used to import the list into memory and the next 43 second to put them in to the list.

If the address-list is only cosmetic then we have 10 seconds for clearing memory and 19 for reading into memory makes 29 seconds
Sequential is that 36 seconds for removal and 19 for reading into memory which makes 55 seconds. So the saving is almost 45%.

The only way to test if the protection is faster up with the magic line is to setup a machine to ping from a address with a matching IP that is in the prepared Blacklist. Then you can see if the exposure is shortened or lengthened.

The BartosP line can could be used in a mutation file to avoid collisions which will stop the script.
Last edited by msatter on Fri Jul 07, 2017 11:59 am, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:56 am

I thought of that. Diff won't work. Everyone would have to always be current. Some will update as soon as an update is available. Others will only update daily. Some even update weekly, even though the list expires after 24 hours.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 07, 2017 12:03 pm

How did you got the idea Dave, to shorten the commands to two characters so that you saves again almost 50% in the transfer?!?!?!?!

Does this also work: ald a1.0.128.0/17 t 1d Nope it did not but this did: add li dynamicBlacklist ad 1.0.128.0/17 ti 1d

An BIG advantage is that the free memory (RAM) went from 163MB free up to 192MB free so almost 30MB less wasted space with the new format. This also improves the read speed and so less exposure.

I am happy that your Blacklist is getting more and more efficient all the time. I think you will see a seizable drop in traffic generated by all of us. :D
Last edited by msatter on Fri Jul 07, 2017 2:50 pm, edited 5 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jul 07, 2017 12:10 pm

Minimum version:
:do {/ip fi ad add address=101.231.46.34 list=blackmail ti=25h} on-error={set [fi wh address=101.231.46.34 list=blackmail] ti=25h}
BTW Dave,

If someone is not updating everyday then router needs to load full update script. There is nothing to do with it.
All others could use smaller script with timeout update and adding new entries.
Real admins use real keyboards.
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Fri Jul 07, 2017 2:11 pm

kind of:
* saving list version in global var;
* sending that var in fetch URL;
* server decides whether it should send full list or just incremental one.
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 07, 2017 2:47 pm

When there was reboot in meantime the global var is lost and the the full update has to be provided.

I am still and fan of keeping the blacklist file an reuse it on reboot. It will be overwritten when a full or like you suggest a incremental update is provided.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Fri Jul 07, 2017 3:30 pm

Well, it's possible to use some 'special' entry in blacklist (like '255.255.255.255/32 disabled=yes') and save the version in its comment
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jul 07, 2017 4:13 pm

Change from
add li=dynamicBlacklist ad=1.0.128.0/17 ti="1d"
to IDDBL=IntrusDaveDynamicBlackList
add l=IDDBL a=1.0.128.0/17 t=25h
saves statistically 27% of size but breaks current filters as list name changes
Real admins use real keyboards.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 07, 2017 5:03 pm

Change from
add li=dynamicBlacklist ad=1.0.128.0/17 ti="1d"
to IDDBL=IntrusDaveDynamicBlackList
add l=IDDBL a=1.0.128.0/17 t=25h
saves statistically 27% of size but breaks current filters as list name changes
You can slim it even more and be backwards compatible with this:
a l=dynamicBlacklist a=xxx.xxx.xxx.xxx/xx t=1d
and the suggested one:
a l=IDDBL a=1.0.128.0/17 t=25h
An other thing on removing and importing. After reboot I have 193MB free after import and when I use remove it drops 188MB. Importing it drops to 177MB and then is becomes stable in 188,177,188,177......
Last edited by msatter on Fri Jul 07, 2017 5:24 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Fri Jul 07, 2017 5:22 pm

I have an idea :idea: for you:
:local l "dynamicBlacklist"
/ip f a
a l=$l a=127.0.0.1
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Jul 07, 2017 5:30 pm

The International Microtik Obsfucated Code Contest :-)
Real admins use real keyboards.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 07, 2017 5:42 pm

I have an idea :idea: for you:
:local l "dynamicBlacklist"
/ip f a
a l=$l a=127.0.0.1
:shock:

And so, we saved more than 3MB per transfer and with 1,3MB now we are a lot closer to the 129KB of a deflated/compress original file. That is a really good result in a short time archived. Thanks to every one and Dave can now focus on the DNS version after first checking this out of it also works with the older devices.

I hope that Dave will not put the comment line back.
Last edited by msatter on Fri Jul 07, 2017 8:47 pm, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 07, 2017 7:41 pm

I have an idea :idea: for you:
:local l "dynamicBlacklist"
/ip f a
a l=$l a=127.0.0.1
I like this. going to see how much it slows things down.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 07, 2017 10:00 pm

I think I've found a viable balanced solution. I'll be posting the first beta of the new system later today.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 07, 2017 11:12 pm

Setting my alarm clock for tomorrow morning. :)

I am not in a hurry and take the time you need.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 08, 2017 2:01 am

I am thinking out loud to reduce the extra data is caused the way Bartoszp did propose the updates. Now every line has an double address due to the way it works.

What if the 56 addresses that can fit in array then a :for reads out the array and imports the addresses. You need little more than 803 of those blocks in a RSC file to cover 45000 address.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 2:11 am

too much work for the slow units. I think I have a solution.
It's slow... but there is no "unprotected time".
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 3:22 am

New script is live. grab the installer in the first post.
Make sure you remove any old schedules and scripts.

Remember this is an RC. it may have bugs.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 3:28 am

So - this process is VERY slow. The initial import is quick, but the updates take a very long time. The upside is that the entries are left in place so that their is no gap in protection.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 7:11 am

not very happy with the speed... Still trying to figure a good way to do this.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 08, 2017 10:17 am

Thanks Dave, I did not yet ran it and I was puzzled by only having the hourly and reset file after executing the install script. Then changed the install script so that it will not erase the other files.

I see in there that a specific port is used to connect to the DNS. This can be the cause that the other two files/scheduler are not kept because my Mikrotik will block a such direct call and I have to allow that port to go out.

I will check that later today when I have time again.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
bajodel
Long time Member
Long time Member
Posts: 545
Joined: Sun Nov 24, 2013 8:30 am
Location: Italy

Re: Blacklist Filter update script

Sat Jul 08, 2017 1:22 pm

@IntrusDave - I was testing/reading your last (beta) 'blacklistUpdate' script, at the end (quoted section below)
..[CUT]..
# Turn the logging back on
:if (\$blDebug = 1) do={ \$log t=\"Enabling firewall info logging...\"; }
/system logging set numbers=0 topics=\$cl;
..[CUT]..
I cannot figure out what is the [ $cl ] variable after 'topic'

---- |edit| ----

Opss.. found it >> :local cl [ /system logging get number=0 value-name=topics ] , it was 'obfuscated' in installation script by tab escaping (\t) :(
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 08, 2017 7:01 pm

Thanks Dave and I have the Beta up and running including the scheduling. I had to make a tiny hole in my firewall to use your DNS to get the serial number....however I did not had any hits on the new Blacklist until I remembered that the name of the list had changed. :oops:
After adapting the firewall rules it caught it first identified trespassers and dropped them.....I mean ignored them.

Reboot worked and the list was imported again after downloading.

Compliments on the new format of the address list and with clever use of the "limited" tools in scripting, and you came up with an elegant solution...again. :D
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 7:25 pm

Thank you. It's definitely still a beta.

I'm really not happy with the update process. I like using the "functions" to make the list smaller, and it works well on my x86 and CHR boxes, but even my CCR1016 in my datacenter struggles with the process.

using BartoszP's concept of "add, update if error" is great, but REALLY slow. I tried flipping it around.. Trying to update first, then add if it's an error, didn't work. The It wouldn't report an error if the entry didn't exist, and therefore wouldn't run the "add" in the on-error section.

I'll have to try that concept again. I have to admit, i had just smoked a huge joint and was pretty high - so I may have made a mistake.

I may just end up writing in some code to use the new process on high end CPUs, and stick to the remove-then-add on the low end units.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 9:41 pm

were there thoughts about BGP feed?..
Okay, I give. Can you point me to a basic setup for BGP. I don't even know where to start.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 08, 2017 9:59 pm

Hi Dave, if you type ASN in the search box above the thread you will see two postings by Zerobyte explaining it's basics last year.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 10:17 pm

Well, right off the bat, BGP fails me.
Peers do not support dynamic IPs.
This is a show stopper for me, as most of the routers I deal with are dynamic.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 08, 2017 10:37 pm

I will say that the BGP method would be simpler to manage over a large distribution, and the implementation on the client side is brain-dead simple:
enable BGP (if not already using BGP) with any private ASN other than 64567. (or just use their real ASN if they're already running BGP).
in-filter=accept all -> action=set route type=blackhole
out-filter=discard all
enable strict RPF in IP options.

Can you post an export rsc to give me a basic BGP setup to drop incoming packets from 10.252.0.7/32?
I'm hoping you can give me a starting point so I can understand how this works.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 09, 2017 12:50 am

I stopped the beta version and reactivated the current version. The problem was that only a few hundred addresses had a timer running and all the others were on zero. In the log I did see less and less hits on blacklist so I got suspicious and checked the list and scrolled down.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 09, 2017 2:56 am

It looks like we’ve uncovered a bug. The timers on dynamic entries aren’t removing the entries when they reach 0.

I’ll change the script to do the remove and add when I get home tonight.

Going to sit in a pool for the evening. It’s 110°F right now. I can’t think anymore.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 09, 2017 10:34 am

That is realy hot and we have now a moderate temperature of 23 Celsius and at night is 14 Celsius so no complaints from my side ;-) Except I woke up early to have an other look and going now to have breakfast but not after I want to share this and I had a look at your latest version of the Bartozp/Dave way initiated by Chupaka of using the :do.

So for the current version you can reduce the size of the dynamic.rsc even more to under 1MB instead of the old size of well over 4MB:
# Medium Blacklist Generated on Sa=t Jul  8 02:00:16 PDT 2017 by Intrus Technologies
:global blSerial 60
:global blDate 1499504416
:local i do={/ip f a a l=dynamicBlacklist t=25h a=$a }

$i a=1.0.128.0/17
$i a=1.1.128.0/17
$i a=1.2.128.0/17
$i a=1.4.128.0/17
$i a=1.9.69.35/32
.
.
.
I now know why I don't get any entries in the log....that is because it are dynamic entries in the address-list. I had to remove a list which was static and so always loaded on reboot. When removing that I got a entries in the log file.

So if this goes up for all the devices then meddling with the log is not needed any more.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 09, 2017 7:51 pm

Now that the import/export is moved to the server side script generation, I can make changes on the fly without the need to update the script.

So, I've returned to the old "remove, then add" method. The "add, or update" was never completing on low end routers. Even CCR's were taking 30~45 minutes.
I have a few other ideas, but my hands are tied until MT fixes the dynamic entries not timing out.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 09, 2017 7:54 pm

Oh, also... The changes resulted in new list sizes. the "small list" (#1) is only 46kb now (down from 118kb). Medium (#2) is 860kb (down from 2.2M), large is 4M (down from 12M).
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sun Jul 09, 2017 8:48 pm

Another idea:

1. Assume that we full table which expires soon

2. As Dave knows diff from last to current list so he prepares script which starts with
# remove entries removed by diff
/ip firewall address-list
:do { remove [find where address=192.168.1.0/24 list=dynamicBlacklist]}
:do { remove [find where address=192.168.2.0/24 list=dynamicBlacklist]}
.....
all entries which are "valid" now will be removed ... we are still protected against all old "offenders"

3. Next part is to update full table ... for existing entries there is no change and their timer goes down, new ones are added
# update list with new one entries
/ip firewall address-list
add address=192.168.3.1 list=dynamicBlacklist ti=25h
add address=192.168.3.1 list=dynamicBlacklist ti=25h
.....
we do not update timeout during this step to speed it up

4. update all timeouts with
:foreach i in=[ find where list=dynamicBlacklist ] do={set $i ti=25h }
5. Voilà ... the list is updated.

The script should declare some timestamp which informs about last imported address list.
It could be
add address=20170709 list=dynamicBlacklistTimeSstampFullTable
add address=20170709 list=dynamicBlacklistTimeStampDaily ti=25h

yes, yes ... ROS allows it.
Each day we can update daily timestamp after list update.
After reboot there is no any dynamic entries so there is no dynamicBlacklistTimeStampDaily so therefore we know that we need to
download full script but ... as we do not remove this full script then we could try to import this full script with
:set ts :put [/ip firewall address-list get [/ip firewall address-list find where list=dynamicBlacklistTimeSstampFullTable] address]]
:do {/import file-name=$ts.rsc }
Checking existence of dynamicBlacklistTimeSstampFullDaily we know if it succeded.
If yes then import diff script and update the table.
If not then we need to import brand new full script, import list and update dynamicBlacklistTimeSstampFullTable timestamp.

Of course all code obsfucation could be used to shrink files.

EDIT:

Meanwhile Dave has changed script generation.
My idea seems to be obsolete but .... do not use timout for dynamicBlacklist and the idea seems to be possible for realiztion.
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 09, 2017 8:56 pm

The issue with that is that with 200,000+ entries the [find where address=xxx.xxx.xxx.xxx] is really REALLY slow. Each list causes RouterOS to check EVERY entry each time. so you are looking at 200,000*200,000 loops. That's 40+ Billion loops.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sun Jul 09, 2017 8:58 pm

I know, but using diff file makes these numbers much, much lower.
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Jul 09, 2017 9:07 pm

diff would work, if I can guarantee that every router will get every update. If someone misses an update, the whole process is screwed. It would require a complete do-over on the backend, and I would have to build the scripts in realtime to deal with differences in versions.

Still far too many only update once a week. I simply can not assume that every router will update every hour.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sun Jul 09, 2017 9:25 pm

No.
If we save last full update locally then we import it and download only diff file which name is determined from locally saved file date/name.
If there is no diff_today_saveddate.rsc then we import full list.
Effect = full list every weekend
diff files = every day.
Real admins use real keyboards.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 09, 2017 9:35 pm

I wrote earlier how you can do that but that needs the full update to remain on the device until the next full update. In time in between is lives from update files add/remove.
You can number the update files internal and if the downloaded update file is out of sequence, missed one or more update, a full update is done.

You still provide full update and update fles. You can sequence the updates in 6, 12, 24 and 30 hours so it will cover all schedules. So you end up with providing four update files and one full file.

If a update is missed and it not more than 30 jours you can provide the 30 hours file so that not a full remove and full read have to be done.

See it as an moving wave and were the with of the wave are the updates and if drop of then you get a kick in the butt, launching you again on top of the wave.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jul 10, 2017 2:08 am

If you don't use a timeout then the list becomes static, generates log entries on add/removing etc., however that eliminates the reloading of the list on reboot. There should be a startup script checking if the if the retained list is up-to-date and if not update it. If update is not possible for any reason then should be an warning that the blacklist is not working optimal.

Then the elephant in the room. When the blacklist is static, where is it saved and because it is even kept on reboot or power-fail I assume it will be in the flash and that is the place where we don't prefer to have it.

Having it dynamic offers a lot advantages as, no need for tinkering with the log, and the timeout time can be set further in future.
You have to re-sync forcefully at an set time or when the previous count of lines in the dynamicBlacklist is different than it should be on Dave's full update list. In account have to be taken the frequency the client uses to get his updates.
If an 30hour update is requested than a second count have to be done after the update. If still the list is out of sequence, different total, than a full update is initiated.....the kick on top of the wave.

The client, is so self healing if there is a problem.

The size of the updates are not a that big problem any more and now we can tackle the blackout of the blacklist.

I have not update the scripts or blacklist today because I had again 6 strikes.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jul 10, 2017 5:57 pm

At the end of the blTemp.rsc you swap the old list name for the new list name but only for incoming traffic. If someone also uses the list for outgoing traffic then the last four lines should be:
/ip firewall filter set src-address-list=intrusBL [find where src-address-list="dynamicBlacklist"]
/ip firewall raw set src-address-list=intrusBL [find where src-address-list="dynamicBlacklist"]
/ip firewall filter set dst-address-list=intrusBL [find where dst-address-list="dynamicBlacklist"]
/ip firewall raw set dst-address-list=intrusBL [find where dst-address-list="dynamicBlacklist"]
I am back on the Beta and saw that it was set to 1 hour refresh and very nice to see that you use the minimal format for the file and it is now considerable smaller than before.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Mon Jul 10, 2017 10:41 pm

Can you post an export rsc to give me a basic BGP setup to drop incoming packets from 10.252.0.7/32?
I'm hoping you can give me a starting point so I can understand how this works.
A few things to note - the implementation is different depending on whether the client and server peer using eBGP or iBGP.
iBGP doesn't need any kind of route filtering at all to distribute a blackhole route IF you have a pre-configured "blackhole" next hop IP (e.g. 169.254.255.254. . .)
This is because iBGP does not modify the "next hop" addresses of prefixes within your AS. So if router1 originates/learns-from-eBGP/redistributes a prefix, with next hop of a.b.c.d, then advertises this prefix to router2, router2 will not install the route using router1 as the next hop - it will install the route with a.b.c.d as the next hop. (recursive lookup)

So if all of your routers are configured with a standard static route:
dst=169.254.255.254/32 type=blackhole
Then all you need to do to blackhole a destination within your AS using iBGP is to inject a route into your BGP table whose next hop is 169.254.255.254.

The filter rules I gave are used when you have an eBGP session - basically, your server would be its own ASN (choose some private ASN)
Then any network that wishes to subscribe to the blacklist will need to use some other ASN than the one you chose. (these can even be public if the remote user has a live ASN)
the server's configuration is a bit different than this script for the clients, but a Mikrotik configured as a "client" would need a configuration basically as follows:

eBGP multihop enabled (your server needs this enabled as well - because by default eBGP only works with directly-connected routers using the IP addresses of their interfaces on that connection - i.e. IP TTL = 1 for eBGP unless you enable multihop)
The config I give here assumes that the client trusts the server not to make mistakes - this is probably not best practice for any operators who wish to run this on a network of much size due to the risk of having something valid get blackholed... but getting started, it's easy to follow along when the configuration is simple.
/ip settings
set rp-filter=strict
/routing bgp instance
set default as=65530 router-id=10.10.10.10
/routing bgp peer
add in-filter=BlackholeDestination multihop=yes name=BlacklistServer1 out-filter=NoRoutes \
    remote-address=192.0.2.100 remote-as=65000 ttl=default
/routing filter
add action=accept chain=BlackholeDestination set-type=blackhole
add action=discard chain=NoRoutes
Obviously this is the very most basic minimum to get it working on the client side.
On the server side, you'd want to use the NoRoutes in-filter because you don't want any clients of the list distribution to ever be able to inject anything into the list, accidentally or maliciously.

The server's configuration might be a tad different based on how you wanted to implement it - say as a standalone server whose entire purpose is to publish the blacklist, but not to participate in any global routing. In this case, you could just configure it to redistribute static, and then create a static blackhole route for each blacklist prefix.
Finally, you could use a community ID on the server to make sure that the server will only send stuff that you intended to be black hole routes (i.e. if you use "redistribute static" then EVERY static route's going to get advertised, not just the blackhole routes)
/routing bgp instance
set default as=65000 router-id=192.0.2.100 redistribute-static=yes
/routing bgp peer
add in-filter=NoRoutes multihop=yes name=Client1 out-filter=OnlyRedistributeBlackholes \
  remote-address=10.10.10.10 remote-as=65530 ttl=default
/routing filter
add action=discard chain=NoRoutes
add action=accept bgp-communities=65000:666 chain=OnlyBlackholes
add action=discard chain=OnlyBlackholes
To blackhole a prefix on the server, add a static route like this:
/ip route add bgp-communities=65000:666 distance=1 dst-address=172.16.66.0/26 type=blackhole

The "type=blackhole" is not actually important on the server. What IS important is the bgp-communities=65000:666 part, because the out-filter for each peer should be "OnlyRedistributeBlackholes" - which matches routes having this community applied to them.

65000:666 -> 65000 = your server's ASN, and 666 is arbitrary - it's whatever value you want to use to mean "blackhole community" (it's like saying "category 666")

Hope this is enough to get your experiments off the ground.
Before you go production with anyone who's using it in production, you'd probably want to put some sanity filters on your outputs - e.g. if you have a list of "never blackhole these things" - you should make a set of static routes to those blocks but use a different community, such as 65000:777 and insert a rule earlier in the output chain which has action=discard if bgp-communities=65000:777 matches.
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jul 11, 2017 10:14 am

For the routers that also use IPv6 the blacklist does not cover it. I have a rule for the IPv6 firewall that catches a lot illegal requests. Lets say you have a webserver and mailserver running then you can use this line. You can adopt the port to your liking and if you don't have any services running then you omit the ports. The interface I use for reaching the internet is pppoe-out1 and if you use any other port than you have to use that one.

Put at the top of the RAW filtering.
chain=prerouting action=drop in-interface=pppoe-out1 dst-port=!25,80,443,554 log=yes log-prefix="TCP hacker" protocol=tcp tcp-flags=syn,!fin,!rst,!ack,!urg,!ece,!cwr 
UDP is more difficult and you could identify all the ports that are used to acces the internet. You need port 53 and 123 for DNS and NTP port 546 and 547 for obtaining your IPv6 address from you ISP and if you have also VOIP then a range of number of ports have to be allowed 5060-5070,7078-7098.
chain=prerouting action=drop in-interface=pppoe-out1 src-port="" dst-port=!53,546,547,5060-5070,7078-7098 port="" log=yes log-prefix="hacker drop" protocol=udp
A big advantage is that IPv6 is more difficult to just scan for hosts because of the sheer numbers of IPv6 addresses so the attacker has to use know addresses of hosts that run services.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue Jul 11, 2017 3:29 pm

Hi all,

Im using this list and sometimes my ip addreses comes with list. what should i do ? and further more some customers cant see their cameras when their ip comes in blacklist they cant connect to their system.
Im using only this rule at raw table to drop;
chain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL 
one of my customer has an internet at another country and his ip also in black list and he can not accses his system. I would like to learn that, with this rule in raw table Im thinking I only block incoming from these src address list but i cant ping any of them also. should i select an in-interface in here ?

Thanks
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jul 11, 2017 3:43 pm

Hi all,

Im using this list and sometimes my ip addreses comes with list. what should i do ? and further more some customers cant see their cameras when their ip comes in blacklist they cant connect to their system.
Im using only this rule at raw table to drop;
chain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL
one of my customer has an internet at another country and his ip also in black list and he can not accses his system. I would like to learn that, with this rule in raw table Im thinking I only block incoming from these src address list but i cant ping any of them also. should i select an in-interface in here ?

Thanks
Hi Amt, have a look at an posting by Dave and put the whitelist above blacklist lines

viewtopic.php?f=9&t=98804&p=602090&hili ... st#p602090
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue Jul 11, 2017 4:04 pm

Hi msatter,

thank you very much for your quick answer.
I solve the problem as here but I wonder that when i add my ip block in here like 123.123.32.0/22, is this not make problem to me ? because when i add rule to accept for my ip blocks, blacklisted ip's can attack to my ip range if I true.

further more I wonder that also why i cant ping any of this blacklisted ip. if i disable that rule
chain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL
there is no problem. when i enable rule ping stop. Im trying to drop blacklist ip to can acsses to me. but i want to acsses to them. I need it cause some of my customers have some VPN, DVR and they cant acces to them.

Thanks.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jul 11, 2017 4:31 pm

You have to simplify that for me and I can make head or tail of you text.

So some general stuff, if you want to access your devices outside the your local network you have to look at the dst-address-list (goes outside) and not at the src-address-list (comes from outside). Secondly you have to take in account that once a connection is established RAW filtering is not checked any more, if you have Fasttrack activate (filters).

Your customers have also to whitelist their devices/sites if those are blocked by external addresses in the blacklist.

If you want to be pinged then only accept in the whitelist the ping (ICMP) and all the other protocols are dropped by the blacklist after that. The blacklist rule you state drops all protocols, not only TCP, UDP, ICMP.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue Jul 11, 2017 5:55 pm

Thanks for your explanation msatter. thanks a lot.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 11, 2017 6:53 pm

Hi msatter,

thank you very much for your quick answer.
I solve the problem as here but I wonder that when i add my ip block in here like 123.123.32.0/22, is this not make problem to me ? because when i add rule to accept for my ip blocks, blacklisted ip's can attack to my ip range if I true.

further more I wonder that also why i cant ping any of this blacklisted ip. if i disable that rule
chain=prerouting action=drop log=no log-prefix="" src-address-list=intrusBL
there is no problem. when i enable rule ping stop. Im trying to drop blacklist ip to can acsses to me. but i want to acsses to them. I need it cause some of my customers have some VPN, DVR and they cant acces to them.

Thanks.
Use a filter drop instead of a raw drop.
chain=Filter action=drop in-interface=wan0 connection-state=new src-address-list=intrusBL log=no log-prefix=""
Select your WAN interface for the in-interface. Select "new" for connection state. This will only drop new incoming connections and it will not drop outgoing. This will NOT protect you if you connect to an infected server.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Tue Jul 11, 2017 10:21 pm

Hi IntrusDave,
This will teach me new thing and near what i was tring to learn. Im tring to understand why outgoing traffic also blocked. İf i true, on the rule i should select in interface(that should be wan) and connection state should be new. but my english not enoguh to explain what i need and what i want :( :(. Thanks a lot for sharing your experince with me. And thanks for sharong your blacklist filter update script with us :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 11, 2017 10:24 pm

The RAW rule is not blocking outgoing traffic. But it IS blocking the response from the the remote address.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Tue Jul 11, 2017 11:16 pm

Hi Dave, still learning here also so when a package is destination to out the WAN it goes first through the RAW however the interface on which it should to out is not yet set. This is done in the NAT and that the last step of the travel through the device. Is my assumption on this correct?

If I want to use the blacklist in RAW for traffic destination to the outside then I do not set the out interface and to protect against accidents I leave always local traffic destination to local through in case a glitch is occurring in the blacklist.

The updates are now running smoothly and the list keep up-to-date. It is a lot quieter on the moment and I have just over 100 hits per day on the blacklist. The filter I mentioned earlier had over 1600 in the same time and it is before the blacklist.

As test I am going to put the blacklist to see how the score is then.

Update: the score after a little more than an hour is Blacklist filtered 28 and let 47 through which are caught by my filter rule in RAW. The most tried port is 23 and then on a far distance port 22. The Blacklist is great way to protect services because my filter only looks at tried services that I don't offer.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jul 12, 2017 4:02 pm

I have question on BGP. Is it possible to have clients to sent back IP addresses of attacking addresses up to the main BGP and that after certain threshold the address will be merged into that BGP. An client is only allowed to sent once in the 24h the same address to that you get a balanced threshold for an address to be blacklisted.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4051
Joined: Wed May 11, 2011 6:08 pm

Re: Blacklist Filter update script

Wed Jul 12, 2017 9:55 pm

I have question on BGP. Is it possible to have clients to sent back IP addresses of attacking addresses up to the main BGP and that after certain threshold the address will be merged into that BGP. An client is only allowed to sent once in the 24h the same address to that you get a balanced threshold for an address to be blacklisted.
You should be very very careful in allowing clients to inject blackhole information into a publicly distributed list. One malicious actor could very easily black list tons of legitimate addresses, either by directly advertising addresses into the master list (if he controls a subscribed client), or by sending spoofed packets to a client that will trigger the spoofed source's IP into an automatic blacklist.

I would suggest that at the very minimum, if clients are allowed to inject blackhole info, that your main server should apply a special community to these prefixes so that other clients can filter out "community blackhole destinations" - probably a good solution would be to tag them as you receive them from clients with a community that's unique to each client so that you can learn the same prefix multiple times from multiple clients - and have your own server which watches the "client-added" prefixes. If it sees some threshold number of instances of the same prefix, then this is akin to "confidence points"
When given a spoon,
you should not cling to your fork.
The soup will get cold.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 12, 2017 10:04 pm

For the time being, I'm very happy with the list system and BGP will not be implemented anytime soon. Currently, about 60% of the systems pulling the blacklist are dynamic IP. That number could be MUCH higher, as some ISP's don't force an IP change unless the modem is offline for a few hours.

I will play with BGP for the large list, as it should only be used on routers in front of internet servers. But until I have a firm understanding, I will not even do a beta test.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 13, 2017 1:11 am

The ones that can supply suspected addresses are few but paced on different spots on the globe. The most of the ones I catch are looking for tenet access and I think of filtering them out to have a better impression of what is more serious.

The addresses that are over the the threshold should have max lifetime of one day or shorter.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Chupaka
Forum Guru
Forum Guru
Posts: 8308
Joined: Mon Jun 19, 2006 11:15 pm
Location: Minsk, Belarus
Contact:

Re: Blacklist Filter update script

Thu Jul 13, 2017 11:51 am

I searched a bit about BGP daemons with dynamic neighbours support, and there are only patches for bird/quagga, not merged into mainline.

Ciscos does support dynamic neighbours, but it's a bit overcomplicated to use dedicated hardware for such things :)
Russian-speaking forum: https://forum.mikrotik.by/. Welcome!

For every complex problem, there is a solution that is simple, neat, and wrong.

MikroTik. Your life. Your routing.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Thu Jul 13, 2017 2:33 pm

I was still thinking about the changing time on find of an present address. When i do a find address=xxx.xxx.xxx.xxx then it takes a few seconds. Once I have the requested the .id then it is instantaneous.
:put [find address=x.xx.172.2];                                                                                                
*154c53
:do { /ip firewall address-list add timeout="25h" list=intrusBL address=x.x.172.2} on-error={set *154c53 timeout=25h};

result:
151 D intrusBL x.xx.172.2 jul/13/2017 11:52:58 1d59m50s
How do I call directly the .id this because I think that on the moment of the error the .id is filled and so the set can use the .id (index) directly to change the timeout.

id (internal ID) - hexadecimal value prefixed by '*' sign. Each menu item has assigned unique number - internal ID
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 14, 2017 10:03 pm

released version 2.0.1 with minor improvements.

Old version will not longer function soon. Please use the install script in the first post to update.
Auto-Script-Update is being testing in house. I hope to have the routers updating themselves next week.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 15, 2017 10:02 am

Thanks Dave and I found a minor glitch with the usage of blScriptVersion which is global but also used as local sv.
:local	blListName "intrusBL";
:global	blScriptVersion	"2.0.1";
:local	cc	$blCount;
:local	bn	[ $urlEncode t=[/system resource get board-name ]];
:local	rv	[ $urlEncode t=[/system resource get version ]];
:local	tm	[ /system resource get total-memory ];
:local	cl	[ /system logging get number=0 value-name=topics ]
:local	bs	[ :resolve server=$blDnsHost server-port=$blDnsPort domain-name=127.0.0.3 ]
:local	sv	$blScriptVersion
or do
:if ($blDebug = 1) do={
	:put	"System ID: $si";
	:put	"Board Name: $bn";
	:put	"RouterOS Version: $rv";
	:put	"Total Memory: $tm";
	:put	"Script Version: $sv";
	:log 	info "System ID: $si";
	:log 	info "Board Name: $bn";
	:log 	info "RouterOS Version: $rv";
	:log 	info "Total Memory: $tm";
	:log 	info "Script Version: $blScriptVersion";
An update on the scores: intrusBL medium list: 257 drops, port 22-23: 536 drops and my services rule: 376 drops and the order is first in list IntrusBL, port 22-23 and then the service rule over an period of 19 hours. In a few day's I am going to try the BIG list to what the score is then.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 15, 2017 1:40 pm

I was trying out the list with my other HEXGr2 with 64MB RAM and noticed that I got standard the small list because of the he less memory. I tried the medium list and after two runs I had still 24MB left of the 64MB. The CPU does not like the importing and exporting and stayed at 100% all the time on delete and import.

I think that it is save to assume that the improved way of the list is build-up, is more memory friendly and that the medium can also be available for the 64MB models and up.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 15, 2017 5:14 pm

in my testing, the 64M units are struggling with anything other than the small list. I'm seeing about 60% of the 64M units pull the medium list 10+ times in a row. That is telling me that the 64M units are having kernel panics and rebooting.

At this time, the server is now forcing the small list on 32M and 64M, medium for 128M and 256M, and large for 512M and up.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 16, 2017 11:41 am

Request to be able to allow or disallow the automatic blacklist update script in the addresslist rsc file from the config file.
:global blScriptVersion;
if ($blScriptVersion != "2.0.1") do={
:local sourceServer "https://mikrotikfilters.com/";
:local scriptName "blInstaller.rsc";
.
.
:do { /ip firewall address-list remove [find where list=dynamicBlacklist] } on-error={}
/system script run blacklistUpdate
} else={ :put "script is current" }
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Jul 16, 2017 11:47 am

in my testing, the 64M units are struggling with anything other than the small list. I'm seeing about 60% of the 64M units pull the medium list 10+ times in a row. That is telling me that the 64M units are having kernel panics and rebooting.

At this time, the server is now forcing the small list on 32M and 64M, medium for 128M and 256M, and large for 512M and up.
So I can't even test with the huge...really huge file how the filter scoring is? ......it is really huge! With the medium list in....not that huge...I have 187MB free RAM.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sun Jul 16, 2017 12:21 pm

Plenty of free RAM in 951Ui-2HnD ... 60% free.
IBL.PNG
You do not have the required permissions to view the files attached to this post.
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 17, 2017 2:46 am

I've shut down the old service (pre 2.0 script).

I found that several users were leaching the large list and rebranding it as their own. They were also trying to probe the server side for exploits.

Again, I offer my list as a free service to the MikroTik community. If people continue to abuse it, I will shut it down completely.
(I've also added one of the offending IP's to the blacklist... I'm sure that will get some attention)
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
xlighting
just joined
Posts: 6
Joined: Wed Apr 02, 2014 6:08 pm

Re: Blacklist Filter update script

Mon Jul 17, 2017 5:55 am

Hi Dave:
I'm not able to see any log output when using scheduler, even if the blDebug is set to 1 in .conf file...
however if I manually run "blacklistUpdate", the logging is shown... is there anything I can do to show the log?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 17, 2017 6:24 am

You should be getting the same logging both ways. If not, check the schedule policy and the script policy and make sure all of the boxes are checked.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jo2jo
Forum Veteran
Forum Veteran
Posts: 958
Joined: Fri May 26, 2006 1:25 am

Re: Blacklist Filter update script

Mon Jul 17, 2017 9:54 pm

thank you so much for putting all this work into this FREE project, that you offer for us. Its really great!

Today i started getting the alarm on my rb3011 (great idea to singal updates btw!) , so i checked the log and went and found the required script update.

So im now running the latest from OP of this thread (i updated it as of today, my script has :global blVersion "2017.07.05a"; in the script im running)

however, when running this updated script, im still getting the audio beep alarm, and log msg "Script is outdated"
any ideas?
tks


EDIT: Looks like the fix was to NOT replace the old script with the NEW code, but rather to remove OLD script and run the new Auto Install / Updater script from scratch. (im assuming that the initial way i did it was failing maybe bc i didnt have a blacklist.conf file as thats a new feature) -- either way all is good and its updating again!


EDIT: Feature Request: maybe at some point you could add a line in blacklist.conf for "Additional, user specified, Logging Rules to TEMP disable during bl updates" , so that users can optionally specify ADDITIONAL /sys logging rule numbers to have TEMP disabled (or have !firewall temp appended) during updates... obviously you have the 0 "memory" rule taken care of , but i often have Firewall action=remote rules as well (pointing to offsite syslogd servers) , and it would be nice to disable these as well (i had done this myself on the old script, but i assume any mods i make now to the actual script will be overwritten by the auto-update feature, thus blacklist.conf would be the new, proper place to define such a feature)

thanks again for this GREAT script/feature.
:beep :beep :beep
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Mon Jul 17, 2017 11:09 pm

I've shut down the old service (pre 2.0 script).

I found that several users were leaching the large list and rebranding it as their own. They were also trying to probe the server side for exploits.

Again, I offer my list as a free service to the MikroTik community. If people continue to abuse it, I will shut it down completely.
(I've also added one of the offending IP's to the blacklist... I'm sure that will get some attention)
Hi Dave, I support you block anyone that tries to collect your list and misuse it, however 'poisoning' your list in a way is not good practice.

I hope not that abuse continues because what you setup, with some help of considerable amount of others as I can see in the thread, is used and recognized by many as an valuable addition. Even so valued that they steal the list.

In the new setup you have much more control over the usage of the list and see how that works out now.

Thanks for all the work and effort you put in this all and we keep in putting in complai...suggestions to improve it even more. ;-)
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Tue Jul 18, 2017 1:28 am

Greetings Dave, my thanks for such an important job and sharing it with us. My support in making decisions to block abusive users. Thank you!
RB3011 UiAS (arm)
Best regards
Ricardo
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 18, 2017 2:11 am

So two things... Some users are simply blocked at my firewall, and now two users have been added to the list itself. I don't see this as "poisoning" as they are the ones that were actively trying to find security holes. (They have been trying SQL injections) Given that they are active attacks, I see them as no different than the botnets and spammers that the list is intended to block.

I find it VERY sad that MikroTik users on this forum would stoop this level. And frankly, if the USA passes this current bill that will allow sys-admins to "hack back" then the next time they pull my list, it will include a command to clear their config. until that time - they will remain blacklisted.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Tue Jul 18, 2017 10:35 am

Hi Dave,

first my compliments for your work and all the effort you put into this !!!

I have updated a couple of different devices and most went fine without a glitch.
Now I am struggling with a RB2011UiAS-2HnD and it does not work ...
I installed the blinstall script, run it and the 2 scripts and 2 schedulers are installed.
The counters on both scripts are updated so something happens.
BUT : no download of the tmp file, no new list with ips to block AND no ENV variables visible ... AND nothing in the log, even on general debug.

Any suggestions ?

Regards,

Eddie
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Jul 18, 2017 7:12 pm

Check for the Scheduler and Script Policies. Make sure that all of the boxes are marked.
You do not have the required permissions to view the files attached to this post.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Tue Jul 18, 2017 7:41 pm

Hi Dave,
tnx for the hint, that did it. For some reason the scripts where created with only read, write, test ...

It works now

Eddie
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 19, 2017 12:54 am

Just released 2.0.2 with minor bug fixes. Run the auto-update/install script to update.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
planetcoop
Member Candidate
Member Candidate
Posts: 115
Joined: Thu May 15, 2014 2:32 pm
Location: Sacramento, CA

Re: Blacklist Filter update script

Wed Jul 19, 2017 1:37 am

Just released 2.0.2 with minor bug fixes. Run the auto-update/install script to update.
Im first on 2.0.2.1 :) Dave if you don't mind, please reach out to me: ------@planetcoop.com I am in the general forum running a btest server with Tom and I am seeing real benefits to this list on spam and attackers of the btest.
Last edited by planetcoop on Wed Jul 19, 2017 3:55 am, edited 1 time in total.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 19, 2017 2:11 am

Just pushed out 2.0.2.2 :)
new auto-script-update script is included. It pulls the current version from the server and updates if needed.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
foxxiu7
just joined
Posts: 5
Joined: Sun Aug 25, 2013 3:30 am

Re: Blacklist Filter update script

Wed Jul 19, 2017 5:57 am

Hi Dave,

First of all thanks for an amazing job and all effort you're putting into this. It's working just fantastic on my hAP-ac router.

A small idea to consider: how about extending firewall filter rules with autoblock functionality for intruders trying to get to a router or network? A dynamic list with banned IP's trying to do excessive pings, scanning ports, attempting DoS attacks, etc?
I'm using your rules from post #2 and having this autoban functionality will just make them more complete and make the network more secure, I think.
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Wed Jul 19, 2017 8:36 am

Hi Dave,
tnx for the updates, it seems to work fine on my RB2011 and my RB1100.

But, I have a very strange problem on my test VM CHR ...
Until 2.0.1 it worked fine, with the latest it wipes my blacklistUpdate and blacklistScriptUpdater ... they are EMPTY ...
Screen Shot 2017-07-19 at 07.29.28.png
in winbox they show up red because they are empty ...
nothing in the logs

even when I copy the content back into the scripts from my RB1100 and run again, the scripts are empty after the run

Eddie
You do not have the required permissions to view the files attached to this post.
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
xlighting
just joined
Posts: 6
Joined: Wed Apr 02, 2014 6:08 pm

Re: Blacklist Filter update script

Wed Jul 19, 2017 10:22 am

eddie: give your install script ALL policies(check all boxes, as Dave's picture show above), then you will get the correct content :)
Hi Dave,
tnx for the updates, it seems to work fine on my RB2011 and my RB1100.

But, I have a very strange problem on my test VM CHR ...
Until 2.0.1 it worked fine, with the latest it wipes my blacklistUpdate and blacklistScriptUpdater ... they are EMPTY ...

Screen Shot 2017-07-19 at 07.29.28.png
in winbox they show up red because they are empty ...
nothing in the logs

even when I copy the content back into the scripts from my RB1100 and run again, the scripts are empty after the run

Eddie
 
xlighting
just joined
Posts: 6
Joined: Wed Apr 02, 2014 6:08 pm

Re: Blacklist Filter update script

Wed Jul 19, 2017 10:37 am

Hi Dave:
the update scripts only require read+write+test+policy to run properly(tested with un-check those policies one-by-one), but I see your install script is trying to import update scripts with all policies.. so if the install script does not have all policies checked, it will ends up with EMPTY import files (that is what Eddie was showing)....

I assume most people would prefer only grant a minimum policy set, so would you mind change your install file, to import scripts with only read+write+test+policy?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 19, 2017 4:33 pm

Unfortunately, taking away the permissions ends with empty scripts. Taking away ANY of them causes issues - I do not know why. You *SHOULD NOT* need "password" or "sensitive", but removing them causes the failure.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 19, 2017 4:36 pm

Hi Dave,

First of all thanks for an amazing job and all effort you're putting into this. It's working just fantastic on my hAP-ac router.

A small idea to consider: how about extending firewall filter rules with autoblock functionality for intruders trying to get to a router or network? A dynamic list with banned IP's trying to do excessive pings, scanning ports, attempting DoS attacks, etc?
I'm using your rules from post #2 and having this autoban functionality will just make them more complete and make the network more secure, I think.
It's a bit beyond the scope of the blacklist. But I do agree.
Right now, I don't use any auto-ban because a current bug in RouterOS. The dynamic address-lists are not expiring as expected, which will cause a lot of false positives.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Wed Jul 19, 2017 6:01 pm

Please do not mix sugar and salt in one script :-)

List is a list. Period.
Rules should be not installed automatically as they could have influence on all other rules.
Where to install them? At the beggining, before all others? At the end?
Big NO, NO, NO for mixing things in this script.
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 19, 2017 6:51 pm

No worries, I have no intention of including rules beyond the basic examples provided in the initial posts.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jul 19, 2017 8:00 pm

So two things... Some users are simply blocked at my firewall, and now two users have been added to the list itself. I don't see this as "poisoning" as they are the ones that were actively trying to find security holes. (They have been trying SQL injections) Given that they are active attacks, I see them as no different than the botnets and spammers that the list is intended to block.

I find it VERY sad that MikroTik users on this forum would stoop this level. And frankly, if the USA passes this current bill that will allow sys-admins to "hack back" then the next time they pull my list, it will include a command to clear their config. until that time - they will remain blacklisted.
I have an ethical standpoint in this and what laws enable is not always sane. If you attack the attacker than you find yourself both back on the same level.

I assume that you have excellent means to defend yourself in this all. Also the IP address you are thinking to can easily be an VPN or use an other mean for hide their real address and if not then they are not that smart.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jul 19, 2017 8:15 pm

Hi Dave,

First of all thanks for an amazing job and all effort you're putting into this. It's working just fantastic on my hAP-ac router.

A small idea to consider: how about extending firewall filter rules with autoblock functionality for intruders trying to get to a router or network? A dynamic list with banned IP's trying to do excessive pings, scanning ports, attempting DoS attacks, etc?
I'm using your rules from post #2 and having this autoban functionality will just make them more complete and make the network more secure, I think.
I have been there and you are building a list of your own however the chance to have a secondary hit on that address is small. There are many devices in the net that are trying yo have a response.
Better is to only allow the services that you offer and protect those very well. All the rest of the traffic you drop. I have the blacklist running which filters 2048 tries and after that I have a filter port 22 and 23 which result in another 2163 hits and then I have the service filter that then filters an other 1280 tries which makes over 5000 tries in a little more than four days.

The blacklist is an valued addition in the whole concept but like a virusscanner living in the past not the now and here.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Wed Jul 19, 2017 8:32 pm

"I have the blacklist running which filters 2048 tries and after that I have a filter port 22 and 23 which result in another 2163 hits and then I have the service filter that then filters an other 1280 tries which makes over 5000 tries in a little more than four days."

Greetings msatter, can you share those filter rules for a beginner? Thank you !
RB3011 UiAS (arm)
Best regards
Ricardo
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jul 19, 2017 8:46 pm

"I have the blacklist running which filters 2048 tries and after that I have a filter port 22 and 23 which result in another 2163 hits and then I have the service filter that then filters an other 1280 tries which makes over 5000 tries in a little more than four days."

Greetings msatter, can you share those filter rules for a beginner? Thank you !
Only use the first one TCP and thee has as avaiable services mail and website.

viewtopic.php?f=9&t=98804&p=607503&hilit=Raw#p607503

Because I don't access my router from outside on port 23 and 22 I drop those a simple rule also again with my WAN as entry point pppoe-out1 and your wan could have a different name.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Thu Jul 20, 2017 4:37 am

Thank you msatter !
RB3011 UiAS (arm)
Best regards
Ricardo
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Thu Jul 20, 2017 9:16 am

Morning,

tnx for explaining the script rights issue, to bad we are struggling with that, for now it works here.

@Dave
I noticed the script got updated to 2.0.3 in the past 12 hours, it would be nice to see some kind of changelog if possible ?

Keep up the good work !

Eddie
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
nico599
just joined
Posts: 2
Joined: Mon Jun 12, 2017 11:42 am

Re: Blacklist Filter update script

Thu Jul 20, 2017 12:43 pm

Hi all
i'm running in my ccr-1009-8G-1S-1S+
log is show notthing
but
Script List show this messeage https://goo.gl/yYE2do Image
messeage is "
LOG 【;(eval (eval /putmessage=$t) (eval /log warningmessage=$t))】
urlEncode【;(eval (eval /localname=$temp) (eval /forcounter=$i;do=;(eval (eval /localname=$char;value=(eval (eval /pickbegin=$i;counter=$t))) (eval /ifcondition=(= $char );do=;(eval (eval /setname=$char;value=%20) /)) (eval /ifcondition=(= $char -);do=;(eval (eval /setname=$char;value=%2D) /)) (eval /ifcondition=(= $char /);do=;(eval (eval /setname=$char;value=%2D) /)) (eval /ifcondition=(= $char &);do=;(eval (eval /setname=$char;value=%26) /)) (eval /ifcondition=(= $char =);do=;(eval (eval /setname=$char;value=%3D) /)) (eval /setname=$temp;value=( . $temp $char)) /);from=0;to=(- (eval (eval /lenvalue=$t)) 1)) (eval /returnvalue=$temp) /)】

how can i do?
 
User avatar
inteq
Member Candidate
Member Candidate
Posts: 102
Joined: Wed Feb 25, 2015 8:15 pm

Re: Blacklist Filter update script

Thu Jul 20, 2017 2:27 pm

Thank you for the script, but I have to say that, as least in my limited testing, I stumbled upon too many blocked gmail servers.
I couldn't even send an email from my gmail account to my corporate address.
The worst part is that gmail somehow didn't even alert me that the message did not go through. Even after one day.
So I have to pass on this one.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 20, 2017 6:40 pm

Hi all
i'm running in my ccr-1009-8G-1S-1S+
log is show notthing
but
Script List show this messeage https://goo.gl/yYE2do Image
messeage is "
LOG 【;(eval (eval /putmessage=$t) (eval /log warningmessage=$t))】
urlEncode【;(eval (eval /localname=$temp) (eval /forcounter=$i;do=;(eval (eval /localname=$char;value=(eval (eval /pickbegin=$i;counter=$t))) (eval /ifcondition=(= $char );do=;(eval (eval /setname=$char;value=%20) /)) (eval /ifcondition=(= $char -);do=;(eval (eval /setname=$char;value=%2D) /)) (eval /ifcondition=(= $char /);do=;(eval (eval /setname=$char;value=%2D) /)) (eval /ifcondition=(= $char &);do=;(eval (eval /setname=$char;value=%26) /)) (eval /ifcondition=(= $char =);do=;(eval (eval /setname=$char;value=%3D) /)) (eval /setname=$temp;value=( . $temp $char)) /);from=0;to=(- (eval (eval /lenvalue=$t)) 1)) (eval /returnvalue=$temp) /)】

how can i do?
set the blDebug in the config to 1

the code you pasted is the environment, not script list. those are the functions used for url encoding and displaying the log, when enabled.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 20, 2017 6:40 pm

Morning,

tnx for explaining the script rights issue, to bad we are struggling with that, for now it works here.

@Dave
I noticed the script got updated to 2.0.3 in the past 12 hours, it would be nice to see some kind of changelog if possible ?

Keep up the good work !

Eddie
release notes are in the first post. 2.0.3 is included there.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 20, 2017 6:43 pm

Thank you for the script, but I have to say that, as least in my limited testing, I stumbled upon too many blocked gmail servers.
I couldn't even send an email from my gmail account to my corporate address.
The worst part is that gmail somehow didn't even alert me that the message did not go through. Even after one day.
So I have to pass on this one.
Yes, unfortunately, Google is now allowing spammers to use their servers for a price. You are welcome to create a whitelist of servers that you do not want blocked. Unfortunately Google is using their size to try and force admins to stop using block lists. They make money on spam. For this reason, I do not use or support google.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Thu Jul 20, 2017 11:03 pm

I go away for a week and everything has changed. :shock:

@IntrusDave, thank you again for all your work on this blacklist.

Unfortunately for me, the automated scripting is now too intrusive and is itself a serious security risk, so I'm out. If in the future you resume publishing a blacklist of addresses/networks that I can import using my own scripting I will probably use that. Meanwhile, I will just use the service from squidblacklist.org that repackages a few public lists and has not caused me any false positive problems.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 20, 2017 11:37 pm

I go away for a week and everything has changed. :shock:

@IntrusDave, thank you again for all your work on this blacklist.

Unfortunately for me, the automated scripting is now too intrusive and is itself a serious security risk, so I'm out. If in the future you resume publishing a blacklist of addresses/networks that I can import using my own scripting I will probably use that. Meanwhile, I will just use the service from squidblacklist.org that repackages a few public lists and has not caused me any false positive problems.
Add this to the config file. Auto-update is not disabled by default, and can be enabled by setting this to "yes"
:global blScriptUpdate "no";
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
ilivlad
just joined
Posts: 14
Joined: Tue Mar 12, 2013 2:02 pm

Re: Blacklist Filter update script

Fri Jul 21, 2017 1:19 am

Thanks Dave for the great work!

One thing I would like to see is maybe to add an entry to config for update interval,
once an hour is a bit excessive for me (or its just me :) )
Originally it was once a day and it was ok for me.

Thanks again!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 21, 2017 2:10 am

The script is called once an hour, however that only means that you will make a single DNS lookup to see if the filters have changed. If there is no change, then no update is downloaded. If the DNS returns a newer serial number than the current installed list, then the new list is downloaded.

The list is regenerated several times though the day. A regeneration is triggered when more than 100 addresses change, or if more than 10 full subnets change.

So, polling each hour isn't much bandwidth at all (just a DNS lookup).
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 21, 2017 2:27 am

The frequency can be determined by yourself by setting that in the scheduler to a time that suits you. And as Dave wrote the inquiry itself is not big.

@Dave, if you only have additions to the list and no removals then you could send only those additions, without a removal of all addresses. And thinking even further on that, you can do each 6 or 12 hour total removal of the list and in the hours in between only the additions.

You keep maximum protection and limited time false positives but reduce the traffic considerable on both ends.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
ilivlad
just joined
Posts: 14
Joined: Tue Mar 12, 2013 2:02 pm

Re: Blacklist Filter update script

Fri Jul 21, 2017 10:18 am

Thanks for your reply guys!
I did set the scheduler time for 24 hours but that gets overwritten with the script update.

My concern is towards writes to the storage device and possible bad blocks.
For example, mine rb2011 is serving me for some 20 months.
Im using Dave's script for a month now and I got 1.5 million total sector writes for this time period.
And I had 2 million total before.
Im not saying this is too much but perhaps guys from Mikrotik can advise where we should keep NAND wear and tear.
Btw msatter idea of updating only the difference sounds good, add the new ones, remove whitelisted, if
thats less expensive Mbyte wise, if Dave didnt already take that into account.
Thanks guys for the good work again!
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 21, 2017 10:53 am

You can now disable the auto updates and Dave write earlier about that for the 2.04 update.

The list are dynamic addresses and are flushed on reboot and have a look at your available RAM after loading the list. It should be in RAM and not in Flash.

When you add the removals to a whitelist it would require a extra rule in firewall beside the already static-whitelist I got. The main fear of Dave is that thing would get out of sync.

However this could work and dynamic and static can be in the same addresslist as for the white and the black list and Dave only removes the dynamic ones on update and less frequent all dynamic address white or black.

sequence: White containing static+dynamic
Black static+dynamic

Problem could be when adding to a list and the IP already exist then you get an error and the script stops. However on smaller updates "on-error" can be used due to the limited nummber of addresses added.

*****update***** You can selectively remove only the dynamic addresses by using this line:
:do { /ip firewall address-list remove [find where dynamic && list=intrusBL] } on-error={}
Last edited by msatter on Sat Jul 22, 2017 9:57 am, edited 2 times in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
rllavona13
just joined
Posts: 7
Joined: Mon Nov 28, 2016 12:41 am

Re: Blacklist Filter update script

Fri Jul 21, 2017 5:39 pm

Sorry the Script is broken? We used it for months now, today we notice that the script is deleting itself and creates 3 empty scripts.
rllavona
Jr Network Engineer
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 21, 2017 5:55 pm

The previous version has been disabled because of abuse. Please remove all the blacklist scripts, and run the installer from the first post.
It provides you with a much more stable and flexible platform. Once installed, read over the .conf file and make changes to suit your needs.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 21, 2017 5:57 pm

I've updated the server side to prevent units with 64M or less from pulling list 3. It's simply too big and causes the units to panic with an out of memory error. I watched one unit download the list and reboot more than 30 times last night, until I forced it to grab list two on the server side.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
boldsuck
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Sun Sep 01, 2013 1:07 am
Location: Germany

Re: Blacklist Filter update script

Fri Jul 21, 2017 6:20 pm

Thank you for the work and offering the service to us other MikroTik users.
I found that several users were leaching the large list and rebranding it as their own. They were also trying to probe the server side for exploits.

Again, I offer my list as a free service to the MikroTik community. If people continue to abuse it, I will shut it down completely.
That would be a pity but understandable.
Or in the script a key query. The key is only available by e-mail request, with forum user name. You only send the key to forum users who are members of n-years or who have x-post's.
But that would be a lot of work :(
So two things... Some users are simply blocked at my firewall, and now two users have been added to the list itself. I don't see this as "poisoning" as they are the ones that were actively trying to find security holes. (They have been trying SQL injections) Given that they are active attacks, I see them as no different than the botnets and spammers that the list is intended to block.

I find it VERY sad that MikroTik users on this forum would stoop this level.
+1
╰_╯ Ciao Marco!
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Jul 21, 2017 6:35 pm

I've updated the server side to prevent units with 64M or less from pulling list 3. It's simply too big and causes the units to panic with an out of memory error. I watched one unit download the list and reboot more than 30 times last night, until I forced it to grab list two on the server side.
The huge list is taking a lot of memory and 64 is not enough certainly if you do a remove and read in. I am not trying for the first time the huge list and it took 7 minutes to push it into the addresslist.

It took extra 40MB in RAM than the medium list so not for 64MB routers as you experienced for yourself. On a reboot I am left with 112MB of free RAM and that will go down when the first refresh is a fact.

Update: with the huge list loaded and after a refresh I had 82MB left so about 100MB used extra compared to the medium list. The 7 minutes reload is a big factor to me and I am sticking to the medium list for the moment.
Last edited by msatter on Sat Jul 22, 2017 9:46 am, edited 1 time in total.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Jul 21, 2017 6:48 pm

for those interested, the DNS now holds the list sizes.
{
  :local list1 [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.4 ];
  :local list2 [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.5 ];
  :local list3 [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.6 ];
  :put "List 1 Entries: $list1\n\rList 2 Entries: $list2\n\rList 3 Entries: $list3";
  :log warning "List 1 Entries: $list1\n\rList 2 Entries: $list2\n\rList 3 Entries: $list3";
}
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sat Jul 22, 2017 10:34 am

OK writing this I see a problem with the blacklist containing static and dynamic addresses because you can get collisions on the complete import and it will stop it and leave you with a incomplete blacklist loaded. So for now the request to have already the dynamic added to the removal to have it in place in case a solution is found for the collisions in the blacklist. I don't see Mikrotik enabling on-error standard when :do is put in a variable and so an easy solution for the collisions in only the blacklist is unsure

Here starts my original request:

Request to be able to have static addresses in the blacklist and on complete refresh only the dynamic addresses are removed?
:do { /ip firewall address-list remove [find where dynamic && list=intrusBL] } on-error={}
[/i]

This can also be used for a whitelist setup (intrusWL) for freed IP addresses that not longer have to be blacklisted. Saving time on reloading the huge list to often and eventually the medium list.
:do { /ip firewall address-list remove [find where dynamic && list=intrusWL] } on-error={}
In the filtering/RAW firewall the whitelist is above the blacklist and have an accept as action. The blacklist still has an drop and both are able to contain dynamic and static addresses at the same time.

On a complete reload both lists are cleared with the only dynamic settings and the blacklist is populated again with the new addresses to block. This can happen once a day due to the 1d+1hour timeout.
Every hour the client contacts the server for an up update and the server decides if it will give a update or a complete list. The update list does not have a remove in it and only add addresses. Because the number of additions is not that big the lines can be writen out completely and you can use on-error if someone has an static address in the list that matches the dynamic address to be added.
Adding addresses is done in the blacklist but also in the whitelist and the whitelist will contain besides static addresses also dynamic addresses of those addresses that don't have to block any more. Those addresses will stay in the whitelist until the next complete remove is done and they are not needed any more because the complete blacklist does not contain them any more...or thay must have been naughty again in the meantime.

Thanks to Jo2jo for the whitelist idea.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
RFCOM
just joined
Posts: 3
Joined: Mon Jul 24, 2017 7:31 pm

Re: Blacklist Filter update script

Mon Jul 24, 2017 8:12 pm

Hello,

I need your help, I had the previous script working and I could see the whole list, after this change I am copying the new script but it does not work, I get the following error.


[mkadmin@MikroTik] > :global blScriptVersion;
[mkadmin@MikroTik] > if ($blScriptVersion != "2.0.1") do={
{... :local sourceServer "https://mikrotikfilters.com/";
{... :local scriptName "blInstaller.rsc";
{... .
{... .
{... :do { /ip firewall address-list remove [find where list=dynamicBlacklist] } on-error={}
{... /system script run blacklistUpdate
{... } else={ :put "script is current" }
script is current
[mkadmin@MikroTik] > :do {
{... :local currentScriptVersion [ :resolve server=mikrotikfilters.com server-port=6502 domain-name=127.0.0.2 ]
{... :put "Installing blacklistUpdate script version: $currentScriptVersion";
{... :local sourceServer "https://mikrotikfilters.com/";
{... :local scriptName "blInstaller.rsc";
{... :put "Downloading update script...";
{... :do {
{{... /tool fetch url="$sourceServer$scriptName" mode=https dst-path="/$scriptName";
{{... } on-error={
{{... :put "Error. Download failed";
{{... }
{... :put "Importing update script...";
{... :do {
{{... /import "$scriptName";
{{... } on-error={
{{... :put "import failed. unknown error.";
{{... }
{... :put "Removing update script...";
{... :do {
{{... /file remove "$scriptName";
{{... } on-error={}
{... :put "Update Complete.";
{... }
failure: dns server failure

Thanks

BR

Note: I'm not an expert, I'm sorry
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 24, 2017 10:00 pm

Either your firewall is blocking DNS to my server, or your IP is blocked by the list already.

What is your public IP?
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Wed Jul 26, 2017 3:27 pm

A update on the score and RC contains a fix for the not timing out of the address list entries.

In 6.41RC3 has the following fix:
firewall - properly remove "address-list" entry after timeout ends;
I keep away for a while from this because I get less speed by dropping master-slave and going to bridge-'slave'

My score after a new start and now 4 days and 14 hours I have the following score. The score is in the position in the filtering list:

drop port 22,222,2222,2323 2776 hits
drop IntrusBL (blacklist) 1970 hits
drop unknown services 1281 hits

static whitelist in/out 180/231 hits
static blacklist 0 hits

I don't use external ports 22,222,2222,2323 so those are always blocked.

I am going now try switching unknown services and the blacklist and reset the counters. The blacklist blocks a lot of traffic that looks legit but is not and so it is not reaching the second line of defence at service level. I had some false positives but those are easy added to the static whitelist that I have now.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 187
Joined: Wed May 20, 2015 10:14 pm
Location: Quito

Re: Blacklist Filter update script

Wed Jul 26, 2017 6:21 pm

Hi Dave, can you help me to check if my public IP is banned on your list?? The script was working fine, but now i've trying several things to make it work again with your updates, but it allways shows me a dns error. If I post my public IP here, it makes me an easy target :) so, i'm wondering if i can send you that info via any message, I also need more information about your DNS, because in our Core Router I blocked DNS requests from any server that is not allowed, so i need an IP of your DNS server to check if that can help me.
MikroTik Soporte y Consultoría - Español / English +593 98 709 3502
https://www.safenet.ec/consultoria.html/ soporte@safenet.ec
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jul 26, 2017 10:01 pm

The DNS address is "mikrotikfilters.com" port 6502. The IP changes, based on current load - so if you add it to an address-list, just put the domain name and let it resolve. The port shouldn't be blocked, unless you are doing level 7 DNS filtering.

you can post the last 3 octets of your IP, and still remain fairly anonymous.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
sri2007
Member Candidate
Member Candidate
Posts: 187
Joined: Wed May 20, 2015 10:14 pm
Location: Quito

Re: Blacklist Filter update script

Thu Jul 27, 2017 3:58 am

Great thanks for your help!! I added a new address-list associated with your domain and it works, thanks for your help! It seems that i'm not banned.
MikroTik Soporte y Consultoría - Español / English +593 98 709 3502
https://www.safenet.ec/consultoria.html/ soporte@safenet.ec
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Jul 27, 2017 8:05 pm

I've updated to 2.0.5. This update moves the server to tcp port 6501 and udp port 6502. These ports are excluded from being blocked by the list on my end, and should allow users on "bad" subnets to pull the list.

make sure to give positive rating if you are using and like this service
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
hhgttg42
just joined
Posts: 8
Joined: Wed Oct 12, 2016 4:48 am

Re: Blacklist Filter update script

Sat Jul 29, 2017 4:48 pm

Hi Dave,

Have you or anyone else noticed that their routerboard emits a chime (alternating high/low like a siren 2-3 times) around when the logging is turned back on and the script completes?? I can't find anything in your scripts or in my setup that would be causing this...it's beeping at least at 9:33am/pm EDT daily like clockwork...

I am running your latest script without modification on a RB951G-2HnD.

Thank you.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Jul 29, 2017 9:33 pm

Yes. If you notice in your log, it is telling you that the script is out of date. The server inserts an alarm into the script when your local script version is out of sync with the server. You can run the code in the first post to keep your script up to date with the latest big fixes.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Mon Jul 31, 2017 5:43 am

Due to a bug in RouterOS, versions below 6.36 are now blocked on the server side.
It appears they they are not able to compare the local blacklist serial with the server side serial.
This is causing the 6.35 and earlier routers to update constantly.

So, to save the NAND as well as bandwidth, I've chosen to block them and attempt to alert the owner.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Thu Aug 03, 2017 4:06 am

Did you make a typo? Its telling me versions below 6.38 are blocked. im on 6.37.5
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Fri Aug 04, 2017 11:18 am

My score after a new start and now 8 days and 19 hours I have the following score. The score written in the position in the RAW filtering list:

drop port 22,222,2222,2323 = 5555 hits
drop unknown services = 5409 hits
drop IntrusBL (blacklist) = 2820 hits

static whitelist in/out = 407/591 hits
static blacklist = 0 hits

So the Blacklist adds, from the already filtered traffic, an other 20% and in my case the most on port 25 (mail) which is a really good score.
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Fri Aug 04, 2017 11:39 am

My busiest mail filtering router router. Stats for 27 days of uptime.
52 000 all SMTP/IMAP/POP access tries.
8600 unique sources of traffic
1200 blocked by my own list
26 000 blocked by Dave's list.
No client's complains for missing mails
smtpmm.PNG
You do not have the required permissions to view the files attached to this post.
Real admins use real keyboards.
 
RFCOM
just joined
Posts: 3
Joined: Mon Jul 24, 2017 7:31 pm

Re: Blacklist Filter update script

Fri Aug 04, 2017 8:37 pm

Either your firewall is blocking DNS to my server, or your IP is blocked by the list already.

What is your public IP?
181.225.100.117
190.253.66.37
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Aug 04, 2017 9:12 pm

Either your firewall is blocking DNS to my server, or your IP is blocked by the list already.

What is your public IP?
181.225.100.117
190.253.66.37
Your ISP /AS262186 is UCEPROTECT-Level3 listed for hosting a total of 462 abusers.
Your ISP COLOMBIA TELECOMUNICACIONES S.A. ESP/AS3816 is UCEPROTECT-Level3 listed for hosting a total of 5478 abusers.
Unfortunately, both of your IP's fall into ASN's that are blocked.
If you are using script version 2.0.5, then you should be able to pull the current blacklist, as the DNS and HTTPS servers are on unfiltered ports.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
RFCOM
just joined
Posts: 3
Joined: Mon Jul 24, 2017 7:31 pm

Re: Blacklist Filter update script

Fri Aug 04, 2017 9:57 pm

Either your firewall is blocking DNS to my server, or your IP is blocked by the list already.

What is your public IP?
181.225.100.117
190.253.66.37
Your ISP /AS262186 is UCEPROTECT-Level3 listed for hosting a total of 462 abusers.
Your ISP COLOMBIA TELECOMUNICACIONES S.A. ESP/AS3816 is UCEPROTECT-Level3 listed for hosting a total of 5478 abusers.
Unfortunately, both of your IP's fall into ASN's that are blocked.
If you are using script version 2.0.5, then you should be able to pull the current blacklist, as the DNS and HTTPS servers are on unfiltered ports.
Thanks, Put the version 2.0.5 and OK. is posible that you share ip rules and filters again or update?

Thanks again
 
hhgttg42
just joined
Posts: 8
Joined: Wed Oct 12, 2016 4:48 am

Re: Blacklist Filter update script

Sat Aug 05, 2017 6:13 pm

Yes. If you notice in your log, it is telling you that the script is out of date. The server inserts an alarm into the script when your local script version is out of sync with the server. You can run the code in the first post to keep your script up to date with the latest big fixes.
Thank you for the insight Dave. I'll keep an eye on that.
 
drzen
Frequent Visitor
Frequent Visitor
Posts: 80
Joined: Thu Aug 09, 2007 3:59 pm
Location: Pordenone, Italy
Contact:

Re: Blacklist Filter update script

Wed Aug 09, 2017 5:22 pm

Great script and hard works. Thanks.
A question about security: all go rights but what if your server is violated? For example a malicious code add "/system reset-configuration" or others dangerous commands in front of a downloaded lists?

This is the last doubt before adopting your solution.

thanks in advance.
v.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Aug 09, 2017 5:40 pm

That would truly be bad.
My solution is to isolate the server that generates the scripts from the rest. The generation server is not accessible from the internet. For me to access it, I must connect to my firewall via VPN, then SSH to the database server, then ssh to the blacklist server. The second layer of protection is a 3rd server that parses the lists to check for any commands other than the expected before it is encrypted and passed to the web server for distribution. Last step is a custom nginx module that decrypts the list on the fly to send it out to the firewall.

I've been working on other solutions to push out the list, but have yet to find a good process that is simple and available to all users / firewalls.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
drzen
Frequent Visitor
Frequent Visitor
Posts: 80
Joined: Thu Aug 09, 2007 3:59 pm
Location: Pordenone, Italy
Contact:

Re: Blacklist Filter update script

Wed Aug 09, 2017 5:45 pm

That would truly be bad.
I've been working on other solutions to push out the list, but have yet to find a good process that is simple and available to all users / firewalls.
THE solution is to output only raw ip addresses' list. But this solution collides with the length of variables in routeros. :(

Thanks for your efforts.
(Rate my posts? If you want... no pressure...)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Aug 09, 2017 5:48 pm

THE solution is to output only raw ip addresses' list. But this solution collides with the length of variables in routeros. :(

Thanks for your efforts.
Unfortunately, it's only a solution if it's possible. The amount of 4kb files needed to be downloaded and processes would cause so much wear on the NAND and take up so many filesystem blocks, it would kill most of the routers pulling the list.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
foxxiu7
just joined
Posts: 5
Joined: Sun Aug 25, 2013 3:30 am

Re: Blacklist Filter update script

Thu Aug 10, 2017 5:58 am

Unfortunately, it's only a solution if it's possible. The amount of 4kb files needed to be downloaded and processes would cause so much wear on the NAND and take up so many filesystem blocks, it would kill most of the routers pulling the list.
Is it an option to change script to download file to USB drive and use external flash drive instead of internal NAND?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Aug 10, 2017 6:21 am

Is it an option to change script to download file to USB drive and use external flash drive instead of internal NAND?
Yes, the path is set in the config file.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
foxxiu7
just joined
Posts: 5
Joined: Sun Aug 25, 2013 3:30 am

Re: Blacklist Filter update script

Thu Aug 10, 2017 6:42 am

Is it an option to change script to download file to USB drive and use external flash drive instead of internal NAND?
Yes, the path is set in the config file.
Awesome! Have to check the config file then.
 
SPKA16
newbie
Posts: 25
Joined: Fri Aug 05, 2016 8:41 pm

Re: Blacklist Filter update script

Wed Aug 16, 2017 8:51 pm

First of all, thanks for the list! Been using it for a while and started to use it on costumer routers as well! Appreciate all the hard work!

Today at work we noticed a (strong) increase in upload from the router and noticed that Winbox was constantly keeps loading all records in the firewall/adress-list view when its opened.
See attachment/screenshot of my homerouter where the example it on a LAN interface. Whenever you switch tab on the firewall tab or close it the load on the interface goes away. It is constantly around 5Mbps+ when this view is opened.

Is this a known problem, by design, worth a report to Mikrotik? Don't know if this is since the start or just when the list changed to dynamic entries.
You do not have the required permissions to view the files attached to this post.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Aug 16, 2017 9:23 pm

This is expected. Because the address list is dynamic, the timeout is constantly changing. This causes WinBox to reload the entire address list output every second. I'll see if I can find a fix, but I don't think there is much you can do about it, other than not leaving the Address List tab open.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Moblaw
just joined
Posts: 7
Joined: Sun Aug 27, 2017 7:56 pm

Re: Blacklist Filter update script

Tue Sep 05, 2017 7:15 pm

I'm trying to get it to work, just can't get the address-list (intrusBL) or (dynamicBlacklist) for that matter, to be filled out.

When I execute the "# Import Intrus Managed Filter Lists
# © 2017 David Joyce, Intrus Technologies" script via /system script run updateBlacklist, which contains the script from the first page.

I get:
invalid value for argument server:
invalid value for argument ipv6-address
failure: bad name
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Sep 05, 2017 7:21 pm

At this time the script is IPv4 only. The servers are able to deal with IPv6, but the client script is not.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Moblaw
just joined
Posts: 7
Joined: Sun Aug 27, 2017 7:56 pm

Re: Blacklist Filter update script

Tue Sep 05, 2017 7:30 pm

Now getting /system script run blacklistUpdate
expected command name (line 16 column 1)

I wasn't using IPV6 before, I've disabled the interface now, but cant get the script to properly script as executeable.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Tue Sep 05, 2017 7:32 pm

have you tried remove the scripts and schedules and reinstalling?
There is not much I can do to help, as I have no access to your router.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Moblaw
just joined
Posts: 7
Joined: Sun Aug 27, 2017 7:56 pm

Re: Blacklist Filter update script

Tue Sep 05, 2017 9:17 pm

have you tried remove the scripts and schedules and reinstalling?
There is not much I can do to help, as I have no access to your router.
Agree.

I got it working. Fault was this firewall rule "Disallow anything from anywhere on any interface" on input. Disabled it, and the script loaded fine

Its kinda obivious, that, that rule blocks the script from executing. tbh, I hard a hard time point out, in which senario, that rule would be beneficial.

Thanks man. You have my full and sincere gratitude, for your Work done scripting and "posting".

I had to adjust the update-interval, 10min is to often and 3 hours is fine for me.

I'm using my router front end, and pfsense af IDS backend. So what the "intrusBL list" doesn't catch, pfsense pfBlocker & snort will, maybe.

Would it be realistic for me to use front end, without an "ids" or pfsense? Since you mention using it at hospitals etc.
 
mhyll
just joined
Posts: 8
Joined: Wed Feb 01, 2017 3:50 pm

Re: Blacklist Filter update script

Wed Oct 11, 2017 6:57 pm

When can I see the source blacklists? From where you are taking IP addresses? Can I exclude some blacklist sources?

Thanks!

PS: Are you accepting some donations? This stuff is bloody hell good. :)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Oct 11, 2017 7:07 pm

The source isn't open. It's a collection of 140 routers and servers running as honeypots.
You can exclude addresses by creating a whitelist address-list, and a rule to accept those addresses before the blacklist drop rules.

I don't feel that donations are needed, but thank you for the offer.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 1303
Joined: Sat Dec 24, 2016 11:17 am
Location: jo.overland at gmail.com

Re: Blacklist Filter update script

Wed Nov 08, 2017 7:57 pm

@IntrusDave
In post one, can you add information on what this script does, who it is for etc?
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 09, 2017 3:30 am

The scripts are for maintaining an address list that is intended to help filter out as much of the bad crap on the internet as possible. This includes spam, viruses, hackers, etc. the script contacts my servers and pulls the latest list of known bad IPs and subnets. The list comes in three sizes. The smallest is meant for home users. It just filters botnets and such. The medium list adds spam hosts and is intended for small to medium businesses. The large filters everything that we can, over 200,000 entries and is only intended for the larger CCR routers protecting servers.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 1303
Joined: Sat Dec 24, 2016 11:17 am
Location: jo.overland at gmail.com

Re: Blacklist Filter update script

Thu Nov 09, 2017 9:01 am

I did found this out after reading the thread, but edit post #1 and add this info there.
People like me, that opens a thread to see what is this about, should not need to read long down the thread to see what its about.
For you and other posters here its obvious, but not for my mother....
 
How to use Splunk to monitor your MikroTik Router

MikroTik->Splunk
 
 
User avatar
hilton
Long time Member
Long time Member
Posts: 635
Joined: Thu Sep 07, 2006 5:12 pm
Location: Jozi (aka Johannesburg), South Africa

Re: Blacklist Filter update script

Thu Nov 09, 2017 9:17 am

The list comes in three sizes. The smallest is meant for home users. It just filters botnets and such. The medium list adds spam hosts and is intended for small to medium businesses. The large filters everything that we can, over 200,000 entries and is only intended for the larger CCR routers protecting servers.
Dave, which list do we get?
Regards
Hilton
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 09, 2017 6:40 pm

For you and other posters here its obvious, but not for my mother....
It's not really intended for your mother. :)
Intended for network admins that don't really need help or explanation.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Nov 09, 2017 6:41 pm

Dave, which list do we get?
That's your choice. Select the list that fits your needs, and set it in the config file.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
amity2kare
newbie
Posts: 35
Joined: Tue Feb 13, 2007 4:24 pm
Location: INDIA

Re: Blacklist Filter update script

Fri Dec 15, 2017 12:58 pm

@IntrusDave

Brilliant script. Worked on my CCR1072 from the word go. Karma from my side.

Regards
 
arkbyte
just joined
Posts: 2
Joined: Thu Dec 21, 2017 6:47 pm

Re: Blacklist Filter update script

Thu Dec 21, 2017 6:50 pm

This blacklist is blocking, among other things, Github. It has been for a while.
It's a great idea, but clearly is not curated or monitored. I would recommend not using it.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Thu Dec 21, 2017 10:56 pm

This blacklist is blocking, among other things, Github. It has been for a while.
It's a great idea, but clearly is not curated or monitored. I would recommend not using it.
Welcome to the board. Not sure why your first post would be to trash someone's work instead of asking a question about it, but okay..

I'd recommend that you read a bit more before posting next time. After reading the thread and notes, you would notice that the list is designed to block *incoming* connections. If it is preventing you from accessing a website, then you have implemented your firewall rules wrong. Note that the list DOES NOT provide rules, only an address list. you supply the rules on how you would like to use it.

That said, I suppose that I need to point out, the list isn't blocking your access to websites, you are blocking access.

If you insist on using the list for outgoing connections, as many do, you will want to also create a whitelist of addresses that you do not want blocked. (as pointed out several times in this topic)

As for GitHub being blocked, if it's being blocked, then it's for good reason. Most of the time its because a server is hosting ads with malicious content, or the site's mail servers are being used for spam.

Anyway. Please fix your rules to block incoming connections, not outgoing. The list is intended to prevent incoming connections from IP's that have been proven to have malicious intent.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
arkbyte
just joined
Posts: 2
Joined: Thu Dec 21, 2017 6:47 pm

Re: Blacklist Filter update script

Fri Dec 29, 2017 7:14 pm

I'm sorry you feel that comment 'trashed your work'. I did not ask a question because I don't have one. I pointed out that your list contains erroneous entries. Based on your response, I'd say you don't have much tolerance for criticism.

The IP 192.30.255.112 hosts three (and only three) domains: githubcom, www.github.com, and shop.github.com and none of those are attacking your honeypots. It is not listed on a spam bl https://mxtoolbox.com/SuperTool.aspx?ac ... n=toolpage, and is not serving malicious ads (as far as anyone knows).
More than anything, you should take one look at the fact your list includes github and say, "Oh, well that's obviously not right," instead of deflecting with nonsense about one of the top 100 websites https://www.alexa.com/siteinfo/github.com trying to infect you with malware.

Additionally, please don't insult my intelligence. I block incoming AND outgoing communications to known hostile servers for what I believe most enterprise admins would agree is a pretty obvious reason: users in the organization need to be protected from these places. You claim that the list blocks servers which spread malware through ads; well guess what, if you allow outgoing connections to it, your users are still going to pick up that malware, even if the host can't hammer your ssh. So you adding malware hosts is literally pointless in that use case. Jesus Christ, how is that not obvious?

Lastly, yes. We could manually whitelist mistakes from your blocklist. But the list should be managed at the source rather than the endpoints. Especially considering you're using this list with your clients.

Though I'm not using it, I greatly I appreciate your work on this. I think it's fantastic that you made this script and set up these servers, and allow people to access it completely free of charge. I have no problem with that. Only with the way you responded to a legitimate issue.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Fri Dec 29, 2017 7:59 pm

As I’m on vacation, I’ll keep it short. The IP you listed has been serving a malformed PDF with a known Microsoft Edge exploit.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
acortesguasch
just joined
Posts: 7
Joined: Tue Dec 19, 2017 6:04 pm

Re: Blacklist Filter update script

Tue Jan 09, 2018 1:28 pm

Beginning to tinker with Mikrotik and I found this topic.
Just a big thanks for all the work put into it.
To Be Continued...
 
w0lt
Member
Member
Posts: 484
Joined: Wed Apr 02, 2008 2:12 pm
Location: Minnesota USA

Re: Blacklist Filter update script

Wed Jan 17, 2018 12:52 am

Has anyone had difficulty getting a "Blacklist" update today?

Thanks,

-tp
MTCNA - 2011

" The Bitterness of Poor Quality Remains Long After the Sweetness of Low Price is Forgotten "

Image
 
Ryo
just joined
Posts: 3
Joined: Thu Jan 11, 2018 8:00 am

Re: Blacklist Filter update script

Wed Jan 17, 2018 1:36 am

Has anyone had difficulty getting a "Blacklist" update today?

Thanks,

-tp
yup, it show dns server failure
 
mducharme
Trainer
Trainer
Posts: 799
Joined: Tue Jul 19, 2016 6:45 pm

Re: Blacklist Filter update script

Wed Jan 17, 2018 2:57 am

Has anyone had difficulty getting a "Blacklist" update today?

Thanks,

-tp
Yes it appears to be failing today.
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Wed Jan 17, 2018 5:49 am

Has anyone had difficulty getting a "Blacklist" update today?

Thanks,

-tp
Yes, today does not work well!
RB3011 UiAS (arm)
Best regards
Ricardo
 
servaris
newbie
Posts: 46
Joined: Tue May 20, 2014 4:30 pm
Location: Planet Earth
Contact:

Re: Blacklist Filter update script

Wed Jan 17, 2018 5:53 pm

The script seems to not be working.

Ran blacklistUpdate script in terminal.
Log displays:
10:46:56 script,warning Checking server for current blacklist serial number.
10:46:56 script,warning Blacklist is already up to date. Nothing to do.
10:46:56 system,info log rule changed by admin
10:46:57 script,error Download failed. Received bytes.
The firewall list intrusBL is empty.
intrusBL.png
Do you have a fix for this?
Thanks.
You do not have the required permissions to view the files attached to this post.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jan 17, 2018 7:20 pm

Very sorry about that guys. I had to upgrade some server hardware, so I migrated the VM's to a different server. The new server didn't import the DNS vm. The old server if back online now and the VM's returned to their home. All should be good now.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
mducharme
Trainer
Trainer
Posts: 799
Joined: Tue Jul 19, 2016 6:45 pm

Re: Blacklist Filter update script

Wed Jan 17, 2018 8:12 pm

Very sorry about that guys. I had to upgrade some server hardware, so I migrated the VM's to a different server. The new server didn't import the DNS vm. The old server if back online now and the VM's returned to their home. All should be good now.
I had to manually lower the serial number to get the blacklist back, it thought it had the latest blacklist but it was actually empty. (under Scripts in the Environment tab)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Wed Jan 17, 2018 8:23 pm

I had actually started looking into moving the service to a distributed cloud to prevent this, but It looks like I may be shutting down my business and taking over running a nonprofit.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
servaris
newbie
Posts: 46
Joined: Tue May 20, 2014 4:30 pm
Location: Planet Earth
Contact:

[SOLVED] Re: Blacklist Filter update script

Wed Jan 17, 2018 9:47 pm

Thanks to mducharme for pointing out the fix to the failed updates.

Go to system -> scripts
Click on Environment

Negatively increment your blSerial by 1. To be clear, the blSerial WAS 1516197642 and now its 1516197641 as shown below.
script-environment.png
You do not have the required permissions to view the files attached to this post.
 
amity2kare
newbie
Posts: 35
Joined: Tue Feb 13, 2007 4:24 pm
Location: INDIA

Re: Blacklist Filter update script

Thu Jan 18, 2018 10:25 pm

Script works fine at my end. However the address list entries (IntrusBL) disappear in a couple of hours. I have been noticing this behavior since i installed this script and have tried upgrading my routeros version as well but to no avail. My current config is routeros 6.39.3 on CCR1072-1G-8S+.
 
Jacka
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Jan 13, 2011 11:34 am

Re: Blacklist Filter update script

Fri Jan 19, 2018 4:23 pm

The script is still operational ? It's not working in my case. The is no code inside the script...

Image
 
User avatar
frank333
Member Candidate
Member Candidate
Posts: 117
Joined: Mon Dec 18, 2017 12:17 pm
Location: italy Router model: RB3011UiAS-RM

Re: Blacklist Filter update script

Mon Jan 22, 2018 12:02 am

The script works very well! Thanks IntrusDave, you are a Wizard Master! :)
 
Jacka
Member Candidate
Member Candidate
Posts: 112
Joined: Thu Jan 13, 2011 11:34 am

Re: Blacklist Filter update script

Mon Jan 22, 2018 9:59 am

The script works very well! Thanks IntrusDave, you are a Wizard Master! :)
Then, what I'm doing wrong? Can smb help me out. Thank you.
 
User avatar
frank333
Member Candidate
Member Candidate
Posts: 117
Joined: Mon Dec 18, 2017 12:17 pm
Location: italy Router model: RB3011UiAS-RM

Re: Blacklist Filter update script

Mon Jan 22, 2018 11:33 am

Then, what I'm doing wrong? Can smb help me out. Thank you.
Jacka,
  • eliminates old variables in the environment
1.png
  • eliminates all blacklists scripts and scheduler
  • reboot router
  • rewrites the launch script
3.png
  • starts the blacklistUpdate script --->run script
  • Must work, it controls ip-->firewall-->address list list
PS:What router and what ros version do you have?
You do not have the required permissions to view the files attached to this post.
Last edited by frank333 on Mon Jan 29, 2018 3:30 am, edited 1 time in total.
 
amity2kare
newbie
Posts: 35
Joined: Tue Feb 13, 2007 4:24 pm
Location: INDIA

Re: Blacklist Filter update script

Mon Jan 22, 2018 2:35 pm

Script works fine at my end. However the address list entries (IntrusBL) disappear in a couple of hours. I have been noticing this behavior since i installed this script and have tried upgrading my routeros version as well but to no avail. My current config is routeros 6.39.3 on CCR1072-1G-8S+.
Any help IntrusDave?
 
kakaxa
just joined
Posts: 15
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter update script

Thu Feb 01, 2018 12:00 pm

Thanks @IntrusDave, great work!
 
kakaxa
just joined
Posts: 15
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter update script

Fri Feb 02, 2018 8:21 pm

Help me please.
The script works fine, but after adding records to the dynamic sheet, after 15-20 minutes, these records are deleted
Help me understand what this is happening? in the log or what is not

Sorry for my English

-------
Помогите пожалуйста.
Скрипт работает отлично, но после добавления записей в динамический лист, спустя 15-20 минут, эти записи удаляются
Помогите понять от чего такое происходит? в логах ни чего нет
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Feb 03, 2018 4:51 am

Make sure you blDataPath does not start with a /
i.e. it should read "disk1/blTemp.rsc" NOT "/disk1/blTemp.rsc"
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
nichky
Long time Member
Long time Member
Posts: 526
Joined: Tue Jun 23, 2015 2:35 pm

Re: Blacklist Filter update script

Sat Feb 03, 2018 6:14 am

Thanks :)
Nikola Suminoski
MikroTik Consultan
MTCRE l MTCWE

!) Safe Mode is your friend;
 
kakaxa
just joined
Posts: 15
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter update script

Sat Feb 03, 2018 9:59 am

Make sure you blDataPath does not start with a /
i.e. it should read "disk1/blTemp.rsc" NOT "/disk1/blTemp.rsc"
I'm use the default, path is not change.
On 6.41 work fine, or 6.39.3 the list is cleared after 15min :(

UPD: update mikrotik to 6.41 - script works, list nod cleared
 
dehghanimeysam
just joined
Posts: 1
Joined: Sat Feb 10, 2018 4:41 am

Re: Blacklist Filter update script

Sat Feb 10, 2018 4:56 am

Thank you for providing this
 
mkx
Forum Guru
Forum Guru
Posts: 2965
Joined: Thu Mar 03, 2016 10:23 pm

Re: Blacklist Filter update script

Sun Mar 11, 2018 11:16 pm

A huge thank you from me!

And a question: does anybody know of a similar list, covering IPv6 address filtering?
BR,
Metod
 
specky
just joined
Posts: 1
Joined: Sun Aug 24, 2014 5:43 pm

Re: Blacklist Filter update script

Mon Mar 19, 2018 11:47 pm

Great script

But its caught a few false positives in the mix from a service we use called Everycloud.
Thanks
 
Grolski
just joined
Posts: 1
Joined: Mon Mar 19, 2018 8:05 pm

Re: Blacklist Filter update script

Thu Mar 22, 2018 1:22 pm

Hello,

I am very impressed with the effectiveness with the IP blacklist by IntrusDave and the scripts he has written. It is the most effective from the ones (the usual suspects) I have used so far.

I have two issue/questions (as none expert) that someone may easily solve.

1. I Drop all Input into my router with some exceptions to allow management. This is effective but also blocks the script to contact the servers and download the latest ip blacklist. What is the best solution to overcome this issue. Remove the Drop all input towards my router or enter an exception rule to allow the script to work. The attempts that I have tried (Layer 7 protocol with URL) were not successful. :(
2. Is there a easy way to extend the Timeout of Dave's list so one could run the script every two days or so? Maybe it is unwise to do so and that could also be an answer.

Regards
Wilko
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Mar 24, 2018 5:32 pm

It's been a nice run. Almost 3 years, and over 2200 active users. But I am shutting down the this service. With the new US tax laws and this new US "sex trafficking law" (which isn't really about sex trafficking) I simply can't afford to keep the service running. Bandwidth and rack space is just too expensive now, and I'm making $0.

Thank you all for the support.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
RyperX
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu May 21, 2015 11:14 am

Re: Blacklist Filter update script

Sat Mar 24, 2018 6:12 pm

Thank you for providing this service for so long, worked really well!

Maybe you could create a guide how it would be possible to create such lists by his own or where you fetch all the information?
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Sat Mar 24, 2018 6:23 pm

It's been a nice run. Almost 3 years, and over 2200 active users. But I am shutting down the this service. With the new US tax laws and this new US "sex trafficking law" (which isn't really about sex trafficking) I simply can't afford to keep the service running. Bandwidth and rack space is just too expensive now, and I'm making $0.

Thank you all for the support.
bw is cheap i can get a server for 8 bucks a month with 12TB of bw. i feel its more than cost as to why you're stopping it.

Need any help?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sat Mar 24, 2018 7:40 pm

bw is cheap i can get a server for 8 bucks a month with 12TB of bw. i feel its more than cost as to why you're stopping it.

Need any help?
If you think $8/month is the cost of a real server, a real firewall, real rack space, real bandwidth, real maintenance, real electricity... Then you are either delusional or have never owned/operated a true network. My servers are not shared VPS servers at some bulk hosting company. My firewalls are not software firewalls. For the last 17 years I have maintained a 48U rack, with 10gbps redundant fiber, a diesel generator with enough fuel for 7 days, a double-conversion UPS with 8 hours of backup time. The rack holds 13 servers, the smallest is an 8 core xeon with 8TB storage and 64GB ram. The largest being dual 12 core xeon, 384GB ram and 64TB. I have 3 CCR1016's and 2 CCR1072's.

So no, $8 doesn't cover it.

I'm also betting that 95% of the 2200 users of the list would not accept pulling a script from a shared server and then running it on their routers. The amount of security and isolation required is far beyond what you can do with a bulk host. Each step of the process to collect the data, build the list, and then push out the list is isolated from the net, not even on the same network that is connected to the internet.

Not to mention the amount of posts, PMs, and emails I get from users demanding that I make changes or run things a different way. It's just not worth it anymore.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Sat Mar 24, 2018 10:01 pm

I would be fine with keeping the service alive. Having a 48u rack for such a piss easy and small script is a bit outrageous. I think you're the delusional one. I"m sure you used that rack for way more than this script.

I'm simply saying you can keep the script stuff online for way cheaper if you wanna still help the community.
 
User avatar
Rhoos
just joined
Posts: 11
Joined: Sun Dec 20, 2015 3:48 pm
Location: Costa Rica
Contact:

Re: Blacklist Filter update script

Sun Mar 25, 2018 12:31 am

Thank you Dave for such a great service provided all these years; would it be possible for you to put at the disposal of "premium" users of this forum your IP blacklist system so that someone else can continue to provide the service?

Richard
RB3011 UiAS (arm)
Best regards
Ricardo
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Mar 25, 2018 2:00 am

I would be fine with keeping the service alive. Having a 48u rack for such a piss easy and small script is a bit outrageous. I think you're the delusional one. I"m sure you used that rack for way more than this script.

I'm simply saying you can keep the script stuff online for way cheaper if you wanna still help the community.
Never said my company was dedicated to the blacklist. It's BS like this that helped me decide to shut it down.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
chippers
newbie
Posts: 25
Joined: Tue Apr 02, 2013 7:45 am

Re: Blacklist Filter update script

Sun Mar 25, 2018 2:29 am

Hey Dave, really sorry to hear you are shutting down but completely understand. Just like to thank you for the script and for me personally, I used it on multiple devices with multiple customers and it must have saved me more than a few times from bad things happening. I used to monitor the firewall rule connected to the script and see the bytes get blocked. It was amazing how high it got at times.

Echoing others here, I would encourage you to throw everything on GitHub or similar, if for nothing else, for us all to learn from your scripting and infra experience, its how the community grows. Once again, thanks for the excellent script and your (free) support of this product. I wish you well in your new role.
 
User avatar
Deantwo
Member Candidate
Member Candidate
Posts: 295
Joined: Tue Sep 30, 2014 4:07 pm

Re: Blacklist Filter update script

Sun Mar 25, 2018 2:31 am

So this is why have gotten notification emails from this thread all day.
Sorry to hear you are shutting down Dave, but with what you explain it is understandable.

I ended up making my own blacklist script last year, so I never actually used your service much more than as a test. But I at least want to thank you for the help and inspection you gave me. ^^

I'd glady share a few snippets from my scripts if people are interested. Mostly I made a C# program that create a dynamic blacklist file every 4 or so hours from different public blacklists and format them into a single script files for my routers to download.
I wish my FTP was FTL.
 
Taylor
newbie
Posts: 33
Joined: Wed Aug 13, 2014 12:27 am

Re: Blacklist Filter update script

Sun Mar 25, 2018 4:01 am

I would be fine with keeping the service alive. Having a 48u rack for such a piss easy and small script is a bit outrageous. I think you're the delusional one. I"m sure you used that rack for way more than this script.

I'm simply saying you can keep the script stuff online for way cheaper if you wanna still help the community.
Never said my company was dedicated to the blacklist. It's BS like this that helped me decide to shut it down.
You did, otherwise why would we care if you had to pay for a 48u rack? Why even mention that? That has nothing to do with this script.

I'd advise you to publish your scripts and collections on github and maybe someone will pick it up and use an $8/m VDS for us ;)

But i'm fairly certain you wont.
 
jgro
newbie
Posts: 46
Joined: Sat Jun 10, 2017 7:33 am

Re: Blacklist Filter update script

Sun Mar 25, 2018 4:47 am

It's been a nice run. Almost 3 years, and over 2200 active users. But I am shutting down the this service.
Thank you, Dave, for a valiant effort.

For everyone who was using Dave's Blacklist, let me recommend the Malicious IP blacklist from SquidBlackList.org, available for download from https://www.squidblacklist.org/download ... licous.rsc . I've been using it for a while and have not run into any problems because of it.

You can download and import it with a simple script:
# File path for squid blacklist. Change to use attached storage if available (e.g. "disk1/bl/drop.malicious.rsc")
:local sblPath "flash/bl/drop.malicious.rsc"
 

:log warning "Downloading squidblacklist malicious BL to $sblPath"

/tool fetch address=www.squidblacklist.org host=www.squidblacklist.org mode=http src-path=/downloads/drop.malicious.rsc dst-path=$sblPath  ;


:log warning "Importing squidblacklist malicious BL from $sblPath"

/import  $sblPath  ;
The downloaded blacklist does not actually block anything, it just creates 3 address lists you can do what you want with (1 for each of the 3 sources they use to compose the final list). I have:
/ip firewall raw
add action=drop chain=prerouting comment="Attack from sbl dshield" log=yes log-prefix="BL dshield" src-address-list="sbl dshield"
add action=drop chain=prerouting comment="Attack from sbl blocklist.de" log=yes log-prefix="BL blocklist.de" src-address-list="sbl blocklist.de"
add action=drop chain=prerouting comment="Attack from sbl spamhaus" src-address-list="sbl spamhaus"
 
Ryo
just joined
Posts: 3
Joined: Thu Jan 11, 2018 8:00 am

Re: Blacklist Filter update script

Sun Mar 25, 2018 7:22 pm

Hi Dave, thanks for ur work.

but i think mod should unpin this thread because this script/service is no longer working.
 
msatter
Forum Guru
Forum Guru
Posts: 1240
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter update script

Sun Mar 25, 2018 9:14 pm

Thanks Dave for the free ride and it worked very good. Learned a lot with which I can help other...even with my limiting knowledge of scripting.

Gonna miss the flawless working of it!
Two RB760iGS (hEX S) in series. One does PPPoE and both do IKEv2.
Running:
RouterOS 6.46Beta / Winbox 3.20 / MikroTik APP 1.3.4
Having an Android device, use https://github.com/M66B/NetGuard/releases (no root required)
 
amity2kare
newbie
Posts: 35
Joined: Tue Feb 13, 2007 4:24 pm
Location: INDIA

Re: Blacklist Filter update script

Sun Mar 25, 2018 9:24 pm

Thanks Dave,

It's people like you who keep the community alive with their contributions. Best of luck with your new role.

Regards
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Sun Mar 25, 2018 11:49 pm

Thank you Dave.
Could you be so kind and shere your valuable technology? Could you publish all scripts?
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1290
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter update script

Sun Mar 25, 2018 11:57 pm

Thank you Dave.
Could you be so kind and shere your valuable technology? Could you publish all scripts?
Unfortunately no. The server side (contrary to what a few hear think) isn't just a "script" it's a network of over 300 honeypots and some very advanced AI code to analyze threats. That system is proprietary and is still in use for the paying clients that I have left. Even if it wasn't in use, it's not just a simple script that I can post. Nor do I want to give away thousands of hours of code.

there is a chance that I will bring it back, but on a low cost subscription basis. though it's doubtful.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Mon Mar 26, 2018 12:04 am

I understand your decision about server side. What about just client side code?
Real admins use real keyboards.
 
eddieb
Member Candidate
Member Candidate
Posts: 137
Joined: Thu Aug 28, 2014 10:53 am
Location: Netherlands

Re: Blacklist Filter update script

Mon Mar 26, 2018 11:46 am

Thanks for all the effort you put into this Dave.
I was using your service for over a year and it helped me to keep my network save.
I surely would consider a small subscription fee to keep it

Eddie
Running 6.45.6 (stable) on :
CCR1009-8G-1S (2x ipsec/l2tp site-to-site, ipsec/l2tp roadwarrior, dhcpd, dns), CRS125-24G-1S, RB1100, RB962UiGS-5HacT2HnT (10pc), RB931-2nD, RB951, RB750GL ,RB2011UAS-RM, CHR running dude (CHR running in VirtualBox on OSX)
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Mon Mar 26, 2018 12:08 pm

Thank you for providing this great service .....
 
User avatar
amt
Long time Member
Long time Member
Posts: 526
Joined: Fri Jan 16, 2015 2:05 pm

Re: Blacklist Filter update script

Mon Mar 26, 2018 12:12 pm

It's been a nice run. Almost 3 years, and over 2200 active users. But I am shutting down the this service.
Thank you, Dave, for a valiant effort.

For everyone who was using Dave's Blacklist, let me recommend the Malicious IP blacklist from SquidBlackList.org, available for download from https://www.squidblacklist.org/download ... licous.rsc . I've been using it for a while and have not run into any problems because of it.

You can download and import it with a simple script:
# File path for squid blacklist. Change to use attached storage if available (e.g. "disk1/bl/drop.malicious.rsc")
:local sblPath "flash/bl/drop.malicious.rsc"
 

:log warning "Downloading squidblacklist malicious BL to $sblPath"

/tool fetch address=www.squidblacklist.org host=www.squidblacklist.org mode=http src-path=/downloads/drop.malicious.rsc dst-path=$sblPath  ;


:log warning "Importing squidblacklist malicious BL from $sblPath"

/import  $sblPath  ;
The downloaded blacklist does not actually block anything, it just creates 3 address lists you can do what you want with (1 for each of the 3 sources they use to compose the final list). I have:
/ip firewall raw
add action=drop chain=prerouting comment="Attack from sbl dshield" log=yes log-prefix="BL dshield" src-address-list="sbl dshield"
add action=drop chain=prerouting comment="Attack from sbl blocklist.de" log=yes log-prefix="BL blocklist.de" src-address-list="sbl blocklist.de"
add action=drop chain=prerouting comment="Attack from sbl spamhaus" src-address-list="sbl spamhaus"
what should i use for storage ? I have 72 core and there is no any attached storage. should i add for this job ? now its using flash. is make any problem for using flash for this job ?

thanks.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1715
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter update script

Mon Mar 26, 2018 12:25 pm

It is temporary locations to download ... it does not matter where it is ... after importing lists script could be removed form flash, disk etc.
Real admins use real keyboards.
 
User avatar
acortesguasch
just joined
Posts: 7
Joined: Tue Dec 19, 2017 6:04 pm

Re: Blacklist Filter update script

Mon Mar 26, 2018 3:54 pm

DAve, thank you fo your insight and the servce you provided. It will be missed.
I just started in the MikroTik world and your script was one of the first things I studied in order to improve my RouterOS knowledge.

Thank you very much :)
To Be Continued...
 
User avatar
Steveocee
Forum Guru
Forum Guru
Posts: 1110
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: Blacklist Filter update script

Mon Mar 26, 2018 5:49 pm

I'm totally gutted that I've now only seen this thread at the end of it's life. I was looking for something exactly like this and was under the impression nothing was about.

@OP You really need to market this as a subscription based list, judging by the amount of people who have said thanks I'm sure you'd make something out of it, whether that offsets your costs only you can determine though.
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
idoch
just joined
Posts: 3
Joined: Mon Mar 26, 2018 6:54 pm

Re: Blacklist Filter update script

Mon Mar 26, 2018 7:06 pm

First off: a Hearty THANK YOU to Dave for putting this together and sharing with the community for all this time. The usefulness of this list is undeniable and it's sad to see it go. Many networks and their users are less safe today because of the end of this list. :(

On another note, Dave - I hope that you DO begin a subscription service for this list. While the list has been demonstrably useful; I have always been reluctant to use this list in a production / business capacity BECAUSE it's completely free and BECAUSE there is no agreement, or "consideration" for using the list. Many of us are happy to pay for the service, I (for one) didn't even know that there was an option to pay for the list!.
 
User avatar
jspool
Member
Member
Posts: 393
Joined: Sun Oct 04, 2009 4:06 am
Location: Oregon

Re: Blacklist Filter update script

Mon Mar 26, 2018 8:50 pm

Thanks Dave! Its always a battle protecting ones network. The people that use free antivirus would likely be the ones that expect such a service free and the people that understand that you get what you pay for would support a paid service. Personally I prefer a BGP based solution. Since there isn't anything worth while in that arena I decided to build my own BGP blocklist system. So far beta testing has been good. On average its blocking 55,000 prefixes. As Dave mentioned it takes a considerable amount of time coding and deduplication across all of the available sources. I am integrating my own honeypot collectors now since I always seem to have a considerable number of attackers that are not present on existing lists.
 
andcz
just joined
Posts: 5
Joined: Tue Mar 20, 2018 10:36 am

Re: Blacklist Filter update script

Tue Mar 27, 2018 9:39 am

How about MikroTik company will pick up this effort, and provide the service to all the MikroTik owners ?

That would be great (and I will be even totaly willing to pay extra, like a per-year subscription or such),
and
most importantly,
this will provide a specific chain of trust - on getting the correct IP black-list from the manufacturer, that could be actually trusted.

The active black-list is a must-have for anyone running any network.

Also, there are many free, respectable services, that do publish blacklists coming from honeypots.
Example: https://project.turris.cz/en/greylist
So there should be not so much issue on getting the inputs for the official service.

I do definitelly vote for this. Anyone else ?
 
User avatar
Steveocee
Forum Guru
Forum Guru
Posts: 1110
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: Blacklist Filter update script

Tue Mar 27, 2018 10:46 am

How about MikroTik company will pick up this effort, and provide the service to all the MikroTik owners ?

That would be great (and I will be even totaly willing to pay extra, like a per-year subscription or such),
and
most importantly,
this will provide a specific chain of trust - on getting the correct IP black-list from the manufacturer, that could be actually trusted.

The active black-list is a must-have for anyone running any network.

Also, there are many free, respectable services, that do publish blacklists coming from honeypots.
Example: https://project.turris.cz/en/greylist
So there should be not so much issue on getting the inputs for the official service.

I do definitelly vote for this. Anyone else ?
I highly doubt MikroTik are going to take on a project like this. Maybe it could be something we do as a community?
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
andcz
just joined
Posts: 5
Joined: Tue Mar 20, 2018 10:36 am

Re: Blacklist Filter update script

Tue Mar 27, 2018 11:19 am

I'm in a need of such a service, thus my reaction. No offense given, I hope :)

In today's world, this shall be a native service provided with the platform.
Whichever manufacturer of routing platforms is going to provide this as a basic service, is then going to win the market.

Sure, it might be simple money to just build the hw platform with a software stack,
and let the users to build and run add ons;
yet given the actual (and future) world-wide situation, everyone shall
not only be looking for a market differentiator (which such a service could and will provide), but also to ensure the core cyber security features.
Having a official and long-term ip-black-list as a service, would be a great leap forward.
This should come from a trusted source (as the platform manufacturer would be, to a certain degree) and be a part of a standard installation (configurable, of course).

Running this from a community source is always a temporary and non-audited solution, as this thread has shown.
Sure, this original source was a side result (if I read the topic correctly) of a cyber security outfit, using all the side ways on how to remove attacks to their customer's infrastructure,
which required pushing as many RB owners to adapt the distributed list; so it in fact paid for itself (this way around).
And why not. In the end, it helped everybody. But once it stopped to be useful to the OP, the service ceased.
This is why I have this rather pushy approach to ask the manufacturer to provide this service on their own.

Just my 5 cents. Hope no one would get any bad feeling from this, not my intention.
 
User avatar
Steveocee
Forum Guru
Forum Guru
Posts: 1110
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: Blacklist Filter update script

Tue Mar 27, 2018 11:36 am

I agree with you, I would love to have a service like this available, I can think of multiple places where it would be employed.

It's not just a bunch of hardware though is it? What happens about false positives and somewhere legitimate is not reachable? Who will administer that, the manufacturer is then employing bodies to vet the lists and handle inbound queries on domains and what if something isn't picked up and it causes an issue, as a client you would hold the vendor responsible for X amount of loss of kit/earnings etc. It's a vast ocean to dive into because of a users project no longer being maintained.

Also what about when the worlds internet gears finally turn and all this investment into a platform becomes null overnight with a move to IPv6? Complete change/upgrade.

You could wait for a long time for a manufacturer to do this, which they won't as they're making money from selling their kit, or you can go to a real RBL provider and pay for a list or you hook onto community versions such as this was.
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
i4jordan
Frequent Visitor
Frequent Visitor
Posts: 75
Joined: Mon Sep 02, 2013 1:42 am

Re: Blacklist Filter update script

Wed Mar 28, 2018 11:12 pm

I also would pay for such a service, no problem.
Maybe you can make something with a pay per device/year option?

In any way thank you for the intrusBL service!
 
kakaxa
just joined
Posts: 15
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter update script

Thu Mar 29, 2018 6:51 pm

at whom the record set remained(blacklist) share please.
Dave - you many thanks that shared such remarkable set
=================
что ни у кого не сохранился лист?
поделитесь остатками пожалуйста

Who is online

Users browsing this forum: No registered users and 9 guests