Community discussions

 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Blacklist Filter (Development Topic)

Wed Jul 11, 2018 3:15 am

Hey guys, I wanted to have this a little more open of a discussion, so I made a new thread.

So I'm starting to plan out the new system and I'm going SQL based. The old system used a boat load of regex, awk, grep, etc. It was pretty dirty, but it worked. The lists generated were stored in a flat file. The new system is going to be way more flexible.

My thoughts are this...
- SQL based realtime list generation
- Subscriber managed private black/white lists (configured per device)
- Subscriber selectable list size (for 32M, 64M, 256M, 512M, 1G+ device)
- Subscriber selectable country blocking (for devices that have enough memory)
Moving to SQL will give this functionality, it will also allow the server to update the blacklists in realtime without blocking downloads. I haven't yet found a way to do non-blocking updates on the client side. (Sorry, no BGP - too complicated to manage, this needs to be fully automated)

So, this is all still only on paper, so if anyone has more ideas, lets hear it.

Here is a form to fill out if you are interested in being notified:
https://goo.gl/forms/UQMYqKJ54E0iV35l2
Last edited by IntrusDave on Sat Aug 04, 2018 11:19 pm, edited 1 time in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
msatter
Forum Veteran
Forum Veteran
Posts: 900
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Blacklist Filter (Development Topic)

Wed Jul 11, 2018 10:20 am

I was charmed by your previous implementation of using DNS to determine which version of the list and partial update (add's) should be provided.
RB760iGS (hEX S) with the SFP being cooled.
Running:
RouterOS 6.44Beta17 / Winbox 3.18 / MikroTik APP 0.69
Cooling a SFP module: viewtopic.php?f=3&t=132258&p=671105#p671105
 
hhgttg42
just joined
Posts: 6
Joined: Wed Oct 12, 2016 4:48 am

Re: Blacklist Filter (Development Topic)

Wed Jul 11, 2018 4:17 pm

[...]
- SQL based realtime list generation
[...]
How would this translate into update frequency for the clients?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Wed Jul 11, 2018 7:41 pm

I do plan on continuing to use DNS for versioning. Ultimate goal will be to have the client send the last update date and time, then request the just the changes from that point.

The effect on the client side would be that the client determines it's own update schedule.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
tigro11
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Tue Feb 20, 2018 12:31 am

Re: Blacklist Filter (Development Topic)

Wed Jul 25, 2018 10:36 pm

we use flash-start dns
http://www.flashstart.com/
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Wed Jul 25, 2018 11:49 pm

This topic is for discussion of the development for the replacement blacklist service that I build a few years ago.
Please limit posts to that topic, as your dns filtering does not help with the development at all.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
tigro11
Frequent Visitor
Frequent Visitor
Posts: 51
Joined: Tue Feb 20, 2018 12:31 am

Re: Blacklist Filter (Development Topic)

Thu Jul 26, 2018 12:16 am

sorry
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 191
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 8:02 am

I am looking forward to it and will definitely be a paying customer!!!!
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 8:27 am

So, I *JUST* started coding this week, so this is really rough.. like pre-alpha. This is going to take some time.. I can code in python, php, perl, and C. but just because I can, doesn't mean I enjoy it. I really hate coding, it's boring and frustrating. Anyway..

I have the database being populated IPs from all the big blacklist sources, I haven't written the module for pulling in the honeypot data. I'm holding off on that because I also want to either rewrite the honeypot code, or move to an open source honeypot with an API.
I'm also pulling in some whitelists. The .rsc will import dynamic address-lists named "IntrusBL" and "IntrusWL". I simply added two RAW rules, 1 to accept the WL one to drop the BL.
I also haven't built any of the accounting, or config system.

If you would like to use the pre-alpha, this script will pull it for you. just change the "priority" to 1, 2, or 3. 1 being the smallest list (about 2k), 2 being middle (18k), and 3 being the whole thing (over 135,000)
I don't recommend the priority 3 list unless you are running servers open to the world with a router that has at least 1GB RAM.
The script doesn't collect anything from your end. yet. As it nears beta, the accounting system will be in place that will require at minimum, the software ID, ether1 MAC address, and maybe CPU type. I will need these to positively identify the router so that the server can generate the router's customized list. I will also be including an opt-in option to provide some "router demographics" so I can generate stats on models/ram/etc.

So, here it is. I make NO PROMISES that it works all the time. my personal router is updating itself every hour, and my development network router is updating every time I make a commit to the source code.

please note that enabling Cloud DDNS is required, no exceptions.
:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/secureFetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;
Last edited by IntrusDave on Mon Aug 20, 2018 2:54 am, edited 2 times in total.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1606
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 9:16 am

Dave,

Thank you for your job.

Testing:
[admin@RBTEST] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl.mikrotikfilters.com/fetch.php\?priority=1";
status: failed

failure: closing connection: <500 Internal Server Error> 35.236.78.203:443 (4)
[admin@RBTEST] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl.mikrotikfilters.com/fetch.php?priority=1";
status: failed

failure: closing connection: <500 Internal Server Error> 35.236.78.203:443 (4)
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 9:29 am

fixed. having issues with mysql terminating
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1606
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 9:58 am

Fixed.
Real admins use real keyboards.
 
aboiles
newbie
Posts: 35
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 10:35 am

[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 10:38 am

[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
you are missing the "?" in the url.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
kakaxa
just joined
Posts: 12
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 11:21 am

i can't put "?" in terminal
ctrv^v reset "?" in url-address
 
sid5632
Member Candidate
Member Candidate
Posts: 253
Joined: Fri Feb 17, 2017 6:05 pm

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 12:40 pm

Use \?

The script in post #9 was wrong.
 
kakaxa
just joined
Posts: 12
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 1:26 pm

Use \?

The script in post #9 was wrong.
Thanks Sid
 
acortesguasch
just joined
Posts: 5
Joined: Tue Dec 19, 2017 6:04 pm

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 2:17 pm

Just incorporated the script to one router, 12h interval. For now, it seems to works well. We'll see.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Thu Aug 02, 2018 6:20 pm

Use \?

The script in post #9 was wrong.
The script is not wrong, it's intended to be a script, NOT command line.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jo2jo
Forum Veteran
Forum Veteran
Posts: 924
Joined: Fri May 26, 2006 1:25 am

Re: Blacklist Filter (Development Topic)

Fri Aug 03, 2018 8:26 am

I loved your service and used it, im def. willing to pay once you are live. Do you have anywhere we can sign up for an email alert or some info once the paid service is done/live? (subbing to this thread or the other main/closed thread will produce alot of "noise") tks
Last edited by jo2jo on Sat Aug 04, 2018 4:07 am, edited 1 time in total.
:beep :beep :beep
 
expert
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Sun Dec 04, 2016 1:22 pm

Re: Blacklist Filter (Development Topic)

Fri Aug 03, 2018 12:19 pm

Hi, since I'm interested about the blacklist service and in order to evaluate whether it's useful to me, I'd like to know, what exactly is blacklisted?
Who/what created such list of IPs? Thanks in advance.
 
hhgttg42
just joined
Posts: 6
Joined: Wed Oct 12, 2016 4:48 am

Re: Blacklist Filter (Development Topic)

Fri Aug 03, 2018 6:23 pm

So, I *JUST* started coding this week, so this is really rough.. like pre-alpha. This is going to take some time.. I can code in python, php, perl, and C. but just because I can, doesn't mean I enjoy it. I really hate coding, it's boring and frustrating. Anyway..
I hear that. Thank you Dave! I will be trying this out tonight to give you some more stress-testing data. Cheers!
 
jo2jo
Forum Veteran
Forum Veteran
Posts: 924
Joined: Fri May 26, 2006 1:25 am

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 4:03 am

Hi, since I'm interested about the blacklist service and in order to evaluate whether it's useful to me, I'd like to know, what exactly is blacklisted?
Who/what created such list of IPs? Thanks in advance.
the dev of this script/list uses both publicly available lists of "bad" ips (spamhaus , malcode ect), as well as his own "honeypot" devices which look for public IPs that are doing suspicious activities (then adds those IPs to his own "private" list, for distribution to ppl running his script , before he closed the service). Def was a great service + script, and one i plan on paying for once he re-launches.

I think in the main forum thread (ie not this new paid/development thread), the dev lists some of these sources he uses.
:beep :beep :beep
 
aboiles
newbie
Posts: 35
Joined: Sat Nov 07, 2015 6:52 pm

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 6:42 pm

Script is no longer functioning, no updates since last night.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1606
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 7:25 pm

Dave,

Have you considered using external to your network honeypots as source of offending IPs?
I use as the first frontier such RAW drop rules and all the time there are some IPs on the list of attackers.
add action=add-src-to-address-list address-list=RAWATTACK2 address-list-timeout=127m chain=prerouting comment=RAW2ADD in-interface-list=WAN_LIST log-prefix="RAW2ADD: " src-address-list=RAWATTACK
add action=drop chain=prerouting comment=RAW2 in-interface-list=WAN_LIST log-prefix="RAW2: " src-address-list=RAWATTACK2
add action=add-src-to-address-list address-list=RAWATTACK address-list-timeout=37m chain=prerouting comment=RAW1ADD dst-port=8291,21,22,23,2000,7547,11211,135,137-139,548,80,8080,81,37215 in-interface-list=WAN_LIST log=yes log-prefix="RAW1: " protocol=tcp
add action=add-src-to-address-list address-list=RAWATTACK address-list-timeout=37m chain=prerouting comment=RAW1ADD dst-port=8291,21,22,23,2000,7547,11211,135,137-139,548,80,8080,81,37215 in-interface-list=WAN_LIST log=yes log-prefix="RAW1: " protocol=udp
add action=drop chain=prerouting disable=yes  comment=RAW1 in-interface-list=WAN_LIST log-prefix="RAW1: " src-address-list=RAWATTACK
Real admins use real keyboards.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 8:26 pm

Dave,

Have you considered using external to your network honeypots as source of offending IPs?
I use as the first frontier such RAW drop rules and all the time there are some IPs on the list of attackers.
I'm not quite sure I follow what you are saying. I'm always open to more sources. The new system is very modular. So importing another source is as simple as coding an import module for it.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 8:27 pm

Script is no longer functioning, no updates since last night.
It's running right now.
As stated above, it's still very pre-alpha, so I can't promise that it stays running while I'm making large code changes.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
jo2jo
Forum Veteran
Forum Veteran
Posts: 924
Joined: Fri May 26, 2006 1:25 am

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 8:48 pm

dave, maybe make a google forums entry where users (who plan to pay once you go live, which i realize may be a good while away) can submit their email address to you, easily and securely (easy for you to create/keep i mean).
This way when you launch, you can send an email to all those who submitted, and you will have a decent amount of funding coming in at start (vs ppl forgetting about it or loosing track of this thread).
just an idea. tks
:beep :beep :beep
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 1606
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 9:36 pm

...Have you considered using external to your network honeypots as source of offending IPs? ...
I'm not quite sure I follow what you are saying. I'm always open to more sources. The new system is very modular. So importing another source is as simple as coding an import module for it.
Could it be possible to send to you lists of attacking IPs from my routers?
Real admins use real keyboards.
 
kakaxa
just joined
Posts: 12
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 9:46 pm

IntrusDave thx for u service
please prompt how to change a timeout of blacklist lifetime. for example for 7 days
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 10:59 pm

please prompt how to change a timeout of blacklist lifetime. for example for 7 days
The lists are set for a max timeout of 24 hours. This is required so that false positives are not blocked for too long. The system is designed to be update every 1~6 hours.
Once the system goes public, each user will be able to configure the timeout for each router.
Could it be possible to send to you lists of attacking IPs from my routers?
Yes, I am working on that too. My plan is that the routers will add IP's to a dedicated address-list, and then a script will submit that list to the server, just as the honeypots do.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 11:18 pm

Here is a form to fill out for those that want to be notified


https://goo.gl/forms/UQMYqKJ54E0iV35l2
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
kakaxa
just joined
Posts: 12
Joined: Thu Feb 01, 2018 5:46 am

Re: Blacklist Filter (Development Topic)

Sat Aug 04, 2018 11:22 pm

The lists are set for a max timeout of 24 hours. This is required so that false positives are not blocked for too long. The system is designed to be update every 1~6 hours.
Once the system goes public, each user will be able to configure the timeout for each router.
thanks for explanation, Dave
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 191
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter (Development Topic)

Mon Aug 06, 2018 5:47 am

Just put the script on my home CCR1009 and am sooooo stoked to be using your service again. Just the piece of mind will be huge for me. Will move it into production on my work Tiks after testing a few days at home. EDIT: Also Dave can you educate us on the Priority Levels 1,2,3 that are part of the service, what determines what IP address makes it in to which priority, how are they prioritized?

Dave do you have an email address or a way to touch bases off line? I am not sure why I can no longer send private messages on the forum anymore....
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Mon Aug 06, 2018 5:26 pm

currently, the priorities are pretty basic.
#1 is a short list of about 2000, consisting of just the most common botnet attacks. If I end up offering a free tier, this will be it.
#2 is a longer list of 30,000 to 40,000 IP's and subnets that includes #1, also adds most of the more common crap out there.
#3 is the largest list of 120,000 to 150,000+ IP's and subnets, includes #1 and #2, includes all "known" spammers, as well as unassigned subnets, proxies, etc.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
43north
Member Candidate
Member Candidate
Posts: 191
Joined: Fri Nov 14, 2014 7:06 am

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:24 am

currently, the priorities are pretty basic.
#1 is a short list of about 2000, consisting of just the most common botnet attacks. If I end up offering a free tier, this will be it.
#2 is a longer list of 30,000 to 40,000 IP's and subnets that includes #1, also adds most of the more common crap out there.
#3 is the largest list of 120,000 to 150,000+ IP's and subnets, includes #1 and #2, includes all "known" spammers, as well as unassigned subnets, proxies, etc.
Thanks for the info. I have been running priority 3 on my 1009 for a couple days now. First time I have used RAW rules as well. Working like a champ!!! Your list is catching everything before anything hits my "blacklist" that I have built over time from things my router has personally seen. Super awesome. Keep up the good work! Once again THANK YOU Dave.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:51 am

If anyone wants to help out more, I need more routers to report some stats to the server. This is part of the health monitoring and alerting system. If you paste the code into a terminal window, it will setup the script and start reporting.
/system scheduler
add interval=1m name=reportStatus on-event="/system script run reportStatus" policy=read,write,policy,test start-time=startup
/system script
add name=reportStatus owner=djoyce policy=read,test source=":local pa\
\_\"\"; :local pb \"\"; :local pc \"\"; :local pd \"\"; :local pe \"\"; :local pf \"\"; :local postdata \"\";\r\
\n:set pa [:tostr [ /system routerboard get ]]; :set pb [:tostr [ /system license get ]];\r\
\n:set pc [:tostr [ /system resource get ]]; :set pd [:tostr [ /system health get ]];\r\
\n:set pe [:tostr [/system identity get ]]; :set postdata [:toarray \"\$pa;\$pb;\$pc;\$pd;\$pe\"];\r\
\n/tool fetch mode=https url=\"https://bl.mikrotikfilters.com/hwstats.php\" http-method=post http-data=\"data=\$postdata\
\" output=file dst-path=hwdata.txt;"
Here is a sample from my personal firewall on what it reports:

board-name=RB1100AHx4 Dude Edition;
current-firmware=6.43rc51;
factory-firmware=3.36.3;
firmware-type=al2;
model=RouterBOARD 1100Dx4;
routerboard=true;
serial-number=735B073F0D77;
upgrade-firmware=6.43rc51;
features=;
nlevel=6;
software-id=NYLS-9KPC;
architecture-name=arm;
board-name=RB1100AHx4 Dude Edition;
build-time=Aug\/01\/2018 09:43:29;
cpu=ARMv7;
cpu-count=4;
cpu-frequency=1400;
cpu-load=0;
factory-software=6.38.4;
free-hdd-space=98365440;
free-memory=1012338688;
platform=MikroTik;
total-hdd-space=134479872;
total-memory=1073741824;
uptime=10:00:11;
version=6.43rc51 (testing);
current=488;
power-consumption=115;
psu1-voltage=243;
psu2-voltage=242;
temperature=53;
voltage=236;
name=Home_Firewall;
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Steveocee
Forum Veteran
Forum Veteran
Posts: 746
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 1:29 pm

Just put this onto my CHR home router. Had to fiddle the script a little bit to make it work though which I expected I may need to;
Note, disk1 is not present and I had to add in a "?" after the "fetch.php"
/tool fetch mode=https dst-path=/blacklist/filters.rsc url="https://bl.mikrotikfilters.com/fetch.php\?priority=3";
/import file-name=blacklist/filters.rsc
/file remove blacklist/filters.rsc

In the rsc file it has 4 filter rules at the bottom which didn't apply, I take it you need to add these in manually? Oddly doing a copy & paste didn't add them in so I made these;
/ip firewall raw
add action=drop chain=prerouting comment="DROP intrusBL" src-address-list=intrusBL
add action=drop chain=prerouting comment="DROP intrusBL" dst-address-list=intrusBL

Have stuck the fetch and remove commands into a script (intrus-bl-updater) and added into scheduler running once every 12 hours (a bit longer than suggested I know).

Added in the system reporter as well, it was set to report every minute though so have altered that slightly to 12H intervals

Amazing work Dave!
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
grusu
Frequent Visitor
Frequent Visitor
Posts: 80
Joined: Tue Aug 13, 2013 7:35 am
Location: Bucharest, Romania

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 3:30 pm

Hi Dave,

In first list first address is 255.255.255.255 . Is that right?

Thanks,
Geo
 
tippenring
Member Candidate
Member Candidate
Posts: 147
Joined: Thu Oct 02, 2014 8:54 pm
Location: St Louis MO
Contact:

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 5:04 pm

If anyone wants to help out more, I need more routers to report some stats to the server. This is part of the health monitoring and alerting system. If you paste the code into a terminal window, it will setup the script and start reporting.
Running on my home router. Do you really want it reporting every minute?
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 5:28 pm

Running on my home router. Do you really want it reporting every minute?
The reporting and monitoring service is reported every minute. The client side can change that, depending on that type of response time they want.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 5:32 pm

Hi Dave,

In first list first address is 255.255.255.255 . Is that right?

Thanks,
Geo
Yes. Once the system is complete, you will be able to whitelist if needed. I filter 255.255.255.255 because I'm on a cable network and I see a crap-load of broadcast trash.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
expert
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Sun Dec 04, 2016 1:22 pm

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 6:17 pm

I see everybody here is amazed how great service it is, but has anybody think about security risks of such service?
Importing third-party script to your router without any validation?

I wonder why this list is not provided as plain list of IPs and let everybody implement custom script parsing and validating the input.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 6:21 pm

You are 100% welcome to not use it. If you don't have anything to add to the topic, or any input on the development process, please find another topic to post in.

If you followed the previous versions thread, you would see that this has been covered in no less than 5 posts.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
User avatar
Steveocee
Forum Veteran
Forum Veteran
Posts: 746
Joined: Tue Jul 21, 2015 10:09 pm
Location: UK
Contact:

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 6:50 pm

I see everybody here is amazed how great service it is, but has anybody think about security risks of such service?
Importing third-party script to your router without any validation?

I wonder why this list is not provided as plain list of IPs and let everybody implement custom script parsing and validating the input.
The script is readily available to download and inspect before hand because any self respecting person would do that rather than blindly running it.
Dave has been here for years providing this service to users in the community and is extremely well trusted, just don't pi$$ him off and you'll be fine :lol:
Steve "Steveocee" Carter
PC Gamer, Airsofter, MikroTik Nerd
My Website - My MikroTik Tutorials
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:00 pm

I do understand the concern, but after 3 years, I'm tired of explaining myself.
I do what I do because I love the internet and I want it to be a safer place.
My company is based on this principle. I understand that you guys don't know me personally, and you have to trust me.
But do know that once this service goes public, you will have a license agreement that both sides agree to.

That said... I'm curious if the people that question the safety of my service run Windows or macOS...?
Microsoft has displayed a complete disregard for user's privacy and safety. Windows 10 forces updates even if you don't want them. Microsoft has FOUR TIMES invalidated my Volume License Keys because they changed the terms of the contract, effectively telling over 1500 users that the version of windows they are running may be pirated, only to lose a court case and be force to unban the VLK.

....and I'm the bad guy.

Anyway. Use it or don't use it. I'm not interested in providing a service to people that just tech-savvy enough to complain, but not enough to understand and see what's going on.

*MY* service will NEVER be a flat text file for anyone to download and parse. I'm spending thousands in time and money to develop the service, and I would at least like to break even.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:02 pm

on the flip side, if anyone is in Southern California (Rancho Cucamonga / Ontario / Pomona / San Bernardino) you are hit me up and I'd love to grab coffee and chat.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
expert
Frequent Visitor
Frequent Visitor
Posts: 90
Joined: Sun Dec 04, 2016 1:22 pm

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:04 pm

The script is readily available to download and inspect before hand because any self respecting person would do that rather than blindly running it.
Dave has been here for years providing this service to users in the community and is extremely well trusted, just don't pi$$ him off and you'll be fine :lol:
I don't really downgrade all the effort author put into the service and I still think it can be useful, however definitely not as directly importable (and scheduled!) script.
In my opinion, after every download it must be inspected for malicious content (what if the origin was hacked in meantime?).

You are 100% welcome to not use it.
I will not use it. Or maybe I will, but before I will implement another script that extracts IPs from the file and apply them one by one.
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:09 pm

Due to people like this guy ^^^ the beta list will be limited to the "free" list of about 1800 IP's.
I do not want my 150,000 IP's collected by my honeypots being used for other people's services.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA
 
IntrusDave
Forum Guru
Forum Guru
Topic Author
Posts: 1247
Joined: Fri May 09, 2014 4:36 am
Location: Rancho Cucamonga, CA

Re: Blacklist Filter (Development Topic)

Tue Aug 07, 2018 7:11 pm

I will not use it. Or maybe I will, but before I will implement another script that extracts IPs from the file and apply them one by one.

Now THAT is funny. I can picture you looking through a list of 150,000 IP address every 24 hours.
That's the WHOLE point of this, to have near-realtime protection that doesn't require manually parsing 15 didn't blacklist sources.
But hey, good luck. Now, please don't post here anymore.
David Joyce
Network & Security Engineer
Intrus Technologies, LLC.
Rancho Cucamonga, CA, USA

Who is online

Users browsing this forum: No registered users and 9 guests