Blacklist Filter (Development Topic)

IntrusDave · Mon Sep 10, 2018 8:21 pm

I'm in a holding pattern while my lawyer researches the EU "GDPR" laws.
It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.
That means that anyone with a honeypot running on their router will be able to delete any IP's in the database that their router may have submitted.
While this may not sound like a big deal, it's ENTIRELY possible for a user to fake an update and delete the whole database.

43north · Mon Sep 10, 2018 9:25 pm

I'm in a holding pattern while my lawyer researches the EU "GDPR" laws.
It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.
That means that anyone with a honeypot running on their router will be able to delete any IP's in the database that their router may have submitted.
While this may not sound like a big deal, it's ENTIRELY possible for a user to fake an update and delete the whole database.

Thanks for the update Dave.

Rico40 · Tue Sep 11, 2018 11:23 am

I also thank you for the update.

idoch · Fri Sep 21, 2018 5:24 pm

I'm in a holding pattern while my lawyer researches the EU "GDPR" laws.
It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.
That means that anyone with a honeypot running on their router will be able to delete any IP's in the database that their router may have submitted.
While this may not sound like a big deal, it's ENTIRELY possible for a user to fake an update and delete the whole database.

Hey Dave,
I am looking forward to the new service. Obligatory - I am not a lawyer - but... You're over thinking this. An IP address itself is not "personally identifiable" -- until it is associated with other specific data that could be personally identifiable to a natural person (cookies, MAC, RFID, etc.). Classification of an IP as the source of infection, malicious behavior, etc. (or any of the behavior a honeypot would flag) is not at all identifiable to a natural person. Check into Recital 26 in full https://gdpr-info.eu/recitals/no-26/ Here's a piece:

"The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes."

If you get Honeypot information secondhand under general classifications of "port scanner" or "wordpress prober" or "SSH Brute Force" -- I can't think of much that could be less personally identifiable to a natural person. You don't know any personally identifiable information about their interactions with the honeypot - all you have is an IP address and a general label. They can't force you to pretend that IP doesn't exist or that you couldn't find their ISP and report them to abuse@ or that you can't make decisions based on anonymized data based on "reputation" of a network -- which (by itself) is not identifiable to a natural person.

Absolute worst case: If your lists were created with an automatic timeout or expiration that was less than the 24 hour requested removal; wouldn't that be compliant? The IP would have been removed from the list after 24 hours (requested or not) -- if they get added back for ongoing bad behavior that's their problem. The list doesn't "track" them at all; each list is an independent serial number.

idoch · Fri Sep 21, 2018 10:30 pm

Showing you further that you (and your subscribers) are in the clear...
https://gdpr-info.eu/recitals/no-49/

szir · Fri Oct 19, 2018 12:14 pm

I'm new to MikroTik and I just found this thread.
I like the work that you do.
Security is important to me, so I would like to use your list. I put together a couple of rules for brute force prevention, (also reported some on abuseipdb) but blocking IPs with malicious activity that others found would be nice.

I read someone suggested DNS for updating the block list (instead of downloading a script). I would also like that. One problem I see is that as far as I know you cannot put an IP range into a DNS A record, which would make blocking whole subnets harder.
msatter suggested DNS as a means to ease the traffic generated by distributing the list.
I would like it for a different reason.

Security is important for me so I don't like the idea of downloading a script form an external source and running it on a schedule. I'm sure you are a nice and trustworthy guy, but I don't know you and don't know what security you put in place that prevents (and will prevent at any point in the future) others from hijacking your update script file. I don't want my router to become part of a botnet because it "accidentally" downloaded the wrong script...

Using DNS to update the list would eliminate the need to download and run unknown scripts, the worst that I can imagine happen in case someone takes over the DNS is that they could block innocent IPs/censor the Internet. (There might be some other way that doesn't require auto-running a downloaded script.)

IntrusDave · Fri Oct 19, 2018 5:02 pm

If using DNS is a requirement for you, I suggest you look elsewhere for the service.
I have no plans to use DNS for this service. It's not a viable distribution method.
DNS is not able to send a response of 200,000+ IP addresses.
BGP is also not going to happen, as it require a large amount of labor on both ends to configure.

I've gotten the server side stable enough to move forward. Though I may be changing the pricing.
My current thought is keeping the small list free for all..
Medium list will be accessible via donations.
Full list and custom configuration will be accessible via monthly subscription.

Steveocee · Fri Oct 19, 2018 6:42 pm

I've watched list "2" slowly grow over time, I think it was "only" around 14,000 entries when you first started this thread off and now it is up to 23,500+ entries. Seriously amazing stuff Dave.

Rico40 · Fri Oct 19, 2018 9:00 pm

Dave since when will be donated for a medium list?

IntrusDave · Fri Oct 19, 2018 9:40 pm

I’m not sure I understand the question

Rico40 · Fri Oct 19, 2018 9:57 pm

I ask since when we pay for the product.
I'm sorry but I'm not good at English.

IntrusDave · Fri Oct 19, 2018 10:52 pm

The pay service will begin on the 1st of the year

Rico40 · Fri Oct 19, 2018 11:22 pm

OK, I'm waiting for information on how to transfer money.

jausovec · Tue Oct 30, 2018 12:24 pm

Hi.

Can someone sum up the latest script/instructions on how to install the new service (and uninstall the old one

)?
Or are we not so far yet, that we could start using it in home environment?
I am also avaiting info about how/where to we can make the payment for the service.

Thank you

boldsuck · Fri Nov 02, 2018 2:02 pm

It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.

This fucking GDPR

This law harms citizens more than it helps. A shot in the knee.
Some good forums has closed and from Germany you can not order anything from some shops in Switzerland anymore.

@Dave
Do you have IP blacklists from squidblacklist.org in priority 1 or 2?

@all
Because here is increasingly asked for payment:
Just fill out the form by Dave, if you haven't yet. Then you will receive an e-mail in time.
https://goo.gl/forms/UQMYqKJ54E0iV35l2

@jausovec
Disable or delete the old Blacklist script(s) and scheduler and fetch the new.
Post Nr.9 in this topic:
viewtopic.php?f=9&t=136666#p677573
Adjust 'destPath' and 'priority'.
Fix new schedulers or adjust the old ones.

HZsolt · Sat Nov 03, 2018 12:35 pm

For IntusDave:
Do you have any problem or do you update? I run your script but the script didn't download nothing.

I thank you for your help!

Steveocee · Sat Nov 03, 2018 1:27 pm

For IntusDave:
Do you have any problem or do you update? I run your script but the script didn't download nothing.

I thank you for your help!

Are you running IP > Cloud ? Would be the "easiest" thing to check at this point as it is a prerequisite.

HZsolt · Sat Nov 03, 2018 1:43 pm

For IntusDave:
Do you have any problem or do you update? I run your script but the script didn't download nothing.

I thank you for your help!
Are you running IP > Cloud ? Would be the "easiest" thing to check at this point as it is a prerequisite.

Yes, IP -> Cloud is running. DDNS Enabled and updated.

IntrusDave · Sat Nov 03, 2018 5:47 pm

The server is hosted on the google cloud platform. It appears that Google has oversold the zone that my servers are in, and my servers have been taking offline to allow others to run. I'll be moving the server to a different zone ASAP.

IntrusDave · Sat Nov 03, 2018 7:07 pm

@Dave
Do you have IP blacklists from squidblacklist.org in priority 1 or 2?

squidblacklist.org is not included, as it is a pay service. The IP's they have on the free lists are pretty much duplicates of my list 2.

IntrusDave · Sat Nov 03, 2018 8:10 pm

I bought a new network "level 3" honeypots online. These are my first located in "hostile" countries. Bringing the current Level 3 list to over 189,000 entries.

HZsolt · Sat Nov 03, 2018 8:15 pm

The server is hosted on the google cloud platform. It appears that Google has oversold the zone that my servers are in, and my servers have been taking offline to allow others to run. I'll be moving the server to a different zone ASAP.

Thanks!

Your blacklist works well again!

Thanks!

IntrusDave · Sat Nov 03, 2018 8:17 pm

No problem, and I'm sorry about the downtime.
I've changed the policy on my servers so that Google can not longer preempt mine to make room for higher paying customers. Not happy that it's adding another $50/month onto my bill, but I can't have them dropping me because someone bigger wants my cpu or memory.

IntrusDave · Sat Nov 03, 2018 8:56 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

HZsolt · Sat Nov 03, 2018 9:30 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

How much memory use on routers with these Blacklist Filters?

IntrusDave · Sun Nov 04, 2018 9:20 am

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.

HZsolt · Sun Nov 04, 2018 10:55 am

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.

What do you think about this service?: viewtopic.php?t=137632

acortesguasch · Sun Nov 04, 2018 12:54 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

Any estimate regarding the end-of-service date as it is? I have to convince my boss of the benefits of being a paying member of the community and I do not want to be caught offguard.

Keep the fantastic work!

IntrusDave · Sun Nov 04, 2018 5:53 pm

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.
What do you think about this service?: viewtopic.php?t=137632

I think it's a ripoff of my project by someone that hasn't been a part of the community as long as I have. I think I am far more transparent in the development process.

boldsuck · Sun Nov 04, 2018 6:05 pm

squidblacklist.org is not included, as it is a pay service. The IP's they have on the free lists are pretty much duplicates of my list 2.

OK, now I'll be clear here

Thanks.
Will test how much RAM a RB2011 needed. Only with priority 2
or priority 1 + drop.malicious.rsc

PS:
SBL Malicious IP Blacklist from: https://www.squidblacklist.org is free of charge.

IntrusDave · Sun Nov 04, 2018 6:05 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.
Any estimate regarding the end-of-service date as it is? I have to convince my boss of the benefits of being a paying member of the community and I do not want to be caught offguard.

Keep the fantastic work!

The goal is NO end-of-service date. This started out as a project out of a personal need, I shared it with the community because I thought others could use it too. Last year, the original service hit just over 17,000 active devices. I realized that several LARGE businesses were using the service and SELLING IT to their customers. At that point I felt that it was "fair" for me to be paying several hundred per month for servers and honeypots out of pocket, while others were making money off it it.

My vision is to have a service that is simple to implement, stable, secure, fast, and self-sustaining. Once the income is able to cover the expenses, and it is able to keep running without me, then I can focus on bringing new features and adding support for new platforms.

I don't want or plan to get rich from this. I want to provide a valuable service at a price that anyone can afford.

As for yearly payments - At this time, I am going to keep it monthly. I don't want to accept a payment for a year of service before I know the service will be able to sustain itself.

IntrusDave · Sun Nov 04, 2018 6:07 pm

squidblacklist.org is not included, as it is a pay service. The IP's they have on the free lists are pretty much duplicates of my list 2.
OK, now I'll be clear here Thanks.
Will test how much RAM a RB2011 needed. Only with priority 2
or priority 1 + drop.malicious.rsc

PS:
SBL Malicious IP Blacklist from: https://www.squidblacklist.org is free of charge.

I compared mine with that one - All of the IP's in that are also in mine. The key difference is the delivery method. My .rsc is much smaller and processed faster.

boldsuck · Sun Nov 04, 2018 6:31 pm

The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.

As for yearly payments - At this time, I am going to keep it monthly. I don't want to accept a payment for a year of service before I know the service will be able to sustain itself.

Would be nice to be able to select priority 1, although you have paid for priority 2.

Uh, monthly. I hope the monthly payment can be automated.

acortesguasch · Sun Nov 04, 2018 6:38 pm

Any estimate regarding the end-of-service date as it is? I have to convince my boss of the benefits of being a paying member of the community and I do not want to be caught offguard.

Keep the fantastic work!
The goal is NO end-of-service date. This started out as a project out of a personal need, I shared it with the community because I thought others could use it too. Last year, the original service hit just over 17,000 active devices. I realized that several LARGE businesses were using the service and SELLING IT to their customers. At that point I felt that it was "fair" for me to be paying several hundred per month for servers and honeypots out of pocket, while others were making money off it it.

When I asked about the end-of-service as it is I was referring when the current scripts will be deactivated and only working via Patreon.

I think I understood since the beginning the kind of project you are running, for you and for the Community, and I cannot see any flaw in you reasoning. You are giving a lot to the Community and it is only fair to try to cover expenses.

IntrusDave · Sun Nov 04, 2018 7:49 pm

My goal is January first.

HZsolt · Sun Nov 04, 2018 8:18 pm

My goal is January first.

After January first the current your script will not work?

IntrusDave · Sun Nov 04, 2018 9:07 pm

The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.

As for yearly payments - At this time, I am going to keep it monthly. I don't want to accept a payment for a year of service before I know the service will be able to sustain itself.
Would be nice to be able to select priority 1, although you have paid for priority 2.

Uh, monthly. I hope the monthly payment can be automated.

Payment is automated via the Patreon page. Each Tier includes the Tier below it. I've updated the tires on the page to better explain what you get.

Rico40 · Sun Nov 04, 2018 10:00 pm

How to make a payment from Poland?

IntrusDave · Sun Nov 04, 2018 10:01 pm

How to make a payment from Poland?

https://www.patreon.com/IntrusTechnologies

You can go to this Patreon page to sign up.

Rico40 · Sun Nov 04, 2018 10:02 pm

OK thanks.

hhgttg42 · Mon Nov 05, 2018 5:05 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

I'm already signed up! Thanks again for the great service Dave.

tippenring · Mon Nov 05, 2018 5:53 pm

OK, now I'll be clear here Thanks.
Will test how much RAM a RB2011 needed. Only with priority 2
or priority 1 + drop.malicious.rsc

I'm using the priority 2 list on an RB2011. Memory is fine. I currently have free 74MB of 128MB with ~30k blacklist entries. The RB2011 is more CPU starved when it's updating the list or if you view the address list.

boldsuck · Mon Nov 05, 2018 7:21 pm

Payment is automated via the Patreon page. Each Tier includes the Tier below it. I've updated the tires on the page to better explain what you get.

Perfect thank you.

Mikrotik forum logged me off last night again and again.

I'm using the priority 2 list on an RB2011. Memory is fine. I currently have free 74MB of 128MB with ~30k blacklist entries. The RB2011 is more CPU starved when it's updating the list or if you view the address list.

I've deleted most of the drop.malicious.rsc address-list entrys (~30k blocklist.de) and switched to priority 2 (also about ~30k). I have 7 MB more Ram free. (Free Memory now 55MB)

anav · Mon Nov 05, 2018 9:45 pm

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.
What do you think about this service?: viewtopic.php?t=137632
I think it's a ripoff of my project by someone that hasn't been a part of the community as long as I have. I think I am far more transparent in the development process.

Sounds like an emotional response based on little fact. You would be better served by recognizing and supporting a like minded fellow provider who decided to share his work for his customers/clients with the community at large. The development cycle taken matches up with what I would have done, first being exposed to Josh Haven and the various sources, such as the Firehol lists, magically available whether you were on this less and less green earth or not. On his own he matured his script skills to be able to create the database and program with the flexibility and 'scalability' of his clients in mind (hex to larger units) which is critical to many of us with lesser units (not companies with fat wallets). Its stable, it works and its phukking affordable. He obviously has put much time and effort into the program, considering the servers required and the fail over and many other detailed minutia it takes to run a credible service. As to transparency, what are you referring too? He opened up his development to anybody that was interested for testing purpose. He clearly outlines his sources which do not contain 'secret' honeypots.

Don't get me wrong, I support what you are attempting to provide as much as the next person. I just hate to see unwarranted antagonism. If I had the skills I would be tempted to do the same and provide such a service. Good luck on progress in the next months!!

IntrusDave · Mon Nov 05, 2018 9:51 pm

If you want to support him, please support him in his topics.
If you would like to be involved in the me-vs-him debate, please at least do a little background research on the history.
My project has been the subject of "IP Theft" several times, and I do my best to keep my server side tech hidden now because of that.
I shut down my project once before, one of the many factors was other projects taking my lists and pushing them out as their own.

People are free to choose what they want to use, but if you want to talk about his stuff, please do so elsewhere.

IntrusDave · Mon Nov 05, 2018 10:03 pm

Tech Note: 6.44beta20 causes some issues, make sure you update to the current beta, if you are running the betas.

Rico40 · Tue Nov 06, 2018 12:33 am

I'm already signed up payment from January?

RackKing · Tue Nov 06, 2018 6:39 pm

So maybe a dumb question... I did have a look a the Patreon page. What level would you recommend to an integrator like who would offer this to his customers as part of a annual service offering? I would bill them directly and purchase your service. I suppose I could buy a tier and then upgrade as I cross that threshold? Will it be easy to see how many "routers" I have left to use?

I think this is a great project and thank you for all your efforts.

IntrusDave · Tue Nov 06, 2018 10:12 pm

Their are no dumb questions!

I will for sure have a UI for you to manage your routers. My goal is to have the UI finished by mid December. Though I just missed a pretty big deadline, so I may end up having to hire another developer to work on the front end while I continue on the back end. The UI is expected to use the Patreon login, then provide you with the total number of routers you can enroll, as well as the currently enrolled and management of the serial numbers. Authentication of the routers will be handled by the email address you subscribed with, along with the serial number of the router. Each router will have one unique address entry (randomly generated) that will allow me to find users that are misusing or claiming my lists as their own.

Cooperdale · Sun Dec 30, 2018 6:16 pm

Hello, are there any news on this? I can't wait for this service to come alive.

Rico40 · Sun Dec 30, 2018 8:44 pm

The service works well, the first payment has been sent.

boldsuck · Mon Dec 31, 2018 11:03 pm

The service works well, the first payment has been sent.

My Also,

but where to submit my router's Cloud DNS name?

Rico40 · Mon Dec 31, 2018 11:36 pm

let's wait for a message from Dave.

IntrusDave · Tue Jan 01, 2019 10:36 am

Hey guys, sorry for the radio silence, it’s been a pretty tough year but I’m trying to survive it.

I’ve been trying to get the automated registration process done but have ran into a few pretty nasty issues that I didn’t foresee. I’m going to put it on hold and process things manually until I can get some cash to pay another coder to work with me.

Going to spend the rest of the week with my kids, and then start fresh Monday morning.

sjoram · Sun Jan 27, 2019 2:42 pm

I found your Patreon.

I looked at the different 'tiers' - $10 currently works out about £7.50 a month...I'd be more than happy to support your work.

However, I do have a couple of questions (others with knowledge of your project may also have views) - sorry if this is not the best place to ask, but it seemed most likely to get the right visibility:

- I'm a home user, with ROS on two RouterBOARDs. Just about to swap both out with new RB750Gr3. I believe these have 256MB RAM, can these handle the largest tier?
- I found out about your work as I was one of the silly people that when a novice to ROS didn't include a PPPoE WAN interface on the default drop rules AND had some ROS services open to the wild, without port knocking. Also lax with updates, so Winbox vulnerability got exploited on one box. Yes, I know... Now fixed.
- One box particularly has a plethora of open dst-nat rules - mail server and the likes. I'm thinking that despite now having the default drop rules correct such that ROS itself is less vulnerable, your project may still be a benefit in protecting against unknown/unpatched exploits from known rogue addresses, not just for ROS but also the services behind those dst-nat rules... Would you agree?

IntrusDave · Tue Feb 12, 2019 3:21 am

Hey everyone. I'm sorry for being out of touch. Here is an update.

So, life has been pretty rough over the last 12-14 months. I've lost most of my clients, the new government tax code is killing me, and server costs keep going up. On the personal side, I've been dealing some some pretty serious health issues, and I'm not entirely sure how this is going to end for me.

That said, I can't in good conscience bill people for something that I can't guarantee that I can keep working on. I do have my notebook with me all the time, but some of the drugs are simply preventing me from thinking clear enough to do any coding at all.

So, I'm going to leave it open to all, and hope that at least a few of you will donate to keep the servers up and running.
I will do my very best to continue to improve the lists and the script.

I'd like to leave this PayPal Donation button here. Use if you like, don't if you don't like.

Again, as long as the service can support itself (about $200/month USD) I'll leave it running.
I have also taught my 14 year old daughter how to keep the servers running (she's already studding quantum physics) so that if the worst case happens, she can keep it running for you all.

I can certainly promise that NONE of the donations will go to my medical bills. I don't want anyone thinking that I'm using donations for anything other than what they expect.

Thank you, and I hope to keep posting.

https://www.paypal.com/cgi-bin/webscr?c ... source=url

BTW - this is the code that I use, and I prefer anyone to use for the script... make note of the path.

:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/fetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;

sjoram · Tue Feb 12, 2019 10:27 am

Hi Dave,

Very sorry to hear of the challenges that life has thrown at you of late. I sincerely wish you and your family all the very best.

Thank you for your work on this, you know yourself how much demand your servers have seen, so I am sure this is benefiting and making life easier for a lot of people!

Rico40 · Tue Feb 12, 2019 9:14 pm

Dave donates funds to patreon and here he will also donate

IntrusDave · Fri Feb 15, 2019 12:59 am

Humans are truly awful.
While at my treatment yesterday, I dozed off. (I'm there for 6 hours every 3 days)
While sleeping, someone stole my backpack with my 6 month old notebook.

I'll still be doing some coding at home, but it's hard to sit at my desktop. I'll keep you all posted.

ihave · Fri Feb 15, 2019 2:45 am

That is truly awful indeed!
Sorry to hear that.

Can someone sum up the latest script/instructions on how to install the new service?
The old install script with blInstaller.rsc doesn't show anything in the log.

Filterimport.rsc is running fine and updating the address list. Am I only missing the scheduler or did the BlInstaller.rsc install additional things I am still missing?

Thank you

boldsuck · Fri Feb 15, 2019 8:10 pm

Can someone sum up the latest script/instructions on how to install the new service?

Script is 4 post higher.

ihave · Fri Feb 15, 2019 9:18 pm

Script is 4 post higher.

That script is running fine. I am just wondering about the frequency of the scheduler and maybe other things that were installed with the old installer.

boldsuck · Sun Feb 17, 2019 3:07 pm

That script is running fine. I am just wondering about the frequency of the scheduler and maybe other things that were installed with the old installer.

OK. Mmmmh, frequency of the scheduler:
That makes everyone different. I update every 12 hours. I think Dave updates the blacklists every hour.

I summarize everything for the people who are new here:

- Enable ddns. I have configured the SNTP client and do not need time here.

/ip cloud set ddns-enabled=yes update-time=no

- The script (destPath and priority may need to be adjusted):

/system script add dont-require-permissions=no name=blacklistScript owner=admin policy=read,write,policy,test source=\
\n"# Intrus Technologies blacklist installer/updater\
\n# \A92017 David Joyce, Intrus Technologies\
\n\
\n:local destPath \"filterImport.rsc\";\
\n:local priority \"2\";\
\n\
\n#If you do not want to delete the script directly after importing, remove the comment and paste one in the last line\
\n#:do { /file remove \$destPath } on-error={};\
\n\
\n:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] \".\"]];\
\n/tool fetch mode=https url=\"https://bl.mikrotikfilters.com/fetch.php\?priority=\$priority\" http-method=post http-data=\"\$sn\" dst-path=\"\$destPath\" output=file;\
\n/import file-name=\$destPath;\
\n/file remove \$destPath;\
\n"

- The sheduler (After every reboot and then every 12 hours at 6:00 and 18:00) Adjust times as you want:

/system scheduler add interval=12h name=blacklistScriptUpdate on-event="/system script run blacklistScript" policy=read,write,policy,test start-date=jan/01/1970 start-time=06:00:00
/system scheduler add name=blacklistScriptUpdateOnBoot on-event=":delay 30;system script run blacklistScript" policy=read,write,policy,test start-time=startup

- The firewall rule:
Others also filter outgoing traffic (dst-address-list) and/or other interfaces. I block all port scanners, bruteforcer and blacklists in the beginning in the RAW chain. (Connection tracking would be a waste of time

I prefer to use the saved computing time for tarpit rules. I tarpit all WAN connections to unused TCP ports of the router.

/ip firewall raw add action=drop chain=prerouting comment="Drop WAN connections from 'intrusBL' blacklisted hosts <- Src. Address List: intrusBL" in-interface-list=WAN src-address-list=intrusBL

Hope that helps

boldsuck · Sun Feb 17, 2019 4:15 pm

@Dave

Wow, 14 year old woman is interested in coding! Very nice.

Crap. Stolen laptop is one of the worst cases.
I do not know exactly "New US government tax" but I think that's what Mr. Trump introduced. (He needs money for his wall. In Germany, we are glad that the wall is gone!)

Should we still run the reportStatus script? (That from #Post 37)

IntrusDave · Fri Mar 01, 2019 3:16 am

The ReportStatus is 100% optional. It's mostly for stats and being able to ty and tailor the lists to the bulk of the routers. Surprisingly, their are far more CloudCore routers running the blacklist now than the smaller units.

I wanted to thank those of you whom have reached out with support. Every little bit helps! It's been a few weeks and I'm starting to grow hair again... Just when I was getting used to being bald. It won't be long though, treatments start again in 6 weeks.

IntrusDave · Tue Apr 02, 2019 6:05 pm

Good Morning and happy April.

A few updates for you this morning.

Server is running stable with no issues other than the random LetsEncrypt cert error.
Once the list's income manages to cover it's costs, I'll be switching to Digicert for ssl certs.
I've begun work on adding IPv6 to the service, it's pretty easy on the router side, changes things quite a bit on the backend
For those following, the police found my notebook, though it's completely wrecked. they are keeping it as evidence. They didn't find any of my tools.
as for the service pricing, I'm still keeping it as a donation only service. Again, I don't want to bill for a service that I may not be around to keep running.

Rico40 · Tue Apr 02, 2019 9:12 pm

It can be paid by Patreon?

msatter · Tue Apr 02, 2019 11:18 pm

Humans can be truly awfull but using you undergoing your treatment to steal from you then there are no words to describe my feelings about that.

I am sorry to read that you are ill and that the outcome is uncertain. I wish all the strength to overcome this horrible time in your life.

IntrusDave · Wed Apr 03, 2019 1:10 am

It can be paid by Patreon?

Yes, somewhere up above, there is a Patreon link, as well as PayPal links.

IntrusDave · Wed Apr 03, 2019 1:11 am

Humans can be truly awfull but using you undergoing your treatment to steal from you then there are no words to describe my feelings about that.

I am sorry to read that you are ill and that the outcome is uncertain. I wish all the strength to overcome this horrible time in your life.

Thank you. No worries, I’ve survived everything else life has tossed my way. Even an earthquake that brought my apartment down on me.

If I could just keep my fingers and toes warm, I would be happy.

IntrusDave · Fri Apr 26, 2019 12:23 am

Just an update... I'm still here, and I have hair again.

Well, I have hair for a few weeks at least.

I'm still working on the IPv6 version of the list. My home ISP has finally managed to provide a stable v6 connection, so I'll be able to start testing.

boldsuck · Sun May 05, 2019 4:59 pm

I'm still working on the IPv6 version of the list. My home ISP has finally managed to provide a stable v6 connection, so I'll be able to start testing.

IPv6 Yeah! That's very good news.
Have meanwhile a few servers on the run. (Debian, Static IP & IPv6)
If you want to test something on it ...
can I give you access. I only need a pub ssh-key from you.

IntrusDave · Tue May 28, 2019 5:43 am

I regret to inform you all that I have shut down the servers.
With over 35,000 routers hitting the server every few hours, and only 10 supporters (totally $50/month), the expenses are not even remotely being covered. I'm putting out almost $500/month now just in bandwidth costs.

I'm moving the code to my in home server where it will just be supporting my personal units now.

it was a good run. I tried to keep it going, I thought that more people would be willing to help, but sadly not.

wanos · Tue May 28, 2019 2:02 pm

We truly appreciated all your efforts Dave. Well, too few of us apparently did. So many want something for nothing. Sucks.

Thank-you for that and hope all goes well for you and your family.

hhgttg42 · Tue May 28, 2019 4:41 pm

Sorry to hear that Dave. Please let us know if you ever change your mind!

Best of luck.

HZsolt · Tue May 28, 2019 8:08 pm

[DELETED]

IntrusDave · Tue May 28, 2019 8:25 pm

Dude, really? My lively-hood is going down the drain, and you pop up to help push the knife in. Nice.

Rico40 · Wed May 29, 2019 8:11 am

I resigned from patreon in favor of donate paypal.

Chupaka · Sat Jun 08, 2019 11:47 am

Dave, any chance to get the code open-sourced? I mean, maybe someone would like to donate servers/bandwidth instead of money...

pe1chl · Sat Jun 08, 2019 12:43 pm

I regret to inform you all that I have shut down the servers.
With over 35,000 routers hitting the server every few hours, and only 10 supporters (totally $50/month), the expenses are not even remotely being covered. I'm putting out almost $500/month now just in bandwidth costs.

I'm curious how much bandwidth your service consumes... there is another poster complaining about $500/mon bandwidth cost for his service, but frankly I cannot understand how this can happen.
For that $50/mon I would get about 10TB/mon of bandwidth at local cloud hosting companies, and with 35000 routers that would be 285MB per router per month.
When that isn't enough by a factor of 10 I would seriously consider revising the method for distributing the data... e.g. some form of incremental updating.

msatter · Sat Jun 08, 2019 1:41 pm

I helped with an earlier version and it is should be incremental and your get the changes you missed since the last sucessful update you had.

The sheer number of routers connecting still can give a heavy bandwith usage.

Dave is doing a great job despite his personal set backs.

viewtopic.php?t=98804

Rico40 · Sat Jun 08, 2019 8:19 pm

it's more about reducing the cost of the service.

pe1chl · Sat Jun 08, 2019 9:25 pm

Please show a calculation of how much bandwidth per month you need per router using the service, and how much the cost for different amounts of bandwidth per month is.

IntrusDave · Thu Jun 13, 2019 9:00 pm

Sorry, but I feel no need to disclose my stats and financial needs for a service that is free.
I can tell that you 4 servers, 120 honeypots, a CDN, storage and the bandwidth needed for all of it is quite a lot.
I wont be open sourcing the code either. it's 100% written by my with no use of any open source code.
It has a use to me still and I will be keeping it for myself.

suszi · Thu Jun 13, 2019 9:28 pm

the problem is, with RB, that ip firewall raw action=drop doent work with src-address-list=intrusBL
according to documentation:
address-list (string; Default: ) Name of the address list to be used. Applicable if action is add-dst-to-address-list or add-src-to-address-list
https://wiki.mikrotik.com/wiki/Manual:IP/Firewall/Raw

/ip firewall raw
add action=drop chain=prerouting comment="DROP intrusBL" src-address-list=intrusBL
add action=drop chain=prerouting comment="DROP intrusBL" dst-address-list=intrusBL

any way to use RAW? or just regular firewall rule?

suszi · Thu Jun 13, 2019 9:29 pm

I'm updating my second portscanners (TCP SYN) list manually - where can I send it, to be included ?
maybe there is a better way ?

msatter · Thu Jun 13, 2019 11:05 pm

The ADD in the rules is there to add the line to the RAW section in the firewall. After thst it not used anymore.

Dropping unwanted traffic is most efficient in RAW and so it won't reach connection tracking.

Chupaka · Fri Jun 14, 2019 12:38 am

the problem is, with RB, that ip firewall raw action=drop doent work with src-address-list=intrusBL
according to documentation:
address-list (string; Default: ) Name of the address list to be used. Applicable if action is add-dst-to-address-list or add-src-to-address-list
https://wiki.mikrotik.com/wiki/Manual:IP/Firewall/Raw
Code: Select all
/ip firewall raw
add action=drop chain=prerouting comment="DROP intrusBL" src-address-list=intrusBL
add action=drop chain=prerouting comment="DROP intrusBL" dst-address-list=intrusBL
any way to use RAW? or just regular firewall rule?

Address-list, src-address-list and dst-address-list are three different parameters. You're talking about one and look at the description of another.

boldsuck · Thu Jun 20, 2019 10:42 pm

I thought that more people would be willing to help, but sadly not.

me too.

I have now unsubscribed at Patreon. ;-(

Thanks for everything Dave. You and this thread inspired me to refine my own raw bruteforce & portscan rules.

I'm trying to do something similar. With fail2ban (0.10 has ipv6 support) My servers run as fail2ban reporting service anyway. For months there is already a UBNT ER-Pro 8 around here. On EdgeOS, packages from the debian archives can be installed. Fail2ban should then be easy to install on the router.

Who is online