Community discussions

MikroTik App
 
TheSirStumfy
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 90
Joined: Sun Oct 14, 2018 7:54 pm

Large blacklists for firewall

Fri Feb 14, 2020 12:14 pm

Hello everyone,

Does anyone have any experience with large block lists?

I am running an email server and get hit with brute force password attacks from IPs that are commonly found in blacklists.

Although the server features and is set up for automatic lockout of IPs that do multiple attempts at passwords, i would like to move this to firewall.

Does anyone have any experience in what kind of a hit on performance a 11k line blacklist makes to the router? The router is a RB3011.

regards
 
User avatar
pcunite
Forum Guru
Forum Guru
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: Large blacklists for firewall

Fri Feb 14, 2020 3:14 pm

11K should be fine. I have 4,000 on an RB3011 and its no trouble. Use RAW rules something like this:

/ip firewall raw
add action=drop chain=prerouting disabled=yes in-interface=ether1 src-address-list=PortScanners
add action=add-src-to-address-list address-list=PortScanners address-list-timeout=2w chain=prerouting disabled=yes dst-port=10,25,333 in-interface=ether1 protocol=udp src-address-list=!WinboxAllow
add action=add-src-to-address-list address-list=PortScanners address-list-timeout=2w chain=prerouting disabled=yes dst-port=10,25,333 in-interface=ether1 protocol=tcp src-address-list=!WinboxAllow
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Fri Feb 14, 2020 3:17 pm

Just try it out and used the following example.RSC file to import your IP list:
:global i do={ /ip firewall address-list add list=blacklist-email timeout=35w3d13:13:56 address="$a" }
:do { /ip firewall address-list remove [find where list=blacklist-email] } on-error={}
$i a=x.x.x.x
$i a=x.x.x.x
$i a=x.x.x.x
.
.
.
$i a=x.x.x.x
$i a=x.x.x.x
$i a=x.x.x.x
This will import the ip address (represented by x.x.x.x) very efficiently.
 
TheSirStumfy
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 90
Joined: Sun Oct 14, 2018 7:54 pm

Re: Large blacklists for firewall

Fri Feb 14, 2020 3:50 pm

Ok, thank you, i see that you both use address list timeouts.

Im i correct in understanding that this is so IPs that never get detected in the list get removed?

Also i see pre routing is used instead of input, this is to save routing overhead of CPU right?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Large blacklists for firewall

Fri Feb 14, 2020 6:29 pm

Timeouts are used so that config doesn't get written to NV storage. Neither this part consumes space in exported config.

Pre-routing is used so that these connections get dropped for both input and forward ... and they get dropped as soon as possible.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 12:50 pm

If I can, I suggest a change.
:global listname "blacklist-email"
:global i do={
    :global listname
    /ip firewall address-list
    :if ( ([:len [find where list="$listname" and address="$1"]] = 0) and ([:typeof [[:parse ":return $1"]] ] ~ "(ip|ip-prefix)") ) do={
        add address="$1" list="$listname" timeout=35w
    } 
}
/ip firewall address-list remove [find where list="$listname"]
$i x.x.x.x
$i x.x.x.x
.
.
.
$i x.x.x.x
$i x.x.x.x
Last edited by rextended on Sat Aug 14, 2021 7:00 pm, edited 2 times in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Thu Aug 12, 2021 1:32 pm

@rextended you are on my ignore list so I don't read your postings anymore. Really sad that I had to resort to that.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 1:39 pm

@rextended you are on my ignore list so I don't read your postings anymore. Really sad that I had to resort to that.
So with no explanation and no reason?

Of course there are really idiotic people in the world...

For me the dawn will continue to be there tomorrow,
even for you,
but you will miss a star in the firmament of the night that writes the history of this world.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 3:07 pm

thanks for the stock tip in this thread rextended. I made $100,000 Euros! Too bad msatter missed out. :-)

Okay this thread is similar to another one rextended where you were adamant not to pull third party sites into a script etc etc.
How is this one different? Its not clear to me where the black list is coming from? What did I miss?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 3:10 pm

Is for self-made blacklist.
having already a file with all ip inside, add the script on the top
and on front of each ip put "$i "
then save the file as .rsc and import it.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 3:13 pm

where you were adamant not to pull third party sites into a script etc etc.
To be precise, there is no problem in importing blacklists from sites, reading the file and following an internal import script,
but about the problem of "IMPORT SCRIPT WITH ROUTEROS COMMANDS" inside, instead of just IP
Last edited by rextended on Thu Aug 12, 2021 3:15 pm, edited 1 time in total.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 3:14 pm

-TX-
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Thu Aug 12, 2021 3:16 pm

-TX-
If you see this... is a revolution... :mrgreen:
viewtopic.php?f=9&t=177530
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Fri Aug 13, 2021 11:21 pm

@rextended you are on my ignore list so I don't read your postings anymore. Really sad that I had to resort to that.
So with no explanation and no reason?

Of course there are really idiotic people in the world...

For me the dawn will continue to be there tomorrow,
even for you,
but you will miss a star in the firmament of the night that writes the history of this world.
Excellent example why I put you on my ignore list or how it is called here I go a foe.

I really hope you will be banned here soon!!!
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 12:55 am

msatter no need to get nasty, and besides, Netherlands will be under water before volcanoes destroy Italy so you just may be looking for a home soon.
Dont cut off your options!!!

Personally I have grown fond of italien poetry in broken english, almost romantic.
How could anyone not eat up and smile when the Italian high jumper tickled the ears of the newly crowned 100 metre champion at the olympics.
They are different but in a good way LOL.

PS, Your ignore list doesnt work very well. I am sure rextended has a script for you that will work.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Sat Aug 14, 2021 1:52 am

We are sending truckloads of Euro's to Italy despite they are richer than we are in the Netherlands. We are buying that country without owning it after sending all our money.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 2:02 am

@green-gelid You can also add "no Money" to your signature :lol:
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Sat Aug 14, 2021 12:32 pm

PS, Your ignore list doesnt work very well. I am sure rextended has a script for you that will work.
The ignore function works very well here. If you want to read a ingnored posting, it depends on which one of the two fingers you use for the screen.
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 5:04 pm

If I can, I suggest a change.
@rextended --- I tried your code on a test file have 50 thousand IP address entries and found that @msatter code was dramatically faster to process .... approximately 4 times faster than your code. Also FYI, your code generates the following error condition
failure: empty list name not allowed
Did you try your code out on a test file before publishing your code here? May I suggest that you give it a try and compare your code to @msatter code ... see for yourself :)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 6:30 pm

The only way to obtain the error "failure: empty list name not allowed" is if you do not set list name, is your fault.
The script is more slow because check every ip or ip-prefix is valid before try to insert on address-list.
On this way the script do not stop if one ip or ip-prefix are wrong, simply "skip".

On the other "faster" script, if are some error, the script stop execution and do not finish the import.
Probably is faster for that reason...
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 6:37 pm

The only way to obtain the error "failure: empty list name not allowed" is if you do not set list name, is your fault.
@rextended .... Did YOU test your script?

You defined the list name with your global declaration
:global list "blacklist-email"
and that is the name I used ..... so I suggest you test it out and see for yourself.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 6:48 pm

I removed the :log info "$listname $1" I add to do the debug and I removed also a extra line.
You have perfect reason, but please do not say I do not test it before,
at max say I'm ain idiot because I can't do copy & paste. For me is less offensive...
I fix the previopus post.

Also routeros if present a item named like a variable, misunderstand name and variable.
Renamed list to listname.
On router without any other address-lists do not create any problem.
On this way also on already presents address-list do not cause problem.

the ":global listname" inside the function "i" is for declare global variable inside the function, but not change his value.
#############
:global i do={
    :global listname
#    :log info "$listname $1"
    /ip firewall address-list
#############
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 7:11 pm

I removed the :log info "$listname $1" I add to do the debug and I removed also a extra line.
You have perfect reason, but please do not say I do not test it before,
at max say I'm ain idiot because I can't do copy & paste. For me is less offensive...
No offense meant on my part and I apologize if I hurt your feelings.

IMO, you are a very smart techie and I for 1 appreciate your contributions . Your integrity checks are outstanding but insofar as RoS is concerned and the current hardware IMO the performance penalty is significant .... perhaps when Tik Hardware performance improves in the future the kind of integrity checks you have provided will be far more performance efficient. In any case I TIP my Hat to YOU !!!

If you are wondering on the performance hit in processing 50 thousand IP addresses ... using your code took 22 minutes while msatter's code took 2 minutes and this is on a CCR1009 ...
Last edited by mozerd on Sat Aug 14, 2021 7:48 pm, edited 1 time in total.
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: Large blacklists for firewall

Sat Aug 14, 2021 7:35 pm

How can @mozerd not spam his signature on a blacklist topic?
He can't.
i find your picking on @rextendeds coding here hilarious. Considering this is in your history: viewtopic.php?f=3&t=173628&p=849231#p849491
You never answered, did you pay for your mistake? Did you admit to your client that you ordered the wrong switch (and that he has to pay more for the right one)? Not that i'd care, just curious about your work quality too, as you are curios about rextendeds.
@rextended spams a bit too much. post hunts almost every topic, considers himself some MikroTik Diety and considers the rest of us just some plebs, if he'd stick to only coding (because that part he does right) and used a different tone in his replies maybe we'd like him more, but his wording in most of the replies is poor. There are other users around here that made a lot of great scripts over the years, even hosting them on github and don't have this attitude on the forums.
Anyway, cheers.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 8:17 pm

Znevna, your input is not welcome.
In fact Mozerd was kind enough to report his findings, its called going the extra mile and actually testing the scripts that were offered.
Its also called communicating and passing on knowledge. Perhaps some assumptions were not correct but he did find some errors,
that rextended then noticed and fixed. Its all good!
They are both okay with it and so am I.
I will add my two cents, I admire both Mozerd and rextended for their expertise and as folks who take the time to help others and do the best they can as volunteers.
Your negative input is not helpful!
Truth be told, no one gives two craps how you feel.
Have a good day on tick tock or reddit or wherever touchy feely people hang out!!

As for me, back on topic, one has to weigh the IP checking vice skipping and the resultant time lag to process 22 seconds or 2 seconds.
If this is being run once a day, who gives a flying PH, if it happens every few minutes then that is a different story, or if the processing prevents other work from getting done.
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: Large blacklists for firewall

Sat Aug 14, 2021 8:32 pm

yes yes anav, we know you keep pushing people to use his services, we know. chill.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Large blacklists for firewall

Sat Aug 14, 2021 10:26 pm

yes yes anav, we know you keep pushing people to use his services, we know. chill.
Any service that appears to be of good value I will always recommend, its called communicating. There is nothing in it for me.
Whatever personal issues you may have, get over it.
 
User avatar
jvanhambelgium
Forum Veteran
Forum Veteran
Posts: 985
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Large blacklists for firewall

Sat Aug 14, 2021 11:17 pm

If you are wondering on the performance hit in processing 50 thousand IP addresses ... using your code took 22 minutes while msatter's code took 2 minutes and this is on a CCR1009 ...
[/quote]
Yeah, stringent validation probably takes its toll here. So how does it work on a Tilera CPU ? 1 core being occupied 100% fully for 2 minutes (script without any validation) vs 22 minutes (stringent validation)
Would like to test if it starts cooking my ARM CPU on a RB3011.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Sun Aug 15, 2021 5:48 am

@mozerd

On the 2 minutes you are unprotected, like the flaw on the shiled from star trek... :mrgreen:

I wonder why you always have to import the 50,000 addresses by hand all the time.

With many tools you can compare actual and new list, and sent only new items and deleting what can be removed.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2855
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Large blacklists for firewall

Sun Aug 15, 2021 11:06 am

As far as I understand your pros&cons:

A. You are unproteced only if this list is THE ONLY PROTECTION and it gets removed?
B. The problem is that import causes high utilization which leds to lower responsivity (chocking) of a router.

Am I right?

IMHO the solution is to mark as a comment imported data with timestamp and store it in a global variable.
Then you import the new list which is appended to the old one and you get even "overprotected for a moment.
The dumber import is the faster and "better" it is.
Next step is clearing the old list using the old timestamp, store the new timestamp in a variable and then start list lifecycle again.

BTW .. you routers should be protected by other rules so there is only small chance to be fully "unproteced".
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Large blacklists for firewall

Sun Aug 15, 2021 1:27 pm

I wonder why you always have to import the 50,000 addresses by hand all the time.
@rextended -- thanks for asking :D
For my MOAB Clients the actual number of IP Address in a blacklist is over 600 million broken down into IPSETS -- so the number of actual IPSET entries can vary between 30 - 60 thousand -- this is dynamic in nature as my system checks 3 times -- every 8 hours. I do the validly/integrity checks on my server using Perl Tools which make the job lightning fast -- on the server the whole process takes 3 minutes. The Process is as follows: Gets the Lists - yes there are many lists , combine the lists, remove the commentary, remove any duplicates, check the validly/integrity of the addresses then produce the IPSET and sort ascending then produce final for distribution. MOAB produces 2 types of blacklist ... one for memory constrained devices and the other for devices that are well equipped. For memory constrained devices [like the Hex or hAPac2] the ipset has between 6-18 thousand entries.

On Tik Routers the process of deleting the expired [dropped] ones then adding new entries takes a very long time === so its just MUCH faster to delete the blacklist and import the fresh one. :) So the Fresh one when imported into a Tik Router must be done QUICKLY .... very quickly ... in the case of MOAB ipsets generally takes about 1.5 minutes for the process -- deleting the blacklist followed by adding the ipset.

Blacklist is just one form of protection that is added to your Firewall -- the Firewall is your FIRST form of protection if done properly then the blacklist becomes very useful as an added form of protection.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Sun Aug 15, 2021 2:11 pm

@Mozerd I read that you where also providing protection for VOIP.

From the site: https://itexpertoncall.com/promotional/moab.html#prime:
Engineered for MikroTik RouterOS Firewall.

voipTIK - "Wow ...its working ...a perfect voip shield" a direct quote from one of our clients

Is your public facing PBX's getting a lot of SIP/VoIP attacks?

If your answer is YES then you need to consider our voipTIK Blacklist service - a perfect voip shield to protect your business and PBX's against VoIP Fraud and minimize the risks of attacks on your Telephony Server.
Is not so that most VOIP based system, are using a provider who sit in between you and the party calling? This way, you could just allow the needed IP addresses (of your provider) in and exclude all other that seek a connection.
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Large blacklists for firewall

Sun Aug 15, 2021 2:31 pm

@Mozerd I read that you where also providing protection for VOIP.
.......
Is not so that most VOIP based system, are using a provider who sit in between you and the party calling? This way, you could just allow the needed IP addresses (of your provider) in and exclude all other that seek a connection.
My voipTIK blacklist service is not made for end users ... this blacklist is made for VoIP Service Providers.

So yes you are correct in your comment insofar as the end user is concerned.

The backstory to my voipTIK blacklist: One of my MOAB clients who happens to be a major VoIP service provider in their country requested if I could produce a voip blacklist for them. Once I produced that blacklist they were very pleased with the results as all attack on their SIP servers stopped. They could have very easily produced their own blacklist system but did not want the cost of maintaining ... the very same is true for MOAB. They estimated the cost of maintaining it was greater than the cost of the subscription which is why I get the business.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Large blacklists for firewall

Sun Aug 15, 2021 5:13 pm

@mozerd, I understand what you wrote, and thanks for taking your time to reply.

Just one idea, ignoring device with less memory:
1) import blacklist with another name everytime, like moab-202108151614 [no on-error find or delete or duplicate check required]
2) swap name used on rules, search where list ~"moab-[0-9]{12}" and replace with moab-20210815 [no on-error find or delete or duplicate check required]
3) delete old list where list ~"moab-[0-9]{12}" and list != "moab-202108151614" [no on-error required, and the protection is continuosly present]
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: Large blacklists for firewall

Sun Aug 15, 2021 5:48 pm

@rextended, a very good idea :) thx.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Large blacklists for firewall

Sun Aug 15, 2021 7:18 pm

yes yes anav, we know you keep pushing people to use his services, we know. chill.
Any service that appears to be of good value I will always recommend, its called communicating. There is nothing in it for me.
Whatever personal issues you may have, get over it.
I have to admit that I can only admire that you protect rextended that much.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Large blacklists for firewall

Sun Aug 15, 2021 8:06 pm

@rextended, a very good idea :) thx.
It will be interesting to see how long it takes and the load etc........
 
roxanaschram
just joined
Posts: 10
Joined: Sat Oct 10, 2020 7:59 am

Re: Large blacklists for firewall

Thu Sep 02, 2021 12:19 am

Hello everyone,

Does anyone have any experience with large block lists?

I am running an email server and get hit with brute force password attacks from IPs that are commonly found in blacklists.

Although the server features and is set up for automatic lockout of IPs that do multiple attempts at passwords, i would like to move this to firewall.

Does anyone have any experience in what kind of a hit on performance a 11k line blacklist makes to the router? The router is a RB3011.

regards


We have around 85K in our router's address list. It is a CCR1009. I'm not sure how it is in comparison for performance. Our address list grows and shrinks a little every day. We have 3 lists in the routers here because there are servers behind them. I created a script that takes the ASN of every data center I've collected over the last 2 years and grabs all the IPv4 blocks assigned to them and then creates a script that's about 8MB in size that can be imported into a Mikrotik router. We block all of those IP's on port 80,443. Then we use the firehol_3 level file which updates daily. Then we have a whitelist so that services like Stripe can still reach the servers inside even if their IP falls within an IP block associated to the datacenter ASN we have blocked. Our routers only get to about 10% usage under SUPER heavy loads.

Who is online

Users browsing this forum: DanMos79 and 62 guests