Community discussions

MikroTik App
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 6:28 pm

Hello all,

Backstory
For years now I have been creating address lists and using them to block bad actors (bots, spammers, etc) on my firewalls.
What I essentailly do is:
remove [/ip firewall address-list find list=my_address_list]
add list=my_address_list address=<an ip>
add list=my_address_list address=<another ip>
...

The problem
Thing is that as we're moving forward my lists are getting fairly large, and import times are now close to 10 minutes on overclocked CCR1072s.
There have been numerous threads on the topic of efficiently loading large address lists, but there is no better solution than the one I stated above.
Furthermore, for the time that we're waiting for the lists to delete and reload, we essentially have a small security hole in that time.

What I propose
Extend the CLI for /ip firewall address-list :
- Add an "empty" command, so we can empty an address list without addinf the overhead of a "find"
- Add a "set from file" command so we can directly add from a txt address list
- Add an "append from file"


Would be nice if the devs gave this a thought, it would make many of our lives much easier! :-)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18961
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 6:39 pm

end of input chain
action=drop chain=input

end of forward chain
action=drop chain=forward

what makes your public IP so interesting to spammers/hackers/bots ??
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

For a cheap service that does what you think you are trying to do.................. way better.
https://itexpertoncall.com/promotional/moab.html
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 6:49 pm

I'm not very certain you understand what my proposal aims to resolve.

It's not something "I think I am trying to do", I already am doing what I need for my environment (I'm dropping 300 new connections per sec atm).

Implementation of my proposal will make loading of lists much more quick and efficient.
 
DigiMasTer
just joined
Posts: 3
Joined: Fri Jan 04, 2019 8:52 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 6:57 pm

i've been waiting years for a feature like this, i hope they consider it
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18961
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 7:09 pm

You didnt answer my question so I can only follow up with sarcasm as a decent question was ignored.

what makes you think your so special? 300 dropped connections, does that make you a big hero?
do you have a clue of how many bots are actively spamming the internet per second???

I am not saying you do not need protection, but you havent indicated why ( a real reason ), like hosting a game server or something.
If you were a real business, you would have
a. a business class ISP account where much of the proper protections from DDOS are rightly associated upstream to an actual capable entity.
b. have an edge router that is far more expensive at the internet and probably with more paid services for such things.......
c. have stuff hosted at a virtual data centre where they take care of all that stuff.......

Call me not convinced...........
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 7:23 pm

This thread is clearly a feature request that would make it much more efficient to import large address lists.
It is not something personal or something that only *I* need, as I stated it's an issue that's been brought up time and again - it's only a forum/Google search away to prove that.

I'm not sure where the personal attack is coming from, plus what you've suggested up until now is completely and utterly off-topic.
It is not about what list is best (MOAB, FireHOL lists, Abseipdb, Cleantalk, etc), it is not about if my upstream networks are neutral or if they do filtering, if I use a DDoS scrubbing centers or not, how many firewalls I have using ECMP to handle the load, what I do specifically et cetera. We are not here to make assumptions and judge if someone needs something just because we don't believe they do.
If you do not have something constructive to add, please refrain from posting on this thread altogether.
Closing, I call upon on a mod to clear this thread up!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 7:46 pm

...what you've suggested up until now is completely and utterly off-topic...
It's your point of view
We are not here to make assumptions and judge if someone needs something just because we don't believe they do.
It's a forum, users forum, everyone can write about that what think, on constructive way, the opposite of your way...
Closing, I call upon on a mod to clear this thread up!
It's like you must clear your mind, about authority...


It is clear that you do not know how a forum works, this is not a MikroTik help center.
If you want to make inquiries contact support@mikrotik.com directly,
otherwise expect comments from the forum, on this users forum.
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 7:55 pm

This is not a support request.

This is a feature request. I am giving an example what I use right now (and what many other users use), and how it could be better! :-)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18961
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 7:55 pm

And note rextended didnt say
"otherwise expect 'expert' comments from the forum, on this users forum.

I dont profess to be an expert but so far youve done a piss poor job of stating the why you need all these fancy configurations...........
Therefore, the take away is that why should I support your call for MT to spend resources on an unknown, whilst there are far better things they should be putting their limited resources towards.
Convince me that its a good idea........ is all that I am asking, because I dont understand the need.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:02 pm

remove [/ip firewall address-list find list=my_address_list]
add list=my_address_list address=<an ip>
add list=my_address_list address=<another ip>
Better to do, on speed point of view,
mark current elements with a prefix on comment (change comment is more fast than delete the entry)
try to add, one by one, new items, on fail (on-error) remove prefix on comment (is already present)
delete only items where prefix is present (are not present on new list)
on this way the list is active and the protection is never interrupted

...but you want do things on correct way, use another program to generate the script to remove and add only the required addresses
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:10 pm

Alright, let's take a look at it.

I took the time to do a small Google search of how people implement their lists, it's pretty much the same way I do it currently, and it works.
- https://github.com/kd7lxl/blacklist-service
- https://github.com/Samhamsam/blocklist_mikrotik
- https://github.com/pwlgrzs/Mikrotik-Blacklist
- https://github.com/multiduplikator/mikrotik_blocklist
- https://github.com/mihaiv/mikrotik-bloc ... s/mikrotik

(I will find more if I spend more time looking, and it's the same way MOAB works too)

All these scripts, as stated in the first post, work by first clearing a given address list, and then recreating them anew using the new updated list (let's call it definitions).
At this time, this is very resource intensive as the method is rather suboptimal and has considerable computational overhead.
In the case of a 200k list, it takes approximately 3 minutes to clear on a CCR1072 at 1.2GHz, and about 4 minutes to re-import a fresh one, giving a total of 7 minutes run time. I have tested this on a CCR2004, which obviously is faster, takes about half the time, and also "spams" our Log file with removal/addition actions.

At this point in time this isn't really a problem for me, but it can become a problem in general for anyone using this feature. Furthermore, if we just "split" the list into multiplies and have multiple address lists to lighten the load will have an effect of increasing the computational complexity of firewall processing, since we will introduce a new B-tree.

Being able to directly import an ipset (pure list of IPs delimited by a newline character) by having a ready, much more optimized function closer to whatever mechanism RouterOS uses (iptables afaik) would make it much, much better and enable us to import even larger lists.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:11 pm

I will give you a hint: swap lists, by delete current and rename the new list to take it's place.

In this forum the handling of big lists are discussed by many many people, for years. All the nooks and crannies have been looked into and being explored.

Reading files up to 63KB is possible and if you split your list up in parts then you can read those numbers of files in using a loop.

Append form file is the default way of importing in ROS.
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:11 pm

Better to do, on speed point of view,
mark current elements with a prefix on comment (change comment is more fast than delete the entry)
try to add, one by one, new items, on fail (on-error) remove prefix on comment (is already present)
delete only items where prefix is present (are not present on new list)
on this way the list is active and the protection is never interrupted

...but you want do things on correct way, use another program to generate the script to remove and add only the required addresses
That is indeed something I am considering implementing, as a solution for now and it will be quick.

What do you think though for a fully "clean" way as I'm saying?
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:15 pm

I will give you a hint: swap lists, by delete current and rename the new list to take it's place.

In this forum the handling of big lists are discussed by many many people, for years. All the nooks and crannies have been looked into and being explored.

Reading files up to 63KB is possible and if you split your list up in parts then you can read those numbers of files in using a loop.

Append form file is the default way of importing in ROS.
The 63KB limitation only exists when loading a file as a variable/or iterating through it as I remember right.
What most of us do nowadays is just create a large rsc file and pretty much do close to what you're saying. :D
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:17 pm

What do you think though for a fully "clean" way as I'm saying?
is the worst way,

@msatter has wroted many post about that and implemented a fast way for upload files.
Also with my method for read files bigger than 63KB
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:19 pm

The 63KB limitation only exists when loading a file as a variable/or iterating through it as I remember right.
What most of us do nowadays is just create a large rsc file and pretty much do close to what you're saying. :D
Do not mix read directly .rsc file and use some sources to generate mikrotik commands, like SPAMHAUS Drop list...
https://www.spamhaus.org/drop/drop.txt
 
stathisch
just joined
Topic Author
Posts: 16
Joined: Wed Oct 26, 2016 12:29 am

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:27 pm

The 63KB limitation only exists when loading a file as a variable/or iterating through it as I remember right.
What most of us do nowadays is just create a large rsc file and pretty much do close to what you're saying. :D
Do not mix read directly .rsc file and use some sources to generate mikrotik commands, like SPAMHAUS Drop list...
https://www.spamhaus.org/drop/drop.txt
I am already doing that, I have a script that takes in lists like the one you mentioned or ipsets from FireHOL and create rsc files with Mikrotik commands. Then with a scheduler download and run.

Again, this is not a support request -- what you guys propose is fine and works flawlessly currently.
The thing is doing it faster and more efficiently!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 8:28 pm

Pray and hope...

We lost "hope" on many things on routeros (like RegEx on :find for cite another...)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18961
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 9:03 pm

Back of the line, I want firewall address lists available in routing rules, could care less about useless blocking crap. ;-)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Feature Request: address-list empty , set-from-file , etc

Mon Mar 28, 2022 11:13 pm

Like block directly it on BGP of IP Transit provider...
No firewall and "no routing"...

Who is online

Users browsing this forum: No registered users and 20 guests