Backstory
For years now I have been creating address lists and using them to block bad actors (bots, spammers, etc) on my firewalls.
What I essentailly do is:
The problem
Thing is that as we’re moving forward my lists are getting fairly large, and import times are now close to 10 minutes on overclocked CCR1072s.
There have been numerous threads on the topic of efficiently loading large address lists, but there is no better solution than the one I stated above.
Furthermore, for the time that we’re waiting for the lists to delete and reload, we essentially have a small security hole in that time.
What I propose
Extend the CLI for /ip firewall address-list :
Add an “empty” command, so we can empty an address list without addinf the overhead of a “find”
Add a “set from file” command so we can directly add from a txt address list
Add an “append from file”
Would be nice if the devs gave this a thought, it would make many of our lives much easier!
You didnt answer my question so I can only follow up with sarcasm as a decent question was ignored.
what makes you think your so special? 300 dropped connections, does that make you a big hero?
do you have a clue of how many bots are actively spamming the internet per second???
I am not saying you do not need protection, but you havent indicated why ( a real reason ), like hosting a game server or something.
If you were a real business, you would have
a. a business class ISP account where much of the proper protections from DDOS are rightly associated upstream to an actual capable entity.
b. have an edge router that is far more expensive at the internet and probably with more paid services for such things…
c. have stuff hosted at a virtual data centre where they take care of all that stuff…
This thread is clearly a feature request that would make it much more efficient to import large address lists.
It is not something personal or something that only I need, as I stated it’s an issue that’s been brought up time and again - it’s only a forum/Google search away to prove that.
I’m not sure where the personal attack is coming from, plus what you’ve suggested up until now is completely and utterly off-topic.
It is not about what list is best (MOAB, FireHOL lists, Abseipdb, Cleantalk, etc), it is not about if my upstream networks are neutral or if they do filtering, if I use a DDoS scrubbing centers or not, how many firewalls I have using ECMP to handle the load, what I do specifically et cetera. We are not here to make assumptions and judge if someone needs something just because we don’t believe they do.
If you do not have something constructive to add, please refrain from posting on this thread altogether.
Closing, I call upon on a mod to clear this thread up!
It’s a forum, users forum, everyone can write about that what think, on constructive way, the opposite of your way…
It’s like you must clear your mind, about authority…
It is clear that you do not know how a forum works, this is not a MikroTik help center.
If you want to make inquiries contact support@mikrotik.com directly,
otherwise expect comments from the forum, on this users forum.
And note rextended didnt say
"otherwise expect ‘expert’ comments from the forum, on this users forum.
I dont profess to be an expert but so far youve done a piss poor job of stating the why you need all these fancy configurations…
Therefore, the take away is that why should I support your call for MT to spend resources on an unknown, whilst there are far better things they should be putting their limited resources towards.
Convince me that its a good idea… is all that I am asking, because I dont understand the need.
Better to do, on speed point of view,
mark current elements with a prefix on comment (change comment is more fast than delete the entry)
try to add, one by one, new items, on fail (on-error) remove prefix on comment (is already present)
delete only items where prefix is present (are not present on new list)
on this way the list is active and the protection is never interrupted
…but you want do things on correct way, use another program to generate the script to remove and add only the required addresses
(I will find more if I spend more time looking, and it’s the same way MOAB works too)
All these scripts, as stated in the first post, work by first clearing a given address list, and then recreating them anew using the new updated list (let’s call it definitions).
At this time, this is very resource intensive as the method is rather suboptimal and has considerable computational overhead.
In the case of a 200k list, it takes approximately 3 minutes to clear on a CCR1072 at 1.2GHz, and about 4 minutes to re-import a fresh one, giving a total of 7 minutes run time. I have tested this on a CCR2004, which obviously is faster, takes about half the time, and also “spams” our Log file with removal/addition actions.
At this point in time this isn’t really a problem for me, but it can become a problem in general for anyone using this feature. Furthermore, if we just “split” the list into multiplies and have multiple address lists to lighten the load will have an effect of increasing the computational complexity of firewall processing, since we will introduce a new B-tree.
Being able to directly import an ipset (pure list of IPs delimited by a newline character) by having a ready, much more optimized function closer to whatever mechanism RouterOS uses (iptables afaik) would make it much, much better and enable us to import even larger lists.
I will give you a hint: swap lists, by delete current and rename the new list to take it’s place.
In this forum the handling of big lists are discussed by many many people, for years. All the nooks and crannies have been looked into and being explored.
Reading files up to 63KB is possible and if you split your list up in parts then you can read those numbers of files in using a loop.
Append form file is the default way of importing in ROS.
The 63KB limitation only exists when loading a file as a variable/or iterating through it as I remember right.
What most of us do nowadays is just create a large rsc file and pretty much do close to what you’re saying.
I am already doing that, I have a script that takes in lists like the one you mentioned or ipsets from FireHOL and create rsc files with Mikrotik commands. Then with a scheduler download and run.
Again, this is not a support request -- what you guys propose is fine and works flawlessly currently.
The thing is doing it faster and more efficiently!
