Is there any way to use some kind of “dynamic” ip address lookup within the firewall (with eg DNSBL check) in the instead of using the build-in static address lists?
The objective is to move out all the static address lists to a server since they’ve grown to big for RoS.
Background to my question is that we’re using ip backlists from Squid/FireHOL and the import administration is becoming rather tedious and shaky because RoS (6.42 long-term) doesn’t seem to cope well with volumes like +50K lines on our CCRs. As an example we quite often get really weird results from the address-list tab in Winbox 3.18 and intermittent sluggish respons time even though the CPU is almost idling (also using the terminal)
There is also a security risk since there are no smart ways (that doesn’t take forever) to update the address list online without wipe and reload.
Hi
I developed MOAB extracted from FireHOL] and have a fairly large number of users where none so far have complained of any intermittent sluggish response times …
Insofar as ip address lookup within the firewall (with eg DNSBL check) — IMO that would impose a significant performance hit plus setting something like that up locally requires significant time and resources adding another point of failure.
Well, IMO some few ms really doesn’t really matter during the initial connection setup compared to the regular “internet” latency. The latency of our internal DNS servers for a single RR is about 30 micro seconds but I don’t mind using the RoS build in DNS for this purpose.
What other options are there? It doesn’t have to be a DSNBL check but any kind of “dynamic” lookup will do.
So, IYO what is the “working” (not technical) limit of no lines in the address lists? According to our experience problems are arising when reaching +40K rows and above…
Any thoughts about a secure way to update the address list online that doesn’t take forever ? The wipe and clean method is highly insecure because the lengthy import times that is direct related to the huge import volumes.
There aint new fish in the pond, its fetch and script remove and add, and until known limitations are removed not sure more can be done. Mozerd has eked out performance optimums with his setup is my understanding.
For a few pennies (or cups of coffee a month) the MOAB is excellent value for most of us who dont script and fetch or wish to play such games and a whole lot more maintenance…
@anav: I have absolutely no opinion about MOAB since I really don’t know anything about it and furthermore it’s not the subject of this discussion. With respect, please keep focus to my original question regarding how to manage problems related to huge address lists in RoS, etc. Many thanks in advance!
So, any thought or ideas regarding my previous questions related to RoS?
None, sorry. I dont have time to play silly games with lists. I wasted lots of time looking at various lists and attempting smallish items and realized I was only fooling myself if I thought I was actually doing something productive LOL. Good luck though!
Untill Tik removes the limitation I mentioned earlier the only way is with .rsc The other issue is dupes that generate TIK errors if imported … so I get all the data from FireHOL,then with Perl I extract the dupes, order the addresses in accending numerical order then create the RSC ready for import. – I do that 3 times each day over 24hour timeframe – FireHOL has a lot of dups, averaging between 5k and 10K based on the lists I use for MOAB
The lists I focus on are Level1, Level2, Level3, webclient, webserver and coinbl. Importing my lists for the CCR takes close to 45 seconds, and on devices like the hEX or the hAPac2 – a smaller list – takes close to 2 minutes. On CHR, its lightning fast – ask @Chupaka
If you are a Ubiquiti user [ER-X, ER-L etc, you have all the tools built in to do the whole sheband inside of 30 sconds] – get the lists, extract the addresses, order the addresses, extract the dupes, and populate — very nice and THAT is what i would like to have for MikroTik and for my clients — done this way I would not charge 1 cent . and the I would not have to pay my webhosts etc.
Yeah, hope they plan to do something about it or create a more manageable/flexible solution in the future.
How many lines (give or take) is the end result? I’m wondering since I’m interested to use the Spamhaus drop/edrop “real-time” lists. Will the total aggregate work on a CCR do you think?
Absolutely brilliant, altering the block-rules is of course the fastest and most secure way to do it! Why didn’t I think about it myself! Now I only need to perform some tests to figure out the actual storage limitation on a CCR as I need room for both current and new lists simultaneously…
My CCR list now contains 68,342 lines of ip addresses + 10 other lines of code – remmber a lot of these are CIDR’s or approx 630 million UNIQUE IP addresses.
For memory contrained MikroTik devices like the hEX and the hAPac2 list now contains 16,770 lines of ip addresses + 10 other lines of code
FireHOL Level 1 conatins pretty well 100% of SpamHouse – 36K ip addresses
Reports from my Clients is that MOAB traps close to 40 million hits a week some weeks a lot more. … and so far no issues with performnace of whatsoever nature.
No no, not at all! You did really help me in this case!
Some times the simple solution is the best but I’ve been focusing on a complete different point of view and forgot to think “outside the box”. Sorry if I was vague and overused the emojis but It was actually a genuine thank you!
And yeah, it’s probably time to start evaluate another solution more aimed at the firewall part or IPS/IDS.
and furthermore it’s not the subject of this discussion. With respect, please keep focus to my original question regarding how to manage problems related to huge address lists in RoS, etc. Many thanks in advance!
Why didn’t I think about it myself!