Is there any way to use some kind of "dynamic" ip address lookup within the firewall (with eg DNSBL check) in the instead of using the build-in static address lists?

The objective is to move out all the static address lists to a server since they've grown to big for RoS.

Background to my question is that we're using ip backlists from Squid/FireHOL and the import administration is becoming rather tedious and shaky because RoS (6.42 long-term) doesn't seem to cope well with volumes like +50K lines on our CCRs. As an example we quite often get really weird results from the address-list tab in Winbox 3.18 and intermittent sluggish respons time even though the CPU is almost idling (also using the terminal)

There is also a security risk since there are no smart ways (that doesn't take forever) to update the address list online without wipe and reload.

Any thoughts and ideas are welcome!

Is there any way to use some kind of "dynamic" ip address lookup within the firewall (with eg DNSBL check) in the instead of using the build-in static address lists?

The objective is to move out all the static address lists to a server since they've grown to big for RoS.

Background to my question is that we're using ip backlists from Squid/FireHOL and the import administration is becoming rather tedious and shaky because RoS (6.42 long-term) doesn't seem to cope well with volumes like +50K lines on our CCRs. As an example we quite often get really weird results from the address-list tab in Winbox 3.18 and intermittent sluggish respons time even though the CPU is almost idling (also using the terminal)

There is also a security risk since there are no smart ways (that doesn't take forever) to update the address list online without wipe and reload.

Any thoughts and ideas are welcome!

Hi
I developed MOAB extracted from FireHOL] and have a fairly large number of users where none so far have complained of any intermittent sluggish response times ...

Insofar as ip address lookup within the firewall (with eg DNSBL check) --- IMO that would impose a significant performance hit plus setting something like that up locally requires significant time and resources adding another point of failure.

IMO, MikroTik should address viewtopic.php?t=138652 ... by removing the current limitation making it far more efficient and secure.

Insofar as ip address lookup within the firewall (with eg DNSBL check) --- IMO that would impose a significant performance hit plus setting something like that up locally requires significant time and resources adding another point of failure.

Well, IMO some few ms really doesn't really matter during the initial connection setup compared to the regular "internet" latency. The latency of our internal DNS servers for a single RR is about 30 micro seconds but I don't mind using the RoS build in DNS for this purpose.

What other options are there? It doesn't have to be a DSNBL check but any kind of "dynamic" lookup will do.

So, IYO what is the "working" (not technical) limit of no lines in the address lists? According to our experience problems are arising when reaching +40K rows and above...

Any thoughts about a secure way to update the address list online that doesn't take forever ? The wipe and clean method is highly insecure because the lengthy import times that is direct related to the huge import volumes.

Btw, what lists are you using from FireHOL (iplists.firehol.org) ?

There aint new fish in the pond, its fetch and script remove and add, and until known limitations are removed not sure more can be done. Mozerd has eked out performance optimums with his setup is my understanding.
For a few pennies (or cups of coffee a month) the MOAB is excellent value for most of us who dont script and fetch or wish to play such games and a whole lot more maintenance......

@anav: I have absolutely no opinion about MOAB since I really don't know anything about it and furthermore it's not the subject of this discussion. With respect, please keep focus to my original question regarding how to manage problems related to huge address lists in RoS, etc. Many thanks in advance!

So, any thought or ideas regarding my previous questions related to RoS?

None, sorry. I dont have time to play silly games with lists. I wasted lots of time looking at various lists and attempting smallish items and realized I was only fooling myself if I thought I was actually doing something productive LOL. Good luck though!

Thanks! Regarding MT blocking capabilities, I take it you've been there, done that and found the bitter dead end!

Besides MT, did you solve it any other way?

I remeber seeing somewhere that addresslist can be feed an dns and it will do resolution on it's own (basically keeping itself updated)

Documented (a big word for just small syntax note) in the meantime: https://wiki.mikrotik.com/wiki/Manual:I ... dress_list

Any thoughts about a secure way to update the address list online that doesn't take forever ? The wipe and clean method is highly insecure because the lengthy import times that is direct related to the huge import volumes.

Btw, what lists are you using from FireHOL (iplists.firehol.org) ?

why not:
1. load new one into new list
2. update rule to use new
3. remove old

Any thoughts about a secure way to update the address list online that doesn't take forever ? The wipe and clean method is highly insecure because the lengthy import times that is direct related to the huge import volumes.

Btw, what lists are you using from FireHOL (iplists.firehol.org) ?

Untill Tik removes the limitation I mentioned earlier the only way is with .rsc The other issue is dupes that generate TIK errors if imported ... so I get all the data from FireHOL,then with Perl I extract the dupes, order the addresses in accending numerical order then create the RSC ready for import. -- I do that 3 times each day over 24hour timeframe -- FireHOL has a lot of dups, averaging between 5k and 10K based on the lists I use for MOAB

The lists I focus on are Level1, Level2, Level3, webclient, webserver and coinbl. Importing my lists for the CCR takes close to 45 seconds, and on devices like the hEX or the hAPac2 -- a smaller list -- takes close to 2 minutes. On CHR, its lightning fast -- ask @Chupaka

If you are a Ubiquiti user [ER-X, ER-L etc, you have all the tools built in to do the whole sheband inside of 30 sconds] -- get the lists, extract the addresses, order the addresses, extract the dupes, and populate --- very nice and THAT is what i would like to have for MikroTik and for my clients --- done this way I would not charge 1 cent . and the I would not have to pay my webhosts etc.

I remeber seeing somewhere that addresslist can be feed an dns and it will do resolution on it's own (basically keeping itself updated)

Documented (a big word for just small syntax note) in the meantime: https://wiki.mikrotik.com/wiki/Manual:I ... dress_list

Thanks for the tip but unfortunately it only seems to translate DNS Names into address lists without any further dynamic functionality.

Any thoughts about a secure way to update the address list online that doesn't take forever ? The wipe and clean method is highly insecure because the lengthy import times that is direct related to the huge import volumes.

Btw, what lists are you using from FireHOL (iplists.firehol.org) ?

Untill Tik removes the limitation I mentioned earlier the only way is with .rsc The other issue is dupes that generate TIK errors if imported ... so I get all the data from FireHOL,then with Perl I extract the dupes, order the addresses in accending numerical order then create the RSC ready for import. -- I do that 3 times each day over 24hour timeframe -- FireHOL has a lot of dups, averaging between 5k and 10K based on the lists I use for MOAB

The lists I focus on are Level1, Level2, Level3, webclient, webserver and coinbl. Importing my lists for the CCR takes close to 45 seconds, and on devices like the hEX or the hAPac2 -- a smaller list -- takes close to 2 minutes. On CHR, its lightning fast -- ask @Chupaka

Yeah, hope they plan to do something about it or create a more manageable/flexible solution in the future.

How many lines (give or take) is the end result? I'm wondering since I'm interested to use the Spamhaus drop/edrop "real-time" lists. Will the total aggregate work on a CCR do you think?

Any thoughts about a secure way to update the address list online that doesn't take forever ? The wipe and clean method is highly insecure because the lengthy import times that is direct related to the huge import volumes.

Btw, what lists are you using from FireHOL (iplists.firehol.org) ?

why not:
1. load new one into new list
2. update rule to use new
3. remove old

Absolutely brilliant, altering the block-rules is of course the fastest and most secure way to do it! Why didn't I think about it myself! Now I only need to perform some tests to figure out the actual storage limitation on a CCR as I need room for both current and new lists simultaneously...

Anyhow Sebastia, thank you very much!

Great explanation Mozerd! Much thanks. I hope the dude that lives in the little red car is paying attention!!

Absolutely brilliant, altering the block-rules is of course the fastest and most secure way to do it! Why didn't I think about it myself! Now I only need to perform some tests to figure out the actual storage limitation on a CCR as I need room for both current and new lists simultaneously...

Anyhow Sebastia, thank you very much!

I catch a hint of sarcasm?

Then again you're the one using a router to do very specialised firewall functionality... Go figure

Yeah, hope they plan to do something about it or create a more manageable/flexible solution in the future.

How many lines (give or take) is the end result? I'm wondering since I'm interested to use the Spamhaus drop/edrop "real-time" lists. Will the total aggregate work on a CCR do you think?

My CCR list now contains 68,342 lines of ip addresses + 10 other lines of code -- remmber a lot of these are CIDR's or approx 630 million UNIQUE IP addresses.
For memory contrained MikroTik devices like the hEX and the hAPac2 list now contains 16,770 lines of ip addresses + 10 other lines of code
FireHOL Level 1 conatins pretty well 100% of SpamHouse -- 36K ip addresses

Reports from my Clients is that MOAB traps close to 40 million hits a week some weeks a lot more. ..... and so far no issues with performnace of whatsoever nature.

Absolutely brilliant, altering the block-rules is of course the fastest and most secure way to do it! Why didn't I think about it myself! Now I only need to perform some tests to figure out the actual storage limitation on a CCR as I need room for both current and new lists simultaneously...

Anyhow Sebastia, thank you very much!

I catch a hint of sarcasm?

Then again you're the one using a router to do very specialised firewall functionality... Go figure

No no, not at all! You did really help me in this case!

Some times the simple solution is the best but I've been focusing on a complete different point of view and forgot to think "outside the box". Sorry if I was vague and overused the emojis but It was actually a genuine thank you!

And yeah, it's probably time to start evaluate another solution more aimed at the firewall part or IPS/IDS.

Yeah, hope they plan to do something about it or create a more manageable/flexible solution in the future.

How many lines (give or take) is the end result? I'm wondering since I'm interested to use the Spamhaus drop/edrop "real-time" lists. Will the total aggregate work on a CCR do you think?

My CCR list now contains 68,342 lines of ip addresses + 10 other lines of code -- remmber a lot of these are CIDR's or approx 630 million UNIQUE IP addresses.
For memory contrained MikroTik devices like the hEX and the hAPac2 list now contains 16,770 lines of ip addresses + 10 other lines of code
FireHOL Level 1 conatins pretty well 100% of SpamHouse -- 36K ip addresses

Reports from my Clients is that MOAB traps close to 40 million hits a week some weeks a lot more. ..... and so far no issues with performnace of whatsoever nature.

Sounds promising! Regarding Spamhaus, are we talking about the same lists here? I was thinking of "drop" (http://iplists.firehol.org/?ipset=spamhaus_drop) and "e-drop" (http://iplists.firehol.org/?ipset=spamhaus_edrop) ...

Yea, check Overlaps of firehol_level1 with other IP
http://iplists.firehol.org

Thanks for the info and walk of shame for me If one care to read carefully, It's actually stated with clear text regarding firehol_level1

"To accomplish this, we include the following IP lists:
. . .
spamhaus drop and edrop - Don't Route Or Peer IPs
. . .
"

According to firehol Linux ipsets are affected only by the number of different subnets". I suppose this also applies to RoS since it utilize iptables, right??

"If you are going to use this IP list as a blocklist / blacklist at a firewall, its size can be important for the performance of the firewall. Keep in mind that the performance of Linux netfilter / iptables firewalls that use ipsets (like FireHOL does), is not affected by the size of an ipset. Any number of entries can be added and the firewall will just do one lookup for every packet checked against the ipset. Linux ipsets are affected only by the number of different subnets in an ipset. FireHOL solves this by automatically reducing the number of unique subnets on all hash:net ipsets (check this article for more information on how this is done)."

Anyhow, i've done some experimenting using the firehol iprange tool. It does amazing work to optimize dups and concatenate subnets or the opposite to reducing subnets by expanding to ip-address only.

I tested iprange to merge and optimize "concatenated.out" from 198 558 rows to "optimized.out" with 145 124 rows and actually managed to load in about 4min30sec on a CCR and it works as long as you don't use winbox to lists the address list.