A number of users have contacted me via email and requested that I make the prerequisites a little clearer to under stand. I now have done that so please check the link again. and thanks to ALL for the feedback. Updated Prerequisites
Sorry but I can’t help myself not to ask couple of questions:
Can you clear up a little bit how does user/owner of router handle security - i.e. limiting your RSC to not create new users, open ports etc? Downloading 3rd party RSC can cause unpredictable and serious issues as it can completely rule the device.
If it is really just blacklist, you can distribute it as txt/csv list of addresses. Everyone can easily create script to download and implement the list on scheduled basis. That way, every user knows exactly what the script does and there is guarantee that it will not do anything else because it is not capable of anything else.
I can see that you offer for example hAP ac^2 as “capable router firewall appliance”. What performance impact can be expected on such device after you add those 600 million IPs into? are there some test results based on clearly defined scenario which can be replicated by everyone so we can confirm those numbers?
A blocklist for MikroTik should be distributed using DNS address lists.
There are two limitations that limit that method:
when the blocklist contains subnets, there is no efficient method to transfer them.
solution: MikroTik should lookup TXT records besides A records, and when they are valid textual subnet notation, load them.
like: IN TXT 192.168.0.0/24
the number of DNS entries returned is limited too much. I think the limit is in the built-in DNS resolver which has a limit on reply size.
(the actual number of addresses varies depending on the length of the DNS name used to query them)
I hope MikroTik addresses these limitations so it will be easier to manage address lists on many routers.
It is possible, but it is quite impractical because you need another step to transfer the information from the
routing table maintained by BGP to a place where you can actually use it, i.e. an address list.
Maybe another useful feature suggestion: an address-list item that refers to a routing-table name, and that
automatically gets loaded by all addresses that appear in that routing table.
Another problem is that a BGP association is always 2-way so you both need to set the address of the central
server providing the information AND in the central server YOUR IP address has to be configured. That is a
problem when your IP address is not static. This could be overcome by setting up a VPN that allows a dynamic
client address (L2TP/IPsec, SSTP, OpenVPN) but that adds yet another layer of complexity.
your point is very valid. Since I am not running a criminal enterprise - the subscriber to my service will need to have explicit trust that my blacklist scripts will not violate their trust. People who subscribe to my service do not want to create or manage there blacklists.
Currently, I have a small number of users with hAPac2 devices [and hEX], subscribing to MOAB, who - so far - are very pleased with the performance in their environments. The hAPac2 that I install/configure clearly outlines the limitations i.e, supports up to 5 users + all needed peripherals – while the hEX that I install/configure supports 10 users and up to 20 connected devices — I have not done any benchmarks under load – I much prefer that my users report back to me if there are performance issues, Perhaps some will come here and provide their endorsement – most [if not all] are very busy with their lives
MOAB is derived ffrom FireHOL, which I make clear in my advertising – you can check it out at here
[EDIT] I did fail to mention that I do have ONE user located in Northern Europe who supports a large number of CCR’s in the field who is using MOAB [for the last 2 months] all of them supporting several thousand users . I just requested that he come here and post his experiences but – so far – he has declined to do so and he requested that I be vague so I cannot state exactly where he is located. He could very easily run his own blacklist mechanism and is well aware of FireHOL as many others are – he got a significant discount based on his number of routers – he chose to subscribe and so far he seems very pleased – no one in his group is complaining of any performance issue attributed to MOAB.
I was wondering who will come with this idea
Well, this is common issue for all services - to make sure that users will not share the product. In this case, it simply can’t be done. If users can manage the router, then they have access to those rules and they can export it and share it. RSC can’t protect it at all.
For example Snort.org have nice approach with giving some basic list of rules for free and limiting better up-to-date list for subscribed users.
Though, i still dont think it is good idea to simply block so many IP addresses. Chance of false-positive is too high and it will end up similarly to sorbs.net - easy to get in, hard to get out, legit services blocked, nobody to blame…
@mozerd:
Thanks for reply! I really appreciate it.
I have no doubt that your intentions are pure and you don’t plan to hack your customers, however, some man-in-the-middle or even angry employee can feel different about this. We unfortunately don’t live in perfect world and attacks are happening. It would be quite sad to inadvertently help attackers while you are trying to stop them, just because your script had too much access.
I see. If you ever get any benchmark (simple iperf test with {transmitter}–{device under test}—{receiver} layout would be great), let us know. Or - if you want - I am willing to do this and share my findings. I understand you offered free trial for local users. I am not really interested in full-blown subscription or even prolonged trial, but if it helps, I can simply dedicate one of my testing routers and try it for couple of days and then give you the trial licence back. Let me know If this sounds interesting. If yes, I will send the request via email.
Anyway, I wish you and your business all best. Hopefully, you will encounter any security issues
Yes I am interested – please do send in your request based on the MOAB Prerequisite’s – I appreciate your participation.and look forward to the results of your testing.
Of course it has zero functionality. Block some people because they appear to have bad intentions, and as a result block some legitimate users and still allow a lot of people with really bad intentions into your system because they happen to be not (yet) on the list.
However, I am interested in general mechanisms to manage large address lists under RouterOS, hence my additions to the topic.
Hopefully some method will become available that works better than importing a .rsc file. Preferably a DNS based address list
“without” limits (or more reasonable limits).
Great concept!!
Much thanks and have been using it with no issues.
When I first started out on my hex, on my own, I found some available firehol lists… and started reading about spamhouse, dshield, malcode. country lists and other lists.
They would all pump out files to use.
Then I came accross Josh Haven… http://joshaven.com/resources/tricks/mikrotik-automatically-updated-address-list/
Wow, a resource that looked at some major lists (not countries though) and provided them in almost a ready to use format.
In summary, if someone one wants to provide a stable, server based, blacklist for free that is tailored to ones equipment and seems to grab the best of whats available out there, then I and many others would be very grateful. (Normis, seems almost ready to volunteer, seeing as it so easy… )
In the meantime I will continue to use the service here that is so low cost - less than what I pay for coffee at Tim Hortons in a month. Since its not tied to a service offering that could disappear at any time (josh) and one that is more complete, and is supported by someone who is looking after many clients (responsible individual) and is not in the business of increasing their security risk (plus being Canadian lol). I am not worried about such issues. I am more concerned about a gazillion other sites to which I use for transactions and mikrotik for their next security blunder LOL.
I am also investigating another avenue, which purports to access ‘closed’ lists and does layer 7 programming and targets TOR nodes.
You get what you pay for though as it is also not free. Seems very good so far. https://axiomcyber.com/shield/
FYI update – I still have 7 slots open for the Free Trial Period that expires September 30, 2018
If you want to participate in the free trial then PLEASE review the MOAB prerequisites link and send me an email with the information requested. If you have any questions post them here. My email address is found in the first opening post of this thread.
Hi,
I finally had chance to test the service and I must say, that performance impact on hAP ac^2 was negligible.
All tests were done with iperf in ubuntu. I used TCP connection and default window sizes (512k) and always performed 2 tests - one with “-r” param for separate RX/TX testing, second with “-d” param for duplex test where RX and TX was tested at the same time.
Directly connected computers:
TX = 914Mbps
RX = 939Mbps
Duplex TX = 703Mbps
Duplex RX = 677Mbps
In following tests, TX means LAN → NAT → WAN , RX means WAN → NAT → LAN
Defconf without fasttrack:
TX = 574Mbps
RX = 579Mbps
Duplex TX = 388Mbps
Duplex RX = 376Mbps
Defconf with fasttrack:
TX = 923Mbps
RX = 933Mbps
Duplex TX = 777Mbps
Duplex RX = 620Mbps
MOAB without fasttrack:
TX = 523Mbps
RX = 509Mbps
Duplex TX = 264Mbps
Duplex RX = 331Mbps
MOAB with fasttrack:
TX = 928Mbps
RX = 937Mbps
Duplex TX = 786Mbps
Duplex RX = 586Mbps
I am aware that my computers were probably not strong enough to handle full gigabit of iperf traffic. Unfortunately, I couldn’t do better. Anyway, as you can see, hAP ac^2 handles the list really well. Especially when you have fasttrack enabled, there is literary no difference between defconf and MOAB. Without fasttrack MOAB cause approximately 50Mbit of speed reduction against defconf. This will obviously be noticeable only if your router is already bottleneck and your connection is faster than your router can handle.
Couple of other things I noticed:
As you can read from MOAB prerequisite page, you are supposed to manually add two “drop” rules - one in Raw table, second in Filter table
Raw drop rule uses list of approximately 11 thousand entries
Filter drop rule uses list of approximately 6 thousand entries
Drop rules are based on interface, instead of interface-list. However, “bogon exclusion list” rule is based on interface-list=WAN. I believe it would be better to use same approach for all rules.
Downloads are protected by HTTP-Auth, so your initial setting script contain username and password to access the data
As I was worried earlier, the list is really distributed as RSC full of commands to add entries. This might be more optimized by distributing simple text file and parsing it directly in router. It will make downloaded file smaller and also remove possible risk from downloading malicious script
There is some attempt to minimize downloading by firstly fetching smaller TXT files which either have some content or is empty. However, as there are no parameters submitted while downloading these “diffs” files, it simply cannot truly represent difference between already applied settings in router and current list on the server. What “diff” it really represents is pure mystery to me
if I manually run the downloader script again and again, my lists were downloading again and again (but they should not as I already had newest version applied)
I would expect the diff file to be dynamically generated based on last version downloaded by the specified username. That would obviously require some back-end with database to store info, which version was downloaded by each user last time
Finally, I would like to thank Mozerd for providing free trial so I was able to do the test.