Is there a way in a mikrotik to load a list of websites (i.e. from urlblacklist.com) to provide for content filtering? I read about something to do with the HTTP proxy that could accomplish this, but I am not sure about the syntax.
The same goes for writing a script to block specific websites without having to type them all in. I’m sure it’s possible, but I am unaware of the syntax.
Thank you in advance for your help.
Yes, you can do this with proxy access list. Use any texteditor (Excel or Calc are more preffered), put the list of sites to one column, other column should contain correct syntax of the RouterOS (e.g. ip proxy access add action=deny dst-host=).
Create file and paste it to the console or save it as *.rsc and upload to the router FTP, use import *.rsc to load the configuration.
I assume the entry in the text editor will read as such:
ip proxy access add action=deny dst-host=
is this correct?
-NSG
When importing the .rsc file (which was originally saved as .txt and I only changed the file extension) what is the specific command? The file is named domains.rsc and when I try the command
/import domains.rsc
or
import domains.rsc
I get a system crashed error.
Where in here is my error?
This is the specific error message if it helps.
Console has crashed or maybe system is completely buys; please log in again
Which version do you use, when file import does not work (How many rules do you have at the file) ?
Yes, you are correct about the syntax.
The MT in question has 2.9.44 installed.
There are 527,171 rules on it. Do I need to chop that into pieces? If so, how small a batch of pieces?
I have chopped off the first 50,000 rules. I am attempting to import it in that size batch. So far, I entered
import domains1.rsc
at the Terminal Prompt. This hasn’t crashed with an error, but 2 hours later it is still hanging in the terminal screen.
A new terminal doesn’t give me a command line.
Is this good or bad?
Is there a limit to the number of rules that can be imported through an .rsc file?
how many is too many?
i would think 500,000 rules is too many to really be usable.
OT:
nathanscottgrey, if there’s no answer under your post, please use edit function instead of adding new posts.
Thank You 
Well, if your number is reached 500,000. Probably you have to think about reverse approach, allow few sites and block all of them that are included at 500,000.
There should be a limit, but it should not influence common tasks.
This is about content filtering for Public Wi-Fi access. I can’t only allow a few sites. The amount I am trying to block is far less than the amount I am trying to leave open access to.
Well, 527,171 all of them are different and do not have any common on their syntax ?
There is no commonality to the syntax beyond the vast majority being .com sites. Is the issue that the Mikrotik just can’t handle that many rules (despite there being enough storage space to do so)? Or, do I need to make the .rsc files smaller and import it in minimal chunks?
If I can get it done then I can save it as a backup and just restore the backup on future configurations.
half a million rules for each http connection to processes seems a little excessive…
I’m not up to date on the latest and greatest content filtering techniques… But is there some type of DNS based CBL (composite blocking list) much like the RBL/CBL’s used for blocking known email spammers for content filtering? That to me sounds like a more feasible/efficient solution then adding 500,000+ rules to a web-proxy.
I’ve seen DNS servers that are able to suggest alternative URL’s for mistyped URL’s.. You’d think you could modify a DNS server to only service requests to non offensive websites using a CBL. Any offensive content could be redirected to a website saying its blocked due to the terms of service etc etc.. Have your DHCP server hand out that DNS server, and block all dns requests not going to the content filtered DNS.
I’d think something like I suggested would be available, if not I better get a copyright cause it sounds like a good idea!
This may not be the answer you’re looking for, but I just don’t see how effective a static rule set will be against offensive material on the internet that grows and changes daily..
Ok…
I have been sitting on this idea, not wanting to muddy the watters, but…
How about this…
Use OpenDNS or similar.. This will block the porn etc by name.. (using proxy with dns pointing to OpenDNS)
A handy option would be an option to evaluate entries to see if an IP address is being requested.. IE
http://10.0.0.234/XXXX.aspx
Intercept the IP Address portion, reverse DNS the address and check the name… (allow / disallow failed lookups to pass as an option)..
This would place the load on the lookup portion wich is built on a database system that is designed to do lookup opperations rather than clogging up a router with thousands of entries that must be evaluated one by one…
This should get you well along the way to a clean system…
OpenDNS allows for custom allow / deny lists…
The forward lookup method works prety well, as most coders use URLs and not IPs to referance objects in their site..
This will block most attempts by IP…
Another feature could be adding by (FIFO) a list of denied IPs obtained durring the IP checking process.
IE an address is entered in the URL, the proxy looks it up and finds that it belongs to http://www.sex.com…
The domain is checked against OpenDNS (forward method / name allow/disallow) and lets say for example that this address is dis allowed.. or allowed as the case may be.
Once this has been established, add the address to a local buffer / list (address list would work)
The next time a request is made by IP, the local lists can be checked before the “round trip” is made to the name servers..
This will save time.. (like a cache).. then eather age the entry out by time or by FIFO based on cache size..
Nothing will “get them all” but I think this will get very close !!!
Craig
I am trying to do a combination of OpenDNS and a content filter. I need the fallback as my customers who ask for it are all using DHCP on their connections and I have no way to constantly update the entries remotely as I have to register with OpenDNS using their public IP address (when I log into the OpenDNS page it checks my public IP to see if I am allowed to set up blocking on that network). I can’t physically go to all my sites every time their DNS changes. Ideas on how to get around this?
DHCP from whom ??
Could be time warner, comcast, verizon, the list goes on. I want to do this at multiple sites and most have different isps.