Steps to setup Mikrotik Router for content filtering. I should mention this is on a quad core pc with 4GB of Ram and 500GB of hard drive space running routeros v4.11 x86 Level 6 license as a central router.
I did many searches on this topic and found no clear example of how to do this except people suggesting using squid proxy or other hardware. This still a work in progess, and I welcome thoughts and opinions on how to improve it.
Problem : Have client that want access to porn and some that want it filtered.
Step 1.) Setup router to let everyone on the internet, including all QoS, firewall filtering, and Userman Test as in other tutorials.
Step 2.) In Userman Test, under profiles, limitations, under constraints set address list to “kid_friendly” on the profile used by client wanting content filtered. You may have to setup two different profiles and limitations. One profile and limitation with rate limits and address list set to something like “normal” and another profile and limitation with the address llist set to “kid_friendly”. Userman Test is very picky and all profiles might have to be removed, reboot router, then added back for the list to populate correctly. Address list associations are somehow cached and do not take affect until router is rebooted.
Step 3.) Add these rules to the firewall filter:
;;; block adult www
chain=forward action=add-dst-to-address-list dst-address=!10.0.0.0/8 address-list=adult_block
address-list-timeout=0s layer7-protocol=kid_friendly-www
;;; block adult www
chain=forward action=drop dst-address=!10.0.0.0/8 src-address-list=kid_friendly layer7-protocol=kid_friendly-www
Step 4.)Now add this rule to the firewall NAT:
;;; Kid Friendly Web Proxy - BAD Sites
chain=dstnat action=redirect to-ports=8080 protocol=tcp src-address-list=kid_friendly dst-address-list=adult_block
dst-port=80
Step 5.) Now add words to the Layer 7 - Protocol to catch and filter:
4 kid_friendly - WWW ^.(get|GET).+(choice|words|to|filter).$
Step 6.) Enable and configure web proxy as suggested in other tutorials. Then add under access add these two rules.
;;; Not for kids URL (Enter word with vertical bar)
Dst. Host : (choice|words|to|filter) deny
;;; Block All
Dst. Host : deny
This may be the hard way of doing it but so far it is working. There are some false positives, in which the dst host address can manually be added and placed before the web proxy rules denying access. In this setup, only clients wanting to be filtered will be filtered, and then only hit the web proxy if a webpage is requested from the adult_blocked list. It also allows other clients that does not wish to be filtered to help populate the block list. I hope some of the experts will chime in with some advice. Hope this helps someone else.
All of these have to be added before any denied rules. I could not post the denied rules. If I did I would have been banned, I am filtering on some pretty nasty words. As you can see some websites have multiple entries to keep the formatting. I am pretty amazed at how many websites have the same ip address. Guess everyone but the small websites use datacenters for hosting. Feel free to fine tune these rules to suit your needs.
Just discovered a flaw in the Userman Test Database. All was working fine for a day. I had two scripts scheduled one to reboot the router every 24 hours and one script thirty minutes after reboot to rebuild the Userman Database. Reboot of the router seems to not have affected the address list, but a rebuild of the database corrupts the address list associations making every profile have the address list kid friendly even though the userman web gui shows otherwise. I had to assign a new address list kid_ok to every profile with the old kid_friendly address list and reboot the router. This seems to reassociate the address lists back correctly. For now going to disable the script rebuilding the database until a better solution is found.
On side note, search engines and news websites seem to give the false positives. Working on enforcing VSS (very strict searching) with google, yahoo, youtube, bing, and ask.
Ended up creating three differen levels of filtering. Adult, teen, and kid. Adult - no filtering, teen - filtering with OpenDNS, and Kid - filtering almost everything and redirecting Search Engines to kid friendly search engines.
In winbox, under web Proxy - > Cache tab…I just left it blank as the web proxy is not really a web proxy for caching websites in this setup. It is a filter web proxy, basically helping to filter out bad web sites.
Woow thats great topic, what Im searching for. Im searching content and URL filter with "AdminBlockedPage". I tried using by ip firewall filter/ content filtering function. It was working nice but my customer couldnt know, that page is blocked or it is problem of network connection.
Now Im trying with proxy option and still cant solve my problem. For example adult requests redirect to proxy and the proxy access rule DENY is catching those connections (hit count increases) but doesnt block and no error page, just customers successful connect to adult pages. Thats very very strange… I have tried to change address-list-timeout, but still not blocking. Some times it works, maybe blocking 1 connection of 15 requests.
Multipath, what can you advise me in this case? Ofcourse now I can copy your settings and use it. But I think if use temporary address-list, it doesn`t give the false positives. Sorry for my bad english. Please Multi advise me.
you can also simply use http://www.opendns.com/ as your DNS server, and turn on adult-filtering in there. They have made pretty good lists that you won’t have to maintain. and RouterOS configuration is simple - just add them as your DNS servers, and make DST-NAT redirect to intercept all DNS requests.
my company does this with several customers of ours throughout the state. the web url proxy list works great most of the time. however i am running into issues where the list will have a url (*.taxwise.com) for example set to be allowed and yet it still gets blocked and does not show a count as an attempt to get there nor does the blocked list count go up either. my deny rule is to deny all websites except the ones in the list that are set to allow. if anyone would like to see my rules let me know and i will be happy to post them.
The problem with Open DNS is that there’s no google image filtering. You can see anything and everything with Open DNS active. It drives me nuts that they claim to be able to filter porn on their site, it’s absurd.
Only way to block undesirable images is to block google entirely.
To block google images, you can force Safesearch by adding a static DNS entry for Google - whilst not 100% foolproof , it does block pretty much everything required.
I cant quite recall the IP address at the moment, but its something like :
/ip dns static add name=google.com address=216.239.38.120
Obviously you would need to add the same for each Google domain you want to intercept.
Hello, I’m new to the forum, friends. First of all I apologize for my English.
I am having difficulties related to this topic.
Would there be the possibility of having “domains.txt” files linked to the access lists?
Example:
/ip proxy access>
add action=deny dst-host=“blacklists/social.txt”
add action=deny dst-host=“blacklists/adult.txt”
add action=deny dst-host=“blacklists/chat.txt”
add action=deny dst-host="blacklists/games.txtIs there correct syntax?