Community discussions

 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Ad Blocking hosts file

Thu Jan 21, 2010 5:09 pm

I have been using DD-WRT and have switched to MikroTik products and os.

I have been using a script for DD-WRT that loads a hosts file of unwanted urls of ad and malware url's. http://www.mvps.org/winhelp2002/hosts.htm

Does anyone have a script for RouterOS for this ?
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Thu Jan 21, 2010 5:18 pm

My current DD-WRT script

________________

_rogue=0.0.0.0
echo -e "#!/bin/sh\nn=1\nwhile sleep 60\ndo\n\twget -q -O - http://www.mvps.org/winhelp2002/hosts.txt | grep \"^127.0.0.1\" | grep -v localhost | awk '{print \"$_rogue\\\t\"\$2}' | tr -d '\\\015' >/tmp/dlhosts\n" >/tmp/write_dlhosts
echo -e "\t[ \`grep -il doubleclick /tmp/dlhosts\` ] && break\n\t[ \$n -gt 5 ] && break\n\tlet n+=1\ndone\n[ -e /jffs/hosts ] && cat /mmc/hosts >>/tmp/dlhosts\n[ -e /opt/etc/hosts ] && cat /opt/etc/hosts >>/tmp/dlhosts\nkillall -HUP dnsmasq" >>/tmp/write_dlhosts
chmod +x /tmp/write_dlhosts
/tmp/write_dlhosts &
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Ad Blocking hosts file

Thu Jan 21, 2010 5:39 pm

The current RouterOS scripting language has a file limit of 4KB (or rather, variables can't contain more than 4096 bytes, and the only way to access a file's contents is to assign all of its contents to a variable). So you can't have a script manipulate that hosts file since it's over 600KB.

You can use outside hosts to convert it, though. In Perl something like this would work:
perl -e 'print "/ip dns;remove [find];\n";while(<>){unless(m/^#/){split;print "add address=127.0.0.1 name=".@_[1]."\n";}}' hosts.txt
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Fri Jan 22, 2010 12:18 pm

wow... really ?

That sucks. So there is no way to do this on a schedule completely inside the router ?

Im kinda stunned

Ok lets say I convert the file outside the router. How do I get it into the router into the internal hosts file ?
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Mon Feb 01, 2010 1:24 am

I would like to revisit this thread..

I would really like to block ads and bad sites using DNS in the router.

Assuming I have the list how do I get it into my static DNS list ? I also need to send the bad url's to 127.0.0.1 kinda thing. I assume using DNS to do this would be better then using firewall rules ?

I still ultimately want to use a scheduled script to pull the data, parse it for use, then get into into my DNS list as 127.0.0.1 destinations.

This would be a very useful script for many people.
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Ad Blocking hosts file

Mon Feb 01, 2010 1:40 am

I just tried this out. On a webserver you have access to, download the hosts file and use the below to convert it into a RouterOS script:
perl -e 'print "/ip dns static; remove [find];\n";while(<>){if(m/^127.0.0.1/){split;print "add address=127.0.0.1 name=".@_[1]."\n";}}' hosts.txt > hosts.rsc
Then you can schedule a script on the router that uses "/fetch" to download the file that adds all the host entries, and then use "/import" to import it.

However, don't do that. I tried this on my 750G. It took 5 minutes to import all those entries, and it nearly died with only 1MB of RAM left. Removing them is still going on at a rate of about 200 a second. It racked up nearly 20,000 NAND sector writes. A massive hosts file like you want to use simply doesn't scale to at least the entry level routers, and I'd say that's indicative that even the big ones won't handle things well as the built in DNS server isn't optimized for serving that many static records.
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Mon Feb 01, 2010 2:41 am

wow....

Well thank you for discovering this...

Sorry it almost melted your 750G !

Man this is quite the task...

Its so easy with a OLD linksys 54G-TM running DD-WRT. Download the file, convert it, copy, restart the DNS server.

Im amazed that a 3 year old retail router + DD-WRT can do this so easily and Mikrotik its just about impossible to do on a 750G.

Amazing... I keep thinking there must be a way...
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Mon Feb 01, 2010 2:54 am

From DD-WRT discussion

"An additional option is to have DNSMasq read an ADDITIONAL hosts file (for those who use the hosts method of ad blocking)"

http://www.dd-wrt.com/phpBB2/viewtopic. ... sc&start=0

Its really easy with the ability to get to the *nix hosts file directly using a option with DNSMasq.

They use it on a very wide range of retail entry level routers, some quite old, so i dont think there is a limitation doing it hardware wise with a 750G. It seems its a software limitation.

I would like better options for DNS like the ones you get with DNSMasq.
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Ad Blocking hosts file

Mon Feb 01, 2010 3:04 am

Do a feature request with support, they may not read this as it's a community forum.

For what it's worth I understand why you want this feature but personally I'd prefer they work on functionality related to routing. If you need a DNS server you can always install a separate DNS server
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Ad Blocking hosts file

Mon Feb 01, 2010 3:11 am

[...]

However, don't do that. I tried this on my 750G. It took 5 minutes to import all those entries, and it nearly died with only 1MB of RAM left. Removing them is still going on at a rate of about 200 a second. It racked up nearly 20,000 NAND sector writes. A massive hosts file like you want to use simply doesn't scale to at least the entry level routers, and I'd say that's indicative that even the big ones won't handle things well as the built in DNS server isn't optimized for serving that many static records.
Sounds like a bug to me. 1800 bytes per entry it's more than just inefficient...
 
fewi
Forum Guru
Forum Guru
Posts: 7734
Joined: Tue Aug 11, 2009 3:19 am

Re: Ad Blocking hosts file

Mon Feb 01, 2010 3:22 am

It's about half that, RAM went from about 15000kb free to about 1000kb free, there's a total of 14883 entries in that hosts file so that's more or less a kilobyte per entry.

Still way too much, though. Could be a bug or just a horribly inefficient implementation. I doubt it was written with 15K entries in mind.
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Ad Blocking hosts file

Mon Feb 01, 2010 5:51 am

Even with less entries - when perhaps memory is not an issue - reading or writing 1KB wastes CPU cycles. Someone should send MT a note about this :)
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Mon Feb 01, 2010 7:00 am

yea I am sure they never thought about 15K static entries !

I am a newbie. I would think deep down in Mikrotik there is Linux ? No ? I guess we have no access to the OS ? and Files ? I suppose its a license issue to simply use DNSMasq if it is *nix down in there.. There must be a dns hosts file in there someplace. If only we could get to it..

Wait, stupid question. The file is 606KB. How does that occupy so much RAM ? Wow your right something is weird.

I would like this capability in a $70 router :) Plus I think this capability is useful for all sorts of DNSBL kinda uses besides ad blocking. Having the router download a realtime DNSBL list from somewhere could be useful to stop users from going to malware sites for example.

The DD-WRT implementation of DNSMasq is nice. Considering how cool Mikrotik is,,, it needs a better DNS server :) Can't have DD-WRT out do Mikrotik !
 
reinerotto
Member
Member
Posts: 437
Joined: Thu Dec 04, 2008 2:35 am

Re: Ad Blocking hosts file

Mon Feb 01, 2010 10:57 am

So actually you could think about using an upstream proxy server for your MT-Box, to remove ads. There are various opensource proxies available to do that, like privoxy, for example. I do it this way, and it has the charme, that I also added virus/malware filtering (clamAV) thru another chained proxy, and dansguardian proxy for content filtering. No access to adult sites etc. :-) squid is running on the proxy as well, of course.
Automatic updating of malware/virus defs is standard; clamAV also has the option of using googles safe browsing data.
Because of transparent proxy on MT-box, no user action necessary.

When you are serious about protecting your users, I think this is the better solution. And it works :-)
Last edited by reinerotto on Tue Feb 02, 2010 11:23 am, edited 2 times in total.
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Mon Feb 01, 2010 6:11 pm

But look how much more effort and expense that is.

Just downloading, parsing and then copying a file is SOOOOO much easier and simplier not to mention cheaper. No user configuration. Applies to all users instantly. Auto updating.
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Ad Blocking hosts file

Sun Feb 21, 2010 8:55 am

Btw, I was successful adding all 14.7k entries to proxy access file. Interestingly safe mode failed with stack overflow but import succeeded. Memory used by the list- about 46MB...
 
User avatar
Xymox
Member
Member
Topic Author
Posts: 387
Joined: Thu Jan 21, 2010 5:04 pm
Location: Phoenix, Arizona US
Contact:

Re: Ad Blocking hosts file

Sun Feb 21, 2010 9:23 am

Besides safe mode issue and huge memory usage did this work ok ? did it cause any weird traffic slowdowns doing lookups or anything weird like that ?

I know have a 450G and its got lots of free ram and a micro-sd card.
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Ad Blocking hosts file

Sun Feb 21, 2010 11:44 pm

It works fine. I suspect that DNS would work ok too. It's just that both waste a lot of RAM per entry so the blocklist is not suitable for <128MB RAM routerboards...

[...]

EDIT2: I spoke to soon. I had to remove all the rules due to random reboots and hangs
 
birendersinghbudhwar
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Thu Oct 29, 2009 5:15 am
Location: Rohtak, HR, INDIA

Re: Ad Blocking hosts file

Thu Apr 01, 2010 6:35 pm

Enjoy the this feature with this DNS server 121.246.169.96 and forget about the scripting just enjoy it i am using this dns in my miktotik and other routers and system and all hackers, ads, phishing site etc are out of our network.

Birender
Birender Singh Budhwar
Network / Server Administration
 
sigxcpu
newbie
Posts: 27
Joined: Wed Sep 14, 2011 12:54 pm

Re: Ad Blocking hosts file

Wed Sep 21, 2011 12:25 pm

Enjoy the this feature with this DNS server 121.246.169.96 and forget about the scripting just enjoy it i am using this dns in my miktotik and other routers and system and all hackers, ads, phishing site etc are out of our network.

Birender
And all ad requests are redirected to a VERY slow loading page that does 302 redirect to http://stats.wifi4india.com/awstats.

Why would somebody prefer a very slow loading web page instead of fast ads?
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6283
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Ad Blocking hosts file

Wed Sep 21, 2011 12:37 pm

one misconception - variable can hold a lot of data (definitely more than 4K), just working with files in scripts has this limitation of 4K size.
 
birendersinghbudhwar
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Thu Oct 29, 2009 5:15 am
Location: Rohtak, HR, INDIA

Re: Ad Blocking hosts file

Thu Oct 20, 2011 9:49 am

Enjoy the this feature with this DNS server 121.246.169.96 and forget about the scripting just enjoy it i am using this dns in my miktotik and other routers and system and all hackers, ads, phishing site etc are out of our network.

Birender
And all ad requests are redirected to a VERY slow loading page that does 302 redirect to http://stats.wifi4india.com/awstats.

Why would somebody prefer a very slow loading web page instead of fast ads?
I don't know about your case but in our case our users are getting

Ads and harmful content is blocked Powered By Wifi4india

check a sample screen shot of this page attached.
You do not have the required permissions to view the files attached to this post.
Birender Singh Budhwar
Network / Server Administration
 
User avatar
dunga
Member Candidate
Member Candidate
Posts: 254
Joined: Fri Jan 23, 2009 9:51 am
Location: Nigeria

Re: Ad Blocking hosts file

Wed Nov 02, 2011 3:37 pm

how is this possible to block advert or adbanner in mikrotik, s tht some adverts on websites will not show when opened. There is this software adzapp which works in linux.

how do we implement this cus these sites consume bandwidths like hell. Is there a sway again how to reduce the consumption of bandwidth from sites or downloading files that have mp3, mp4 etc
 
birendersinghbudhwar
Frequent Visitor
Frequent Visitor
Posts: 98
Joined: Thu Oct 29, 2009 5:15 am
Location: Rohtak, HR, INDIA

Re: Ad Blocking hosts file

Thu Nov 03, 2011 11:25 am

how is this possible to block advert or adbanner in mikrotik, s tht some adverts on websites will not show when opened. There is this software adzapp which works in linux.

how do we implement this cus these sites consume bandwidths like hell. Is there a sway again how to reduce the consumption of bandwidth from sites or downloading files that have mp3, mp4 etc
you can set the DNS rule for specific IP address only so that peoples cant change their DNS if they change their own then there will be no internet you can set mikrotik to send request only to specific IP address on which DNS is running so users are binded to your DNS server only and have enjoyed.

Birender
Birender Singh Budhwar
Network / Server Administration
 
User avatar
dunga
Member Candidate
Member Candidate
Posts: 254
Joined: Fri Jan 23, 2009 9:51 am
Location: Nigeria

Re: Ad Blocking hosts file

Thu Nov 03, 2011 2:44 pm

you can set the DNS rule for specific IP address only so that peoples cant change their DNS if they change their own then there will be no internet you can set mikrotik to send request only to specific IP address on which DNS is running so users are binded to your DNS server only and have enjoyed.

Birender
How do i set them i need a detail way of achieving it. I need a tested and working stuff, so that i dont go messing up things.

Thanks
 
H2009
Member Candidate
Member Candidate
Posts: 137
Joined: Tue Oct 26, 2010 8:46 am

Re: Ad Blocking hosts file

Wed Nov 20, 2013 11:23 am

Hi I would like to follow this as well - i can see ubnt they have a script to remove it from there side - wondering if anyone is good enough to convert it to usage with for ROS.


http://community.ubnt.com/t5/EdgeMAX/Ad ... 39#U623239
To show support for those with food allergies/intolerances please download the application from www.biteappy.com
 
cusco
newbie
Posts: 34
Joined: Tue Jun 29, 2010 2:34 pm

Re: Ad Blocking hosts file

Thu Mar 27, 2014 9:25 pm

Hi,

Because http://winhelp2002.mvps.org/hosts.txt has DNS names instead of IP's (seems logic), can there be a entry in L7 that detects the DNS querie on these hosts?

How does one add a L7 entrye for a DNS lookup of.. say.. google.com?

We had this lists of DNS in a RB1200 and we noticed performance issues (CPU peaks at times), not stating the time for winbox to list static DNS entries.
Doesn't seem the best way to go at it, but I'm not sure if L7 would perform better!


Any ideas?
 
reinerotto
Member
Member
Posts: 437
Joined: Thu Dec 04, 2008 2:35 am

Re: Ad Blocking hosts file

Sun Mar 30, 2014 10:17 pm

Use your own DNS-server on a separate LINUX box, with transparent squid cache.
Then you convert host.txt to named.conf.local (there is a script to do that) and feed it into your bind9.

Your resulting named.conf.local will contain lines like this one:
zone "ad-srv.net" { type master; notify no; file "/etc/local_server.db"; };

and /etc/local_server.db should be like:

; NULL Zone File for Ad Servers
;
$TTL 36000
@ in soa localhost. postmaster.localhost. (
2002110101 ;serial
3600 ;refresh
1800 ;retry
604800 ;expiration
3600 ) ;minimum

;
; Zone NS records
;

@ NS localhost.

A 127.0.0.1
* IN A 127.0.0.1


May be, you can do something like this directly using MTs DNS-server.
 
cusco
newbie
Posts: 34
Joined: Tue Jun 29, 2010 2:34 pm

Re: Ad Blocking hosts file

Wed Jun 10, 2015 3:42 pm

have a look at
https://github.com/aziraphale/routeros-dns-adblock

I found it usefoul
 
User avatar
NetVicious
Member Candidate
Member Candidate
Posts: 109
Joined: Fri Nov 13, 2009 3:30 pm
Location: Spain

Re: Ad Blocking hosts file

Wed Jul 22, 2015 3:19 pm

Hi!

I got here looking for one configuration for RouterOS to set an external zone to the Routerboard DNS server.

I read here something about dnsmasq or creating a new zone. This it's possible using RouterOS?

I want to say to the RouterOs DNS if it gets a query asking for domain.com it should ask for that domain the dns 192.168.1.1 as example instead of going out and querying the root dns servers.
. . //\/ e t . \/ i c i o u s ..
 
MikroDik69
just joined
Posts: 6
Joined: Tue Nov 19, 2013 5:49 am

Re: Ad Blocking hosts file

Thu Apr 21, 2016 9:58 am

have a look at
https://github.com/aziraphale/routeros-dns-adblock

I found it usefoul
Can you elaborate on how to use that? The readme instructions aren't clear / simple enough for me to understand.
 
cusco
newbie
Posts: 34
Joined: Tue Jun 29, 2010 2:34 pm

Re: Ad Blocking hosts file

Wed Jul 20, 2016 3:01 pm

have a look at
https://github.com/aziraphale/routeros-dns-adblock

I found it usefoul
Can you elaborate on how to use that? The readme instructions aren't clear / simple enough for me to understand.
Hi.
This is basically a set of PHP scripts, that allow you to generage .rsc files (to import into routerOS and run them)
so in command line in linux with php:
 cusco@devel ~$ git clone https://github.com/aziraphale/routeros-dns-adblock.git
Cloning into 'routeros-dns-adblock'...
remote: Counting objects: 61, done.
remote: Total 61 (delta 0), reused 0 (delta 0), pack-reused 61
Unpacking objects: 100% (61/61), done.

 cusco@devel ~$ cd routeros-dns-adblock/

Change process.php as you wish (make BIND9_OUTPUT false)
devel ~/routeros-dns-adblock$ php -q process.php
NOTE: Removing duplicate hosts is ENABLED (via crc32).

spam404                                 => 240.0.0.5       ... 4590 hosts (821.99ms) (2 files)

Total duration: 822.11ms
Total hosts:    4590
Total files:    2
Peak RAM use:   1.50 MB

devel ~/routeros-dns-adblock$ cat script.spam404-*

So I just parsed spam404 (commented the others on process.php)
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24268
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Ad Blocking hosts file

Wed Jul 20, 2016 3:31 pm

You have to be on macOS or Linux to do these commands, if you have no access  to such, I have attached the output to this post (have not tested), so you can see what it does. 

The commands on my Mac were: 
git clone https://github.com/aziraphale/routeros-dns-adblock.git
cd routeros-dns-adblock/
now open the file process.php in a text editor and find this line:
define('BIND9_OUTPUT', true);
change it, so that it says now:
define('BIND9_OUTPUT', false);
now you can run this command:
php -q process.php
After that, you will have a few new RouterOS import files:
/ip dns static

add address=240.0.0.3 name="banners164.hpg.com.br"
add address=240.0.0.3 name="banners165.hpg.com.br"
add address=240.0.0.3 name="banners166.hpg.com.br"
add address=240.0.0.3 name="banners167.hpg.com.br"
add address=240.0.0.3 name="banners168.hpg.com.br"
add address=240.0.0.3 name="banners169.hpg.com.br"
add address=240.0.0.3 name="banners17.hpg.com.br"
add address=240.0.0.3 name="banners170.hpg.com.br"
add address=240.0.0.3 name="banners171.hpg.com.br"
add address=240.0.0.3 name="banners172.hpg.com.br"
add address=240.0.0.3 name="banners173.hpg.com.br"
add address=240.0.0.3 name="banners174.hpg.com.br"
add address=240.0.0.3 name="banners175.hpg.com.br"
You will need a powerful machine to use this, since it will import 55000 static DNS entries into your router. 
You do not have the required permissions to view the files attached to this post.
No answer to your question? How to write posts
 
cusco
newbie
Posts: 34
Joined: Tue Jun 29, 2010 2:34 pm

Re: Ad Blocking hosts file

Wed Jul 20, 2016 4:50 pm

You have to be on macOS or Linux to do these commands, if you have no access  to such, I have attached the output to this post (have not tested), so you can see what it does. 

The commands on my Mac were: 
...
Thank you normis, your explanation sounds a lot better :-)
 
Metroiss
just joined
Posts: 1
Joined: Sat Jul 30, 2016 6:27 pm

Re: Ad Blocking hosts file

Sat Jul 30, 2016 6:37 pm

I've successfully entered a large list of DNS entries into my Static DNS and I found that the problem isn't the large number of entries, its the TTL resetting in the dynamic server.  When I added the host list into static all of the TTL are set for 1 day, however, when I watch the entries populated in the dynamic list i noticed that they are reinitialized about every 3 seconds (TTL constantly goes from 23:59:57 to 1d) causing the CPU to go to 100% for a few seconds.  Not sure if this is a bug, none of they dynamically assigned DNS entries are reset, they live out their TTL and die.  If the static TTLs were given a chance to last a day then the router would only use the CPU for a few seconds once a day to repopulate the static DNS entries.  Is there a setting I'm missing or is this a bug?  If this is fixed they only limitation on the router would be how much ram you have and these host files are not very large, I believe i set mine at 8 meg and have a list of over 6500.

Who is online

Users browsing this forum: MSN [Bot] and 178 guests