We forced to filter over 50000 URLs (and this number are growing) due to laws in Russia. If we wont do it - it will cause fines and ISP license suspension.
Is it possible to do this on Mikrotik?
We currently have CCR1009-8G-S1-S1+ with a 30% CPU load.
You could use OpenDNS or something like that if you need to block by specific URL. Or if you have your own DNS server, do something similar to OpenDNS.
In routerOS to block this in DNS cache is not a good idea.
You can use address lists in the RB too, if you have no problem to block by IP. this will be very fast, but you need all the IPs
Of course blocking a URL is not the same as blocking a hostname or blocking an IP address.
So what is the real requirement? Blocking a URL would mean you block e.g. http://host.example.com/illegal/content but at the same
time you allow http://host.example.com/content/that/is/ok.
Blocking a hostname would mean that all access to host.example.com is to be blocked no matter what the exact URL is.
Blocking an IP would mean that you can safely block all addresses that result when you lookup host.example.com in DNS,
and you don’t mind that a lookup of anotherhost.example.com that returns the same IP addresses means that anotherhost.example.com
is blocked as well (even though it was not on your list of hosts to block).
We have the IPs (over 130000 of them) but if we block them - entire websites will be blocked.
We need to block specific pages somehow.
For example: some-site.com/suicide_tutorial.html is in block list.
But some-site.com/everything_else is SFW.
Just imagine if it is Youtube.
Manipulating DNS seems a good idea for the sites that blocked entirely.
pe1chl,
Yes, you understand it correctly.
I looked at hardware for traffic filtering - it’s too expensive for a small provider.
You’ll need to redirect all traffic through a transparent web proxy such as Squid, using a RB for this is not the right tool. Keep in mind that filtering HTTPS at the URL level is not possible.
As R1CH correctly writes: for blocking http URLs you can use a transparent proxy, but it will not work for https.
As most services (including youtube) are now migrating to https or have already done so, it will become impossible to do what you want.
It is a controversial topic: the proponents of https have the opinion that https is better exactly because of this.
They do not want intermediate systems to be able to inspect and manipulate traffic.
However, what these people often do not understand is that in some situations such inspection is required for legal
or network-management reasons, and they effectively lock out people from internet because the only workable solution
is to stop operation and disconnect all users. That is a pity, but we cannot change it. The move to encryption is irreversible.
I think your only option (and this is unrelated to MikroTik) is to stop operation as long as the legal requirements to not change.
What is we use a dedicated server for a filtering purposes?
We install DNS-server and some transparent proxy on it.
Then we put a list of domains that are referenced in the blocklist as a static entries to DNS-server and assign them an IP address of the filtering server.
Then on the port 80 we will have a transparent proxy, that will check if current HTTP URL is in the blocklist. If so - it will redirect to the page with “site is blocked” text on it, otherwise it will pass it.
All we need in this case - is to change DNS server on mikrotik to a filtering server’s one.
All HTTPS unfortunately will be blocked without URL checking. There’s just no other way. Luckily, there’s none of very popular HTTPS sites in the blocklist.
Could you tell me if it’s possible?
PS: not used transparent proxy before, so I don’t know its limitations.
PS: sorry for asking a non-mikrotik-related question.
Those laws were came to power in 2012. How long we should wait for the things to change? Many providers had enabled filtering and already stopped complaining about these laws.
Yes it is certainly possible to do that. A Linux server with Squid and a DNS resolver (bind9) will do the job.
You have to realize that its days are numbered because “everyone” is switching to https, because it is now
very easy to obtain certificates and search engines and browsers are trying to move people to https by
preferring https links and by displaying more and more warnings for sites without https.
But of course that will affect you and your competitors the same way.
Also, in countries like that the users will normally use some VPN service that will take them around such
local limitations. So it is not effective at all. However, that is probably not your problem either.
(until you are told to block these VPN services. that is very, very difficult! if not impossible)
Thanks, I’ll try this. But again, how proxy will work with a list of over 50000 blocked URLs? How do I store them - DB or a text file?
Also, in countries like that the users will normally use some VPN service that will take them around such
local limitations. So it is not effective at all. However, that is probably not your problem either.
(until you are told to block these VPN services. that is very, very difficult! if not impossible)
Yes, that’s correct. VPN, proxy, tunneling and other methods aren’t restricted yet, and we aren’t forced to block them.
You can use a squid redir plugin, there are several of them, e.g. dansguardian.