I have HTTP and HTTPS traffic on my LAN going to the internet.
I need to be able to look at the domain the traffic is destined to, and
compare it to a list of domains to determine if the traffic is social media,
or business, or porn, or something else.
Based on what category it’s in, I then want to throttle that traffic.
So all social media traffic might be limited to 1Mbps (for everyone, not per user),
all youtube traffic might be limited to 5Mbps, etc.
My ideas so far on how to get it done:
To categorize traffic, I have a subscription to squidblacklist, which provides a
list of domains for each category.
Squidblacklist has about 40 categories, and about 2 million domains in total that they categorize.
HTTP traffic has a HOST field the router can look at, and compare against the list of domains in each squidblacklist category.
HTTPS traffic has an SNI field the router can look at to do the same.
Any matched HTTP/HTTPS connections will be marked with Mangle rules. Then queues will be setup that match those markings to limit each category to some specific speed.
Is this possible?
Note: if I need to redirect the traffic to a linux box to do some/all of the work in linux, I can do that
As far as I know, using the tls-host rule would require me to create 2 million firewall rules, as there are 2 million hosts I’d be checking for.
At least a few thousand, if I narrow it down to only certain categories I care about.
I’m fairly certain if every single packet going through a mikrotik has to be checked against a few thousand, to 2 million firewall rules, the mikrotik will catch on fire.
Save yourself the trouble, much of what you seek is done here for mere pennies
Search MOAB on this forum.
I would use it but personally use this currently (as I can write it off for taxes). https://axiomcyber.com/shield/
As far as I can tell, neither MOAB, nor https://axiomcyber.com/shield/ has anything to do with traffic shaping.
They block traffic to and from dangerous IPs.
My use case has nothing to do with security. I need to rate limit (not block) traffic based on category.
Also, I’m dealing with domain names - not IPs like MOAB and axiom seem to. Since I can’t make an address list of domain names on a Mikrotik, the best I could do is convert 2 million domain names to several million IPs (each domain might resolve to several IPs), on a regular basis (to keep it up-to-date), add them all to different mikrotik address lists, and use packet marking and queues based on IPs.
It would take several days for my fast desktop to resolve 2 million domains to IPs (in a single thread). I have no clue how many years it would take a Mikrotik to do it, but it’s not going to be very practical to have the mikoritk do it on a regular basis.
Haha. Exact requirements. Good one.
All verbal. All loosely defined. After I build it, if he doesn’t like it, he’ll make me change it.
Like I said - we need to be able to throttle speed based on category.
Category being netflix, social media, youtube, etc. His words.
We have a network appliance that can do it, so he knows it’s possible.
The network appliance is too big and expensive to use for this project.
Either the Mikrotik can do it, or a Linux box can do it, or we need some
cheap network appliance to do it.
Well, if you are going to use layer7 then get the biggest CPU with most memory.
The problem is not the equipment its the CEO for not creating and enforcing IT policy that simply ’
states, use of P2P, NETFLIX, YOUTUBE is expressly forbidden.
Offenders will get two warnings and then will be terminated.
That should clear up the issues making your job merely one of detecting and logging evidence.
Dealing with employees using customer resources for personal uses is not on for most businesses, other than personal banking.
Protecting the network is not just an IT phenomena, there is education and consequences!!
The users don’t work for our company. They are our customers.
We can tell them “if you’re using our service, it’s going to have some restrictions”.
We can’t terminate anyone. The best we can do is terminate our services, and if
we do that, then we don’t have a customer, which is shooting ourselves in the foot.
With respect, I don’t want this to turn into a debate over business practices. I
don’t make any decisions for the company. It is what it is. I can either complete
the task I was assigned, or I can’t.