Method to mark youtube and facebook connections

RouterOS General
1

I have 800 internet users and want to mark youtube and facebook connections to prioritize traffic.
I ran script for whatsapp, to fetch DNS cache and it worked very well…
But for youtube and facebook it did not.
I have CCR1016

The option i have it to use:
1-

  • Layer7 protocols

. I am affraid that CCR1016 wan’t be able to handle that much of matching processing power for 800 users. It shows internal ip addresses also
2-

  • DNS fetching technique. Did not show any ip address, while whatsapp for it does

3-

  • manual configuration for youtube in address list where mikrotik will put the ips dynamically. Did not show all ip addresses for youtube and facebook

2 and 3 failed.

i want to know:

  • the best method to do that for youtube and facebook


  • why DNS fetching technique did not work for youtube and facebook with whatsapp script below


:foreach i in=[/ip dns cache find] do={
:local bNew "true";
:local cacheName [/ip dns cache all get $i name] ;
:if ([:find [/ip dns cache get $i name] "whatsapp"] > 0) do={
:local tmpAddress [/ip dns cache get $i address] ;
:if ( [/ip firewall address-list find ] = "") do={
/ip firewall address-list add address=$tmpAddress list=WHATSAPP-DYN-DNS-IPS disabled=no comment=$cacheName; 
} else={
:foreach j in=[/ip firewall address-list find ] do={
:if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
:set bNew "false";
}
}
:if ( $bNew = "true" ) do={
/ip firewall address-list add address=$tmpAddress list=WHATSAPP-DYN-DNS-IPS disabled=no comment=$cacheName; 
}
}
}
}

Thanks

2

Maybe look into using TLS-Host

3

TLS-host matcher is similarly resource-intensive as L7 matcher - it uses pattern matching as well, except that the pattern syntax is more limited (thus simpler to execute) and that it takes place only after parsing the packet for being a TLS Client Hello and for containing the server_host “extension” which the ****

tls-host

matcher looks into.

But neither L7 matching nor TLS-host matching should be a resource-killer if used carefully. In particular, I would suggest to use that rule to match only TCP packets from client to server which are not connection-marked yet and their size exceeds, say, 80 bytes, and connection-mark all of them, even those which do not match any TLS host.

When a https session is established, it looks as follows:

client                    server
   ------- SYN (empty) ----->
   <--- SYN,ACK (empty) -----
   ------ ACK (empty) ------>
   --- ACK (Client Hello) -->



So we need to let through the first two packets from the client, which are small as they carry no TCP payload, without matching them for TLS host. The first packet with a TCP payload contains either the TLS Client Hello (in case of https connections) or the http GET (in case of plaintext http connections), and this is the one on whose contents we base the decision on priority group. So a row of rules translating the match patterns to connection marks expressing priority categories must be applied on this first “significant” packet of each connection, and what is important, packets mathing none of the rules must get a connection mark as well. Due to that, any further packets belonging to these connections will not be matched on TLS host or L7 pattern either.

So the whole setup for marking Youtube connections for priority handling would look as follows:

/ip firewall mangle
add chain=prerouting action=mark-packet new-packet-mark=priorityA connection-mark=priorityA passthrough=no
... similar rules for lower priorities here ...
add chain=prerouting action=mark-packet new-packet-mark=priorityZ connection-mark=priorityZ passthrough=no
#so now any already connection-marked packets, which constitute the vast majority of traffic, have been let through, without inspecting even for protocol to save as much CPU as possible
#so what got up to this point must have no connection mark
add chain=prerouting action=accept  packet-size=0-80
#skip further processing of all packets from LAN which are too small to be useful for TLS host matching; again, we're matching as little as possible to save CPU
add chain=prerouting protocol=tcp action=mark-connection in-interface-list=LAN tls-host=*youtube* new-connection-mark=priorityA passthrough=yes
...similar rules for lower priorities here...
add chain=prerouting protocol=tcp action=mark-connection in-interface-list=LAN new-connection-mark=priorityZ passthrough=yes
... additional rules changing the connection mark to a lower priority one for connections which have already transported more than X bytes of data may be placed here ...
... all the rules translating connection mark to packet mark may be repeated here to speed up connection establishment a tiny bit ...
4

Thanks Sindy,

I will check and test your post and revert back soon. :wink:

5

There was a mistake in my example above in how the ****

packet-size

match was configured. The correct rule to accept packets smaller than 81 bytes is

add chain=prerouting action=accept packet-size=0-80

(now fixed above). I was mislead by the RouterOS context help saying

PacketSize ::= [!]Min[-Max]
Min,Max ::= 0..65535 (integer number)

>

which implies that if only a single value is provided, it is the

```text
Min

one, but it seems to be an exact size instead.

Also the ****

tls-host

match needs to be wildcarded.

6

Hi Sindy,

the configuration above did not work

:frowning:

7

Have you analyzed which part did not work? Use ****

/ip firewall mangle print stats

to see whether the rules with

tls-host

work.

My problem during testing was that my Android phone has established a TLS session to Youtube once forever, so I had to disconnect it from the network and connect it again to generate a new TLS session establishment. Check this too if you cannot see any matches on the ****

tls-host

matching rules.

8

I restarted RB, to kill all current connections. Closed my browser. And tested again.
Facebook worked, but not youtube.

9

Youtube “not worked” means that the rule with ****

tls-match=*youtube*

had 0 count of packets after the web page has been open or that the rule has matched but the proper priority queue wasn’t chosen?

10

Hi Sindy,

yes the count was “0” for both youtube and facebook (i did the tests again). Below is the configuration:

0    chain=prerouting action=accept protocol=tcp in-interface=ether2-LAN packet-size=0-80 log=no log-prefix="" 

 1    chain=prerouting action=mark-connection new-connection-mark=FACEBOOK-CONN passthrough=yes protocol=tcp 
      in-interface=ether2-LAN log=no log-prefix="" tls-host=*facebook* 

 2    chain=prerouting action=mark-connection new-connection-mark=YOUTUBE-CONN passthrough=yes protocol=tcp 
      in-interface=ether2-LAN log=no log-prefix="" tls-host=*youtube* 

 3    chain=prerouting action=mark-connection new-connection-mark=YOUTUBE-CONN passthrough=yes protocol=tcp 
      in-interface=ether2-LAN log=no log-prefix="" tls-host=*googlevideo*
11

Given that for me it works like a charm (after I’ve made the proper interface a member of ****

/interface list LAN

:wink: ), I assume some of your other firewall rules interfere with these.

Please post the output of ****

/ip firewall export

, after systematically replacing any public IP address you don’t want to reveal by a unique distinctive pattern such as

my.public.ip.1

.

12

I tested two configurations. Method 1 did not work at all, while method 2 worked successfully.

Method #1:

add action=add-dst-to-address-list address-list=youtube address-list-timeout=none-dynamic chain=prerouting in-interface=all-ppp protocol=tcp tls-host=*.youtube.com
add action=add-dst-to-address-list address-list=facebook address-list-timeout=none-dynamic chain=prerouting in-interface=all-ppp protocol=tcp tls-host=*.facebook.com

Method #2:

add action=add-src-to-address-list address-list=youtube address-list-timeout=none-dynamic chain=prerouting in-interface=ether1-wan  layer7-protocol=youtube
add action=add-src-to-address-list address-list=facebook address-list-timeout=none-dynamic chain=prerouting in-interface=ether1-wan layer7-protocol=facebook

This section is for traffic categorization for download and upload which will be used later with queue tree:

For Downstream Traffic

add action=mark-connection chain=forward in-interface=ether1-wan new-connection-mark=client-downstream-conn passthrough=yes
add action=mark-packet chain=forward connection-mark=client-downstream-conn new-packet-mark=client-downstream-pkt passthrough=yes

add action=mark-packet chain=forward new-packet-mark=rx-youtube-pkt passthrough=no src-address-list=youtube in-interface=ether1-wan
add action=mark-packet chain=forward new-packet-mark=rx-facebook-pkt passthrough=no src-address-list=facebook in-interface=ether1-wan

For Upstream Traffic

add action=mark-connection chain=forward in-interface=all-ppp new-connection-mark=client-upstream-conn passthrough=yes
add action=mark-packet chain=forward connection-mark=client-upstream-conn new-packet-mark=client-upstream-pkt passthrough=yes

add action=mark-packet chain=forward dst-address-list=youtube new-packet-mark=tx-youtube-pkt packet-mark=client-upstream-pkt passthrough=no out-interface=ether1-wan
add action=mark-packet chain=forward dst-address-list=facebook new-packet-mark=tx-facebook-pkt packet-mark=client-upstream-pkt passthrough=no out-interface=ether1-wan

P.S. Method 2 did not work if i omitted the in and out interfaces. In this case for downstream traffic in-interface should be ether1-wan and for upstream traffic, out-interface should be ether1-wan for method 2 to work successfully.
I don’t know why…i have to investigate.:slight_smile:

Queue Tree looks like this:
SNAG- 2018-05-01 08 26 01.png
Best Regards.