Community discussions

MikroTik App
 
mladen074
just joined
Topic Author
Posts: 8
Joined: Mon Nov 27, 2017 3:54 pm

Mark the traffic for YouTube, Facebook, etc.

Sun Dec 10, 2017 1:29 pm

Hi all,

I've figured the easy way to mark the traffic related to the YouTube, Facebook, etc.

Introduction:
So far, I read on the forum about people manually maintaining huge lists of ip addresses for this purpose. Starting from v6.36 RouterOS allows adding domain names to address-lists. This means we can simply add youtube.com to the address list and RouterOS will automatically resolve this hostname to the IP address(es) and place it into the address list. This helped a lot, since we could just add for example youtube.com to the address list, hoping that it would distinguish the desired traffic automatically. But, this wouldn't work, because yt, fb and others are using what's called a CDN (Content Delivery Network) for their content delivery and only use the main domain for the web UI. Further more, these CDNs are usually geographically dispersed around the world, in order to provide consumers with the closest servers for content delivery.
These CDN hostnames/ip addresses are what we are looking for to mark/filter/limit the traffic to/from it.

The problem:
Find a way to automatically update all the CDN ip addresses (place them into an address list), so we could mark/filter/limit the traffic coming from/to these addresses. If we take a look at one youtube video request, we'll see that the content delivery usually comes from "googlevideo" CDN network. Hostnames like these are usually seen in your connection list (depending on the country you are coming from):
r5.sn-ncc-cxbe.googlevideo.com
r15.sn-c0q7lnek.googlevideo.com
r5.sn-4g5ednsl.googlevideo.com
It's apparent that the subdomains are quite difficult to predict (which I guess was intentional, since they don't want their service to be easily blocked). But one thing is predictable there and it's the domain googlevideo.com itself. The similar case is with facebook and the other big players.

One of the possible solutions:
Now, we'd like to automate the discovery of all these hosts on such CDN networks, which our users are visiting. We could sniff the content of the HTTP requests to youtube.com and see which CDN hosts are being offered for content delivery, but this is CPU consuming and lately even not possible, due to most of the websites switching to HTTPS anyway. Another way could be that we sniff the DNS requests for the hostnames on these CDN domains, and get the hostnames/ip addresses we need. The issue with this approach is that our users might be using some public dns servers (like Google's 8.8.8.8 for example), so we would need to sniff all the DNS traffic in order to discover new CDN hostnames. Sniffing such traffic also requires CPU to parse the DNS requests/responses. Unless we redirect all DNS requests through our DNS service, running on our router. This way, our router will serve as a DNS server for our users, no matter which public DNS server users choose to use. It will make sure we always have the DNS cache populated with the current CDN hostnames being used by our users. The downside of this approach is the increased CPU usage, due to the increased DNS traffic, which now our router needs to handle.

So, in short, we could solve this problem in the following way. First, we redirect all the DNS requests to our DNS server, running on the router:
/ip firewall nat
add disabled=no chain=dstnat protocol=udp dst-port=53 action=redirect to-ports=53
add disabled=no chain=dstnat protocol=tcp dst-port=53 action=redirect to-ports=53

We create a simple script, which will filter all the dns cache entries, looking for the entries we need (like "googlevideo.com") and will update our Address List named "social":
:foreach i in=[/ip dns cache find name~"googlevideo.com"] do={ :do { /ip firewall address-list add list=social address=[/ip dns cache get $i name]; } on-error={} }
The part
/ip dns cache find name~"googlevideo.com"
looks for all the dns cache entries having "googlevideo.com" in them. After that we iterate through found etries, using foreach, adding each entry's name to our address list named "social". Note the "on-error" part at the end of the line. Starting from v6.2 RouterOS scripting has ability to catch run-time errors. This is the way to tell the RouterOS not to stop when it encounters a duplicate entry in our address list. That's the reason why we have double "do" construction, too.

At this point, we can either mark/filter/limit all the packets coming/going from/to these ip addresses.

If you have any idea how can we improve the solution to this problem, please leave a comment.
 
ukitnam
just joined
Posts: 19
Joined: Thu Aug 10, 2017 4:56 pm

Re: Mark the traffic for YouTube, Facebook, etc.

Sat Jan 12, 2019 4:51 pm

nice. but do we need to run this on scheduler? if so, do we need to clear the previous addresslist? how frequent is it recommended to run this on scheduler?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Sat Jan 12, 2019 5:06 pm

So the example catches google.video.com but what about all the ones we dont know?
 
neutronlaser
Member
Member
Posts: 445
Joined: Thu Jan 18, 2018 5:18 pm

Re: Mark the traffic for YouTube, Facebook, etc.

Sat Jan 12, 2019 9:31 pm

 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: Mark the traffic for YouTube, Facebook, etc.

Sun Jan 13, 2019 12:18 am

Interesting approach. Just need to determine most "basis" name structures used for streaming.
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Mon Jan 14, 2019 3:16 pm

in firewall mangle, i just use tls-host to detect if its *.googlevideo.com then write it to a address list, from then i can limit the speed.

not sure for Facebook though.

add action=add-dst-to-address-list address-list=YOUTUBE address-list-timeout=\
12h chain=forward comment="add youtube to address list" dst-port=443 \
packet-mark=no-mark protocol=tcp tls-host=*.googlevideo.com
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Fri Jan 18, 2019 11:23 pm

Pegasus is this a working application of your idea??
Context: Ensure those using Yyutube are sent do a secondary WAN connection.

In order to properly identify traffic heading for youtube probably requires manglink/marking of some sort. Its a two step process in general.
One is to mark connections to youtube, then two, you attach a route marking rule to that traffic (marking connections is more efficient in that one marks the connection and all packets are then
Thankfully MT now has a tls-host option to help with the fact that most traffic is now https!

First step will be to identify find users who are attempting to access youtube and add the destination addresses to a firewall address list.
/ip firewall mangle
add action=add-dst-to-address-list address-list=youtube_users address-list-timeout=\
12h chain=prerouting dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.googlevideo.com passthrough=yes

/ip firewall mangle
add action=add-dst-to-address-list address-list=youtube_users address-list-timeout=\
12h chain=prerouting dst-port=443 interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.youtube.com passthrough=yes

/ip firewall mangle
add action=mark-connection chain=prerouting dst-address-list=youtube_users in-interface-list=lan\
connection-mark=no-mark new-connection-mark=markUtube passthrough=yes
add action=mark-routing chain=prerouting connection-mark=markUtube\
new-routing-mark=routeUtube passthrough=no

In plain english, the router will look at packets without any markings, ie this is done before sending traffic anywhere (not routed or going anywhere direct) but just showing up at any interface covered by the "in-interface-list". If the packets have no markings and has a destination port of 443 (and has a domain identification of youtube.com for example) the associated destination address of the connection is added to the applicable address list. The we say step to the next rule............
The next rule states, if the packets has no markings and has a destination port of 443 (and has a domain identification of googlevideo.com for example) the associated destination address of the connection is added to the applicable address list. The we say step to the next rule............

Next the router will look for any connections intended for the destination address from our created list (identified by our now populated firewall address rule). We set it up such that that the packet being inspected is the first in a new connection without any connection markings because we want the router to only look at new connections (and ignore allready marked traffic).
The router will mark the connection with the mark (markUtube) and any of its associated packets will not be inspected and will automatically receive the temporary mark.
Then you tell the router look at the next prerouting rule for this connection
The router then sees that for any connection (and associated packets) with the connection mark of markUtube attach another kind of mark, a routing mark to that connection (and its associated packets). Then you tell the router basically there are no more marking rules that apply.

So how do we use this special routing mark in IP Route rules.
We make use of the Routing mark option available (see winbox).

++++++++++++++++++++++++++++++++++++++++++++

Simple example failover:
/ip route
add check-gateway=ping distance=2 gateway=Gateway IP of WAN2 (assumes/translates to destination default entry of 0.0.0.0/0 if using winbox)
add distance=3 gateway=Gateway IP of WAN1 (assumes/translates to destination default entry of 0.0.0.0/0 if using winbox)
add distance=1 gateway=Gateway IP of WAN1 routing-mark=routeUtube

Complex example recursive failover
/ip route
add check-gateway=ping distance=2 gateway=8.8.4.4
add distance=2 dst-address=8.8.4.4/32 gateway=GatewayIP of WAN2
add distance=3 gateway=Gateway IP of WAN1
add distance=1 gateway=Gateway IP of WAN1 routing-mark=routeUtube

Three things not sure of in this approach.

1. Do I Need to state distance=1 to ensure users that connect to youtube go out the secondary WAN or is the mark rule enough??
2. Should I use no-mark packets for the first mangle rule or is sticking with no-mark connections more efficent/optimal/accurate??
3. Should I be using jump chain rules here? I'm not sure if I have used passthrough correctly??
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Mon Jan 21, 2019 7:35 am

@anav, im not particularly sure on your setup because my network is simple. But i think you are in the right track.

1. You would need to pre-route mark (Mangle) any tls-host=*googlevideo.com so that you can process it in QUEUE.

2. Since you have mark in on prerouting table, you can create a rule going out to WAN1 or WAN2 as you have mentioned based on this mangle marking

3. Process the speed in queue tree
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 11:58 am

I also tried implementing you tube Traffic control via this and its absolutely not working.

TSL host thing is totally useless in this case and doesnt pick actual IP of video stream *.googlevideo.com *.youtube.com give me about 4 ip to my address list, but when i start youtube video it comes from some other ip which idk how to detect and mangle.
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 12:34 pm

I also tried implementing you tube Traffic control via this and its absolutely not working.

TSL host thing is totally useless in this case and doesnt pick actual IP of video stream *.googlevideo.com *.youtube.com give me about 4 ip to my address list, but when i start youtube video it comes from some other ip which idk how to detect and mangle.
lol its not useless, you would either.

1. mark all ip address that *googlevideo.com is using and add address to address-list. use queue tree as well to limit bandwidth

2. mark that packets and use queue tree to limit the bandwidth
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 12:45 pm

I also tried implementing you tube Traffic control via this and its absolutely not working.

TSL host thing is totally useless in this case and doesnt pick actual IP of video stream *.googlevideo.com *.youtube.com give me about 4 ip to my address list, but when i start youtube video it comes from some other ip which idk how to detect and mangle.
lol its not useless, you would either.

1. mark all ip address that *googlevideo.com is using and add address to address-list. use queue tree as well to limit bandwidth

2. mark that packets and use queue tree to limit the bandwidth
But its NOT working, simple as that.

add action=add-dst-to-address-list address-list=YOUTUBE_SERVER address-list-timeout=12h chain=prerouting protocol=tcp tls-host=*.googlevideo.com
It picks some IPs but they are not the video stream im recieving from youtube
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 1:59 pm

I also tried implementing you tube Traffic control via this and its absolutely not working.

TSL host thing is totally useless in this case and doesnt pick actual IP of video stream *.googlevideo.com *.youtube.com give me about 4 ip to my address list, but when i start youtube video it comes from some other ip which idk how to detect and mangle.
lol its not useless, you would either.

1. mark all ip address that *googlevideo.com is using and add address to address-list. use queue tree as well to limit bandwidth

2. mark that packets and use queue tree to limit the bandwidth
But its NOT working, simple as that.

add action=add-dst-to-address-list address-list=YOUTUBE_SERVER address-list-timeout=12h chain=prerouting protocol=tcp tls-host=*.googlevideo.com
It picks some IPs but they are not the video stream im recieving from youtube
it should be the first rule in your mangle. mine working perfect.
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:16 pm

I also tried implementing you tube Traffic control via this and its absolutely not working.

TSL host thing is totally useless in this case and doesnt pick actual IP of video stream *.googlevideo.com *.youtube.com give me about 4 ip to my address list, but when i start youtube video it comes from some other ip which idk how to detect and mangle.
lol its not useless, you would either.

1. mark all ip address that *googlevideo.com is using and add address to address-list. use queue tree as well to limit bandwidth

2. mark that packets and use queue tree to limit the bandwidth
But its NOT working, simple as that.

add action=add-dst-to-address-list address-list=YOUTUBE_SERVER address-list-timeout=12h chain=prerouting protocol=tcp tls-host=*.googlevideo.com
It picks some IPs but they are not the video stream im recieving from youtube
it should be the first rule in your mangle. mine working perfect.
It is first rule, and by now it has 26 ip addresses in address list, still none of them match the one im getting video from
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:17 pm

Maybe google is using and additional dns structure. What ip's are being streamed from? which doman is that? You can contribute to the thread.
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:34 pm

Maybe google is using and additional dns structure. What ip's are being streamed from? which doman is that? You can contribute to the thread.
I figured it, its streaming it over UDP actualy for me, i had TCP protocol as TLS matcher requires it and this of course didint work for me.

I added this script to my scheduler which pull googlevideo dns records and adds them to adress-list and now it works perfect.
:local myServers { "googlevideo" }
:local myListName "YoutubeServers"
:local myTimeout "12:00:00"

/ip dns cache all {
    :foreach i in=$myServers do={
        :foreach j in=[find where (name~$i)] do={
            :do {
                :local myName [get $j name]
                :local myType [get $j type]
                :local myData [get $j data]

                :if ($myType = "A") do={
                    :do {
                        /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                    } on-error={
                        :put "DNSCACHE: Error on $myName $myData $myType"
                    }
                }

                :if ($myType = "CNAME") do={
                    :local currentName $j
                    :local nextName [find where (name=$myData && type="CNAME")]

                    :while ($nextName != "") do={
                        :set currentName $nextName
                        :resolve $currentName
                        :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                    }

                    :resolve [get $currentName data]

                    :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                        :set myData [get $k data]
                        :do {
                            /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                        } on-error={
                            :put "DNSCACHE: Error on $myName $myData $myType"
                        }
                    }
                }
            } on-error={
                :put "DNSCACHE: Missing Entry"
            }
        }
    }
}
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:37 pm

I should state that I am still working on the best code for this and please find it below.
It does not address perhaps catching the traffic but it was an effort for more efficiency and more accurate setup.

Give this a shot and see if it yields better results...........
/ip firewall mangle
{youtube}
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.googlevideo.com

add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.youtube.com

{netflix}
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.netflix.com

{primevideo}
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.primevideo.com

{spotify}
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.spotify.com

Second step:
/ip firewall mangle
add action=add-dst-to-address-list address-list=Streaming_users chain=moveTOstep2\
address-list-timeout=12h passthrough=yes
add action=mark-connection chain=moveTOstep2 dst-address-list=Streaming_users in-interface-list=lan\
connection-mark=no-mark new-connection-mark=markStreamers passthrough=yes
add action=mark-routing chain=moveTOstep2 connection-mark=markStreamers\
new-routing-mark=routeStreamers passthrough=no
Last edited by anav on Tue Jan 22, 2019 3:14 pm, edited 1 time in total.
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:39 pm

yeah good stuff, i noticed that when you are using Mobile app, it uses UDP 443 instead of TCP.

For desktop, i believe that google QUIC protocol is disabled by default, hence should work with TCP. (in where tls-host only works)
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 12319
Joined: Thu Mar 03, 2016 10:23 pm

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:45 pm

{primevideo}
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.primevideo.com.com
Isn't this one dot-com (bubble) too many?

Or is primevideo.com (single .com) from completely unrelated can of worms?
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 2:52 pm

yeah good stuff, i noticed that when you are using Mobile app, it uses UDP 443 instead of TCP.

For desktop, i believe that google QUIC protocol is disabled by default, hence should work with TCP. (in where tls-host only works)
It streams to my Windows 10 PC (Chrome) in UDP protocol also.

The above script works fine, only small issue is that it needs to run in loop and there might be some delay before it starts catching new IP depending on loop time you set in scheduler.I set it to 1min now.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 3:15 pm

Thanks mkx, fixed the typo.
What is different from what your (ivicask) script does compared to what the code snippet I have does?
Can it be incorporated somehow if I am missing something?

Also based on the above should we be adding an UDP rule for each??
[Edit: since posting it has been clarified that this does not work for UDP and the affected additions are italicized because strike out seems to be missing from this forum]

/ip firewall mangle
{youtube}
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.googlevideo.com
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=udp tls-host=*.googlevideo.com


add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.youtube.com
add chain=prerouting action=jump jump-target=moveTOstep 2 dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=udp tls-host=*.youtube.com
Last edited by anav on Tue Jan 22, 2019 5:12 pm, edited 1 time in total.
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 3:31 pm

@anav, tls-host only works for TCP, you should use ivicask script to read googlevideo.com dns from catch and write it to address list
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 3:36 pm

@anav, tls-host only works for TCP, you should use ivicask script to read googlevideo.com dns from catch and write it to address list
Thats basically what OP @mladen074 did but in simpler script, i actually jumped to lasts posts and missed the first post from him :)

Not sure which one is better if any difference tho.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 5:10 pm

Okay so let me get this straight,
A. Use script to catch UDP traffic going to those sites (create address list)
B. Use my borrowed code with latest changes to catch TCP traffic going to those sites (create address list)
C. All Jump to my last two rules which can be used to mangle address list traffic (Mark and route OR Mark and drop OR etc..............)

In other words looking for a strategy to solve this sticky business.

I also dont understand how the script works :-(
Since it involves DNS stuff, I get lost and fast.
Its especially intimidating because there are many ways for people to setup and use their DNS I am afraid the script will bugger up configurations or not work.

Put in another way.
a. what are the inputs to the script?
b. what DNS sturcture has to be in place for the script to work (dynamic, static, special firewall rules, ????)
c. what are the outputs?

For example if the output is simply an address list and I call this address list Streaming_users, then my last two rules can be applied and it should work??

The difference I see is that the Mangle is happening in prerouting but I am not sure if the script is doing the same and HOW does one interleave the two????
 
pegasus123
Frequent Visitor
Frequent Visitor
Posts: 58
Joined: Tue Jul 24, 2018 7:02 am

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 6:06 pm

Okay so let me get this straight,
A. Use script to catch UDP traffic going to those sites (create address list)
B. Use my borrowed code with latest changes to catch TCP traffic going to those sites (create address list)
C. All Jump to my last two rules which can be used to mangle address list traffic (Mark and route OR Mark and drop OR etc..............)

In other words looking for a strategy to solve this sticky business.

I also dont understand how the script works :-(
Since it involves DNS stuff, I get lost and fast.
Its especially intimidating because there are many ways for people to setup and use their DNS I am afraid the script will bugger up configurations or not work.

Put in another way.
a. what are the inputs to the script?
b. what DNS sturcture has to be in place for the script to work (dynamic, static, special firewall rules, ????)
c. what are the outputs?

For example if the output is simply an address list and I call this address list Streaming_users, then my last two rules can be applied and it should work??

The difference I see is that the Mangle is happening in prerouting but I am not sure if the script is doing the same and HOW does one interleave the two????

a. what are the inputs to the script? - no inputs necessary 3 variables are set
:local myServers { "googlevideo" } - look for *googlevideo* in /ip dns cache
:local myListName "YoutubeServers" - add all ip address to address list "YoutubeServers"
:local myTimeout "12:00:00" - TTL timeout of 12hours (it will auto delete in 12 hours)

b. what DNS sturcture has to be in place for the script to work (dynamic, static, special firewall rules, ????) - i do not know whether this script will work if you are not using your router as a DNS server (Allow remote requests). look at /ip dns cache

c. what are the outputs? - in ip firewall address list, you should see "YoutubeServers" if working


This is a quick way to find google servers. This means that the address only only contains googlevideo servers hence, you just need to mark the source/destination addresses on mangle.
You do not even need to use tls-host anymore.

Now the script can be set to run every 1 min, 5 mins, 10 mins. Just know that you will probably experience delay in marking youtube packets due to the servers have to wait to be added to address list
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 6:45 pm

Hmmm Okay, so that sounds promising.
However what you are telling me is that initial traffic will ALWAYS get out and not be rerouted because its done in real time not prerouting.
Also, the script is not timed to user access but to a rote timing scheme that will run regardless if streaming is done (while sleeping and or at work/school etc).

What if.........
A. USE TLS HOST as another determinant to run the script.

A. Run script every 5 min
B. Run script everytime any of my tls host rules get a hit.

IM assuming that prime video or netflix or whatever connections are initially done on TCP 443 and then switch to UDP for the actual streaming.
Therefore if we can nail that first bit, then we can quickly invoke the script and be more efficient?

Thoughts?
 
ivicask
Member
Member
Posts: 434
Joined: Tue Jul 07, 2015 2:40 pm
Location: Croatia, Zagreb

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 7:54 pm

Hmmm Okay, so that sounds promising.
However what you are telling me is that initial traffic will ALWAYS get out and not be rerouted because its done in real time not prerouting.
Also, the script is not timed to user access but to a rote timing scheme that will run regardless if streaming is done (while sleeping and or at work/school etc).

What if.........
A. USE TLS HOST as another determinant to run the script.

A. Run script every 5 min
B. Run script everytime any of my tls host rules get a hit.

IM assuming that prime video or netflix or whatever connections are initially done on TCP 443 and then switch to UDP for the actual streaming.
Therefore if we can nail that first bit, then we can quickly invoke the script and be more efficient?

Thoughts?
You can use TSL + this script + anything else you want, simple make them fill the same youtube adress list and thats the best result you can get.

U can also set script to lower time like 5 sec, if router isnt under high load or has large amount of cached entries it shouldnt be problem, or simple test during usual high loads if CPU spikes when u run the script.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 9:33 pm

Firstly this question or assumption was not answered...
If my assumption that prime video or netflix or whatever connections are initially done on TCP 443 and then switch to UDP for the actual streaming.
Can we use that to invoke the script and be more efficient?

Secondly,
How would I use the TLS detection of youtube.com tcp traffic, to initiate the script you made to run??
I assume its create new script that if detect run script 'ivicask'.

Finally, there is something called quicc traffic or something (possibly just a fancy way of saying UDP) that we can exploit in our rules.
I was trolling MUMs and came across this

QUIC runs on UDP and runs encrypted BUT before settling.....
uses 'ClientHello' that has a parameter called "SERVER NAME" (as tls).

The associated wireshark LINE contains a standard time/source/destination etc and the protocol is QUIC and
the info part is as follows: Client Hello, PKN: 1, CID: xxxxxxxxxxxxxxx

The blowup of that line (the more detailed look) contains the amplifying information.
Tag: CHLO (Client Hello)
Tag number: 26
padding: 0000
Tag/value: PAQ
Tag/value: SNI (Server Name Indication) (l=14): www.google.com
Tag/value: STK

So how do we make use of this interesting information for UDP based traffic??????
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 9:37 pm

I believe that at this moment it's only possible through L7 matching or maybe content matching (?), unless MT provides that as a option, like they did for TCP.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Tue Jan 22, 2019 10:26 pm

https://en.wikipedia.org/wiki/HTTP/3

Well MT better get on with it!

QUIC aims to be nearly equivalent to a TCP connection but with much-reduced latency. It does this primarily through two changes that rely on the understanding of the behaviour of HTTP traffic.[12]

The first change is to greatly reduce overhead during connection setup. As most HTTP connections will demand TLS and compression, QUIC makes the exchange of setup keys and supported protocols part of the initial handshake process. When a client opens a connection, the response packet includes the data needed for future packets to use the encryption or any desired compression. This eliminates the need to set up the TCP connection and then negotiate the transmission protocols via additional packets. Other protocols can be serviced in the same way, combining together multiple steps into a single request-response. This data can then be used both for following requests in the initial setup, as well as future requests that would otherwise be negotiated as separate connections.[12]

Obviously we are EFFED if thinking TCP is going to do anything for us. :-(

Before I jump off the bridge, are you saying that we can still catch all the traffic via the dns cache method of ivicasks script??
What I am not understand being a nooblet is if DNS chatter is also encrypted - can a destination address be parsed from such traffic??

Finally, if nothing else, why not just simply in pre-routing detect any traffic using UDP and port 443?
 
microtikgroup
newbie
Posts: 36
Joined: Mon Aug 23, 2021 11:03 am
Location: SF CA or BGC PH
Contact:

Re: Mark the traffic for YouTube, Facebook, etc.

Thu Oct 28, 2021 12:06 pm

Maybe google is using and additional dns structure. What ip's are being streamed from? which doman is that? You can contribute to the thread.
I figured it, its streaming it over UDP actualy for me, i had TCP protocol as TLS matcher requires it and this of course didint work for me.

I added this script to my scheduler which pull googlevideo dns records and adds them to adress-list and now it works perfect.
:local myServers { "googlevideo" }
:local myListName "YoutubeServers"
:local myTimeout "12:00:00"

/ip dns cache all {
    :foreach i in=$myServers do={
        :foreach j in=[find where (name~$i)] do={
            :do {
                :local myName [get $j name]
                :local myType [get $j type]
                :local myData [get $j data]

                :if ($myType = "A") do={
                    :do {
                        /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                    } on-error={
                        :put "DNSCACHE: Error on $myName $myData $myType"
                    }
                }

                :if ($myType = "CNAME") do={
                    :local currentName $j
                    :local nextName [find where (name=$myData && type="CNAME")]

                    :while ($nextName != "") do={
                        :set currentName $nextName
                        :resolve $currentName
                        :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                    }

                    :resolve [get $currentName data]

                    :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                        :set myData [get $k data]
                        :do {
                            /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                        } on-error={
                            :put "DNSCACHE: Error on $myName $myData $myType"
                        }
                    }
                }
            } on-error={
                :put "DNSCACHE: Missing Entry"
            }
        }
    }
}
what about Netflix among others. can you just lump sum all of them in one place and fire that scheduler?

Who is online

Users browsing this forum: Bing [Bot], kraken and 49 guests