Community discussions

MikroTik App
 
Peter0
just joined
Topic Author
Posts: 3
Joined: Sat Jan 19, 2019 7:49 pm

URL Based Load Balancing

Sat Jan 19, 2019 11:03 pm

Good Evening

First forum post. Go easy. I am looking for some advice on whether what I am attempting to achieve is feasible. Additionally, pointers to any resources / guidance that might help me create my desired solution.

After looking at various devices I might consider, I am settling on using a Mikrotik hEX (RB750GR3)

I live in a rural location with poor fixed line broadband. However, I have also installed faster, but bandwidth limited 4G based mobile broadband. I also have children who like the faster 4G based broadband and so consume the available 4G based bandwidth streaming funny cat videos ;-)

I am looking to install a WAN router of some sort. 2 ports minimum, 3 would be useful. The router will need to support URL Policy Based Routing to send traffic to appropriate broadband connections. What I think I need to achieve is something like:

ADSL Broadband --------> | ....................................................................................................................... | User 1
..........................................|........................................................................................................................ | User 2
..........................................| <---------- Mikrotik hEX (RB750GR3) ---------> | <-------------- Deco M5 -------- | User 3
4G Broadband ------------> | ................................................................................ (Wireless Meshing).......| User 4
..........................................|........................................................................................................................ | User 5
ANO Broadband -------- > |........................................................................................................................ | User 6, User Etc, Etc

I want to be able allow all users on the right hand side to use the faster but usage capped 4G Broadband by default and route certain types of high bandwidth demanding traffic over the slower but uncapped ADSL broadband. For example, all YouTube, Netflix, Prime Video, Spotify traffic should always go via the ADSL Broadband. N.B. I have identified a possible other 4G connection that can access UK Netflix traffic at zero bandwidth cost.

Additionally, it would be useful to route Users who exceed a "fair use" bandwidth to the ADSL Broadband until the next day/week. Basically to manage the children’s online activities when they get out of hand. This is a less important requirement.

I note from various articles and discussions here and elsewhere that attempting to shape traffic based on URL's is difficult and constantly changes. Notwithstanding these challenges, I would like to have a go. I think some variation of these two articles is what I need to do:

https://wifitechtalk.com/mikrotik-youtube-throttling/

https://wiki.mikrotik.com/wiki/Per-Traf ... _Balancing

The first article I thinks shows a way of creating a self-propagating list of IP addresses used by Netflix / Youtube for video streaming.?

The second I think shows a way of using particular routes based on the traffic you marked with a specific tag/address list?

Before I rush off and but my Mikrotik hEX (RB750GR3), is what I am attempting achievable? And, am I looking in the right place for configuration ideas?

Many thanks
Peter
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19379
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: URL Based Load Balancing

Sun Jan 20, 2019 7:27 pm

I think its possible, in the sense that
a. the hex can accommodate multiple wan 5 ports (3 wans 2 port for lan)
b. the hex can identify traffic and route it to a specific lan
c. beyond my scope is your bandwidth allocation cap and resulting change to routing but I would guess its possible.
I wil see what I can dream up here for my fellow git, I mean Brit LOL (me-->colonial Canadian)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Context: Ensure those using Youtube. netflix etc are sent do a secondary WAN connection (ADSL) where 4G is the primary.

In order to properly identify traffic heading for youtube etc. probably requires manglink/marking of some sort. Its a two step process in general.
One is to mark connections to youtube, then two, you attach a route marking rule to that traffic (marking connections is more efficient in that one marks the connection and all packets are then
Thankfully MT now has a tls-host option to help with the fact that most traffic is now https!
I dont think we need to separate out the route rule for different stream, so all marked connections will be using same route rule.

First step will be to identify find users who are attempting to access youtube etc... and add the destination addresses to a firewall address list.
/ip firewall mangle
{youtube}
add action=add-dst-to-address-list address-list=Streaming_users address-list-timeout=\
12h chain=prerouting dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.googlevideo.com passthrough=yes

add action=add-dst-to-address-list address-list=Streaming_users address-list-timeout=\
12h chain=prerouting dst-port=443 interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.youtube.com passthrough=yes

{netflix}
add action=add-dst-to-address-list address-list=Streaming_users address-list-timeout=\
12h chain=prerouting dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.netflix.com passthrough=yes

{primevideo} Note that prime music may be harder and in that case would have to force all amazon.com or amazon.uk traffic to asdl)
add action=add-dst-to-address-list address-list=Streaminge_users address-list-timeout=\
12h chain=prerouting dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.primevideo.com.com passthrough=yes

{spotify}
add action=add-dst-to-address-list address-list=Streaming_users address-list-timeout=\
12h chain=prerouting dst-port=443 in-interface-list=lan\
connection-mark=no-mark protocol=tcp tls-host=*.spotify.com passthrough=yes

Second step, for all identified traffic apply a connection mark and a routing mark to any traffic with said list.
/ip firewall mangle
add action=mark-connection chain=prerouting dst-address-list=Streaming_users in-interface-list=lan\
connection-mark=no-mark new-connection-mark=markStreamers passthrough=yes
add action=mark-routing chain=prerouting connection-mark=markStreamers\
new-routing-mark=routeStreamers passthrough=no

In plain english, the router will look at packets without any markings, ie this is done before sending traffic anywhere (not routed or going anywhere direct) but just showing up at any interface covered by the "in-interface-list". If the packets have no markings and has a destination port of 443 (and has a domain identification of youtube.com for example) the associated destination address of the connection is added to the applicable address list. The we say step to the next rule............
The next rule states, if the packets has no markings and has a destination port of 443 (and has a domain identification of googlevideo.com for example) the associated destination address of the connection is added to the applicable address list. The we say step to the next rule............ and so on.........

After identifying all possible addresses, the router will then look for any connections intended for the destination address from our created list (identified by our now populated firewall address rule). We set it up such that that the packet being inspected is the first in a new connection without any connection markings because we want the router to only look at new connections (and ignore already marked traffic).
The router will mark the connection with the mark (markStreamer) and any of its associated packets will not be inspected and will automatically receive the temporary mark.
Then you tell the router look at the next prerouting rule for this connection
The router then sees that for any connection (and associated packets) with the connection mark of markStreamer attach another kind of mark, a routing mark routeStreamer to that connection (and its associated packets). Then you tell the router basically there are no more marking rules that apply.

So how do we use this special routing mark in IP Route rules.
We make use of the Routing mark option available (see winbox).

++++++++++++++++++++++++++++++++++++++++++++

Simple example failover:
/ip route
add check-gateway=ping distance=2 gateway=Gateway IP of WAN2 (4G connection)
add distance=3 gateway=Gateway IP of WAN1 (ADSL)
add distance=5 gateway=Gateway IP of WAN3 (not sure what other WANs you have or how they will get used so just a place marker for now)
add distance=1 gateway=Gateway IP of WAN1 routing-mark=routeStreamer

Complex example recursive failover
/ip route
add check-gateway=ping distance=2 gateway=8.8.4.4
add distance=2 dst-address=8.8.4.4/32 gateway=GatewayIP of WAN2 (Your 4G connection)
add distance=3 gateway=Gateway IP of WAN1 (Your ADSL connection)
add distance=1 gateway=Gateway IP of WAN1 routing-mark=routeStreamer

A number of things not sure of in this approach.

1. Do I Need to state distance=1 to ensure users that connect to youtube etc. go out the secondary WAN or is the mark rule enough??
2. Should I use no-mark packets for the first mangle rule or is sticking with no-mark connections more efficient/optimal/accurate??
3. Should I be using jump chain rules here? I'm not sure if I have used pass through correctly??
Last edited by anav on Sun Jan 20, 2019 8:14 pm, edited 2 times in total.
 
Peter0
just joined
Topic Author
Posts: 3
Joined: Sat Jan 19, 2019 7:49 pm

Re: URL Based Load Balancing

Sun Jan 20, 2019 7:58 pm

I wil see what I can dream up here for my fellow git, I mean Brit LOL (me-->colonial Canadian)
Thank you. Though if we are going to be "Colonial" about it I should fess up to being Irish but residing pretty much all my life across the Irish Sea. I blame the parents, they got wanderlust when I was but a baby;-)

Cheers
Peter
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19379
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: URL Based Load Balancing

Sun Jan 20, 2019 8:15 pm

Hahaha, sorry for the cliche, but do you wake up every morning and have a bowl of Lucky Charms and say, Top of the Morning to ya? ;-P
 
Peter0
just joined
Topic Author
Posts: 3
Joined: Sat Jan 19, 2019 7:49 pm

Re: URL Based Load Balancing

Mon Jan 21, 2019 1:26 am

Hahaha, sorry for the cliche, but do you wake up every morning and have a bowl of Lucky Charms and say, Top of the Morning to ya? ;-P
Thank you ever so much for your advice. I will be sure to start here once my hEX arrives.

And Lucky Charms and Top of the Morning? I grew up in England, think I'm English, act English. ;-)

Cheers
Peter

Who is online

Users browsing this forum: broderick, fxcd, jaclaz, zalciukaz and 116 guests