Auto Throttle During Congestion

ccmks · Fri Feb 19, 2021 4:28 pm

Hello,

I am not sure if this can be achieve.

I am living in one of the developing country where bandwidth is a very precious thing there. My total bandwidth is 60 Mbps, so during night time is the most congested period and it is always hitting its bandwidth capacity. After doing some analysis on traffic, I found out majority of my users are using Youtube, Facebook and Instagram. The bandwidth is wasted on them watching the content through smartphone and sometimes, they ended up fell asleep while keep the content played and hogging the traffic for others. My current setup is to limit the traffic that my majority user use which are Youtube, Facebook and Instagram to 1,5 Mbps per device despite of having higher bandwidth assigned.

This setup is working so far, but I would like to make it better by only enforcing the traffic restriction mentioned above during congested period as right now the restriction is on all the time. I need this to be dynamically trigger restriction on if the total bandwidth is reaching above 40 Mbps in last 5 minutes, and trigger restriction off if it is less than 40 Mbps in last 5 minutes (I am not sure if this is the best parameter to trigger restriction, I am open to any suggestion if you have better one). Basically I still want the user to enjoy high resolution streaming during non-congested period and throttle the streaming to 1,5 Mbps during congested period.

How can I achieve that?

Thank you for your answer.

ccmks · Sat Feb 20, 2021 3:02 pm

Anyone?

sindy · Sat Feb 20, 2021 3:23 pm

The answer is scripting. In my opinion, it is easier to use hysteresis rather than to calculate 5 minute traffic averages - configure two different threshold values far enough from each other for activation and deactivation of the bandwidth throttling. Let's say activate it if traffic exceeds 45 Mbit/s and deactivate it if it falls below 35 Mbit/s.

So you can schedule a simple script to run every minute, which will compare the current download bandwidth on the uplink port to the two thresholds depending on the presence of a dynamic address-list item. If the address list item doesn't exist, which indicates that throttling is inactive, and the threshold value for its activation is exceeded, the script will create the address list item and activate the throttling; if the address list item exists and the threshold value for its deactivation is undershot, the script will remove the address list item and deactivate the throttling.

The current download speed in bits per second can be obtained as ([/interface monitor-traffic wan-interface-name once as-value]->"rx-bits-per-second") .

The address-list item used to indicate the throttling state can actually directly control the throttling if you use mangle rule matching on that address-list to set a packet-mark value used for strong throttling, whereas a subsequent rule matching on packet-mark=no-mark will set the packet-mark for light throttling, and the address value of the address-list item will be set 0.0.0.0/0. If you create that address-list item with a lifetime of some hours, the configuration file will not be re-written to the flash on every change.

ccmks · Sat Feb 20, 2021 8:35 pm

Thank you for answering. I don't know how to make script.

Can you tell me how to make one or give me the script example? I am not good at making script.

Thank you

sindy · Sat Feb 20, 2021 9:52 pm

Copy-paste the following code to a text editor, replace wan-interface-name by the actual name of your WAN interface, and then copy-paste the result into the command line window of the Mikrotik:

/system script
add name=throttle source=":local onThreshold 45000000\
    \n:local offThreshold 35000000\
    \n:local isOn ([:len [ip firewall address-list find where list=throttle]] > 0)\
    \n:local wanRate ([/interface monitor-traffic wan-interface-name once as-value]->\"rx-bits-per-second\")\
    \n:if \$isOn do={\
    \n  if (\$wanRate<\$offThreshold) do={\
    \n    /ip firewall address-list remove [find where list=throttle]\
    \n  }\
    \n} else={\
    \n  if (\$wanRate>\$onThreshold) do={\
    \n    /ip firewall address-list add list=throttle address=0.0.0.0/0 timeout=3h\
    \n  }\
    \n}\
    \n\
    \n"
    
/system scheduler
add interval=1m name=throttle on-event="/system script run throttle" start-time=startup

You can then edit the threshold values to reach your goal, and modify the mangle rules the way I've suggested in my previous post. Without the mangle rules, the script will have no effect but creation and removal of the address list row depending on download bitrate.

ccmks · Sun Feb 21, 2021 2:17 pm

Thank you for providing me the script.

If we just checking on bandwidth usage on interface periodically every 1 minutes, would you think that it would trigger the throttle too early? What I mean is when the script is checking the interface bandwidth and the interface might happen to reach the threshold briefly for only 1 second then the throttle triggered. What do you think if it calculates the average of 5 minutes of bandwidth usage?

Also, can I have different version of the script which action is to disable or enable the queue tree rule I made during throttle instead of using address-list? I would like to compare how it works.

Thank you

sindy · Sun Feb 21, 2021 3:36 pm

What do you think if it calculates the average of 5 minutes of bandwidth usage?

Averaging definitely helps suppress overreaction, but is an overreaction which activates the limitation for just one minute really so important that the code needs to be made more complex (and thus a coding mistake more likely)? Besides, calculating an average of 5 samples during 5 minutes would probably be much less effective as a filter than calculating an average of, say, 55 samples during 55 seconds. Plus I consider safer to let the scheduler start a one-time script every minute than to let it start the script running in an endless loop just once at startup, so you'd have to use global variables to let the individual runs of the script inherit information from the previous ones.

So if you want to somewhat filter out random peaks of traffic, replace the single line

:local wanRate ([/interface monitor-traffic wan-interface-name once as-value]->"rx-bits-per-second")

in the original script by the following lines:

:local wanRate 0
:for i from=0 to=54 do={
  :set wanRate (wanRate+([/interface monitor-traffic wan-interface-name once as-value]->"rx-bits-per-second"))
  :delay 1s
}
:set wanRate (wanRate/55)

Also, can I have different version of the script which action is to disable or enable the queue tree rule I made during throttle instead of using address-list? I would like to compare how it works.

I was assuming that the queue in the tree matches on some packet-mark value, hence not assigning that packet-mark value in mangle (because the mangle rule assigning the packet-mark doesn't match if the address-list it refers to is empty) effectively inactivates the queue. But if you want to disable and enable the queue in configuration, causing the config file to be written again with each change, just add the corresponding commands to the script right after the ones creating and removing the address list item:

/ip firewall address-list remove [find where list=throttle]
/queue tree disable name-of-the-queue

and

/ip firewall address-list add list=throttle address=0.0.0.0/0 timeout=3h
/queue tree enable name-of-the-queue

ccmks · Sun Feb 21, 2021 3:58 pm

I was assuming that the queue in the tree matches on some packet-mark value, hence not assigning that packet-mark value in mangle (because the mangle rule assigning the packet-mark doesn't match if the address-list it refers to is empty) effectively inactivates the queue. But if you want to disable and enable the queue in configuration, causing the config file to be written again with each change, just add the corresponding commands to the script right after the ones creating and removing the address list item:

I currently have the rule throttling enforce already, but it is on all the time regardless the network is congested or not, which means that I have the packet mark in the queue against client IP already. I just would like to make it simpler by just eliminating the action to enable or disable IP address-list at this moment since I have setup pre-defined client to have packet marked on certain ip address-list. Just need the queue to be disable or enable when the threshold is either reached or below. I see your point now on not having to calculate average on 5 mins. Will try to use your advice on checking the interface every one minute.

I am kinda confused on replacing the script below, can you show me the whole script which reflecting this version (including the disable and enable queue and filter random threshold)?

Again, scripting is not my forte, and I am sorry about it.

I appreciate your time, effort and patience dealing with my ignorant request.

Thank you

sindy · Sun Feb 21, 2021 4:20 pm

First remove the existing scheduler and script, and then repeat the whole exercise again, this time replacing wan-interface-name (once) and name-of-the-queue (twice).

/system script
add name=throttle source=":local onThreshold 45000000\
    \n:local offThreshold 35000000\
    \n:local isOn ([:len [ip firewall address-list find where list=throttle]] > 0)\
    \n:local wanRate 0\
    \n:for i from=0 to=54 do={\
    \n  :set wanRate (\$wanRate+([/interface monitor-traffic wan-interface-name once as-value]->\"rx-bits-per-second\"))\
    \n}\
    \n:set wanRate (\$wanRate/55)\
    \n:if \$isOn do={\
    \n  if (\$wanRate<\$offThreshold) do={\
    \n    /ip firewall address-list remove [find where list=throttle]\
    \n    /queue tree enable name-of-the-queue\
    \n  }\
    \n} else={\
    \n  if (\$wanRate>\$onThreshold) do={\
    \n    /ip firewall address-list add list=throttle address=0.0.0.0/0 timeout=3h\
    \n    /queue tree disable name-of-the-queue\
    \n  }\
    \n}\
    \n\
    \n"
    
/system scheduler
add interval=1m name=throttle on-event="/system script run throttle" start-time=startup

But still, even if you use both src-address-list and dst-address-list in the mangle rules assigning the packet-mark, you can do the following instead of disabling and enabling the rule:

before:
chain=xxx src-address-list=listA dst-address-list=listB (...) action=mark-packet new-packet-mark=throttle-queue

after:
chain=xxx src-address-list=!throttle action=mark-packet new-packet-mark=shadow
chain=xxx src-address-list=listA dst-address-list=listB (...) packet-mark=no-mark action=mark-packet new-packet-mark=throttle-queue

ccmks · Sun Feb 21, 2021 4:52 pm

Is this script look good?

/system script
add name=throttle source=":local onThreshold 45000000\
    \n:local offThreshold 40000000\
    \n:local wanRate 0\
    \n:for i from=0 to=54 do={\
    \n  :set wanRate (\$wanRate+([/interface monitor-traffic LAN once as-value]->\"rx-bits-per-second\"))\
    \n}\
    \n:set wanRate (\$wanRate/55)\
    \n:if \$isOn do={\
    \n  if (\$wanRate<\$offThreshold) do={\
    \n    /queue tree disable youtube-non-VIP\
	\n    /queue tree disable Facebook-non-VIP\
    \n  }\
    \n} else={\
    \n  if (\$wanRate>\$onThreshold) do={\
 	\n    /queue tree enable youtube-non-VIP\
	\n    /queue tree enable Facebook-non-VIP\
    \n  }\
    \n}\
    \n\
    \n"

Note:
youtube-non-VIP and facebook-non-VIP are the queue to limit traffic to respective site. If it is enabled, it means it will throttle the traffic to it, otherwise it will not throttle.

I removed the address-list part as I just need the script to enable or disable the queue I made when the trigger happens

I tried applying the script and try to run manually without schedule it first. But it is not working.

sindy · Sun Feb 21, 2021 6:19 pm

Of course it is not working. The address list item in my script was also playing a role of a state variable, from which the isOn was derived. So once you've removed it, isOn is undefined, so the if now probably always takes the else path.

So replace the $isOn by (!([/queue tree get youtube-non-VIP disabled])).

ccmks · Mon Feb 22, 2021 3:59 am

/system script
add name=throttle source=":local onThreshold 45000000\
    \n:local offThreshold 40000000\
    \n:local wanRate 0\
    \n:for i from=0 to=54 do={\
    \n  :set wanRate (\$wanRate+([/interface monitor-traffic CCARENET once as-value]->\"rx-bits-per-second\"))\
    \n}\
    \n:set wanRate (\$wanRate/55)\
    \n:if (!([/queue tree get youtube-non-VIP disabled])) do={\
    \n  if (\$wanRate<\$offThreshold) do={\
    \n    /queue tree disable youtube-non-VIP\
	\n    /queue tree disable Facebook-non-VIP\
    \n  }\
    \n} else={\
    \n  if (\$wanRate>\$onThreshold) do={\
 	\n    /queue tree enable youtube-non-VIP\
	\n    /queue tree enable Facebook-non-VIP\
    \n  }\
    \n}\
    \n\
    \n"

I changed to this, but it only disable the queue rule regardless it is reaching on or off threshold. What could be wrong?

sindy · Mon Feb 22, 2021 12:25 pm

after the
\n:set wanRate (\$wanRate/55)\
row, add another one:
\n:put $wanRate\
to visualize the result of the throughput measurement. Then you can run the script manually and before each run, update the thresholds to see the whether it does what is expected. I did it this way and it worked for me. I've also double-checked now that the boolean expression (!([/queue tree get youtube-non-VIP disabled])) works properly.

ccmks · Tue Feb 23, 2021 3:34 pm

Thanks for helping me out. I couldn't get it to work, for some reason, it only read the first if condition and action (in this case, it disable the queue).

What do you think if we separate into 2 scripts and run at same time and same interval? Only 1 script will be applied no matter which condition currently happening

Thank you

sindy · Tue Feb 23, 2021 4:07 pm

Thanks for helping me out. I couldn't get it to work, for some reason, it only read the first if condition and action (in this case, it disable the queue).

This looks to me as if the thresholds were not matching the actual speeds and it always seemed that no throttling is required.

I have imported the script you've posted above and modified the interface name and queue names.

I don't know how it happened but the :delay 1s fell out of the script I've posted before, so the average was calculated from 55 samples taken in the same second or two subsequent ones at best. So maybe you were "lucky" to always hit a second where the traffic was low, no idea.

Are you normally using Winbox or command line? Editing an already imported script is possible, and the export format of the script is hard to read, so using the format shown in edit window when telling you what to change would be better.

On command line, you can use
/system script edit throttle source
In WinBox, the script is editable in the GUI but the formatting is terribly broken at least in v3.20, so even there I'd recommend to open the command line window by pressing [New Terminal] and running the above command in the terminal window. Maybe that will cure your scriptophobia :)

So first, change

:for i from=0 to=54 do={
  :set wanRate ($wanRate+([/interface monitor-traffic CCARENET once as-value]->"rx-bits-per-second"))
}

into

:for i from=0 to=54 do={
  :set wanRate ($wanRate+([/interface monitor-traffic CCARENET once as-value]->"rx-bits-per-second"))
  :delay 1s
}

and try again (don't be surprised that now the script will be silent for 55 seconds). If it is not enough, change

:set wanRate ($wanRate/55)

into

:set wanRate ($wanRate/55)
:put $wanRate

and run the script again, to see what the calculated average actually is.

What do you think if we separate into 2 scripts and run at same time and same interval? Only 1 script will be applied no matter which condition currently happening

Doing this won't help anything, as the issue is not in processing the if ... do{} ... else {} as you probably suspect but in the actual traffic values and the thresholds - I've checked again that the logic of the script can handle both directions. So splitting the same logic with the same threshold values into two scripts cannot change anything about the outcome.

Auto Throttle During Congestion

Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Re: Auto Throttle During Congestion

Who is online