Community discussions

MikroTik App
  4. RouterOS
  5. Beginner Basics

[ISSUE, DUAL WAN] Bandwith Management with Mangle, Prerouting and Routing Marks

Post Reply
 
User avatar
AryanProphet
just joined
Topic Author
Posts: 5
Joined: Tue May 03, 2022 1:16 pm

[ISSUE, DUAL WAN] Bandwith Management with Mangle, Prerouting and Routing Marks

Wed Nov 16, 2022 12:49 pm

Hello Everyone,

I am trying to get some bandwidth management done. I have a 50 MBit DSL on ISP_01, and a 200 Mbit metered connection on ISP_02. The logic should be the following, when the bandwidth from ISP_01 reaches 50 MBit, new connections should be routed over to ISP_02.

The scripts that update the gateways on the default routes on to_ISP1 and to_ISP2 are working perfectly. I have to use a script to update the gateways, I know the default route can be added via the DHCP client, but it does not allow me to set options on the route. The bandwidth scripts on the bandwidth monitor are working too.

The issue I am facing is:
Code: Select all
add action=mark-routing chain=prerouting comment=balancer connection-state=new disabled=yes new-routing-mark=to_ISP1 passthrough=no

When I ping 8.8.8.8, only the first packet goes through ISP_01, the second one does not... when I kill the connection on /ip/firewall/connections, again only 1 packet goes through... when I enable new and related... does also not work... so my question is... what do I have to set here so that all new connections go to ISP_01 using my config. This is the step where I am stuck for now, the next step to this would be adding a disabled mangle rule before the to_ISP1 rule that get activated by the bandwidth monitoring that sends the routing via to_ISP2. If that makes sense.

Any help is greatly appreciated. For reference, the whole config below.

Thanks!
Ben

Code: Select all
/interface bridge
add name=bridge_LAN
/interface ethernet
set [ find default-name=ether1 ] name=ISP_01
set [ find default-name=ether2 ] name=ISP_02
set [ find default-name=ether3 ] name=LAN_01
set [ find default-name=ether4 ] name=LAN_02
set [ find default-name=ether5 ] name=LAN_03
set [ find default-name=sfp1 ] disabled=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=dhcp_pool0 ranges=192.168.77.11-192.168.77.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=bridge_LAN name=dhcp1
/port
set 0 name=serial0
/routing table
add disabled=no fib name=to_ISP1
add disabled=no fib name=to_ISP2
/system logging action
set 0 memory-lines=100
/interface bridge port
add bridge=bridge_LAN interface=LAN_01
add bridge=bridge_LAN interface=LAN_02
add bridge=bridge_LAN interface=LAN_03
/ip address
add address=192.168.77.1/24 interface=bridge_LAN network=192.168.77.0
/ip dhcp-client
add add-default-route=no interface=ISP_01
add add-default-route=no interface=ISP_02
/ip dhcp-server network
add address=192.168.77.0/24 dns-server=192.168.77.1 gateway=192.168.77.1
/ip firewall mangle
add action=mark-routing chain=prerouting comment=balancer connection-state=new disabled=yes new-routing-mark=to_ISP1 passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ISP_01
add action=masquerade chain=srcnat out-interface=ISP_02
/ip route
add comment=to_ISP_01 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.81.1 pref-src="" routing-table=to_ISP1 scope=30 suppress-hw-offload=no target-scope=10
add comment=to_ISP_02 disabled=no distance=2 dst-address=0.0.0.0/0 gateway=192.168.82.1 pref-src="" routing-table=to_ISP2 scope=30 suppress-hw-offload=no target-scope=10
/system clock
set time-zone-name=Europe/Berlin
/system scheduler
add interval=1s name=to_ISP on-event="to_ISP1\r\
    \nto_ISP2" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=nov/15/2022 start-time=16:37:05
/system script
add dont-require-permissions=no name=to_ISP1 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":local newgw [ip\
    \_dhcp-client get [find interface=\"ISP_01\"] gateway];\r\
    \n:local routegw [/ip route get [find comment=\"to_ISP_01\"] gateway ];\r\
    \n:if (\$newgw != \$routegw) do={\r\
    \n         /ip route set [find comment=\"to_ISP_01\"] gateway=\$newgw;\r\
    \n}"
add dont-require-permissions=no name=to_ISP2 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":local newgw [ip\
    \_dhcp-client get [find interface=\"ISP_02\"] gateway];\r\
    \n:local routegw [/ip route get [find comment=\"to_ISP_02\"] gateway ];\r\
    \n:if (\$newgw != \$routegw) do={\r\
    \n         /ip route set [find comment=\"to_ISP_02\"] gateway=\$newgw;\r\
    \n}"
/tool traffic-monitor
add interface=ISP_01 name=LB1 on-event=":log warning \"LB Debug: ISP1 overloaded, switching to ISP2\";\r\
    \n/ip firewall mangle set [find comment=\"balancer\"] connection-mark=ISP2_conn\r\
    \n" threshold=50000000 traffic=received
add interface=ISP_01 name=LB2 on-event=\
    ":log warning \"LB Debug: ISP1 ok, switching back to ISP1\";\r\
    \n/ip firewall mangle set [find comment=\"balancer\"] connection-mark=ISP1_conn\r\
    \n" threshold=30000000 traffic=received trigger=below
Last edited by AryanProphet on Wed Nov 16, 2022 1:02 pm, edited 3 times in total.
Top
 
pe1chl
Forum Guru
Forum Guru
Posts: 10231
Joined: Mon Jun 08, 2015 12:09 pm

Re: Mangle, Prerouting, Routing Marks

Wed Nov 16, 2022 1:00 pm

You are doing the mark only for connection-state=new, which it is only for the first packet. So what you observe is correct.
Usually in situations like this (2 ISPs and using marking) it is best to first set a connection-mark on the conditions you desire (and set passthrough=yes), then in another rule match on that connection-mark and apply the routing-mark when it is there.
That way all packets of the same connection will receive the same routing-mark.
Top
 
User avatar
AryanProphet
just joined
Topic Author
Posts: 5
Joined: Tue May 03, 2022 1:16 pm

Re: [ISSUE, DUAL WAN] Bandwith Management with Mangle, Prerouting and Routing Marks

Wed Nov 16, 2022 5:24 pm

Hey pe1chl! Thanks for the hint.

I have adapted my configuration inspiring myself with: Example 3 (PCC)
https://help.mikrotik.com/docs/display/ ... ingWithNTH

What I changed on the mangle rules:
Code: Select all
/ip firewall mangle
add action=accept chain=prerouting comment=ISP_01_dst dst-address=192.168.81.0/30 in-interface=bridge_LAN
add action=accept chain=prerouting comment=ISP_02_dst dst-address=192.168.82.0/30 in-interface=bridge_LAN
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ISP_01 new-connection-mark=ISP1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=ISP_02 new-connection-mark=ISP2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge_LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes dst-address-type=!local in-interface=bridge_LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface=bridge_LAN new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2_conn disabled=yes in-interface=bridge_LAN new-routing-mark=to_ISP2 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn disabled=yes new-routing-mark=to_ISP2 passthrough=yes

I'd like to have the ISP2 mangle rules disabled until the bandwidth limit is reached.

The issues I am facing now is when I have the mangle rules for ISP2 disabled I don't have internet, and when I kill all the active connections it does not seem to go through the ISP1 mangle rules, so something still seems to be wrong. When I enable the mangle rules for ISP2, it works again... how is it decided what goes through what WAN? Why is does it not work when ISP2 mangle rules are disabled? Many question marks around my head.

The actual full configuration:
Code: Select all
/interface bridge
add name=bridge_LAN
/interface ethernet
set [ find default-name=ether1 ] name=ISP_01
set [ find default-name=ether2 ] name=ISP_02
set [ find default-name=ether3 ] name=LAN_01
set [ find default-name=ether4 ] name=LAN_02
set [ find default-name=ether5 ] name=LAN_03
set [ find default-name=sfp1 ] disabled=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=dhcp_pool0 ranges=192.168.77.11-192.168.77.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=bridge_LAN name=dhcp1
/port
set 0 name=serial0
/routing table
add disabled=no fib name=to_ISP1
add disabled=no fib name=to_ISP2
/system logging action
set 0 memory-lines=100
/interface bridge port
add bridge=bridge_LAN interface=LAN_01
add bridge=bridge_LAN interface=LAN_02
add bridge=bridge_LAN interface=LAN_03
/ip address
add address=192.168.77.1/24 interface=bridge_LAN network=192.168.77.0
/ip dhcp-client
add add-default-route=no interface=ISP_01
add add-default-route=no interface=ISP_02
/ip dhcp-server network
add address=192.168.77.0/24 dns-server=192.168.77.1 gateway=192.168.77.1
/ip firewall mangle
add action=accept chain=prerouting comment=ISP_01_dst dst-address=192.168.81.0/30 in-interface=bridge_LAN
add action=accept chain=prerouting comment=ISP_02_dst dst-address=192.168.82.0/30 in-interface=bridge_LAN
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ISP_01 new-connection-mark=ISP1_conn
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=ISP_02 new-connection-mark=ISP2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge_LAN new-connection-mark=ISP1_conn per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes dst-address-type=!local in-interface=bridge_LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface=bridge_LAN new-routing-mark=to_ISP1
add action=mark-routing chain=prerouting connection-mark=ISP2_conn disabled=yes in-interface=bridge_LAN new-routing-mark=to_ISP2 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1
add action=mark-routing chain=output connection-mark=ISP2_conn disabled=yes new-routing-mark=to_ISP2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ISP_01
add action=masquerade chain=srcnat out-interface=ISP_02
/ip route
add comment=to_ISP_01 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.81.1 pref-src="" routing-table=to_ISP1 scope=30 suppress-hw-offload=no target-scope=10
add comment=to_ISP_02 disabled=no distance=2 dst-address=0.0.0.0/0 gateway=192.168.82.1 pref-src="" routing-table=to_ISP2 scope=30 suppress-hw-offload=no target-scope=10
/system clock
set time-zone-name=Europe/Berlin
/system scheduler
add interval=1s name=to_ISP on-event="/system script run to_ISP1\r\
    \n/system script run to_ISP2" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=nov/15/2022 start-time=16:37:05
/system script
add dont-require-permissions=no name=to_ISP1 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":local newgw [ip dhcp-client get [find interface=\"ISP_01\"] gateway];\r\
    \n:local routegw [/ip route get [find comment=\"to_ISP_01\"] gateway ];\r\
    \n:local dstgw (\$newgw.\"/30\");\r\
    \n\r\
    \n:if (\$newgw != \$routegw) do={\r\
    \n         /ip route set [find comment=\"to_ISP_01\"] gateway=\$newgw;\r\
    \n         /ip firewall mangle set [find comment=\"ISP_01_dst\"] dst-address=\$dstgw;\r\
    \n}\r\
    \n"
add dont-require-permissions=no name=to_ISP2 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":local newgw [ip dhcp-client get [find interface=\"ISP_02\"] gateway];\r\
    \n:local routegw [/ip route get [find comment=\"to_ISP_02\"] gateway ];\r\
    \n:local dstgw (\$newgw.\"/30\");\r\
    \n\r\
    \n:if (\$newgw != \$routegw) do={\r\
    \n         /ip route set [find comment=\"to_ISP_02\"] gateway=\$newgw;\r\
    \n         /ip firewall mangle set [find comment=\"ISP_02_dst\"] dst-address=\$dstgw;\r\
    \n}\r\
    \n"
add dont-require-permissions=no name=clear_log owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="/system logging action\r\
    \nset 0 memory-lines=1\r\
    \nset 0 memory-lines=100"
/tool traffic-monitor
add interface=ISP_01 name=LB1 on-event=":log warning \"LB Debug: ISP1 overloaded, switching to ISP2\";\r\
    \n/ip firewall mangle set [find comment=\"balancer\"] connection-mark=ISP2_conn\r\
    \n" threshold=50000000 traffic=received
add interface=ISP_01 name=LB2 on-event=":log warning \"LB Debug: ISP1 ok, switching back to ISP1\";\r\
    \n/ip firewall mangle set [find comment=\"balancer\"] connection-mark=ISP1_conn\r\
    \n" threshold=30000000 traffic=received trigger=below
Top
 
User avatar
AryanProphet
just joined
Topic Author
Posts: 5
Joined: Tue May 03, 2022 1:16 pm

Re: [ISSUE, DUAL WAN] Bandwith Management with Mangle, Prerouting and Routing Marks

Wed Nov 16, 2022 6:40 pm

Ok, I got it working with disabling the connection classifier. :wink:
Top
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: [ISSUE, DUAL WAN] Bandwith Management with Mangle, Prerouting and Routing Marks

Wed Nov 16, 2022 6:45 pm

First, you have routes only in routing tables to_ISP1 and to_ISP2, the routing table main (consisting of routes without any routing-mark value) is empty. And in mangle, you only assign routing-mark values in chain prerouting. In summary, this means that the Mikrotik itself has no route to use for its own traffic.

At the same time, you tell the clients to use the Mikrotik itself as their DNS server, so they send a DNS query, Mikrotik looks it up in its DNS cache, finds nothing there, and needs to send its own DNS query to its upstream DNS server but it cannot, for the lack of routes. So the clients report this as "internet unavailable".

If you ping numeric IP addresses from the clients and it depends on enabling or disabling the mangle rules for ISP2 whether the pings get responded or not, that's another case, but with the same root cause. For traffic from the LAN hosts, you assign one of the two connection-mark values (which you later translate to a routing-mark) depending on the value of per-connection-classifier. Packets that match the 2/0 condition are routed via ISP 1, but packets that do not match it get no routing-mark so they don't get routed at all. If you had the routing table main, it would be used as the emergency fallback, but that's currently not the case.

For this reason and several others, your new mangle rules seem to me like a blind copy-paste. So I would suggest to let the DHCP client for ISP 1 add the default route to routing table main, remove the routing table to_ISP1 completely, and only use a single action=mark-connection rule and a single action=mark-routing rule to direct traffic via routing table to_ISP2.

I personally don't like writing to the flash unless actually necessary; if you enable and disable rules depending on used bandwidth on ISP 1, every change means a write. Instead, I would add a match condition src-address-list=enable to the action=mark-connection rule, and the bandwidth tracking scripts would add (or remove) a dynamic item to that address list. As duplicate entries are not supported, you have to use the following construct to ignore errors in script execution when adding the address:
:do command={/ip/firewall/address-list/add list=enable address=0.0.0.0/0 timeout=35w} on-error={nothing}
For removal, a plain /ip/firewall/address-list/remove [find where list=enable] is safe - if nothing is found, nothing is removed and it is not considered an error.
Top
Post Reply

Who is online

Users browsing this forum: erlinden and 47 guests
  4. RouterOS
  5. Beginner Basics