Community discussions

MikroTik App
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

3-ISPs Load Balancing - need help

Sun Feb 18, 2024 10:55 pm

Hello everyone. Need some help on triple ISP config with PCC load balancing. Looking for assistance to please look at the mangle & routing rules. Also the DHCP-client script. Config & concerns are below.

HEX on ROS 7.13.4 using winbox
ISP1: ether1 (100/100)
ISP2: ether2 (100/100)
ISP3: ether3 (100/100)
Lan Bridge: ether 4 & ether5

Concerns:
Mangle rules: I'm not sure if the rules are correct, the ordering of the rules, using passthrough properly, etc. For some reason, when WAN-3 is disabled (interface disabled or cable unplugged), the prerouting mangle rules for ether3 (ISP3 / to_ISP3) still have traffic, also shown in connection tracking. I delete the connections but still connections repopulate like it's connected. The output chain for ether2 & ether3 barely have any traffic. Most is on ether1, a lot of it. Leading to believe load balancing isn't working as intended because I mucked something up.

dhcp-client: The dhcp-client creates a default route for each ISP (not sure I should be doing this or it's ok). I also create new routes for routing tables to_ISPs. Found a script to update the gateway & also delete the connections when the gateway IP changes. Problem is, neither work. The GW is not being updated nor are the connections being killed/deleted. I don't know anything about scripts. I've tried copy/paste script in the dhcp-client advanced tab in winbox & also using the terminal.
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN

/interface bridge port
add bridge="Lan Bridge" interface=ether4
add bridge="Lan Bridge" interface=ether5

/ip firewall connection tracking
set tcp-established-timeout=12h loose-tcp-tracking=no

/ip settings
set max-neighbor-entries=4096 rp-filter=loose

/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=ether3 list=WAN
add interface="Lan Bridge" list=LAN

/routing table
add fib name=to_ISP1
add fib name=to_ISP2
add fib name=to_ISP3

/ip dhcp-client 
add interface=ether1 add-default-route=yes script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP1\"] gateway=\$\"gateway-address\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]" use-peer-dns=no use-peer-ntp=no
add interface=ether2 add-default-route=yes default-route-distance=2 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP2\"] gateway=\$\"gateway-address\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]" use-peer-dns=no use-peer-ntp=no
add interface=ether3 add-default-route=yes default-route-distance=3 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP3\"] gateway=\$\"gateway-address\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]" use-peer-dns=no use-peer-ntp=no

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked disabled=no
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid disabled=no
add action=accept chain=input comment="defconf: accept ICMP" disabled=no protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback" disabled=no dst-address=127.0.0.1
add action=accept chain=input comment="admin access" disabled=no in-interface-list=LAN src-address-list=admin
add action=accept chain=input comment="allow LAN DNS queries-TCP" disabled=no dst-port=53 in-interface-list=LAN protocol=tcp
add action=accept chain=input comment="allow LAN DNS/NTP queries-UDP" disabled=no dst-port=53,123 in-interface-list=LAN protocol=udp
add action=drop chain=input comment="drop all else" disabled=no
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related disabled=no hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked disabled=no
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid disabled=no
add action=accept chain=forward comment="internet access" disabled=no in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat disabled=no
add action=accept chain=forward comment="allow multi-subnet access" disabled=no dst-address-list=expected-address-from-LAN src-address-list=expected-address-from-LAN in-interface-list=LAN out-interface-list=LAN
add action=drop chain=forward comment="drop all else" disabled=no

/ip firewall mangle
add action=accept chain=prerouting comment="Lan Bridge access" dst-address-list=expected-address-from-LAN in-interface-list=LAN
add action=mark-connection chain=prerouting comment="mark all new incoming connections" connection-mark=no-mark connection-state=new in-interface=ether1 new-connection-mark=ISP1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=ether2 new-connection-mark=ISP2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=ether3 new-connection-mark=ISP3_conn passthrough=yes
add action=mark-connection chain=prerouting comment="divide traffic into three groups" connection-mark=no-mark connection-state=new dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=src-address-and-port:3/0
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=src-address-and-port:3/1
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP3_conn passthrough=yes per-connection-classifier=src-address-and-port:3/2
add action=mark-routing chain=prerouting comment="distribute traffic across ISP's" connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP3_conn in-interface-list=LAN new-routing-mark=to_ISP3 passthrough=yes
add action=mark-routing chain=output comment="force connection to table" connection-mark=ISP1_conn new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=output connection-mark=ISP3_conn new-routing-mark=to_ISP3 passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="ISP1 - Masquerade" out-interface=ether1
add action=masquerade chain=srcnat comment="ISP2 - Masquerade" out-interface=ether2
add action=masquerade chain=srcnat comment="ISP3 - Masquerade" out-interface=ether3

/ip route
add check-gateway=ping gateway=ISP1-IP routing-table=to_ISP1 distance=1 comment="ISP1"
add check-gateway=ping gateway=ISP2-IP routing-table=to_ISP2 distance=1 comment="ISP2"
add check-gateway=ping gateway=ISP3-IP routing-table=to_ISP3 distance=1 comment="ISP3"
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Sun Feb 18, 2024 11:20 pm

Can you confirm that ISP1,2,3 are providing static fixed WANIPs, or dynamic WANIps
Are they all from the same provider?
Do you have any external traffic going to the router itself (aka wireguard handshake for example)
Do you have any LAN servers that external traffic expects to hit...........
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Mon Feb 19, 2024 12:04 am

Can you confirm that ISP1,2,3 are providing static fixed WANIPs, or dynamic WANIps
Are they all from the same provider?
Do you have any external traffic going to the router itself (aka wireguard handshake for example)
Do you have any LAN servers that external traffic expects to hit...........
Hello Mesquite. Thank you for looking.
Yes, the ISP's are all dynamic
Different providers for each ISP
No wireguard (had it but removed it).
There is a server on the Lan Bridge that is accessible externally via port forwarding (I didn't provide that NAT rule for it)
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Mon Feb 19, 2024 1:07 am

Okay, well since you have servers in the mix External traffic to LAN, then we have to account for that traffic to ensure its goes back out the same WAN it came in.
Since you have no external traffic to the router itself,VPN etc, there is no need to mangle that traffic.

1. The other thing I noted is that you have hoisted in the fact that for actual IP routes where we need to identify the gateway IP of ISP1,2,3 we need to ensure that the current gateway IP is used and thus we need scripts, However your scripts are long winded and seem to involve removing connection marks............... I am not aware of this effort, and IMHO its added complexity not required or desired. So I will propose what I would do for each of the three IP DHCP Client script settings. Just modify the other two by changing the parameter number used.

Select in IP DHCP client : In DHCP TAB --> Add default route=yes. In Advanced TAB --> Set default route distance=255. What we are doing is forcing the update of the gateway IP and not interfering with any of our manually created routes. In advance tab script area:

:if ($bound=1) do={
:local gw [ /ip dhcp-client get $"gateway-address" ]
/ip route set [ find comment="ISP1-MainTable" gateway!=$gw ] gateway=$gw
/ip route set [ find comment="ISP1-SpecificTable" gateway!=$gw ] gateway=$gw
}

Next lets tackle the ROUTES......
We need to establish the basic IP routes for the router on the main table.
They are missing!!

/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=Current_ISP1-gatewayIP-address routing-table=main distance=1 comment="ISP1-MainTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=Current_ISP2-gatewayIP-address routing-table=main distance=2 comment="ISP2-MainTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=Current_ISP3-gatewayIP-address routing-table=main distance=3 comment="ISP3-MainTable"


Then the specific tables you have created already............... check-gateway=ping is not useful here, each table is separate and availability determined on main table execution.
Neither is distance.

/ip route
add dst-address=0.0.0.0/0 gateway=Current_ISP1-gatewayIP-address routing-table=to_ISP1 comment="ISP1-SpecificTable"
add dst-address=0.0.0.0/0 gateway=Current_ISP2-gatewayIP-address routing-table=to_ISP2 comment="ISP2-SpecificTable"
add dst-address=0.0.0.0/0 gateway=Current_ISP3-gatewayIP-address routing-table=to_ISP3 comment="ISP3-SpecificTable"


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Lets tackle the servers issue.........
2. Cover off Server traffic before any PCC rules!!

The first thing to do is mark traffic entering the WAN externally and being forwarded to the LAN.
/ip mangle
add chain=forward action=mark-connections connection-mark=no-mark in-interface=ether1 \
new connection-mark=WAN1-conn passthrough=yes
add chain=forward action=mark-connections connection-mark=no-mark in-interface=ether2 \
new connection-mark=WAN2-conn passthrough=yes
add chain=forward action=mark-connections connection-mark=no-mark in-interface=ether3 \
new connection-mark=WAN3-conn passthrough=yes


Then we route that traffic out the same WAN.
/ip mangle
add chain=prerouting action=mark-routing connection-mark=WAN1-conn \
new-routing-mark=to_ISP1 src-address-list=Servers passthrough=no
add chain=prerouting action=mark-routing connection-mark=WAN2-conn \
new-routing-mark=to_ISP2 src-address-list=Servers passthrough=no
add chain=prerouting action=mark-routing connection-mark=WAN3-conn \
new-routing-mark=to_ISP3 src-address-list=Servers passthrough=no


What we are doing is saying any of the external traffic with those markings being replied to by the servers needs to go out the specific wan table.
You could also simply use in-interface-list=LAN instead, your choice........

In any case in the order of mangle rules make sure the above ones are entered first!

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Now lets look at those PCC rules. The first thing we do is mark connections not going to local address types and being forwarded from the LAN.

(edit fixed chains to prerouting)
add chain=prerouting action=mark-connection connection-mark=no-mark dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes
per-connection-classifier=src-address-and-port:3/0
add chain=prerouting action=mark-connection connection-mark=no-mark dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes
per-connection-classifier=src-address-and-port:3/1
add chain=prerouting action=mark-connection connection-mark=no-mark dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP3_conn passthrough=yes
per-connection-classifier=src-address-and-port:3/2


Next we route-mark the connection marked traffic back through the specific table and thus WAN..........

add chain=prerouting action=mark-routing in-interface-list=LAN connection-mark=ISP1_conn \
new-routing-mark=to_ISP1 passthrough=no
add chain=prerouting action=mark-routing in-interface-list=LAN connection-mark=ISP2_conn \
new-routing-mark=to_ISP2 passthrough=no
add chain=prerouting action=mark-routing in-interface-list=LAN connection-mark=ISP3_conn \
new-routing-mark=to_ISP3 passthrough=no


DONE......... the rest of the mangle rules you have are not required.

MODIFY FORWARD CHAIN FASTRACK RULE TO:

add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related disabled=no hw-offload=yes connection-mark=no-mark
Last edited by Mesquite on Thu Feb 22, 2024 4:06 am, edited 1 time in total.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Mon Feb 19, 2024 1:35 am

Thank you Mesquite. I will adjust the config & report back shortly
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Mon Feb 19, 2024 3:45 am

A couple of things to think about once we get the rudimentary setup working properly.
Did you want recursive routing, meaning check the connectivity not from router to ISP but from router do public DNS on the web.
Rare to happen but its possible the router and the ISP talk but the ISP to the internet is broken and thats why people use recursive routing.

Second is failover. Right now its basic failover based on the setup in the main routes

If ISP1 fails its pCC traffic will go to ISP2.
If ISP2 failts its pCC traffic will go to ISP1
if ISP3 fails its pCC traffic will go to ISP1

By changing the distances on the main routes you can select how traffic is re-routed upon a failure.
There is another option a bit more complex..................
What If you wanted the PCC traffic from a failed WAN connection to be split between the two remaining WANs........
Thus this is something to explore as well.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 3:42 am

Greetings Mesquite

Apologies for the delay in response. I made the changes. I ended up losing access to the hex. I couldn't login using the IP 192.168.88.1. I was able to regain access by using the mac method in the neighbors tab in winbox. I disabled the mangle rules & was able to gain access again using the IP. Narrowed it down to the first set of forward rules that's blocking the access via IP. Disabled the following first rule set (2 more rules for ISP's 2 & 3):

add action=mark-connection chain=forward comment=\
"mark all new incoming connections" connection-mark=no-mark \
connection-state="" disabled=yes in-interface=ether1 new-connection-mark=\
WAN1-conn passthrough=yes

Logged in via mac method in neighbors tab, all mangle rules enabled:
Looking at the routes, there are 2 sets of default routes (for main table) for each ISP. One set that the dhcp-client auto creates & the manual ones I added. The routes are all connected, no routes indicating unreachable. Also, ISP2 isn't working, no traffic & the config seems to favor ISP3, even after multiple reboots, disabling interfaces and/or dhcp clients etc. ISP3 ends up with most (roughly 90%) of the traffic. Goal is to spread all of the traffic among all 3 ISP's equally. But if one goes out, to use whichever ISP's are working.
Screen Shot 2024-02-19 at 8.27.44 PM.png

On the recursive front, I definitely would be interested in that. I have tried to understand it in the past but abandoned it. But I read another thread the other day where you laid it out perfectly. This thread here: viewtopic.php?t=204636 Now a days with fiber becoming more of a standard (which one of my ISP is), the GW will ping even though the connection is down. So I definitely think recursive is the way to go. Enough of my rambling haha

Config:
# 2024-02-19 19:46:46 by RouterOS 7.13.4
# software id = FBAJ-N40M
#
# model = RB750Gr3
# serial number = xxxxxx
/interface bridge
add comment="Lan Bridge" name="Lan Bridge"
/interface ethernet
set [ find default-name=ether1 ] comment="ISP1 - Buckeye - Load Balancing"
set [ find default-name=ether2 ] comment="ISP2 - T-Mobile - Load Balancing"
set [ find default-name=ether3 ] comment="ISP3 - AT&T - Load Balancing"
set [ find default-name=ether4 ] comment="TP-Link Network"
set [ find default-name=ether5 ] comment="Mikrotik Network"
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add comment="Lan Bridge" name=LAN ranges=192.168.88.20-192.168.88.150
/ip dhcp-server
add address-pool=LAN interface="Lan Bridge" name=dhcp1
/routing table
add fib name=to_ISP1
add fib name=to_ISP2
add fib name=to_ISP3
/interface bridge port
add bridge="Lan Bridge" interface=ether4
add bridge="Lan Bridge" interface=ether5
/ip firewall connection tracking
set loose-tcp-tracking=no tcp-established-timeout=12h
/ip settings
set max-neighbor-entries=4096 rp-filter=loose
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=ether3 list=WAN
add interface="Lan Bridge" list=LAN
/ip address
add address=192.168.88.1/24 interface="Lan Bridge" network=192.168.88.0
/ip dhcp-client
add comment="ISP1 - Buckeye - DHCP Client" default-route-distance=255 \
    interface=ether1 script=":if (\$bound=1) do={\r\
    \n:local gw [ /ip dhcp-client get \$\"gateway-address\" ]\r\
    \n/ip route set [ find comment=\"ISP1-MainTable\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP1-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP2 - T-Mobile - DHCP Client" default-route-distance=255 \
    interface=ether2 script=":if (\$bound=1) do={\r\
    \n:local gw [ /ip dhcp-client get \$\"gateway-address\" ]\r\
    \n/ip route set [ find comment=\"ISP2-MainTable\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP2-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP3 - AT&T - DHCP Client" default-route-distance=255 interface=\
    ether3 script=":if (\$bound=1) do={\r\
    \n:local gw [ /ip dhcp-client get \$\"gateway-address\" ]\r\
    \n/ip route set [ find comment=\"ISP3-MainTable\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP3-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
/ip dhcp-server network
add address=192.168.88.0/24 comment="Lan Bridge" dns-server=192.168.88.1 \
    gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1
/ip firewall address-list
add address=192.168.88.200-192.168.88.205 comment="Mikrotik - Admin Devices" list=admin
add address=192.168.88.0/24 comment="Mikrotik hEX Network" list=\
    expected-address-from-LAN
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback" \
    dst-address=127.0.0.1
add action=accept chain=input comment="admin access" in-interface-list=LAN \
    src-address-list=admin
add action=accept chain=input comment="allow LAN DNS queries-TCP" dst-port=53 \
    in-interface-list=LAN protocol=tcp
add action=accept chain=input comment="allow LAN DNS/NTP queries-UDP" \
    dst-port=53,123 in-interface-list=LAN protocol=udp
add action=drop chain=input comment="drop all else"
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-mark=no-mark connection-state=established,related hw-offload=\
    yes
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=accept chain=forward comment="internet access" in-interface-list=\
    LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" \
    connection-nat-state=dstnat
add action=drop chain=forward comment="drop all else"
/ip firewall mangle
add action=mark-connection chain=forward comment=\
    "mark all new incoming connections" connection-mark=no-mark \
    connection-state="" disabled=yes in-interface=ether1 new-connection-mark=\
    WAN1-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    connection-state="" disabled=yes in-interface=ether2 new-connection-mark=\
    WAN2-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    connection-state="" disabled=yes in-interface=ether3 new-connection-mark=\
    WAN3-conn passthrough=yes
add action=mark-routing chain=prerouting comment="force connection to table" \
    connection-mark=WAN1-conn disabled=yes in-interface-list=LAN \
    new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2-conn disabled=\
    yes in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN3-conn disabled=\
    yes in-interface-list=LAN new-routing-mark=to_ISP3 passthrough=no
add action=mark-connection chain=forward comment=\
    "divide traffic into three groups" connection-mark=no-mark \
    connection-state="" disabled=yes dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes \
    per-connection-classifier=src-address-and-port:3/0
add action=mark-connection chain=forward connection-mark=no-mark \
    connection-state="" disabled=yes dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes \
    per-connection-classifier=src-address-and-port:3/1
add action=mark-connection chain=forward connection-mark=no-mark \
    connection-state="" disabled=yes dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP3_conn passthrough=yes \
    per-connection-classifier=src-address-and-port:3/2
add action=mark-routing chain=prerouting comment=\
    "distribute traffic across ISP's" connection-mark=ISP1_conn disabled=yes \
    in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn disabled=\
    yes in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP3_conn disabled=\
    yes in-interface-list=LAN new-routing-mark=to_ISP3 passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat comment="ISP1 - Buckeye - Masquerade" \
    out-interface=ether1
add action=masquerade chain=srcnat comment="ISP2 - T-Mobile - Masquerade" \
    out-interface=ether2
add action=masquerade chain=srcnat comment="ISP3 - AT&T - Masquerade" \
    out-interface=ether3
/ip route
add comment=ISP1-MainTable disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=ISP1-GW-IP pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment=ISP2-MainTable disabled=no distance=2 dst-address=0.0.0.0/0 \
    gateway=ISP2-GW-IP pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment=ISP3-MainTable disabled=no distance=3 dst-address=0.0.0.0/0 \
    gateway=ISP3-GW-IP pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment=ISP1-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=ISP1-GW-IP pref-src="" routing-table=to_ISP1 scope=30 \
    suppress-hw-offload=no target-scope=10
add comment=ISP2-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=ISP2-GW-IP pref-src="" routing-table=to_ISP2 scope=30 \
    suppress-hw-offload=no target-scope=10
add comment=ISP3-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=ISP3-GW-IP pref-src="" routing-table=to_ISP3 scope=30 \
    suppress-hw-offload=no target-scope=10
You do not have the required permissions to view the files attached to this post.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 5:02 am

Okay, the IP Routes Table is exactly correct.
If one was to follow the main table, traffic would get routed out WAN1 as its primary (and working hence black).
The two farther distance routes wan2 and wan3 are blue because they are not being used in the main table at the moment ( on-standby)
The specific tables and routes for wan1,2,3 are black hence connected and will be used for traffic through those tables. All good!!

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

What I dont understand is why you have disabled all the mangling??
THe mangle rules should not interfere with accessing the router itself etc. but let me think about why it could............
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 5:26 am

Okay upon review, I erred in the first half of second set of mangle rules......................
The initial PCC rules.....
FROM:
add action=mark-connection chain=forward comment=\
"divide traffic into three groups" connection-mark=no-mark \
connection-state="" disabled=yes dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes \
per-connection-classifier=src-address-and-port:3/0
add action=mark-connection chain=forward connection-mark=no-mark \
connection-state="" disabled=yes dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes \
per-connection-classifier=src-address-and-port:3/1
add action=mark-connection chain=forward connection-mark=no-mark \
connection-state="" disabled=yes dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP3_conn passthrough=yes \
per-connection-classifier=src-address-and-port:3/2


TO:
add action=mark-connection chain=prerouting comment=\
"divide traffic into three groups" connection-mark=no-mark \
connection-state="" disabled=yes dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes \
per-connection-classifier=src-address-and-port:3/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
connection-state="" disabled=yes dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes \
per-connection-classifier=src-address-and-port:3/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
connection-state="" disabled=yes dst-address-type=!local \
in-interface-list=LAN new-connection-mark=ISP3_conn passthrough=yes \
per-connection-classifier=src-address-and-port:3/2


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
See if that solves the problem. If not then next step is to put this mangle rule as the first mangle rule in the order of mangle rules.

/ip firewall mangle
add chain=prerouting action=accept src-address-list=192.168.88.0/24 dst-address-type=local
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 6:24 am

Greeting Mesquite

That worked! Changing the chain to prerouting vs forward. I can access via IP. All ISP's have equal amounts of traffic shown in interfaces. But, the counters in the mangles rules are at 0 for the first 2 sets of rules. Everything seems to be working though.

Does it matter is the connection marks are different from the first 2 sets of rules vs the last 2 sets of rules?
First 2 sets connection mark: WAN1-conn
Last 2 sets connection marks: ISP1_conn
Screen Shot 2024-02-19 at 11.22.21 PM.png
You do not have the required permissions to view the files attached to this post.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 1:39 pm

No it makes little difference, perhaps more items for the CPU to keep track of. I personally like the differentiation as its really different traffic we are marking, the similarity is we push that traffic to the same routes, via using the same table.

So the PCC is working, great news!! As for mangle rule counters?? Think that was for any external traffic to the routers............ has that been exercised or tested??
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 8:27 pm

Hello Mesquite

You were right! I put the hex in a live environment & the counters in the first 2 sets of mangle rules are now working.
I personally like the differentiation as its really different traffic we are marking, the similarity is we push that traffic to the same routes, via using the same table.
I like the way you think. Perfect!

One small thing, the gateway IP's are not updating in the manually created routes. The default route that the DHCP-client creates is updated with the new IP, but the manual routes don't update with the script in the dhcp-client. I disabled/enabled interface, dhcp-client & also rebooted. Neither has updated the manual routes. Script below:
:if ($bound=1) do={
:local gw [ /ip dhcp-client get $"gateway-address" ]
/ip route set [ find comment="ISP3-MainTable" gateway!=$gw ] gateway=$gw
/ip route set [ find comment="ISP3-SpecificTable" gateway!=$gw ] gateway=$gw
}
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 9:17 pm

Can you poste your IP ROUTES protecting the actual WANIPs of course.
Im looking for the line with distance of 255 especially, should be three of them in blue ( possibly withe Dd as the first entry column)
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 9:36 pm

I printed the routes. When I export using "export verbose file=" or "export file=", the defaults created by dhcp-client don't show up. Only the manually created routes do. Let me know if there's a way to print it properly. If not, if you'd like, I can create the command similar to the other routes

D d 0.0.0.0/0 xxx.xxx.xx.x 255
D d 0.0.0.0/0 xxx.xxx.xx.x 255
D d 0.0.0.0/0 xxx.xxx.xx.x 255

All are the same but here is ISP3:
add distance=255 dst-address=0.0.0.0/0 gateway=ISP3-GW-IP pref-src="" routing-table=main scope=30 target-scope=10 suppress-hw-offload=no vrf-interface=ether3
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 10:56 pm

Okay will have to sort out proper syntax............ its like working through mud!!
I am assuming that you are using the letters instead of the actual number showing correct to keep wan information private???
gateway=ISP3-GW-IP
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 11:07 pm

Correct. I'll put it back to test that way you'll see the full config, along with the IP info. Allow me some time to do this. I'm not in front of it at the moment. I'll provide the full unedited config later tonight. Is there a certain way you want me to export the config? Or is the standard "export file=" ok to provide?
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 11:17 pm

standard export is fine.........i use notepad++
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Tue Feb 20, 2024 11:24 pm

standard export is fine.........i use notepad++

Sounds good. I'll reply back in a few hours with the full config. Thank you Mesquite
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Wed Feb 21, 2024 12:09 am

Try this iteration......... for the three scripts.......... should work.

:if ($bound=1) do={
:local gw $"gateway-address"
/ip route set [ find comment="ISP1-MainTable" gateway!=$gw ] gateway=$gw
/ip route set [ find comment="ISP1-SpecificTable" gateway!=$gw ] gateway=$gw
}
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Wed Feb 21, 2024 12:30 am

Try this iteration......... for the three scripts.......... should work.

:if ($bound=1) do={
:local gw $"gateway-address"
/ip route set [ find comment="ISP1-MainTable" gateway!=$gw ] gateway=$gw
/ip route set [ find comment="ISP1-SpecificTable" gateway!=$gw ] gateway=$gw
}

Genius! That worked, but only after a reboot. I waited a few minutes & it didn't update. After the reboot, both manually created routes updated for ISP3 (only one that I tested). Once I get home, I'll try it again after changing the GW & will report back. Do you still want the full config when I post?
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Wed Feb 21, 2024 12:38 am

Negative if all three ISPs, when rebooted, provide the proper gateway IP in IP routes we are good to go.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help  [SOLVED]

Wed Feb 21, 2024 5:45 am

Hello Mesquite

Happy to report the dhcp-client script is updating all routes, didn't have to reboot this time. Changed the GW two more times for sake of sanity haha. The pcc load balancing is working perfect. As a matter of fact everything is working perfectly. Thank you so much for your time & expertise!!
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Wed Feb 21, 2024 11:14 pm

Awesome, I need we could get there, the journey is the fun part!
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Wed Feb 21, 2024 11:33 pm

Awesome, I need we could get there, the journey is the fun part!

Hi Mesquite. Indeed it is & was. Thank you for everything & helping me out with this. It's working perfectly. For shits & giggles, I randomly moved the cables around earlier today (ISPs/WANs) between ether1-ether3 & the routes updated instantly. Very happy!! Soon enough I'll try to tackle the recursive portion. I didn't want to continue this thread & be bothersome lol. I believe I understand your post in the recursive thread (viewtopic.php?t=204636) about the posting in github. I've seen that exact same post & couldn't understand it for the life of me until you cleared it all up. That's another task for another day.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Thu Feb 22, 2024 1:12 am

Hello Mesquite

I went ahead & implemented the recursive as well. Hopefully it's correct if I understood your directions. I also changed the dhcp-client script to update ISPx-Recursive instead of ISPx-MainTable.


Screen Shot 2024-02-21 at 6.09.38 PM.png
You do not have the required permissions to view the files attached to this post.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Thu Feb 22, 2024 4:03 am

Sweet!! Looking good!

Now for the icing on the cake!!!
Have each WAN, when it fails, spread its traffic evenly to the two remaining ISPs vice just one.

Step One:
Increase the PCC mangles. The idea is is not to have 1/3 of the traffic go to each WAN, but 2/6 of the traffic go to each WAN and thus when one fails we send 1/6 to WANX and 1/6 to WANY :-)
Need six fib tables to start.

Thus our PCC tables would look like the following: So create the 6 highlighted fib tables
6/0 --> AthenB traffic to WAN1 ( 1/6 )
6/1 --> AthenC traffic to WAN1 ( 1/6 )
6/2 --> BthenA traffic to WAN2 ( 1/6 )
6/3 --> BthenC traffic to WAN2 ( 1/6 )
6/4 --> CthenA traffic to WAN3 ( 1/6 )
6/5 --> CthenB traffic to WAN3 ( 1/6 )

Thus the mangle rules would consist not of three connection marking and three route marking rules (six in total) but six and six so 12 in total.
Basically just modify what you have by changing the table names and using the PCC qualifiers on the left 6/0 thru 6/5

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

So nothing drastically hard, next is routes...... If you recall we had 3 basic routes initially, with two additional specific table routes for basic PCC and basic failover, which were changed to six for the recursive plus the two specific table routes.
WE KEEP the six recursive main table routes, that does not change, but we now need two routes per specific table. Think of it this way, each PCC will generate two routes., since we now have six tables that means 12 routes in total for particular tables..........

Note that since we have the table identified twice ( in two rules ) we differentiate which gateway will be used first, and which will be used as failover if the first one is not available.
We use check-gateway=ping on the first one because as soon as that WAN comes backup we want to start using it for PCC again.
Also, you will need to add comment line to each line of the config........ suggesting something simple for both recursive ones and these ones..
You dont need to but would separate them out
For recursive keep the ones you have.......... and for the new specific tables.

.:if ($bound=1) do={
:local gw $"gateway-address"
/ip route set [ find comment="ISP1-MainTable" gateway!=$gw ] gateway=$gw
/ip route set [ find comment="ISP1-ST" gateway!=$gw ] gateway=$gw
}


add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useAthenB distance=2 comment="ISP1-ST"
add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useAthenB distance=3 comment="ISP2-ST"

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useAthenC distance=2 comment="ISP1-ST"
add dst-address=0.0.0.0/0 gateway=ISp3-gatewayIP routing-table=useAthenC distance=3 comment="ISP3-ST"

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useBthenA distance=2 comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useBthenA distance=3 comment="ISP1-ST"

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useBthenC comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=useBthenC distance=3 comment="ISP3-ST"

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=useCthenA distance=2 comment="ISP3-ST"
add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useCthenA distance=3 comment="ISP1-ST"

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=useCthenB distance=2 comment="ISP3-ST"
add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useCthenB distance=3
comment="ISP2-ST"
Last edited by Mesquite on Thu Feb 22, 2024 4:46 am, edited 2 times in total.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Thu Feb 22, 2024 4:33 am

That's great news! Thanks for your stamp of approval! Now off to update the config on the active router. It never ends haha
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Thu Feb 22, 2024 4:46 am

Maybe you spoke to soon, read above again LOL new info.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Thu Feb 22, 2024 5:35 am

Apparently so lmao. Putting more work on me eyyyyyy. Alright, I'm up for it. Will let you know how it goes tomorrow
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 1:22 am

Ok, here's what I got. I haven't configured yet, just "made" the config for verification prior to executing. With creating the new routes that check-gateway=ping, would that affect the recursive GW's?

I changed the tables for AthenB to 1then2 and so on. Just to keep it uniform with the numerical values already being used.
"ISP1-MainTable" can't be updated with the dhcp-client script due to recursive DNS pings. It was replaced with "ISP1-Recursive".
Hopefully I understood your directions.
/routing table
add fib name=to_ISP1
add fib name=to_ISP2
add fib name=to_ISP3
add fib name=1then2
add fib name=1then3
add fib name=2then1
add fib name=2then3
add fib name=3then1
add fib name=3then2

#No changes made below. Not sure what to change the "new-routing-mark" to. But the routes "to_ISP1" and so on are now missing. Do I double these up as well?
/ip firewall mangle
add action=mark-connection chain=forward comment=\
    "mark all new incoming connections" connection-mark=no-mark in-interface=\
    ether1 new-connection-mark=WAN1-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    in-interface=ether2 new-connection-mark=WAN2-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    in-interface=ether3 new-connection-mark=WAN3-conn passthrough=yes
add action=mark-routing chain=prerouting comment="force connection to table" \
    connection-mark=WAN1-conn in-interface-list=LAN new-routing-mark=to_ISP1 \
    passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2-conn \
    in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN3-conn \
    in-interface-list=LAN new-routing-mark=to_ISP3 passthrough=no

#Changes made below
/ip firewall mangle
add action=mark-connection chain=prerouting comment=\
    "divide traffic into six groups" connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=1then2_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=1then3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=2then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/2
add action=mark-connection chain=prerouting comment=\
    "divide traffic into three groups" connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=2then3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=3then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/4
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=3then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/5
add action=mark-routing chain=prerouting comment=\
    "distribute traffic across ISP's" connection-mark=1then2_conn \
    in-interface-list=LAN new-routing-mark=1then2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=1then3_conn \
    in-interface-list=LAN new-routing-mark=1then3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=2then1_conn \
    in-interface-list=LAN new-routing-mark=2then1 passthrough=no
add action=mark-routing chain=prerouting comment=\
    "distribute traffic across ISP's" connection-mark=2then3_conn \
    in-interface-list=LAN new-routing-mark=2then3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=3then1_conn \
    in-interface-list=LAN new-routing-mark=3then1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=3then2_conn \
    in-interface-list=LAN new-routing-mark=3then2 passthrough=no

#No changes made below (the default routes created by dhcp-client (distance=255) don't get printed using "export file=")
/ip route
add check-gateway=ping comment=ISP1-MainTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=8.8.4.4 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping comment=ISP2-MainTable disabled=no distance=2 dst-address=0.0.0.0/0 gateway=9.9.9.9 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping comment=ISP3-MainTable disabled=no distance=3 dst-address=0.0.0.0/0 gateway=185.228.168.9 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add comment=ISP1-Recursive disabled=no distance=1 dst-address=8.8.4.4/32 gateway=192.168.87.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP2-Recursive disabled=no distance=2 dst-address=9.9.9.9/32 gateway=192.168.86.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP3-Recursive disabled=no distance=3 dst-address=185.228.168.9/32 gateway=192.168.85.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11

#Changes made below
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=1then2 distance=2 comment="ISP1-SpecificTable"
add dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=1then2 distance=3 comment="ISP2-SpecificTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=1then3 distance=2 comment="ISP1-SpecificTable"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=1then3 distance=3 comment="ISP3-SpecificTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then1 distance=2 comment="ISP2-SpecificTable"
add dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=2then1 distance=3 comment="ISP1-SpecificTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then3 comment="ISP2-SpecificTable"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=2then3 distance=3 comment="ISP3-SpecificTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=3then1 distance=2 comment="ISP3-SpecificTable"
add dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=3then1 distance=3 comment="ISP1-SpecificTable"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=3then2 distance=2 comment="ISP3-SpecificTable"
add dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=3then2 distance=3 comment="ISP2-SpecificTable"

#No changes made below. Replaced routes comment above "ISP1-ST" to "ISP1-SpecificTable" that was already in the script
/ip dhcp-client
add comment="ISP1 - Buckeye - DHCP Client" default-route-distance=255 \
    interface=ether1 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP1-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP1-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP2 - T-Mobile - DHCP Client" default-route-distance=255 \
    interface=ether2 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP2-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP2-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP3 - AT&T - DHCP Client" default-route-distance=255 interface=\
    ether3 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP3-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP3-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 3:19 am

Wow, very impressive, you nailed it. I am going to start asking you todo all my configs!!.
You kept the original recursive routes on the main table as required.
you kept the tables and mangles and the specific table routes to ensure server traffic was not entangled with PCC traffic.
you correctly setup the mangles for PCC and correctly applied the routes
you correctly applied the scripts in terms of identifying the correct comments so each gateway IP would be updated.

On the way back from the pool, I had a thought LOL. One might be tempted to change the order of mangles such that its more like the order

6/0 --> AthenB traffic to WAN1 ( 1/6 )
6/1 --> BthenA traffic to WAN2 ( 1/6 )
6/2 --> CthenA traffic to WAN3 ( 1/6 )
6/3 --> AthenC traffic to WAN1 ( 1/6 )
6/4 --> BthenC traffic to WAN2 ( 1/6 )
6/5 --> CthenB traffic to WAN3 ( 1/6 )

Assuming all WANS are up............ the first connection would go to WAN1, then to WAN2, then to WAN3, then repeat, right now the first two connections go to wan1, the next two go to wan 2 and the next two go to wan3 and it repeats. If you can wrap your head around that LOL.
Last edited by Mesquite on Fri Feb 23, 2024 4:39 am, edited 1 time in total.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 3:21 am

Lol. You posted right after I updated. Look again lol. I may have mucked it up

Update. On second thought. Don't look yet. Let me make the changes to the incoming mangle rules & change what I updated lol...
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 3:25 am

Okay I see one error in the last PCC mark connection......
add action=mark-connection chain=prerouting connection-mark=no-mark \
connection-state="" dst-address-type=!local in-interface-list=LAN \
new-connection-mark=3then1_conn passthrough=yes per-connection-classifier=\
src-address-and-port:6/4

add action=mark-connection chain=prerouting connection-mark=no-mark \
connection-state="" dst-address-type=!local in-interface-list=LAN \
new-connection-mark=3then1_conn passthrough=yes per-connection-classifier=\
src-address-and-port:6/5


The last PCC mark connection rule should be 3 then 2 !!!
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 3:28 am

Why did you remove the SPECIFIC Table routes?
The clue is if you kept them in the tables listing created, you should be using them!
Remember this is for server traffic so you should use them............
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 3:30 am

Why did you remove the SPECIFIC Table routes?
The clue is if you kept them in the tables listing created, you should be using them!
Remember this is for server traffic so you should use them............

Yup, I'm reverting. Standby
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 3:42 am

Are you sure about this?

6/0 --> AthenB traffic to WAN1 ( 1/6 )
6/1 --> BthenA traffic to WAN2 ( 1/6 )
6/2 --> CthenA traffic to WAN3 ( 1/6 )
6/3 --> Athen traffic to WAN1 ( 1/6 ) <--- Missing letter
6/4 --> BthenC traffic to WAN2 ( 1/6 )
6/5 --> CthenB traffic to WAN3 ( 1/6 )
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 4:39 am

Scratch the last post. I misunderstood your direction. Don't know what I was thinking lol. Config is updated below...
/routing table
add fib name=to_ISP1
add fib name=to_ISP2
add fib name=to_ISP3
add fib name=1then2
add fib name=1then3
add fib name=2then1
add fib name=2then3
add fib name=3then1
add fib name=3then2

#No changes made
/ip firewall mangle
add action=mark-connection chain=forward comment=\
    "mark all new incoming connections" connection-mark=no-mark in-interface=\
    ether1 new-connection-mark=WAN1-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    in-interface=ether2 new-connection-mark=WAN2-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    in-interface=ether3 new-connection-mark=WAN3-conn passthrough=yes
add action=mark-routing chain=prerouting comment="force connection to table" \
    connection-mark=WAN1-conn in-interface-list=LAN new-routing-mark=to_ISP1 \
    passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2-conn \
    in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN3-conn \
    in-interface-list=LAN new-routing-mark=to_ISP3 passthrough=no

#Changes made & mangle rules reordered
/ip firewall mangle
add action=mark-connection chain=prerouting comment=\
    "divide traffic into six groups" connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=1then2_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=2then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=3then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=1then3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=2then3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/4
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=3then2_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/5
add action=mark-routing chain=prerouting comment=\
    "distribute traffic across ISP's" connection-mark=1then2_conn \
    in-interface-list=LAN new-routing-mark=1then2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=1then3_conn \
    in-interface-list=LAN new-routing-mark=1then3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=2then1_conn \
    in-interface-list=LAN new-routing-mark=2then1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=2then3_conn \
    in-interface-list=LAN new-routing-mark=2then3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=3then1_conn \
    in-interface-list=LAN new-routing-mark=3then1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=3then2_conn \
    in-interface-list=LAN new-routing-mark=3then2 passthrough=no

#No changes made (the default routes created by dhcp-client (distance=255) don't get printed using "export file=")
/ip route
add check-gateway=ping comment=ISP1-MainTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=8.8.4.4 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping comment=ISP2-MainTable disabled=no distance=2 dst-address=0.0.0.0/0 gateway=9.9.9.9 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping comment=ISP3-MainTable disabled=no distance=3 dst-address=0.0.0.0/0 gateway=185.228.168.9 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add comment=ISP1-Recursive disabled=no distance=1 dst-address=8.8.4.4/32 gateway=192.168.87.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP2-Recursive disabled=no distance=2 dst-address=9.9.9.9/32 gateway=192.168.86.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP3-Recursive disabled=no distance=3 dst-address=185.228.168.9/32 gateway=192.168.85.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP1-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.87.1 pref-src="" routing-table=to_ISP1 scope=30 suppress-hw-offload=no target-scope=10
add comment=ISP2-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.86.1 pref-src="" routing-table=to_ISP2 scope=30 suppress-hw-offload=no target-scope=10
add comment=ISP3-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.85.1 pref-src="" routing-table=to_ISP3 scope=30 suppress-hw-offload=no target-scope=10

#Changes made
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=1then2 distance=2 comment="ISP1-ST"
add dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=1then2 distance=3 comment="ISP2-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=1then3 distance=2 comment="ISP1-ST"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=1then3 distance=3 comment="ISP3-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then1 distance=2 comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=2then1 distance=3 comment="ISP1-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then3 comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=2then3 distance=3 comment="ISP3-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=3then1 distance=2 comment="ISP3-ST"
add dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=3then1 distance=3 comment="ISP1-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=3then2 distance=2 comment="ISP3-ST"
add dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=3then2 distance=3 comment="ISP2-ST"

#Changes made. Added "ISP1-ST" to update routes
/ip dhcp-client
add comment="ISP1 - Buckeye - DHCP Client" default-route-distance=255 \
    interface=ether1 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP1-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP1-ST\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP1-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP2 - T-Mobile - DHCP Client" default-route-distance=255 \
    interface=ether2 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP2-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP2-ST\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP2-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP3 - AT&T - DHCP Client" default-route-distance=255 interface=\
    ether3 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP3-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP3-ST\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP3-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
Last edited by MTNick on Fri Feb 23, 2024 4:50 am, edited 4 times in total.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 4:39 am

The missing letter was simply a typo....
You moved the rules around perfectly, but the point was not to move the 6/0 etc numbering, which as the rest should be in order 6/0 through 6/5, they dont follow the rule they were with previously.

Correct for the routing mark rules, they dont change.
Correct no change for the associated routes.
Last edited by Mesquite on Fri Feb 23, 2024 4:49 am, edited 1 time in total.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 4:48 am

Ahhhh got ya now. Apologies. Once again, I misunderstood. Updating...

Updated the config in the above post
Last edited by MTNick on Fri Feb 23, 2024 4:51 am, edited 1 time in total.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 4:51 am

Well think about it, the reason to do this last change was to spread out the new connections 1 to one ISP, then to the next ISP then to the next ISP, vice the first two connections to wan1, the next two etc..............
Therefore logically one has to not move the 6/0 from whatever is the first rule, 6/1 to the second rule, 6/3 to the third rule and so on.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 5:00 am

I over think things. That's for sure. I make one change & think it'll need changed in some way lol. Case in point the previous config I posted & then mucked it up. Over thinking it
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 5:20 am

Trust me, you have gotten this far faster than I................ and your attention to detail is very good!!
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 6:34 am

Thanks! Appreciate the confidence booster. You explain things very well & in detail. It's been a task to take these last few days lol. But like you said, we'll get there. Does this revision (fingers crossed) get the Mesquite stamp of approval??

/routing table
add fib name=to_ISP1
add fib name=to_ISP2
add fib name=to_ISP3
add fib name=1then2
add fib name=1then3
add fib name=2then1
add fib name=2then3
add fib name=3then1
add fib name=3then2

#No changes made
/ip firewall mangle
add action=mark-connection chain=forward comment=\
    "mark all new incoming connections" connection-mark=no-mark in-interface=\
    ether1 new-connection-mark=WAN1-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    in-interface=ether2 new-connection-mark=WAN2-conn passthrough=yes
add action=mark-connection chain=forward connection-mark=no-mark \
    in-interface=ether3 new-connection-mark=WAN3-conn passthrough=yes
add action=mark-routing chain=prerouting comment="force connection to table" \
    connection-mark=WAN1-conn in-interface-list=LAN new-routing-mark=to_ISP1 \
    passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2-conn \
    in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN3-conn \
    in-interface-list=LAN new-routing-mark=to_ISP3 passthrough=no

#Changes made & mangle rules reordered
/ip firewall mangle
add action=mark-connection chain=prerouting comment=\
    "divide traffic into six groups" connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=1then2_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=2then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=3then1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=1then3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=2then3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/4
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" dst-address-type=!local in-interface-list=LAN \
    new-connection-mark=3then2_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:6/5
add action=mark-routing chain=prerouting comment=\
    "distribute traffic across ISP's" connection-mark=1then2_conn \
    in-interface-list=LAN new-routing-mark=1then2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=1then3_conn \
    in-interface-list=LAN new-routing-mark=1then3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=2then1_conn \
    in-interface-list=LAN new-routing-mark=2then1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=2then3_conn \
    in-interface-list=LAN new-routing-mark=2then3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=3then1_conn \
    in-interface-list=LAN new-routing-mark=3then1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=3then2_conn \
    in-interface-list=LAN new-routing-mark=3then2 passthrough=no

#No changes made (the default routes created by dhcp-client (distance=255) don't get printed using "export file=")
/ip route
add check-gateway=ping comment=ISP1-MainTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=8.8.4.4 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping comment=ISP2-MainTable disabled=no distance=2 dst-address=0.0.0.0/0 gateway=9.9.9.9 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping comment=ISP3-MainTable disabled=no distance=3 dst-address=0.0.0.0/0 gateway=185.228.168.9 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add comment=ISP1-Recursive disabled=no distance=1 dst-address=8.8.4.4/32 gateway=192.168.87.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP2-Recursive disabled=no distance=2 dst-address=9.9.9.9/32 gateway=192.168.86.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP3-Recursive disabled=no distance=3 dst-address=185.228.168.9/32 gateway=192.168.85.1 pref-src="" routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add comment=ISP1-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.87.1 pref-src="" routing-table=to_ISP1 scope=30 suppress-hw-offload=no target-scope=10
add comment=ISP2-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.86.1 pref-src="" routing-table=to_ISP2 scope=30 suppress-hw-offload=no target-scope=10
add comment=ISP3-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.85.1 pref-src="" routing-table=to_ISP3 scope=30 suppress-hw-offload=no target-scope=10

#Changes made
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=1then2 distance=2 comment="ISP1-ST"
add dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=1then2 distance=3 comment="ISP2-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=1then3 distance=2 comment="ISP1-ST"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=1then3 distance=3 comment="ISP3-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then1 distance=2 comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=2then1 distance=3 comment="ISP1-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then3 comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=2then3 distance=3 comment="ISP3-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=3then1 distance=2 comment="ISP3-ST"
add dst-address=0.0.0.0/0 gateway=192.168.87.1 routing-table=3then1 distance=3 comment="ISP1-ST"
add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=3then2 distance=2 comment="ISP3-ST"
add dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=3then2 distance=3 comment="ISP2-ST"

#Changes made. Added "ISP1-ST" to update routes
/ip dhcp-client
add comment="ISP1 - Buckeye - DHCP Client" default-route-distance=255 \
    interface=ether1 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP1-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP1-ST\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP1-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP2 - T-Mobile - DHCP Client" default-route-distance=255 \
    interface=ether2 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP2-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP2-ST\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP2-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
add comment="ISP3 - AT&T - DHCP Client" default-route-distance=255 interface=\
    ether3 script=":if (\$bound=1) do={\r\
    \n:local gw \$\"gateway-address\"\r\
    \n/ip route set [ find comment=\"ISP3-Recursive\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP3-ST\" gateway!=\$gw ] gateway=\
    \$gw\r\
    \n/ip route set [ find comment=\"ISP3-SpecificTable\" gateway!=\$gw ] gate\
    way=\$gw\r\
    \n}" use-peer-dns=no use-peer-ntp=no
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Fri Feb 23, 2024 2:58 pm

add check-gateway=ping dst-address=0.0.0.0/0 gateway=192.168.86.1 routing-table=2then3 ............ comment="ISP2-ST"
add dst-address=0.0.0.0/0 gateway=192.168.85.1 routing-table=2then3 distance=3 comment="ISP3-ST"

( missing distance=2 but would work as the default is 1 lol )


Theoretically you could use the same comment ( a single comment ) for each ISP case requiring gateway substitution ( recursive, specific, ST ) and thus your script would have only one entry vice 3 different ones for each ISP - the executed action is the same!!
But to understand what we were doing it was necessary to keep them separate and its good for clear understanding.

Similarly we could remove the tables ISP1, ISP2, ISP3 ( the specific used for servers) and use available tables due to PCC for equivalent effect.
But to understand what we were doing it was necessary to keep them separate and its good for clear understanding.

for example...... take wan1.
add action=mark-routing chain=prerouting comment="force connection to table" \
connection-mark=WAN1-conn in-interface-list=LAN new-routing-mark=1then2 \
passthrough=no


add comment=ISP1-SpecificTable disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.87.1 pref-src="" routing-table=1then2 scope=30 suppress-hw-offload=no target-scope=10

+++++++++++++++++++++++++++++++++
If it was me I would keep the separate tables for clarity, but likely truncate the use of comments to ONE comment for each ISP, to update the gateway IP.
Last edited by Mesquite on Sat Feb 24, 2024 8:40 pm, edited 1 time in total.
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Sat Feb 24, 2024 5:26 pm

Hello Mesquite.

Happy to report that this is working beautifully. I've been testing it before reporting back. Server is not behind it, just some local resources mainly streaming YouTubeTV. I took your advise on changing the route comments & the dhcp-client script to only include 1 command to update "ISPx-SpecificTables" routes.

I can't say this enough, thank you for everything. The way you explain things makes it easy to follow. Appreciate all the help you've provided. As a matter of fact, from all the Mikrotik forum geniuses including yourself, ANAV (firewall), and others with various things. Thank you all for being here for us!
Screen Shot 2024-02-24 at 10.04.41 AM.png
Screen Shot 2024-02-24 at 10.07.25 AM.png
Screen Shot 2024-02-24 at 10.08.07 AM.png
You do not have the required permissions to view the files attached to this post.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Sat Feb 24, 2024 8:46 pm

To be honest, the geniuses are behind me, I am just channeling great advice I have gotten from others over the years.
Glad its working well.
You may want to play with PCC settings aka both addresses and ports as that combo is said to be the optimal for spreading connections among the WANs, but potentially get in the way of institutions like banks that are leery of connections coming at them from different WANS they think are related.....

I have enjoyed working with you, great pics and attention to detail and self-effort on your part goes a long WAY!!
 
MTNick
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 77
Joined: Fri Nov 24, 2023 6:43 am

Re: 3-ISPs Load Balancing - need help

Sun Feb 25, 2024 12:16 am

Thanks Mesquite. Appreciate the kind words. Went ahead & configured the main router with the final version. Thanks for the heads up to look at pcc setting if any web activity that isn't working as it should. Time will tell

To summarize what this thread accomplished:
1. 3 ISP (dynamic) w/Load balancing
2. PCC equal across all ISP's
3. ISP Recursive Failover
4. Script in dhcp-client to update gateways in routes when/if the dynamic IP's change
 
kivimart
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Thu Oct 10, 2013 3:06 pm

Re: 3-ISPs Load Balancing - need help

Mon Feb 26, 2024 10:42 am

This is just perfect and very nicely done, is there a possibility to show a final export?

I agree i must open my eyes on this Monday morning :-)
Last edited by kivimart on Mon Feb 26, 2024 3:47 pm, edited 1 time in total.
 
Mesquite
Member
Member
Posts: 420
Joined: Tue Jan 23, 2024 9:16 pm

Re: 3-ISPs Load Balancing - need help

Mon Feb 26, 2024 1:30 pm

?????? He has provided updated scripts throughout. ????
best documented work I have seen in some time even indicated if changes were made.
The pics are also germane and excellent.

Exactly what part dont you understand??
 
monotsc
just joined
Posts: 11
Joined: Sat Jun 30, 2018 7:15 am
Location: indonesia

Re: 3-ISPs Load Balancing - need help

Sun Mar 17, 2024 9:00 pm

LB with different ISP, did you guys consider how DNS based ISP optimized CDN like akamai for example ?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19409
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: 3-ISPs Load Balancing - need help

Sun Mar 17, 2024 11:19 pm

If you have an issue please start a new thread
 
monotsc
just joined
Posts: 11
Joined: Sat Jun 30, 2018 7:15 am
Location: indonesia

Re: 3-ISPs Load Balancing - need help

Mon Mar 18, 2024 12:21 am

@anav ahh no...it just in my experience, LB with diff ISP need small adjustment with static routing/fw addresss-list for each ISP networks so that packet with dst ISP-A networks not route via ISP-B networks which often result with more latency and less throughput

but it will be different story if client use DNS server that ignore eDNS like cloudflare 1.1.1.1

Who is online

Users browsing this forum: smx52, sutrus and 14 guests