Community discussions

MikroTik App
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

MultiWAN with RouterOS

Wed Jan 18, 2023 8:01 pm

Title:
MultiWAN with RouterOS

Welcome:
This article aims to bring clarity to the daunting and confusing task of routing multiple WAN and ISP provider connections in and out of your network. We will be using RouterOS version 7 firmware to accomplish these techniques. Examples will include Static IP wan connections, DHCP assignments, PPPoE encapsulations, and using LTE as a backup or main internet connection. Each style has its own unique requirements to get the most out of the WAN availability. What happens when one goes down? Do you want to dedicate one ISP to a certain type of traffic? How does incoming traffic reach internal targets? These solutions and more will be discussed.

MultiWAN is not SD-WAN:
Utilizing more than one internet provider is a way to bring more capacity and uptime to your network. If you have latency sensitive traffic, it can be helpful to put it on another WAN connection. However, multiple wan connections by themselves don't necessarily create a seamless experience for end applications because of the way current TCP protocols are designed. An ISP failover may result in temporary application disruption.

Advanced applications can cope if they are programmed a certain way but streaming style applications are very suspect of background changes. There is a way, using a layer of external servers and routers, to achieve that high level of ISP float technique, where the applications are not aware of what is happening below. Illustrating this is not the purpose of this document which instead focuses on a more rudimentary way to manage multiple internet providers given you only have one router to manage.

Why MultiWAN?
This sounds like a trick question. If a single provider had amazing latency, enormous bandwidth, and never experienced any downtime, why indeed would there be the need for another network? I suppose there would not be much of a need. But in the real world networks aren't everything we wish they were. Sometimes internet providers can only offer so much latency, or they go down from events taking place in the area.

Sometimes it is more cost effective to put an application on one ISP and the main bulk of your network on another. There is also the possibility that you have a physically different connection to a network resource and pathway you own and control. Still another scenario is for when networks come and go based on where you might be. The router maybe a moving device and thus different wireless networks come and go. They don't fail exactly, they just are not there for awhile.

MultiWAN.png

MultiWAN Overview:
There is a lot to cover so make sure you read slowly. This article series will illustrate a scenario that will have several moving parts. You will not implement everything because your environment is different and custom to you. This means you will not be able to simply copy/past the examples. You will need to add, remove, or customize what has been presented. To do that, you will need to understand the mono sized router example.

The example shows a single router having many WAN connections. You will not have this exact arrangement. Do you have LTE? If not, do not implement that section of the example. A best effort has been made to create a single example that can easily shed various parts that are not needed. This article assumes a VLAN environment. So, consider that a requirement in your learning journey.


Disclaimer:
What follows is my best understanding of how to implement the stated goals in RouterOS v7 based on the generous feedback from many forum community members. I am a student and spend time in the forums to learn and give back when I can. I am not an expert nor am I even a forum guru as my forum title humorously states. There are far more qualified persons in attendance. My skill is taking what others have shared and then building a presentation around their thoughts and techniques. Thank you. Note that this article is new and has many mistakes. It will be updated many times until the community considers it the gold standard.
You do not have the required permissions to view the files attached to this post.
Last edited by pcunite on Fri Feb 03, 2023 12:16 am, edited 4 times in total.
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 8:02 pm

Example1:
We will lead into this topic using a generic example that covers the majority of situations. We have four WAN connections and need a method to determine when a primary connection is down. When primary is deemed down, the router should switch the environment over to another WAN network. Because we have multiple WAN connections, some traffic is configured to use a specific WAN. Static, DHCP, PPPoE, and LTE style WAN types are shown. Remove or rearrange assignments that don't apply to you.

Example1.png

Are we down?
Finding out if a network is down is tricky. The absolute best way would be an application inside your network that makes connections to resources outside your network for every type of application you deem important. That would be a very special application. Lacking that, a low cost, easy to use, and included with RouterOS option, is to use a technique called recursive nexthop lookup (aka recursive routing). This just means that we validate the entire path to a host, instead of just the one connected to our router directly. This way, your WAN's connection to the internet and thus outside of your ISP network, can be verified.

A caveat of course is that the host you are checking is outside your ISP and is itself not down. Because a single host could be down it is therefore highly recommended to check two separate hosts. You could check as many as you feel warrant a decision. Here we show two.

Gateway Ping
Route verification is performed with ping checks. Every ten seconds a ping is sent to a remote Host. Failing that, another ping is attempted. Two failed ping replies will set the route as invalid and unreachable. The check-gateway parameter helps us to accomplish this but not on its own. It is necessary to link two route commands with each other.

Scope and Target Scope
To validate a path to a remote host, we use route entries that are connected to each other over a Scope and Target Scope arrangement. These two parameters are an unfortunate abstraction that we as network administrators must deal with. I'll attempt an explanation. Think of scope as the area of your concern. How much of an area, how far, how deep, and how wide of an area do you care about? This is your scope size.

Scope.png

Target Scope would then be the next area you care about. Since we are only concerned with next hop paths, the way we tell RouterOS to use a particular route as a next hop to be validated, is to set the Target-Scope higher than the Scope, effectively increasing the size of your standard scope. It takes two command lines to show this awkward representation and linking.

Forum member anav has beautifully hacked this concept by always setting the default Scope to 10 and using the Target Scope parameter to change the relationship. Note that two linked route entries is enough for validation as shown in the simplified diagram. But in our examples, you will note that we also add one more route for return traffic. This is because we want the ability to use the other ISP connections even when primary is up.

If we are up
When both networks are up, we show other networks being utilized instead of leaving them always idle. We also enable traffic to connect remotely into the network from any available WAN. Also shown is having certain traffic always leave out of a specific WAN.


TypesOfWAN.png

DHCP WAN Type
If you have a DHCP connection to your ISP, you will note that we use a static route in the example. Attached to your DHCP client, is a curious script provided by rextended (who is author?). With a static route, the script becomes necessary if the ISP changes the IP Address or gateway for any reason. The script keeps your dhcp client and manually added route in sync.

When this event occurs, the script will fire and compare the client gateway value with any route that has a comment of ISP2_Monitor. The script reads like so: if this dhcp client has an ip address, search through all routes for a route in which the gateway does not match our own and which has a comment of "ISP2_Monitor". If both conditions are true, change the gateway value.

LTE WAN Type
If you have LTE availability in your area, the best way to utilize this service is with an LTE enabled MikroTik router running ROS v7 which has better support for modems and behavior. To us, these hockey pucks are radios and that's about all. To that end, enable the passthrough interface feature in the APN configuration. However, if the LTE hardware only has one ethernet interface (or you only want to use one interface), you'll loose the ability to manage the LTE unit. This is not a problem, however with a simple VLAN between the two devices. Our example shows this arrangement. See the LTE Router Example linked below.


MultiWAN Router Example1
Example1.rsc

LTE Router Example
LTE_Router.rsc
You do not have the required permissions to view the files attached to this post.
Last edited by pcunite on Thu Sep 21, 2023 6:27 pm, edited 41 times in total.
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 8:02 pm

Reserved
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 8:03 pm

Reserved
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 8:03 pm

Reserved
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 10:26 pm

Excellent pcunite..........if only my posts were so well put together as your approach LOL.
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 11:36 pm

@anav,

That means a lot. Your help on the forums is felt and you've helped me personally. You know more than me! These long form articles take a long time to produce. So, don't feel bad about that. We all have our own strengths. This is a way for me to give back.

Give me a few weeks to get this article in order and then we can hammer on it. I'm writing, producing content as I go. Then I'll publish out the syntax.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5326
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: MultiWAN with RouterOS

Wed Jan 18, 2023 11:41 pm

If it is the same quality as your vlan series, it will be a great contribution !
Subscribed to get the updates.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Thu Jan 19, 2023 6:34 am

Good idea! Some significant number of posts on the forum involve some kinda "MultiWAN situation". And @anav was going to run out of letters in his "new user" post.

I like the approach starting with the 4 WAN types you have. Think focus on "recursive lookup" (vs "netwatch techniques") for upstream failure detection seems a good call... I use RRs (with PBR) today, but with all the new netwatch detection mechanism in 7.7, I can see a well-design script doing better than RR – but as generic/"general purpose" approach might be tough to explain. Now I guess, depending on when and what form BFD comes, that may offer a 3rd failure detection option – but that's still not in V7 to even consider.

Anyway, I'll follow with interest. Glad someone is taking this on – MT's docs have long lacked on any kinda of respectable "user guide", and offer very limited cookbook/recipe style docs to even hint at some canonical network architectures/designs.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: MultiWAN with RouterOS

Thu Jan 19, 2023 12:00 pm

@pcunite

Bravo.
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Thu Jan 19, 2023 2:53 pm

... focus on "recursive lookup" (vs "netwatch techniques") for upstream failure detection seems a good call ... with the new netwatch detection mechanism in 7.7, I can see a well-design script doing better than RR ... when BFD comes, that may offer a 3rd failure detection option.

I agree. I hope to see a new netwatch routine make it into this series. In the opening graphic, the text bubble states: "recursive lookup or netwatch techniques". I wanted to lead with RR because it seems more popular on the forums. So, I speak to that. It is a mess to make sense of in the route menu, however. I would personally prefer a script, but that is going pretty deep into the woods. Actually, I would prefer an application in RouterOS to identify and process a failed network. I do hope to engage yourself and rextended to create something easy to follow as an alternative to RR. I'll have some ideas to offer when we get to that section. BFD will take the forum by storm because sockets are the best way to identify and process downed networks.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5326
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: MultiWAN with RouterOS

Thu Jan 19, 2023 3:06 pm

Is PCC also going to be part of the scope ?
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Fri Jan 20, 2023 9:35 pm

Is PCC also going to be part of the scope ?

A perfectly cooked beef tenderloin served with buttered mashed potatoes, lemon roasted asparagus, and ranch covered salad spears is not enough for you? You want ... dessert too?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Sat Jan 21, 2023 1:31 am

Pssst he's Belgium has a France complex, dont mention desserts!!!
 
holvoetn
Forum Guru
Forum Guru
Posts: 5326
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: MultiWAN with RouterOS

Sat Jan 21, 2023 10:15 am

1 i don't like dessert
2 isn't pcc suitable for multi wan ?
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Sat Jan 21, 2023 3:07 pm

I'm sure they'll be time for a debate about load balancing, e.g. PCC vs ECMP vs .... But in reality you almost always want some failure detection, regardless if a failover or load balancing case.

Perhaps a nod to the "check-gateway=ping" that's required in routes, before explaining the scopes and recursive routing? That's actually what triggers/blocks the route recursion.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Thu Feb 02, 2023 6:45 pm

pcunite I noticed on ex1, if I am not mistaken, you are using 1.1.1.1 for an ISP address?? This is not an ideal choice as that is the IP address for clouldflare DNS services and I happen to use this as a host to check my recursive routes......... very confusing when I saw that next to ISP............ ?????? Please change...
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Thu Feb 02, 2023 7:05 pm

you are using 1.1.1.1 for an ISP address?

Opps, yes I had that incorrect in the diagram. Thank you. The rsc file was correct, however. DNS Host1 is 1.1.1.1, it is what the example is pinging. The ISP is 10.1.1.1 and the recursive route is naturally 1.1.1.1.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Thu Feb 02, 2023 7:10 pm

Can you explain how your script works ( ip dhpc script in first example)
a. what each command is invoking but in english and not script language, in other words right the script in words,
b. what does it do functionally
c. why is it needed.
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Fri Feb 03, 2023 9:58 pm

Okay, Example1 is ready for intensive criticism. What could be more clear? Are there any errors?

A note about Example1:
This is a recursive routing example. It is supposed to stand on its own as a deliverable. It answers the question: "How do I wire up multiple WANs and make sending/receiving, custom routes in/out, and fail-over just work?" It is simple and will get people going with the basics. However, let's see the kinda of questions we get.

For Example 2, maybe we have a scripting version (Netwatch) instead of recursive? Example3 would be WAN Load Balancing per holvoetn request?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Wed Feb 22, 2023 9:26 pm

I dont understand your mangling on example 1. You mark in-interface WAN traffic and then you mangle it again???? The second one makes no sense to me.
It thought you want to ensure in-interface=LAN1 and interface=LAN2 go out the right WAN correct............
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Wed Feb 22, 2023 11:22 pm

For Example 2, maybe we have a scripting version (Netwatch) instead of recursive?
I still think you devalue just a basic "check-gateway=ping" as an option 1. That does something for little effort (e.g. just the next hop – better than either 0 checking) – want more? move on to recursive routing...

I still wish MT do something like this:
viewtopic.php?t=192844
as it allow better scripting of liveness routing & avoid recursive routes in more cases. Netwatch is become quite sophisticated of late. And, if there were a feature to link a route to a netwatch status, that let you define the monitored host in netwatch. A poor man's BFD.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: MultiWAN with RouterOS

Wed Feb 22, 2023 11:59 pm

Agreed. As for the feature request, I concur. Regarding example 2, why not add PCC and perhaps a @Sindy optimization to minmized the cpu load when the number of mangle rules becomes sgnificant. 1) viewtopic.php?t=134048#p659676, 2) viewtopic.php?p=962647#p962647
 
holvoetn
Forum Guru
Forum Guru
Posts: 5326
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: MultiWAN with RouterOS

Thu Feb 23, 2023 12:21 am

Hmm... I asked about adding pcc before and my nose was practically gone :shock:

:lol:
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: MultiWAN with RouterOS

Thu Feb 23, 2023 12:29 am

Haha ... don't be sorry, what can I do to ease your pain? :-D
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Thu Feb 23, 2023 2:16 am

Caress holvoe with your wings to ease his pain, but don't accidentally nip him in the nose with those fangs....... or are those front hanging testicles, hard to tell, maybe just dripping globules of phlegm. =-)
 
CountGoosila
just joined
Posts: 1
Joined: Thu Feb 23, 2023 5:38 am

Re: MultiWAN with RouterOS

Thu Feb 23, 2023 7:03 am

Oh man, this is gold dust! Please continue development of this topic.
 
cyayon
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Wed Aug 24, 2022 9:39 am

Re: MultiWAN with RouterOS

Wed Jul 12, 2023 9:11 pm

Hi,

I encounter a strange issue today.
I had a failure on my primary link and another failure on my dynamic ip update script for my secondary link...

When the primary link came back, i tested my secondary link input rules and routes (when my 2 links are up and running).
A simple curl from an internet test server to my secondary ip link (LTE) took very long time (at least 20-30s) to answer.
However, output (NAT) from LAN to internet via the secondary link do not have any issue and work as expected.
Of course i do not have any issue on my primary link (input/output).

I defined a simple route in main table to force output to the server where i tested input (curl) and after that, it worked as expected. The requests on the secondary link answer very quick as expected.
In conclusion, there seems to be an routing issue, but i do not understand where i made a mistake.

My router is a CCR2116 with RouterOS 7.7.
Here is the config in attachment, could you please check if everything is fine ?

EDIT1 : i tried to use tools/Torch to check, and as soon as i enable Torch, everything work perfectly ! I stop Torch issue come back again !
EDIT2 : upgraded to the last 7.10.2, no change. I even tried to completely disable recursive monitoring for primary link, (and only use distance) no change
EDIT3 : in real failover (when primary link is really down), no issue, the secondary ip (failover link) work and answer very quickly

thanks !
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Thu Jul 13, 2023 7:15 pm

Cyayon DO NOT DOUBLE POST. Besides you should know better, this thread is to discuss the user article not solve your issues which CAN BE FOLLOWED HERE:

viewtopic.php?p=1012786#p1012786


( I would say our newly minted mods are asleep on the job)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Mon Oct 30, 2023 2:13 am

The point Larsa is making is that simple works and works well.

Example one, just check-gateway=ping without recursive is probably what most users need
How often does the connection between ISP and WWW go down for me, well I cannot recall it every happening.

Example two, with recursive to show how to accomplish the principle of checking the connection from isp to www not just router to ISP.

Example three, would be to use a system script in conjunction with srcnat and maquerade to efficiently close connections when changing WANs.
The reason we would want this approach is because one is not happy with waiting 10 seconds twice..................... and thus the script can check every 3 or 5 seconds as desired
See I3 if interested ;-)

Example four, would be netwatch which would accomplish same??

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Note the above assumes a simple primary failover scenario, not including
a. some users or subnets to one wan
b. external users coming in etc..
c. any mangling or routing rules and associated extra IP routes involved.

IN other words any talk of PCC in this article would be redonkulous and should be in a separate article as the combinations grow quickly.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Mon Oct 30, 2023 4:30 am

The point Larsa is making is that simple works and works well.
[...]
Example four, would be netwatch which would accomplish same??
Where netwatch-approach starts making sense IMO is with multiple LTE uplinks. In particular the new ICMP check that offers the ability to trigger failure on latency. This issue is LTE will often be able ping...but with significant RF/backhaul contention one LTE link may be dramatically better/worse at a particular time than another. While not exact, a poor LTE connection will show up in metrics such as jitter and RTT. But this is awful hard to explain since what to set and what to do depend on having some previous data from monitoring ICMP times. And the ICMP check will fail if ANY of the monitored parameters are out of range (even the default/unset ones), so tweaking these to avoid false-positives is kinda trial-and-error process.

Now if you have a relatively stable terrestrial internet option, I'm with @anav check-gateway=ping and setting distance= (or default-route-distance= in DHCP/PPP) is likely sufficient in a majority of cases. While recursive routing is more complete, complexity+mistakes in config can also cause outages (which against the goal to prevent them).
 
ashpri
Member Candidate
Member Candidate
Posts: 154
Joined: Sun Oct 14, 2018 3:11 am

Re: MultiWAN with RouterOS

Tue Oct 31, 2023 6:08 am

Bravo to pcunite for this post. I tried to follow the mikrotik wiki for dual wan in ros7 and they talk about virtual hops with no explanation.

I have managed to intregrate PCC Load Balance in my ros7.10.2 setup using pcunite's as base. This is just the mangle. The routing section is the same.

/ip firewall mangle

add action=accept chain=prerouting dst-address-list="RFC1918" src-address-list="RFC1918"

---

Note: WAN1 = PPOE
add action=mark-connection chain=prerouting connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface-list=LAN new-routing-mark=out-WAN1 passthrough=yes

---

Note: WAN2 = ether5 dhcp
add action=mark-connection chain=prerouting connection-state=new in-interface=ether5 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface-list=LAN new-routing-mark=out-WAN2 passthrough=yes

---

add action=mark-connection chain=input connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=out-WAN1 passthrough=yes

---

add action=mark-connection chain=input connection-state=new in-interface=ether5 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=out-WAN2 passthrough=yes

------------

Some thoughts and explanations of my logic:

For the longest time (around 2 days :) ) I was confused whether I should use the same or different naming for the connection-mark for the mangles for pcc and the rest, but I realise that they are different connections, so using the same connection mark name is acceptable. Using WAN1 as example:

1. This is the fwd-chain connection from WAN1 to LAN.
add action=mark-connection chain=prerouting connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes

2. This is the fwd-chain connection from LAN to WAN1.
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface-list=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:2/0

3. This is the input-chain connection from WAN1 directly to the router (winbox).
add action=mark-connection chain=input connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes

Since they are all different connections (diff source and target), using the same connection mark is acceptable since they will not over-ride each other. Correct me if I am wrong in my understanding above, or if I am using incorrect terms.

-----------

PCC Load Balance is tested on my 2 WANs and is working great. Load balance only works for download, not upload. I wonder if you can PCC LB Uploads as well.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Tue Oct 31, 2023 2:54 pm

I am sure you are over complicating it LOL.
In any case the real fun begins in routing.
How will you ensure that traffic from 2/0 or 2/1 and if three WANS 3/0, or 3/1 or 3/2 will be covered by the other WAN.
In other words if one is forcing traffic to a WAN that is not available, how do you deal with that! :-)

The answer is easy to see without recursive first, and then you can add recursive into the mix.
 
ashpri
Member Candidate
Member Candidate
Posts: 154
Joined: Sun Oct 14, 2018 3:11 am

Re: MultiWAN with RouterOS

Wed Nov 01, 2023 6:14 am

In any case the real fun begins in routing.
How will you ensure that traffic from 2/0 or 2/1 and if three WANS 3/0, or 3/1 or 3/2 will be covered by the other WAN.

I am following pcunite's recursive routing and so far when I disable WAN 1 and WAN 2 the failover seems to be reasonably seamless (except for the first minute or so after a WAN goes down).

/ip route

# Note: The next 6 routing rules are for WAN1 (recursive by checking 2 hosts, 8.8.8.8 and 208.67.222.222)

add disabled=no distance=11 dst-address=8.8.8.8/32 gateway=PPOE-WAN1 routing-table=main \
    scope=10 suppress-hw-offload=no target-scope=11

add disabled=no distance=12 dst-address=208.67.222.222/32 gateway=PPOE-WAN1 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=11

add check-gateway=ping disabled=no distance=11 dst-address=0.0.0.0/0 gateway=8.8.8.8 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=12

add check-gateway=ping disabled=no distance=12 dst-address=0.0.0.0/0 gateway=208.67.222.222 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=12

add disabled=no distance=11 dst-address=0.0.0.0/0 gateway=8.8.8.8 \
    routing-table=out-WAN1 scope=10 suppress-hw-offload=no target-scope=12

add disabled=no distance=12 dst-address=0.0.0.0/0 gateway=208.67.222.222 \
    routing-table=out-WAN1 scope=10 suppress-hw-offload=no target-scope=12

----------

# Note: The next 6 routing rules are for WAN2 (recursive by checking 2 hosts, 8.8.4.4 and 208.67.220.220)

add disabled=no distance=21 dst-address=8.8.4.4/32 gateway=192.168.18.1 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=11

add disabled=no distance=22 dst-address=208.67.220.220/32 gateway=192.168.18.1 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=11

add check-gateway=ping disabled=no distance=21 dst-address=0.0.0.0/0 gateway=8.8.4.4 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=12

add check-gateway=ping disabled=no distance=22 dst-address=0.0.0.0/0 gateway=208.67.220.220 \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=12

add disabled=no distance=21 dst-address=0.0.0.0/0 gateway=8.8.4.4 \
    routing-table=out-WAN2 scope=10 suppress-hw-offload=no target-scope=12

add disabled=no distance=22 dst-address=0.0.0.0/0 gateway=208.67.220.220 \
    routing-table=out-WAN2 scope=10 suppress-hw-offload=no target-scope=12
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Wed Nov 01, 2023 7:40 am

I am sure you are over complicating it LOL.
Well, it is a complicated topic. ;)

I believe in this case, the PCC will send to routing table... but if route failed, then should still go main – even if PCC originally directed to a route table – since route table have no valid routes when check-gateway=ping fails.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Wed Nov 01, 2023 7:11 pm

asphri start your own thread if you want specific help on your config. I will cease and desist here as the thread is for the article...........
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Wed Nov 01, 2023 7:30 pm

asphri start your own thread if you want specific help on your config. I will cease and desist here as the thread is for the article...........
Fair point. Load balancing seems like an extension of the failover described here... But I'd imagine it be popular topic... if YouTube view counts are any indication, the PCC YouTube video has relatively high 35K views. But the video doesn't cover failure detection...
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Thu Nov 02, 2023 12:54 am

I do and I am adding to it with a second method, first is
A, B, C, D ( op decides below the order deemed fit )
1-->2-->3-->4
2-->1-->3-->4
3-->1-->2-->4
4-->1-->2-->3

The approach above basically asks the OP: to decide if WANA fails, ITS LOAD should go to WANB and if WANB also fails then WAN As load should go to C etc....
We also cover the scenario for each WAN so that at the end of the day, there are no PCC packets that are dropped.

The Second approach is to say if WANA fails, how do I distribute ITS load amongst the other ThREE......... NOT just dump on ONE wan!!!

See I4 LOL.
 
ashpri
Member Candidate
Member Candidate
Posts: 154
Joined: Sun Oct 14, 2018 3:11 am

Re: MultiWAN with RouterOS

Thu Nov 02, 2023 2:14 am

asphri start your own thread if you want specific help on your config. I will cease and desist here as the thread is for the article...........

Well I was responding to your: "In other words if one is forcing traffic to a WAN that is not available, how do you deal with that!" and hopefully to help others since my PCC mangles seem to integrate well (so far) with pcunite's guide.

I chose to add my thoughts to generate discussion and to help others with understanding mangles along the way. Since we all find tinkering with mikrotik is fun (I added a 2nd isp to my house just to test PCC LB and I might add a 3rd to test your conjecture that LB not dump its load on 1 WAN connection in the case of 3 or more WANs and 1 failure). :)

Essentially I feel that my posts builds on pcunite's guide and did generate discussion. Thanks for your thought provoking previous post.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Thu Nov 02, 2023 3:11 am

Not my topic, but I think all contributions are welcome. Just the sub-discussions get confusing to readers sometimes.

The simpler approach to load balancing with recursive routes is just taking @pcunite's example and setting the distance= to be the same. This creates an ECMP-based load balancing. ECMP is not as controllable as PCC but routes with same distance are load balanced by default without more mangles:
Because results of the forwarding decision are cached, packets with the same source address, destination address, source interface, routing mark and ToS are sent to the same gateway. This means that ECMP route does not perform pure per-connection balancing, but it can be used to load balance connections if at least one of previously mentioned parameters is different than previous connection.
...and you can "cheat" ECMP using additional recursive routes with same distance and same gateway (but different canary/"8.8.8.8") to achieve asymmetric load balancing.
 
S8T8
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Thu Sep 15, 2022 7:15 pm

Re: MultiWAN with RouterOS

Fri Nov 03, 2023 7:23 pm

Hello, thanks pcunite for this very informative and well-explained post!
I'm using a different approach, without routing tables and mangle rules, different scope values, adapting suggestions (probably in the wrong way) from @sindy posts .
add comment="WAN1" distance=1 dst-address=1.1.1.1 gateway=(WAN1.Gateway) scope=10 target-scope=10
add comment="WAN2" distance=1 dst-address=8.8.8.8 gateway=(WAN2.Gateway) scope=10 target-scope=10
add check-gateway=ping comment="ISP1" distance=1 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=30 target-scope=11
add check-gateway=ping comment="ISP2" distance=3 dst-address=0.0.0.0/0 gateway=8.8.8.8 scope=30 target-scope=11

Few questions;
- what is the advantage of having routing tables?
- "Server that will only use ISP2" in Example1, is possible to achieve the same goal just with routing rule?
- /ip firewall connection tracking set loose-tcp-tracking=no
loose-tcp-tracking , there are reasons to disable it? Based on this topic and this article is suggested to leave it on
loose-tcp-tracking=no -> Stric
loose-tcp-tracking=yes -> Loose
Last edited by S8T8 on Sun Nov 05, 2023 11:25 pm, edited 1 time in total.
 
cyasistemas
just joined
Posts: 1
Joined: Sat Nov 04, 2023 12:24 am

Re: MultiWAN with RouterOS

Sat Nov 04, 2023 1:24 am

it is possible the software engineers from mikrotik program a easy multi wan/failover feature in router os? i saw it in other brands like tp-link
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Sat Nov 04, 2023 3:05 am

Nope, its up to the MT user to figure it out. They give you all the tools to do it..............
 
User avatar
pcunite
Forum Guru
Forum Guru
Topic Author
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: MultiWAN with RouterOS

Sun Nov 05, 2023 4:31 am

Few questions;
1: what is the advantage of having routing tables?
2: "Server that will only use ISP2" in Example1, is possible to achieve the same goal without routing rule?
3: firewall tracking set loose-tcp-tracking=no there are reasons to disable it? Based on this topic

There are those who can better explain. However, I'll try.

Questions 1 & 2:
In our mangle rules we note the interface packet are entering/exiting and assign them to a table via new-routing-mark=ISP2, etc. This allows us to force an internal server to send data out and get replies back via this "table" concept. If we did not do this, the server would send data out the "default" route or any other other than our preferred ISP2. If you do not have this requirement, naturally you don't need to do all this.

Question 3:
I may have gotten this wrong. Test and report.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 12:30 am

Q3. No you are correct, NOT CHECKED< we want it OFF. Its a potential security risk if checked ON, when the router crashes and recovers.............
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 3:22 am

Q3. No you are correct, NOT CHECKED< we want it OFF. Its a potential security risk if checked ON, when the router crashes and recovers.............
Agree that likely best default. I wouldn't security risk however...more hygiene.

All Multi-WAN roads go through firewall connection tracking, so the "loose-tcp-tracking=no" is will cause WAN re-selection if the TCP 3-way handshake doesn't match the TCP connection state in /ip/firewall/connections. So the alternative choice =yes will be more sticky.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 4:36 am

Good hygiene=less illness=troops-are-available=better security LOL

/ip firewall: Connection Tab settings (Tracking Button): Do NOT check box for Loose Tracking

It's one thing if you reboot an ISP's router, it's one thing that a home user's router reboots or crashes.
What do you care about trying to restore sessions that already exist?
By now the network has been interrupted for minutes, the reboot is not instantaneous, and the router does not know the previous NATs when it is rebooted,
so what does it do? It puts them at random... and may "open" connections to unsolicited ports or services, which is a security hazard.
 
S8T8
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Thu Sep 15, 2022 7:15 pm

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 7:28 pm

Questions 1 & 2:
Thanks @pcunite, appreciated, you know better than many of us!
Question 3 we had confirmation from @anav and @Amm0 that loose-tcp-tracking=no is OK.
Question 1, your point is clear.
Question 2, as your example, the server is routed only via specific ISP or Wireguard from my example, this could be a solution to use without mangle rules?
#pasted from @anav post
/routing table add  fib name=useWG
/ip route add dst-address=0.0.0.0/0 gateway=wg0  routing-table=useWG
/routing rule add src-address=192.168.10.20/32  action=lookup table=useWG
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 8:02 pm

Question 3 we had confirmation from @anav and @Amm0 that loose-tcp-tracking=no is OK.
I don't know of any cases where it isn't ok. But I'm sure there exist some mis-behaving TCP application/middlebox that don't follow rules where it be a might an issue.

The high-level is the /ip/firewall/connections is what stores the "MultiWAN decision" – so it's any of conntrack settings may have some effect on failover logic (e.g. there are configurable UDP timeouts too)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 9:22 pm

Questions 1 & 2:
Thanks @pcunite, appreciated, you know better than many of us!
Question 3 we had confirmation from @anav and @Amm0 that loose-tcp-tracking=no is OK.
Question 1, your point is clear.
Question 2, as your example, the server is routed only via specific ISP or Wireguard from my example, this could be a solution to use without mangle rules?
#pasted from @anav post
/routing table add  fib name=useWG
/ip route add dst-address=0.0.0.0/0 gateway=wg0  routing-table=useWG
/routing rule add src-address=192.168.10.20/32  action=lookup table=useWG
YES but do consider that if 192.168.10.20/32 needs to receive or send traffic to local subnets on the router you will need more routing rules before this one that forces ALL its traffic to wireguard.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3169
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 10:50 pm

Few questions;
2: "Server that will only use ISP2" in Example1, is possible to achieve the same goal without routing rule?
Questions 1 & 2:
If you do not have this requirement, naturally you don't need to do all this.
Sure, but the routing tables are akin to the using VLAN bridging for LANs. If you only have one LAN, you technical don't need do deal with VLANs...but if you setup VLANs at start, you setup for future changes without dramatic changes. Similar with routing tables, plenty of ways to use firewall rules (or WG allowed-addresses, etc.) avoid the need...but there is some logically consistency in using tables.

e.g. the through line between @pcunite's article series is to use the various RouterOS "tables" (address-list, interface-list, bridge vlans, etc.) to define the network specifics to avoid need to make more complex changes to the firewall/routing
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18958
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: MultiWAN with RouterOS

Mon Nov 06, 2023 11:04 pm

For firewall rules --- when Firewall rules apply to groups of users................

interfaces are best for two or more whole subnets ( exception: management subnet may the only subnet in such an interface )
Dst or Src address are best to describe a single subnet
Firewall addresses are best for any situation requiring a large number of users in a subnet (but not the whole subnet), or a bunch of users across several subnets (with or without whole subnets).

Who is online

Users browsing this forum: No registered users and 6 guests