I dont understand your mangling on example 1. You mark in-interface WAN traffic and then you mangle it again??? The second one makes no sense to me.
It thought you want to ensure in-interface=LAN1 and interface=LAN2 go out the right WAN correct…
I still think you devalue just a basic “check-gateway=ping” as an option 1. That does something for little effort (e.g. just the next hop – better than either 0 checking) – want more? move on to recursive routing…
I still wish MT do something like this:
as it allow better scripting of liveness routing & avoid recursive routes in more cases. Netwatch is become quite sophisticated of late. And, if there were a feature to link a route to a netwatch status, that let you define the monitored host in netwatch. A poor man’s BFD.
Agreed. As for the feature request, I concur. Regarding example 2, why not add PCC and perhaps a @Sindy optimization to minmized the cpu load when the number of mangle rules becomes sgnificant. 1) http://forum.mikrotik.com/t/static-default-route-im-missing-something/119183/1 2) http://forum.mikrotik.com/t/static-default-route-im-missing-something/119183/30
Hmm… I asked about adding pcc before and my nose was practically gone 
Haha … don’t be sorry, what can I do to ease your pain? 
Caress holvoe with your wings to ease his pain, but don’t accidentally nip him in the nose with those fangs… or are those front hanging testicles, hard to tell, maybe just dripping globules of phlegm. =-)
Oh man, this is gold dust! Please continue development of this topic.
Hi,
I encounter a strange issue today.
I had a failure on my primary link and another failure on my dynamic ip update script for my secondary link…
When the primary link came back, i tested my secondary link input rules and routes (when my 2 links are up and running).
A simple curl from an internet test server to my secondary ip link (LTE) took very long time (at least 20-30s) to answer.
However, output (NAT) from LAN to internet via the secondary link do not have any issue and work as expected.
Of course i do not have any issue on my primary link (input/output).
I defined a simple route in main table to force output to the server where i tested input (curl) and after that, it worked as expected. The requests on the secondary link answer very quick as expected.
In conclusion, there seems to be an routing issue, but i do not understand where i made a mistake.
My router is a CCR2116 with RouterOS 7.7.
Here is the config in attachment, could you please check if everything is fine ?
EDIT1 : i tried to use tools/Torch to check, and as soon as i enable Torch, everything work perfectly ! I stop Torch issue come back again !
EDIT2 : upgraded to the last 7.10.2, no change. I even tried to completely disable recursive monitoring for primary link, (and only use distance) no change
EDIT3 : in real failover (when primary link is really down), no issue, the secondary ip (failover link) work and answer very quickly
thanks !
router1.rsc (5.69 KB)
Cyayon DO NOT DOUBLE POST. Besides you should know better, this thread is to discuss the user article not solve your issues which CAN BE FOLLOWED HERE:
http://forum.mikrotik.com/t/modifying-route-distance-dual-wan/163194/50
( I would say our newly minted mods are asleep on the job)
The point Larsa is making is that simple works and works well.
Example one, just check-gateway=ping without recursive is probably what most users need
How often does the connection between ISP and WWW go down for me, well I cannot recall it every happening.
Example two, with recursive to show how to accomplish the principle of checking the connection from isp to www not just router to ISP.
Example three, would be to use a system script in conjunction with srcnat and maquerade to efficiently close connections when changing WANs.
The reason we would want this approach is because one is not happy with waiting 10 seconds twice… and thus the script can check every 3 or 5 seconds as desired
See I3 if interested 
Example four, would be netwatch which would accomplish same??
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Note the above assumes a simple primary failover scenario, not including
a. some users or subnets to one wan
b. external users coming in etc..
c. any mangling or routing rules and associated extra IP routes involved.
IN other words any talk of PCC in this article would be redonkulous and should be in a separate article as the combinations grow quickly.
Where netwatch-approach starts making sense IMO is with multiple LTE uplinks. In particular the new ICMP check that offers the ability to trigger failure on latency. This issue is LTE will often be able ping…but with significant RF/backhaul contention one LTE link may be dramatically better/worse at a particular time than another. While not exact, a poor LTE connection will show up in metrics such as jitter and RTT. But this is awful hard to explain since what to set and what to do depend on having some previous data from monitoring ICMP times. And the ICMP check will fail if ANY of the monitored parameters are out of range (even the default/unset ones), so tweaking these to avoid false-positives is kinda trial-and-error process.
Now if you have a relatively stable terrestrial internet option, I’m with @anav check-gateway=ping and setting distance= (or default-route-distance= in DHCP/PPP) is likely sufficient in a majority of cases. While recursive routing is more complete, complexity+mistakes in config can also cause outages (which against the goal to prevent them).
Bravo to pcunite for this post. I tried to follow the mikrotik wiki for dual wan in ros7 and they talk about virtual hops with no explanation.
I have managed to intregrate PCC Load Balance in my ros7.10.2 setup using pcunite’s as base. This is just the mangle. The routing section is the same.
/ip firewall mangle
add action=accept chain=prerouting dst-address-list="RFC1918" src-address-list="RFC1918"
---
Note: WAN1 = PPOE
add action=mark-connection chain=prerouting connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface-list=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface-list=LAN new-routing-mark=out-WAN1 passthrough=yes
---
Note: WAN2 = ether5 dhcp
add action=mark-connection chain=prerouting connection-state=new in-interface=ether5 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface-list=LAN new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface-list=LAN new-routing-mark=out-WAN2 passthrough=yes
---
add action=mark-connection chain=input connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=out-WAN1 passthrough=yes
---
add action=mark-connection chain=input connection-state=new in-interface=ether5 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=out-WAN2 passthrough=yes
Some thoughts and explanations of my logic:
For the longest time (around 2 days 
) I was confused whether I should use the same or different naming for the connection-mark for the mangles for pcc and the rest, but I realise that they are different connections, so using the same connection mark name is acceptable. Using WAN1 as example:
-
This is the fwd-chain connection from WAN1 to LAN.
add action=mark-connection chain=prerouting connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes -
This is the fwd-chain connection from LAN to WAN1.
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local
in-interface-list=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:2/0 -
This is the input-chain connection from WAN1 directly to the router (winbox).
add action=mark-connection chain=input connection-state=new in-interface=PPOE-WAN1 new-connection-mark=WAN1_conn passthrough=yes
Since they are all different connections (diff source and target), using the same connection mark is acceptable since they will not over-ride each other. Correct me if I am wrong in my understanding above, or if I am using incorrect terms.
PCC Load Balance is tested on my 2 WANs and is working great. Load balance only works for download, not upload. I wonder if you can PCC LB Uploads as well.
I am sure you are over complicating it LOL.
In any case the real fun begins in routing.
How will you ensure that traffic from 2/0 or 2/1 and if three WANS 3/0, or 3/1 or 3/2 will be covered by the other WAN.
In other words if one is forcing traffic to a WAN that is not available, how do you deal with that!
The answer is easy to see without recursive first, and then you can add recursive into the mix.
I am following pcunite’s recursive routing and so far when I disable WAN 1 and WAN 2 the failover seems to be reasonably seamless (except for the first minute or so after a WAN goes down).
/ip route
# Note: The next 6 routing rules are for WAN1 (recursive by checking 2 hosts, 8.8.8.8 and 208.67.222.222)
add disabled=no distance=11 dst-address=8.8.8.8/32 gateway=PPOE-WAN1 routing-table=main \
scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=12 dst-address=208.67.222.222/32 gateway=PPOE-WAN1 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add check-gateway=ping disabled=no distance=11 dst-address=0.0.0.0/0 gateway=8.8.8.8 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping disabled=no distance=12 dst-address=0.0.0.0/0 gateway=208.67.222.222 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add disabled=no distance=11 dst-address=0.0.0.0/0 gateway=8.8.8.8 \
routing-table=out-WAN1 scope=10 suppress-hw-offload=no target-scope=12
add disabled=no distance=12 dst-address=0.0.0.0/0 gateway=208.67.222.222 \
routing-table=out-WAN1 scope=10 suppress-hw-offload=no target-scope=12
----------
# Note: The next 6 routing rules are for WAN2 (recursive by checking 2 hosts, 8.8.4.4 and 208.67.220.220)
add disabled=no distance=21 dst-address=8.8.4.4/32 gateway=192.168.18.1 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=22 dst-address=208.67.220.220/32 gateway=192.168.18.1 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add check-gateway=ping disabled=no distance=21 dst-address=0.0.0.0/0 gateway=8.8.4.4 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add check-gateway=ping disabled=no distance=22 dst-address=0.0.0.0/0 gateway=208.67.220.220 \
routing-table=main scope=10 suppress-hw-offload=no target-scope=12
add disabled=no distance=21 dst-address=0.0.0.0/0 gateway=8.8.4.4 \
routing-table=out-WAN2 scope=10 suppress-hw-offload=no target-scope=12
add disabled=no distance=22 dst-address=0.0.0.0/0 gateway=208.67.220.220 \
routing-table=out-WAN2 scope=10 suppress-hw-offload=no target-scope=12
Well, it is a complicated topic.
I believe in this case, the PCC will send to routing table… but if route failed, then should still go main – even if PCC originally directed to a route table – since route table have no valid routes when check-gateway=ping fails.
asphri start your own thread if you want specific help on your config. I will cease and desist here as the thread is for the article…
Fair point. Load balancing seems like an extension of the failover described here… But I’d imagine it be popular topic… if YouTube view counts are any indication, the PCC YouTube video has relatively high 35K views. But the video doesn’t cover failure detection…
I do and I am adding to it with a second method, first is
A, B, C, D ( op decides below the order deemed fit )
1–>2–>3–>4
2–>1–>3–>4
3–>1–>2–>4
4–>1–>2–>3
The approach above basically asks the OP: to decide if WANA fails, ITS LOAD should go to WANB and if WANB also fails then WAN As load should go to C etc…
We also cover the scenario for each WAN so that at the end of the day, there are no PCC packets that are dropped.
The Second approach is to say if WANA fails, how do I distribute ITS load amongst the other ThREE… NOT just dump on ONE wan!!!
See I4 LOL.
Well I was responding to your: “In other words if one is forcing traffic to a WAN that is not available, how do you deal with that!” and hopefully to help others since my PCC mangles seem to integrate well (so far) with pcunite’s guide.
I chose to add my thoughts to generate discussion and to help others with understanding mangles along the way. Since we all find tinkering with mikrotik is fun (I added a 2nd isp to my house just to test PCC LB and I might add a 3rd to test your conjecture that LB not dump its load on 1 WAN connection in the case of 3 or more WANs and 1 failure).
Essentially I feel that my posts builds on pcunite’s guide and did generate discussion. Thanks for your thought provoking previous post.
Not my topic, but I think all contributions are welcome. Just the sub-discussions get confusing to readers sometimes.
The simpler approach to load balancing with recursive routes is just taking @pcunite’s example and setting the distance= to be the same. This creates an ECMP-based load balancing. ECMP is not as controllable as PCC but routes with same distance are load balanced by default without more mangles:
Because results of the forwarding decision are cached, packets with the same source address, destination address, source interface, routing mark and ToS are sent to the same gateway. This means that ECMP route does not perform pure per-connection balancing, but it can be used to load balance connections if at least one of previously mentioned parameters is different than previous connection.
…and you can “cheat” ECMP using additional recursive routes with same distance and same gateway (but different canary/“8.8.8.8”) to achieve asymmetric load balancing.