New User Pathway To Config Success

anav · Tue Jan 18, 2022 9:20 pm

NEW USER PATHWAY TO SUCCESS [ if you find a way to run into problems despite the guides, let me know where to improve

]

HAVING ISSUES AND NEED HELP READ THIS FIRST BEFORE POSTING - viewtopic.php?p=908118

PUBLIC BANDWIDTH TEST SERVERS - viewtopic.php?t=104266

A. Configuring Off Bridge
B. Firewall
C. VLAN Filtering
D. AP/Switch RoS Setup
E. Destination NAT (Port Forwarding, hairpin nat & pihole/adguard)
F. Wireguard
G. Backup/Export Config
H. NetInstall & Protected Router Boot Mode (PRBM)
I1. IP Route Multi-WAN & Recursive
I2. NETWATCH Non-Recursive Option.
J. Directing Users out A Specific WAN with Routing Rules - (no mangling required)
J2. Ensuring Router Responds to requests from Same WAN ( no mangling required )
K1. Introduction to Mangling
K2. Directing Firewall Address List out A Specific WAN (with mangling)
L. Multi-WAN Input To Servers (with mangling)
M1. First Access to MT including WINBOX
M2. Beginner Basic Random Topics ( viewtopic.php?t=191442 )``
M3. PPPOE ISP with VLAN
N. Zerotier
O. Load Balancing PCC/Bandwidth
P. Switch Chip Vlans
Q. Mangling and Allowing Fastrack
R. Discovery/Cast Wireguard - EOIP VXLAN
S. OSPF Video Tutorials V7 --> https://www.youtube.com/playlist?list=P ... 3oaBTlWK3_
T. VRRP (HA) ( viewtopic.php?t=196051 )
Z. Super Useful Scripts
?. Capsman NOT INCLUDED! - not recommended for new users (besides being of a patriarchal nomenclature)

JUST STARTING TO CONFIG YOUR MIKROTIK DEVICE OR SIMPLY BROWSING - SEE BELOW

The first thing a new user should do is go straight to Para M. ( Para = Paragraph ) and watch the linked videos. If you have a grasp on the basics of winbox then the following order of help topics is designed to get you up and running safely and efficiently (A & B), and upon success one can venture into more interesting configurations that include vlans and using MT devices for other than routing (C & D) and finally, when ready, to start to tackle slightly more advanced topics/requirements (E & F) such as remotely and securely connecting to your device.

WARNING: Do not use QUICKSET to configure your router.
One may often use quickset to set the mode of the router (such as WISP mode for capac), or use the quickie VPN setup, but NEVER go back!!!

A. CONFIGURING (off) BRIDGE - viewtopic.php?t=181718 (all MT devices)

B. FIREWALL - viewtopic.php?t=180838

C. VLAN FILTERING read this --->[/b] ^^^- viewtopic.php?t=143620 ( pcunite's bible )

For nerds who wish to understand the bridge/vlan filtering construct, this is an excellent reference from The Chosen One: viewtopic.php?f=2&t=173692

To understand Bridge in a nutshell: Courtesy of Sindy, the Jakob Ingebrigtsen of MT Support.
You use a bridge interface whenever you want to connect multiple devices to the same IP subnet. Whether you use VLANs or not is a separate thing. You can use a dedicated bridge for each (V)LAN, you can use multiple bridges with multiple VLANs on each, or a single bridge for all your VLANs - it's up to your actual needs. But the bridge still forwards frames between its ports up to their MAC addresses only, and if the Mikrotik device on which this bridge exists wants to talk IP to the external devices connected to the bridge, you have to attach an IP address to the appropriate interface of the router - to the bridge-facing interface of the router directly or to the /interface vlan attached to it, depending on the rest of the configuration.

So again in different words - the interface of the router that is "connected" to the matching port of the virtual switch must have an IP address if you want that the router had access to devices in the same (V)LAN which is configured as an access one on that port of the virtual switch. If vlan-filtering is off, what came in tagged stays tagged on egress and what came in tagless stays tagged on egress; if vlan-filtering is on, you can tag frames on ingress and untag them on ingress. But that has nothing to do with attaching or not attaching IP addresses to interfaces.

When an IP stack sends a packet via an L2-capable interface, that packet gets encapsulated into a tagless ethernet frame (leaving wireless aside to simplify things). If the interface is the tagless end of the /interface vlan pipe, the frame gets an 802.1Q VLAN header as it passes through the pipe, so at the tagged end of the pipe, it is now a VLAN frame.

In the opposite direction, the pipes are "picky" - they only forward VLAN frames whose VID matches the one the pipe is configured to use, and untag them as they deliver them to their tagless end. They ignore frames with other VIDs.

You can attach the /interface vlan to an L2-capable interface directly, and you can also attach an IP address to it, but only if that L2-capable interface is not a member port of a bridge or a member link of a bond (exceptions exist but making use of them has side effects and causes incompatibility with less punky devices than Mikrotik is). At this step, we still think about the interface to which the IP address and/or /interface vlan pipes are attached as about the switch-facing interface of the router.

The next step is to see what happens to the frame as it enters the bridge via the router-facing port of the bridge. And here the difference between vlan-filtering=no and vlan-fltering=yes comes into play. With no, the bridge behaves like a "non-managed" (dumb) switch - it only cares about MAC addresses and nothing else. So if a frame came in tagless through one port, it will stay tagless on the bridge and as it egresses via another port(s). With yes, the bridge behaves like a "managed switch", so each of its ports can be an "access", "trunk", or "hybrid" one, depending on the settings. An access port only accepts tagless frames on ingress, tags them with the configured VID, and ignores tagged ones; a trunk port only accepts tagged frames (and may only accept those that bear the permitted VIDs), and a hybrid port is a combination of both behaviours. "Inside" the bridge, all frames are tagged, but if a tagless frame comes in through to a port with a certain pvid and egresses via another port with the same pvid, it will be tagless also at egress, as it gets tagged on ingress, passes through the bridge tagged, and then gets untagged on egress.

The router-facing port of the bridge behaves exactly the same like any other port of that bridge. So if you attach an IP address "to the bridge itself" (in fact, to the bridge-facing interface of the router), it will end up being reachable via the VLAN which is configured as pvid "on the bridge itself" (in fact, the router-facing port of the bridge). If the "bridge" port is in pure trunk mode, such an address will not be reachable at all. If you attach an IP address to an /interface vlan that is attached to the bridge-facing interface of the router, that address will be reachable via the corresponding VLAN if that VLAN is permitted also on the "router-facing port of the bridge" (which means that the router-facing port of the bridge is listed in the tagged list for that VLAN under /interface bridge vlan).

***** Did you Turn ON VLAN filtering after configuring /ip bridge port and /ip bridge vlan settings *****

^^^My advice is not to config/copy pcunite's formats verbatim from the first link above, its stilted and confusing. Instead simply using winbox, work through the Configuration Steps logically.

FIRST though, take one etherport and configure it so that it is accessible off the bridge to avoid getting locked out!!. SEE PARA A. above!!!
Add Bridge (or modify default Bridge as required)
Add VLANS with the parent interface being the Bridge
Create Subnet Structure for each VLAN (and likely modify the default subnet to be one of the VLANs)
Construct /interface bridge ports ( etherports and WLANs as applicable )
Construct /interface bridge vlans ( tagged and untagged **** Should match up with /interface bridge ports as a cross-check )
Make Changes to LAN Interface List ( remove bridge and add all vlans typically)
Add Management/Base Interface List & applicable members (Base Vlan and off bridge etherport for example)
Adjust Firewall Rules as necessary (Base List to Input Chain, LAN List to Input Chain for DNS, etc. ( required router services ))
Go to CLI and run export and see if any errors crop up.
Turn on bridge vlan filtering.

NOTE: **** To understand why its best to manually assign untagging, please read post #3 - viewtopic.php?p=956155#p956155

Note: MT help topics have vastly improved on this topic and can be found at:
https://help.mikrotik.com/docs/display/ ... NFiltering
https://help.mikrotik.com/docs/display/ ... VLAN+Table

Videos For setting up MT
https://www.youtube.com/watch?v=pdpFAxwocTo&t=467s
MT part1- https://www.youtube.com/watch?v=US2EU6cgHQU&t=299s
MT part2- https://www.youtube.com/watch?v=YMwOrc0LDP8

VLANS & HYBRID PORTS

Rule of Thumb: A Port (any port) can ONLY HAVE ONE "1" untagged vlan.

-An access port can only have one untagged vlan.
-A hybrid port can only have one untagged vlan (and as many tagged vlans as required - and obviously not tagged on the same vlan as untagged).
-A trunk port can only have one or more tagged vlans.

(example)
.......................

/bridge port ports
add bridge=bridge interface=ether5  pvid=10  {hybrid}
add bridge=bridge interface=ether6  pvid=10  {access}
add bridge=bridge interface=ether7  pvid=11  {access}
add bridge=bridge interface=ether8            {trunk}

/interface bridge vlan
add bridge=bridge tagged=bridge,ether5,ether8  untagged=ether7  vlan-ids=11
add bridge=bridge tagged=bridge  untagged=ether5,ether6  vlan-ids=10
add bridge=bridge tagged=bridge,ether8 vlan-ids=14,15,16

...........................

D. AP SWITCH SETUP (using any RoS device as an AP/Switch) - viewtopic.php?t=182276

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

E. DESTINATION NAT & Port Forwarding - viewtopic.php?t=179343

F. WIREGUARD - viewtopic.php?t=182340

G. BACKUP/EXPORT CONFIG - https://help.mikrotik.com/docs/display/ROS/Backup / https://help.mikrotik.com/docs/pages/vi ... Id=2555971
(courtesy of mkx)
The most direct method is "BACKUP" - a binary file, created by the admin. This file is really only intended to restore a configuration on the very same device, preferably running the same version of ROS. If restored to another ROS version, there's a built-in configuration converter which mostly does a good job (but not always). If restored to another device of same type, MAC addresses are overwritten. Restoring it to a device of different model doesn't work (either creates random results or fails altogether).
Backup Function ---> Select Files in winbox.

Then next and most common method is called "EXPORT". This is an admin created text export of the current configuration. This text file shows commands needed to configure the device to its current state. This export lacks some items that make complete setup: users, passwords, certificates, SSH keys, etc. The benefit of using it is that it's text file, so it's pretty easy to apply it to a different device ... one has to adjust commands and that's it. It can be applied verbatim to a device, but a devices configuration needs to be completely blank beforehand.
Export Function ---> Select New Terminal in winbox. and use the apppropriate syntax.

Export Syntax For Admin
/export verbose file=anyname { shows every possible entry - rarely used }
/export show-sensitive file=anyname { useful for most admin work and NOT to be posted }
Export Syntax For Forum (public viewing)
/export hide-sensitive file=anyname { for version 6 }
/export file=anyname { for version 7 }

Note1:The EXPORT format is the preferred communication tool for forum members when discussing problems (screenshots are requested if needed).
Caution: One should not expose public IPs or public gateway IPs in any posted Export file or screenshot!
Note2: Export is best viewed using NOTEPAD++

----> Where to find DEFAULT CONFIG - /system default-configuration print

H. NETINSTALL & PROTECTED ROUTER BOOT MODE (PBRM)

NETINSTALL should be used if any security concerns arise OR if your firmware version seems to be acting strangely or the firmware version is really old!!
PBRM is mainly activated/used by those 'loaning' routers to customers and want to retain control over the firmware. viewtopic.php?t=181158
FOR MAC - https://tangentsoft.com/mikrotik/wiki?n ... 20on%20EL9

Another way of getting your unit(s) up to date is to netinstall devices. I would say it's preferred because it also installs recent default configuration while upgrades convert running configuration. A few versions between your running ROS and current one brought some big changes and it is possible for upgrade process to fail to convert old config to new one. In addition, new default config makes much more secure setup in certain aspects. In other words, if you are many firmware versions behind, netinstall is the least painless way to get to the latest config and is more secure.

Sage tips/advice on NetInstall if having difficulties......
a. Regarding Etherboot for all devices, the most error-free method is - to press the reset button, keep it pressed, power on the device, and wait until the device shows up in the NetInstall window, then release the button. This holds true for the Audience as well.
b. Try to have a direct cable connection between computer and device (laptop to device for example).
c. A trick which helps for some is to only use a dumb switch in between, nothing else connected.
d. Also on THE COMPUTER being used to run netinstall, there may be issues if there are multiple NETWORK interfaces. If experiencing difficulties on the PC, it is best to ensure all other NETWORK interfaces are turned off.
e. Enable NETBIOS, over TCP/IP ( active vice automatic or disabled )!
f. Apparently (fart, belch) some users have success using a LINUX computer vice Windows......... hard to believe but worth it if no luck otherwise !

User Comment: In my experience, Netinstall often fails to detect devices after the ethernet link to the MT device goes down and up again while netinstall is running.
Either place a dumb switch between the PC and the MT box to netinstall, so the PC ethernet link stays up during reboot while entering Etherboot.
Or first put the MT box in Etherboot mode, wait for the link to come up and start Netinstall after the link is up.

General Refs:
https://help.mikrotik.com/docs/display/ROS/Netinstall
https://help.mikrotik.com/docs/pages/vi ... Id=1409054
https://wiki.mikrotik.com/wiki/Manual:E ... set_button
RESET VIDEO FROM MikroTik - https://www.youtube.com/watch?v=6Unz92rABs8

I1. IP ROUTE - Multi-WAN & RECURSIVE { rp-filter=loose NOT strict }

There is much confusion on IP routes for the new beginner in part because we dont know how it got there in the first place. Mikrotik default settings in IP DHCP settings, typically for ether1, include two items of interest. Use ISP "PEER" DNS (check box selected) and use ISP Route ( Add Default Route is selected to "yes" ). Thus an IP Route is already provided (the default route) if the admin selects YES. One can uncheck this selection and enter manual routes which is usually the case once you get past one ISP provider. This section will focus on using manual routes which means ensuring the selection for Add Default Route is set to "no". The manual routes below will work regardless of firmware (6/7) chosen.

DUAL WAN - FAILOVER ( basic )
/ip route
add check-gateway=ping comment=Primary ISP distance=5 dst-address=0.0.0.0/0 gateway=Primary-gatewayIP
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=Secondary-gatewayIP

The syntax check-gateway=ping, tells the router to ping the gateway IP every 10 seconds or so. In this example, the router will still switch to ISP2, if ISP1 is not available ( and this could take awhile on a long lease), and the router will keep checking to see if WAN1 becomes available again, and if it does will switch router internet traffic back to ISP1 due to the fact that the shorter distance of ISP1 will also point the router to selecting ISP1 if both are available. THe key component here is distance. The route chosen all the time if both are available will be the one with the shortest distance.

Recursive Routing
In its simplest form/def'n - A recursive route is simply one that points to another route in your routing table instead of to a directly connected link.
Recursive routing occurs when a route either static route or dynamically learned has a next-hop that is not directly connected to the local router. This second lookup in the routing table is called recursion. A recursive route in general is one that points to a next-hop which isn't directly connected to the router. A directly connected link would be something that has your ISP gateway IP in the route. An internal route would be one on the router itself, (like all the subnet or vlan routes - dst-address=vlansubnet gwy=vlan-name)

WHY RECURSIVE? --> Many experienced admins don't trust that the router to ISP connection as the best way to ensure connectivity to the WWW. The reason is that often the ISPs internal network connection can be up (modem to next hop within the ISP network) but the ISPs connection to the internet is down. Thus the router will think the gateway is working when nothing on the internet can be reached. Admins solve this issue with something called "Recursive Routing".

To accomplish this, the admin, by use of the DNS addreses as the gateways, verifies the WAN connection works (is live). We ping those addresses to establish if the connection works and then transform the connection to the ACTIVE state. In the examples we are using a common DNS provider IP address such as 1.0.0.1, and 9.9.9.9.

TWO RULES OF THUMB (scope & target scope):
First Rule. The resolving route (DIRECT - connected route) with dst-address TO the "real WWW IP (dns site)" and with local ISP gateway IP, has Target-Scope=X and the recursive route (INDIRECT - external route) with gateway IP VIA the "real work WWW gateway IP (dns site)" has Target-Scope=X+1. In other words, the farther one gets from the router, the TS increases by one.

Second Rule. Between the same two routes being compared, the Direct , connected route, with local ISP gateway IP (resolving route) has to have a SCOPE that is equal to or less than the TARGET SCOPE of the recursive route. In other words, the scope of the route must be equal or less than the target scope of the next farthest route.

FARTHEST ROUTE: SCOPE= (doesnt matter) / TARGET SCOPE=Y+2 (recursive route)
CLOSER ROUTE: SCOPE= Y+2 or less / TARGET SCOPE=Y+1 (recursive route)
CLOSEST ROUTE: SCOPE=Y+1 or less / TARGET SCOPE=Y (gateway=ISP, resolving route)
INTERNAL ROUTE: ( within router, scope is not used, no recursive action at all )

Video LInk - https://www.youtube.com/watch?v=eTmpBAAW_pQ

DUAL WAN - RECURSIVE
/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=11 target-scope=11
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

Note1: As you can see TS as you get closer to the router decreases by 1.
Note2: As you can see the scope of the next route is equal to or less than the TS of the preceding route.

USING TWO RECURSIVE ROUTES - FLAT
Additionally many admins also like to use at least two different public IPs to confirm connectivity (removes the rare but not rare enough occasion that DNS providers are not available). Thus as per the below example, a secondary DNS is checked to verify indeed that ISP1 is not available, when the first DNS checked is not working. If both DNS recursive routes show as not available, the router will select the Secondary WAN for all traffic. In the meanwhile the router will continue checking the availability of both DNS recursive routes for connectivity and will switch back to ISP1 if at least one of them is available.

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11
+++++++++++++++++++
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12
add distance=4 dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11
+++++++++++++++++++
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

Note: As you may see, its easiest just to set all the scopes the same and in this case to 10, which ensures the Second Rule from above is followed.

USING TWO RECURSIVES - NESTED
Finally, admins can accomplish the same effect as the "flat" approach above by layering or nesting the recursive routes with each other with the same outcome. This is a more complex setup but is deemed to be more efficient overall especially if using multiple routing tables (academic and not required for new users). It is included for those interested. The tricky part is that we will use an address that really doesnt exist to force the router to look for a routing, in this case randomly choose 10.10.10.10/32. Thus we create a top rule with no apparent path and then recursively look for a path through both DNS addresses which then are resolved by the direct route through the ISP gateway. So although it appears we are using three Other addresses in nested recursive routing we are still only using 2 KNOWN DNS addresses and one 'BOGUS" address. The bogus address is used to force the router to be recursive for both the KNOWN Dns addresses. In other words, the flat approach is Recursive to Resolving (two steps), nested is Recursive to Recursive to Resolving (three steps).

/ip route
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14
++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
+++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
+++++++++++++++
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30

Note: All Primary routes have same distance, only the last Secondary route has a higher distance.

Example of two WANS both using recursive nested routing not just the primary and relying on 4 different external sites for checking.

distance=1 dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14
distance=2 dst-address=0.0.0.0/0 gateway=11.11.11.11 scope=10 target-scope=14

add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1scope=10 target-scope=13
add check-gateway=ping dst-address=11.11.11.11/32 gateway=8.8.8.8 scope=10 target-scope=13
add check-gateway=ping dst-address=11.11.11.11/32 gateway=208.67.222.222 scope=10 target-scope=13

add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
add dst-address=8.8.8.8/32 gateway=SecondaryISP-gatewayIP scope=10 target-scope=12
add dst-address=208.67.222.222/32 gateway=SecondaryISP-gatewayIP scope=10 target-scope=12

DTRIPLE WAN - RECURSIVE Courtesy of AzDsL - This is a very challenging scenario. In this case, we have three WANS, where WAN1=dist1 WAN2=dist2 and WAN3=dist3. We have a number of subnets and the various subnets are connected to specific WANs. Subnets A&B (standard users) go to WAN1, Subnets C&D (VIP users) go to WAN2 and Subnet E&F (Other users) go to WAN3. One can only get so far before running into the wall of a common issue --> whenever a specific WAN is down and gets back, its assigned traffic won't be automatically routed back to is original gateway.

So how do we ensure that the specific conditions are met:
if WAN1 fails all gwy1 traffic goes to WAN2 and if WAN2 fails this traffic goes to WAN3
If WAN 2 fails all gwy2 traffic goes to WAN1 and if WAN1 fails this traffic goes to WAN3
if WAN3 fails all gwy3 traffic goes to WAN1 and if WAN1 fails this traffic goes to WAN2

As per other examples we need 3 tables.
/ip tables
add name=ISP1-route fib
add name=ISP2-route fib
add name=ISP3-route fib

and we need Routing Rules.
/ip routing rules
add src-address=10.100.100.0/24 action=lookup table=ISP1_route comment=STANDARD
add src-address=10.100.200.0/24 action=lookup table=ISP1_route comment=STANDARD
add src-address=10.200.123.0/24 action=lookup table=ISP2_route comment=VIP
add src-address=172.20.234.0/24 action=lookup table=ISP2_route comment=VIP
add src-address=192.168.111.0/24 action=lookup table=ISP3_route comment=OTHERS
add src-address=192.168.222.0/24 action=lookup table=ISP3_route comment=OTHERS

The final piece of the Recursive puzzle is modifying and adding failover routes for each of the routing rules with priority distance.
Where we start with the example of Dual WAN recursive above except we have a third virtual recursive IP:
STARTING POINT:
distance=1 dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 { main table }
distance=2 dst-address=0.0.0.0/0 gateway=11.11.11.11 scope=10 target-scope=14 { main table }
distance=3 dst-address=0.0.0.0/0 gateway=12.12.12.12 scope=10 target-scope=14 { main table }

ADD:
distance=1 dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 routing-table=ISP1-route
distance=2 dst-address=0.0.0.0/0 gateway=11.11.11.11 scope=10 target-scope=14 routing-table=ISP1-route
distance=3 dst-address=0.0.0.0/0 gateway=12.12.12.12 scope=10 target-scope=14 routing-table=ISP1-route

distance=1 dst-address=0.0.0.0/0 gateway=11.11.11.11 scope=10 target-scope=14 routing-table=ISP2-route
distance=2 dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 routing-table=ISP2-route
distance=3 dst-address=0.0.0.0/0 gateway=12.12.12.12 scope=10 target-scope=14 routing-table=ISP2-route

distance=1 dst-address=0.0.0.0/0 gateway=12.12.12.12 scope=10 target-scope=14 routing-table=ISP3-route
distance=2 dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 routing-table=ISP3-route
distance=3 dst-address=0.0.0.0/0 gateway=11.11.11.11 scope=10 target-scope=14 routing-table=ISP3-route

Then the rest of the routes are required, six with target scope of 13, and the last six with target scope of 12.

I1. NETWATCH - A Non-Recursive Option

J1. DIRECTING Users out Specific WAN Using Routing Rules { no mangling }

In multi-WAN setups, it is often that the admin needs to direct a whole subnet or one or more users out a specific WAN. This is typically applied when the users are supposed to go out a Primary WAN but a few users need to go out the Secondary WAN. The same approach by the way is used in Wireguard configs, when one has to force some users into the wireguard tunnel for internet (at a remote site) because the bulk of users locally go out the local WAN for internet. If you need a group of IPs and its too numerous for this method, check out the next paragraph which addresses ensuring an address list goes out a specific WAN.

Note: In Ver6 firmware Routing Rules take precedence over Mangling. In Ver7 firmware the reverse is true.

GENERAL EXAMPLE
Version 6 approach
Solution: Consists of two steps: ONE - create an additional route for the secondary WAN (copy of the original one) and add a routing-mark!, and TWO - create an associated Routing Rule which will direct the User to that Routing Mark via its Table selection parameter.
[Step1]
dst-address=0.0.0.0/0 gwy=ISP1 distance=5 check-gateway=ping
dst-address=0.0.0.0/0 gwy=ISP2 distance=10
dst-address=0.0.0./0 gwy=ISP2 distance=10 routing-mark=useSecondary

[Step 2]
/ip routing rule
add src-address=SUBNET or IPaddress Action=Lookup-only-in-table table=useSecondary

Version 7 approach
Solution: Very similar to the Version 6 approach but we have to physically create the new table and adjust the additional route by changing from the default table=main, to table=new table constructed ( and no use of route-marking ).

[Additional Step]
/routing table add fib name=useSecondary

Note1: Action=Lookup-only-in-table means the router will NOT look for any other routing for said traffic in the main table, if the Secondary WAN is not available, and the traffic will be dropped.
Note2: If the admin wants the users to be able to access the Primary WAN, if the Secondary WAN is down, then use ACTION=Lookup.

MULTIPLE USERS
If you have more than one user, that needs to be 'redirected', then simply create MULTIPLE Routing Rules! .

/routing rule add src-address=IP1 action=lookup  table=useSecondary   { for first IP }
/routing rule add src-address=IP2 action=lookup  table=useSecondary   { for second IP }
...
...
/routing rule add src-address=IPXX action=lookup  table=useSecondary   { for last IP }

.........................

Note3 (see below) : Routing Rule order counts much like firewall rules etc. This may come into play when you have more complex requirements.

COMPLEX CASE: [ Three Subnets ]
THREE SUBNETS. Primary WAN is for subnet A and subnet B, but users on subnet C should go out Secondary WAN. In addition, one user on subnetC (the exception) should actually follow the same routing as subnets A&B. In terms of failover, users from subnet A and B should be able to use ISP2, if ISP1 is not available. However, users from subnet C should not have access to ISP1 if ISP2 is not available.

In terms of planning purposes, it should be clear that we are going to have to redirect Subnet C users to WAN2. In addition we are going to have redirect Exception User on Subnet C to mimick routing that users on Subnets A,B follow. Therefore we will have to create additional routes for both WANS, to be able to force users as needed. (subnet C to wan2, exception user on subnet C to wan1)

dst-address=0.0.0.0/0 gwy=ISP1 table=main distance=5 check-gateway=ping
dst-address=0.0.0.0/0 gwy=ISP2 table=main distance=10
dst-address=0.0.0.0/0 gwy=ISP1 table=main distance=5 check-gateway=ping table=usePrimary
dst-address=0.0.0./0 gwy=ISP2 distance=10 table=useSecondary

/routing table add name=usePrimary fib
/routing table add name=useSecondary fib

/routing rule add src-address=Exception IP action=lookup table=usePrimary { for exception user on subnet C }
/routing rule add src-address=subnetC action=lookup-only-in-table table=useSecondary { for subnet C }

Order is key: If we had the routing rules order reversed, the exception user would have been captured by that Routing Rule, since located in subnet C and forced to WAN2, thus we had to match that user to the correct routing rule, FIRST, to ensure traffic went to WAN1

SERVER EXAMPLE: [ Local User Access to server NOT EXTERNALLY FACING ]
It is thusly possible to direct a server out a specific WAN. Lets say we have a VOIP server 10.10.10.2 that has been directed out WAN2.

/routing table add name=useWAN2
/routing rule  add src-address=10.10.10.2/32  action=lookup-only-in-table table=useWAN2
/ip route
dst-address=0.0.0.0/0 gateway=WAN2ISP  table=useWAN2

However a problem arises! The local LAN users should also be able to access the Server but with the rules set above, all responses to LAN users would be directed out to WAN2 and not back to the users. The solution is to create another route rule that identifies traffic being sent to the Server and informs the router which table such traffic should use (table main). The router, via table main, will select the most appropriate routing which will be the interface for the subnet the user sent the request from, and the returns will go back to the local user. ORDER IS IMPORTANT........... In this case ensure the more generic rule is stated first!!

/routing rule add dst-address=10.10.10.2/32 action=lookup-only-in-table table=main
/routing rule add src-address=10.10.10.2/32 action=lookup-only-in-table table=useWAN2

SUBNET TO SUBNET: [ Local Users need to go out WAN2 but ALSO reach other subnets on the LAN ]
This same problem arises, when the admin also needs to allow subnets to talk to one another when they are being "forced" out a specific WAN. As demonstrated above, use of additional ROUTE RULES does the job with the use of destination addresses!

In this example, subnet 10.20.0.0 needs to go out WAN2 vice WAN1 (which the other two subnets use as WAN1 is primary by distance). However we still want the subnet to be able to have a path to the other two subnets. The key is to provide routing rules for such traffic PRIOR to the useWAN2 route rule (and destination address as applicable).
...........

/routing rule
add dst-address=192.168.0.0/16 action=lookup-only-in-table table=main 
add dst-address=10.0.0.0/8 action=lookup-only-in-table table=main
add src-address=10.20.0.0/22 action=lookup table=useWAN2

.........................

J2. ENSURING Same WAN for Return Traffic { no mangling }

When one wants to ensure incoming requests TO THE ROUTER (services such as wireguard) in a multi-wan setup, are returned via the correct same WAN, then
one can use Routing Rules to accomplish this task. Unlike mangling, Routing Rules are considered prior to Routing. Manging(in output chain) is considered after Routing

Hence lets look at two WANs.
/add fib table=preferWAN1
/add fib table=preferWAN2
/routing rule add src-address=WANIP1 action=lookup-only-in-table table=preferWAN1 { we ensure wan1 replies via ISP1 }
/routing rule add src-address=WANIP2 action=lookup-only-in-table table=preferWAN2 { we ensure wan2 replies via ISP2 }
/ip route
add dst-addres=0.0.0.0 gateway=ISP1 routing-table=preferWAN1
add dst-addres=0.0.0.0 gateway=ISP2 routing-table=preferWAN2

Note: If ISP provides a dynamic WANIP one will need scripts to update the src-address and most likely gateway IP as well.

Routing Recap: Order for Routing is
a. Routing Table
b. Best matching dst-address
c. route activity
When does distance come into effect?
Among all routes with identical routing table name and same dst-address, the active one is chosen among all those that can work ( interface up, check gateway says its available) distance determines which route is chosen.

(Note: Keep in mind that routing marks assigned in mangling take precedence over those used in Routing Rules.)
................................

K1. Introduction to Basic Mangling -
TWO WANS and TWO LANS - both LANs contain at least one server
In the previous example we were only concerned about local traffic being controlled out the WAN. Now we are going to add the requirement to ensure external traffic coming in a particular WAN leaves the same WAN. This will be accomplished by two sets of mangle rules in the pre-routing chain, basically marking packets entering each WAN and then associating those packets with a route mark that equates to the associated WANX routing-table we created previously for forcing LANX out the specific WANX. We will apply this two TWO LANS and TWO WANS.

Note: In Ver7 firmware Mangling takes precedence over Routing Rules. In Ver6, the reverse is true.

Requirements: LAN1 needs to go out WAN1 and LAN2 needs to go out WAN2. Additionally we have servers on both LANS that external users access and thus we need to ensure incoming user return traffic, after hitting the servers, goes back out the same WAN.
Notes
1. Both WANs are from the same ISP so there is no failover discussion.
2. We only need two create two tables as they will be used both by mangle rules for external traffic (routing-mark) and routing rules for internally originated traffic. (table=) as well as for the IP ROUTEs directing traffic out each gateway and corresponding table.

Tables:
/routing-table add fib name=viaISP1
/routing-table add fib name=viaISP2

Mangles
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=ether1 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=ether2 new-connection-mark=WAN2 passthrough=yes
=============================================================================
add action=mark-routing chain=prerouting connection-mark=WAN1 \
new-routing-mark=viaISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 \
new-routing-mark=viaISP2 passthrough=no

Routing Rules
/routing rules
add src-address=LAN1-subnet action=lookup-in-table-only table=viaISP1
add src-address=LAN2-subnet action=lookup-in-table-only table=viaISP2

/ip route
add distance=1 gwy=WAN1 table=main {standard route}
add distance=1 gwy=WAN2 table=main {standard route}
add distance=1 gwy=WAN1 table=viaISP1 { route for both external return traffic and local originated traffic on LAN1 to go out WAN1}
add distance=1 gwy=WAN2 table=viaISP2 route for both external return traffic and local originated traffic on LAN2 to go out WAN2}

K2. MULTI-WAN & Firewall Address List & Mangling (PCC) }

In this type of scenario you EITHER have a group of internal/local IP addresses (could be something less than a subnet or bunches of IPs from different subnets) where route rules are just to onerous OR you have a bunch of specific destination addresses that when selected from ANY user, should go out a specific WAN. In these cases its typical to disable fasttrack as mangling and fasttrack are not compatible.

SOURCE ADDRESS BASED EXAMPLE
Assuming two separate PPPOE connections for a dual WAN scenario and a firewall address list called special-group, where this group needs to go out pppoe-2

/ip route
add gateway=pppoe-out1
add gateway=pppoe-out2 
add gateway=pppoe-out2  routing-mark=specific-users (<--- V 6+++ V7 --->) add gateway=pppoe-out2  table=specific-users
/ip firewall mangle
add chain=prerouting  src-address-list=special-group  action=mark-routing  new-routing-mark=specific-users { (same for v6 and v7)}

..............

If you wish to keep fasttrack for the rest of the traffic, modify two copies of the regular established,related rule and put them in front of the fasttrack rule - one rule for originating traffic and one rule for returning traffic. The modification is the addition of the firewall address list!
add action=accept chain=forward connection-state=established,related src-address-list=special-group
add action=accept chain=forward connection-state=established,related dst-address-list=special-group

DESTINATION ADDRESS BASED EXAMPLE
Assuming two separate PPPOE connections for a dual WAN scenario and a firewall address list of destination addresses called external-group, where any traffic headed towards these destinations needs to go out pppoe-2.

/ip route
add gateway=pppoe-out1
add gateway=pppoe-out2 
add gateway=pppoe-out2 routing-mark=dest-exception
/ip firewall mangle
add chain=prerouting dst-address-list=external-group  action=mark-routing  new-routing-mark=dest-exception

If you wish to keep fastrack for the rest of the traffic repeat the process as shown in the previous example above.........
add action=accept chain=forward connection-state=established,related dst-address-list=external-users
add action=accept chain=forward connection-state=established,related src address-list=external-users

L. MULTI-WAN Input To Servers & Mangling { rp-filter=loose NOT strict / fastrack=disabled }

For discussion purposes we are describing a scenario where the Admin has one or more servers behind the router and they should be accessible from both WAN connections, regardless of which is Primary or which is Secondary. How the users reach the servers is up to the admin. Basically the idea is to capture the incoming traffic from external users and then ensure the traffic goes out the same WAN it came in on! (Thanks to Sob) the following script forms the basis for this V7 approach:

/routing table
add name=ISPx_route fib
/ip route
add dst-address=0.0.0.0/0 gateway=<ISPx gateway> table=main  { comment:  For local originated traffic }
add  dst-address=0.0.0.0/0 gateway=<ISPx gateway>  table=ISPx_route  { comment: For external originated traffic to servers } { V7 method } 
/ip firewall mangle
add chain=prerouting in-interface=WANx connection-state=new action=mark-connection new-connection-mark=WANx_conn
add chain=prerouting in-interface-list=LAN connection-mark=WANx_conn action=mark-routing new-routing-mark=ISPx_route

...................................
The only difference between V7 and V6, is that in V6 there is no separate table and one adds the routing-mark in the ip route not the table!

/Iip route
add  dst-address=0.0.0.0/0 gateway=<ISPx gateway>  routing-mark=ISPx-route  { comment: For external originated traffic to servers }

..................................

Note: Both approaches (using connection-state=new OR connection-mark=no-mark) are equally valid!

Example 1: DUAL WAN FAILOVER
...................................................

/ip routing table
add name=ISP1_route fib
add name=ISP2_route fib
/ip route
add check-gateway=ping comment=Primary ISP distance=5 dst-address=0.0.0.0/0 gateway=Primary-gatewayIP 
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=Secondary-gatewayIP
add comment=Primary ISP distance=5 dst-address=0.0.0.0/0 gateway=Primary-gatewayIP  table=ISP1_route
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=Secondary-gatewayIP  table=ISP2_route
/ip firewall mangle
add chain=prerouting in-interface=WAN1 connection-state=new action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN1_conn action=mark-routing new-routing-mark=ISP1_route
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN2_conn action=mark-routing new-routing-mark=ISP2_route

......................
Note: The Table and Mangle creation are identical for all examples.

Example 2: DUAL WAN FAILOVER RECURSIVE
.................

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10  target-scope=12 table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10  target-scope=12 table=ISP1_route
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main
..............................................................................................
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30  table=ISP2_route

....................

Example 3: DUAL WAN FAILOVER TWO RECURSIVE (flat)
.....................

/ip route
add check=gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=ISP1_route
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main
....................................................................................................................................................................
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=main
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=ISP1_route
add distance=4 dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main
..............................................................................................
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30  table=ISP2_route

Example 4: DUAL WAN FAILOVER TWO RECURSIVES (nested)
................................

dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 table=main
dst-address=0.0.0.0/0 gateway=10.10.10.10 scope=10 target-scope=14 table=ISP1_route
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping dst-address=10.10.10.10/32 gateway=9.9.9.9 scope=10 target-scope=13
add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
add check-gateway=ping dst-address=10.10.10.10/32 gateway=1.0.0.1 scope=10 target-scope=13 
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
..................................................................................................................................................
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30  table=ISP2_route

.....................................................

Example 5: TRIPLE WAN FAILOVER ONE RECURSIVES & PCC MANGLING (flat)
....................................................
a. if WAN2 fails, send WAN2 traffic to Wan3, if Wan3 then fails, send WAN2 & WAN3 traffic to Wan4
b. if WAN3 fails, send WAN3 traffic to Wan2, if Wan2 then fails, send WAN3 &WAN2 traffic to Wan4
c. If WAN4 fails, send WAN4 traffic to Wan2, if Wan2, then fails, send WAN4 & WAN2 traffic to Wan3

This should get you close in terms of IP routes.
Starting Point Farthest Hop.
...................................

/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=10 target-scope=14 { main table }
add check-gateway=ping dst-address=0.0.0.0/0 gateway=39.39.39.39 scope=10 target-scope=14 {main table }
add check-gateway=ping dst-address=0.0.0.0/0 gateway=8.8.4.4 scope=10 target-scope=14 {main table }

Add Cases for Failure..... ( W2--> 3 then 4 / W3--> 2 then 4 / W4--> 2 then 3 )
distance=1 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=10 target-scope=14 routing-table=prefer-ISP2
distance=2 dst-address=0.0.0.0/0 gateway=39.39.39.39 scope=10 target-scope=14 routing-table=prefer-ISP2
distance=3 dst-address=0.0.0.0/0 gateway=8.8.4.4 scope=10 target-scope=14 routing-table=prefer-ISP2

distance=1 dst-address=0.0.0.0/0 gateway=39.39.39.39 scope=10 target-scope=14 routing-table=prefer-ISP3
distance=2 dst-address=0.0.0.0/0 gateway=1.1.1.1 scope=10 target-scope=14 routing-table=prefer-ISP3
distance=3 dst-address=0.0.0.0/0 gateway=8.8.4.4 scope=10 target-scope=14 routing-table=prefer-ISP3

distance=1 dst-address=0.0.0.0/0 gateway=8.8.4.4 scope=10 target-scope=14 routing-table=prefer-ISP4
distance=2 dst-address=0.0.0.0/0 gateway=1.1.1.1.1 scope=10 target-scope=14 routing-table=prefer-ISP4
distance=3 dst-address=0.0.0.0/0 gateway=39.39.39.39 scope=10 target-scope=14 routing-table=prefer-ISP4

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ADD NEXT HOP (Direct LINK)
.......................................

/ip route
add dst-address=1.1.1.1/32 gateway=192.168.12.1 scope=10 target-scope=13
add dst-address=39.39.39.39/32 gateway=192.168.13.1 scope=10 target-scope=13
add dst-address=8.8.4.4/32 gateway=192.168.14.1 scope=10 target-scope=13

.............................................

MANGLING - 3 WANS WITH PCC ( with failover and one PPP network internally )

A. Have to ensure any users coming in externally for mainly the internal server, on any WAN, go back out the same WAN.
This rule has two purposes, to mark incoming traffic headed toward a server etc, or to the router itself!
..........

add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark="" in-interface=\
WAN3 new-connection-mark=WAN3_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=WAN4 new-connection-mark=WAN4_conn passthrough=yes

................
B. Next we have to ensure all users originating traffic heading for the WANs are both connection marked and PCCd. Since we have the standard LAN and anther ppp subnet to worry about (not really an interface) we need to use a firewall address list vice and interface list approach.
/ip firewall address list
add address=192.168.15.0/24 list=ALL-users
add address=192.168.20.0/24 list=ALL-users

And then the PCC rules and marking connections for the internal originating traffic.
.......................

add action=mark-connection chain=prerouting connection-mark=no-mark \
dst-address-type=!local src-address-list=ALL-users new-connection-mark=\
WAN2_conn passthrough=yes per-connection-classifier=\
both-addresses-and-ports:4/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
dst-address-type=!local src-address-list=ALL-users new-connection-mark=\
WAN3_conn passthrough=yes per-connection-classifier=\
both-addresses-and-ports:4/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
dst-address-type=!local src-address-list=ALL-users new-connection-mark=\
WAN3_conn passthrough=yes per-connection-classifier=\
both-addresses-and-ports:4/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
dst-address-type=!local src-address-list=ALL-users new-connection-mark=\
WAN4_conn passthrough=yes per-connection-classifier=\
both-addresses-and-ports:4/3

C. Next, for the LAN and PPP traffic identified we want to associate that marked traffic withe the specific routes accordingly!!! It is not totally necessary to include the LANSIDE members just for information.
....................

add action=mark-routing chain=prerouting connection-mark=WAN2_conn \
src-address-list=ALL-users new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn \
src-address-list=ALL-users new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN4_conn \
src-address-list=ALL-users new-routing-mark=to_WAN4 passthrough=yes

..................

D. Lastly we want to ensure any traffic that was headed for the router itself through the WANs,
gets sent back out the same interface.
Note: Return traffic from the Router itself does not go through preouting and thus we must use output chain.
.........................

add action=mark-routing chain=output connection-mark=WAN3_conn \
new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN4_conn \
new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn \
new-routing-mark=to_WAN2 passthrough=yes

...........................

M1. First Time Access & WINBOX (must watch videos for the new user)
First Access/Config - https://www.youtube.com/watch?v=Q9qwgKrw-0g
Access/Quickset - https://www.youtube.com/watch?v=JR8OJPNhwjM
{Please note that quickset should only be used to set the initial mode of the router ( capac --->wisp mode) and NEVER touched again!}
More on Winbox - https://www.youtube.com/watch?v=MKVJYxhmSkQ
Basic Firewall - https://www.youtube.com/watch?v=6boYA7xdjZY&t=1376s
Basic Routing - https://www.youtube.com/watch?v=FBknCFkH0J8&t=21s
Static Routes - https://www.youtube.com/watch?v=ZYeMuYBAVrQ&t=214s
Intro to CLI - https://www.youtube.com/watch?v=EYCjuvTd3dY
CLI Tips - https://www.youtube.com/watch?v=fSt1UrNcx08

M2. Beginner Topics Random - viewtopic.php?t=191442

M3. When your ISP assigns you a PPPOE with VLAN20 and the WAN port used is your SFP port.
.........

interface vlan add vlan-id=20 interface=sfp1 name=vlan20
interface pppoe-client add interface=vlan20 name=PPPoE_Out user=xxxxx password=yyyyyy use-peer-dns=yes add-default-route=yes

........

N. ZEROTIER (courtesy of Amm0)
viewtopic.php?t=183424
Excellent Video from Mikrotik - https://www.youtube.com/watch?v=60uIlyF8Z5s
https://help.mikrotik.com/docs/display/ROS/ZeroTier

O. LOAD BALANCE
https://mum.mikrotik.com/presentations/US12/steve.pdf via PCC
https://mum.mikrotik.com/presentations/US12/tomas.pdf via Bandwidth
Note: For Dynamic IPs, and Fib Routes, use interface name (ether2, pppoe-out1 etc.) as gateway IP is unknown.

P. SWITCH CHIP VLANS
Switch Chip Features - https://help.mikrotik.com/docs/display/ ... p+Features
CRS1 / CRS2 -Switches - https://help.mikrotik.com/docs/pages/vi ... =103841835
CRS3 / CRS5 / CCR2 Switches - https://help.mikrotik.com/docs/display/ ... p+features
L3 Hardware Offloading - https://help.mikrotik.com/docs/display/ ... Offloading
Other Devices (older) Basic - https://help.mikrotik.com/docs/display/ ... switchchip

other refs:
https://www.youtube.com/watch?v=Rj9aPoyZOPo - Vlans using the Switch Chip
https://www.youtube.com/watch?v=rvQ6o4RfnoU - Configure Vlan on Switch Chip
https://www.youtube.com/watch?v=YLtGQAQ8iS0 - CRS3XX Step by Step

CRS1xx VLAN Example

###############################################################################
# Recommended reading
# https://wiki.mikrotik.com/wiki/Manual:Basic_VLAN_switching
#
# Notes: Start with a reset (/system reset-configuration)
#
# Based on: https://forum.mikrotik.com/viewtopic.php?t=143620
###############################################################################

#######################################
# Naming
#######################################

# name the device being configured
/system identity set name="CRS1xx_Switch"

#######################################
# VLAN Overview
#######################################

# 10 = BLUE
# 20 = GREEN
# 30 = RED
# 99 = BASE (MGMT) VLAN

#######################################
# Bridge
#######################################

# create one bridge
/interface bridge add name=BR1 protocol-mode=none

# add "all" ports to this one bridge
/interface bridge port
add bridge=BR1 interface=ether1
add bridge=BR1 interface=ether2
add bridge=BR1 interface=ether3
add bridge=BR1 interface=ether4
# and so on until you get to 24 ...

#######################################
# -- Access Ports --
#######################################

# ingress behavior, egress dynamically handled
/interface ethernet switch ingress-vlan-translation
add customer-vid=0 new-customer-vid=10 ports=ether2
add customer-vid=0 new-customer-vid=20 ports=ether3
add customer-vid=0 new-customer-vid=30 ports=ether4

#######################################
# -- Trunk Ports --
#######################################

# ingress behavior
# L2 switching only, Bridge (aka switch1-cpu) not needed as tagged member (except for BASE_VLAN)
/interface ethernet switch vlan
add ports=ether1,ether2 vlan-id=10
add ports=ether1,ether3 vlan-id=20
add ports=ether1,ether4 vlan-id=30
add ports=switch1-cpu,ether1 vlan-id=99

# egress behavior
# L2 switching only, Bridge (aka switch1-cpu) not needed as tagged member (except for BASE_VLAN)
/interface ethernet switch egress-vlan-tag
add tagged-ports=ether1 vlan-id=10
add tagged-ports=ether1 vlan-id=20
add tagged-ports=ether1 vlan-id=30
add tagged-ports=switch1-cpu,ether1 vlan-id=99

#######################################
# VLAN Security
#######################################

# drop traffic that does not follow the above port layout
/interface ethernet switch set forward-unknown-vlan=no


#######################################
# IP Addressing & Routing
#######################################

# LAN facing Switch's IP address on a BASE_VLAN
/interface vlan add interface=BR1 name=BASE_VLAN vlan-id=99
/ip address add address=192.168.0.2/24 interface=BASE_VLAN network=192.168.0.0

# The Router's IP this switch will use
/ip route add distance=1 gateway=192.168.0.1

#######################################
# MAC Server settings
#######################################

# Ensure only visibility and availability from BASE_VLAN, the MGMT network
/interface list add name=BASE
/interface list member add interface=BASE_VLAN list=BASE
/ip neighbor discovery-settings set discover-interface-list=BASE
/tool mac-server mac-winbox set allowed-interface-list=BASE
/tool mac-server set allowed-interface-list=BASE

........................................

Q. MANGLING & ALLOWING FASTTRACK
viewtopic.php?t=134048#p659676

R. DISCOVERY/CAST WIREGUARD EOIP VXLAN
viewtopic.php?t=194646

Z. SUPER USEFUL SCRIPTS

Script Syntax Info:
(i) if you want to use an output of one command as a value for another command (or expression), you must put that "one command" into [], such as
:put [:resolve www.google,com]
(ii) if you want to put more commands as a parameter of another command, you have t use {}, such as
:if (x=y) do={command1 ; command2 ; command3}
(iii) :do { put there command you want to run; } on-error={ put command to be run on error}

Scripts
a. Capture LOGGED IPs into Firewall Address list (Vers 1.5 Jotne/Rext) - viewtopic.php?p=905420#p906705
b. SCHEDULER USE (not script) Turn off / on Interface at specific times (CZFAN) - viewtopic.php?t=131494#p645744
c. REMOVE OLD Connections when WAN interface changes (Rextended) - viewtopic.php?f=9&t=154606&p=853803&hil ... ve#p853803
d. Use a DYNDNS provider such as DYNU (Rextended) - viewtopic.php?p=961067#p961068
e. SEND AN EMAIL for when a New User/Device connects - viewtopic.php?t=191991#p974371
f. Force IP CLOUD update through WANX - viewtopic.php?p=962282&hilit=force+cloud+to+use#p963399
g. WIREGUARD CURRENT USERS
/interface/wireguard/peers/print detail where last-handshake < [:totime "3m"]
h. Security: REMOVE Logged in (BUT INACTIVE) USERS - viewtopic.php?t=198410#p1018445

SecCon · Tue Jan 25, 2022 2:38 pm

hates maybe... 1 + 1 = 2 . Period. Hate diff ways to do things or how things can be done in diff ways. Settle on ONE method.

mozerd · Tue Jan 25, 2022 2:56 pm

IMO the path you created here is excellent especially for the newbie .... Tik should make this a STICKY plus Tik should consider a similar approach in their new and improved docs which are sorely lacking information of value.

anav · Tue Jan 25, 2022 3:46 pm

hates maybe... 1 + 1 = 2 . Period. Hate diff ways to do things or how things can be done in diff ways. Settle on ONE method.

My point with maybe, is that the OP always finds ways to screw things up, they are the best testers!!!

404Network · Mon Mar 07, 2022 5:10 am

[edit:nm]

pcunite · Sat Apr 23, 2022 10:38 pm

anav,
I'm really enjoying your IP ROUTE - Multi-WAN section. I appreciate the updates and clarification. Its a lot of work to produce something like this, keeping things as simple as possible. Would like to see support added for how to allow Multi-WAN inbound support (ROS v6 & v7) too. This requires mangle based on what I've read. This simplest way while still supporting the Recursive option would be valuable.

Regarding WireGuard, something that I've been doing is putting a WireGuard server behind a MikroTik firewall. I've had to do this because I can not move to ROS v7 yet in production. Thus I had to find a way to support both a pure Linux based wireguard instance (I was building Ubuntu RPi servers for a while) and also a MikroTik functioning only as a wireguard endpoint. I really like using the MikroTik option. But how to do this took some trial and error. I would like to get that to you and see if you would like to incorporate it into your topic designs about WireGuard. If so, do you want the article here, or in another post?

Sob · Sat Apr 23, 2022 10:53 pm

It's simple, for each WAN do something like (v7):

/routing table
add name=ISPx_route fib
/ip route
add routing-table=ISPx_route dst-address=0.0.0.0/0 gateway=<ISPx gateway>
/ip firewall mangle
add chain=prerouting in-interface=WANx connection-state=new action=mark-connection new-connection-mark=WANx_conn
add chain=prerouting in-interface-list=LAN connection-mark=WANx_conn action=mark-routing new-routing-mark=ISPx_route
add chain=output connection-mark=WANx_conn action=mark-routing new-routing-mark=ISPx_route

For v6 it's almost the same, only you don't need to create routing table and route's routing-table parameter is named routing-mark.

anav · Sat Apr 23, 2022 11:39 pm

I dont get the requirement multi-wan input, do you mean to ensure all traffic coming in on one WAN PORT is forced out the same WANPORT.

Sob needs to explain each line to me, not what it does but its purpose in the scheme of things........ besides syntax tell me 'the story'.

So far I see mark any new connection coming in on a particular WAN,
then he assigns a routing mark to that connection mark.
He does both these steps in prerouting, okay.

What I dont get is the last line??
It appears that he is marking outgoing traffic with the same naming convention for routing mark.........
Not sure how any of this helps...

IF I go out WAN2, the return traffic is going to come in on WAN2.

IF I come in on WAN1, going out on WANX will be predicated on routes.

+++++++++++++++++++++++++++++++
Hmm pcunite were you talking PRIMARY and FAILOVER, or PCC load balanced routes???

Sob · Sat Apr 23, 2022 11:56 pm

Unless I misunderstood the request, multi-WAN router with e.g. 1.2.3.4 on WAN1 and 2.3.4.5 on WAN2, internal webserver with forwarded ports 80 and 443, and it should be possible to reach it using both public addresses. So you mark new incoming connections based on WAN interface, and then mark routing for responses. The one in prerouting is for server (traffic from LAN), other in output is for router itself, e.g. if you'd run VPN server on it, which should be also accessible using both addresses (this can be also done using routing rules).

anav · Sun Apr 24, 2022 12:34 am

PCUNITE, I cover your topics for wireguard already.

Check out para's 7 & 8
viewtopic.php?p=906311#p906311

anav · Sun Apr 24, 2022 12:39 am

Unless I misunderstood the request, multi-WAN router with e.g. 1.2.3.4 on WAN1 and 2.3.4.5 on WAN2, internal webserver with forwarded ports 80 and 443, and it should be possible to reach it using both public addresses. So you mark new incoming connections based on WAN interface, and then mark routing for responses. The one in prerouting is for server (traffic from LAN), other in output is for router itself, e.g. if you'd run VPN server on it, which should be also accessible using both addresses (this can be also done using routing rules).

Hi Sob, so you are thinking primary and failover and where the external users should be able to access either WAN to get to server.
The right solution there is to give users the dyndns URL to the ACTIVE WANIP. So as MYNETNAME, if the primary ISP goes down the secondary ISP should take over be it the IP cloud or some other service.

In other words, I can paint at least 5 different scenarios and its up to pcunite to come clean on the requirements, not playing a chasing game on this one LOL

Sob · Sun Apr 24, 2022 12:54 am

Two disadvantages:

1) DDNS change is not instant. There can be delay before router updates it, also on client side it takes some time before old cached record expires (not much, but it's not zero).
2) If there's long open connection (e.g. to VPN server) and it was made to backup WAN, you may not want to break it when primary comes back (although sometimes maybe you do, e.g. if backup has much worse parameters than primary, then it would be worth it to reconnect there).

pcunite · Sun Apr 24, 2022 2:05 am

Unless I misunderstood the request, multi-WAN router with e.g. 1.2.3.4 on WAN1 and 2.3.4.5 on WAN2, internal webserver with forwarded ports 80 and 443, and it should be possible to reach it using both public addresses.
I can paint at least 5 different scenarios and its up to pcunite to come clean on the requirements ...

Sob is correct. I'm asking about making connections to the router from multiple WAN inputs. There could be many reasons why someone would want this. I don't intend to challenge them. I'm asking about how to do it because this is an academic thread.

anav · Sun Apr 24, 2022 2:42 am

I get that but I need to understand the requirements to talk about the config. Saying I need multiwan inputs is bereft of detail............
PCUNITE, see new para K!

pcunite · Wed Apr 27, 2022 9:46 pm

I get that, but I need to understand the requirements to talk about the config ... PCUNITE, see new paragraph K!

Excellent! I shall test this soon. So, you are correct, I need an example. I shall present one here:

Some customers don't actively use fail-over and they don't actively load balance, not traditionally speaking. Instead, they use ISP1 for all standard traffic in their environment and an ISP2 is used only for VoIP and media streaming services. Thus, they have two egress and ingress WAN providers. However, both WAN1 and WAN2 host servers (applications listening on ports) that need to be reached from the outside. Thus even when ISP1 is primary, secondary ISP providers are much in use and need external control, monitoring, and accessibility by admins and users alike.

anav · Wed Apr 27, 2022 11:10 pm

I am of the same boat, I have some uses for WAN2 only, in that case route rules work fine, no mangling required.
Really the only reason to mangle if is you have incoming traffic on both wans to servers and you have a complex set of users using the different wans.

pcunite · Fri Apr 29, 2022 3:41 pm

I'm attempting to configure the Multi-WAN Input To Servers (with mangling) option. I appeal for assistance as I'm unable to get this to work.

Ether1 is ISP1, Ether3 is ISP3, (Ether2 is not plugged in at the moment). While pinging ISP3's static IP, I can see ping attempts being counted in the following rule:

add action=mark-connection chain=prerouting connection-state=new disabled=no in-interface=ether3 new-connection-mark=WAN3_conn passthrough=yes

but they never make it to new-routing-mark:

add action=mark-routing chain=prerouting connection-mark=WAN3_conn disabled=no in-interface-list=LAN new-routing-mark=ISP3_route passthrough=yes

Here is my sanitized configuration. Anything stand out to you?

#
# April 29, 2022 RouterOS 6.47.10
# model = RB4011iGS+
#

/system identity
set name=WAN_FAIL_OVER_WITH_REMOTE

#
# BRIDGE
#

/interface bridge
add admin-mac=AA:BB:CC:DD:EE:FF auto-mac=no name=BR1 protocol-mode=none

/interface bridge port
add bridge=BR1 interface=ether4
add bridge=BR1 interface=ether5
add bridge=BR1 interface=ether6
add bridge=BR1 interface=ether7
add bridge=BR1 interface=ether8
add bridge=BR1 interface=ether9
add bridge=BR1 interface=ether10
add bridge=BR1 interface=sfp1

/interface bridge settings set allow-fast-path=no

/interface ethernet
set [ find default-name=ether1 ] comment=ISP1_PPPoE
set [ find default-name=ether2 ] comment=ISP2_LTE
set [ find default-name=ether3 ] comment=ISP3_T1_Static
set [ find default-name=ether6 ] comment=""
set [ find default-name=ether7 ] comment=""
set [ find default-name=ether8 ] comment=""
set [ find default-name=sfp-sfpplus1 ] name=sfp1

# MGMT access to wAP R ac
/interface vlan
add comment="LTE MGMT" interface=ether2 name=BASE_VL vlan-id=99

/interface list
add name=WAN
add name=LAN
add name=BASE

#
# DHCP standard stuff
#

/ip dhcp-server option
add code=66  name="ZeroConfig Server Path" value="'https://10.2.2.2:8089/zccgi'"
add code=15  name=domain-name value="'MyDomain.com'"
add code=119 name=domain-search value="'MyDomain.com'"

/ip pool
add name=CORP_POOL ranges=10.1.1.20-10.1.1.254
add name=VOIP_POOL ranges=10.2.2.2-10.2.2.254
add name=BASE_POOL ranges=192.168.3.20-192.168.3.254

/ip dhcp-server
add address-pool=CORP_POOL bootp-support=none disabled=no interface=BR1 lease-time=2d name=CORP_DHCP
add address-pool=VOIP_POOL bootp-support=none interface=BR1 lease-time=2d name=VOIP_DHCP
add address-pool=BASE_POOL bootp-support=none interface=BR1 lease-time=2d name=BASE_DHCP

/ip dhcp-server network
add address=10.1.1.0/24 comment=CORP dhcp-option=domain-name,domain-search dns-server=192.168.3.1 domain=MyDomain.com gateway=10.1.1.1
add address=10.2.2.0/24 comment=VOIP dns-server=192.168.3.1 gateway=10.2.2.1
add address=192.168.3.0/24 comment=BASE dns-server=192.168.3.1 gateway=192.168.3.1

/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.4.4,149.112.112.112

#
# Internet interfaces of a dynamic nature
#

/interface pppoe-client
add comment=ISP1_PPPoE disabled=no interface=ether1 name=PPPoE1 password="password" user="User"

/ip dhcp-client
add comment=ISP2_LTE   disabled=no add-default-route=no use-peer-dns=no use-peer-ntp=no interface=ether2 script="\r\n# Use for a clients with \"Add Default Route\" unchecked!\r\n:if (\$bound=1) do={\r\n    /ip route set [/ip route find where gateway!=\$\"gateway-address\" and comment~\"ISP2_LTE_Route\"] gateway=\$\"gateway-address\"\r\n\t:log info \"ISP2_LTE_Route gateway updated\";\r\n}"

/interface list member
add interface=ether1  list=WAN
add interface=ether2  list=WAN
add interface=ether3  list=WAN
add interface=PPPoE1  list=WAN
add interface=BR1     list=LAN
add interface=BASE_VL list=LAN

#
# IP Settings
#

/ip settings
set allow-fast-path=no rp-filter=strict secure-redirects=no tcp-syncookies=yes

#
# IP Assignments
#

/ip address
add address=10.1.1.1/24    interface=BR1 network=10.1.1.0 comment=Corp
add address=10.2.2.1/24    interface=BR1 network=10.2.2.0 comment=VoIP
add address=192.168.3.1/24 interface=BR1 network=192.168.3.0 comment=Base
add address=192.168.4.1/24 interface=BASE_VL network=192.168.4.0 comment="LTE MGMT"
add address=ISP3.x.x.x/27  interface=ether3 network=ISP3.x.x.x comment=ISP3_T1_Static

#
# IP Routes
#

/ip route

# ISP1
add distance=1 gateway=ISP1.x.x.x routing-mark=ISP1_Route comment=ISP1
add check-gateway=ping distance=1 gateway=ISP1.x.x.x      comment=ISP1
add check-gateway=ping distance=1 gateway=8.8.8.8 routing-mark=ISP1_Route scope=10 target-scope=12 comment=Host1
add check-gateway=ping distance=2 gateway=9.9.9.9 routing-mark=ISP1_Route scope=10 target-scope=12 comment=Host2
add distance=1 dst-address=8.8.8.8/32 gateway=ISP1.x.x.x scope=10 target-scope=11   comment=ISP1_check
add distance=2 dst-address=9.9.9.9/32 gateway=ISP1.x.x.x scope=10 target-scope=11   comment=ISP1_check
# ISP2
add distance=3  gateway=ISP2.x.x.x routing-mark=ISP2_Route                          comment=ISP2_LTE_Route
add distance=10 gateway=ISP2.x.x.x scope=10 target-scope=30 routing-mark=ISP2_Route comment=ISP2_LTE_Route
#ISP3
add distance=1 gateway=ISP3.x.x.x routing-mark=ISP3_Route comment=ISP3_T1_Static_Route
add distance=3 gateway=ISP3.x.x.x                         comment=ISP3_T1_Static_Route
#Other
add check-gateway=ping comment=Host1 distance=1 gateway=8.8.8.8 scope=10 target-scope=12
add check-gateway=ping comment=Host2 distance=2 gateway=9.9.9.9 scope=10 target-scope=12
#Other2
add comment=WireGuard distance=1 dst-address=10.0.60.0/24 gateway=10.1.1.201

/ip firewall address-list
add address=0.us.pool.ntp.org list=1PoolNtpOrg comment=NTP_POOL
add address=10.1.1.0/24    list=LANSUBNETS
add address=10.2.2.0/24    list=LANSUBNETS
add address=192.168.3.0/24 list=LANSUBNETS
add address=192.168.4.0/24 list=LANSUBNETS

/ip firewall filter
add action=accept chain=input   comment="Allow Estab & Related" connection-state=established,related
add action=accept chain=input   comment="Allow LAN DNS" dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=input   comment="Allow Ping" protocol=icmp
add action=accept chain=input   comment="Allow MGMT to MikroTik" in-interface=BR1
add action=drop chain=input     comment=Drop
add action=accept chain=forward comment="Allow Estab & Related" connection-state=established,related
add action=accept chain=forward comment="Allow LAN Internet Access" connection-state=new in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="Allow port forwards" connection-nat-state=dstnat in-interface-list=WAN
add action=drop chain=forward   comment=Drop

#
# Mangle rules to Mark ISP_Routes
#

/ip firewall mangle
add action=mark-connection chain=prerouting comment=Mark_ISP connection-state=new disabled=no in-interface=ether1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_conn disabled=no in-interface-list=LAN new-routing-mark=ISP1_route passthrough=yes
add action=mark-connection chain=prerouting connection-state=new disabled=no in-interface=ether2 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn disabled=no in-interface-list=LAN new-routing-mark=ISP2_route passthrough=yes
add action=mark-connection chain=prerouting connection-state=new disabled=no in-interface=ether3 new-connection-mark=WAN3_conn passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn disabled=no in-interface-list=LAN new-routing-mark=ISP3_route passthrough=yes

# Other markings for QoS behaviour
add action=mark-connection chain=prerouting comment=DNS connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=postrouting connection-mark=DNS new-packet-mark=DNS passthrough=no
add action=mark-connection chain=prerouting comment=VOIP new-connection-mark=VOIP passthrough=yes src-address=10.2.2.2
add action=change-dscp chain=prerouting disabled=yes new-dscp=40 passthrough=yes port=5060-5062,10000-10500 protocol=udp
add action=mark-packet chain=prerouting connection-mark=VOIP new-packet-mark=VOIP passthrough=no
add action=mark-connection chain=prerouting comment=QUIC connection-state=new new-connection-mark=QUIC passthrough=yes port=80,443 protocol=udp
add action=mark-packet chain=prerouting connection-mark=QUIC new-packet-mark=QUIC passthrough=no
add action=mark-connection chain=prerouting comment=UDP connection-state=new new-connection-mark=UDP passthrough=yes protocol=udp
add action=mark-packet chain=prerouting connection-mark=UDP new-packet-mark=UDP passthrough=no
add action=mark-connection chain=prerouting comment=ICMP connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=prerouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-connection chain=postrouting connection-state=new new-connection-mark=ICMP passthrough=yes protocol=icmp
add action=mark-packet chain=postrouting connection-mark=ICMP new-packet-mark=ICMP passthrough=no
add action=mark-packet chain=postrouting comment=ACK new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack
add action=mark-packet chain=prerouting new-packet-mark=ACK packet-size=0-123 passthrough=no protocol=tcp tcp-flags=ack

/ip firewall nat
add action=masquerade chain=srcnat comment="Default masquerade" out-interface-list=WAN
add action=dst-nat chain=dstnat comment="Force DNS" dst-port=53 protocol=udp to-addresses=192.168.3.1 to-ports=53
add action=dst-nat chain=dstnat dst-port=53 protocol=tcp to-addresses=192.168.3.1 to-ports=53

anav · Fri Apr 29, 2022 11:00 pm

Your routes are not setup correctly! The gateway to the actual ISP is only used once per ISP set of routes!!
From this .......
.........

# ISP1
add distance=1 gateway=ISP1.x.x.x routing-mark=ISP1_Route comment=ISP1
add check-gateway=ping distance=1 gateway=ISP1.x.x.x      comment=ISP1
add check-gateway=ping distance=1 gateway=8.8.8.8 routing-mark=ISP1_Route scope=10 target-scope=12 comment=Host1
add check-gateway=ping distance=2 gateway=9.9.9.9 routing-mark=ISP1_Route scope=10 target-scope=12 comment=Host2
add distance=1 dst-address=8.8.8.8/32 gateway=ISP1.x.x.x scope=10 target-scope=11   comment=ISP1_check
add distance=2 dst-address=9.9.9.9/32 gateway=ISP1.x.x.x scope=10 target-scope=11   comment=ISP1_check

To THIS
.................

add check-gateway=ping distance=1 gateway=8.8.8.8     scope=10 target-scope =12
add check-gateway=ping distance=1 gateway=8.8.8.8 routing-mark=ISP1_Route  scope=10 target-scope=12  comment=Host1
add distance=1 dst-address=8.8.8.8/32 gateway=ISP1.x.x.x scope=10 target-scope=11 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping distance=1 gateway=9.9.9.9   scope=10 target-scope=12
add check-gateway=ping distance=2 gateway=9.9.9.9 routing-mark=ISP1_Route scope=10 target-scope=12 comment=Host2
add distance=2 dst-address=9.9.9.9/32 gateway=ISP1.x.x.x scope=10 target-scope=11

.............

Then looking at your secondary and tertiary fall back routes..............
From this
.........

# ISP2
add distance=2  gateway=ISP2.x.x.x routing-mark=ISP2_Route                          comment=ISP2_LTE_Route
add distance=10 gateway=ISP2.x.x.x scope=10 target-scope=30 routing-mark=ISP2_Route comment=ISP2_LTE_Route
#ISP3
add distance=1 gateway=ISP3.x.x.x routing-mark=ISP3_Route comment=ISP3_T1_Static_Route
add distance=3 gateway=ISP3.x.x.x                         comment=ISP3_T1_Static_Route

.................

To THis (had one too many routing-mark assigned on isp2 routes)
............

add distance=3  gateway=ISP2.x.x.x   scope=10 target scope=30  comment=ISP2_LTE_Route
add distance=3 gateway=ISP2.x.x.x  routing-mark=ISP2_Route scope=10 target-scope=30
#ISP3
add distance=4 gateway=ISP3.x.x.x routing-mark=ISP3_Route  scope=10 target-scope=30
add distance=4 gateway=ISP3.x.x.x   scope=10 target-scope=30  comment=ISP3_T1_Static_Route

..............

Finally get rid of these NOT required......
#Other
add check-gateway=ping comment=Host1 distance=1 gateway=8.8.8.8 scope=10 target-scope=12
add check-gateway=ping comment=Host2 distance=2 gateway=9.9.9.9 scope=10 target-scope=12

anav · Fri Apr 29, 2022 11:13 pm

Final comment since you mangle ICMP traffic there is no guarantee that ping will work, based on the errors so far LOL. Suggest removing that mangling, but perhaps you have legit reasons for frigging with it

Sob · Sat Apr 30, 2022 1:55 am

@pcunite: Multi WAN and rp-filter=strict are not friends.

pcunite · Sat Apr 30, 2022 3:45 am

@pcunite: Multi WAN and rp-filter=strict are not friends.

Oh my goodness. The rp-filter manual states: Warning: strict mode does NOT work with routing tables. Opps! Now that I've changed this, I can access internal servers now from both WAN ports! Woo-hoo! Adding input/output rules took care of ping. Very interesting. What is a work around for strict mode (RFC3704)?

This PDF states: "Underlying principle of rp_filter is to block outbound traffic if the IP does not belong to the subnet that resides on the LAN" Okay, so I need a list of what I consider valid? Other comments I see are: "if the interface is not the best reverse path the packet check will fail" Well, true I guess because it is going out a different interface from default. I think a firmware update should address this. These packets can be tracked.

anav · Sat Apr 30, 2022 4:04 am

As per my amended Guide, for multi-wan rp-filter=loose, and of course if mangling involved fastrack=disabled!
@pcunite just ensure that tcp tracking is turned on, AND uncheck the box for loose tracking - WE WANT strict tracking, which makes our invalid rule have more teeth.

Sob · Sat Apr 30, 2022 12:45 pm

Yes, common suggestion is to use rp-filter=loose, but according to the thread linked by @pcunite, if it takes into account default routes (that cover any possible address), it's useless for this kind of multi WAN configs.

anav · Sat Apr 30, 2022 5:34 pm

Yes, common suggestion is to use rp-filter=loose, but according to the thread linked by @pcunite, if it takes into account default routes (that cover any possible address), it's useless for this kind of multi WAN configs.

You know very well that being obtuse is not helpful and will cause me to post again! ;-P
RP filter loose will NOT cause problems in multi-wan scenarios.
RP filter strict will very likely cause problems in multi-wan scenarios.

Hence the guidance I will continue to provide in multi-wan scenarios is to ensure RP-filter is set to loose.
Yes there may be some ways around it, but until someone like pcunite plays with it, tests it, finds the boundary conditions etc etc, then the guidance wont change. An article is not good enough.
Of course if Sindy provides the guidance, then I wont have to wait for pcunite.

Sob · Sat Apr 30, 2022 6:58 pm

It's not that rp-filter=loose would cause problems, only in this scenario it may not provide any benefits over rp-filter=no (default).

anav · Sat Apr 30, 2022 7:30 pm

Certainly no EXTRA benefits, thats for sure but then also recommended tcp tracking strict (which makes invalid rule stronger).

pcunite · Mon May 02, 2022 3:23 pm

Yes there may be some ways around it, but until someone like pcunite plays with it, tests it, finds the boundary conditions etc etc, then the guidance wont change. An article is not good enough. Of course if Sindy provides the guidance, then I wont have to wait for pcunite.

I know so very little. Very grateful to this forum for the help. I'm good at formatting documents! But I lack proper understanding. Anav is a master at tearing up configurations (editing), and mkx, sindy, Sob, Zacharias, pe1chl, and so many others (forgive me if I don't mention your name) have the actual expertise. Then a host of silent professionals too timid to speak what they know. Maybe they can't. Look at my QoS article that rots. I simply don't know enough to fix it.

It would be cool if we could form a website, and thereby crowdsourse the documentation and formatting the way we know is best, and then backport it into the wiki (assuming MikroTik would accept it). I don't care if someone uses our hard work as long as knowledge is shared. Silence is hindering MikroTik. People just need answers, not arrogance. So, imagine a monthly posting schedule to were we address every issue, but it is formatted on a website that makes it easy to learn and understand the concepts.

It took me an entire month to prove the VLAN article. It took the creation of a lab, testing, seeking help from strangers, stringing words together, and making images. But with the help of all of us, maybe we could create something truly special?

Larsa · Mon May 02, 2022 3:37 pm

It would be cool if we could form a website, and thereby crowdsourse the documentation and formatting the way we know is best, and then backport it into the wiki (assuming MikroTik would accept it). I don't care if someone uses our hard work as long as knowledge is shared. Silence is hindering MikroTik. People just need answers, not arrogance. So, imagine a monthly posting schedule to were we address every issue, but it is formatted on a website that makes it easy to learn and understand the concepts.

Concur, a very good suggestion! Have had the same thoughts. There are so many good solutions found in the forums that should be gathered to a common place as "best practice" or something like it. Or maybe Sob/Sindy might want to write "the" book "RoS Best Practice"...

pcunite · Mon May 02, 2022 5:30 pm

There are so many good solutions found in the forums that should be gathered to a common place as "best practice" or something like it. Or maybe Sob/Sindy might want to write "the" book "RoS Best Practice"...

I like, "MikroTik, the definitive guide"

Well, let's have a Zoom meeting. I invite sindy, mkx, anav, amm0, larsa, chupaka, pe1chl, sob, and zacharias. I invite anyone willing and capable in being apart of the formation of a new website and articles series that addresses the most pressing configuration needs the MikroTik community faces. These will ultimately be turned into a PDF and or a printed book. The weary MikroTik sojourner need no longer to despair. "MikroTik, the definitive guide. Best practices from the community. Volume 1"

Mon May 02, 2022 5:35 pm

Not as knowledgeable as the others but I can proof-read. Easier with time-zone differences too.

If I don't understand something right away, there is a possibility some more beefing might be needed

anav · Mon May 02, 2022 8:02 pm

Hmmm, I am not sure I can add any academic knowledge to the endevour. I am as stated good at tearing things up so if you need paper shredded................
One thing is to think about the audience and intentions. I am trying to make things easier for beginners and helping them on a journey to understand what the config means, which is more important than getting it right the first time (copy and paste someone elses works doesnt get one very far as I can attest too)..

pcunite · Mon May 02, 2022 8:57 pm

One thing is to think about the audience and intentions. I am trying to make things easier for beginners and helping them on a journey to understand what the config means, which is more important than getting it right the first time (copy and paste someone elses works doesnt get one very far as I can attest too)..

Well, the website would actually be for you. It will not replace what you do in the forums, but augment the "how". So, imagine that you use your vast expertise with endless bad configurations, and help us build proper ones that make sense. You will still be in the forums (the website is not a replacement for the official forums) and be linking to documents you helped create.

The links you have here are a mess (not your fault). Imagine linking to something that has good formatting and lengthy or short topics depending on the user you're working with. Now, I'm trying to sell you! Haha, but if it doesn't make sense, we don't have to do it.

You did a lot of linking to my VLAN article and I do appreciate it. But it was created for you and the community. But the community needs more than I can or even you can provide. It needs a series of answers offsite but that directs them back here as well.

pcunite · Mon May 02, 2022 9:31 pm

Not as knowledgeable as the others but I can proof-read. Easier with time-zone differences too.

All of our past experiences bring something to the table. I'm involved with MikroTik mostly from dealing with phone systems. I'm extremely weak in other areas. When I need help, I'm lost really quick. I think a lot of people could use a website_to_PDF presentation that is enjoyable to read and leads you along many of the common concepts that get asked here alot.

It is such a waste for some of the forum's brightest minds to wrestle in the weeds while trying to simultaneously teach concepts in a forum with bad configurations and methodology over and over. I was needing help recently and I had to spend hours reading many random threads only to come up with an invalid answer. This is madness.

So, imagine an improved format of the vlan article. I've rightly received criticism on the syntax format I used because it doesn't mirror the export command. I was going for a balance between maximum shortest line + easy on the eyes + teaching a concept. But creating a version that closely mirrors export would help to prevent tripping up new users who are trying to edit the configuration. On the proposed website, I could show different styles because I could create efficient links, formatting, and versioning.

But I'm useless without the knowledge that comes from others here. I don't have the answers people need, I'm not good at working with bad configurations, yadda, yadda.

pcunite · Mon May 02, 2022 9:43 pm

I am trying to make things easier for beginners and helping them on a journey to understand what the config means, which is more important than getting it right the first time (copy and pasting someone else's works doesn't get one very far as I can attest too)

Very true. The VLAN article in many ways has failed. Something better is needed and I don't think the forum itself is the answer. The amount of work you're doing is incredible and I wonder if you can sustain it. In my line of work we call it "customer journey". I do think more teaching is needed but alongside the syntax (and maybe with GUI examples). I don't know. Something needs to be different because I can't get the help I need. Is the forum truly for helping or contesting? I think we can help and it not be a great burden on everyone.

So, a website under the control of the forum's top editors (fifteen of us?) and we build, test, create, and promote configurations. We promote the knowledge here that was built, presented, and hosted elsewhere. We crowdsource the RouterOS user experience. Something MikroTik is unable or unwilling to do. I think I could get a corporate sponsor to pay for the hosting etc.

Sob · Tue May 03, 2022 1:23 am

I have to confess something (semi-seriously), I'm in this forum to hide from work (not in a dishonest way, that I'd get paid for doing something else while being here, it's all my "free" time) and other responsibilities, procrastination is the popular term for it.

Related to that, I suffer from perfectionism. I can point out a mistake in someone's config, give them hints, or play with some config because it's fun, etc. It's easy, because there are no commitments or anything (although... I tend to not abandon threads and sometimes I get the feeling that I should have left some to someone else; but it doesn't happen very often). But it's completely different to create serious quality document, because such thing should ideally explain everything, and it adds up. It quickly turns into work, and... see the beginning. I have writing up some things (like @anav is doing) in my virtual TODO list for years, but somehow I can't get to it.

Also related, I hate to promise anything to anyone, because when I do and it doesn't go as planned, it doesn't help with everything else. So while I may provide some help or feedback, meaning that I'm not against it, rather don't expect too much from me.

anav · Tue May 03, 2022 2:47 am

Yes Please SOB, leave me the easy ones, and just take the hard ones!!

pcunite · Tue May 03, 2022 3:29 am

I have to confess something ...
<SNIP>
... it's completely different to create serious quality document ...

Great points. Yeah, sometimes I check out the forums to decompress too. And making a website with perfect configurations is a lot of effort! If you have a full time life, its just another thing to get done before bed. I get that. Well, its fun to hope we might get good documentation someday!

pcunite · Thu May 12, 2022 1:12 am

A question about Section J. DIRECTING Users out Specific WAN. Say I'm directing a specific IP address out WAN2 (10.2.2.2 is a VoIP server). That's great, however, the LAN can no longer access it. So, do I need a rule like this?

/ip route rule add action=lookup-only-in-table dst-address=10.2.2.0/24 table=main

I need something to ensure that LAN access to that IP which is forced out WAN2, is still accessible from other default LAN traffic that is default routed out WAN1. I think this would be an incredibly desirable feature.

anav · Thu May 12, 2022 1:22 am

Not sure what you mean?
Routing traffic out a specific WANIP, should not interfere with LAN to LAN traffic or VLAN to VLAN traffic which is more controlled by firewall rules.

The reason I can say this is because Vlan to Vlan traffic is more specific than 0.0.0.0/0 and thus the default routes (DAC) created by IP addresses and gateways of existing vlans or subnets will take precedence of the IP route out WAN1 or WAN2. The traffic should simply flow, if your firewall rules allow it.

Maybe I am overlooking something here, so dont quote me as being correct, its just a counter thought at the moment, trying to work through the logic!

pcunite · Thu May 12, 2022 1:27 am

Not sure what you mean?
Routing traffic out a specific WANIP, should not interfere with LAN to LAN traffic or VLAN to VLAN traffic which is more controlled by firewall rules.

I may have something mis-configured. Will test and report back.

Sob · Thu May 12, 2022 3:01 am

Of course it can interfere. If you use routing rule saying that internal address X should use specific routing table, which contains only default route to internet, then even traffic from X to other local subnets will be sent to internet according to given routing table. And @pcunite's fix with another routing rule is correct.

anav · Fri May 13, 2022 4:21 pm

Hmmm...
I so badly wanted to state Sob was wrong but after going around in circles...........

Ahhhhhhhhhhh I see your doing it by DESTINATION ADDRESS not by source address!!
Okay, now it makes sense.
Rather smart of you pcunite.............

Also the potential implications of forcing an IP out a specific WANIP cannot be overstated!!
But if you only had read my Para J, "SERVER EXAMPLE"

pcunite · Sun May 15, 2022 11:52 pm

Ahhhhhhhhhhh I see your doing it by DESTINATION ADDRESS not by source address!!
Okay, now it makes sense.
Rather smart of you pcunite.............

Also the potential implications of forcing an IP out a specific WANIP cannot be overstated!!
But if you only had read my Para J, "SERVER EXAMPLE"

Haha, well the credit goes to Sob. I saw a post he made a while back. That is why collections of information are so helpful. I've spent hours trying to sort all this out. When I get everything done, maybe I'll produce a monster document on how to really setup a multi-wan scenario that covers what most people need. Lots of practical things people need to know. There is a lot to it.

anav · Mon May 16, 2022 3:05 pm

Too difficult to cover every scenario but thats why the current explanation in the user article spans 3 paragraphs I, J, K
At some point in time, we can get together to revamp what is there into something more coherent........ both vlans and the other topics as you wish to progress.
We dont need the other trash talkers, sob mkx etc.... involved because they will be sure to correct all our faux pas, once such a revamp is done and thus we can expect to publish and then make the necessary changes that will come. Its like a running race when you pass the first place runner and are happily in the lead but your strategy involves knowing that others will be kicking by you and thus you have to mentally prepare to up the game shortly! Probably can look at doing this starting in July..... I expect that skype sessions will be in order.

Sob · Tue May 17, 2022 2:44 am

Don't cover every scenario, just explain how it works. It's great about machines, they simply follow exact steps and rules, no moods or whims, tell them what to do and they will do it.

Btw, info about "DUAL WAN - Simple" is wrong. First, switch to ISP2 will happen only if ISP1 fails in certain way, e.g. if DHCP is not able to renew lease. Then the gateway won't be reachable, because client won't have address and subnet that contains gateway. In this case, router will use route to ISP2. But if there's long DHCP lease and client isn't renewing it yet, gateway can be dead as Dodo and router won't notice. That's what check-gateway option is for. Second, when ISP1's route with lower distance becomes available, router will switch to it immediately. In short, you want following "DUAL WAN - FAILOVER", because "DUAL WAN - Simple" is pretty useless.

anav · Tue May 17, 2022 5:00 am

Done, good clarification that the check ping has nothing to with driving router selection it merely informs the router the route is available and the distance is the driving factor in the example.

Sat Jul 02, 2022 1:31 am

i think this topic deserves to be sticky at the top of the forum like others useful guides

rextended · Sat Jul 02, 2022 1:32 am

mmm..... can be....

"Anav's guide for successful configuration" ???

Sound better than an all uppercas title...

....

Sat Jul 02, 2022 8:02 am

+1

I hate all uppercase.
It's shouting !!

TheLorc · Sun Jul 31, 2022 2:33 am

Para M? What is that? Cant find it when i google?

rextended · Sun Jul 31, 2022 3:19 am

Seriously?

Para M. = Paragraph M.

TheLorc · Sun Jul 31, 2022 8:19 pm

Seriously?

Para M. = Paragraph M.

Yes, I thought he meant some Mikrotik youtuber or something

anav · Tue Aug 02, 2022 1:45 am

Reasonable pt. if 1s nat'v langa aint the queens E.
Fixed!!!

Fri Aug 05, 2022 9:35 am

@anav

Para H - Netinstall
Maybe worthwhile to also mention the Linux approach as alternative if Windows fails (which is the case for quite a lot of users) ?

anav · Fri Aug 19, 2022 4:04 pm

Done!

pcunite · Wed Jan 18, 2023 8:09 pm

Hello fellow forum compatriots! I have another article series I'm putting together and would like your valuable knowledge and skill to refine it for the benefit of the forum. When finished, it should be linked from here. This thread is a nice collection of user created content and shows the best documentation for MikroTik products.

The topic I've chosen to focus on is MultiWAN and is available as a work in progress here. Will take a month or more most likely to get everything worked out and tested. Your critiques are necessary to make it as accurate and useful as it can be.

The credit for the techniques and knowledge go to you. I'm merely putting the document together.

Look forward to your help.

DEGABB · Tue Jun 13, 2023 2:32 am

@anav I would like to draw your attention to K1. Introduction to basic mangling.
You have made a mistake in the mangling rules.
If you do this:
add action=mark-routing chain=prerouting connection-mark=WAN1 \
new-routing-mark=viaISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 \
new-routing-mark=viaISP2 passthrough=no
then all incoming packet to be place in dedicated routing tables (viaISP1,viaISP2), but they will never reach LAN, causing no route to LAN in these tables.
So you need to add the in-interface=LAN parameter to exclude remote packets from being placed in these tables.
Proper rules:
add action=mark-routing chain=prerouting connection-mark=WAN1 \
in-interface=LAN1 new-routing-mark=viaISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 \
in-interface=LAN1 new-routing-mark=viaISP2 passthrough=no

anav · Sat Jun 17, 2023 2:53 pm

Incorrect!

(1) The Mangling in this example is ONLY used to
a. marked connections for traffic coming in on each WAN
b. mark routing for this traffic.
The tables created will ensure any RETURN traffic for external originators coming in on WAN1 and heading for LAN servers, will go back out WAN1 and any traffic that comes in on WAN2, will have its return traffic go out WAN2.

(2) To ensure LAN1 goes out only WAN1 and LAN2 goes out only WAN2 is done via Routing Rules and the same tables used above.
There is no need to mangle for LAN subnets in this scenario.
The only complication that may arise is that the admin may want LAN1 to be accessible by LAN2 for example....... and thus we have to ensure such traffic is allowed before being captured by the routing rule and sent out WAN2 instead....... ( order counts )

/routing rules
add dst-address=LAN1 action=lookup-only-in-table table=main
add src-address=LAN1 action=lookup-only-in-table table=useWAN1
add src-address=LAN2 action=lookup-only-in-table table=useWAN2

anav · Sat Jun 17, 2023 3:00 pm

Reading your post more carefully, I think where you are erring in your thinking is that mangling is the same as routing, it is not!
We are simply putting marks on traffic for later use.
The traffic has a destination and that is port based and the public IP.
We mark it on the way in is all. The traffic is still inbound and routed by the destination nat rule, stating for port XX, send to internal IP x.x.x.x.x
(assuming we have the generic firewall rule permitting port forwarding). The router is aware of the destination IP and has a direct route to it.
The return traffic is what the router will look at for IP routes since the originating IP is not local, it will look for an IP route, see the route marking and ensures it goes out the proper WAN.

vincej · Fri Aug 04, 2023 8:58 pm

add dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
add dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=12
add dst-address=8.8.8.8/32 gateway=SecondaryISP-gatewayIP scope=10 target-scope=12
add dst-address=208.67.222.222/32 gateway=SecondaryISP-gatewayIP scope=10 target-scope=12

does this mean when PrimaryISP is down, 9.9.9.9 & 1.0.0.1 won't be accessible, and 8.8.8.8 & 208.67.222.222 will not be accessible as well when SecondaryISP is down?

anav · Sat Aug 05, 2023 1:50 am

No it means that the path to them is blocked somehwere along the line via the ISP being used.
( google and other DNS sites are still up and available, just not through the ISP).

We use recursive routing in order to ensure that although the connection to the ISP from your router may be good, we are then confirming that the routing from ISP to the WWW is also good.

broderick · Sat Aug 19, 2023 1:32 pm

@anav.
Hi

I took a look at: L. MULTI-WAN Input To Servers & Mangling { rp-filter=loose NOT strict / fastrack=disabled }

You wrote: Note: The Table and Mangle creation are identical for all examples
Just to make sure that I got it right, I wanted to know whether I also need to add these lines for example 2, 3, and 4:

ip routing table
add name=ISP1_route fib
add name=ISP2_route fib
.....

/ip firewall mangle
add chain=prerouting in-interface=WAN1 connection-state=new action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN1_conn action=mark-routing new-routing-mark=ISP1_route
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN2_conn action=mark-routing new-routing-mark=ISP2_route

I mean, Example 3: DUAL WAN FAILOVER TWO RECURSIVE (flat), does it need to be?

ip routing table
add name=ISP1_route fib
add name=ISP2_route fib

/ip route
add check=gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=ISP1_route
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main
....................................................................................................................................................................
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=main
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=9.9.9.9 scope=10 target-scope=12 table=ISP1_route
add distance=4 dst-address=9.9.9.9/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main
..............................................................................................
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30  table=ISP2_route

/ip firewall mangle
add chain=prerouting in-interface=WAN1 connection-state=new action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN1_conn action=mark-routing new-routing-mark=ISP1_route
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface-list=LAN connection-mark=WAN2_conn action=mark-routing new-routing-mark=ISP2_route

Thanks

anav · Sat Aug 19, 2023 2:55 pm

Good question and all scenario specific to multiple wans with failover!
In general, what we are doing is marking connections for each incoming WAN and then stating any traffic leaving the LAN with that mark should be routed to the same WAN it came in on.
So yes to your question!

However, since I wrote that article I prefer a different format. Both are equally interchangeable in this scenario.

article:
add chain=prerouting in-interface=WAN1 connection-state=new action=mark-connection new-connection-mark=WAN1_conn
Now:
add chain=prerouting in-interface=WANx connection-mark=no-mark action=mark-connection new-connection-mark=WANx_conn

broderick · Sat Aug 19, 2023 9:22 pm

In general, what we are doing is marking connections for each incoming WAN and then stating any traffic leaving the LAN with that mark should be routed to the same WAN it came in on.
So yes to your question!

okay.
I decided to bring my understanding of the Mikrotik device one step further, and I started reading articles and examples about policy routing and mangle, but it's often hard to make sense of each step in these types of setups (What exactly is this for? How does it work? Why should I choose one setup over another, etc). Unfortunately, the Mikrotik wiki is not always very helpful, in my humble opinion.

Thank you

anav · Sat Aug 19, 2023 11:12 pm

Post a new thread when you get halfway stuck

vincej · Wed Aug 23, 2023 12:53 pm

No it means that the path to them is blocked somehwere along the line via the ISP being used.
( google and other DNS sites are still up and available, just not through the ISP).

We use recursive routing in order to ensure that although the connection to the ISP from your router may be good, we are then confirming that the routing from ISP to the WWW is also good.

It doesn't seem to work like you described during a certain situation, for example, when Secondary-ISP route is down, but the Secondary-ISP gateway is up (that's why we use recursive routing in the first place), all traffic to 8.8.8.8 and 208.67.222.222 is forced to traverse through Secondary-ISP. No issues when replicating it in a scenario when the Secondary-ISP gateway is down.

My config, isp1 (distance 1) is pppoe-globe-residential, isp3 (distance 2) is pppoe-globe-biz-mivesa, isp2 (distance 3) is 192.168.253.1 to lte router, following the config based on "DTRIPLE WAN - RECURSIVE Courtesy of AzDsL"

# 2023-08-23 17:49:30 by RouterOS 7.11
# software id = 9PDL-AUGS
#
# model = RB750Gr3
# serial number = D
/ip route
add comment="isp2 route" distance=3 gateway=12.12.12.12 routing-table=main \
    scope=10 target-scope=14
add comment="isp2 route" distance=3 gateway=12.12.12.12 routing-table=isp1 \
    scope=10 target-scope=14
add comment="isp2 route" distance=3 gateway=12.12.12.12 routing-table=isp2 \
    scope=10 target-scope=14
add comment="isp2 route" distance=3 gateway=12.12.12.12 routing-table=isp3 \
    scope=10 target-scope=14
add check-gateway=ping comment="isp1 health check" dst-address=10.10.10.10/32 \
    gateway=9.9.9.9 scope=10 target-scope=13
add check-gateway=ping comment="isp1 health check" dst-address=10.10.10.10/32 \
    gateway=1.0.0.1 scope=10 target-scope=13
add check-gateway=ping comment="isp3 health check" disabled=no distance=1 \
    dst-address=11.11.11.11/32 gateway=8.8.8.8 pref-src="" routing-table=main \
    scope=10 suppress-hw-offload=no target-scope=13
add check-gateway=ping comment="isp3 health check" disabled=no distance=1 \
    dst-address=11.11.11.11/32 gateway=208.67.222.222 pref-src="" \
    routing-table=main scope=10 suppress-hw-offload=no target-scope=13
add check-gateway=ping comment="isp2 health check" disabled=no distance=1 \
    dst-address=12.12.12.12/32 gateway=8.8.4.4 pref-src="" routing-table=main \
    scope=10 suppress-hw-offload=no target-scope=13
add check-gateway=ping comment="isp2 health check" disabled=no distance=1 \
    dst-address=12.12.12.12/32 gateway=4.2.2.2 pref-src="" routing-table=main \
    scope=10 suppress-hw-offload=no target-scope=13
add comment="isp2 route" disabled=no distance=1 dst-address=8.8.4.4/32 \
    gateway=192.168.253.1 pref-src="" routing-table=main scope=10 \
    suppress-hw-offload=no target-scope=12
add comment="isp2 route" disabled=no distance=1 dst-address=4.2.2.2/32 \
    gateway=192.168.253.1 pref-src="" routing-table=main scope=10 \
    suppress-hw-offload=no target-scope=12
add comment="isp1 route" distance=1 gateway=10.10.10.10 routing-table=main \
    scope=10 target-scope=14
add comment="isp1 route" distance=1 gateway=10.10.10.10 routing-table=isp1 \
    scope=10 target-scope=14
add comment="isp1 route" distance=1 gateway=10.10.10.10 routing-table=isp2 \
    scope=10 target-scope=14
add comment="isp1 route" distance=1 gateway=10.10.10.10 routing-table=isp3 \
    scope=10 target-scope=14
add comment="isp1 route" dst-address=9.9.9.9/32 gateway=\
    pppoe-globe-residential scope=10 target-scope=12
add comment="isp1 route" dst-address=1.0.0.1/32 gateway=\
    pppoe-globe-residential scope=10 target-scope=12
add comment="isp3 route" disabled=no distance=2 dst-address=0.0.0.0/0 \
    gateway=11.11.11.11 pref-src="" routing-table=main scope=10 \
    suppress-hw-offload=no target-scope=14
add comment="isp3 route" disabled=no distance=2 dst-address=0.0.0.0/0 \
    gateway=11.11.11.11 pref-src="" routing-table=isp1 scope=10 \
    suppress-hw-offload=no target-scope=14
add comment="isp3 route" disabled=no distance=2 dst-address=0.0.0.0/0 \
    gateway=11.11.11.11 pref-src="" routing-table=isp2 scope=10 \
    suppress-hw-offload=no target-scope=14
add comment="isp3 route" disabled=no distance=2 dst-address=0.0.0.0/0 \
    gateway=11.11.11.11 pref-src="" routing-table=isp3 scope=10 \
    suppress-hw-offload=no target-scope=14
add comment="isp3 route" disabled=no distance=1 dst-address=8.8.8.8/32 \
    gateway=pppoe-globe-biz-mivesa pref-src="" routing-table=main scope=10 \
    suppress-hw-offload=no target-scope=12
add comment="isp3 route" disabled=no distance=1 dst-address=208.67.222.222/32 \
    gateway=pppoe-globe-biz-mivesa pref-src="" routing-table=main scope=10 \
    suppress-hw-offload=no target-scope=12

anav · Wed Aug 23, 2023 4:33 pm

If you have a config that needs help, post it as a new thread and I will have a look.

Who is online