Community discussions

 
lrn23
just joined
Topic Author
Posts: 24
Joined: Mon Jan 07, 2019 10:24 am

routing - 3x GW, failover

Fri May 10, 2019 1:46 pm

Hi guys,
I'm trying to setup my router and I don't know which way I should go. I have 3 different ISP connected, I want to have 1 ISP as a default one, one subnet should use another ISP and next subnet should use the last IPS. In case of outage on some ISP link, there should be working automatic failover. I have few questions now. How to achieve this routing logic? Is the use of "ip route rule" desired way how to setup things like this? I found one example: https://awarmanf.wordpress.com/2010/01/ ... cyrouting/. Is this good configuration? Or should I do it only with mangle and "ip route"? And what about failover? Is this good working solution: https://wiki.mikrotik.com/wiki/Advanced ... _Scripting? Thank you for every suggestion!
 
pe1chl
Forum Guru
Forum Guru
Posts: 5369
Joined: Mon Jun 08, 2015 12:09 pm

Re: routing - 3x GW, failover

Fri May 10, 2019 2:23 pm

It is the basic way of configuring it, yes. Use policy routing to route depending on your local subnet, use multiple default gw at different distance to achieve your failiver.
You need to decide what criteria you want to use for "not working ISP".
You can use ping or arp check of their end of the connection (check= on default route), or you can check some system further down on internet and use recursive routing.
 
lrn23
just joined
Topic Author
Posts: 24
Joined: Mon Jan 07, 2019 10:24 am

Re: routing - 3x GW, failover

Tue May 14, 2019 4:14 pm

OK, thank you! :) So, this could be a working configuration? I have to be sure, because I have to configure this in remote location so I can't test it properly :(

WAN interface ISP1 - 10.0.1.1
WAN interface ISP2 - 10.0.2.1
WAN interface ISP3 - 10.0.3.1

LAN subnets:
192.168.1.0/24
192.168.2.0/24
192.168.3.0/24
/ip route
add dst-address=8.8.8.8 gateway=10.0.1.1 scope=10
add dst-address=8.8.4.4 gateway=10.0.2.1 scope=10
add dst-address=1.1.1.1 gateway=10.0.3.1 scope=10

add distance=1 gateway=8.8.8.8 routing-mark=IPS1 check-gateway=ping
add distance=2 gateway=8.8.4.4 routing-mark=IPS1 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS1 check-gateway=ping

add distance=1 gateway=8.8.4.4 routing-mark=IPS2 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS2 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS2 check-gateway=ping

add distance=1 gateway=1.1.1.1 routing-mark=IPS3 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS3 check-gateway=ping
add distance=3 gateway=8.8.4.4 routing-mark=IPS3 check-gateway=ping

/ip route rule
add dst-address=192.168.1.0/24 action=lookup table=main
add dst-address=192.168.2.0/24 action=lookup table=main
add dst-address=192.168.3.0/24 action=lookup table=main

add dst-address=10.0.1.0/24 action=lookup table=main
add dst-address=10.0.2.0/24 action=lookup table=main
add dst-address=10.0.3.0/24 action=lookup table=main

add src-address=10.0.1.0/24 action=lookup table=IPS1
add src-address=10.0.2.0/24 action=lookup table=IPS2
add src-address=10.0.3.0/24 action=lookup table=IPS3

add routing-mark=IPS1 action=lookup table=IPS1
add routing-mark=IPS2 action=lookup table=IPS2
add routing-mark=IPS3 action=lookup table=IPS3

/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=IPS1 passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=IPS2 passthrough=no src-address=192.168.2.0/24
add action=mark-routing chain=prerouting new-routing-mark=IPS3 passthrough=no src-address=192.168.3.0/24
 
NetWorker
Frequent Visitor
Frequent Visitor
Posts: 61
Joined: Sun Jan 31, 2010 6:55 pm

Re: routing - 3x GW, failover

Tue May 14, 2019 6:46 pm

Not quite. You either do it with mangle or with routing rules as pe1chl suggested. Also, you're missing the default gateways.
/ip route
#first add the default gateways with distance for failover
add dst-address=0.0.0.0/0 gateway=10.0.1.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.0.2.1 distance=2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.0.3.1 distance=3 check-gateway=ping

#next add the routes for each subnet.
add dst-address=0.0.0.0/0 gateway=10.0.1.1 check-gateway=ping routing-mark=ISP1
add dst-address=0.0.0.0/0 gateway=10.0.2.1 check-gateway=ping routing-mark=ISP2
add dst-address=0.0.0.0/0 gateway=10.0.3.1 check-gateway=ping routing-mark=ISP3

#finally add the rules for each subnet
/ip route rule
add src-address=192.168.1.0/24 action=lookup table=ISP1
add src-address=192.168.2.0/24 action=lookup table=ISP2
add src-address=192.168.3.0/24 action=lookup table=ISP3
Logic is as follows:
You add three default gateways. All traffic that doesn't have a routing mark will be routed to ISP1. If that connection fails, it will use ISP2 and if both ISP1 and ISP2 are offline, it will use ISP3.
Next you add the three routes that each subnet will follow. If all is running, each connection will follow it's own route. If a ping check fails, both routes will deactivate; the one with the routing mark and the default one. Now, all traffic with that routing mark will follow the default gateway logic.
Finally you add the three rules for each subnet.

That should be all you need though I can't test it for you.
 
NetWorker
Frequent Visitor
Frequent Visitor
Posts: 61
Joined: Sun Jan 31, 2010 6:55 pm

Re: routing - 3x GW, failover

Tue May 14, 2019 7:21 pm

Forgot to point out two things. First was already covered by pe1chl. If you only ping the gateway and there's a problem further down the line in the ISP, those routes will stay up but you'll drop all the traffic. I.e. the failover won't happen. More complex checking schemes will require scripts.

Second, bear in mind that failover will dump all the traffic from two lansubnets on a single connection. I.e. suppose ISP2 fails, now all the traffic from Subnets 1 and 2 will be on ISP1. This could lead to dropped traffic on the ISP1 line. Now if you use the highest and most reliable line as ISP1 and put all your heavy consumers in lansubnet1, if that line ever fails, your ISP2 line will be taking an unacceptable load and drop so much traffic that the failover is hardly doing you any favors. Therefore, assuming similarly reliable lines, it's best to also use load balancing schemes in multiwan scenarios. This however adds complexity and is not as simple as discussed above.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5369
Joined: Mon Jun 08, 2015 12:09 pm

Re: routing - 3x GW, failover

Tue May 14, 2019 8:19 pm

Forgot to point out two things. First was already covered by pe1chl. If you only ping the gateway and there's a problem further down the line in the ISP, those routes will stay up but you'll drop all the traffic. I.e. the failover won't happen. More complex checking schemes will require scripts.
That is not really true. You can set a specific route for a single IP via some ISP and with a ping check, then add a default route via that specific IP.
This is called "recursive routing". It can deliver failover with checking further down the line without requiring scripting.

However, note that:
- you should also add routes to the same destination with higher distance via the other ISPs to keep that particular IP reachable when some ISP is down.
(and it is not really advisable to use widely-used IPs like 8.8.8.8 for the checking, to avoid user problems in that case)
- by using a ping check of a remote system, you make the usability of the link dependent on the availability of the remote system. when it is down, your ISP link will be declared down even when it really isn't. remember that the remote system may decide that it had enough of the pings and just block them, which you won't notice when you do not monitor the router closely.
 
sindy
Forum Guru
Forum Guru
Posts: 3527
Joined: Mon Dec 04, 2017 9:19 pm

Re: routing - 3x GW, failover

Tue May 14, 2019 9:03 pm

I have to be sure, because I have to configure this in remote location so I can't test it properly :(
This indicates to me that you access the device remotely, and therefore you can lock yourself out if you make a mistake.
So first, whatever change you do should be done using safe mode (Ctrl-X in command line toggles safe mode for changes done in the same command line window; a button in WebFig and/or Winbox does the same for changes done using that GUI, but you can use safe mode only on one interface at a time).
Second, the way you use the failover between your 3 WANs so far is not sufficient to access it remotely via WAN. There are two possibilities how to deal with this:
  • the simpler to configure one is to configure a static route towards the IP from which you'll be configuring into the default routing table (i.e. without a routing-mark) and also one per each routing-mark you'll be using, all via the same WAN through which you'll be accessing the device remotely,
  • a more complex but more useful one is not to assign the routing-mark directly but to use connection-mark as a base for it, which allows you to let any connection from outside be answered via the same WAN interface through which it came in - see details here (start reading that post from the last paragraph which gives you the context otherwise scattered across the topic)
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
NetWorker
Frequent Visitor
Frequent Visitor
Posts: 61
Joined: Sun Jan 31, 2010 6:55 pm

Re: routing - 3x GW, failover

Tue May 14, 2019 11:16 pm

- by using a ping check of a remote system, you make the usability of the link dependent on the availability of the remote system. when it is down, your ISP link will be declared down even when it really isn't. remember that the remote system may decide that it had enough of the pings and just block them, which you won't notice when you do not monitor the router closely.
This is the main reason I dislike recursive routing. You don't have control over how many pings are sent, how many are dropped and you need to add a route for each remote address you want to ping. A script allows full control over the ICMP part and to ping up any number of hosts before deciding the connection is actually down. Further it allows to e-mail or even to enable disabled interfaces periodically to see if the connection was restored.

Which is what I meant by "more complex schemes". I find there is a place for recursive routing for those wanting nearly instantaneous failover and have a remote place they control to ping over a line that rarely if ever fails. But if you don't have control over what you're pinging or partial line failures are a possibility I always prefer to script.
 
lrn23
just joined
Topic Author
Posts: 24
Joined: Mon Jan 07, 2019 10:24 am

Re: routing - 3x GW, failover

Fri May 17, 2019 4:11 pm

Thank you guys!

So that example here https://awarmanf.wordpress.com/2010/01/ ... cyrouting/ is wrong? There are route rules combined with mangle..

Failover decisions based on pinging gateway does not seems to me like a great idea. There is usualy working gateway and problem is "further down the line" like NetWorker said. That's why I was counting with recoursive routing and pinging something in internet, because I have no experience with scripting on MikroTik. And using IPs like 8.8.8.8... I know it's not ideal, but it usually works :) Are there any GOOD examples of failover scripts? I'll learn how to script on ROS and I'll rework this logic later.

NetWorker: Thanks for corrections. I understand that way better now. But I think I have to go with mangle, because I obviously need connection-marks like sindy suggested.
sindy: Thank you for great advices. Yes, I have to configure that device remotely. And yes, I need to access more devices in that area remotely, so I think connection-marking should be my choice. Thanks for that link. It's well explained there. So I have to configure it this way? (Yes, I have failover with recursive routing again, but I'll rework it as soon as I learn something about scripting on ROS :))
/ip route
add dst-address=8.8.8.8 gateway=10.0.1.1 scope=10
add dst-address=8.8.4.4 gateway=10.0.2.1 scope=10
add dst-address=1.1.1.1 gateway=10.0.3.1 scope=10

add distance=1 gateway=8.8.8.8 check-gateway=ping
add distance=2 gateway=8.8.4.4 check-gateway=ping
add distance=3 gateway=1.1.1.1 check-gateway=ping

add distance=1 gateway=8.8.8.8 routing-mark=IPS1 check-gateway=ping
add distance=2 gateway=8.8.4.4 routing-mark=IPS1 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS1 check-gateway=ping

add distance=1 gateway=8.8.4.4 routing-mark=IPS2 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS2 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS2 check-gateway=ping

add distance=1 gateway=1.1.1.1 routing-mark=IPS3 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS3 check-gateway=ping
add distance=3 gateway=8.8.4.4 routing-mark=IPS3 check-gateway=ping

/ip firewall mangle
add chain=prerouting connection-state=established,related action=accept
add chain=prerouting connection-state=established,related in-interface=WAN1
add chain=prerouting connection-state=established,related in-interface=WAN2
add chain=prerouting connection-state=established,related in-interface=WAN3
add chain=prerouting connection-mark=handling-1 action=mark-routing new-routing-mark=IPS1
add chain=prerouting connection-mark=handling-2 action=mark-routing new-routing-mark=IPS2
add chain=prerouting connection-mark=handling-3 action=mark-routing new-routing-mark=IPS3

add chain=prerouting src-address=192.168.1.0/24 connection-state=new action=mark-connection new-connection-mark=handling-1 passthrough=yes
add chain=prerouting src-address=192.168.2.0/24 connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-2 passthrough=yes
add chain=prerouting src-address=192.168.3.0/24 connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-3 passthrough=yes

add chain=prerouting connection-mark=handling-1 action=mark-routing new-routing-mark=IPS1
add chain=prerouting connection-mark=handling-2 action=mark-routing new-routing-mark=IPS2
add chain=prerouting connection-mark=handling-3 action=mark-routing new-routing-mark=IPS3

As soon as I would have this configuration working, I'll have to look at queues, because one of those subnets is guest network and I have to limit users a little bit. But I have to study it a little bit in a first place. I have no experience with queues yet. And thats the same like in the case of potencional load balancing (ISP2 - ISP3) which NetWorker mentioned. But I'll read a little bit about that in the first place... Thank you guys for your help. I really appreciate it!
 
NetWorker
Frequent Visitor
Frequent Visitor
Posts: 61
Joined: Sun Jan 31, 2010 6:55 pm

Re: routing - 3x GW, failover

Sat May 18, 2019 6:19 am

Glad to help!

Didn't read all your code since the Mrs. is nagging to go to bed lol. But I still find you're missing the default gateways? Unless you have something else in mind?

Anyway, I posted an interface watchdog script a couple of years ago that would reset an interface since we were having issues with one of our DSL lines crashing from time to time.
You can find it here: viewtopic.php?f=9&t=122664

But check out the wiki on scripting first. It makes pretty easy reading and if you've ever scripted in anything from a batch to some language you'll find scripting in routerOS a piece of cake!
To me it's arguably routerOS' strongest feature!

Oh and one more thing. If you do mangle, don't do policy routing. If you ever make a mistake, with 3 WAN, 30+ mangle rules, 10+ different connection marks and then some routing rules, it can take hours to track down. So if you do mangle, do mangle and be tidy about it. Use the comments to label what each rule does! That way, if someone ever rings you up with "hey, I can't connect to whatever" all you need to do is run down your mangle list and find the mistake instead of looking in two places at the same time even though it's almost the same thing (the routing rule part I mean).

edit: correct link
 
sindy
Forum Guru
Forum Guru
Posts: 3527
Joined: Mon Dec 04, 2017 9:19 pm

Re: routing - 3x GW, failover

Sat May 18, 2019 9:33 am

Terminologically, it is not policy routing versus mangle rules. Policy routing is anything that chooses a route depending on anything else in addition to the destination address, regardless the particular method you use to achieve this.

But I agree that a clear structure is a key to long-term maintainability of the configuration.

Specially for your ambitious goal, there is a caveat - you can have only a single connection-mark per each connection, so if you need to use them both for policy routing via routing-marks and for QoS via packet-marks, you will have to use combined connection-marks.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
lrn23
just joined
Topic Author
Posts: 24
Joined: Mon Jan 07, 2019 10:24 am

Re: routing - 3x GW, failover

Wed May 22, 2019 4:14 pm

NetWorker: No, I don't miss them. It's done with recursive routing and it works this way. But thank you very much for the link and tips! I really appreciate it!

sindy: I see :) Thanks! Combined connection-marks? Mmm nice! I'm already scared! :D Btw... That connection-mark related post is really great and informative! But isn't there a typo? In the first mangle rule you are commenting mid-connection packet with no connection-mark, but that no-mark condition is missing in that rule...
 
sindy
Forum Guru
Forum Guru
Posts: 3527
Joined: Mon Dec 04, 2017 9:19 pm

Re: routing - 3x GW, failover

Wed May 22, 2019 7:22 pm

But isn't there a typo? In the first mangle rule you are commenting mid-connection packet with no connection-mark, but that no-mark condition is missing in that rule...
Spot-on. Thank you, fixed.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
sindy
Forum Guru
Forum Guru
Posts: 3527
Joined: Mon Dec 04, 2017 9:19 pm

Re: routing - 3x GW, failover

Wed May 22, 2019 7:31 pm

Combined connection-marks? Mmm nice! I'm already scared!
Nothing to be really scared about, it's rather a matter of boring routine so it is easy to make a mistake in it, see here.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5369
Joined: Mon Jun 08, 2015 12:09 pm

Re: routing - 3x GW, failover

Wed May 22, 2019 9:02 pm

I had that problem too, solved it in a similar way, but at that time I asked MikroTik and it was sort of promised (as always) that version 7 would have multiple marks support :D
It should be "easy to do" based on the underlying kernel support, it is more or less of an oversight that this is not possible right now.
(the connection- and packet marks in the kernel are 32-bit values on which bit-masking and compare operations are supported, so single bits or groups of bits can be used for marks)

But of course it is tricky to change it lateron, because "mark packet" which now means "remove previous mark if any and assign new mark" cannot be changed to "add this mark to the packet" without risk of breaking existing setups....
So there either has to be a new "add mark to packet" operation or some other conversion of existing configuration.

Who is online

Users browsing this forum: No registered users and 57 guests