Hi guys,
I'm trying to setup my router and I don't know which way I should go. I have 3 different ISP connected, I want to have 1 ISP as a default one, one subnet should use another ISP and next subnet should use the last IPS. In case of outage on some ISP link, there should be working automatic failover. I have few questions now. How to achieve this routing logic? Is the use of "ip route rule" desired way how to setup things like this? I found one example: https://awarmanf.wordpress.com/2010/01/ ... cyrouting/. Is this good configuration? Or should I do it only with mangle and "ip route"? And what about failover? Is this good working solution: https://wiki.mikrotik.com/wiki/Advanced ... _Scripting? Thank you for every suggestion!
Re: routing - 3x GW, failover
It is the basic way of configuring it, yes. Use policy routing to route depending on your local subnet, use multiple default gw at different distance to achieve your failiver.
You need to decide what criteria you want to use for "not working ISP".
You can use ping or arp check of their end of the connection (check= on default route), or you can check some system further down on internet and use recursive routing.
You need to decide what criteria you want to use for "not working ISP".
You can use ping or arp check of their end of the connection (check= on default route), or you can check some system further down on internet and use recursive routing.
Re: routing - 3x GW, failover
OK, thank you! 
So, this could be a working configuration? I have to be sure, because I have to configure this in remote location so I can't test it properly 
WAN interface ISP1 - 10.0.1.1
WAN interface ISP2 - 10.0.2.1
WAN interface ISP3 - 10.0.3.1
LAN subnets:
192.168.1.0/24
192.168.2.0/24
192.168.3.0/24
So, this could be a working configuration? I have to be sure, because I have to configure this in remote location so I can't test it properly WAN interface ISP1 - 10.0.1.1
WAN interface ISP2 - 10.0.2.1
WAN interface ISP3 - 10.0.3.1
LAN subnets:
192.168.1.0/24
192.168.2.0/24
192.168.3.0/24
Code: Select all
/ip route
add dst-address=8.8.8.8 gateway=10.0.1.1 scope=10
add dst-address=8.8.4.4 gateway=10.0.2.1 scope=10
add dst-address=1.1.1.1 gateway=10.0.3.1 scope=10
add distance=1 gateway=8.8.8.8 routing-mark=IPS1 check-gateway=ping
add distance=2 gateway=8.8.4.4 routing-mark=IPS1 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS1 check-gateway=ping
add distance=1 gateway=8.8.4.4 routing-mark=IPS2 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS2 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS2 check-gateway=ping
add distance=1 gateway=1.1.1.1 routing-mark=IPS3 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS3 check-gateway=ping
add distance=3 gateway=8.8.4.4 routing-mark=IPS3 check-gateway=ping
/ip route rule
add dst-address=192.168.1.0/24 action=lookup table=main
add dst-address=192.168.2.0/24 action=lookup table=main
add dst-address=192.168.3.0/24 action=lookup table=main
add dst-address=10.0.1.0/24 action=lookup table=main
add dst-address=10.0.2.0/24 action=lookup table=main
add dst-address=10.0.3.0/24 action=lookup table=main
add src-address=10.0.1.0/24 action=lookup table=IPS1
add src-address=10.0.2.0/24 action=lookup table=IPS2
add src-address=10.0.3.0/24 action=lookup table=IPS3
add routing-mark=IPS1 action=lookup table=IPS1
add routing-mark=IPS2 action=lookup table=IPS2
add routing-mark=IPS3 action=lookup table=IPS3
/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=IPS1 passthrough=no src-address=192.168.1.0/24
add action=mark-routing chain=prerouting new-routing-mark=IPS2 passthrough=no src-address=192.168.2.0/24
add action=mark-routing chain=prerouting new-routing-mark=IPS3 passthrough=no src-address=192.168.3.0/24
Re: routing - 3x GW, failover
Not quite. You either do it with mangle or with routing rules as pe1chl suggested. Also, you're missing the default gateways.
Logic is as follows:
You add three default gateways. All traffic that doesn't have a routing mark will be routed to ISP1. If that connection fails, it will use ISP2 and if both ISP1 and ISP2 are offline, it will use ISP3.
Next you add the three routes that each subnet will follow. If all is running, each connection will follow it's own route. If a ping check fails, both routes will deactivate; the one with the routing mark and the default one. Now, all traffic with that routing mark will follow the default gateway logic.
Finally you add the three rules for each subnet.
That should be all you need though I can't test it for you.
Code: Select all
/ip route
#first add the default gateways with distance for failover
add dst-address=0.0.0.0/0 gateway=10.0.1.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.0.2.1 distance=2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.0.3.1 distance=3 check-gateway=ping
#next add the routes for each subnet.
add dst-address=0.0.0.0/0 gateway=10.0.1.1 check-gateway=ping routing-mark=ISP1
add dst-address=0.0.0.0/0 gateway=10.0.2.1 check-gateway=ping routing-mark=ISP2
add dst-address=0.0.0.0/0 gateway=10.0.3.1 check-gateway=ping routing-mark=ISP3
#finally add the rules for each subnet
/ip route rule
add src-address=192.168.1.0/24 action=lookup table=ISP1
add src-address=192.168.2.0/24 action=lookup table=ISP2
add src-address=192.168.3.0/24 action=lookup table=ISP3
You add three default gateways. All traffic that doesn't have a routing mark will be routed to ISP1. If that connection fails, it will use ISP2 and if both ISP1 and ISP2 are offline, it will use ISP3.
Next you add the three routes that each subnet will follow. If all is running, each connection will follow it's own route. If a ping check fails, both routes will deactivate; the one with the routing mark and the default one. Now, all traffic with that routing mark will follow the default gateway logic.
Finally you add the three rules for each subnet.
That should be all you need though I can't test it for you.
Re: routing - 3x GW, failover
Forgot to point out two things. First was already covered by pe1chl. If you only ping the gateway and there's a problem further down the line in the ISP, those routes will stay up but you'll drop all the traffic. I.e. the failover won't happen. More complex checking schemes will require scripts.
Second, bear in mind that failover will dump all the traffic from two lansubnets on a single connection. I.e. suppose ISP2 fails, now all the traffic from Subnets 1 and 2 will be on ISP1. This could lead to dropped traffic on the ISP1 line. Now if you use the highest and most reliable line as ISP1 and put all your heavy consumers in lansubnet1, if that line ever fails, your ISP2 line will be taking an unacceptable load and drop so much traffic that the failover is hardly doing you any favors. Therefore, assuming similarly reliable lines, it's best to also use load balancing schemes in multiwan scenarios. This however adds complexity and is not as simple as discussed above.
Second, bear in mind that failover will dump all the traffic from two lansubnets on a single connection. I.e. suppose ISP2 fails, now all the traffic from Subnets 1 and 2 will be on ISP1. This could lead to dropped traffic on the ISP1 line. Now if you use the highest and most reliable line as ISP1 and put all your heavy consumers in lansubnet1, if that line ever fails, your ISP2 line will be taking an unacceptable load and drop so much traffic that the failover is hardly doing you any favors. Therefore, assuming similarly reliable lines, it's best to also use load balancing schemes in multiwan scenarios. This however adds complexity and is not as simple as discussed above.
Re: routing - 3x GW, failover
That is not really true. You can set a specific route for a single IP via some ISP and with a ping check, then add a default route via that specific IP.Forgot to point out two things. First was already covered by pe1chl. If you only ping the gateway and there's a problem further down the line in the ISP, those routes will stay up but you'll drop all the traffic. I.e. the failover won't happen. More complex checking schemes will require scripts.
This is called "recursive routing". It can deliver failover with checking further down the line without requiring scripting.
However, note that:
- you should also add routes to the same destination with higher distance via the other ISPs to keep that particular IP reachable when some ISP is down.
(and it is not really advisable to use widely-used IPs like 8.8.8.8 for the checking, to avoid user problems in that case)
- by using a ping check of a remote system, you make the usability of the link dependent on the availability of the remote system. when it is down, your ISP link will be declared down even when it really isn't. remember that the remote system may decide that it had enough of the pings and just block them, which you won't notice when you do not monitor the router closely.
Re: routing - 3x GW, failover
This indicates to me that you access the device remotely, and therefore you can lock yourself out if you make a mistake.I have to be sure, because I have to configure this in remote location so I can't test it properly
So first, whatever change you do should be done using safe mode (Ctrl-X in command line toggles safe mode for changes done in the same command line window; a button in WebFig and/or Winbox does the same for changes done using that GUI, but you can use safe mode only on one interface at a time).
Second, the way you use the failover between your 3 WANs so far is not sufficient to access it remotely via WAN. There are two possibilities how to deal with this:
- the simpler to configure one is to configure a static route towards the IP from which you'll be configuring into the default routing table (i.e. without a routing-mark) and also one per each routing-mark you'll be using, all via the same WAN through which you'll be accessing the device remotely,
- a more complex but more useful one is not to assign the routing-mark directly but to use connection-mark as a base for it, which allows you to let any connection from outside be answered via the same WAN interface through which it came in - see details here (start reading that post from the last paragraph which gives you the context otherwise scattered across the topic)
Re: routing - 3x GW, failover
This is the main reason I dislike recursive routing. You don't have control over how many pings are sent, how many are dropped and you need to add a route for each remote address you want to ping. A script allows full control over the ICMP part and to ping up any number of hosts before deciding the connection is actually down. Further it allows to e-mail or even to enable disabled interfaces periodically to see if the connection was restored.- by using a ping check of a remote system, you make the usability of the link dependent on the availability of the remote system. when it is down, your ISP link will be declared down even when it really isn't. remember that the remote system may decide that it had enough of the pings and just block them, which you won't notice when you do not monitor the router closely.
Which is what I meant by "more complex schemes". I find there is a place for recursive routing for those wanting nearly instantaneous failover and have a remote place they control to ping over a line that rarely if ever fails. But if you don't have control over what you're pinging or partial line failures are a possibility I always prefer to script.
Re: routing - 3x GW, failover
Thank you guys!
So that example here https://awarmanf.wordpress.com/2010/01/ ... cyrouting/ is wrong? There are route rules combined with mangle..
Failover decisions based on pinging gateway does not seems to me like a great idea. There is usualy working gateway and problem is "further down the line" like NetWorker said. That's why I was counting with recoursive routing and pinging something in internet, because I have no experience with scripting on MikroTik. And using IPs like 8.8.8.8... I know it's not ideal, but it usually works Are there any GOOD examples of failover scripts? I'll learn how to script on ROS and I'll rework this logic later.
NetWorker: Thanks for corrections. I understand that way better now. But I think I have to go with mangle, because I obviously need connection-marks like sindy suggested.
sindy: Thank you for great advices. Yes, I have to configure that device remotely. And yes, I need to access more devices in that area remotely, so I think connection-marking should be my choice. Thanks for that link. It's well explained there. So I have to configure it this way? (Yes, I have failover with recursive routing again, but I'll rework it as soon as I learn something about scripting on ROS)
As soon as I would have this configuration working, I'll have to look at queues, because one of those subnets is guest network and I have to limit users a little bit. But I have to study it a little bit in a first place. I have no experience with queues yet. And thats the same like in the case of potencional load balancing (ISP2 - ISP3) which NetWorker mentioned. But I'll read a little bit about that in the first place... Thank you guys for your help. I really appreciate it!
So that example here https://awarmanf.wordpress.com/2010/01/ ... cyrouting/ is wrong? There are route rules combined with mangle..
Failover decisions based on pinging gateway does not seems to me like a great idea. There is usualy working gateway and problem is "further down the line" like NetWorker said. That's why I was counting with recoursive routing and pinging something in internet, because I have no experience with scripting on MikroTik. And using IPs like 8.8.8.8... I know it's not ideal, but it usually works
Are there any GOOD examples of failover scripts? I'll learn how to script on ROS and I'll rework this logic later.NetWorker: Thanks for corrections. I understand that way better now. But I think I have to go with mangle, because I obviously need connection-marks like sindy suggested.
sindy: Thank you for great advices. Yes, I have to configure that device remotely. And yes, I need to access more devices in that area remotely, so I think connection-marking should be my choice. Thanks for that link. It's well explained there. So I have to configure it this way? (Yes, I have failover with recursive routing again, but I'll rework it as soon as I learn something about scripting on ROS
)Code: Select all
/ip route
add dst-address=8.8.8.8 gateway=10.0.1.1 scope=10
add dst-address=8.8.4.4 gateway=10.0.2.1 scope=10
add dst-address=1.1.1.1 gateway=10.0.3.1 scope=10
add distance=1 gateway=8.8.8.8 check-gateway=ping
add distance=2 gateway=8.8.4.4 check-gateway=ping
add distance=3 gateway=1.1.1.1 check-gateway=ping
add distance=1 gateway=8.8.8.8 routing-mark=IPS1 check-gateway=ping
add distance=2 gateway=8.8.4.4 routing-mark=IPS1 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS1 check-gateway=ping
add distance=1 gateway=8.8.4.4 routing-mark=IPS2 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS2 check-gateway=ping
add distance=3 gateway=1.1.1.1 routing-mark=IPS2 check-gateway=ping
add distance=1 gateway=1.1.1.1 routing-mark=IPS3 check-gateway=ping
add distance=2 gateway=8.8.8.8 routing-mark=IPS3 check-gateway=ping
add distance=3 gateway=8.8.4.4 routing-mark=IPS3 check-gateway=ping
/ip firewall mangle
add chain=prerouting connection-state=established,related action=accept
add chain=prerouting connection-state=established,related in-interface=WAN1
add chain=prerouting connection-state=established,related in-interface=WAN2
add chain=prerouting connection-state=established,related in-interface=WAN3
add chain=prerouting connection-mark=handling-1 action=mark-routing new-routing-mark=IPS1
add chain=prerouting connection-mark=handling-2 action=mark-routing new-routing-mark=IPS2
add chain=prerouting connection-mark=handling-3 action=mark-routing new-routing-mark=IPS3
add chain=prerouting src-address=192.168.1.0/24 connection-state=new action=mark-connection new-connection-mark=handling-1 passthrough=yes
add chain=prerouting src-address=192.168.2.0/24 connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-2 passthrough=yes
add chain=prerouting src-address=192.168.3.0/24 connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-3 passthrough=yes
add chain=prerouting connection-mark=handling-1 action=mark-routing new-routing-mark=IPS1
add chain=prerouting connection-mark=handling-2 action=mark-routing new-routing-mark=IPS2
add chain=prerouting connection-mark=handling-3 action=mark-routing new-routing-mark=IPS3
As soon as I would have this configuration working, I'll have to look at queues, because one of those subnets is guest network and I have to limit users a little bit. But I have to study it a little bit in a first place. I have no experience with queues yet. And thats the same like in the case of potencional load balancing (ISP2 - ISP3) which NetWorker mentioned. But I'll read a little bit about that in the first place... Thank you guys for your help. I really appreciate it!
Re: routing - 3x GW, failover
Glad to help!
Didn't read all your code since the Mrs. is nagging to go to bed lol. But I still find you're missing the default gateways? Unless you have something else in mind?
Anyway, I posted an interface watchdog script a couple of years ago that would reset an interface since we were having issues with one of our DSL lines crashing from time to time.
You can find it here: viewtopic.php?f=9&t=122664
But check out the wiki on scripting first. It makes pretty easy reading and if you've ever scripted in anything from a batch to some language you'll find scripting in routerOS a piece of cake!
To me it's arguably routerOS' strongest feature!
Oh and one more thing. If you do mangle, don't do policy routing. If you ever make a mistake, with 3 WAN, 30+ mangle rules, 10+ different connection marks and then some routing rules, it can take hours to track down. So if you do mangle, do mangle and be tidy about it. Use the comments to label what each rule does! That way, if someone ever rings you up with "hey, I can't connect to whatever" all you need to do is run down your mangle list and find the mistake instead of looking in two places at the same time even though it's almost the same thing (the routing rule part I mean).
edit: correct link
Didn't read all your code since the Mrs. is nagging to go to bed lol. But I still find you're missing the default gateways? Unless you have something else in mind?
Anyway, I posted an interface watchdog script a couple of years ago that would reset an interface since we were having issues with one of our DSL lines crashing from time to time.
You can find it here: viewtopic.php?f=9&t=122664
But check out the wiki on scripting first. It makes pretty easy reading and if you've ever scripted in anything from a batch to some language you'll find scripting in routerOS a piece of cake!
To me it's arguably routerOS' strongest feature!
Oh and one more thing. If you do mangle, don't do policy routing. If you ever make a mistake, with 3 WAN, 30+ mangle rules, 10+ different connection marks and then some routing rules, it can take hours to track down. So if you do mangle, do mangle and be tidy about it. Use the comments to label what each rule does! That way, if someone ever rings you up with "hey, I can't connect to whatever" all you need to do is run down your mangle list and find the mistake instead of looking in two places at the same time even though it's almost the same thing (the routing rule part I mean).
edit: correct link
Re: routing - 3x GW, failover
Terminologically, it is not policy routing versus mangle rules. Policy routing is anything that chooses a route depending on anything else in addition to the destination address, regardless the particular method you use to achieve this.
But I agree that a clear structure is a key to long-term maintainability of the configuration.
Specially for your ambitious goal, there is a caveat - you can have only a single connection-mark per each connection, so if you need to use them both for policy routing via routing-marks and for QoS via packet-marks, you will have to use combined connection-marks.
But I agree that a clear structure is a key to long-term maintainability of the configuration.
Specially for your ambitious goal, there is a caveat - you can have only a single connection-mark per each connection, so if you need to use them both for policy routing via routing-marks and for QoS via packet-marks, you will have to use combined connection-marks.
Re: routing - 3x GW, failover
NetWorker: No, I don't miss them. It's done with recursive routing and it works this way. But thank you very much for the link and tips! I really appreciate it!
sindy: I see Thanks! Combined connection-marks? Mmm nice! I'm already scared! 
Btw... That connection-mark related post is really great and informative! But isn't there a typo? In the first mangle rule you are commenting mid-connection packet with no connection-mark, but that no-mark condition is missing in that rule...
sindy: I see
Thanks! Combined connection-marks? Mmm nice! I'm already scared! Btw... That connection-mark related post is really great and informative! But isn't there a typo? In the first mangle rule you are commenting mid-connection packet with no connection-mark, but that no-mark condition is missing in that rule...Re: routing - 3x GW, failover
Spot-on. Thank you, fixed.But isn't there a typo? In the first mangle rule you are commenting mid-connection packet with no connection-mark, but that no-mark condition is missing in that rule...
Re: routing - 3x GW, failover
Nothing to be really scared about, it's rather a matter of boring routine so it is easy to make a mistake in it, see here.Combined connection-marks? Mmm nice! I'm already scared!
Re: routing - 3x GW, failover
I had that problem too, solved it in a similar way, but at that time I asked MikroTik and it was sort of promised (as always) that version 7 would have multiple marks support
It should be "easy to do" based on the underlying kernel support, it is more or less of an oversight that this is not possible right now.
(the connection- and packet marks in the kernel are 32-bit values on which bit-masking and compare operations are supported, so single bits or groups of bits can be used for marks)
But of course it is tricky to change it lateron, because "mark packet" which now means "remove previous mark if any and assign new mark" cannot be changed to "add this mark to the packet" without risk of breaking existing setups....
So there either has to be a new "add mark to packet" operation or some other conversion of existing configuration.
It should be "easy to do" based on the underlying kernel support, it is more or less of an oversight that this is not possible right now.
(the connection- and packet marks in the kernel are 32-bit values on which bit-masking and compare operations are supported, so single bits or groups of bits can be used for marks)
But of course it is tricky to change it lateron, because "mark packet" which now means "remove previous mark if any and assign new mark" cannot be changed to "add this mark to the packet" without risk of breaking existing setups....
So there either has to be a new "add mark to packet" operation or some other conversion of existing configuration.
Who is online
Users browsing this forum: akakua, shahzaddj1 and 175 guests