Community discussions

 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Mikrotik Dual WAN Failover

Mon Sep 03, 2018 7:43 pm

Hello guys,

I've recently bough a mikrotik router and the model I chose as let's say my teaching router thanks to the help of some colleagues from the forum is hAP ac2. If someone is interested of something about the need of the router - here is the thread I made viewtopic.php?t=138342.

However, one of the critical things I needed was the dual WAN support. I need it only as a fail-over without load balance. This weekend I had few hours to play with the new router and to try to make a simple setup.

Here is the current situation about the ISPs:

1. Main link - 100Mbps PPPoE directly to the hAP
2. Back up link - 50 Mbps ADSL - phone line->Modem->hAP
3. Static local IPs from both providers.

Here is what I've done so far:

1. I stepped on the default configuration of the hAP and from there I've tried to build up what I need. Firstly I've changed rotuer's IP 192.168.0.1 and created a new DHCP server wit primary DNS 192.168.0.1 and secondary 8.8.8.8, added pool in the needed range.

2. Excluded the ether2 port from the bridge and added it to the list of WANs so other rules can apply to it.

3. I've created a ppoe client for port1 with the needed user and password while using peer DNS and default route / ppoe-out was also added to the WAN list.

4. Static IP for the port2 - 192.168.x.x, added route with gateway (the ADSL modem)

5. Static DNS - same as the adsl modem

6. Distance of the main link is 1 and distance of the backup - 2

So in this configuration everything seems to work for now even if I don't know if there is something missed and the test was done only by disconnecting the WAN ports. However this configurations is apart from the PCC tutorial and I'm not sure if the PCC could be applied to this case when there is one static and one pppoe? Could I use as gateway the whole pppoe-out in the PCC wiki scenario? I've read several posts and wikis about the dual WAN scenario but it seems that most of them are using load balance and they are mostly for static addresses. There are also the mangle rules which I haven't had the time to study more carefully. I saw that the preferred way of dual WAN fail-over is the PCC but what about the mangle rules, they seems to use it too? What would be the best way to configure dual WAN fail-over in my case and is my configuration by far worth a dime?

I'm using the default settings for the firewall applied to both WAN ports via the WAN list, same for NAT.

Thanks for the help in advance guys. It's a great device with plenty of settings and I like it. Still it would be great if they've added something as the quick config for dual WAN options as it's a common thing now days. This router would be used for experiments for now until I start to feel a bit more comfortable with the OS and I get my knowledge together.
Last edited by draid on Mon Sep 03, 2018 10:22 pm, edited 4 times in total.
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 1168
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa
Contact:

Re: Mikrotik Dual WAN Failover

Mon Sep 03, 2018 8:02 pm

PCC is for load balancing, from your description, you do not need that.

Then I would also change the ADSL Modem to bridge mode and configure ADSL PPPoE on the Mikrotik.

The do not use the "Add default Gateway"in the PPPoE settings, instead create static default routes with a distance of 1 and 2, 2 for the adsl and use "check gateway " on the static routes
MTCNA, MTCTCE, MTCRE & MTCINE
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Mon Sep 03, 2018 9:04 pm

...the dual WAN support. I need it only as a fail-over without load balance.
As @CZFan has said already: if so, don't bother about PCC, as PCC is here first for load distribution, and only as a side effect it provides some kind of failover. Leaving out PCC will relieve you from having to understand the mangle rules for the moment.

... the test was done only by disconnecting the WAN ports.
Which is also the weakest point of that configuration. Even though the handover interface of WAN1 is Ethernet, there may still be something between your Ethernet port and the actual Internet which may fail without your Ethernet interface going down, and in such case the route via that interface will stay up so no failover will happen. So here I recommend this article explaining how to monitor that accesss to internet is really possible via each WAN. The Mikrotik wiki describes the same coniguration but in a much less explanatory way.

Still it would be great if they've added something as the quick config for dual WAN options as it's a common thing now days. This router would be used for experiments for now until I start to feel a bit more comfortable with the OS and I get my knowledge together.
I agree it would be great - but a great waste of developers' efforts that could be better spent on features which cannot be achieved by configuration. Every user has a different environment, so WAN1 may be anything out of (PPPoE, static IP configuration, DHCP) on anything out of (ethernet, wireless), leaving aside LTE with its two modes (serial or Ethernet emulation) and so can be the WAN2, and every user has different requirements, e.g. a mere failover in your case and load distribution in someone else's case. Others may want one of those basic approaches for most of the traffic but some services to be accessed solely via one of the WANs. So I personally like the current approach where QuickSet is for people who have bought Mikrotik by chance and the real configuration interface is for those who have chosen it for its flexibility. Flexibility means a lot of things can be configured, and without an understanding what each setting is necessary for it is close to impossible to answer properly all what a configuration wizard would have to ask.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Mon Sep 03, 2018 10:41 pm

PCC is for load balancing, from your description, you do not need that.

Then I would also change the ADSL Modem to bridge mode and configure ADSL PPPoE on the Mikrotik.

The do not use the "Add default Gateway"in the PPPoE settings, instead create static default routes with a distance of 1 and 2, 2 for the adsl and use "check gateway " on the static routes
I though about something like this but if I set the ADSL modem to bridge I'd need the password so I'll be able to create the pppoe from the mikrotik. Sadly I don't have this information and the ISP won't give it to me if requested. Normally I won't even have access to their device and would be forced to manage it by their limited web but however I have access to the modem. I'd probably be able even to recover the password for the pppoe but it may be too much. Other problem that will result directly from that is the DVR which needs some ports forwarding so I'd have to configure it too.

According to the Add default gateway - I'm a bit confused as the local address is static but in the default settings it takes the remote address. I'll have to try this otherwise the pppoe route is still set to 1 and the adsl to 0 - that's for the 0.0.0.0/0 dest.
As @CZFan has said already: if so, don't bother about PCC, as PCC is here first for load distribution, and only as a side effect it provides some kind of failover. Leaving out PCC will relieve you from having to understand the mangle rules for the moment.

Which is also the weakest point of that configuration. Even though the handover interface of WAN1 is Ethernet, there may still be something between your Ethernet port and the actual Internet which may fail without your Ethernet interface going down, and in such case the route via that interface will stay up so no failover will happen. So here I recommend this article explaining how to monitor that accesss to internet is really possible via each WAN. The Mikrotik wiki describes the same coniguration but in a much less explanatory way.
Thanks for the link, I'm going to check it for sure and I'll try to make this work. According to the quick settings - it is true that it's a bit of a hard work to implement it. Still even if it's a bit tricky to set things up and you'll need a lot of reading and network knowledge I really like this product. You can learn a lot from it.

At the bottom line it turns out that I can use the distance difference and monitoring to realize the setup. It's great as I was just preparing to read about the PCC and mangle rules for the next time I have free time.
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 1168
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa
Contact:

Re: Mikrotik Dual WAN Failover

Tue Sep 04, 2018 12:27 am

Just a note when keeping ADSL modem in router mode, you must not use nat / masquerade between else you will have a double NAT situation than can cause issues
MTCNA, MTCTCE, MTCRE & MTCINE
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Tue Sep 04, 2018 11:19 am

@CZFan, sorry... although it is true that multiple NAT does cause issues in rare cases (in 99,9% situations it is just as bad as a single NAT), you cannot just disable src-nat (masquerade) between Mikrotik and the ADSL modem, but you also have to add route(s) to Mikrotik's LAN subnet(s) to the ADSL modem. Otherwise the modem would send packets for these subnets back up the WAN (which is its default gateway).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 2:56 pm

Hello guys,

I haven't had much time recently to play with the fail-over but today I had some time and I decided to test the fail-over scenario from the article sindy posted here. I think that I'm facing a problem and I'm not exactly sure where it comes from.

First of all I want to say that I'm continuing with the article after I have set my settings for both ISPs. Here are the things I've done before continuing with the first method from the article.

1. Port 1&2 are set as WAN ports.
2. Port 3-5 are in a LAN bridge.
3. DHCP server for the bridge is set with the needed pool and the router used as DNS.
4. PPPoE-out for the main link is set with the needed credentials (Use peer DNS = true Add default route = false), Ethernet port 1.
5. Created route Dest. Add. 0.0.0.0/0, GW PPPoE-out.

At this point I'm having internet through the ppoe and everything works fine.

6. Add new address to the address list for the second ISP 192.169.1.2/24, network 192.168.1.0, Ethernet port 2.
7. Created route Dest. Add. 0.0.0.0/0, GW 192.168.1.1 via Ethernet port 2.
8. DNS server set to 192.168.1.1 (the ADSL modem), Allow remote request = true.

WAN1 and WAN2 alongside with pppoe are added to the WAN list so the NAT and firewall rules can apply to all of them.
At this point I want to continue with the monitoring of the gateways explained in the article (its method one)

/ip route
add dst-address=8.8.8.8 gateway=PPPoE-out scope=10
add dst-address=8.8.4.4 gateway=192.168.1.1 scope=10

At this point both addresses are reachable

/ip route
add distance=1 gateway=8.8.8.8 check-gateway=ping
add distance=2 gateway=8.8.4.4 check-gateway=ping

And here comes the problem, once I set these two routes they are both unreachable. I know that I'm missing something and probably I don't need the dest in point 5 and 7 but I'm not able to figure it out. Probably there is some kind of a conflict but I've tried everything I could imagine for now and it seems not to work. Could you please give me an advice about where I'm in fact messing the things up.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 3:47 pm

First of all, the recursive routing on which the scriptless failover is based does not work if a route's gateway is set to anything else than an IP number anywhere in the recursive chain. So you cannot use the interface name (PPPoE-out) as a gateway for dst-address=8.8.8.8, you have to use the IP address provided by the PPPoE server.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 6:10 pm

First of all, the recursive routing on which the scriptless failover is based does not work if a route's gateway is set to anything else than an IP number anywhere in the recursive chain. So you cannot use the interface name (PPPoE-out) as a gateway for dst-address=8.8.8.8, you have to use the IP address provided by the PPPoE server.
Fair enough but it isn't working even with the static address of the second ISP. I mean the 0.0.0.0/0 with GW 8.8.8.8 is still unreachable.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 7:14 pm

When I say "you must use as gateway the IP address provided by the PPPoE server", I have in mind the address which that PPPoE server provides as a gateway, not the one it assigns to you. Is it what you mean by "static address of the second ISP"?

Normally, where you are a PPPoE client, the server assigns you your own address and indicates its own IP address which you may use as a gateway for anything you want to send via that server. But in most cases, you can use the interface name as well; recursive routing on Mikrotik is one of the exceptions where you can't. I have seen you have set add-default-route in /interface pppoe-client to no, but when you do that, you won't learn the gateway address. So you have to set it to yes for a while to learn the address "manually", or keep it on yes and set default-route-distance to e.g. 10 and add a blackhole route with a lower distance, so the final set of default routes would be
dst-address=0.0.0.0/0 gateway=8.8.8.8 distance=1
dst-address=0.0.0.0/0 gateway=8.8.4.4 distance=2
dst-address=0.0.0.0/0 type=blackhole distance=9
dst-address=0.0.0.0/0 gateway=the.ip.from.isp distance=10
Then, you would use an on-up script from a /ppp profile attached to the /interface pppoe-client to copy the gateway IP from the route with distance=10 to the route with dst-address=8.8.8.8/32. But it only makes sense to do it this complex way if the PPPoE server doesn't provide the same gateway IP address each time.

Where you are a DHCP client, you must use the IP address provided by the DHCP server as a default gateway (or use the routing table provided by the DHCP server as Option 121 but that's out of scope of this).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
Sob
Forum Guru
Forum Guru
Posts: 3867
Joined: Mon Apr 20, 2009 9:11 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 8:48 pm

There's also the trick with locally set remote address. Simply put one in PPP profile used by PPPoE client and then use it as gateway. I found it some time ago in this forum and although it looks completely wrong at first (how can I set remote address when I don't control remote side, right?) it works. The used address is not actually used by anything by default, no packets are sent to it, so it don't matter what you put there. Importatnt part is that it's static. And I think it was possible to go even one step further and use 8.8.8.8 as this remote address/gateway and check-gateway=ping with it. I don't remember the details, it probably had to be done with routing filter to add check-gateway option.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 8:58 pm

The used address is not actually used by anything by default, no packets are sent to it, so it don't matter what you put there.
So what you are saying is that
  • you don't need to retrieve the real "remote" address from the PPPoE client, so add-default-gateway may stay at no
  • you can assign different "remote" addresses to different PPPoE clients, which makes it possible to use the recursive next-hop search and thus scriptless failover even on several PPPoE connections even if the servers assign the same remote addresses to them
?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
Sob
Forum Guru
Forum Guru
Posts: 3867
Joined: Mon Apr 20, 2009 9:11 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 16, 2018 9:46 pm

Yes. In other words, if ISP would be giving you random 10.x.y.z every time you connect, you can set static 10.1.1.1 in PPP profile and use that. And it will work, because it's PPP, a tunnel where you just feed everything into. On ethernet, gateway IP address is used by ARP, but with PPP it's just a local hint where it is.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Mon Sep 17, 2018 8:04 am

When I say "you must use as gateway the IP address provided by the PPPoE server", I have in mind the address which that PPPoE server provides as a gateway, not the one it assigns to you. Is it what you mean by "static address of the second ISP"?

Normally, where you are a PPPoE client, the server assigns you your own address and indicates its own IP address which you may use as a gateway for anything you want to send via that server. But in most cases, you can use the interface name as well; recursive routing on Mikrotik is one of the exceptions where you can't. I have seen you have set add-default-route in /interface pppoe-client to no, but when you do that, you won't learn the gateway address. So you have to set it to yes for a while to learn the address "manually", or keep it on yes and set default-route-distance to e.g. 10 and add a blackhole route with a lower distance, so the final set of default routes would be
dst-address=0.0.0.0/0 gateway=8.8.8.8 distance=1
dst-address=0.0.0.0/0 gateway=8.8.4.4 distance=2
dst-address=0.0.0.0/0 type=blackhole distance=9
dst-address=0.0.0.0/0 gateway=the.ip.from.isp distance=10
Then, you would use an on-up script from a /ppp profile attached to the /interface pppoe-client to copy the gateway IP from the route with distance=10 to the route with dst-address=8.8.8.8/32. But it only makes sense to do it this complex way if the PPPoE server doesn't provide the same gateway IP address each time.

Where you are a DHCP client, you must use the IP address provided by the DHCP server as a default gateway (or use the routing table provided by the DHCP server as Option 121 but that's out of scope of this).
Ah I was afraid it won't be so straight forword with the PPPoE...

I meant that when I'm using the second ISP settings, everything is static i.e. the adress is 192.168.1.2 and the GW is the ADS modem at 192.168.1.1. Thus with these settings while I set the route with dest 8.8.8.8 throught GW 192.168.1.1 it is reachable and when afterwords I set the dest 0.0.0.0/0 with GW 8.8.8.8 its unreachable.

Sadly I'm afraid that the remote ip is not the same and it may variate (I'll double check it) which seems to make the things even more complicated as it's obvoius that if the remote adress is changing the set up won't work if this adress is not monitored. However I have static IP adresses from both ISPs.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Thu Sep 20, 2018 5:20 pm

There is no point in monitoring the remote IP, it may even not be up at all on the remote end. For the purpose of identifying a local PPPoE tunnel to use by a gateway IP address, you may assign the local alias to the tunnel's remote-address as per @Sob's suggestion. For the purpose of monitoring the WAN link transparency, the monitored addresses should be some immortal addresses further in the internet, so instead of checking just the hop between your router and ISP's PPPoE server, you check the whole path through the ISP up to the internet.

Regarding the gateway IP provided by DHCP on the second WAN, there is again no point in monitoring that address itself but you need to use it as a gateway to the monitored destination in the recursive next-hop search scheme, and you cannot easily assign an alias to it (well, you can in some cases, but in exactly those cases it is pointless to do that). So if the DHCP server runs on the modem+router combo you've got from the ISP, there is a 0.001% chance that the gateway address will ever change; if that box acts as a bridge and the DHCP server is physically located at the other end of the WAN link, the chance that the gateway IP will change is much higher. So in the latter case, you would have to permit the dhcp client to install a default gateway, but you would tell it to set a high distance = low priority value to it and copy the address of that gateway to the individual route to the monitored destination using the script parameter. So each time a new DHCP assignment arrives, you'd check whether the gateway IP has changed as compared to the previous one and if yes, you'd modify the individual route(s) to the monitored IPs.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Fri Sep 21, 2018 11:03 pm

Hello guys,

Thank you all for the precious help. Tonight I had some time to try the things up and everything seemed to work good with one exception. The remote address of the PPPoE is changing. It seems to be either 5 or 12 but it changes.

So What I've done till now:
/ip route
add check-gateway=ping distance=1 gateway=8.8.8.8
add check-gateway=ping distance=2 gateway=8.8.4.4

add distance=1 dst-address=8.8.4.4/32 gateway=192.168.x.x scope=10
add distance=1 dst-address=8.8.8.8/32 gateway=109.x.x.x scope=10
So when the Line 1 fail and then reconnect it may take a different Remote address so the recurse fails as the gateway in line 3 is different. You have mentioned some kind of a script but isn't there an easier way to always take the current remote address and to put it as the GW without scripting? It's really sad that this isn't working with the pppoe interface. I was able to get the address of the GW from the status bar of the pppoe interface. I wasn't completely able to understand the method proposed by @Sob's.

Also I found in an article that the recursive method has the following limitation:
Whatever IP you use as your target is only reachable via the primary route. If the primary route is down, that IP address will be unreachable. If you use 8.8.8.8 to resolve DNS, the DNS service will be down when the primary route is down. Therefore if you use Google for DNS and use 8.8.8.8 as the routing target, you should use a different Google DNS server such as 8.8.4.4 for DNS instead.
The pppoe is using 1.1.1.1 and 8.8.8.8 as DNS and for the ADSL I have to check it because currently I'm using the ADSL address as DNS.

So except the problem with the changing remote address everything seems to work. If I'm able to reslove this too I'd be able to make it with multiple host checks and leave it this way.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Fri Sep 21, 2018 11:55 pm

The remote address of the PPPoE is changing. It seems to be either 5 or 12 but it changes.
You have mentioned some kind of a script but isn't there an easier way to always take the current remote address and to put it as the GW without scripting? It's really sad that this isn't working with the pppoe interface. I was able to get the address of the GW from the status bar of the pppoe interface. I wasn't completely able to understand the method proposed by @Sob's.
You're mixing together the DHCP case with the PPPoE case.
For DHCP (used at your WAN2), there is no other way than a script to get the assigned IP address of default gateway and set it as a gateway in the individual routes to the monitored anchor addresses, but you obviously don't need it because the gateway IP provided by DHCP on WAN2 does not change.
For PPPoE (used at your WAN1), there is a script-less way which @Sob has described: you create a copy of /ppp profile named default, give it a name like my-pppoe-profile, and set the remote-address item in that new profile to some private address which isn't in conflict with any private subnet you use anywhere in your network - say, 10.22.33.44. In /interface pppoe-client configuration, you set the profile item to my-pppoe-profile. And in the individual route(s) to the anchor IP(s) used to monitor PPPoE availability, you use the 10.22.33.44 as a gateway address. This way, the remote-address setting from the /ppp profile my-pppoe-profile overrides the setting which came from the PPPoE server, and so it remains stable even though the PPPoE server sends you a different one each time.

Also I found in an article that the recursive method has the following limitation:
Whatever IP you use as your target is only reachable via the primary route. If the primary route is down, that IP address will be unreachable. If you use 8.8.8.8 to resolve DNS, the DNS service will be down when the primary route is down. Therefore if you use Google for DNS and use 8.8.8.8 as the routing target, you should use a different Google DNS server such as 8.8.4.4 for DNS instead.
This is normal - for any destination address the routes with the longest, i.e. most exactly matching, dst-address prefix are chosen. So if at least one route with dst-address=8.8.8.8/32 exists and is active, routes whose dst-address prefixes also match 8.8.8.8 but are shorter (wider), such as 8.8.8.0/24 or 0.0.0.0/0, are never chosen for delivery of packets to 8.8.8.8. This has two consequences:
  • you must not set check-gateway=ping for the individual routes to monitored anchor addresses, because if you do and the gateway becomes unreachable, the route becomes inactive and the check-gateway pings of the routes one level higher in the recursion start taking another route, ruining the idea of using inaccessibility of the anchor address as indication of network path failure
  • you cannot use the anchor IP for any other purpose than network path monitoring because the anchor IP has to be inaccessible if the path whose availability it monitors is broken, so you cannot set up an alternative route to the anchor IP.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sat Sep 22, 2018 11:09 am

Yes I don't have problem with the WAN2 a its gateway is constant. I'm using the ADSL modem as GW and it won't change. The route to WAN2 is static. The only thing that is changing is the remote address of the PPPoE which I'm using as WAN1 (main link).

The current set is:
WAN 1 - Optic -> media convertor -> Mikrotik at eth1
WAN 2 - phone line->ADSL modem ->Mikrotik at eth2

I have full access to the ADSL modem. I'm only not sure which DNS it was using but now on the Mikrotik I'm using the modem as DNS.That's why I believe the only problem is the PPPoE with it's changing GW. I'll try the proposed workaround for it and I'll write if there is any success as it's not still completely clear for me. If it was possible to use the pppoe interface instead of exact GW it would be way easier...

I'm really interested how in fact the current TP-link failover is in fact realized behind the wizard.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sat Sep 22, 2018 1:08 pm

it's not still completely clear for me.
PPPoE creates a Point-to-Point interface. For all interfaces of this type, there is no actual need to use any address of the remote device because "the remote end of the tunnel" is the only address you need - whatever you send out that interface will end up on the single remote device. This is a difference to Point-to-Multipoint interface where you need an address of a particular device in addition to the name of the interface. For practical reasons, the gateway addresses are configured as IP addresses, which allows to quickly choose the interface by its associated "network" address, and there the IP address of the gateway device is translated into its MAC address.

So a common habit is to use IP address as a gateway even for PPP interfaces although in these cases it actually acts only as an alias to the interface name. The recursive next-hop search needs IP addresses of gateways, that's a fact you have to merely accept :-)

But as the "remote" address of a PPP interface plays no other role in the process than the alias of the interface name, it is only meaningful in the local context of the sending device. Thus you may label the PPP interface with any "remote" address you like. And whilst /interface pppoe-client doesn't have a direct parameter remote-address, it does accept that parameter if provided by means of the profile and uses it to override the value provided by the server.

If it was possible to use the pppoe interface instead of exact GW it would be way easier...
I'll stay polite so I won't translate any of the Czech sayings related to this kind of statements, but it would at least take long to happen (if at all possible because I'm not deep into the recursive next hop search algorithm, so maybe there is some reason which excludes using the interface name as gateway). So if you want it now, use the workaround suggested or use a script. In fact, the next-hop search mechanism was also not originally intended for the failover use. Which BTW also means that the failover may happen up to about 10 seconds after the active WAN path breaks because this is how often the check-gateway pings are sent.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sat Sep 22, 2018 8:08 pm

For PPPoE (used at your WAN1), there is a script-less way which @Sob has described: you create a copy of /ppp profile named default, give it a name like my-pppoe-profile, and set the remote-address item in that new profile to some private address which isn't in conflict with any private subnet you use anywhere in your network - say, 10.22.33.44. In /interface pppoe-client configuration, you set the profile item to my-pppoe-profile. And in the individual route(s) to the anchor IP(s) used to monitor PPPoE availability, you use the 10.22.33.44 as a gateway address. This way, the remote-address setting from the /ppp profile my-pppoe-profile overrides the setting which came from the PPPoE server, and so it remains stable even though the PPPoE server sends you a different one each time.
So I've tried that but sadly while trying to connect it says the connection is terminated and it isn't able to make a connection when I'm using a random remote address.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sat Sep 22, 2018 9:50 pm

Hm, I've just tried it myself and it does work as expected, but the PPPoE client and server discuss the addresses during the startup phase so I assume your ISP's server doesn't accept that your client comes with its own idea what address the server should use at its end and terminates the negotiation.

However, if you know for sure that the ISP randomly chooses from just two remote addresses as you wrote, instead of scripting, you can create two selective routes to the monitored address each with one of those addresses as gateway, they may even have the same distance. You could also try to set up one of them in the profile and see whether it convinces the server to assign it systematically, but even if that works, it is against the concept of redundancy as it is likely that the two remote addresses actually belong to a different ISP's piece of hardware each.

A follow-up, not directly related to the above - as testing the idea of two alternative routes with different gateways suggested above, I've come back to the idea that check-gateway=ping must not be used to make sure that the monitored anchor address would be unreachable if the WAN path it is monitoring is down. In fact, it is the scope and target-scope of the recursive routes higher in the stack what should take care of not using a path whose monitored address is not accessible via a direct route. If the pppoe interface goes down, it doesn't help that you don't check-gateway the selective route and it goes inactive anyway, so a default route is used instead. But the scope and target-scope of the recursive routes higher in the stack should prevent it from being used recursively. However, nothing prevents it from being used directly, so the DNS requests may be sent to an anchor address used to monitor WAN1 even when it is accessible only via the recursive default route via WAN2.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sat Sep 22, 2018 10:50 pm

Sadly today I saw that it's not only these two GWs. They are more then two (yesterday it took only two but today I saw another 2). I though it may be the server side that is the problem with the profile variant as it is trying to establish a connection and imminently afterwords it's terminated. Honestly I didn't expect it to be so difficult to set up the fail-over but obviously if something is dynamic it gets a lot more complicated. I'm running out of ideas, I could put another router which to deal with the pppoe connection but the whole thing would lose its point.

When I set the remote address with the one of the given it managed to connect with the new profile but as a follow up the next connection failed as previously.
 
Sob
Forum Guru
Forum Guru
Posts: 3867
Joined: Mon Apr 20, 2009 9:11 pm

Re: Mikrotik Dual WAN Failover

Sat Sep 22, 2018 11:30 pm

... but the PPPoE client and server discuss the addresses during the startup phase ...
Oops, I didn't test that before. But it's true. RouterOS as PPPoE server doesn't seem to care and I don't see it doing anything with that address. But other implementation surely can.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 23, 2018 12:45 am

OK, so one possibility would be to use a script to generate a ton of routes for the whole range of remote address values the ISP provides.

A better possibility is to use an on-up parameter of the /ppp profile to call a script to update the lowermost recursive route:
/system script
add name=update-pppoe-route source=":local gtw \$\"remote-address\"\
    \n:local rte [/ip route find dst-address~\"8.8.8.8/32\"]\
    \n:if ([/ip route get \$rte gateway]!=\$gtw) do={\
    \n /ip route set \$rte gateway=\$gtw\
    \n}\
    \n"

/ppp profile
add name=my-pppoe on-up=update-pppoe-route

/interface pppoe-client set [find name=your-pppoe-client-interface-name] profile=my-pppoe
This way, the gateway of the route will be set to the remote address value received from the server each time the pppoe-client interface goes up and the currently configured gateway in that route is different.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 23, 2018 10:01 am

OK, so one possibility would be to use a script to generate a ton of routes for the whole range of remote address values the ISP provides.

A better possibility is to use an on-up parameter of the /ppp profile to call a script to update the lowermost recursive route:
/system script
add name=update-pppoe-route source=":local gtw \$\"remote-address\"\
    \n:local rte [/ip route find dst-address~\"8.8.8.8/32\"]\
    \n:if ([/ip route get \$rte gateway]!=\$gtw) do={\
    \n /ip route set \$rte gateway=\$gtw\
    \n}\
    \n"

/ppp profile
add name=my-pppoe on-up=update-pppoe-route

/interface pppoe-client set [find name=your-pppoe-client-interface-name] profile=my-pppoe
This way, the gateway of the route will be set to the remote address value received from the server each time the pppoe-client interface goes up and the currently configured gateway in that route is different.
Oook, this one worked, now it's updating the GW every time in the route. I hoped it would work without scripts and so but at least there is a way. The only thing I changes in the ppp profile was under Change TCP MSS from default to yes as in the default profile it's set to yes.

The only interesting thing is that via WAN2 eth port there are some spikes in the Tx and Rx from time to time so something is passing there, probably it's the connection test? Under load everything seems to pass through WAN1.
/ip route
add check-gateway=ping distance=1 gateway=8.8.8.8
add check-gateway=ping distance=2 gateway=8.8.4.4
add distance=1 dst-address=8.8.4.4/32 gateway=192.x.x.x scope=10
add distance=1 dst-address=8.8.8.8/32 gateway=109.x.x.x scope=10
/ppp profile
add change-tcp-mss=yes name=my-pppoe on-up=update-pppoe-route
/interface ethernet
set [ find default-name=ether1 ] name=WAN1-Ether1 speed=100Mbps
set [ find default-name=ether2 ] name=WAN2-Ether2 speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] speed=100Mbps
Not sure why the ports are set to 100Mbts instead of 1Gbps???
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wlan1
add bridge=bridge comment=defconf interface=wlan2
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=WAN1-Ether1 list=WAN
add interface=WAN2-Ether2 list=WAN
add interface=pppoe-out1 list=WAN
I'm still open to an scriptless method if someone have idea.
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 23, 2018 11:11 am

The only interesting thing is that via WAN2 eth port there are some spikes in the Tx and Rx from time to time so something is passing there, probably it's the connection test?
What is the traffic volume through WAN2? Each route with check-gateway=ping generates one ping request and response every 10 seconds, maybe up to three requests when the monitored IP doesn't respond (which is how netwatch behaves so I'd expect the same approach to be reused also here). Another source of traffic is DHCP renewal whose frequency depends on the lease time choice of the server (i.e. your ADSL modem).

Not sure why the ports are set to 100Mbts instead of 1Gbps???
Me neither, but it is not the default setting. If you use /interface ethernet set [find] speed=1Gbps, the speed will not be limited to 100 Mbit/s any more and the export should stop showing the speed parameter at all as 1Gbps is the default value, which means that the set lines for ether3 to ether5 will disappear from the export completely as they will not contain any non-default setting any more. If you are 120% sure you haven't modified those settings manually (even by mistake), some bug of this or some previously running software version may be responsible.

/interface list member
...
add comment=defconf interface=WAN1-Ether1 list=WAN
Just FYI, making WAN1-Ether1 an /interface list member is pointless unless you have an IP configuration attached directly to it. From the perspective of the IP firewall, only the pppoe-out1 is an IP interface and that WAN1-Ether1 is its underlying physical path is irrelevant for the IP firewall.

I'm still open to an scriptless method if someone have idea.
Out of curiosity, why would you like to get rid of scripts completely? Whereas a script directly controlling the failover itself has to be scheduled for a frequent periodical run, the script updating the route is only triggered by address reassignment which should happen rarely, so it causes a negligible CPU load and flash chip wear.

But thank you for pushing me to think about flash wear again, I've got an idea how to get rid of configuration updates in another design :-)

It's a pity that some PPPoE servers are not tolerant to the solution suggested by @Sob, because it means it cannot be used to resolve a conflict situation where the servers of two PPPoE uplinks provide the same remote-address.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 23, 2018 2:28 pm

What is the traffic volume through WAN2? Each route with check-gateway=ping generates one ping request and response every 10 seconds, maybe up to three requests when the monitored IP doesn't respond (which is how netwatch behaves so I'd expect the same approach to be reused also here). Another source of traffic is DHCP renewal whose frequency depends on the lease time choice of the server (i.e. your ADSL modem).
It's really minor - between 500 and 600 bps more likely around 590. When I put load to it the whole traffic pass through the working WAN i.e. WAN1 where the pppoe is set. Moreover the Dest. 8.8.4.4 through the GW of the ADSL is reachable and only the default route 0.0.0.0/0 is inactive (blue) so I guess that its most likely the ping.
Me neither, but it is not the default setting. If you use /interface ethernet set [find] speed=1Gbps, the speed will not be limited to 100 Mbit/s any more and the export should stop showing the speed parameter at all as 1Gbps is the default value, which means that the set lines for ether3 to ether5 will disappear from the export completely as they will not contain any non-default setting any more. If you are 120% sure you haven't modified those settings manually (even by mistake), some bug of this or some previously running software version may be responsible.
I haven't touched anything instead of setting the second WAN and removing it from the bridge. The interfaces are with active 10/100/1000 (they all have ticks) but It may be the auto negotiation that is doing it. For sure the ADSL is 10/100 and one of the routers I'm using as AP is also 10/100 (it's now connected to the eth3 port) if I manually set it to 1000 half/full it's shown as 1000 in the export.
Just FYI, making WAN1-Ether1 an /interface list member is pointless unless you have an IP configuration attached directly to it. From the perspective of the IP firewall, only the pppoe-out1 is an IP interface and that WAN1-Ether1 is its underlying physical path is irrelevant for the IP firewall.
I know but because I'm stepping on the default settings and the port was listed by default I haven't removed it from there - just added the pppoe-out to the list so the rules can apply.
Out of curiosity, why would you like to get rid of scripts completely? Whereas a script directly controlling the failover itself has to be scheduled for a frequent periodical run, the script updating the route is only triggered by address reassignment which should happen rarely, so it causes a negligible CPU load and flash chip wear.
Mainly because it's something that I'm not familliar with. I'd prefer to know everything that I've done to any settings and as the scripting is a bit advanced in this learning process I'd like to stick to the scriptless settings. However I'll check the syntax of the script language and I'll try to decode the script so I'd be able to reproduce it myself. It seems that in the current settings it will work only if the checking address is 8.8.8.8. As I'd want to realize the recursion with two different hosts just to be more reliable if it somehow happen that the google DNS is down. I'm to see how these settings will work with the script.
But thank you for pushing me to think about flash wear again, I've got an idea how to get rid of configuration updates in another design :-)
It would be great if it had possitive side for you, because you really helped me a lot. It seems that in this forum the community is really open and eager to help to the new users who are not familiar with the ROS. So thank you very much for the help.
It's a pity that some PPPoE servers are not tolerant to the solution suggested by @Sob, because it means it cannot be used to resolve a conflict situation where the servers of two PPPoE uplinks provide the same remote-address.
Yeah it would be great if the suggested by @Sab workaround was possible but at least we learned that it depends on the server side and it could be problematic. It could be even a problem if initially it works but for some reason the ISP decide to change the settings of its servers.

By the way, is the script for the let's say a dynamic IP common to the one you proposed for the pppoe GW monitoring?
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Sep 23, 2018 3:55 pm

I haven't touched anything instead of setting the second WAN and removing it from the bridge. The interfaces are with active 10/100/1000 (they all have ticks) but It may be the auto negotiation that is doing it. For sure the ADSL is 10/100 and one of the routers I'm using as AP is also 10/100 (it's now connected to the eth3 port) if I manually set it to 1000 half/full it's shown as 1000 in the export.
The advertise configuration parameter on one hand and the full-duplex and speed configuration parameters on the other one are used in an exclusive-or manner depending on the auto-negotiation setting. So if you have auto-negotiation set to yes, the speed configuration parameter should be ignored and the negotiated speed should be shown.

Mainly because it's something that I'm not familliar with. I'd prefer to know everything that I've done to any settings and as the scripting is a bit advanced in this learning process I'd like to stick to the scriptless settings. However I'll check the syntax of the script language and I'll try to decode the script so I'd be able to reproduce it myself. It seems that in the current settings it will work only if the checking address is 8.8.8.8. As I'd want to realize the recursion with two different hosts just to be more reliable if it somehow happen that the google DNS is down. I'm to see how these settings will work with the script.
The scripting works with lists, so you can configure a selection condition in the find which matches several routes so the find returns their list, and then the set will be applied to all items on the list. So you may use regular expressions (dst-address~"1.2.3.4|8.7.6.5") or a logical "or" ((dst-address="1.2.3.4" or dst-address="8.7.6.5")) to make the find return IDs of both the route to 1.2.3.4 and the route to 8.7.6.5.

It would be great if it had possitive side for you
It did as I've found an issue in that other design. And it has also pushed me to raise a ticket with support because something in RouterOS behaves counter-intuitively, so the idea I've got has failed because it was based on what the intuitive behaviour would be.

is the script for the let's say a dynamic IP common to the one you proposed for the pppoe GW monitoring?
In principle yes - the scripts are bound to /ip dhcp-client in a slightly different manner than to ppp interfaces (directly rather than via a profile, and a single script is invoked at any change so it has to determine the actual invoking event based on a context variable and choose the corresponding behaviour), but the task is the same - at each assignment or renewal of IP configuration, check whether the new gateway IP is the same like the previously assigned one and if it differs, modify the configuration.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sat Dec 15, 2018 8:35 pm

Greetings guys!

I didn't had a lot of time recently, so the further configuration of the hAP was on hold. As the christmas holidays are getting closer I hope that I'll manage to finalize and test everything that I wanted to do with the router. Currently I think that the failover is finished and it's working as intended with the best reliability it could be. I've changed the dual WAN failover with single gateway ping check to one with multiple. I'd post the current configuration with the hope that if there is any problem you guys would be able to spot it. Also I think that it could be in use of someone else who struggled with a configuration like mine.

I'm using the default router configuration for the firewall. Eth3-5 are bridged, Eth1 is used for the main ISP link (PPPoE) and Eth2 is set for the backup ISP (Static). Eth2 and pppoe-client added to the WAN interface list.
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes port=2200
set api disabled=yes
set api-ssl disabled=yes
/ip dns
set servers=1.1.1.1,1.0.0.1
allow-remote-requests=no
/ip address
add address=192.168.x.x/24 interface=bridge network=192.168.x.x
add address=192.168.x.x/24 interface=ether2 network=192.168.x.x
/ip route
add distance=1 gateway=10.1.1.1
add distance=2 gateway=10.2.2.2
add distance=1 dst-address=8.8.4.4/32 gateway=192.168.x.x scope=10
add distance=1 dst-address=8.8.8.8/32 gateway=x.x.x.x scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=8.8.8.8 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=208.67.220.220 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=8.8.4.4 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=208.67.222.222 scope=10
add distance=1 dst-address=208.67.220.220/32 gateway=x.x.x.xscope=10
add distance=1 dst-address=208.67.222.222/32 gateway=y.y.y.yscope=10
As it turned out the pppoe server wasn't giving the same remote address so the recursive wasn't properly working. Thanks to @sindy who wrote a script I was able to finally set it up. I've added two lines to the script so it could change the remote address not only for one address but for two as the dual WAN with multiple gateway ping check uses 2 for every connection. Here is the script:
:local gtw $"remote-address"

:local rte [/ip route find dst-address~"8.8.8.8/32"]

:local rtf [/ip route find dst-address~"208.67.220.220/32"]
:if ([/ip route get $rte gateway]!=$gtw) do={
 /ip route set $rte gateway=$gtw
}
:if ([/ip route get $rtf gateway]!=$gtw) do={
 /ip route set $rtf gateway=$gtw
}
Currently I'm not able to reduce the time for the second connection to take over and it takes probably around 30 seconds. I've set different DNS addresses than the one used for the failover.

Now I'm trying to deal with the port forwarding because I have OpenVPN on one openmediavault server but sadly till now I wasn't able to deal with it.
/ip firewall nat
add action=dst-nat chain=dstnat dst-address=my-static-address dst-port=xxxx protocol=udp \
to-addresses=Local-address of the server to-ports=xxxx

/ip firewall filter
add chain=forward dst-port=xxxx protocol=udp dst-address=Local-address of the server in-interface=ether1 action=accept
Once I manage to deal with this I'd finally set up the VLANs
 
anav
Forum Guru
Forum Guru
Posts: 2452
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: Mikrotik Dual WAN Failover

Sat Dec 15, 2018 10:25 pm

Your firewall NAT rule looks okay, if you your destination ports are the same as the to ports, you can drop the to=ports and just have the to-adddresses.

The Filter rule looks wrong, all you need is the following:
add action=accept chain=forward comment=\
"Allow Port Forwarding - DSTNAT" connection-nat-state=dstnat
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Dec 16, 2018 9:38 am

Your firewall NAT rule looks okay, if you your destination ports are the same as the to ports, you can drop the to=ports and just have the to-adddresses.

The Filter rule looks wrong, all you need is the following:
add action=accept chain=forward comment=\
"Allow Port Forwarding - DSTNAT" connection-nat-state=dstnat
Sadly It didn't worked even this way. The ports are the same and it's the default 1194
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Dec 16, 2018 12:29 pm

It's always hard to follow up after such a long time (how do the doctors deal with this?), so just a couple of questions which may actually have been answered before:
  • does your Mikrotik have the public address assigned directly to itself? If not, you have to set the port forwarding also on the device(s) between the Tik and the internet
  • if only one of your WANs has a public address, is it the primary one (with distance=1 on the recursive default route? If not, the response packets from the OVPN server are getting sent out the other WAN with the source address which does not belong to it and the next device on the path may not let them through for this reason
  • does your OVPN client use UDP transport? If not, your port forwarding rule won't handle the connection
You say that you're builiding on the default firewall configuration. In its recent versions, the factory default firewall configuration already contains a rule which permits any dst-nat connection, but yours may come from some older RouterOS release (a ROS upgrade doesn't modify existing custom configuration so unless you've reset the machine to defaults with a recent ROS installed, your current configuration is still based on the old default one). So better post your complete current configuration than refer to some unknown default.

Also, check whether /ip firewall nat print stats chain=dstnat shows your dst-nat rule to count packets/bytes when you try to connect the openvpn client. And make sure that when testing, you connect the client from outside, not from the same subnet in which the server is connected, as if you do, the dstnat rule works but the backward path is an L2 one so it bypasses the firewall, unless you add a src-nat rule making the server think that the request comes from the Mikrotik itself. The whole scenario you have to address is often called "hairpin NAT" and the trouble is that if you don't make the server sent the responses back to Mikrotik rather than directly to the client, the client sees the response to be coming from an address different from the one to which it has sent the request so it doesn't recognize the packets as belonging to the same connection.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Dec 16, 2018 1:06 pm

Hello Sindy,

I'm glad you've joined the conversation. I'm going to answer your questions in the order you posted them:
  • The Eth1 is for the PPPoE and it's address is directly coming from the pppoe-out client assigned on Eth1. Eth2 is with static address behind the ADSL modem, however I tried the port forwarding on the old tp-link 480T+ and the VPN connects.
  • Both WANs are with static public addresses but the backup is behind the ADSL modem. The one that is directly connected is the main link and it's with distance 1. Still I could try to disable the settings for the second WAN and see what will happen.
  • You should be ok, as the conf files and certificates are generated by the server and it's set to UDP at the moment. Also as I mentioned with the old router there isn't a problem with the connection.
I'm building on the default firewall and because it turned out that I've limited the access to the router by IP (which I forgot and thought I don't recall the password), so at the end I made a reset, upgraded to the last stable version and started the setting on the base of the def conf (the thing's I've changed are the IP, DHCP, Pools etc.) There is a rule in the firewall:
defconf: drop all from WAN not DSTNATed - I believe you're talking about this one?

The current Nat rule is:
chain=dstnat action=dst-nat to-addresses=Local-address-of-the-server protocol=udp
dst-address=Public-static-address-of-the-pppoe in-interface=pppoe-out1 dst-port=1194 log=yes
log-prefix=""
I'm indeed trying to connect to the server from a machine in the same subnet as the server. It worked with the other router but I'm going to try it from the network of the ADSL. On the server the public address is set to the same address I'm setting the NAT, it's UDP with the standard port and cert/key + password.

Sorry for the long period but I thought it would be better to use the same thread as the information would be in one place in case someone else have a similar problem. Also what do you think of the current failover configuration? I think that in this scenario would be enough to guarantee maximum reliability It would be great if it was possible to force a faster switch in case of problem with the main link but it's still good.

One more side question - what would be better to use port based VLAN or 802.1Q?

Thanks again for the precious help guys!

Edit: It worked when I tried to connect from a different network! Current settings are:
chain=dstnat action=dst-nat to-addresses=local-server-address protocol=udp
in-interface=pppoe-out1 dst-port=1194 log=yes log-prefix=""
No additional firewall rules
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Dec 16, 2018 2:05 pm

There is a rule in the firewall:
defconf: drop all from WAN not DSTNATed - I believe you're talking about this one?
Yes.

I'm indeed trying to connect to the server from a machine in the same subnet as the server.
This definitely needs to be addressed unless you only need it for the testing phase. Either give the server its own subnet or use a src-nat rule (/ip firewall nat add chain=srcnat action=src-nat protocol=udp dst-address=the.lan.ip.of.the.server dst-port=1194 to-addresses=the.ip.of.mikrotik.itself.in.the.lan.subnet.

On the server the public address is set to the same address I'm setting the NAT
Slow down here, I've probably misunderstood what you wrote. As far as I remember, you only tell the openvpn server on which local addresses to listen if you don't want it to listen on all of them. So if you specify any address at all, it should be the local LAN one of the server, not the public one which is not up on the server itself.

Sorry for the long period but I thought it would be better to use the same thread as the information would be in one place in case someone else have a similar problem.
I agree with the intention, but the thread is somehow quite curly and I wasn't sure I've sourced everything from it. After all the doctors structure their case notes in their own way, whereas here different people with different ways of thinking contribute so finding the bit of information you need right now is a challenge.

what do you think of the current failover configuration? I think that in this scenario would be enough to guarantee maximum reliability It would be great if it was possible to force a faster switch in case of problem with the main link but it's still good.
The limiting factor here is the check-gateway ping rate which is hardcoded to 10s. You mention 30 seconds needed for a failover, but you get this because additional factors contribute to the total time:

The first TCP SYN creates a tracked connection in a firewall which gets src-nated to the address of the WAN used. If this first SYN is sent short after the WAN link went down but the check-gateway ping hasn't noticed that yet, the routing still chooses the dead WAN. All subsequent re-transmissions of this SYN are mapped to the same tracked connection, so they get src-nated to the same address of the dead WAN despite being actually routed out via the other WAN, so even if they reach their destination via this second WAN, the destination sends its response to the dead address.

This does not happen for connections which are initiated after the check-gateway ping has already detected the failure.

Any speedup would require scripting, as you would have to
  • ping the anchor addresses in the internet more frequently than once in 10s to notice the failure faster (but doing so has some drawbacks like false negatives if the bandwidth is close to saturation),
  • remove any tracked connections with reply-dst-address set to the address of the dead WAN when you detect it is dead, to prevent the effect described above.

One more side question - what would be better to use port based VLAN or 802.1Q?
It depends on the rest of your network topology. If all your end devices are connected directly to Mikrotik, there is no need to deal with 802.1Q as you may create as many bridges as you want and assign the interfaces to these bridges freely. If you want to connect external switches to provide access ports for the end devices, or to connect devices which make use of 802.1Q internally (such as host servers for virtual guests), you save cables and interfaces by use of VLAN tagging (802.1Q, 802.1ad and others).
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
draid
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Wed Aug 22, 2018 5:42 pm

Re: Mikrotik Dual WAN Failover

Sun Dec 16, 2018 4:25 pm

This definitely needs to be addressed unless you only need it for the testing phase. Either give the server its own subnet or use a src-nat rule (/ip firewall nat add chain=srcnat action=src-nat protocol=udp dst-address=the.lan.ip.of.the.server dst-port=1194 to-addresses=the.ip.of.mikrotik.itself.in.the.lan.subnet.
It was just for testing but with the additional nat rule it works.
Slow down here, I've probably misunderstood what you wrote. As far as I remember, you only tell the openvpn server on which local addresses to listen if you don't want it to listen on all of them. So if you specify any address at all, it should be the local LAN one of the server, not the public one which is not up on the server itself.
Well its a plug-in to the OpenMediaVault server and its port is set to the default OpenVPN, The thing is that if I set the public address to its own local one the VPN server would be accessible only from the local network. The remote address on the server is set to the public static address of my main link so when the clients try to connect to the public address with the needed port it's redirected to the local address of the server. Or at least I think that this is the case. It is possible that my setup has flows but I don't see how it would work without the real public address.

This is the current NAT rule that works for the outside networks:
chain=dstnat action=dst-nat to-addresses= local-server-address protocol=udp dst-port=1194
log=yes log-prefix=""
It works with the static adress as dest. adress and without it set in the rule, honestly don't know which one should be the right.

As for the VLAN - currently it's on a different switch which is attached directly to one of the ports of the Tik. Everything on that switch is should be in a different VLAN with internet access/DHCP. The main VLAN is using another smart switch right after the Tik and from there APs.

Interesting thing about this failover is that when I'm checking it with constant ping from the command prompt it seems that the second link takes over imminently with a singe request timeout but for the internet to properly work it takes few seconds (between 15-30 on average) I don't know if in this case it isn't from the DNS. When the main link is up again it takes a bit more to switch to it but the ping comes way faster then the "internet"
 
sindy
Forum Guru
Forum Guru
Posts: 2699
Joined: Mon Dec 04, 2017 9:19 pm

Re: Mikrotik Dual WAN Failover

Sun Dec 16, 2018 5:28 pm

Well its a plug-in to the OpenMediaVault server and its port is set to the default OpenVPN, The thing is that if I set the public address to its own local one the VPN server would be accessible only from the local network. The remote address on the server is set to the public static address of my main link so when the clients try to connect to the public address with the needed port it's redirected to the local address of the server. Or at least I think that this is the case. It is possible that my setup has flows but I don't see how it would work without the real public address.
OK, so the public address is there for clients to connect to, not for the server to bind at. This way it makes sense.

This is the current NAT rule that works for the outside networks:
chain=dstnat action=dst-nat to-addresses= local-server-address protocol=udp dst-port=1194 log=yes log-prefix=""

It works with the static adress as dest. adress and without it set in the rule, honestly don't know which one should be the right.
That way (without in-interface or dst-address) it handles any connection to udp 1194, even if you'd e.g. have a client in the LAN trying to connect to some external server. So adding the in-interface (or in-interface-list) to the rule with a proper value will save you future headache.

As for the VLAN - currently it's on a different switch which is attached directly to one of the ports of the Tik. Everything on that switch is should be in a different VLAN with internet access/DHCP. The main VLAN is using another smart switch right after the Tik and from there APs.
If everything that belongs to one VLAN is connected to one interface of Mikrotik, albeit via an external switch, and everything what belongs to another VLAN is connected to another interface of Mikrotik, using yet another switch, port-based VLANs on Mikrotik are enough. If the external switches support 802.1Q VLANs, deploying 802.1Q at Mikrotik and those external switches would provide you with more flexibility in physical placement of the connected devices, as you could have ports from both VLANs on both switches.

Interesting thing about this failover is that when I'm checking it with constant ping from the command prompt it seems that the second link takes over imminently with a singe request timeout but for the internet to properly work it takes few seconds (between 15-30 on average) I don't know if in this case it isn't from the DNS. When the main link is up again it takes a bit more to switch to it but the ping comes way faster then the "internet"
The fact that ping recovers quickly is a surprise to me as a single ping sequence should suffer from the same factor like the TCP connections - for TCP and UDP, source and destination IP addresses and ports are used to identify a tracked connection, for ping it is source and destination IP addresses and the identifier field of ICMP echo packets. So again, all packets bearing this same identifier value (and IP addresses of course) are matched to the same tracked connection and maintain the src-nat address until this connection times out. Normally, all icmp echo request packets generated by a single run of the "ping" utility have the same value of the identifier field, so the same ping running "forever" should not ever recover, whereas a new one has a different identifier value and thus creates a new tracked connection, which starts when the primary WAN has already been found to fail, so it gets the src-nat address of the backup one.

As for what "internet" actually means, it may be DNS availability, and if using UDP transport, DNS typically retransmits the query packets rather than sending each new one from another local port, so the same mechanism as described for TCP applies - until a new local port is engaged, all query packets sent from the same one get the same treatment in Mikrotik's firewall.

You should see this if you set up packet sniffing on the backup interface or simply run /ip firewall connection print detail interval=1s dst-address~":53\$" and watch the reply-dst-address field.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.

Who is online

Users browsing this forum: No registered users and 5 guests