Failover Issue

pothi · Fri Feb 01, 2019 7:10 am

Hello all,

Newbie here.

I have Mikrotik hap ac2 with the latest version of Router OS ( version: 6.43.8 ).

I have an ADSL connection as primary and LTE connection via Android Tethering as secondary. I route both ADSL and LTE via Mikrotik. I want to use Mikrotik router for failover. Manual failover is okay for me, and it works too. I have an issue with the (manual) failover.

As a first step, I linked ADSL connection to Mikrotik's Ethernet 1 port and used Webfig->Quickset for the initial configuration.

Then, I connected Android phone via USB tethering, created firewall NAT rule (chain=srcnat action=masquerade out-interface=lte1 log=no log-prefix="").

Modified DHCP clients to never "Add Default Route", so that I can add routes manually.

Deleted existing routes at IP => Route and created them manually. Now, I can switch between both internet connections manually. I can see my public IP has changed whenever I change the route (aka internet connection from ADSL to LTE or vice versa).

The issue is when I switch between ADSL to LTE, all existing connections (for example, ping to 1.1.1.1) that originated from ADSL doesn't switch to LTE automatically (ping command shows timeout). Similarly, when switch from LTE to ADSL, all existing connections that originated from LTE doesn't switch to ADSL automatically. For example, I had to restart the browser in order to use the newer internet connection.

Please know that the same ping command, if I issue from within Mikrotik router (/ping 1.1.1.1), the ping keeps going when I switch the routes. I hope you all can understand what I am trying to say or do.

Here's output of /export hide-sensitive compact

# feb/01/2019 10:27:19 by RouterOS 6.43.8
# software id = KBW2-46HF
#
# model = RBD52G-5HacD2HnD
# serial number = xxxx
/interface bridge
add admin-mac=xxxx auto-mac=no comment=defconf name=bridge
/interface lte
set [ find ] mac-address=xxxx name=lte1
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX disabled=no distance=indoors frequency=auto mode=ap-bridge ssid=MikroTik-20C6B3 wireless-protocol=\
    802.11
set [ find default-name=wlan2 ] band=5ghz-a/n/ac channel-width=20/40/80mhz-XXXX country=india disabled=no distance=indoors frequency=auto mode=ap-bridge ssid=MikroTik-20C6B4 \
    wireless-protocol=802.11
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wlan1
add bridge=bridge comment=defconf interface=wlan2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether2 network=192.168.88.0
/ip dhcp-client
add add-default-route=no comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
add add-default-route=no dhcp-options=clientid,hostname disabled=no interface=lte1
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat out-interface=lte1
/ip route
add distance=1 gateway=192.168.75.1
add distance=2 gateway=192.168.42.129
/system clock
set time-zone-name=Asia/Kolkata
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

What did I miss to do for a smooth failover of internet connection on my PC?

solar77 · Fri Feb 01, 2019 11:07 am

your routing config:

/ip route
add distance=1 gateway=192.168.75.1
add distance=2 gateway=192.168.42.129

as long as the router can reach gateway IP (which does not mean there is internet connection beyond this gateway), the route will be available and distance=1 route (assuming that's your ADSL) is going to be the default and not switching to 192.168.42.129

have a look this
https://wiki.mikrotik.com/wiki/Advanced ... _Scripting

pothi · Fri Feb 01, 2019 2:46 pm

Thanks for the reply.

I went through the aforementioned article and other relevant articles. From my understanding after reading those, existing sessions (especially from ping or from the same website on the same browser) don't switch to the new ISP after the failover. I may be wrong.

anav · Fri Feb 01, 2019 2:57 pm

I think it has to do with your source nat rules
The first one should be out-interface=ether1

Also if either connection has a STATIC WAN IP, then you should use action srcnat,
(this example covers if both are static)
add action=src-nat chain=srcnat out-interface=wan1 to-address=fixedWANIP1
add action=src-nat chain=srcnat out-interface=wan2 to-address=fixedWANIP2

Also have something missing...
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
Where is your lte?
add interface=lte list=WAN ??

For routes and failover this is more typical.......
/ip route
add check-gateway=ping distance=1 gateway=192.168.75.1
add distance=2 gateway=192.168.42.129

The router will select the shortest route but if not available will switch to gateway2 and back when gateway1 comes back on line.

solar77 · Fri Feb 01, 2019 4:16 pm

Thanks for the reply.

I went through the aforementioned article and other relevant articles. From my understanding after reading those, existing sessions (especially from ping or from the same website on the same browser) don't switch to the new ISP after the failover. I may be wrong.

I think you are right. This is because the Connection Tracking . New connection will move to the new route once it's selected but existing established connections will still try the previous route before it times-out. You can change the time out values in IP firewall connections, then Tracking.

solar77 · Fri Feb 01, 2019 4:20 pm

or reset all connections as soon as you move to the 2nd connection, by disable / enable Connection Tracking.

anav · Fri Feb 01, 2019 5:32 pm

Nice Solar77

Is there a difference how connection tracking or already established connections are handled between masquerade and source nat when the Wans change??

Also can you provide a script that clears the connection table upon change of WAN

sebastia · Fri Feb 01, 2019 5:42 pm

The routing will depend on the routing configuration. So if primary route comes back, traffic will be directed that way, but as connection tracking (especially natting) is still set of the backup route, natting won't take place. These connection should then be dropped by "invalid" check.
Also le'ts not forget that the other side knows the connection from "backup IP", and forwarded packets from "primary IP" won't be recognised.

Once existing connection have been detected as dead, the application will setup new connections over primary path.

solar77 · Fri Feb 01, 2019 5:48 pm

hi anav

1st question, to the best of my knowledge, not really. and cannot think why it should be any differences. Masquerade is basically source nat but only change the src-ip to that of the out-interface.

2nd, I believe you can do

/ip firewall connection tracking set enabled=no

which is easy to add to existing script when you already use script. but I use the auto-failover by ping external IP . If I was doing it, I would create a netwatch to external IP, say 4.4.4.4 and if it takes longer than xxxms, I reset all connections. I am sure there are situations where this is going to cause other issues but hay, it's Friday.

@sebastia
once the destination route becomes unavailable , would "established" connection become "invalid" connections? I thought you either have to reset it or wait for it to expire.

sebastia · Fri Feb 01, 2019 8:25 pm

See https://mum.mikrotik.com/presentations/ ... 639302.pdf

1. masq will clear connection tracking table if the related ip becomes inactive. Src-nat won't. slide 25

That is one of the measures as specified by Mikrotik, slide 28+, to ensure no leakage happens = packets don't leave on wrong interface without natting.

pothi · Sat Feb 02, 2019 5:17 am

Also have something missing...
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
Where is your lte?
add interface=lte list=WAN ??

I checked again. LTE is indeed missing when I do "/export hide-sensitive". Not sure why RouterOS doesn't include LTE connections. Probably, a bug or design decision by RouterOS.

Thanks for pointing it out @anav.

or reset all connections as soon as you move to the 2nd connection, by disable / enable Connection Tracking.

Tried to disabling and then enabling Connection Tracking using the code...

/ip firewall connection tracking set enabled=no
:delay 5s
/ip firewall connection tracking set enabled=yes

Source: viewtopic.php?t=103812#p515861

It didn't help. But, it actually helped indirectly to find the right answer from the same thread.

/ip firewall connection remove [find]

The above code indeed removes all existing connections and let the application (ping, browser, etc) create new connections.

I can't thank you enough for showing the right directions @solar77 .

Thanks to @sebastia too. I learned a lot from your answers too.

Some final thoughts:

As per multiple sources (and from @sebastia)...

See https://mum.mikrotik.com/presentations/ ... 639302.pdf

1. masq will clear connection tracking table if the related ip becomes inactive. Src-nat won't. slide 25

My issue is directly related to the ability of clearing the connection tracking table. I used masquerade as well, on both interfaces (in /ip firewall nat). For some reason, RouterOS doesn't clear the connection tracking table when the interface is changed. Could it be a bug? Or does masquerade clear the connection tracking table only if the public IP changes within the same interface? Or is it just how RouterOS is designed (by not clearing the connection tracking table upon routing change from one interface to another interface)?

If anyone clarifies on this, I can create a separate thread to discuss further. Or I can provide further information, if any, to debug further. For now, this issue is resolved by clearing the connection tracking table manually (using the aforementioned command as script).

anav · Sat Feb 02, 2019 3:18 pm

Great last question pothi, I was wondering the same myself!! Lets see if the masters can answer this riddle LOL.

sebastia · Sat Feb 02, 2019 3:25 pm

(not a master...but)

Routing is separate from masquerading. The last one is linked to active ip on the used interface. Only if that ip changes (or is lost) will the connection table be cleared.
A routing change doesn't impact the ip assignment of an interface.
So primary coming up, it's primary route becoming active again, and hence not sending traffic over backup / masq interface will NOT result in clearing of backup masq conn tracking.

anav · Sat Feb 02, 2019 3:30 pm

Yeah but I bet YODA drives an alfa romeo!!!

Thanks, so basically keep masquerade and routing functionality separate (as always) when assessing what the router is doing.

pothi · Tue Feb 05, 2019 5:17 am

Sorry about the delay to respond. For unknown reason, I was unsubscribed from this topic. Subscribed again now.

Thanks @sebastia for clarifying. I understand the core concept now (masq is different from routing).

I was manually switching the interface to LTE whenever I wanted to upload something. My ADSL's upload speed is capped at 750kpbs (even without the cap, the default upload speed in ADSL isn't great in general, compared to LTE). So, basically, I was only changing the routes, manually, even though both routes are still active or ready to serve internet traffic at all the time. Basically, the public IP is never lost or changed on either interface at all the time. So, connection tracking table was never cleared.

Probably, it was all my mistake. I should've mentioned my usecase in OP. That may have cleared lots of confusion about failover, etc. Actually, I shouldn't have used the term "failover". Because, I was only switching the route/s. Nothing related to "failover". My apologies.

Thanks again @anav for your great support throughout this thread.

Failover Issue [SOLVED]

Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue [SOLVED]

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Re: Failover Issue

Who is online