pcc and failover configuration not working on wlan

RouterOS General
1

Hi guys,
I have set up my RB951G-2HnD with dual Internet connections WAN1 and WAN2 and WLAN users. I have done PCC load balancing and failover.
The problem arises when, for example, the first Internet is disconnected and does not automatically switch over to the second Internet. The first interface must be manually disabled to enable the second Internet. After the first Internet connection is reconnected, the Internet remains on the second line of the Internet until it is manually disabled again.
Failover really doesn’t work. I also tried these settings on other Mikrotik device 433UAH and it didn’t make any difference. also I also checked the routing lines several times for Longest Prefix matching and I don’t know where the problem is.

I want the WAN2 to be activated immediately when the WAN1 is disconnected, and after the WAN1 is comes up, the WAN1 will be primary line and serve users.
Also provide users with both Internet broadband when both are connected.

Please guide how the problem solved. Any advise would be greatly appreciated.
The settings are as follows :

Ether2 - WAN1 (192.168.3.2/24), main ISP → 2Mb Bandwidth
Ether3 - WAN2 (192.168.3.10/24) → 8 Mb Bandwidth
wlan1 - local network 192.168.2.1/24

my configuration:

/interface ethernet
set [ find default-name=ether2 ] name=WAN1-ether2
set [ find default-name=ether3 ] name=WAN2-ether3
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa-psk,wpa2-psk eap-methods=“” group-ciphers=
tkip,aes-ccm management-protection=allowed mode=dynamic-keys name=
profile1 supplicant-identity=“” unicast-ciphers=tkip,aes-ccm
wpa-pre-shared-key=“mikrotik” wpa2-pre-shared-key=“mikrotik”
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-g/n channel-width=20/40mhz-Ce
disabled=no frequency=2422 mode=ap-bridge radio-name=MyWifiNet
security-profile=profile1 ssid=MyWifiNet
/ip pool
add name=dhcp_pool0 ranges=192.168.2.20-192.168.2.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=wlan1 lease-time=2d name=
dhcp1
/ip address
add address=192.168.3.2/24 interface=WAN1-ether2 network=192.168.3.0
add address=192.168.10.2/24 interface=WAN2-ether3 network=192.168.10.0
add address=192.168.2.1/24 interface=wlan1 network=192.168.2.0
/ip dhcp-server network
add address=192.168.2.0/24 dns-server=8.8.8.8,4.2.2.4 gateway=192.168.2.1
ntp-server=202.162.32.12
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall mangle
add action=accept chain=prerouting dst-address=192.168.3.0/24 in-interface=
wlan1
add action=accept chain=prerouting dst-address=192.168.10.0/24 in-interface=
wlan1
add action=mark-connection chain=prerouting connection-mark=no-mark
in-interface=WAN1-ether2 new-connection-mark=ISP1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark
in-interface=WAN2-ether3 new-connection-mark=ISP2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark
dst-address-type=!local in-interface=wlan1 new-connection-mark=ISP1_conn
passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark
dst-address-type=!local in-interface=wlan1 new-connection-mark=ISP2_conn
passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=ISP1_conn
in-interface=wlan1 new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2_conn
in-interface=wlan1 new-routing-mark=to_ISP2 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1_conn
new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn
new-routing-mark=to_ISP2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1-ether2
add action=masquerade chain=srcnat out-interface=WAN2-ether3
/ip route
add check-gateway=ping distance=1 gateway=192.168.3.1 routing-mark=to_ISP1
add check-gateway=ping disabled=yes distance=1 gateway=192.168.10.1
routing-mark=to_ISP2
add check-gateway=ping distance=1 gateway=192.168.3.1
add check-gateway=ping distance=2 gateway=192.168.10.1

2

I’m afraid you expect too much from the failover. As your two uplinks use src-nat (masquerade in your case), ongoing connections will fail whenever the uplink they use fails, and they have to be re-established. The reason is that the remote party doesn’t recognize packets coming from another address as belonging to an existing connection and ignores them.

In the particular case of Mikrotik (and any other system whose firewall is based on linux kernel’s netfilter), there is one more thing - the connection tracking module of the firewall keeps src-nating packets of a connection with the source address of the WAN through which the connection has been established even when they are actually routed through the other one, so they are likely to be dropped before even reaching the remote party. And if we talk about UDP connections where the client keeps sending, the connection in the connection-tracking keeps getting refreshed so it never expires and re-establishes, so it keeps the old address.

This is where the masquerade kicks in - it differs from plain src-nat not only in sourcing the IP address to be used for src-nat from the out-interface, it also deletes all connections which use that address from connection tracking if that address changes (via DHCP or other method of dynamic assignment of address) or if the interface bearing that address goes down.

So when you shut down the primary interface, all clients’ outbound connections running through it are removed from connection-tracking and re-establish in the connection tracking with a new src-nat address, but the remote party ignores the packets with the new address, so the client has to initiate new connections. This applies to all TCP sessions and to those UDP sessions which carry some stateful application protocol (such as SIP registration). But when you switch the primary interface on again yet don’t shut down the secondary one, the UDP connections established through the secondary one are not removed from connection tracking, so even if the client attempts to re-establish them, the connection tracking keeps src-nating them to the address of the secondary interface and they never succeed. TCP sessions are a different story, as each new attempt of the client normally (exceptions exist) uses a different port at client side so the connection tracking treats it as a new one.

To overcome this, you would have to schedule a script for periodical run, which would periodically look for connections whose connection-mark says that they should use WANx but their reply-dst-address matches the one of WANy and would remove them whenever WANx would be up (or route through WANx would be active which is easier when your WANs are of different type). But there is a caveat if you use SIP phones and need to use the SIP helper in the firewall (which is not always the case): people report that removing SIP connections from connection tracking doesn’t always work.

3

Dear Sindy,

Thank you very much for your complete explanation. Could you give me a sample script to solve the problem or advice on how to write it? We also use VOIP (SIP) connections.
It is very important for me to solve this problem.

4

Given that there is no standard way to force SIP phones (or PBXes) to re-register when you need them to do so (which is each time when the uplink migrates to another WAN), the question is whether the script is sufficient to resolve your headache.

So one possibility is to configure the phones to re-register every 3 minutes or so and see whether the exchange will accept that, another possibility is to disable and re-enable the switch ports to which the phones are connected and check whether the phones notice that and send a new DHCPDISCOVER and later a new REGISTER etc., but if they are not connected directly to Mikrotik devices, it may not be easy to control the switch ports this way.

In another words, the script is the last thing to deal with. First diagram the network so that it is clear what the overall VoIP architecture is.

There is also another possibility to deal with this if you can run a virtual machine somewhere in a data center (which is supposed to have a more reliable connection to the internet); in this case, you could set up a tunnel to your site via each of the two uplinks and do the NAT on the virtual machine; this way, the outside address seen by the remote devices would be the same regardless which physical uplink would be used.

5

How about writing a script, assuming it’s not a VOIP (SIP) in network at all? Please give me a practical example. Thanks.

6

The script body would say something like

if [/ip route get [find dst-address=0.0.0.0/0 gateway=ip.of.WAN1.recursive.gw] active] do={
  /ip firewall connection remove [find srcnat and connection-mark=via-WAN1 and reply-dst-address~"the.ip.of.wan2"]
}

It assumes that you use e.g. the recursive next-hop search to control the failover, so the default route via WAN 1’s gateway is only active when WAN1 is not only physically up but it is actually possible to deliver a packet to internet through it. If the recursive route is active, which means that internet is accessible via that WAN, the script removes all connections which are connection-marked to use WAN1 but ended up being NATed to WAN2’s IP because they have been initiated while WAN1 was unusable. You may also add the symmetric function (with roles of WAN1 and WAN2 swapped) into the same script. Then, use a scheduler to run this script every 5 seconds or so.

But as said, sometimes connections marked as SIP ones seem not to be removable, so switching off the SIP ALG may be necessary for the removal to work whereas keeping it on may be necessary t for the SIP calls to work, so you may find yourself trapped in an unresolvable contradiction.

7

yeah VOIP is a funny beast during failover scenarios. I have experienced it first hand.

A past thread on this very topic that is the best.
http://forum.mikrotik.com/t/sip-client-cannot-re-register-in-the-sip-server-after-switching-isp-different-nat/115091/1

good thread on voip in general
http://forum.mikrotik.com/t/using-routeros-to-qos-your-network-2020-edition/66683/1

Other interesting MUM presentation on VOIP
https://mum.mikrotik.com/presentations/US17/presentation_4321_1496084451.pdf
https://www.youtube.com/watch?v=tM7wyKdnIKA&feature=youtu.be

Interesting url info for a script to let you know a link is down.
http://gregsowell.com/?p=819

8

I am implementing the above script and will resubmit if I have any problems. Thank you very much for your help sindy and anav.
Sindy, can I contact you by email? If so, email me at the email I mentioned below. Thank you very much.
my email address is : sepehr.parandeh@gmail.com

9

Sindy, I implement your script, also Tomas Kirnak’s script as mentioned in the link below :
https://wiki.mikrotik.com/wiki/Failover_Scripting
I replaced my network parameters in the script. But it didn’t work. One day I worked on it all.
This problem is on my brain and I want to solve it any way I can. please direct me to the right place if you can.

10

I have sent you an e-mail message a few hours after you’ve posted your previous post, have you ever received it?

11

Hi Guys. I am strugling with something and i do not understand whats going on.
I have PCC setup and as soon as i enable the one WAN route my Mikrotik can not ping the internet.

here is what i have. WAN3 is dedicated for PBX only.

/ ip firewall mangle
add chain=prerouting dst-address=192.168.8.0/24 action=accept in-interface=LAN0
add chain=prerouting dst-address=192.168.7.0/24 action=accept in-interface=LAN0
add chain=prerouting dst-address=192.168.9.0/24 action=accept in-interface=LAN0

add chain=prerouting dst-address=192.168.8.0/24 action=accept in-interface=LAN5
add chain=prerouting dst-address=192.168.7.0/24 action=accept in-interface=LAN5

add chain=prerouting dst-address=192.168.8.0/24 action=accept in-interface=LAN4
add chain=prerouting dst-address=192.168.7.0/24 action=accept in-interface=LAN4

add chain=prerouting dst-address=192.168.8.0/24 action=accept in-interface=LAN3
add chain=prerouting dst-address=192.168.7.0/24 action=accept in-interface=LAN3

add chain=prerouting dst-address=192.168.8.0/24 action=accept in-interface=RAW
add chain=prerouting dst-address=192.168.7.0/24 action=accept in-interface=RAW

add chain=prerouting in-interface=WAN1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=WAN2 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface=WAN3 connection-mark=no-mark action=mark-connection new-connection-mark=WAN3_conn

add chain=prerouting in-interface=LAN0 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=LAN0 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting connection-mark=WAN1_conn in-interface=LAN0 action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=LAN0 action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting in-interface=LAN5 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=LAN5 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting connection-mark=WAN1_conn in-interface=LAN5 action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=LAN5 action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting in-interface=LAN4 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=LAN4 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting connection-mark=WAN1_conn in-interface=LAN4 action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=LAN4 action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting in-interface=LAN3 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=LAN3 connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting connection-mark=WAN1_conn in-interface=LAN3 action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=LAN3 action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting in-interface=RAW connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=RAW connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting connection-mark=WAN1_conn in-interface=RAW action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=RAW action=mark-routing new-routing-mark=to_WAN2

add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_conn action=mark-routing new-routing-mark=to_WAN2
add chain=output connection-mark=WAN3_conn action=mark-routing new-routing-mark=to_WAN3
/
/ ip route
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.7.1 routing-mark=to_WAN2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.7.1 distance=2 check-gateway=ping
/
/ ip firewall nat
add chain=srcnat out-interface=WAN1 action=masquerade
add chain=srcnat out-interface=WAN2 action=masquerade
add chain=srcnat out-interface=WAN3 action=masquerade

12

In what you’ve posted I cannot see anything that would explain the behaviour you describe. If I get the description right, the problem begins when you enable one of the routes with a routing-mark, is that right? What is also not clear, is it only the Tik which cannot ping internet, or also the devices connected in various LANs?

Other than that, two points:

  • your rules are overly complex because you don’t make use of the benefits of address-list and interface list aggregators - you repeat the same set of 4 rules for each LANx whereas you could have just one instance of that set if you used in-interface**-list**=pcc-LAN, and LAN0, LAN3, LAN4, LAN5 and RAW were made member interfaces of that list.
  • you seem to be affected by the common misunderstanding of the meaning of dst-address-type=!local. All IP addresses except the own ones of the Mikrotik itself match this condition. There is no address-type value connected-subnet which would match any addresses in a connected subnet, which is what you seem to actually expect from address-type value local, and you have to create an address-list for that purpose (or reach that goal in another way, e.g. using dst-address=!192.168.0.0/16 instead of an address list consisting of just a few particular subnets, which is often possible)

.

Another remark, the goal should be to minimize the number of rules a packet has to be inspected by as it is processed. So over time, I’ve settled on the following:

chain=prerouting connection-mark=no-mark action=jump jump-target=conn-mark

chain=prerouting connection-mark=CM1 action=mark-routing new-routing-mark=RM1 passthrough=no
…
chain=prerouting connection-mark=CMX action=mark-routing new-routing-mark=RMX passthrough=no

chain=conn-mark condition_list_1 connection-mark=no-mark action=mark-connection new-connection-mark=CM1
…
chain=conn-mark condition_list_X connection-mark=no-mark action=mark-connection new-connection-mark=CMX
chain=conn-mark connection-mark=no-mark action=mark-connection new-connection-mark=use-main

So this way, always only the first packet of each connection gets connection-marked; even if it matches none of the connection-mark assignment criteria, the connection gets marked with a “use-main” connection mark in order to avoid its subsequent packets from running through all the rules. After reaching the end of the chain=conn-mark, the processing of that packet continues by the first rule following the action=jump one, so the packet does get its routing-mark (unless it got the “use-main” connection-mark). Packets belonging to already marked connections skip the first rule and go directly to the action=mark-routing rules.

13

Hi

Sorry for the late reply. Yes as soon as i enable the main WAN ie. WAN1 the pings go up and the just no ping. at this point the users also start experiencing no internet browsing. Here is what i have. Please tell me what you think i can change. i am new to these kinds of setups and thought it is better to keep all my network rules apart from one another.

Ok So let me get this right
add chain=prerouting in-interface=WAN1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface=WAN2 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface=WAN3 connection-mark=no-mark action=mark-connection new-connection-mark=WAN3_conn

add chain=prerouting in-interface-list=LAN connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:3/0 action=mark-connection new-connection-mark=WAN1_conn
add chain=prerouting in-interface-list=LAN connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:3/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=prerouting in-interface-list=LAN connection-mark=no-mark dst-address-type=!local per-connection-classifier=both-addresses:3/2 action=mark-connection new-connection-mark=WAN3_conn
add chain=prerouting connection-mark=WAN1_conn in-interface-list=LAN action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface-list=LAN action=mark-routing new-routing-mark=to_WAN2
add chain=prerouting connection-mark=WAN3_conn in-interface-list=LAN action=mark-routing new-routing-mark=to_WAN3
add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_conn action=mark-routing new-routing-mark=to_WAN2
add chain=output connection-mark=WAN3_conn action=mark-routing new-routing-mark=to_WAN3
/ ip route
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.7.1 routing-mark=to_WAN2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.7.1 routing-mark=to_WAN3 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.7.1 distance=2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.9.1 distance=2 check-gateway=ping
/ ip firewall nat
add chain=srcnat out-interface=WAN1 action=masquerade
add chain=srcnat out-interface=WAN2 action=masquerade
add chain=srcnat out-interface=WAN3 action=masquerade

Like that for the firs part Less clutter?
current.rsc (30.1 KB)

14

I think I’ve found your problem, you’ve swapped the gateway IP addresses in the two marked routes:

/ip address
add address=192.168.7.2/24 interface=WAN1 network=192.168.7.0
add address=192.168.8.2/24 interface=WAN2 network=192.168.8.0
add address=192.168.9.10/24 interface=WAN3 network=192.168.9.0

/ip route
add check-gateway=ping distance=1 gateway=192.168.8.1 routing-mark=toWAN1
add check-gateway=ping distance=1 gateway=192.168.7.1 routing-mark=to_WAN2
add check-gateway=ping distance=1 gateway=192.168.9.1 routing-mark=to_WAN3_

15

hi

so i changed that and here is my changes. i am experiansing ither no internet or intermediate internet on client side and on the Mikrotik it pings now.
CurrentNew.rsc (24.4 KB)

16

I cannot see anything wrong in particular. Just to give you a more exact picture, when no route with a matching routing-mark is found for a packet with a routing-mark assigned, the routing looks for a route in table main (i.e. it chooses from routes without any routing-mark assigned, or with routing-mark=main which is the default value). As both your default routes in table main have the same distance, one of them is active but I can’t say which one it is. While all the routes with a routing-mark are disabled, the active one out of those two is used, so the whole path related to that route must be OK.

So I’d say you should enable the routes with routing-mark=to_WANx just one at a time to find out which one is broken. I can see nothing wrong about the rules assigning the connection-marks based on per-connection-classifier and/or the rules translating these connection-marks to routing-marks.

17

Thx at this time i have a better connection i made WAN1 distance 1 and WAN2 distance 2 now i have a problem with my nat to my remote desktop server i see the connection come in but not going back on the same route.

18

To respond via the same WAN through which the connection came in, the routing-marked route via that WAN must be enabled. But the fact that this route&associated network path works for these incoming connections still does not mean that it works for all connections. There may be ICMP-related issues with PMTUD making some sites unreachable through that route.

So I insist on what I’ve said, check all three marked routes one by one (the connections marked to use other routes will be handled by the main routing table) also by connections initiated from LAN side. To get a result faster, you may assign the right connection mark by an additional mark-connection rule placed right before the PCC ones, which will match on a particular LAN address (so connections from that host will always use the marked route).

19

Hi

So here is what i did and now it looks as if it all is working stable for now.

i reset the hole mikrotik and started all from scratch only adding what i need for the time being and still it gave me a few problems so i came back to your posts and started gouping the lans and wans. i am only taging the first packets and monitoring the related as you suggested and wala. internet but still unstable so what i did was not alowing the mikrotik to detect internet i forced the mikrotik to see the wan group as internet and wala stable. thx for the help m8 i owe you a beer