Multiple WAN Failover

RouterOS General
1

Hello,

I am following this guide to try to do failover between 3 WANs and some load balancing also.
I have all 3 interfaces configured: https://goo.gl/cy6kjJ
I have DHCP enabled on all 3 also, and they get IPs: https://goo.gl/ThDbBo
I have even removed these ports from the bridge: https://goo.gl/TPp8zL

However, I can only ping out on the first one, the others won’t ping: https://goo.gl/NJLSyu

2

Not enough information and many things may be wrong.

For example, each DHCP server offers a default gateway along with the other configration, and you have to use the gateway matching the interface when sending packets. Whenever you use several dhcp clients on a single system, you have to deal properly with these multiple gateways, where “properly” depends on a number of factors.

So please provide the output of “/export hide-sensitive” and also of “/ip route print”. The crystal ball is broken.

3

This is a normal situation, read about Policy Base Routing

https://wiki.mikrotik.com/wiki/Policy_Base_Routing

/p

4

Here you go:

[admin@MLM] > /export hide-sensitive
# mar/12/2018 12:15:59 by RouterOS 6.41.2
# software id = Q9KG-WHFF
#
# model = 2011iL
# serial number = xxxxxxxxx
/interface bridge
add admin-mac=6C:3B:6B:BE:19:92 arp=proxy-arp auto-mac=no comment=defconf name=\
    bridge
/interface ethernet
set [ find default-name=ether1 ] name=COMCAST-100M
set [ find default-name=ether2 ] name=DSL-45M
set [ find default-name=ether3 ] name=SATELLITE-30M
set [ find default-name=ether6 ] loop-protect=on
/interface pptp-client
add allow=chap,mschap2 connect-to=router.xxxxx.net disabled=no name=pptp-FMT \
    user=Openvpn-MLM
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=192.168.5.30-192.168.5.254
add name=vpn ranges=192.168.89.2-192.168.89.255
/ip dhcp-server
add address-pool=dhcp bootp-lease-time=lease-time bootp-support=dynamic \
    disabled=no interface=bridge name=DHCP
/ppp profile
set *FFFFFFFE local-address=192.168.5.1 remote-address=dhcp
/interface bridge port
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface l2tp-server server
set enabled=yes use-ipsec=yes
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=COMCAST-100M list=WAN
/interface pptp-server server
set enabled=yes
/interface sstp-server server
set default-profile=default-encryption enabled=yes
/ip address
add address=192.168.5.1/24 comment=defconf interface=ether6 network=192.168.5.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=\
    COMCAST-100M
add dhcp-options=hostname,clientid disabled=no interface=DSL-45M
add dhcp-options=hostname,clientid disabled=no interface=SATELLITE-30M
/ip dhcp-server lease
add address=192.168.5.18 client-id=\
    ff:bc:9a:4a:2d:0:2:0:0:ab:11:f5:a4:38:4a:1a:32:4a:1c mac-address=\
    00:0C:29:AF:4B:A8 server=DHCP
/ip dhcp-server network
add address=192.168.5.0/24 dns-server=192.168.5.1 gateway=192.168.5.1 netmask=\
    24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,4.2.2.2
/ip dns static
add address=192.168.5.1 name=router.lan
add address=192.168.5.18 name=wiki
add address=192.168.5.18 name=docs
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input dst-port=8291 protocol=tcp
add action=accept chain=input comment="allow pptp" dst-port=1723 protocol=tcp
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=\
    out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
/ip route
add distance=1 gateway=COMCAST-100M routing-mark=COMCAST-100M
add distance=1 gateway=DSL-45M routing-mark=DSL-45M
add distance=1 dst-address=192.168.2.0/24 gateway=10.255.255.1
/ip service
set winbox address=0.0.0.0/0
/ppp secret
add name=vpn
add local-address=10.50.1.2 name=WYN remote-address=10.50.1.3
/system clock
set time-zone-name=America/Chicago
/system identity
set name=MLM
/system ntp client
set enabled=yes primary-ntp=129.6.15.28 secondary-ntp=129.6.15.30
/system routerboard settings
set cpu-frequency=750MHz
/tool e-mail
set address=in-v3.mailjet.com from=mikrotik@xxx.com start-tls=yes user=\
    xxxx
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
[admin@MLM] >

and

[admin@MLM] > /ip route print
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          COMCAST-100M              1
 1 A S  0.0.0.0/0                          DSL-45M                   1
 2 ADS  0.0.0.0/0                          68.63.48.1                1
 3  DS  0.0.0.0/0                          75.107.16.1               1
 4  DS  0.0.0.0/0                          99.38.20.1                1
 5 ADC  10.255.255.1/32    192.168.5.1     pptp-FMT                  0
 6 ADC  68.63.xx.0/22      68.63.xx.xx    COMCAST-100M              0
 7 ADC  75.107.xx.0/22     75.107.xx.xx   SATELLITE-30M             0
 8 ADC  99.38.xx.0/22      99.38.xx.xx    DSL-45M                   0
 9 A S  192.168.2.0/24                     10.255.255.1              1
10 ADC  192.168.5.0/24     192.168.5.1     bridge                    0
5

There is a number of issues in your configuration so maybe my comments will be a bit chaotic, sorry for that.

  1. as you have a dhcp-client attached to both COMCAST-100M and DSL-45M, you cannot use these interfaces as routes’ gateways, you must use the IP addresses of the gateway devices connected to them. Assuming that the “default” gateways you get from the DHCP servers on these interfaces are static, you can configure the dhcp-clients for use-default-gateway=no and manually configure the gateway IP addresses you would normally receive as route gateways. If the gateway addresses change (which unlikely but possible), you would have to set the dhcp-clients to accept them with some high distance like 4 and use a dhcp-client script to copy the gateway addresses to the routing tables on each lease renewal.
    Besides, recursive routes can not work with interface names as gateways even if the WAN links would be point-to-point ones - see later what recursive routes are and why you need them.
  2. your masquerade rule says:
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN

However, the interface list WAN has a single member:

/interface list member
add comment=defconf interface=COMCAST-100M list=WAN

This means that only packets leaving via COMCAST-100M are masqueraded. It is fine for packets sent by the Mikrotik itself but a disaster for packets from other devices on its LAN as these leave with the original source address so either the ISP will drop them or the responses from the recipients will go anywhere but back to your network.
3. Leaving aside that the two static routes with dst-address=0.0.0.0/0 wouldn’t work anyway as they use interface as a gateway, you have configured them with routing marks but you do not assign routing marks anywhere in your configuration (no “/ip firewall mangle” rules exist in your configuration at all). Without packet-marking rules, routing of any packet, locally sourced or forwarded one, can only choose from routes with no routing mark, which are the three ones dynamically created by dhcp clients. As all of them have the same priority (distance) and their respective interfaces are all up, only the first one by order of creation (or by lowest IP address value, don’t ask me which one is true) is active:

 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 2 ADS  0.0.0.0/0                          68.63.48.1                1
 3  DS  0.0.0.0/0                          75.107.16.1               1
 4  DS  0.0.0.0/0                          99.38.20.1                1
  1. your firewall rules permit connection fasttracking which is incompatible with policy routing. So if the use of routing marks on the two routes is a stub of policy routing configuration (where particular categories of traffic would use their own order of preference for the three WAN links), you have to limit fasttracking only to connections using the default routing table because otherwise your connections would be initiated via one WAN and continue via another one (or not at all if there would be no route for them in the default routing table), so they would fail.
  2. your LAN IP address configuration is attached to ether6 which is a member port of bridge “bridge”. I am surprised that it works at all. It may be a result of configuration migration during upgrade, but you should change the interface in the /ip address configuration from “ether6” to “bridge” to avoid possible ghostly effects.

So to test that all WAN links are working, always disable two of the three dynamic routes with dst-address=0.0.0.0/0 and try to ping something in the internet (like 8.8.8.8 ) without specifying the interface. Then enable another “default” route and disable the already tested one.

Next, I wonder which tutorial have you used for configuring the WAN failover? The usual approach is to use recursive routes, see this article which explains both why and how.