basic dual WAN configuration do not work

ieleja · April 6, 2020, 7:54am

hAP AC (962UiGS-5HacT2HnT) ROS v6.46.4.

ISP_1 (2 dynamic IP, 500 Mbps),
ISP_2 (1 static IP, 100 Mbps),
2 masquerade, srcnat rules, Out Interfaces ISP_1, ISP_2,
10 rules for internal servers forwarded from ISP_2,
default RouterOS rules

my configuration was worked fine for 5 years, but one ISP_1 (2 dynamic IP, 500 Mbps) do some nightly service works, “broke my internet connection”, router switched to ISP_2 (1 static IP, 100 Mbps), I call service and get technician visit, tell him history, he calls some master technician, he reconfigure something at their site, and ISP_1 internet works, if I directly connect computer to it

but!

if I connect this ISP_1 Ethernet cable to router, router switches to WAN1 - ISP_1, but ISP_2 link stops working - PING stops, internal services from outside are not accessible, if I disable ether1_ISP1, all starts working again after second

it looks like, that when ether1_ISP1 is enabled, all data goes somewhere wrong, but I can’t find where and why

Zacharias · April 6, 2020, 5:04pm

it looks like, that when ether1_ISP1 is enabled, all data goes somewhere wrong, but I can’t find where and why

Thats what i think as well…
Is your ISP 1 IP still Dynamic or Static ?
If it is static, did you make sure your config and your addresses, subnets etc are correct ?

ieleja · April 6, 2020, 5:55pm

ISP_1 it’s dynamic - as my experiments I disable and enable this ether1 and get addresses - 46., 85., 95. and so on. for now I even leave Add Default Route - on, and if I enable ether1 (ISP1), I get working internet through this link with full speed (~500/500), but after second of enable ether1, static link ether5 came unreachable from outside (but if I try ping 1.1.1.1 through ether5 I get ~7ms response)

ieleja · April 6, 2020, 7:59pm

just as experiment I put simple configuration:

# apr/06/2020 22:42:27 by RouterOS 6.46.4
# software id = 5DWL-EDZZ
#
# model = 751G-2HnD

/interface bridge
add admin-mac=00:0C:42:FB:C7:47 auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] disabled=yes
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=wlan1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether5 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether2 network=192.168.88.0
add address=87.87.87.87/24 interface=ether5 network=87.87.87.0
/ip dhcp-client
add comment=defconf disabled=no interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=forward comment="accept forwarded ports" dst-port=80 protocol=tcp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=ether1
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=ether5
add action=dst-nat chain=dstnat comment="services on MPE" dst-address=87.87.87.87 dst-port=80 protocol=tcp to-addresses=192.168.88.80
/ip route
add distance=20 gateway=87.87.87.1
set winbox address=192.168.0.0/16
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Riga
/system ntp client
set enabled=yes primary-ntp=192.168.88.80
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

on spare RB-751G and get the same results, if I enable ether1, ether5 came unreachable from outside, PING stops, forwarded web server on 80 port stops respond, but internet connection to outgoing connections switches from ether5 to ether1 and gets ~500/500 Mbps

I have no idea what’s going on…

ieleja · April 6, 2020, 9:53pm

and for clear experiment, I remove all LAN devices, just connect to RB-751G ether1, ether5 as WANs and one Windows 10 notebook to ether3 as LAN device. results - all as previous - if I turn on ether1, ether5 stops respond from outside

tdw · April 7, 2020, 12:10am

There do not appear to be any mangle rules or additional routing tables to properly support dual WAN operation. Replies to traffic arriving from either ISP1 or ISP2 will return via the best default route to ISP1 - this may have accidentally worked until the nightly engineering works correctly blocked this traffic.

ieleja · April 7, 2020, 12:44am

thanks, it looks possible reason for my problems
I don’t find HOWTO for dual WAN, when one link is with dynamic IP, with possibly changing gateway

tdw · April 7, 2020, 1:27am

I’m sure load balancing with dynamic gateways will have cropped up before in the forums - a script triggered by the DHCP client, or possibly using routing filters.

It depends on what you need - if all traffic will be to/from ISP1, other than replies to that coming from ISP2, instead of full load balancing you could get away with something like
/ip firewall mangle
add chain=prerouting dst-address=87.87.87.87/24 action=accept in-interface=LAN
add chain=prerouting in-interface=ether5 connection-mark=no-mark action=mark-connection new-connection-mark=ISP2_conn
add chain=prerouting connection-mark=ISP2_conn in-interface=LAN action=mark-routing new-routing-mark=to_ISP2
add chain=output connection-mark=ISP2_conn action=mark-routing new-routing-mark=to_ISP2

/ip route
add gateway=87.87.87.1 routing-mark=to_ISP2 check-gateway=ping

basically a cut-down version of https://wiki.mikrotik.com/wiki/Manual:PCC. Note that you will have to disable fasttrack for mangling to work.

ieleja · April 7, 2020, 5:54am

thanks!

1st mangle rule I in dst-address put 87.87.87.0/24 as ‘ip address expected’ or I must use 87.87.87.87?
2nd and 3rd I use In. Interface List=LAN or in-interface=bridge

this corrects “half” of problems one time - if I enable ether1, ether5 no stop answering on PINGs, but inside servers came unreachable from outside

and after next disable/enable eth1, eth5 stopped answer on outside PINGs completely if eth1 is enabled

p.s.
as You say - “load balancing with dynamic gateways will have cropped up before in the forums” - but it was answers and fragments from some configurations, not complete tutorial

but thanks again for trying!

ieleja · April 7, 2020, 10:38am

delete DHCP client and manually added routes and import this

/ip firewall mangle add action=mark-connection chain=prerouting \
comment="Connmark in from ETH" connection-mark=no-mark in-interface=ether5 \
new-connection-mark=conn_eth passthrough=no

/ip firewall mangle add action=mark-connection chain=prerouting \
comment="Connmark in from TET" connection-mark=no-mark in-interface=ether1 \
new-connection-mark=conn_tet passthrough=no



/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Routemark transit out via ETH" connection-mark=conn_eth \
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_eth passthrough=no

/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Routemark transit out via TET" connection-mark=conn_tet \
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_tet passthrough=no



/ip firewall mangle add action=mark-routing chain=output \
comment="Routemark local out via ETH" connection-mark=conn_eth \
dst-address-type=!local new-routing-mark=to_eth passthrough=no

/ip firewall mangle add action=mark-routing chain=output \
comment="Routemark local out via TET" connection-mark=conn_tet \
dst-address-type=!local new-routing-mark=to_tet passthrough=no



/ip route add check-gateway=ping comment="For recursion via ETH" \
distance=1 dst-address=4.2.2.1 gateway=87.87.87.1 scope=10

/ip route add check-gateway=ping comment="Unmarked via ETH" \
distance=2 gateway=4.2.2.1

/ip route add comment="Marked via ETH Main" distance=1 gateway=4.2.2.1 \
routing-mark=to_eth

/ip route add comment="Marked via TET Backup1" distance=2 gateway=4.2.2.1 \
routing-mark=to_tet

/ip route rule add comment="From ETH IP to Inet" src-address=87.87.87.87 table=to_eth



/ip dhcp-client
add add-default-route=no disabled=no interface=ether1 script=":if (\$bound=1) do={\r\
    \n   /ip route remove [ find gateway=\"4.2.2.2\" ]; /ip route remove \
          [ find where dst-address ~\"4.2.2.2\" ]\r\
    \n   /ip route add check-gateway=ping comment=\"For recursion via TET\" \
          distance=1 dst-address=4.2.2.2/32 gateway=\$\"gateway-address\" scope=10\r\
    \n   /ip route add check-gateway=ping comment=\"Unmarked via TET\" \
          distance=1 gateway=4.2.2.2\r\
    \n   /ip route add comment=\"Marked via TET Main\" distance=1 gateway=4.2.2.2 \
          routing-mark=to_tet\r\
    \n   /ip route add comment=\"Marked via ETH Backup1\" distance=2 \
          gateway=4.2.2.2 routing-mark=to_eth\r\
    \n   /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
          out-interface=\$\"interface\" to-addresses=\$\"lease-address\" \
          comment=\"NAT via TET\"\r\
    \n   /ip route rule add comment=\"From TET IP to Inet\" \
          src-address=\$\"lease-address\" table=to_tet \r\
    \n} else={\r\
    \n   /ip route remove [ find gateway=\"4.2.2.2\" ]; /ip route remove \
          [ find where dst-address ~\"4.2.2.2\" ]\r\
    \n   /ip firewall nat remove  [find comment=\"NAT via TET\"]\r\
    \n   /ip route rule remove [find comment=\"From TET IP to Inet\"]\r\
    \n}\r\
    \n" use-peer-dns=no use-peer-ntp=no

anav · April 7, 2020, 11:52am

Config comes LAST. A thorough understand of requirements comes first.

ieleja · February 15, 2024, 9:35pm

after upgrade to v7.* and ISP blocking any ICMP beyond gateway, old problem comes again, if if optics with DHCP is enabled, routing messes and port forwarding don’t work. corrected via adopting DHCP script to v7.*

routing rule

instead

ip route rule

as in v6.*

another correction in “recursion” route, now ISP gateway, not 4.2.2.1, which isn’t pingable from this ISP

Mesquite · February 16, 2024, 3:22pm

Dual wan works very well.
What is not clear is the requirements, so I have to guess.
WAN1 is primary should run most traffic all the time. If WAN1 fails you want users to go to WAN2?
WAN2 is secondary, handles servers all the time. If WAN2 fails you want servers accessible on WAN1?
add distance=1 check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=main comment=MyStatic
add distance=2 check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing table=main comment=MyDynamic
Thats the starting point for two fixed static WANIPs.
If you have a dynamic WANIP you need to ensure the gateway IP is updated via script.
So in the IP DHCP client area under ADVANCED TAB enter the following script:

:if ($bound=1) do={
:local gw [ /ip dhcp-client get $“gateway-address” ]
/ip route set [ find comment=“MyDynamic” gateway!=$gw ] gateway=$gw
}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

If you want to add recursive routing, then it would look like.
add distance=1 check-gateway=ping dst-address=0.0.0.0/0 gateway=1.1.1.1 routing-table=main scope=10 target-scope=12 comment=wan1
add distance=2 check-gateway=ping dst-address=0.0.0.0/0 gateway=9.9.9.9 routing table=main scope=10 target-scope=12 comment=wan2

add distance=1 dst-address=1.1.1.1/32 gateway=ISP1-gatewayIP routing-table=main scope=10 target-scope=11
add distance=2 dst-address=9.9.9.9/32 gateway=ISP2-gatewayIP routing table=main scope=10 target-scope=11 comment=MyDynamic

Since you have servers that need to work on WAN2 all the time, we will need to ensure that external traffic hitting the WANs heading to the servers, goes out the same WAN. We do both WANs in case WAN2 is down and we use WAN1 for servers…

add action=mark-connections chain=forward in-interface=WAN1 connection-mark=no-mark
new connection-mark=from-WAN1 passthrough=yes
add action=mark-connections chain=forward in-interface=WAN2 connection-mark=no-mark
new connection-mark=from-WAN2 passthrough=yes
add action=mark-routing chain=prerouting in-interface-list=Servers connection-mark=from-WAN1
new-routing-mark=to-WAN1 passthrough=no
add action=mark-routing chain=prerouting in-interface-list=Servers connection-mark=from-WAN2
new-routing-mark=to-WAN2 passthrough=no

NOTE: firewall address list Servers, contains the IP addresses of all servers…

Basically this marks any traffic coming on the WAN with the appropriate connection marks.
Then any traffic with these marks coming from the servers (reply traffic), gets sent out the same WAN they came in on.
To complete the config we need tables and routes.

/routing table add fib name=to-WAN1
/routing table add fib name=to-WAN2

And then the routes
add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=to-WAN1
add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=to-WAN2 comment=MyDynamic

++++++++++++++++++++++++++++++++++++++++++++++++++

ieleja · February 19, 2024, 3:59am

Dual wan works very well.

Yes, and in latest post I explain corrections after:

upgrade from v6 to v7,
my ISP with static IP now blocks any IGMP, except I only can ping my GW, not these “1.1.1.1”, “9.9.9.9” and so…

but thanks for answer, I add it to my personal scripting book…