Multiwan setup with Starlink and ip/route check-gateway issue

zentavr · October 25, 2023, 2:49am

Hi,

I’m trying to set up Multi WAN mikrotik router where the second ISP is Starlink with ethernet adapter. The issue is that Starlink provides the default gateway 100.64.0.1 (mac address 00:00:5E:00:01:01) which does not reply at icmp requests. I guess that they use some kind of a load balancing/or switching when the sattelite is passing the region.

The setup looks like this:

# Adding local subnet to the main table
/routing rule
add action=lookup disabled=no dst-address=172.16.46.0/24 table=main

/routing table
add disabled=no fib name=rtab-wan
add disabled=no fib name=rtab-sat

# Marking the incoming connections from every ISP
/ip firewall mangle
add action=mark-connection chain=prerouting comment=WAN-Routing-Mark connection-mark=no-mark in-interface=bridge_wan new-connection-mark=from-wan-internet passthrough=no
add action=mark-connection chain=prerouting comment=SAT-Routing-Mark connection-mark=no-mark in-interface=bridge_sat new-connection-mark=from-sat-internet passthrough=no

## Outgoing transit traffic rules
add action=mark-routing chain=prerouting comment="WAN Outgoing transit traffic to rtab-wan Routing Table " connection-mark=from-wan-internet dst-address-type=!local in-interface-list=!WAN new-routing-mark=rtab-wan passthrough=no
add action=mark-routing chain=prerouting comment="SAT Outgoing transit traffic to rtab-sat Routing Table " connection-mark=from-sat-internet dst-address-type=!local in-interface-list=!WAN new-routing-mark=rtab-sat passthrough=no

## Outgoing local traffic rules
add action=mark-routing chain=output comment="WAN Outgoing Local traffic to rtab-wan Routing Table " connection-mark=from-wan-internet dst-address-type=!local new-routing-mark=rtab-wan passthrough=no
add action=mark-routing chain=output comment="SAT Outgoing Local traffic to rtab-sat Routing Table " connection-mark=from-sat-internet dst-address-type=!local new-routing-mark=rtab-sat passthrough=no

## Dedicated Routing rules per address lists
add action=mark-routing chain=prerouting comment="Users routed via WAN" dst-address-list=!BOGONS new-routing-mark=rtab-wan passthrough=yes src-address-list=Routed-via-WAN place-before=0
add action=mark-routing chain=prerouting comment="Users routed via SAT" dst-address-list=!BOGONS new-routing-mark=rtab-sat passthrough=yes src-address-list=Routed-via-SAT place-before=0

# Adding emergency default route
/interface bridge add name=br-lo comment="Loopback Routing Interface"
/ip route add distance=254 gateway=br-lo comment="Emergency route"

# Adding the route for the 1st testing IP via WAN
/ip route
add check-gateway=ping comment="For recursion via WAN" distance=1 dst-address=4.2.2.1 gateway=80.92.227.42 scope=11
add check-gateway=ping comment="Unmarked via WAN" distance=1 gateway=4.2.2.1 target-scope=11
add comment="Marked via WAN" distance=1 gateway=4.2.2.1 routing-table=rtab-wan target-scope=11
add comment="Marked via SAT" distance=2 gateway=4.2.2.1 routing-table=rtab-sat target-scope=11

# Route rule from local traffic to internet (WAN)
/routing/rule/add action=lookup comment="From WAN IP to Inet" src-address=8.9.7.41 table=rtab-wan

The DHCP client for starlink bridge is done like this:

/ip dhcp-client add add-default-route=no disabled=no interface=bridge-sat

Also there is a script attached to DHCP clients for bridge-sat:

:local ispThis "SAT"
:local ispThisRt "rtab-sat"
:local ispThisGwCheck "arp"
:local ispThisMainRtDistance "2"

:local ispMain "WAN"
:local ispBackup "SAT"

:local checkIp "4.2.2.2"
:local ispMainRt "rtab-wan"
:local ispBackupRt "rtab-sat"

#:local postScript "some-script"

:log info "$ispThis: Bound: $bound, Gateway: $"gateway-address""

:if ($bound=1) do={
    :log info "$ispThis: Removing the old route records"
    /ip route remove [ find gateway="$checkIp" ]
    /ip route remove [ find where dst-address ~"$checkIp" ]

    :log info "$ispThis: Adding actual route records"
    /ip route add check-gateway=$ispThisGwCheck comment="For recursion via $ispThis" distance=1 dst-address=$checkIp gateway=$"gateway-address" scope=11
    /ip route add check-gateway=ping comment="Unmarked via $ispThis" distance=$ispThisMainRtDistance gateway=$checkIp target-scope=11
    /ip route add comment="Marked via $ispBackup" distance=1 gateway=$checkIp routing-table=$ispBackupRt target-scope=11
    /ip route add comment="Marked via $ispMain" distance=2 gateway=$checkIp routing-table=$ispMainRt target-scope=11

    :log info "$ispThis: Setting NAT"
    :if [:tobool ([/ip firewall/nat/ find comment="NAT via $ispThis"])] do={
        :log info "$ispThis:     .... rule had been updated"
        /ip firewall nat set [find comment="NAT via $ispThis"] action=src-nat chain=srcnat ipsec-policy=out,none out-interface=$"interface" to-addresses=$"lease-address"
    } else={
        :log info "$ispThis:     .... rule had been added"
        /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none out-interface=$"interface" to-addresses=$"lease-address" comment="NAT via $ispThis"
    }

    :log info "$ispThis: Setting routing rules"
    :if [:tobool ([/routing/rule find comment="From $ISP IP to Inet"])] do={
        :log info "$ispThis:     .... rule had been updated"
        /routing/rule/set [find comment="From $ispThis IP to Inet"] action=lookup src-address=$"lease-address" table=$ispThisRt
    } else={
        :log info "$ispThis:     .... rule had been added"
        /routing/rule/add action=lookup comment="From $ispThis IP to Inet" src-address=$"lease-address" table=$ispThisRt
    }

    #:log info "$ispThis: Runnit the post action script"
    #/system script run $postScript

    :log info "$ispThis: Activation is done!"
} else={
    :log info "$ispThis: Removing route records"
    /ip route remove [ find gateway="$checkIp" ]
    /ip route remove [ find where dst-address ~"$checkIp" ]

    :log info "$ispThis: Removing firewall NAT"
    /ip firewall nat remove  [find comment="NAT via $ispThis"]

    :log info "$ispThis: Removing routing rules"
    /routing/rule/remove [find comment="From $ispThis IP to Inet"]

    :log info "$ispThis: Cleaning is done!"
}

The problem is that “check-gateway” feature when there is “ping” value not works (because the Starling gateway not accepts ICMPs). I also tried ARP - that somehow not work as well, but I can see the mac-address of the gateway at IP-ARP via Winbox. Could it be the issue because the gateway is somewhere at the space and the RouterOS times out the arp response? Doing arp ping from the routeros console works as well.

How can I check that starlink is up using Mikrotik’s native tools?
Знімок екрана 2023-10-25 о 05.39.47.png

anav · October 25, 2023, 11:55am

Could care less about the config because more important are the requirements.
What is WAn1 and what is WAn2 is it primary and failover? Is it PCC load balancing.
What are the expectations if one of the WANs is not available.
What are the expectations for lan users, some to wan1 and some to wan2
Any VPNs??

/export file=anynameyouwish ( minus router serial # and any public WANIP information )

jvanhambelgium · October 25, 2023, 2:29pm

Put the Starlink in a separate VRF and work from there?
You could issue some health-check to eg. 8.8.8.8 across the Starlink-vrf and make some decisions from there?

Amm0 · October 25, 2023, 4:32pm

I’d think the check-gateway=ping on the 4.2.2.2 is enough with the recursive routing. e.g. check-gateway likely doesn’t be on the starlink interface itself. The CGNAT will drop if there is no starlink & 4.2.2.2 recursive route is already checking internet connectivity.

FWIW, the OP’s 4.2.2.2 is one of the oldest DNS public servers … so it’s same as 8.8.8.8 used in other recursive routing configs - there a few of them conveniently sequential 4.2.2.1 … 4.2.2.6, e.g. 4.2.2.1 is ISP1 canary, 4.2.2.2 is ISP2 canary & n.b. using 8.8.8.8 as canary causes anyone using Google DNS to go out only one ISP, while 4.2.2.1…6 are highly unlikely to be used as DNS servers these days

zentavr · October 25, 2023, 8:20pm

WAN is the PON connection which might be down because of electricity power loss. Starlink is the backup channel (we have Ecoflow battery for this kind of power outgages).

It is not a PCC load balancing (we actually did not thought about that feature yet )

So either one or another WAN Provider should handle the internet connections. The fiber channel is preferable if available.

Actually the main requirement the users would be able to connect to the internet (just office stuff like skype, MS Teams, Slack, Zoom, etc…)

Mikrotik is an IPSec + OpenVPN server at the WAN1 IS (PON Fiber). It’s ok that be unreachable when the main fiber ISP is down.

zentavr · October 25, 2023, 8:21pm

Frankly speaking I hadn’t worked with VRF at all. Would you be so kind to submit any recommendations to start with?

zentavr · October 25, 2023, 8:59pm

As for the check-gateway:

[zentavr@zentavr-mt-rv] /ip/route> export 
# 2023-10-25 23:22:44 by RouterOS 7.11.2
/ip route
# ...
# This "check-gateway" works pretty well. Either "arp" or "ping" works here
add check-gateway=arp comment="For recursion via WAN" distance=1 dst-address=4.2.2.1/32 gateway=A.B.C.65 routing-table=main scope=11
# ...
# This "check-gateway" does not work at all. "arp" works some time and really flappy, but or "ping" does not work at all. The gateway is marked as "unreachable" in 20 seconds.
add check-gateway=none comment="For recursion via SAT" distance=1 dst-address=4.2.2.2 gateway=100.64.0.1 scope=11
...

These IPs had been selected just because of high SLA.