Two providers. Unstable behavior.

RouterOS General
1

Hi.
My config in general: bridge-inet - main ISP, backup-bridge - backup ISP and ISP for some IP-IP tunnel.
I try to create config where my Router can accept connections from main and backup channels. Some times working well, but sometimes I have problem that I cannot understand.
I have two ISP. And my router has three IP-addresses on two external interfaces:
/interfaces list
add interface=bridge-inet list=wan
add interface=bridge-backup list=wan

/ip address
add address=85.XXX.XXX.21/24 interface=bridge-inet network=85.XXX.XXX.0
add address=85.XXX.XXX.20 interface=bridge-inet network=85.XXX.XXX.20
add address=89.YYY.YYY.YYY/24 interface=bridge-backup network=89.YYY.YYY.0

/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=bridge-inet new-connection-mark=main-in passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=bridge-backup new-connection-mark=backup-in passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=main-in in-interface-list=!wan new-routing-mark=maininet passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting connection-mark=backup-in in-interface-list=!wan new-routing-mark=backupinet passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=main-in new-routing-mark=maininet passthrough=yes
/ip firewall mangle add action=mark-routing chain=output connection-mark=backup-in new-routing-mark=backupinet passthrough=yes

/ip firewall nat add action=masquerade chain=srcnat out-interface-list=wan

/ip route add distance=1 gateway=85.XXX.XXX.1
/ip route add distance=2 gateway=89.YYY.YYY.1
/ip route add distance=1 gateway=85.XXX.XXX.1 routing-mark=maininet
/ip route add distance=1 gateway=89.YYY.YYY.1 routing-mark=backupinet

/ip route rule
add action=lookup-only-in-table src-address=89.YYY.YYY.YYY/32 table=backupinet
add action=lookup-only-in-table src-address=85.XXX.XXX.20/32 table=maininet
add action=lookup-only-in-table src-address=85.XXX.XXX.21/32 table=maininet

After these config setup I have lost packets from inet to my main address router:
Ping statistics for 85.XXX.XXX.20:
Packets: Sent = 854, Received = 824, Lost = 30 (3% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 91ms, Average = 6ms

How to troubleshoot?

2
  1. need network diagram.
  2. clearer set of requirements.
    what do you need users or groups of users to be able to do or not do…

What is the relationship between the ISP connections (assuming from diff providers) (failover, primary, secondary, equally used etc…)
What is the relationship between lan subnets and the ISP connections…

  1. full config
    /export hide-sensitive file=anynameyouwish
3

@BlackRat, the setting you highlited is IMO invalid. It’s not logical to have address with network address set to same value. If bridge-inet should use both addresses 85.xxx.xxx.20 and 85.xxx.xxx.21 and when router uses either of WAN addresses it can directly connect to the same subnet (which is logical if xxx-es in addresses), then both addresses should be set with same subnet mask (/24) and same network address (85.xxx.xxx.0).

Note that having multiple addresses on same interface doesn’t preclude routing decisions (selection of next hop) nor use of particular address (if using SRC-NAT), other settings take care of that (routing tables, NAT rules). The way you use SRC-NAT (action=masquerade), it probably instructs ROS to use one of WAN addresses, but when using address 85.xxx.xxx.20 it can’t really access upstream router (because no other IP address is inside same subnet) and those connections fail.

4

It is a /32 address, and usually used for loopback interfaces

5

I have special rule for additional NAT
add action=src-nat chain=srcnat out-interface=bridge-inet src-address=192.168.188.200 to-addresses=85.XXX.XXX.20
because I want to use another external address for my internal Server.

6

Thank You.
Changed to 85.xxx.xxx.20/24.
I will be monitoring. Maybe it was a problem.

7

You can use this as start, removing all your actual routes, route rules and mangles

/ip dns
set servers=1.1.1.1,8.8.8.8

/ip route
add comment="A - 1.1.1.1 must be reachable only from ISP1" distance=1 dst-address=1.1.1.1/32 gateway=85.XXX.XXX.1 scope=10
add comment="B - Recursive Routing, check ping 1.1.1.1 instead of ISP IP" distance=10 gateway=1.1.1.1 check-gateway=ping
add comment="C - ISP2 is alternative Gateway" distance=20 gateway=89.YYY.YYY.1
8

I don't want to create backup ISP for failover (sorry for "main" and "backup"). I want to create equal paths (but 85.XXX.XXX.1 = main ISP) and create IP-IP tunnels to second (as my "backup") backup-bridge interface. I cannot create this config without mangle, etc.

9

Ah, ok I misunderstand for labels :wink:

10

Made ping test. And I think, that reason is wrong settings for additional address for external interface. I should use 85.xxx.xxx.20**/24** and 85.xxx.xxx.21**/24** instead of 85.xxx.xxx.20/32 and 85.xxx.xxx.21/24
Thanks’ to CZFan anв mkx.