Routes Taking Priority over Unrelated Routes

RouterOS General
1

I’ve got a not-so-simple set up here, so let me outline that first:

We have two routers, RTR-001 and RTR-002, both are the RB1100AHx4 model with passive passthrough; and two ISPs, ATT and Comcast. Comcast plugs in to RTR-002 and ATT into RTR-003. Either ISP will passively passthrough to the opposite router in the event that its router loses power or dies. This setup allows for both ISP and router failover.

We have VRRP configured with RTR-002 being the master and RTR-003 being the backup. Both routers plug directly in to the LAN.

Packets are given route marks based on either their destination (e.g. the remote VoIP service is marked with “ATT”), or its source (e.g. tenant LANs are marked with “Comcast”). These marks are used by the route tables. Both routers have nearly identical route tables except for minor differences related to sending traffic to the opposite router. We are using recursive routing to determine which paths are up and where traffic should be sent for each routing mark.

This is where we start to encounter our problem. RTR-002 works perfectly, no issues at all. However, on RTR-003, if I enable any of the routes targeting the “Comcast” mark that aren’t the final “give up and just use ATT” route, all of my ATT routes turn blue and become inactive, despite there being no clear relation between the two sets.

What gets me is that I’ve done this very same setup several times now without issue. I’m wondering if there’s a bug in my device or if I’m really really blind right now.

Here’s the logic:
We use three routes to test whether our links are up:
4.2.2.3 is routed through the local IP of RTR-002 to use Comcast on that router.
4.2.2.2 is routed through Comcast Gateway, on this router (only active when RTR-002 is dead)
4.2.2.1 is routed through ATT Gateway, on RTR-003

RTR-003 has ATT physically connected, not Comcast.

Primary route for all packets marked “Comcast”, with a distance of 1, uses 4.2.2.3 as its gateway so that the traffic is sent to RTR-002

The secondary route, with a distance of 3, uses 4.2.2.2 as its gateway so that the traffic uses Comcast on this router if available.

The tertiary, and last, route uses 4.2.2.1 as its gateway. This is a final resort to send the traffic marked for Comcast over this router’s ATT connection.

ATT is set up in a similar manner just with the gateways in a different order. However, enabling the Primary and Secondary routes for “Comcast” traffic on RTR-003 kills all of my routes marked for ATT on RTR-003 despite no obvious reason.

VRRP address = 192.168.1.1
RTR-002 address = 192.168.1.2
RTR-003 address = 192.168.1.3
Comcast WAN gateway is being presented as 1.1.1.1 for security purposes
ATT WAN gateway is being presented as 2.2.2.2 for security purposes

RTR-002
/ip route export

# mar/30/2018 13:03:20 by RouterOS 6.41.2
[RTR-002] /ip route> export
/ip route
add comment="Primary Route for \"ATT\"" distance=1 gateway=4.2.2.3 routing-mark=ATT
add comment="Secondary Route for \"ATT\"" distance=2 gateway=4.2.2.1 routing-mark=ATT
add comment="Tertiary Route for \"ATT\"" distance=3 gateway=4.2.2.2 routing-mark=ATT
add comment="Primary Route for \"Comcast\" Mark" distance=1 gateway=4.2.2.2 routing-mark=Comcast
add comment="Tertiary Route for \"Comcast\"" distance=3 gateway=4.2.2.3 routing-mark=Comcast
add comment="Secondary Route for \"Comcast\" Mark" distance=4 gateway=4.2.2.1 routing-mark=Comcast
add comment="Default Route using Comcast" distance=1 gateway=1.1.1.1
add comment="Default Route using ATT" distance=2 gateway=2.2.2.2
add comment="Default Route using ATT through Secondary Router" distance=3 gateway=192.168.1.3
add comment="Recursive Route for ATT" distance=1 dst-address=4.2.2.1/32 gateway=2.2.2.2 scope=10
add comment="Recursive Route for Comcast" distance=1 dst-address=4.2.2.2/32 gateway=1.1.1.1 scope=10
add comment="Recursive Route for ATT through Second Router" distance=1 dst-address=4.2.2.3/32 gateway=192.168.1.3 scope=10

/ip route print

[RTR-002] /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  ;;; Primary Route for "ATT"
        0.0.0.0/0                          4.2.2.3                   1
 1   S  ;;; Secondary Route for "ATT"
        0.0.0.0/0                          4.2.2.1                   2
 2   S  ;;; Tertiary Route for "ATT"
        0.0.0.0/0                          4.2.2.2                   3
 3 A S  ;;; Primary Route for "Comcast" Mark
        0.0.0.0/0                          4.2.2.2                   1
 4   S  ;;; Tertiary Route for "Comcast"
        0.0.0.0/0                          4.2.2.3                   3
 5   S  ;;; Secondary Route for "Comcast" Mark
        0.0.0.0/0                          4.2.2.1                   4
 6 A S  ;;; Default Route using Comcast
        0.0.0.0/0                          1.1.1.1               1
 7   S  ;;; Default Route using ATT
        0.0.0.0/0                          2.2.2.2            2
 8   S  ;;; Default Route using ATT through Secondary Router
        0.0.0.0/0                          192.168.1.3               3
 9 A S  ;;; Recursive Route for ATT
        4.2.2.1/32                         2.2.2.2            1
10 A S  ;;; Recursive Route for Comcast
        4.2.2.2/32                         1.1.1.1               1
11 A S  ;;; Recursive Route for ATT through Second Router
        4.2.2.3/32                         192.168.1.3               1
12 ADC  10.10.10.0/24      10.10.10.251    Tenant VLAN               0
                                           Tenant VRRP       
13 ADC  10.20.20.0/24      10.20.20.251    Phone VLAN                0
                                           Phone VRRP        
14 ADC  10.50.50.0/24      10.50.50.251    LAN Bridge (Gue...        0
                                           Guest VRRP        
15 ADC  10.100.100.0/24    10.100.100.251  Public Access VLAN        0
                                           Public Access VRRP
16 ADC  2.2.2.2/30  2.2.2.0  ether10                   0
17 ADC  1.1.1.1/30     1.1.1.0     ether11                   0
18 ADC  192.168.1.0/24     192.168.1.2     CLIENT VLAN                  0
                                           CLIENT VRRP

RTR-003
/ip route export

[RTR-003] > /ip route export
# mar/30/2018 13:02:35 by RouterOS 6.41.2
/ip route
add comment="Primary route for \"ATT\"" distance=1 gateway=4.2.2.1 routing-mark=ATT
add comment="Tertiary route for \"ATT\"" distance=3 gateway=4.2.2.3 routing-mark=ATT
add comment="Secondary route for \"ATT\"" distance=4 gateway=4.2.2.2 routing-mark=ATT
add comment="Tertiary Route for \"Comcast\"" distance=4 gateway=4.2.2.1 routing-mark=Comcast
add comment="Secondary Route for \"Comcast\"" disabled=yes distance=3 gateway=4.2.2.2 routing-mark=Comcast
add comment="Primary Route for \"Comcast\"" disabled=yes distance=1 gateway=4.2.2.3 routing-mark=Comcast
add comment="Default Route using ATT" distance=1 gateway=2.2.2.2
add comment="Default Route Using Comcast through Primary Router" distance=2 gateway=192.168.1.2
add comment="Default Route Using Comcast" distance=3 gateway=1.1.1.1
add comment="Recursive Route for ATT" distance=1 dst-address=4.2.2.1/32 gateway=2.2.2.2 scope=10
add comment="Recursive Route for Comcast" distance=1 dst-address=4.2.2.2/32 gateway=1.1.1.1 scope=10
add comment="Recursive route for Comcast through Primary Router" distance=1 dst-address=4.2.2.3/32 gateway=192.168.1.2 scope=10

/ip route print

[RTR-003] > /ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  ;;; Primary route for "ATT"
        0.0.0.0/0                          4.2.2.1                   1
 1   S  ;;; Tertiary route for "ATT"
        0.0.0.0/0                          4.2.2.3                   3
 2   S  ;;; Secondary route for "ATT"
        0.0.0.0/0                          4.2.2.2                   4
 3 A S  ;;; Tertiary Route for "Comcast"
        0.0.0.0/0                          4.2.2.1                   4
 4 X S  ;;; Secondary Route for "Comcast"
        0.0.0.0/0                          4.2.2.2                   3
 5 X S  ;;; Primary Route for "Comcast"
        0.0.0.0/0                          4.2.2.3                   1
 6 A S  ;;; Default Route using ATT
        0.0.0.0/0                          2.2.2.2            1
 7   S  ;;; Default Route Using Comcast through Primary Router
        0.0.0.0/0                          192.168.1.2               2
 8   S  ;;; Default Route Using Comcast
        0.0.0.0/0                          1.1.1.1               3
 9 A S  ;;; Recursive Route for ATT
        4.2.2.1/32                         2.2.2.2            1
10 A S  ;;; Recursive Route for Comcast
        4.2.2.2/32                         1.1.1.1               1
11 A S  ;;; Recursive route for Comcast through Primary Router
        4.2.2.3/32                         192.168.1.2               1
12 ADC  10.10.10.0/24      10.10.10.252    Tenant VLAN               0
13 ADC  10.20.20.0/24      10.20.20.252    Phone VLAN                0
14 ADC  10.50.50.0/24      10.50.50.252    LAN Bridge (Gue...        0
15 ADC  10.100.100.0/24    10.100.100.252  Public Access VLAN        0
16 ADC  2.2.2.2/30  2.2.2.0  ether11                   0
17 ADC  1.1.1.1/30     1.1.1.0     ether10                   0
18 ADC  192.168.1.0/24     192.168.1.3     WJF VLAN                  0

I’ve been comparing to known working set ups (including RTR-002 as well as other client sites) and I just can’t see what’s different about RTR-003 in this situation. Hoping someone else has some insight or can point out a dumb mistake my eyes keep glossing over.

Please let me know if I need to clarify anything or explain further. I can take screenshots of the route table in Winbox as well, I would just need to spend some time in an image editor sanitizing them.

Thank you

2

To me, the only configuration-related explanation would be that the pings, used by the gateway monitors of the recursive routes to check gateway availability, would get marked with routing mark “Comcast” by mistake. As a result, they would take the routes from the Comcast set instead of the routes with no routing mark as they should. If this is true, then if you enable the primary route in the “Comcast” set, the ping to 2.2.2.2 sent by the gateway monitor of the “Recursive Route for ATT” does not take the direct local route but is sent to 192.168.1.2 via the recursive route via 4.2.2.3; if you enable the secondary route in the “Comcast” set, it is sent to 1.1.1.1 via the recursive route via 4.2.2.2 (which should deliver it, but the received response might get dropped, I don’t know how the NAT rules look like). Only the tertiary route would be harmless as it, surprisingly, would send packets to 2.2.2.2 via recursive 4.2.2.1 via 2.2.2.2.

Regardless whether I’m right or not, thank you for turning my attention to the existence of Ethernet passthrough functionality :slight_smile:

And BTW, exports and prints from the CLI are much more useful for analysis than screenshots. The fact that most people don’t realize that and publish screenshots instead doesn’t mean it is the preferred way.

3

Thank you! Your suspicion sounds very plausible, I will review deeper with a closer eye for this.