I came across various entries looking at the firewall -> connections tab related to an internal Ubiquiti device eg. 192.168.11.5 and the CMR point (192.168.110.253). How come that the external router is even aware of this traffic?

Doing a torch on the incoming port of the central MT1 is not showing any traffic from the management device 192.168.110.253

The IP Addresses shown by the firewall connections are indeed Ubiquiti devices that are managed by the CRM point.
However, these devices are all situated behind either MT3, MT4 or MT5.
So I would assume that this traffic flows via MT2 (Central router) towards MT3,4 or 5, and not towards the border router MT1.

The IP Addresses shown by the firewall connections are indeed Ubiquiti devices that are managed by the CRM point.
However, these devices are all situated behind either MT3, MT4 or MT5.
So I would assume that this traffic flows via MT2 (Central router) towards MT3,4 or 5, and not towards the border router MT1.

Yet somehow they're getting marked as ISP connections in your mangle rules...

At this point I think it would be helpful to start sharing your routing tables as I feel we might get an answer there. Also, confirm gateway of CRM is actually referencing MT2 to begin with.

• Traffic originating from 192.168.110.253 destined for 172.18.3.4 is coming in on MT2:eth10|192.168.110.1/24.
• One would expect traffic destined to 172.18.3.x to leave MT2:eth2|192.168.2.9/29 to MT3:eth5|192.168.2.14/29 as per routing table entry for subnet 172.18.2.0/23.
• Instead it shows up in MT1 firewall thus leaving MT2:eth1|192.168.2.6/29 towards MT1:eth?|192.168.2.1/29
• According to routing table of MT2, MT1 however would normally only route subnets 192.168.[11-14].0/24 as well as the default route 0.0.0.0/0
• In MT1, this traffic this then being marked as internet bound traffic given the ISP1_conn, ISP2_conn and ISP3_conn marks?
• Nevertheless, traffic eventually arrives through MT3 allowing control of the access points.

• Is it just traffic from the CMR and/or to this subnet where you're seeing this behaviour or is it more wide-spread?
• Is there any filtering going on at MT2 that would prevent a direct route to MT3 without looping through MT1?
• I see some VLAN and bridge configurations. No strange loops introduced here?
• Wild guess: Ubiquiti discovery is taking place through L2 UDP broadcast 255.255.255.255:10001 / multicast 233.89.188.1:10001. Could it be that things are going "wrong" here, i.e. untagged multicast traffic traveling through MT1 as rendezvouz point?

Apologies for the late reply; it has been a busy week.

Just to make sure I have established the correct picture of your situation (as it's not your everyday SOHO setup you have running there):

• Traffic originating from 192.168.110.253 destined for 172.18.3.4 is coming in on MT2:eth10|192.168.110.1/24.
• One would expect traffic destined to 172.18.3.x to leave MT2:eth2|192.168.2.9/29 to MT3:eth5|192.168.2.14/29 as per routing table entry for subnet 172.18.2.0/23.
• Instead it shows up in MT1 firewall thus leaving MT2:eth1|192.168.2.6/29 towards MT1:eth?|192.168.2.1/29
• According to routing table of MT2, MT1 however would normally only route subnets 192.168.[11-14].0/24 as well as the default route 0.0.0.0/0
• In MT1, this traffic this then being marked as internet bound traffic given the ISP1_conn, ISP2_conn and ISP3_conn marks?
• Nevertheless, traffic eventually arrives through MT3 allowing control of the access points.

Indeed, not immediately obvious where things are going wrong.

Some questions:

• Is it just traffic from the CMR and/or to this subnet where you're seeing this behaviour or is it more wide-spread?
• Is there any filtering going on at MT2 that would prevent a direct route to MT3 without looping through MT1?
• I see some VLAN and bridge configurations. No strange loops introduced here?
• Wild guess: Ubiquiti discovery is taking place through L2 UDP broadcast 255.255.255.255:10001 / multicast 233.89.188.1:10001. Could it be that things are going "wrong" here, i.e. untagged multicast traffic traveling through MT1 as rendezvouz point?

add dst-address=10.0.0.0/8 type=unreachable
add dst-address=172.16.0.0/12 type=unreachable
add dst-address=192.168.0.0/16 type=unreachable

During the writing of the post I did check again and found NOTHING of this behaviour anymore!!!

For lowering the load on the CPU and block useless traffic I added unreachable routes to my routing table (I started with blackholes but found out later that unreachables were better in terms of trouble shooting).

The routing tables on my routers are distilled from this, the OSPF information and directly connected networks.
Could it be that sometimes the OSPF routes get ditched for a short amount of time, and that this caused the traffic to end up at MT1?
Because before I added the unreachables, when an OSPF neighbor goes down, the 0.0.0.0/0 towards MT1 would have taken over the route for the 172.18.2.0/23 network and traffic towards the 172.18.3.x clients would endup on MT1.
Now since I have the unreachables, when an OSPF goes down, the network goes to unreachable., right?

During the writing of the post I did check again and found NOTHING of this behaviour anymore!!!

OK, that's good as a result, less so as still being unexplained.

For lowering the load on the CPU and block useless traffic I added unreachable routes to my routing table (I started with blackholes but found out later that unreachables were better in terms of trouble shooting).

So as shown in your post above, these are statically configured on all the individual routers yes?

The routing tables on my routers are distilled from this, the OSPF information and directly connected networks.
Could it be that sometimes the OSPF routes get ditched for a short amount of time, and that this caused the traffic to end up at MT1?
Because before I added the unreachables, when an OSPF neighbor goes down, the 0.0.0.0/0 towards MT1 would have taken over the route for the 172.18.2.0/23 network and traffic towards the 172.18.3.x clients would endup on MT1.
Now since I have the unreachables, when an OSPF goes down, the network goes to unreachable., right?

Sure, if it loses the more specific routing information it'll take the default route. But assuming these blackholes were statically configured at MT2, even without the OSPF routing information the packets would still be discarded at MT2.

As for the ditching of OSPF routes: how often does that connection go down? OSPF has quite fast convergence, so the impact would only be minimal (almost: coincidental). When I look at the age of the connections shown in one of your previous posts, it's not a single occasion (e.g.: we see connections timing out 17 hours, but also 19, 23 and 14).