Two interconnected MLAG pairs on CRS326: which RouterOS versions are stable?

Hello,

Is anyone running two interconnected MLAG pairs in production?

Our setup consists of four CRS326-24S+2Q+RM switches:

           MLAG1
Primary-N0  =====  Primary-N1
    |                 |

    | 802.3ad MLAG    |

    |   interconnect  |

    |                 |
Secondary-N0 ===== Secondary-N1
           MLAG2

Both MLAG pairs operate in the same Layer 2 domain.

We have noticed significant differences between RouterOS versions. RouterOS 7.25.1 fixed the BPDU/edge-port issue introduced in 7.24, but an MLAG member state change can still cause high CPU usage, FDB reconstruction and severe packet loss.

I would like to compare production experience:

- switch models;

- currently used RouterOS version;

- last known stable version;

- versions that caused problems;

- behaviour during link failure, reconnection and switch reboot;

- method used to interconnect both MLAG pairs.

Some older reports identify RouterOS 7.6 as stable and versions 7.7-7.9 as problematic in four-switch MLAG environments, but I would like to know how current RouterOS releases behave in similar production networks.

Thank you.

Yeah, the 7.25.1 changelog confirms a fix for the BPDU issue you saw.

To cut the packet loss when the primary reboots, try "mlag-lacp-fallback=no" (new in 7.25) so the secondary keeps the shared system ID and the bond to the other pair doesn’t have to renegotiate. Also worth checking that the interconnect bond isn't "edge=yes" and that bridge priority and STP port settings match within each pair.

The CPU load or FDB rebuild on member state change isn't mentioned in any changelog AFAIK, and it could also be the CRS326 itself since it has a pretty weak single-core CPU. How many hosts and VLANs are you running? Also check what "/tool profile" shows during a member state change. Either way it's worth a ticket with supout.rif from the affected switches while it happens.

Related thread: Multiple MLAG pairs

Thank you, this is a useful suggestion.

All four MLAG bridges currently have:

mlag-lacp-fallback=yes

mlag-init-delay=1m

We will test mlag-lacp-fallback=no on all four switches during the maintenance window on Saturday.

Our issue is not limited to a primary reboot. It can also be triggered by disabling or enabling one member interface while both MLAG peers remain operational.

The environment is relatively small:

- 4x CRS326-24S+2Q+RM;

- approximately 360-500 FDB entries;

- approximately 11 VLANs;

- no large VLAN ranges;

- L3 hardware offloading disabled;

- bridge hardware offload active.

During one controlled test, the FDB count changed as follows:

499 -> 493 -> 417 -> 395 -> 84

Unrelated MAC entries moved to the MLAG peer-port, one peer reached 100% CPU, and the entire Layer 2 domain experienced severe packet loss.

After upgrading to RouterOS 7.25.1, AE1_Interconnect was initially correctly detected as non-edge on all four nodes. However, after multiple member link disable/enable operations, the interconnect on one node was again operationally detected as edge=yes, although its configured value remained edge=auto.

This was followed by severe packet loss and network instability.

We have stopped further testing during production hours. During the maintenance window on Saturday, we will:

- test mlag-lacp-fallback=no on all four switches;

- reproduce the issue using controlled member link changes and switch reboots;

- collect /tool profile output;

- monitor bridge, RSTP, LACP and FDB state on both peers;

- generate supout.rif files while the issue is active.

At this point, we cannot confirm whether mlag-lacp-fallback is related to the incorrect edge-port state or whether these are two separate symptoms.

I will report back after the Saturday maintenance window.

UPDATE

Additional historical information:

A similar stability problem was also present on an earlier RouterOS release 7.24 or 7.25, but the behaviour was different.

On the earlier version, CPU usage increased to approximately 40% on all four switches. None of the switches reached 100% CPU, but RSTP appeared to be continuously reconverging and the entire Layer 2 domain experienced severe instability and packet loss.

Interestingly, disabling and enabling a seemingly random aggregation interface would sometimes restore normal operation.

At that time, the problem did not look like CPU exhaustion. It appeared more closely related to RSTP state, forwarding behaviour or MAC learning across the MLAG topology.

On RouterOS 7.25.1, the BPDU and edge-port behaviour initially improved, but the failure now appears more CPU-intensive: one MLAG peer can reach 100% CPU while the FDB is globally flushed and rebuilt.

We do not yet know whether this is the same underlying MLAG reconvergence issue presenting differently between RouterOS versions, or two separate problems.

To clarify, this is not a temporary CPU spike during FDB reconstruction. Once the failure is triggered, one MLAG peer reaches 100% CPU and remains at 100%, while the network stays severely degraded. Flapping an aggregation interface may restore normal operation, but this is inconsistent and does not always work.

With 360-500 FDB entries and 11 VLANs you're way below the max host recommendation, so forget my weak CPU theory. And since a member flap with both peers up also triggers it, "mlag-lacp-fallback=no" probably won't help that case.

The interconnect turning edge is the interesting part IMO. AFAIK "edge=auto" makes a port edge when it stops receiving BPDUs, and an edge port between two MLAG pairs can loop, which would fit the FDB flush and the stuck CPU. I'd set "edge=no" on AE1_Interconnect on all four switches and watch it while you flap members:

/interface/bridge/port/monitor [find interface=AE1_Interconnect]

On the pair that receives BPDUs on the interconnect, the rx part of tx-rx-bpdu should keep growing on at least one of the two nodes. If it stops after a flap the BPDUs are getting lost. Problem gone with "edge=no" points at the edge detection, problem still there points at the BPDU loss itself. That's a bug for support, and 7.23.8 long-term might be worth a try meanwhile since the BPDU issue was introduced in 7.24. Just be aware that "mlag-init-delay" and "mlag-lacp-fallback" don't exist before 7.25.

Btw, please post back if you get it sorted and how you did it, so others with the same issue can benefit. Thanks!

Quick update:

We have not performed the planned tests yet because the customer has not provided a maintenance window.

The environment is currently running with one interconnect link disconnected and remains stable. As previously mentioned, we are avoiding interface disable/enable operations for now.

VLAN additions and other routine configuration changes are being made without any issues.

When we get a maintenance window, I will perform the planned MLAG tests and share the results. We may also wait for a newer RouterOS release before continuing the investigation.