Multiple MLAG pairs

Hardware
1

Hey Community,

We plan to replace our CRS518-16XS-2XQ core pair with two CRS812-8DS-2DQ-2DDQ-RM running as an
MLAG pair. Before ordering we would like to confirm a few points.

Planned setup:

  • 2x CRS812 DDQ about 1 km apart. The MLAG peer-port is an 802.3ad bond of both 400G QSFP56-DD
    ports, with third-party 400G-LR4-10 (100G Lambda MSA) QSFP-DD modules over single-mode fibre.
  • The QSFP56 ports run at 100G (QSFP28 LR4 and 100G DAC) to two CCR2216-1G-12XS-2XQ.
  • The SFP56 ports carry XS+2733LC15D at 25G/10G towards CRS518 and CRS326 MLAG pairs.
  • Pure L2 core: bridge VLAN filtering, HW-offloaded 802.3ad bonds, L3HW disabled,
    l2mtu 9500 / mtu 9200.

Questions:

  1. MLAG. Is hardware-offloaded MLAG officially supported on the CRS812 (98DX7335)? Is an
    802.3ad bond of the two 400G QSFP56-DD ports a supported peer-port? What exactly did
    "bridge - fix MLAG for CRS8xx series switches" in 7.24.1 fix? Which is the minimum version
    you recommend for a CRS812 MLAG pair? We plan 7.24.4 stable, since the 7.23.x long-term
    branch does not seem to include that fix. Is anyone already running a CRS812 MLAG pair in
    production?
  2. Module power. What is the maximum module power per QSFP56-DD cage and per QSFP56 cage?
    A staff reply in another thread mentioned a total budget of 100 W for all optical modules.
    The datasheet gives 134 W maximum vs 81 W without attachments (about 53 W). Which figure
    applies? Our estimate is about 45 W of optics per unit: 2x 400G LR4-10 at 9-12 W,
    2x 100G LR4, and 8x SFP28.
  3. Third-party single-mode 400G optics. Which speed and FEC settings should a 400G-LR4-10 or
    400GBASE-FR4 QSFP-DD module use (400GAUI-8 host interface, RS(544,514) FEC)? Is it
    speed=400G-baseSR8-LR8 with auto-negotiation disabled? Has MikroTik tested any single-mode
    400G module (DR4, FR4 or LR4) in the CRS812?
  4. MikroTik optics roadmap. Are there plans for a MikroTik single-mode 400G QSFP-DD module
    (FR4 or LR4-10)?

Thank you! Any opinion will be deeply appreciated.

Svetomir

2

Hi,

think that you HAVE TO contact directly with MikroTik as we, the Users, could not answer that question and be 100% sure that all your needs are obeyed.

MT staff more or less follow what is going on the forum but direct question to support should dispel doubts.

3

Not production yet but we're running 2xCRS812 on 7.24.2 (not updated yet) as MLAGs for a new 4 node Ceph Cluster.
But they also only have to sync a total of 8 MACs :sweat_smile:

They're working great but they're also the first Mikrotik Mlag we're running.

Also just using a single 400G DAC as the peer port

sirbryan runs a couple of MLAGs and apparently the biggest issue is with sync of Big Mac tables

4

We run MLAG on quite a few CRS326 24S+ devices, basically running dual 10gig to each server, when updates come out we can just reboot the switch and basically loose one packet if that. 99% of the users don't even know that has happened. It works quite well.

5

Thank you all for the replies!

I was just asking here to see if anyone has already implemented something similar. We have been running several pairs of MLAG devices. And honestly, it has been a rough ride.

Specifically, one of the pairs of CRS326-24S+2Q+RM devices is constantly causing troubles because it is reflecting BUM traffic back through the sending bond. Also I have noticed that there is some problem with BPDUs between the two peers. The secondary does not see any BPDUs from the stack and believes that it is RSTP root bridge. Every attempt that I have made:

  • MLAG bond to the core router,
  • Independent uplinks on each switch with different RSTP path-cost
  • Single homed bond on the primary MLAG member

All of those fail miserably the moment a second slave interface is brought up. The strange things is that I have another pair of absolutely the same devices and it does not behave like this. I started to fear that there is some problem with the MikroTik devices as a whole and wanted to confirm.

I am starting to suspect a hardware problem with the primary MLAG peer but I do not know how I can test it and present it to the MikroTik support.

6

I would check the MT documentation, make sure the priority on both switches is the same and the MAC address (admin mac) of the bridge is NOT the same. Betcha one of those things is buggering it up.

7

Actually this is my set up to light some clarity:

mlag-pairs-overview
mlag-pairs-overview2400×1740 236 KB

Five MLAG pairs, with models and RouterOS versions. Dashed links are currently disabled. We are running on the primaries only, because bringing the secondary uplinks back caused the problems in Pair 3 and I believed it is fleet wide.

This isn't tied to one RouterOS version. We saw the same behaviour when every switch was on 7.19, and it has carried through the upgrades to 7.24.x.

  1. The Pair 3 secondary thought it was the root bridge

For about two weeks, 3B (the Pair 3 secondary) reported itself as the RSTP root:

  • its root bridge ID was its own pair ID;
  • its root path cost was 0;
  • it had no root port;
  • both ends of the peer link were designated ports.

It looked as if the primary's BPDUs were not reaching the secondary over the peer link. On every other pair, the secondary's peer link is its root port and shows the upstream pair as the designated bridge. That includes Pair 1, which uses the same model.

As long as it was in that state, any uplink I enabled on 3B looped instead of blocking. Rebooting 3B didn't fix it. It cleared only after I rebooted the primary, 3A, in the same window where I removed the mlag-id from Pair 3's uplink.

  1. Pair 3 reflects BUM when an uplink towards Core B comes up

Whenever I enable a Pair 3 uplink towards Core B (the red links), other devices log bridge RX looped packet within seconds. It happened 7 out of 7 times:

  • on both 3A and 3B;
  • with and without an mlag-id on Pair 3's side;
  • with hw=no on Core B's port.

Pairs 1 and 2 (MLAG bond ⇄ MLAG bond) have never looped. Pairs 3 and 4 have a separate bond on each switch, with RSTP choosing the active path, and those are the ones that loop.

Has anyone seen an MLAG secondary stop taking BPDUs from its primary and declare itself root, then recover only after the primary is rebooted?
What would MikroTik support want in a ticket: supout.rif files from both peers taken while it's happening, or something else?

Thanks! All ideas are most welcomed. I have been struggling with this for quite some time.

8

Your config is going to be more useful in determining any possible config bugs.

MLAG priorities are being set, correct?
And bridge priorities as well (lower = higher), with each MLAG pair having the same priority?
And just a standard LACP LAG between each pair, with each end configured identically?

I got to a point with a three-deep MLAG setup that kept going awry that I finally just connected the last two each into the upstream MLAG and it solved issues with that pair of switches. (Neither of them had anything redundant plugged in anyway.)

I think there's some hashing + broadcast bugs that I haven't quite nailed down, and that could be exacerbated the more MLAG stacks you introduce into the mix. For example, I have one ESXi host connected to a 2004, and the 2004 is LAG'd to a 326-24S+ MLAG pair. On the other side is a LAG to a 520 with a storage server hanging off one of the 100G ports. No matter what I try, and just from that ESXi host's IP address to the one RDS, it seems initial ARPs don't make it from the hypervisor to the storage server. If I initiate a ping from the other direction, it's fine and works until ARP times out. Pulling out one of the LAGs from the 2004 into the CRS326 stack solves the problem.

9

Besides the priority and LAG checks sirbryan and gmsmstr already mentioned, I'd try to catch the actual STP state while it's broken, before taking the uplink down or rebooting anything.

Pairs 3 and 4 rely on RSTP alone to break the loop in the separate-uplink setup. Without mlag-id the uplink bonds are just regular bridge ports, so BUM coming in over the peer link gets flooded towards them and RSTP has to block the redundant L2 path. On a dual-connected MLAG bond, MLAG doesn't flood that BUM to its local MLAG bond while both links are up.

Since you've already reproduced it both with and without mlag-id on the Pair 3 side, and with a single-homed bond, the topology alone doesn't explain it, so I wouldn't change it again just yet.

For any dual-connected MLAG bond, I'd first make sure path-cost, port priority and edge match on both peers. Also check that edge isn't set to yes on the uplinks or the peer port, and that every VLAN used on the other bridge ports is tagged on the peer port on both nodes.

You already saw root-bridge-id, root-path-cost and root-port on 3B while it thought it was root, so the useful part now is catching them live on both 3A and 3B while it breaks. On the uplinks and the peer port I'd watch role, forwarding, actual-path-cost, designated-bridge-id and tx-rx-bpdu, plus mlag-state on the bridge. I'd check the matching core-side ports at the same time, and grab the same output from Pair 1 as a baseline, especially the secondary's peer port.

You could leave these running in two separate terminal windows while reproducing the problem (narrow the port "[find]" down to the uplinks and peer port, otherwise the output gets very wide):

/interface bridge monitor [find]
/interface bridge port monitor [find]

Once it's actually broken, grab a static snapshot from another terminal and save that output before you take the Core B uplink down again:

/interface bridge monitor [find] once
/interface bridge port monitor [find] once

In that output, if the TX side of tx-rx-bpdu keeps increasing on 3A's peer port but the RX side on 3B's doesn't, 3B isn't receiving or processing the BPDUs from 3A, which would fit what you saw with 3B as root. If they are arriving and designated-bridge-id on 3B's peer port looks like it does on Pair 1's secondary, but RSTP still leaves a redundant L2 path forwarding, that would make an MLAG/RSTP state issue look a lot more likely!

And before rebooting anything, grab a full export and "supout.rif" from both peers while it's still broken, especially 3A, since last time it only cleared after 3A was rebooted. That's what I'd send support, together with the monitor output, the diagram and the exact RouterOS version for each observation. 7.24 had changes to STP/BPDU handling with MLAG, so the version matters.