MLAG client LACP failover handling

RouterOS General
1

I'm preparing for setting up MLAG in my system and have been using containerlab with CHR v7.24rc3 to test things out. I've followed the MLAG setup guide: Multi-chassis Link Aggregation Group - RouterOS - MikroTik Documentation
My test setup is:

  • Two switches with a single peer link (I will be using LACP once I've gotten the basics figured out).
  • Two Linux clients with LACP where eth2 goes to switch 1 and eth3 goes to switch 2.

If I do a ping series from client 2 to client 1, and on client 1 take eth3 down:

  • The pings no longer reach between clients.
  • I can see the packets reach switch 1 peer link, but not the bond.
  • Switch 2 still shows client 1 MAC to be on the bond interface instead of the peer link.
  • Switch 2 shows the client 1 bond as: key=15 flags="ATGS--F-"

If I instead take down the port on switch 2 side instead of on the client:

  • Pings continue working
  • Switch 2 shows client 1 MAC to be on the peer link interface.
  • Switch 2 shows the client bond as: key=0 flags="ATGSCD--" partner-sys-id=AA:C1:AB:5C:7F:5B partner-sys-priority=65535 partner-key=15 partner-flags="ATGSCD--"

In both cases client 1 LACP status shows LACP for eth3 as down, so that side seems to be working fine.
But the switch seems to not really fail over correctly on the MLAG when the state is "ATGS--F-" which should indicate failed. Instead it only seems to react to "key=0" (or something else that I don't know where to look for).

My question is: Is this intentional, an issue/limitation with CHR in containerlab, a configuration issue, or a bug?

I'll continue digging, but before spending lots of time digging deeper I would just want to know if this is expected or not. I'm hoping it's not by design, because I see a lot of cases where MLAG is much worse than regular LACP since it seems to then require a physical link failure and everything else will just fail traffic.

2

Might be something specific to the CHR. That said, worth posting the configuration to check if there's anything in there that needs tweaking.

I've gotten MLAG configurations working between switches just fine, but there may be more tuning required when connected machines over LACP.

3

Sure, here are the configs.

switch1-config.rsc (1.3 KB)

switch2-config.rsc (1.2 KB)

4

After reading MLAG Support On CHR? it seems like it may well be a limitation when using CHR. Although my setup does work partially at least, so better than what's discussed in that thread.

That's too bad though, because with MLAG being so historically finicky and buggy it would have been great to test and compare using a virtual lab. This would kind of "force" me to get a set of test switches I think.

5

I hadn't seen that thread - but it doesn't look promising for the CHR approach. At a glance the configurations look reasonable. The syntax I use is a little different (I'm running RouterOS 7), but appears to be expressing the same things, notably:

/interface bridge
add frame-types=admit-only-vlan-tagged igmp-snooping=yes mlag-peer-port=\
    ether6 mlag-priority=50 name=bridge priority=0x1000 vlan-filtering=yes

vs

/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge1 priority=0x1000 \
    pvid=4094 vlan-filtering=yes

/interface bridge mlag
set bridge=bridge1 peer-port=ether6 priority=50

Are you running RouterOS 6 or 7? I'd forgotten, but I never got it working reliably under 6 but when I retried under 7 it started working.

6

I'm currently testing on v7.24rc3

7

Hmm - that does make the syntax differences rather curious

8

Which version are you on? Because apparently there were breaking MLAG syntax changes in v7.22: Multi-chassis Link Aggregation Group - RouterOS - MikroTik Documentation

I also see that CHR should have gotten MLAG support in that version. Hmm..

9

Ah - that explains the syntax variance then. I was going from some older reference configuration backup flles I had kicking around.

10

Since CHR MLAG support was added in 7.22 I'll try to downgrade to that version.

But can anyone confirm if they have gotten MLAG to work with edge LACP where a link fails in a way that is not seen as complete link loss on the switch side (but the client actually fails the LACP over to just one link)?

11

Same in v7.22.3. Failing on the server side with LACP status:
A port=ether2 key=15 flags="ATGS--F-"
Which is clearly failed, traffic does not switch to pass over the MLAG. But, setting interface disabled on the switch gives LACP status:
port=ether2 key=0 flags="ATGS--F-"
Traffic does start to flow over the MLAG.

12

On my end I've only tested on switch to switch LACP connections. So there may be something specific to the Linux bonding configuration that it doesn't like.

13

I have probably a dozen MLAG setups with switches, all running 7.19.x (4 or 6). So it works and is pretty robust on the switches.

I've seen random oddities with switches on newer releases, so I've held off upgrading those; I haven't tested using CHR's as MLAG pairs/stacks.