We are curently evaluating two switch models (crs520, crs354) for use in our data centers.
This is about how MLAG behaves under failure conditions, here: How is failover handled in case the primary MLAG suddenly stops working?
For this I set up an MLAG connection between two Servers and a pair of identical switches and started transfering data between the servers.
Then I remove power supply from the primary MLAG partner. This results in a connection loss between the servers. It takes 2 minutes for the connection to be re-established. This seems to be an excessive amount of time. I would have expected something on the order of a few seconds at most.
I would like to know:
Has anybody tested the behaviour when the primary MLAG peer fails?
If so, what was your observation?
If you have a significantly shorter failover time: What is your configuration?
The Details of the setup:
I did two tests with a pair of switches as MLAG peers. One test with 2 CRS354 and a second one with 2 CRS520. Both exhibited the same behaviour.
The setup follows your documentation ( Multi-chassis Link Aggregation Group | RouterOS Manual ).
The server's connections to the switches are bond-srv1 and bond-srv2 respectively.
The switch peers are interconnected via an LACP bond called bond-interswitch.
Setting the bridge ports attached to the server to edge=yes makes a lot of sense. Otherwise RSTP is going to have to confirm that nothing on those ports is part of a forwarding domain and it may take a while to confirm that negative response.
I've had good experiences with MLAG between pairs of switches failing over pretty much transparently, but all of the issues I've seen in the forums have been when connected to servers.
Likely nothing new here, since my colleagues and I and several others on the forum have hit the same issue over the past year.
To me this is a pretty serious design flaw in ROS MLAG failover. When the primary disappears, the secondary falls back to its own LACP System ID, so the server may see a different LACP system and rebuild the aggregator instead of simply continuing on the surviving member. That pretty much defeats transparent MLAG failover.
As a workaround, try the same static lacp-system-id on both peers and set the server-facing MLAG bonds as edge ports.
If this is Linux bonding in 802.3ad mode, the long recovery might be aggravated by the default lacp_rate=slow and its 90-second receive timeout.
7.25 beta adds "mlag-lacp-fallback" and "mlag-init-delay", but MikroTik hasn’t documented what they actually do, so we don’t even know if these are intended to address this problem. That makes proper testing pretty much pointless.
I’d definitely open a support ticket and suggest that MikroTik keep the virtual MLAG LACP System ID stable during a normal node failure and handle split-brain separately. We’ve reported related MLAG issues too, but MikroTik still doesn’t seem to fully recognize it, so more reproducible reports would help.
Once fixed, IMO this really should be backported to the current long-term 7.23.x branch.
I would have one related question - ROS documentation specifically says that the ICCP uses untagged L2 frames. The older example for MLAG doesn't use the "admit-only-vlan-tagged" parameter.... so I would be afraid that enabling only tagged traffic on the peer port might affect he ICCP....but the current example says something different...
Thanks, everybody, especially @yasir_pk for all the input. I was able to get the failover time down significantly. Here are the details:
Server 1 runs Debian. LACP rate: fast was already set.
Server 2 runs OpenBSD where I didn't set lacptimeout fast (the equivalent to LACP rate). Once I did, failover time dropped to something between 5 and 8 seconds. So, that was the culprit. When the primary comes back there is a short interuption of 1 to 3 seconds.
I checked interruptions by running netstat -I aggr0 -i -w 1 on the server 2 while doing an scp from server 1 to server 2.
But before I did the above I checked if setting edge=yes would affect the failover time. I did not notice any difference.
I had also tested the behavior when powering down the secondary MLAG peer: This worked already without any hickup before I fixed the lacptimeout setting.
BTW: I did open a ticket a week before I posted here. I got an answer to that roughly the same time the first answer popped up here, and it also pointed me to the LACP timings on the servers.
So far it looks like MLAG is handling this failure fine. (I tested this only on the crs520s.)