Hello,
I have 2 x crs510-8xs-2xq that are linked via 2 x 40Gb bonded interfaces (qsfp28-1-1 & qsfp28-2-1). Ports sfp28-1, sfp28-2, sfp28-3 on each switch connect to a corresponding server (clustered servers). The interfaces on each server are teamed within the operating system. I am trying to replicate the configuration as per mikrotik.com documentation.
I currently have the configuration working as expected, and failure testing of NICs, switches, servers appears successful. My concern is that it only works correctly when I add an mlag-id to the bonded ICCP interface between the two switches, and I'm not sure this is correct. It's certainly not documentated that way.
When I have the mlag-id defined on the bonded interface, a message appears on the bridge port for that interface that says "mlag not connected".
I fear I have something fundamently wrong that allows the system to work correctly.
/interface bonding
add lacp-rate=1sec mode=802.3ad mtu=9000 name=TOR-Bond slaves=qsfp28-1-1,qsfp28-2-1 transmit-hash-policy=layer-3-and-4
add lacp-rate=1sec mlag-id=111 mode=802.3ad mtu=9000 name=server1 slaves=sfp28-1 transmit-hash-policy=layer-3-and-4
add lacp-rate=1sec mlag-id=112 mode=802.3ad mtu=9000 name=server2 slaves=sfp28-2 transmit-hash-policy=layer-3-and-4
add lacp-rate=1sec mlag-id=113 mode=802.3ad mtu=9000 name=server3 slaves=sfp28-3 transmit-hash-policy=layer-3-and-4
/interface bridge
add mlag-peer-port=TOR-Bond mlag-priority=50 name=bridge vlan-filtering=yes
/interface bridge port
add bridge=bridge interface=TOR-Bond pvid=99
add bridge=bridge interface=server1
add bridge=bridge interface=server2
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-8
add bridge=bridge interface=server3
/interface bridge vlan
add bridge=bridge tagged=sfp28-8,TOR-Bond,server1,server2,server3 vlan-ids=10
add bridge=bridge tagged=sfp28-8,server1,server2,server3,TOR-Bond vlan-ids=1
add bridge=bridge tagged=sfp28-8,server1,server2,server3,TOR-Bond vlan-ids=5
Any guidance would be appreciated.
Cheers,
John
The MLAG ID is required on cross-switch bonds and must be the same on both switches
Here's the most stripped down version of an MLAG configuration that I got working:
/interface bridge
add frame-types=admit-only-vlan-tagged mtu=10218 name=bridge1 priority=0x1000 \
pvid=667 vlan-filtering=yes
/interface bonding
add lacp-rate=1sec mlag-id=1 mode=802.3ad name=BOND-LAN slaves=sfp-sfpplus1
add lacp-rate=1sec mode=802.3ad name=INTERCO slaves=qsfpplus1-1,qsfpplus2-1
/interface bridge mlag
set bridge=bridge1 peer-port=INTERCO priority=50
/interface bridge port
add bridge=bridge1 interface=INTERCO pvid=667
add bridge=bridge1 interface=BOND-USR pvid=50
/interface bridge vlan
add bridge=bridge1 comment=MANAGEMENT tagged=INTERCO vlan-ids=101
add bridge=bridge1 comment=WIFI tagged=INTERCO vlan-ids=102
add bridge=bridge1 comment=DMZ tagged=INTERCO vlan-ids=103
add bridge=bridge1 comment=LAN tagged=INTERCO vlan-ids=50
Where the sfp-sfpplus1 on each switch is connected to a third party switch with LACP configured.
Thanks InfraErik, I've been able to get it successfully working. Your example reinforced what I thought was the correct way to do it.
[admin@swea-tor-1] > /interface/bridge/monitor [find name=bridge]
state: enabled
current-mac-address: D0:EA:11:60:58:70
bridge-id: 0x8000.D0:EA:11:60:58:70
root-bridge: no
root-bridge-id: 0x1000.64:9D:99:D1:D8:31
root-path-cost: 500
root-port: TOR-Bond
port-count: 5
designated-port-count: 3
fast-forward: no
mlag-state: connected
mlag-active-role: primary
[admin@swea-tor-2] > /interface/bridge/monitor [find name=bridge]
state: enabled
current-mac-address: D0:EA:11:60:58:70
bridge-id: 0x8000.D0:EA:11:60:58:70
root-bridge: no
root-bridge-id: 0x1000.64:9D:99:D1:D8:31
root-path-cost: 800
root-port: sfp28-8
port-count: 5
designated-port-count: 4
fast-forward: no
mlag-state: connected
mlag-active-role: secondary
The root cause of my issue was that I was using inter-switch bonds which Windows did not like. I removed these mlag bonds and let Windows handle the connections through SET process. Initial testing of shutting down interfaces, rebooting the switches, etc has been successful. Before doing this I believe the mlag bonds on the mikrotiks were working as they should have, they are just not required for a Windows failover cluster.
Good to hear! I remember trying to get my first MLAG setups working and I thought I was going crazy following the documentation, but not getting expected results.
I was switching between going crazy and having a stroke with this MLAG stuff.
I went to our other network vendor, there it is one (1)!!!!! click and a reboot and stacking works. I would never ever do this with MT (unless MT implements something like other vendors do).
Thank you for your helpful and constructive post.
Meanwhile, there are plenty of us with dozens of MikroTik MLAG stacks that are working fine, and once you get the hang of configuration, subsequent stacks are easier to set up.
RouterOS is like a Swiss army knife: a tool for any job, but also easy to cut yourself if you don't use it correctly.
