Damo
October 6, 2023, 6:16pm
1
I am trying to set up MLAG for my two CRS switches that are connected via SFP+24 and then all other ports be in the same VLAN and able to ping eachother. Each baremetal server has a SFP+ cable attached to each CRS (on the same port). The plan is to set up redundancy following the guide https://help.mikrotik.com/docs/display/ROS/Multi-chassis+Link+Aggregation+Group however I can’t get them to be able to ping each other running the following
On each CRS
/interface bonding
add mlag-id=10 mode=802.3ad name=client-bond slaves=sfp-sfpplus9,sfp-sfpplus10
/interface bridge
add name=bridge1 vlan-filtering=yes
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus24 pvid=99
add bridge=bridge1 interface=client-bond
/interface bridge vlan
add bridge=bridge1 tagged=sfp-sfpplus24 vlan-ids=1
/interface bridge mlag
set bridge=bridge1 peer-port=sfp-sfpplus24
Inside bare metal
auto enp133s0f0
iface enp133s0f0 inet manual
auto enp133s0f1
iface enp133s0f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp133s0f0 enp133s0f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet static
address 192.168.0.100/24
gateway 192.168.0.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
What looks wrong? I am fairly new to networking in general apologies.
mkx
October 6, 2023, 7:15pm
2
add mlag-id=10 mode=802.3ad name=client-bond slaves=sfp-sfpplus9,sfp-sfpplus10
Aren’t the mentioned sfp-sfpplus ports used to connect two different servers? You need two MLAG bonds (with different MLAG IDs), one for sfp-sfpplus9 and the other one for sfp-sfpplus10.
Damo
October 6, 2023, 8:11pm
3
Ok so I replace that with
/interface bonding
add mlag-id=10 mode=802.3ad name=client-bond9 slaves=sfp-sfpplus9
add mlag-id=11 mode=802.3ad name=client-bond10 slaves=sfp-sfpplus10
Each bond needs unique name, still they can’t ping.
Damo
October 6, 2023, 8:47pm
4
Ok so I replace that with
/interface bonding
add mlag-id=10 mode=802.3ad name=client-bond9 slaves=sfp-sfpplus9
add mlag-id=11 mode=802.3ad name=client-bond10 slaves=sfp-sfpplus10
Each bond needs unique name, still they can’t ping.
I think it’s working now. I also had to run
/interface bridge port
add bridge=bridge1 interface=client-bond9
add bridge=bridge1 interface=client-bond10
That looks OK to you now?
Damo
October 6, 2023, 9:23pm
5
As far as I can see it’s working. The final config is
/interface bonding
add mlag-id=10 mode=802.3ad name=client-bond9 slaves=sfp-sfpplus9
add mlag-id=11 mode=802.3ad name=client-bond10 slaves=sfp-sfpplus10
/interface bridge
add name=bridgeLAG vlan-filtering=yes
/interface bridge port
add bridge=bridgeLAG interface=sfp-sfpplus24 pvid=99
add bridge=bridgeLAG interface=client-bond9
add bridge=bridgeLAG interface=client-bond10
/interface bridge vlan
add bridge=bridgeLAG tagged=sfp-sfpplus24 vlan-ids=1
/interface bridge mlag
set bridge=bridgeLAG peer-port=sfp-sfpplus24
But speed is low, only 300Mib/s and CPU is hitting 100% on one of the switches (active one). Both switches are CRS326-24S+2Q+RM
mkx
October 7, 2023, 12:32pm
6
Is the bridgeLAG the only bridge on switch? ROS can offload only one bridge to hardware.
Damo
October 7, 2023, 3:41pm
7
No it’s not, so you think if I add everything to bridgeLAG it will be OK?
[admin@MikroTik] /interface/bridge/port> print
Flags: X, I - INACTIVE; H - HW-OFFLOAD
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, PATH-COST, INTERNAL-PATH-COST, HORIZON
# INTERFACE BRIDGE HW PVID PRIORITY PATH-COST INTERNAL-PATH-COST HORIZON
;;; defconf
0 IH ether1 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
1 IH qsfpplus1-1 bridgeLocal yes 1 0x80 10 10 none 10 none
;;; defconf
2 IH qsfpplus1-2 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
3 IH qsfpplus1-3 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
4 IH qsfpplus1-4 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
5 IH qsfpplus2-1 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
6 IH qsfpplus2-2 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
7 IH qsfpplus2-3 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
8 IH qsfpplus2-4 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
9 H sfp-sfpplus1 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
10 IH sfp-sfpplus2 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
11 H sfp-sfpplus3 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
12 H sfp-sfpplus4 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
13 H sfp-sfpplus5 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
14 H sfp-sfpplus6 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
15 IH sfp-sfpplus7 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
16 H sfp-sfpplus8 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
17 X sfp-sfpplus9 bridgeLocal 1 0x80 10 10 none
;;; defconf
18 H sfp-sfpplus10 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
19 H sfp-sfpplus11 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
20 IH sfp-sfpplus12 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
21 H sfp-sfpplus13 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
22 H sfp-sfpplus14 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
23 H sfp-sfpplus15 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
24 H sfp-sfpplus16 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
25 X sfp-sfpplus17 bridgeLocal 1 0x80 10 10 none
;;; defconf
26 IH sfp-sfpplus18 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
27 IH sfp-sfpplus19 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
28 IH sfp-sfpplus20 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
29 IH sfp-sfpplus21 bridgeLocal yes 1 0x80 10 10 none
;;; defconf
30 IH sfp-sfpplus22 bridgeLocal yes 1 0x80 10 10 none
31 sfp-sfpplus24 bridgeLAG yes 99 0x80 10 10 none
32 I client-bond9 bridgeLAG yes 1 0x80 10 10 none
33 I client-bond17 bridgeLAG yes 1 0x80 10 10 none
Damo
October 8, 2023, 8:53am
9
Great, it seems to be working but I am having what looks like path routing issues, some connections are using the 2nd switch and some on the first. I think it’s causing packet loss maybe with the way my layout is set up.
Can I set CRS #1 to be used as priority and only CRS #2 if CRS #1 goes down?
mkx
October 8, 2023, 10:40am
10
From connected device point of view, connections to both MLAG-enabled switches look just like normal LACP (bonding) does. And vice versa. If you wanted to prioritize one of switches, then you have to use LACP mode which supports it.
But then, you may want to rethink your MLAG strategy and perhaps switch over to using RSTP/MSTP to use switches in an active/standby manner.
Damo
October 10, 2023, 8:38pm
11
From connected device point of view, connections to both MLAG-enabled switches look just like normal LACP (bonding) does. And vice versa. If you wanted to prioritize one of switches, then you have to use LACP mode which supports it.
But then, you may want to rethink your MLAG strategy and perhaps switch over to using RSTP/MSTP to use switches in an active/standby manner.
I think the issue is the root port keeps changing. On the CRS #1 port 24 is is being used as root port which sometimes makes me not able to ping CRS #2 .
Should SFP+1 be set to root port on both the CRS? How am I able to set that
