MLAG not working with one upstream

RouterOS RouterOS beta
1

Hello, we are testing MLAG configuration on two CRS305-1G-4S+ and single client (hap lite).
All devices are running RouterOS v7.1.1.
testing_lab.png
If we are running it like on the picture, ping from upstream router will stop after few moments, only way to restore the communication is disabling half of the LACP towards client, thus forsing data through right CRS305.

Upstream router is untagged on port sfp-sfpplus3 and client is on bond0 using vlan1.

MLAG bridge is using untagged vlan777 for ICCP traffic and tagged vlan1 on bonding towards client as well as untagged vlan1 for sfp-sfpplus3.

Is there a way to force traffic through the interconnect between CRS305s ? Because this setup simulates two different upstream providers or link failure.
We have tested same setup on Arista switches with MLAG and they transfer the traffic using interconnect link.

Adding configuration of all 3 devices bellow:

Client:

/interface bonding
add mode=802.3ad name=bonding1 slaves=ether1,ether2 transmit-hash-policy=layer-3-and-4
/interface vlan
add interface=bonding1 name=vlan1 vlan-id=1
/ip dhcp-client
add interface=vlan1

Bond is up on client with active/active

interface/bonding/monitor
numbers: 0
                    mode: 802.3ad
            active-ports: ether1,ether2
          inactive-ports: 
          lacp-system-id: B8:69:F4:67:4F:C9
    lacp-system-priority: 65535
  lacp-partner-system-id: 48:8F:5A:93:91:2D

Configuration of first CRS305

/interface bridge
add name=Bridge-MLAG vlan-filtering=yes
/interface bridge mlag
set bridge=Bridge-MLAG peer-port=sfp-sfpplus4
/interface bridge port
add bridge=Bridge-MLAG interface=Po1
add bridge=Bridge-MLAG interface=sfp-sfpplus4 pvid=777
add bridge=Bridge-MLAG interface=Po2
add bridge=Bridge-MLAG interface=*18
/interface bridge vlan
add bridge=Bridge-MLAG tagged=Po1,Po2 vlan-ids=1
/interface bonding
add mlag-id=100 mode=802.3ad name=Po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-3-and-4
add mlag-id=102 mode=802.3ad name=Po2 slaves=sfp-sfpplus2

Configuration of right CRS305

/interface bridge
add name=Bridge-MLAG vlan-filtering=yes
/interface bridge mlag
set bridge=Bridge-MLAG peer-port=sfp-sfpplus4
/interface bridge port
add bridge=Bridge-MLAG interface=Po1
add bridge=Bridge-MLAG interface=sfp-sfpplus4 pvid=777
add bridge=Bridge-MLAG interface=Po2
add bridge=Bridge-MLAG interface=sfp-sfpplus3
/interface bridge vlan
add bridge=Bridge-MLAG tagged=Po1,Po2 vlan-ids=1
/interface bonding
add mlag-id=100 mode=802.3ad name=Po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-3-and-4
add mlag-id=102 mode=802.3ad name=Po2 slaves=sfp-sfpplus2

Thank you for any hints.

2

Any ideas? Doing something wrong? How to deal with uplink failure from MLAG switch pair?

3

Hi,

This seems to be a common misconfiguration. Try adding the peer-port sfp-sfpplus4 as a VLAN 1 tagged member. In order to send any VLAN traffic over the peer ports, you should include them in those VLANs as a tagged member.

We also added a warning message in our user manual, so this step hopefully does not get skipped.
https://help.mikrotik.com/docs/display/ROS/Multi-chassis+Link+Aggregation+Group#MultichassisLinkAggregationGroup-Quicksetup

4

Thank you for reply, I will test and let you know it it works.

5

Great, just added VLAN on MLAG ports as tagged and it’s working as intended.
Thank you

6

I am having a big problem with MLAG, too.

Currently, I am only working with 2x CRS326-24S+2Q that use peer bond with 2*QSFP. Client is third CRS326-24S+2Q that is connected with two SFP+ (port 1 on each of MLAG switches). All of those are connected on CRS326-24G-2S+RM that acts as management switch.

MLAG status is ok, but I loose connectivity to secondary switch (no matter which one is picked as primary or secondary). Sometimes I even can’t reach client switch.

I am not sure if this is the feature od MLAG on Mikrotik, but here is my config:

Management switch:

# model = CRS326-24G-2S+
# serial number = 
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan100 vlan-id=100
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether21 pvid=100
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether22 pvid=100
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether23 pvid=100
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether24 pvid=100
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlan100 pvid=100
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,vlan100 untagged=ether21,ether22,ether23,ether24 vlan-ids=100
/ip address
add address=10.44.100.100/24 interface=vlan100 network=10.44.100.0
/system identity
set name="mgmt sw"
/system routerboard settings
set boot-os=router-os

Client switch:

# model = CRS326-24S+2Q+
# serial number = 
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan100 vlan-id=100
/interface bonding
add mode=802.3ad name=bonding1 slaves=sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
/interface lte apn
set [ find default=yes ] ip-type=ipv4
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=bonding1 pvid=3000
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether1 pvid=100
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlan100 pvid=100
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,bonding1 vlan-ids=3000
add bridge=bridge1 tagged=bridge1,vlan100 untagged=ether1 vlan-ids=100
/ip address
add address=10.44.100.101/24 interface=vlan100 network=10.44.100.0
/system identity
set name=KT-DC-SW2
/system routerboard settings
set boot-os=router-os

MLAG switch1:

# model = CRS326-24S+2Q+
# serial number = 
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=qsfpplus1-1 ] name="qsfpplus1-1 - MLAG peer"
set [ find default-name=qsfpplus1-2 ] name="qsfpplus1-2 - MLAG peer"
set [ find default-name=qsfpplus1-3 ] name="qsfpplus1-3 - MLAG peer"
set [ find default-name=qsfpplus1-4 ] name="qsfpplus1-4 - MLAG peer"
set [ find default-name=qsfpplus2-1 ] name="qsfpplus2-1 - MLAG peer"
set [ find default-name=qsfpplus2-2 ] name="qsfpplus2-2 - MLAG peer"
set [ find default-name=qsfpplus2-3 ] name="qsfpplus2-3 - MLAG peer"
set [ find default-name=qsfpplus2-4 ] name="qsfpplus2-4 - MLAG peer"
/interface vlan
add interface=bridge1 name=vlan100 vlan-id=100
/interface bonding
add mode=802.3ad name="bonding - MLAG peer" slaves="qsfpplus1-1 - MLAG peer,qsfpplus1-2 - MLAG peer,qsfpplus1-3 - MLAG peer,qsfpplus1-4 - MLAG peer,qsfpplus2-1 - MLAG peer,qsfpplus2-2 - MLAG peer,qsfpplus2-3 - MLAG peer,qsfpplus2-4 - MLAG peer" transmit-hash-policy=layer-2-and-3
add mlag-id=100 mode=802.3ad name="bonding - Po1" slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
add mlag-id=101 mode=802.3ad name="bonding - Po2" slaves=sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
/interface lte apn
set [ find default=yes ] ip-type=ipv4
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/interface bridge mlag
# peer port not running
set bridge=bridge1 peer-port="bonding - MLAG peer"
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether1 pvid=100
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface="bonding - MLAG peer" pvid=777
add bridge=bridge1 interface="bonding - Po1"
add bridge=bridge1 interface="bonding - Po2"
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlan100 pvid=100
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge1 untagged="bonding - MLAG peer" vlan-ids=777
add bridge=bridge1 tagged="bonding - Po1,bonding - Po2,bonding - MLAG peer" vlan-ids=3000
add bridge=bridge1 tagged="bridge1,bonding - MLAG peer" untagged=ether1 vlan-ids=100
/ip address
add address=10.44.100.102/24 interface=vlan100 network=10.44.100.0
/system identity
set name=DC-SW1
/system routerboard settings
set boot-os=router-os

MLAG switch 2:

# model = CRS326-24S+2Q+
# serial number = 
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=qsfpplus1-1 ] name="qsfpplus1-1 - MLAG peer"
set [ find default-name=qsfpplus1-2 ] name="qsfpplus1-2 - MLAG peer"
set [ find default-name=qsfpplus1-3 ] name="qsfpplus1-3 - MLAG peer"
set [ find default-name=qsfpplus1-4 ] name="qsfpplus1-4 - MLAG peer"
set [ find default-name=qsfpplus2-1 ] name="qsfpplus2-1 - MLAG peer"
set [ find default-name=qsfpplus2-2 ] name="qsfpplus2-2 - MLAG peer"
set [ find default-name=qsfpplus2-3 ] name="qsfpplus2-3 - MLAG peer"
set [ find default-name=qsfpplus2-4 ] name="qsfpplus2-4 - MLAG peer"
/interface vlan
add interface=bridge1 name=vlan100 vlan-id=100
/interface bonding
add mode=802.3ad name="bonding - MLAG peer" slaves="qsfpplus1-1 - MLAG peer,qsfpplus1-2 - MLAG peer,qsfpplus1-3 - MLAG peer,qsfpplus1-4 - MLAG peer,qsfpplus2-1 - MLAG peer,qsfpplus2-2 - MLAG peer,qsfpplus2-3 - MLAG peer,qsfpplus2-4 - MLAG peer" transmit-hash-policy=layer-2-and-3
add mlag-id=100 mode=802.3ad name="bonding - Po1" slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
add mlag-id=101 mode=802.3ad name="bonding - Po2" slaves=sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
/interface lte apn
set [ find default=yes ] ip-type=ipv4
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/interface bridge mlag
# peer port not running
set bridge=bridge1 peer-port="bonding - MLAG peer"
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=ether1 pvid=100
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface="bonding - MLAG peer" pvid=777
add bridge=bridge1 interface="bonding - Po1"
add bridge=bridge1 interface="bonding - Po2"
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlan100 pvid=100
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge1 untagged="bonding - MLAG peer" vlan-ids=777
add bridge=bridge1 tagged="bonding - Po1,bonding - Po2,bonding - MLAG peer" vlan-ids=3000
add bridge=bridge1 tagged="bridge1,bonding - MLAG peer" untagged=ether1 vlan-ids=100
/ip address
add address=10.44.100.103/24 interface=vlan100 network=10.44.100.0
/system identity
set name=DC-SW2
/system routerboard settings
set boot-os=router-os

I really want to push this to production, but I am not sure with such beahviour. Maybe is my mistake, but I don’t see anything wrong.

7

Anyone?

8

Hi,

The VLAN 100 configuration seems a bit odd. You have added the VLAN 100 on the bridge and then added this interface in the “/interface bridge port” and “/interface bridge vlan” sections. It should be enough to allow the bridge interface to be a VLAN 100 tagged member on “/interface bridge vlan” menu. See this example.

On both MLAG peer ports, you have set “frame-types=admit-only-untagged-and-priority-tagged”, but they should be able to forward VLAN tagged packets. Try changing to “frame-types=admit-all” on “bonding - MLAG peer” bridge ports.

9

Thank you for your advices. I will apply suggested changes and report back. Thank you.

10

I have applied all configuration changes, but problem remains the same.

11

Thanks for the update.

Perhaps something else is missing or MLAG goes haywire in certain situations.

Please share all the details (network diagram, supout.rif files, source and destination IP/MAC addresses that cannot communicate) to MikroTik support. Hopefully, we can recreate the same behavior in our labs.

12

Thank you. I will provide all those details to support.

13

Hi,
Same model: CRS326-24S+2Q+, same problem:

Firmware: 7.3.1, 7.4, 7.4.1
Any news?
Thanks

14

The symptoms are:
Upon reboot, the switch can boot normally and be available by IP
or
To be available on a different interface until the MLAG rises, although you can connect to the mac address

15

I think the problem comes from the fact that you have configured on the MLAG the hash policy on Layer2+3 (or Layer3+4), can you try to modify on Layer2 only on the MLAG and on the client switch?