MLAG: Frames destined for VRRP MAC are received by the passive firewall

RouterOS General
1

I'm troubleshooting an unusual forwarding behaviour in a MikroTik MLAG setup and would appreciate some help understanding what might cause it.

Topology

  • Two MikroTik CRS309-1G-8S+ switches (spinesw01, spinesw02), RouterOS 7.25.
  • The switches form an MLAG pair using po1000 as the inter-chassis link (ICC).
  • Both firewalls (fw01, fw02) are connected to both spines using LACP.
  • Each firewall has its own LACP bond (bond1), with one physical link to each spine.
  • On the spines:
    • po1: LACP to fw01, mlag-id=1
    • po2: LACP to fw02, mlag-id=2
    • po3: LACP to the leaf switch pair, mlag-id=3
  • fw01 is the active firewall; fw02 is passive. VRRP is configured on fw01/fw02, using grouped VRRP instances. Only the group authority on the management VLAN sends VRRP advertisements.

Problem

During an iperf test between 10.10.0.4 and 10.10.8.4, fw02 receives a large amount of traffic even though it is passive.

The path here is 10.10.0.4 <-> leafsw03 <-> spinesw01 <-> fw01 <-> spinesw01/02 <-> leafsw01/02 <-> 10.10.8.4.

A packet capture on fw02's sfp1 shows frames like:

SRC MAC: 40:3F:8C:CC:74:D7
DST MAC: 00:00:5E:00:01:6C
VLAN:    1080

SRC IP:  10.10.8.4
DST IP:  10.10.0.4

00:00:5E:00:01:6C is the VRRP virtual MAC belonging to fw01.

The corresponding bridge host entry on both spines is:

mac-address=00:00:5E:00:01:6C
vid=1080
interface=po1

This entry is not present most of the time. If added manually TCP retransmissions drop from several thousands to only few hundrets during iperf test. This is only done to test the behaviour as this manual entry defeats the purpose of VRRP (redundancy)

Performance impact

A 60-second iperf test with four parallel streams gives approximately:

Test Throughput TCP retransmissions
10.10.0.4 ↔ 10.10.8.4 (routed through firewalls and spines) 940 Mbit/s 4,418
10.10.8.2 ↔ 10.10.8.4 (same subnet, both connected to the same leaf-pair (with MLAG)) 941 Mbit/s 110

In another routed test, retransmissions increased to 11,376.

During these tests, fw02's CPU usage rises significantly, sometimes reaching 100%. Its counters show approximately 7.4 GB received on sfp1, but only a few MB transmitted.

The connection from the notebook is wired, no wireless is being used in this case.

What I've tried

  1. The VRRP MAC is dynamically learned on po1 on both spines after fw01 sends an ARP reply using that MAC.
  2. When the FDB entry exists, fw02's CPU load and the retransmission count are significantly lower. When it disappears, both increase.

The current bridge-port configuration has hardware offload enabled (H flag), and unknown-unicast-flood=yes on po2.

There are no obvious physical link errors or FCS errors during the tests.

Questions

  • Could this be related to MLAG forwarding, FDB synchronisation, or hardware offloading on the CRS309/Prestera switch chip?
  • Is there a known RouterOS 7.25 issue or configuration detail that could explain this behaviour?
  • What would be the best way to determine why these frames are sent to fw02?

I would particularly appreciate suggestions for diagnosing the actual hardware forwarding behaviour, rather than simply disabling unknown-unicast flooding or adding static FDB entries.

[user@fw01] > /interface/export 
# 2026-11-01 11:01:45 by RouterOS 7.25
# software id = <redacted>
#
# model = CCR2116-12G-4S+
# serial number = <redacted>
/interface list
add name=mgmt
add name=wan
add name=lan
/interface ethernet
set [ find default-name=ether1 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether2 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether3 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether4 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether5 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether6 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether7 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether8 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether9 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether10 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether11 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether12 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether13 ] comment=";;;" l2mtu=1598
set [ find default-name=sfp-sfpplus1 ] comment="Core;spinesw01 sfp-sfpplus1;C0000118;" l2mtu=1598
set [ find default-name=sfp-sfpplus2 ] comment="Core;spinesw02 sfp-sfpplus1;C0000119;" l2mtu=1598
set [ find default-name=sfp-sfpplus3 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=sfp-sfpplus4 ] comment=";;;" disabled=yes l2mtu=1598
/interface bonding
add comment="LACP to spinesw0[12]" lacp-rate=1sec min-links=1 mode=802.3ad name=bond1 slaves=sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
/interface ethernet switch
set switch1 l3-hw-offloading=yes
/interface vlan
add interface=bond1 name=vl-900-uplink vlan-id=900
add interface=bond1 name=vl-999-mgmt vlan-id=999
add interface=bond1 name=vl-1000-int vlan-id=1000
add interface=bond1 name=vl-1010-media vlan-id=1010
add interface=bond1 name=vl-1030-tools vlan-id=1030
add interface=bond1 name=vl-1040-lab vlan-id=1040
add interface=bond1 name=vl-1050-iot vlan-id=1050
add interface=bond1 name=vl-1060-work vlan-id=1060
add interface=bond1 name=vl-1070-ff vlan-id=1070
add interface=bond1 name=vl-1080-virt vlan-id=1080
add interface=bond1 name=vl-1090-ipmi vlan-id=1090
add interface=bond1 name=vl-1100-unk vlan-id=1100
/interface list member
add interface=vl-999-mgmt list=mgmt
add interface=vl-999-mgmt list=lan
add interface=vl-1000-int list=lan
add interface=vl-1010-media list=lan
add interface=vl-1030-tools list=lan
add interface=vl-1040-lab list=lan
add interface=vl-1050-iot list=lan
add interface=vl-1060-work list=lan
add interface=vl-1070-ff list=lan
add interface=vl-1080-virt list=lan
add interface=vl-1090-ipmi list=lan
add interface=vl-900-uplink list=wan
/interface vrrp
add group-authority=self interface=vl-999-mgmt name=vrrp-mgmt priority=254 remote-address=10.10.99.252 sync-connection-tracking=yes vrid=99
add group-authority=vrrp-mgmt interface=vl-1030-tools name=vrrp-tools priority=254 vrid=103
add group-authority=vrrp-mgmt interface=vl-1100-unk name=vrrp-unk priority=254 vrid=110
add group-authority=vrrp-mgmt interface=vl-900-uplink name=vrrp-uplink priority=254 vrid=90
add group-authority=vrrp-mgmt interface=vl-1080-virt name=vrrp-virt priority=254 vrid=108
add group-authority=vrrp-mgmt interface=vl-1060-work name=vrrp-work priority=254 vrid=106
/interface list member
add interface=vrrp-mgmt list=lan
add interface=vrrp-tools list=lan
add interface=vrrp-virt list=lan
add interface=vrrp-work list=lan
add interface=vrrp-uplink list=wan
/interface vrrp
add group-authority=vrrp-mgmt interface=vl-1070-ff name=vrrp-ff priority=254 vrid=107
add group-authority=vrrp-mgmt interface=vl-1000-int name=vrrp-int priority=254 vrid=100
add group-authority=vrrp-mgmt interface=vl-1050-iot name=vrrp-iot priority=254 vrid=105
add group-authority=vrrp-mgmt interface=vl-1090-ipmi name=vrrp-ipmi priority=254 vrid=109
add group-authority=vrrp-mgmt interface=vl-1040-lab name=vrrp-lab priority=254 vrid=104
add group-authority=vrrp-mgmt interface=vl-1010-media name=vrrp-media priority=254 vrid=101
/interface list member
add interface=vrrp-ff list=lan
add interface=vrrp-int list=lan
add interface=vrrp-iot list=lan
add interface=vrrp-ipmi list=lan
add interface=vrrp-lab list=lan
add interface=vrrp-media list=lan


[user@fw02] > /interface/export                         
# 2026-11-01 11:18:59 by RouterOS 7.25
# software id = <redacted>
#
# model = RB3011UiAS
# serial number = <redacted>
/interface list
add name=mgmt
add name=wan
add name=lan
/interface ethernet
set [ find default-name=ether1 ] comment="Core;spinesw02 sfp-sfpplus2;C000005;"
set [ find default-name=ether2 ] comment=";;;" disabled=yes
set [ find default-name=ether3 ] comment=";;;" disabled=yes
set [ find default-name=ether4 ] comment=";;;" disabled=yes
set [ find default-name=ether5 ] comment=";;;" disabled=yes
set [ find default-name=ether6 ] comment=";;;" disabled=yes
set [ find default-name=ether7 ] comment=";;;" disabled=yes
set [ find default-name=ether8 ] comment=";;;" disabled=yes
set [ find default-name=ether9 ] comment=";;;" disabled=yes
set [ find default-name=ether10 ] comment=";;;" disabled=yes
set [ find default-name=sfp1 ] auto-negotiation=no comment="Core;spinesw01 sfp-sfpplus2;C000048;" l2mtu=1598 speed=1G-baseT-full
/interface bonding
add comment="LACP to spinesw0[12]" lacp-rate=1sec min-links=1 mode=802.3ad name=bond1 slaves=ether1,sfp1 transmit-hash-policy=layer-2-and-3
/interface vlan
add interface=bond1 name=vl-900-uplink vlan-id=900
add interface=bond1 name=vl-999-mgmt vlan-id=999
add interface=bond1 name=vl-1000-int vlan-id=1000
add interface=bond1 name=vl-1010-media vlan-id=1010
add interface=bond1 name=vl-1030-tools vlan-id=1030
add interface=bond1 name=vl-1040-lab vlan-id=1040
add interface=bond1 name=vl-1050-iot vlan-id=1050
add interface=bond1 name=vl-1060-work vlan-id=1060
add interface=bond1 name=vl-1070-ff vlan-id=1070
add interface=bond1 name=vl-1080-virt vlan-id=1080
add interface=bond1 name=vl-1090-ipmi vlan-id=1090
add interface=bond1 name=vl-1100-unk vlan-id=1100
/interface list member
add interface=vl-999-mgmt list=mgmt
add interface=vl-999-mgmt list=lan
add interface=vl-1000-int list=lan
add interface=vl-1010-media list=lan
add interface=vl-1030-tools list=lan
add interface=vl-1040-lab list=lan
add interface=vl-1050-iot list=lan
add interface=vl-1060-work list=lan
add interface=vl-1070-ff list=lan
add interface=vl-1080-virt list=lan
add interface=vl-1090-ipmi list=lan
add interface=vl-900-uplink list=wan
/interface vrrp
add group-authority=self interface=vl-999-mgmt name=vrrp-mgmt remote-address=10.10.99.253 sync-connection-tracking=yes vrid=99
add group-authority=vrrp-mgmt interface=vl-1030-tools name=vrrp-tools vrid=103
add group-authority=vrrp-mgmt interface=vl-1100-unk name=vrrp-unk vrid=110
add group-authority=vrrp-mgmt interface=vl-900-uplink name=vrrp-uplink vrid=90
add group-authority=vrrp-mgmt interface=vl-1080-virt name=vrrp-virt vrid=108
add group-authority=vrrp-mgmt interface=vl-1060-work name=vrrp-work vrid=106
/interface list member
add interface=vrrp-mgmt list=lan
add interface=vrrp-tools list=lan
add interface=vrrp-virt list=lan
add interface=vrrp-work list=lan
add interface=vrrp-uplink list=wan
/interface vrrp
add group-authority=vrrp-mgmt interface=vl-1070-ff name=vrrp-ff vrid=107
add group-authority=vrrp-mgmt interface=vl-1000-int name=vrrp-int vrid=100
add group-authority=vrrp-mgmt interface=vl-1050-iot name=vrrp-iot vrid=105
add group-authority=vrrp-mgmt interface=vl-1090-ipmi name=vrrp-ipmi vrid=109
add group-authority=vrrp-mgmt interface=vl-1040-lab name=vrrp-lab vrid=104
add group-authority=vrrp-mgmt interface=vl-1010-media name=vrrp-media vrid=101
/interface list member
add interface=vrrp-ff list=lan
add interface=vrrp-int list=lan
add interface=vrrp-iot list=lan
add interface=vrrp-ipmi list=lan
add interface=vrrp-lab list=lan
add interface=vrrp-media list=lan


[user@spinesw01] > /interface/export 
# 2026-11-01 11:21:31 by RouterOS 7.25
# software id = <redacted>
#
# model = CRS309-1G-8S+
# serial number = <redacted>
/interface list
add name=mgmt
/interface ethernet
set [ find default-name=ether1 ] comment=";;;"
set [ find default-name=sfp-sfpplus1 ] comment="Core;fw01 sfp-sfpplus1;C0000118;"
set [ find default-name=sfp-sfpplus2 ] auto-negotiation=no comment="Core;fw02 sfp1;C000048;" speed=1G-baseT-full
set [ find default-name=sfp-sfpplus3 ] comment="Core;leafsw01 sfp-sfpplus1;C0000116;"
set [ find default-name=sfp-sfpplus4 ] comment="Access;PP-Basement-01 23;C0000113;"
set [ find default-name=sfp-sfpplus5 ] comment="Access;PP-Basement-01 24;C0000112;"
set [ find default-name=sfp-sfpplus6 ] comment=";;;" disabled=yes
set [ find default-name=sfp-sfpplus7 ] comment="Core;spinesw02 sfp-sfpplus7;C0000114;ICC"
set [ find default-name=sfp-sfpplus8 ] comment="Core;spinesw02 sfp-sfpplus8;C0000115;ICC"
/interface bonding
add comment="LACP to fw01" lacp-rate=1sec min-links=1 mlag-id=1 mode=802.3ad name=po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
add comment="LACP to fw02" lacp-rate=1sec min-links=1 mlag-id=2 mode=802.3ad name=po2 slaves=sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
add comment="LACP to leafsw0[12]" lacp-rate=1sec min-links=1 mlag-id=3 mode=802.3ad name=po3 slaves=sfp-sfpplus3 transmit-hash-policy=layer-2-and-3
add comment="LACP to leafsw03" lacp-rate=1sec min-links=1 mode=802.3ad name=po4 slaves=sfp-sfpplus4 transmit-hash-policy=layer-2-and-3
add comment="ICC bond" lacp-rate=1sec min-links=1 mode=802.3ad name=po1000 slaves=sfp-sfpplus7,sfp-sfpplus8 transmit-hash-policy=layer-2-and-3
/interface bridge
add admin-mac=02:00:00:99:00:11 auto-mac=no frame-types=admit-only-vlan-tagged mlag-peer-port=po1000 mlag-priority=50 mtu=1500 name=general-bridge priority=0x1000 vlan-filtering=yes
/interface vlan
add interface=general-bridge name=vl-999-mgmt vlan-id=999
/interface bridge host
add bridge=general-bridge interface=po1 mac-address=00:00:5E:00:01:6C vid=1080
/interface list member
add interface=vl-999-mgmt list=mgmt
/interface bridge port
add bridge=general-bridge edge=yes frame-types=admit-only-vlan-tagged interface=po1
add bridge=general-bridge edge=yes frame-types=admit-only-vlan-tagged interface=po2
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=po3
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=sfp-sfpplus5
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=po4
add bridge=general-bridge edge=no interface=po1000 pvid=100
/interface bridge vlan
add bridge=general-bridge tagged=general-bridge,po1,po2,po3,po4,po1000,sfp-sfpplus5 vlan-ids=999
add bridge=general-bridge tagged=po1,po2,po4,po1000 vlan-ids=900
add bridge=general-bridge tagged=po1,po2,po4,po1000 vlan-ids=1000
add bridge=general-bridge tagged=po1,po2,po3,po4,po1000,sfp-sfpplus5 vlan-ids=1010
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1030
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1040
add bridge=general-bridge tagged=po1,po2,po4,po1000 vlan-ids=1050
add bridge=general-bridge tagged=po1,po2,po4,po1000 vlan-ids=1060
add bridge=general-bridge tagged=po1,po2,po4,po1000 vlan-ids=1070
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1080
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1090
add bridge=general-bridge tagged=po1,po2,po4,po1000 vlan-ids=1100


[user@spinesw02] > /interface/export 
# 2026-11-01 11:21:48 by RouterOS 7.25
# software id = <redacted>
#
# model = CRS309-1G-8S+
# serial number = <redacted>
/interface list
add name=mgmt
/interface ethernet
set [ find default-name=ether1 ] comment=";;;"
set [ find default-name=sfp-sfpplus1 ] comment="Core;fw01 sfp-sfpplus2;C0000119;"
set [ find default-name=sfp-sfpplus2 ] comment="Core;fw02 ether1;C000005;"
set [ find default-name=sfp-sfpplus3 ] comment="Core;leafsw02 sfp-sfpplus1;C0000117;"
set [ find default-name=sfp-sfpplus4 ] comment=";;;" disabled=yes
set [ find default-name=sfp-sfpplus5 ] comment=";;;" disabled=yes
set [ find default-name=sfp-sfpplus6 ] comment=";;;" disabled=yes
set [ find default-name=sfp-sfpplus7 ] comment="Core;spinesw01 sfp-sfpplus7;C0000114;"
set [ find default-name=sfp-sfpplus8 ] comment="Core;spinesw01 sfp-sfpplus8;C0000115;"
/interface bonding
add comment="LACP to fw01" lacp-rate=1sec min-links=1 mlag-id=1 mode=802.3ad name=po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
add comment="LACP to fw02" lacp-rate=1sec min-links=1 mlag-id=2 mode=802.3ad name=po2 slaves=sfp-sfpplus2 transmit-hash-policy=layer-2-and-3
add comment="LACP to leafsw0[12]" lacp-rate=1sec min-links=1 mlag-id=3 mode=802.3ad name=po3 slaves=sfp-sfpplus3 transmit-hash-policy=layer-2-and-3
add comment="ICC bond" lacp-rate=1sec min-links=1 mode=802.3ad name=po1000 slaves=sfp-sfpplus7,sfp-sfpplus8 transmit-hash-policy=layer-2-and-3
/interface bridge
add admin-mac=02:00:00:99:00:12 auto-mac=no frame-types=admit-only-vlan-tagged mlag-peer-port=po1000 mlag-priority=100 mtu=1500 name=general-bridge priority=0x1000 vlan-filtering=yes
/interface vlan
add interface=general-bridge name=vl-999-mgmt vlan-id=999
/interface list member
add interface=vl-999-mgmt list=mgmt
/interface bridge port
add bridge=general-bridge edge=yes frame-types=admit-only-vlan-tagged interface=po1
add bridge=general-bridge edge=yes frame-types=admit-only-vlan-tagged interface=po2
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=po3
add bridge=general-bridge edge=no interface=po1000 pvid=100
/interface bridge vlan
add bridge=general-bridge tagged=general-bridge,po1,po2,po3,po1000 vlan-ids=999
add bridge=general-bridge tagged=po1,po2,po1000 vlan-ids=900
add bridge=general-bridge tagged=po1,po2,po1000 vlan-ids=1000
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1010
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1030
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1040
add bridge=general-bridge tagged=po1,po2,po1000 vlan-ids=1050
add bridge=general-bridge tagged=po1,po2,po1000 vlan-ids=1060
add bridge=general-bridge tagged=po1,po2,po1000 vlan-ids=1070
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1080
add bridge=general-bridge tagged=po1,po2,po3,po1000 vlan-ids=1090
add bridge=general-bridge tagged=po1,po2,po1000 vlan-ids=1100


[user@leafsw01] > /interface/export 
# 2026-11-01 11:22:10 by RouterOS 7.25
# software id = <redacted>
#
# model = CRS326-24G-2S+
# serial number = <redacted>
/interface list
add name=mgmt
/interface ethernet
set [ find default-name=ether1 ] comment="Core;leafsw02 ether1;C000100;" l2mtu=1598
set [ find default-name=ether2 ] comment="Core;leafsw02 ether2;C000144;" l2mtu=1598
set [ find default-name=ether3 ] comment="Access;nas01 eth0;C000145;" l2mtu=1598
set [ find default-name=ether4 ] comment="Access;vhost01 eno1;C000028;" l2mtu=1598
set [ find default-name=ether5 ] comment="Access;vhost02 enp2s0;C000150;" l2mtu=1598
set [ find default-name=ether6 ] comment="Access;vhost03 enp9s0;C000149;" l2mtu=1598
set [ find default-name=ether7 ] comment="Access;vhost04 enp8s0;C000152;" l2mtu=1598
set [ find default-name=ether8 ] comment="Access;monitoring02 eth0;C000004;" l2mtu=1598
set [ find default-name=ether9 ] comment="Access;log01 eth0;C000014;" l2mtu=1598
set [ find default-name=ether10 ] comment="Access;cs01 eth0;C000039;" l2mtu=1598
set [ find default-name=ether11 ] comment="Access;tools02 eth0;C000026;" l2mtu=1598
set [ find default-name=ether12 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether13 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether14 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether15 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether16 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether17 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether18 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether19 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether20 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether21 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether22 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether23 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether24 ] comment="Access;vhost01 iDRAC;;" l2mtu=1598
set [ find default-name=sfp-sfpplus1 ] comment="Core;spinesw01 sfp-sfpplus3;C0000116;" l2mtu=1598
set [ find default-name=sfp-sfpplus2 ] comment=";;;" disabled=yes l2mtu=1598
/interface bonding
add comment="LACP to spinesw0[12]" lacp-rate=1sec min-links=1 mlag-id=1 mode=802.3ad name=po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
add comment="LACP to nas01" lacp-rate=1sec min-links=1 mlag-id=2 mode=802.3ad name=po2 slaves=ether3 transmit-hash-policy=layer-2-and-3
add comment="LACP to vhost01" lacp-rate=1sec min-links=1 mlag-id=3 mode=802.3ad name=po3 slaves=ether4 transmit-hash-policy=layer-2-and-3
add comment="LACP to vhost02" lacp-rate=1sec min-links=1 mlag-id=4 mode=802.3ad name=po4 slaves=ether5 transmit-hash-policy=layer-2-and-3
add comment="LACP to vhost04" lacp-rate=1sec min-links=1 mlag-id=5 mode=802.3ad name=po5 slaves=ether7 transmit-hash-policy=layer-2-and-3
add comment="ICC bond" lacp-rate=1sec min-links=1 mode=802.3ad name=po1000 slaves=ether1,ether2 transmit-hash-policy=layer-2-and-3
/interface bridge
add admin-mac=02:00:00:99:00:21 auto-mac=no frame-types=admit-only-vlan-tagged mlag-peer-port=po1000 mlag-priority=50 mtu=1500 name=general-bridge priority=0x2000 vlan-filtering=yes
/interface vlan
add interface=general-bridge name=vl-999-mgmt vlan-id=999
/interface list member
add interface=vl-999-mgmt list=mgmt
/interface bridge port
add bridge=general-bridge edge=yes interface=ether6 pvid=1080
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether8 pvid=1030
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether9 pvid=1030
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether10 pvid=1030
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether11 pvid=1030
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether24 pvid=1090
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=po1
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=po2 pvid=1010
add bridge=general-bridge edge=yes interface=po3 pvid=1080
add bridge=general-bridge edge=yes interface=po4 pvid=1080
add bridge=general-bridge edge=yes interface=po5 pvid=1080
add bridge=general-bridge edge=no interface=po1000 pvid=100
/interface bridge vlan
add bridge=general-bridge tagged=general-bridge,po1,po1000 vlan-ids=999
add bridge=general-bridge tagged=po1000 vlan-ids=1000
add bridge=general-bridge tagged=ether6,po1,po3,po4,po5,po1000 vlan-ids=1010
add bridge=general-bridge tagged=ether6,po1,po3,po4,po5,po1000 vlan-ids=1030
add bridge=general-bridge tagged=po1000 vlan-ids=1060
add bridge=general-bridge tagged=po1000 vlan-ids=1070
add bridge=general-bridge tagged=po1000 vlan-ids=1100
add bridge=general-bridge tagged=po1,po1000 vlan-ids=1090
add bridge=general-bridge tagged=po1,po1000 vlan-ids=1080
add bridge=general-bridge tagged=ether6,po1,po3,po4,po5,po1000 vlan-ids=1040
add bridge=general-bridge tagged=po1000 vlan-ids=1050


[user@leafsw02] > /interface/export 
# 2026-11-01 11:22:22 by RouterOS 7.25
# software id = <redacted>
#
# model = CRS326-24G-2S+
# serial number = <redacted>
/interface list
add name=mgmt
/interface ethernet
set [ find default-name=ether1 ] comment="Core;leafsw01 ether1;C000100;" l2mtu=1598
set [ find default-name=ether2 ] comment="Core;leafsw01 ether2;C000144;" l2mtu=1598
set [ find default-name=ether3 ] comment="Access;nas01 eth1;C000146;" l2mtu=1598
set [ find default-name=ether4 ] comment="Access;vhost01 eno2;;" l2mtu=1598
set [ find default-name=ether5 ] comment="Access;vhost02 enp1s0;C0000151;" l2mtu=1598
set [ find default-name=ether6 ] comment="Access;vhost03 enx00249b3acc74;C000013;" l2mtu=1598
set [ find default-name=ether7 ] comment="Access;vhost04 enp6s0;C000153;" l2mtu=1598
set [ find default-name=ether8 ] comment="Access;monitoring02 enp1s0u2c2;C000027;" l2mtu=1598
set [ find default-name=ether9 ] comment="Access;log01 eth1;;" l2mtu=1598
set [ find default-name=ether10 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether11 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether12 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether13 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether14 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether15 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether16 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether17 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether18 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether19 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether20 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether21 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether22 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether23 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=ether24 ] comment=";;;" disabled=yes l2mtu=1598
set [ find default-name=sfp-sfpplus1 ] comment="Core;spinesw02 sfp-sfpplus3;C0000117;" l2mtu=1598
set [ find default-name=sfp-sfpplus2 ] comment=";;;" disabled=yes l2mtu=1598
/interface bonding
add comment="LACP to spinesw0[12]" lacp-rate=1sec min-links=1 mlag-id=1 mode=802.3ad name=po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
add comment="nas01 LACP" lacp-rate=1sec min-links=1 mlag-id=2 mode=802.3ad name=po2 slaves=ether3 transmit-hash-policy=layer-2-and-3
add comment="vhost01 LACP" lacp-rate=1sec min-links=1 mlag-id=3 mode=802.3ad name=po3 slaves=ether4 transmit-hash-policy=layer-2-and-3
add comment="vhost02 LACP" lacp-rate=1sec min-links=1 mlag-id=4 mode=802.3ad name=po4 slaves=ether5 transmit-hash-policy=layer-2-and-3
add comment="vhost04 LACP" lacp-rate=1sec min-links=1 mlag-id=5 mode=802.3ad name=po5 slaves=ether7 transmit-hash-policy=layer-2-and-3
add comment="ICC bond" lacp-rate=1sec min-links=1 mode=802.3ad name=po1000 slaves=ether1,ether2 transmit-hash-policy=layer-2-and-3
/interface bridge
add admin-mac=02:00:00:99:00:22 auto-mac=no frame-types=admit-only-vlan-tagged mlag-peer-port=po1000 mlag-priority=100 mtu=1500 name=general-bridge priority=0x2000 vlan-filtering=yes
/interface vlan
add interface=general-bridge name=vl-999-mgmt vlan-id=999
/interface list member
add interface=vl-999-mgmt list=mgmt
/interface bridge port
add bridge=general-bridge edge=yes interface=ether6 pvid=1080
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether8 pvid=1030
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether9 pvid=1030
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=po1
add bridge=general-bridge edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=po2 pvid=1010
add bridge=general-bridge edge=yes interface=po3 pvid=1080
add bridge=general-bridge edge=yes interface=po4 pvid=1080
add bridge=general-bridge edge=yes interface=po5 pvid=1080
add bridge=general-bridge edge=no interface=po1000 pvid=100
/interface bridge vlan
add bridge=general-bridge tagged=general-bridge,po1,po1000 vlan-ids=999
add bridge=general-bridge tagged=ether6,po1,po3,po4,po5,po1000 vlan-ids=1010
add bridge=general-bridge tagged=ether6,po1,po3,po4,po5,po1000 vlan-ids=1030
add bridge=general-bridge tagged=ether6,po1,po3,po4,po5,po1000 vlan-ids=1040
add bridge=general-bridge tagged=po1,po1000 vlan-ids=1080
add bridge=general-bridge tagged=po1000 vlan-ids=1000
add bridge=general-bridge tagged=po1000 vlan-ids=1050
add bridge=general-bridge tagged=po1000 vlan-ids=1060
add bridge=general-bridge tagged=po1000 vlan-ids=1070
add bridge=general-bridge tagged=po1000 vlan-ids=1090
add bridge=general-bridge tagged=po1000 vlan-ids=1100


[user@leafsw03] > /interface/export 
# 2026-11-01 11:22:41 by RouterOS 7.25
# software id = <redacted>
#
# model = CRS326-24G-2S+
# serial number = <redacted>
/interface list
add name=mgmt
/interface bridge
add admin-mac=02:00:00:99:00:23 auto-mac=no frame-types=admit-only-vlan-tagged mtu=1500 name=general-bridge priority=0x3000 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment="Access;PP-Office-01 1;C000133;pc"
set [ find default-name=ether2 ] comment="Access;PP-Office-01 2;C000134;notebook"
set [ find default-name=ether3 ] comment="Access;PP-Office-01 3;C000135;printer"
set [ find default-name=ether4 ] comment="Access;PP-Office-01 4;C000136;" disabled=yes
set [ find default-name=ether5 ] comment="Access;PP-Office-01 5;C000137;notebook work"
set [ find default-name=ether6 ] comment="Access;PP-Office-01 6;C000138;" disabled=yes
set [ find default-name=ether7 ] comment="Access;PP-Office-01 7;C000139;tv"
set [ find default-name=ether8 ] comment="Access;PP-Office-01 8;C000140;" disabled=yes
set [ find default-name=ether9 ] comment="Access;PP-Office-01 9;C000141;"
set [ find default-name=ether10 ] comment="Access;PP-Office-01 10;C000142;"
set [ find default-name=ether11 ] comment=";;;" disabled=yes
set [ find default-name=ether12 ] comment=";;;" disabled=yes
set [ find default-name=ether13 ] comment=";;;" disabled=yes
set [ find default-name=ether14 ] comment=";;;" disabled=yes
set [ find default-name=ether15 ] comment=";;;" disabled=yes
set [ find default-name=ether16 ] comment=";;;" disabled=yes
set [ find default-name=ether17 ] comment=";;;" disabled=yes
set [ find default-name=ether18 ] comment=";;;" disabled=yes
set [ find default-name=ether19 ] comment=";;;" disabled=yes
set [ find default-name=ether20 ] comment=";;;" disabled=yes
set [ find default-name=ether21 ] comment="Core;ff01 br-wan;C000143;"
set [ find default-name=ether22 ] comment="Core;modem01 LAN 1;C000132;"
set [ find default-name=ether23 ] comment=";;;" disabled=yes
set [ find default-name=ether24 ] comment=";;;" disabled=yes
set [ find default-name=sfp-sfpplus1 ] comment="Access;PP-Office-01 24;C000120;"
set [ find default-name=sfp-sfpplus2 ] comment=";;;" disabled=yes
/interface vlan
add interface=general-bridge name=vl-999-mgmt vlan-id=999
/interface bonding
add lacp-rate=1sec min-links=1 mlag-id=1 mode=802.3ad name=po1 slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3
/interface list member
add interface=vl-999-mgmt list=mgmt
/interface bridge port
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether21 pvid=1070
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether22 pvid=900
add bridge=general-bridge edge=no frame-types=admit-only-vlan-tagged interface=po1
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether2 pvid=1000
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether1 pvid=1000
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether3 pvid=1010
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether5 pvid=1060
add bridge=general-bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether7 pvid=1010
add bridge=general-bridge frame-types=admit-only-vlan-tagged interface=ether9
add bridge=general-bridge frame-types=admit-only-vlan-tagged interface=ether10
/interface bridge vlan
add bridge=general-bridge tagged=ether9,ether10,general-bridge,po1 vlan-ids=999
add bridge=general-bridge tagged=po1 vlan-ids=900
add bridge=general-bridge tagged=ether9,ether10,po1 vlan-ids=1000
add bridge=general-bridge tagged=ether9,ether10,po1 vlan-ids=1010
add bridge=general-bridge tagged=ether9,ether10,po1 vlan-ids=1060
add bridge=general-bridge tagged=po1 vlan-ids=1070
2

Could you be so kind to draw a network diagram + assigned IPs
From the wall of text is quite hard to determine the layout.

3

Come on @BartoszP, that's hardly a wall of text. The actual write-up is short and neat, the rest is just the exports we're always asking people to post.

The layout is right there under "Topology" and the traffic path under "Problem":
10.10.0.4 → leafsw03 → spinesw01 → fw01 → spinesw01/02 → leafsw01/02 → 10.10.8.4

With that and the exports, I'd say the picture's already there.

4

@derdeagle My guess is the VRRP MAC just ages out of the FDB on the spines, since with group-authority only vrrp-mgmt sends advertisements (on VLAN 999). On VLAN 1080 the virtual MAC is probably only seen as a source in ARP replies, so after the default 5 min ageing-time the entry is gone and the traffic gets flooded to every port in the VLAN, fw02 included.

A quick A/B test should show if that's the cause by comparing the current setup with one where VLAN 1080 gets its own advertisements:

Test A: current config, but remove the static host entry on spinesw01 first:

/interface/bridge/host/remove [find mac-address=00:00:5E:00:01:6C !dynamic]

Give it more than 5 minutes so the dynamic entry ages out, then run iperf and note retransmits and fw02 CPU. Meanwhile, check on the spines every now and then if the entry is still there:

/interface/bridge/host/print where mac-address=00:00:5E:00:01:6C

In a separate run (the sniffer turns off fast path on fw01 and would mess up the numbers), see which source MAC the routed traffic actually uses:

/tool/sniffer/quick interface=vl-1080-virt ip-address=10.10.0.4 proplist=time,dir,src-mac,dst-mac,src-address,dst-address

If nothing shows up, the traffic is probably L3 hardware offloaded and never hits the CPU.

Test B: let vrrp-virt send its own advertisements (on both fw01 and fw02):

/interface/vrrp/set vrrp-virt group-authority=self

Run the same iperf test again.

If after test B the FDB entry stays and fw02 isn't flooded anymore, that confirms it. The fix is to let every VRRP instance send its own advertisements, but you lose the grouping (instances can then fail over independently). If you need to keep the grouping, a workaround is a scheduled script on fw01 sending a gratuitous ARP from each virtual MAC every minute or so. (check this)

Instead of scripting (which IMO is an ugly workaround), what would really help here is a robust built-in feature that lets the VRRP master keep the virtual MAC address alive on grouped instances. Other users and I have already requested this, so feel free to submit it as a feature request too.

@EdPa, maybe something to consider for a future release?

Anyway, if retransmits are still high after test B, it's something else. Then check tx drops on the 10G to 1G hop (leafsw03 ether2 and the spine ports) with /interface/print stats.

Flooding that keeps going even though the entry is there would point to a bug, so open a ticket with supout.rif and also ask if the v7.25 changelog entry "fix VRRP MAC behavior in MLAG setups" might be related.

Feel free to report back once you've run the tests and heard from support.

EDIT:

This might be worth bringing up with support too!

Same story as in Feature Request: VRRP L2 Keep-alive for Unmanaged Switch Environments, and the MT VRRP docs don't say a word about this side effect. They mention that only the group authority sends advertisements, but not that the other instances' virtual MACs then stop getting refreshed in the switches' FDB. The WAN+LAN example even suggests making LAN the group authority, which matches the setup in that thread where the WAN-side MAC ages out.

5

@Larsa Thank you very much for your reply. I tried your tests out.

Test A

With the static bridge host table entry removed and a reverse (-R) iperf test the CPU load on fw02 (passive) went up again - as expected.

[user@fw01] > /tool/sniffer/quick interface=vl-1080-virt ip-address=10.10.0.4 proplist=time,dir,src-mac,dst-mac,src-address,dst-address
Columns: TIME, DIR, SRC-MAC, DST-MAC, SRC-ADDRESS, DST-ADDRESS
TIME      DIR  SRC-MAC            DST-MAC            SRC-ADDRESS      DST-ADDRESS    
3.409679  ->   18:FD:74:7C:9E:99  40:3F:8C:CC:74:D7  10.10.0.4:45458  10.10.8.4:5201 
3.409686  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45474
3.4097    <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45474
3.409709  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45474
3.409721  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45474
3.409734  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45474
3.409747  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45474
3.409761  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.409773  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.409776  ->   18:FD:74:7C:9E:99  40:3F:8C:CC:74:D7  10.10.0.4:45458  10.10.8.4:5201 
3.409784  ->   18:FD:74:7C:9E:99  40:3F:8C:CC:74:D7  10.10.0.4:45468  10.10.8.4:5201 
3.409787  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.40979   ->   18:FD:74:7C:9E:99  40:3F:8C:CC:74:D7  10.10.0.4:45474  10.10.8.4:5201 
3.409798  ->   18:FD:74:7C:9E:99  40:3F:8C:CC:74:D7  10.10.0.4:45476  10.10.8.4:5201 
3.4098    <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.409813  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.409826  ->   18:FD:74:7C:9E:99  40:3F:8C:CC:74:D7  10.10.0.4:45474  10.10.8.4:5201 
3.409827  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.40984   <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458
3.409853  <-   40:3F:8C:CC:74:D7  00:00:5E:00:01:6C  10.10.8.4:5201   10.10.0.4:45458

Test B

With the group-authority set to self on vrrp-virt the MAC address was instantly present in the bridge host table on the spine switches. No real CPU load increase on fw02 (passive) could be seen and only ~300 TCP retransmits for a 30 second iperf test with ~940 Mbit/s (4 workers).

Yet I do have several entries in the bridge host table even though (as for vrrp-virt) the group-authority is set to vrrp-mgmt. Why is that the case for e.g. vrrp-int, vrrp-media, vrrp-tools, vrrp-ff then?

6

Test B confirms the VRRP MAC just ages out of the FDB. Your sniffer shows fw01 routes with its own MAC (18:FD:74:7C:9E:99), so the virtual MAC never shows up as a source for routed traffic. That means it's not MLAG, FDB sync, HW offload or a specific v7.25 bug, it's a design limitation (a flaw IMO) in how group-authority works: only vrrp-mgmt sends advertisements, so nothing regularly refreshes the virtual MAC on the other VLANs.

As for vrrp-int, vrrp-media, vrrp-tools and vrrp-ff, their entries are only kept alive by fw01's ARP replies (like you noticed in your first post). Those VLANs are busier, so someone ARPs for the gateway often enough and they're basically there by luck. Same problem, it just hasn't bitten you yet. You can check it on fw01, e.g. for vrrp-int (the last byte is the VRID in hex, so swap it for the others):

/tool/sniffer/quick src-mac-address=00:00:5E:00:01:64 proplist=time,interface,dir,src-mac,dst-mac,protocol

So like I said, set group-authority=self, in your case on every instance whose VLAN is tagged on po2, vrrp-uplink included (otherwise WAN traffic can end up at fw02 too). Without the grouping, instances can end up split between fw01 and fw02 and break NAT, so if that worries you, the GARP script still does the job. And remove the static host entry on spinesw01 when you're done.

The ~300 retransmits are most likely caused by 10G bursts hitting a 1G port. That's just TCP doing its thing, and turning on flow control would only make things worse since it pauses the whole 10G uplink and everything else on it. To check, run the same test length and direction as before and look at tx drops on the 1G egress port (leafsw03 ether2 with -R):

/interface/ethernet/print stats where name=ether2

So two things are worth bringing up with support:

  • RFE: let grouped VRRP instances still send their own advertisements or a periodic GARP on their VLAN, so you keep the grouping without the FDB entry aging out (which replaces the need for scripting).
  • MT Doc flaw: the VRRP page should warn that group-authority lets the other instances' virtual MACs age out in the switches.