VRRP brings my entire network down

divB · August 19, 2026, 6:27am

VRRP is driving me nuts for a few weeks now and it does something that should never, ever happen: "Randomly" my entire network goes down and by this I really mean it: All VLANs, no matter if part of a VRRP or not stop working and even traffic from two unrelated hosts in an unrelated VLAN stops working.

I have Router2 which is configured as master:

/interface vrrp
add group-authority=vrrp98 interface=vlan3 name=vrrp3 on-backup="/ip/dhcp-server/disable dhcp-lan" on-master="/ip/dhcp-server/enable dhcp-lan" priority=200 vrid=3
add group-authority=vrrp98 interface=vlan4 name=vrrp4 priority=200 vrid=4
add group-authority=vrrp98 interface=vlan11 name=vrrp11 on-backup="/ip/dhcp-server/disable dhcp-dev" on-master="/ip/dhcp-server/enable dhcp-dev" priority=200 vrid=11
add group-authority=vrrp98 interface=vlan12 name=vrrp12 on-backup="/ip/dhcp-server/enable dhcp-devp" on-master="/ip/dhcp-server/enable dhcp-devp" priority=200 vrid=12
add group-authority=vrrp98 interface=vlan44 name=vrrp44 priority=200 vrid=44
add group-authority=self interface=vlan98 name=vrrp98 priority=200 remote-address=10.227.2.1 sync-connection-tracking=yes vrid=98
add group-authority=vrrp98 interface=vlan99 name=vrrp99 on-backup="/ip/dhcp-server/disable dhcp-adm" on-master="/ip/dhcp-server/enable dhcp-adm" priority=200 vrid=99

Then I have Router1 which is configured as slave:

/interface vrrp
add disabled=yes group-authority=vrrp98 interface=vlan3 name=vrrp3 on-backup="/ip/dhcp-server/disable dhcp-lan" on-master="/ip/dhcp-server/enable dhcp-lan" vrid=3
add group-authority=vrrp98 interface=vlan4 name=vrrp4 vrid=4
add group-authority=vrrp98 interface=vlan11 name=vrrp11 on-backup="/ip/dhcp-server/disable dhcp-dev" on-master="/ip/dhcp-server/enable dhcp-dev" vrid=11
add group-authority=vrrp98 interface=vlan12 name=vrrp12 on-backup="/ip/dhcp-server/enable dhcp-devp" on-master="/ip/dhcp-server/enable dhcp-devp" vrid=12
add group-authority=vrrp98 interface=vlan44 name=vrrp44 vrid=44
add group-authority=self interface=vlan98 name=vrrp98 remote-address=10.227.2.2 sync-connection-tracking=yes vrid=98
add group-authority=vrrp98 interface=vlan99 name=vrrp99 on-backup="/ip/dhcp-server/disable dhcp-adm" on-master="/ip/dhcp-server/enable dhcp-adm" vrid=99

All my interfaces are part of a VRF. As very well known, it's impossible to touch anything VRF related on RouterOS without finding a bug: A new bug I found is that connection sync seems to communicate via main VRF (*). For this reason alone, I added an additional interface vlan98 (along with vrrp98) on each router which is not assigned to any VRF. It acts as group-authority. The other interfaces are all using vrrp98 as group authority. Both routers can feely communicate via vlan98 (10.227.2.1 and 10.227.2.2).

The config as above has now been working for over an hour. Please note that vrrp3 on the slave is disabled.

Now comes the issue: once I enable it, all hell breaks loose. All VLANs, even the ones completely unrelated to vlan3 stop working. The only way to recover is to physically unplug router1 (then disable all VRRPs on router2, then plug back in router1, disable all VRRPs there and then re-enable them on router2).

This is the screenshot of a MacOS client connected to vlan3, pinging router1 (10.227.79.1), router2 (10.227.79.2) and internet (8.8.8.8) at the time the issue occurs:

Hosts remain somewhat reachable but most of the packets are lost and if they respond, the response times are hundreds of milliseconds. Accessing either router via Winbox or SSH hangs forever and eventually times out.

For comparison, this is before I enabled vrrp3 and everything is working:

My central switch is a TP-Link TL-SG1016PE. Many VLAN aware devices are connected via tagged/untagged ports. The most important ones are:

router1 (which is a Mikrotik hEX)
2x TP-Link EAP610 access points
a Netgear GS305E
Two proxmox nodes

The proxmox nodes have a VLAN aware bridge (the usual vmbr0) and one of the nodes runs router2 as CHR instance. That instance does not have tag= set, i.e., it obtains access to all VLAN interfaces.

After much trial and error, the issue only seems to appear once I enable vrrp3. However, there is nothing special about vrrp3 or vlan3 I am aware of.

I understand that this "super GAU" could in principle happen in a split brain scenario where both router1/router2 think they are master, announce their VRRP addresses and then switch then sees two hosts with the same MAC, panics and shuts down ports. This would explain why it brings down the entire network, including VLANs and hosts that have nothing to do with VRRP.

However, I do not see how this can happen here. Via vrrp98 I make sure that there is only one dedicated network for the VRRP and conntrack exchange.

At this point I am just looking for brainstorming of possible ideas what the heck this could cause, what could be special about vlan3 and how to debug further. Note that since I am loosing the whole network it is very hard to check logs in real time or use packet sniffers to see what's going on.

(*) This is despite its interfaces (both vlan and vrrp) being assigned to a VRF. The issue looks similar as identified in countless other VRF related bugs in the past: The traffic goes out in one direction but return traffic is discarded since it's received via an interface which is part of a VRF.

Larsa · August 19, 2026, 6:50am

Just guessing here, but given that unrelated VLANs also suffer massive loss/latency, I’d suspect MAC flapping, a loop, or switch port blocking before VRF/conntrack.

I’d check the switch MAC table, especially whether the VRRP MAC starts bouncing between ports, plus any loop/STP protection events and port counters while enabling vrrp3.

divB · August 19, 2026, 7:58am

guessing is good because logic doesnt get me far here

What could be an explanation that enabling VRRP on VLAN3 could create a loop that's not present otherwise? What could be the reason that only enabling VRRP on vlan3 (but as it seems not vlan11, vlan12, vlan4 and others ... which are otherwise absolutely identical) cause this?

As you may suspect, I cannot check the MAC table of the switch because it's one of these "smart managed crap". Similarly, no event log or similar on the TL-SG1016PE.

rplant · August 20, 2026, 1:09am

It may be worth checking that the firewall on the interfaces allows the VRRP communications in.
(default firewall, interface member of LAN is enough)

I found that having a master vrrp vlan that only did vrrp (no traffic) and using it as a group authority for all the other vrrp's worked well for me.

divB · August 20, 2026, 6:02am

Yes I double checked that already. These are my first rules on each side:



/ip firewall filter
add action=accept chain=input comment=VRRP in-interface=vlan98 protocol=vrrp src-address=10.227.2.1
add action=accept chain=input comment="VRRP: conntrack sync" dst-port=8275 protocol=udp

divB · August 20, 2026, 7:57am

Ok, I did one more long debug session with loads of additional information and insights but zero clue WHAT THE HECK ....

I have disconnected everything from the switch except: (a) The internet VLANs (b) Mikrotik hEX (c) One proxmox node (c) The MacBook directly connected to vlan3 . No change
It seems I can perfectly reproduce the issue: One I enable vrrp3 on Router1, shit hits the fan. Sometimes I need to reboot Router1 for the issue to pop up. But I haven't been able to reproduce with any other vrrp interface
While next to the rack, I noticed something (=a fan) getting louder and louder. It was the promox node
I ssh'ed into the proxmox node from Router2 and found that indeed all CPU cores were at 100%. No particular process (i.e., might have been kernel usage)
I temporarily disabled entire Firewall in proxmox (all ebtables and iptables showed empty). No change
Most strange: When the issue occurs, Tx of all VRRP interfaces on Router2 went pretty high, ~4Mbps. By contrast, all counters on Router1 remained zero. Screenshots:

Router2:

Router1:

In case it's useful, log files filtered by VRRP of Router2:

And Router1:

So yes, it does seem that weirdly something on Proxmox node (on which CHR Router2 runs on) starts to hog CPU and causes issues.

With these additional bits in mind, is there any further debug step I could do?

EDIT: One more bit:

7.) Here are the port statistics of the switch:

Port 8 is hEX (Router1), Port 16 is Proxmox node (with CHR; Router2). Port 3 and 6 are wireless access points (EAP610). I am not sure if RxBadPkt are related to this issue or not but I did power cycle the switch in between and afterwards the issue occurred again.

Larsa · August 20, 2026, 8:38am

Great info which makes me suspect the Proxmox bridge/NIC path even more. All cores hitting 100% at the same time as several Mbit/s suddenly show up on the VRRP interfaces really looks suspicious, like some kind of packet storm or L2 loop.

I'd try capturing the traffic on the Proxmox physical NIC and Linux bridge while enabling vrrp3 and check bridge fdb, ip -s link/ethtool -S as well as softirq usage.

Since the TL-SG1016PE can't show the MAC/FDB table, I'd use port mirroring instead and mirror the hEX and Proxmox ports to another port, then capture the traffic with Wireshark while enabling vrrp3. That should let you see if there's a broadcast/multicast storm, duplicated frames or if the VRRP MAC address is showing up from both directions. Additionally, if you can borrow a proper managed switch that exposes the FDB table, that would be even better.

The RxBadPkt counters on port 16 are interesting, but I wouldn't draw too many conclusions from them.
__

Happy hunting!

Ps..
My bet is that it’s some kind of L2 loop or frame reflection in the Proxmox/CHR path.

If the Wireshark capture confirms that, I’d start by simplifying the Proxmox bridge/VLAN3 setup and removing any duplicate L2 paths or incorrect bridge membership.