RB5009 7.24.5: inner payload corruption over WireGuard (SSH “MAC incorrect”, TLS failures), fixed by downgrading to 7.23.8

RouterOS General
1

Hi all,

I wanted to share a reproducible data-corruption issue I tracked down on an RB5009UG+S+ running RouterOS 7.24.5, in case anyone else is seeing it.

Setup

Home LAN hosts --- CSS318 (SwOS) --- RB5009UG+S+ (7.24.5, arm64) === WireGuard (MTU 1420, IPv4 underlay) === CHR (version: 7.24.5) --- Proxmox Backup Server (VM on same host as CHR)
  • Home hosts are 1500 MTU, on two VLANs routed by the RB5009
  • MSS clamp (change-mss clamp-to-pmtu) on both ends for traffic in/out of the WG interface
  • Default fasttrack rule present

Symptom
Large transfers from the home LAN to the datacenter over the RB5009's WireGuard tunnel randomly fail, roughly once every 5-8 GB:

  • Proxmox backups fail with "backup write data failed: command error: protocol canceled". The PBS side logs "error reading a body from connection", and a packet capture shows PBS sending a TLS alert followed by an RST.
  • scp of a 4 GB random file fails partway with "Connection reset by peer". The receiving sshd logs:
sshd-session: Corrupted MAC on input.
ssh_dispatch_run_fatal: Connection from user root x.x.x.x: message authentication code incorrect

So bytes are arriving changed, and TCP isn't catching it (TcpInCsumErrors on the receiver did not increase during the failing runs). My understanding is that after a successful WireGuard decrypt the inner packet is treated as checksum-verified, so corruption that happens before encryption gets through untouched until the TLS/SSH layer checks it.

Isolation (4 GB random file via scp, repeated 3-5x per path)

Path                                                      Result
Home host A -> RB5009 WG -> CHR -> PBS                    FAIL (MAC incorrect)
Home host B (different VLAN) -> RB5009 WG -> CHR -> PBS   FAIL (MAC incorrect)
Home host -> RB5009 inter-VLAN routing -> home host       PASS 3/3 (no tunnel)
Datacenter host -> PBS (no tunnel)                        PASS 3/3
Home host with its own WG peer -> CHR -> PBS              PASS 3/3 (RB5009 only forwards outer UDP)

So only traffic that the RB5009 itself encrypts into WireGuard gets corrupted. Plain routing through the RB5009 is clean, and the CHR's decrypt/forwarding is clean with another peer.

Things that did NOT fix it

  • Excluding WG traffic from fasttrack (accept established/related in/out the WG interface before the fasttrack rule)
  • Upgrading RouterBOOT firmware from 7.16.2 to 7.24.5 to match RouterOS
  • MTU/MSS: PMTUD works correctly on the path, and the MSS clamp makes no difference

What DID fix it
Downgrading the RB5009 to long-term 7.23.8. After the downgrade: 5/5 clean 4 GB passes (20 GB), where 7.24.5 failed within 1-2 passes every time.

Repro

dd if=/dev/urandom of=rand.bin bs=1M count=4096
for i in 1 2 3 4 5; do scp rand.bin user@<host-behind-tunnel>:/tmp/rand.bin || break; done
# then check the receiver: journalctl | grep -i 'authentication code incorrect'

Has anyone else seen this on 7.24.x, on the RB5009 or other arm64 boards? Happy to test fixes or provide more detail.

2

In this instance, If you haven't already done so, I would recommend contacting Mikrotik support.

Log into Atlassian - Support

(Also can email support@mikrotik.com )

3

@frajdej Thank you for the detailed report.

We tried to reproduce it in the lab with your setup on 7.24.5:

  • WireGuard, MTU 1420, IPv4 underlay
  • MSS clamp-to-pmtu on both ends, default fasttrack rule
  • routed LANs on both sides, hosts at MTU 1500
  • traffic sent from the encrypting side, the same direction as your backups

We moved 80 GiB, every 2 GiB chunk was SHA-256 checked, and none of the 80 chunks was corrupted. At your rate of about one failure per 5–8 GB we would have expected 10 or more hits, so the generic WireGuard and fasttrack path looks fine.

To narrow it down, could you check these on the RB5009 (7.24.5)?

  1. Does the corruption stop with the fasttrack rule disabled?
  2. Does it stop if traffic enters the RB5009 on a plain port, without VLAN or bridge?
  3. Does lowering the WireGuard MTU to 1380 change anything?

Also, please open a support ticket at support@mikrotik.com. Attach a supout.rif from the RB5009, taken on 7.24.5 right after a failed transfer, and your config export. Then we can look at it on the hardware.

4

Thank you for looking into it. I did try it with the fasttrack rule disabled and excluding the WireGuard interface from hitting the fasttrack rule and the issue persisted.

I will upgrade the device back to 7.24.5 next week and try to replicate the issue and will open a ticket with the supout.rif then.

5

@sergejs I was able to reproduce the issue and opened MikroTik ticket SUP-227413