I think this must be the same situation I am seeing with wireguard and it makes sense that it isn’t limited to wireguard but rather anything that requires an ICMP response. Also can confirm that disabling the “send redirects” does nothing in my case.
http://forum.mikrotik.com/t/router-leaking-packets-icmp-marked-for-wireguard-tunnel/167626/1