WAN interface freezes after a while in CHR

RouterOS General
1

I’m running a trial version of CHR on a Proxmox VM on a Qotom rack x86 PC and everything works fine except the one issue.
After some period of time (1 or 8 hours) the interface assigned as WAN becomes unresponsive, no traffic on Tx side, just some lonely ARP requests on Rx.

All other ports are working fine — serving DHCP, accessing LAN and other VMs, WinBox works as usual.
There was one time I saw 25% cpu load during the freeze on “networking” process in profiler (VM has 4 vCPU) but that’s not common.
No issues reported in the logs, just a familiar RouterOS experience but with WAN port just being dead.

There are four PCIe ethernet devices passed through to VM and one virtual linux bridge.
I can fix the issue by disabling and enabling that WAN interface manually and after that everything works fine for a several hours before next freeze.

Things I tried:

  1. Disabling DHCP and using static routes
  2. Upgrading from 7.7 to 7.8rc3
  3. Using different routers on uplink side (Chateau LTE12 @ 7.7 and RB2011 @ 6.49.4)
  4. Trying to use different physical ports as WAN interface
  5. Running netwatch to generate constant traffic
  6. Different lease times for DHCP
  7. Changed ARP options on WAN interface
  8. Disabled/enabled auto negotiation and flow control
  9. Checked that ASPM and other power saving measures are disabled on that PCIe ports.
  10. Resetting to stock configuration and using quick set to setup basic DHCP-client and server.

I’m running a basic quick set settings here, nothing special tweaked yet. Support ticket #[SUP-109266].
Maybe I’m missing something important with running CHR in Proxmox?

/interface bridge
    add name=bridge1

/interface ethernet
    set [ find default-name=ether1 ] disable-running-check=no name=edge # virtual bridge in proxmox
    set [ find default-name=ether1 ] name=eth1-wan

/interface list
    add name=WAN
    add name=LAN

/ip pool
    add name=dhcp ranges=10.42.0.1-10.42.0.255

/ip dhcp-server
    add address-pool=dhcp interface=bridge1 name=dhcp1

/interface bridge port
    add bridge=bridge1 interface=ether2
    add bridge=bridge1 interface=ether3
    add bridge=bridge1 interface=ether4
    add bridge=bridge1 interface=edge

/interface list member
    add interface=eth1-wan list=WAN
    add interface=bridge1 list=LAN

/ip address
    add address=10.42.0.1/16 interface=bridge1 network=10.42.0.0

/ip dhcp-client
    add interface=eth1-wan use-peer-dns=no

/ip dhcp-server lease
    add address=10.42.0.100 client-id=1:ce:10:2b:46:ce:83 mac-address=\
    CE:10:2B:46:CE:83 server=dhcp1

/ip dhcp-server network
    add address=10.42.0.0/16 dns-server=10.42.0.53 gateway=10.42.0.1

/ip firewall nat
    add action=masquerade chain=srcnat out-interface-list=WAN
2

Resolved this issue with the help from excellent MikroTik support team.

There is a bug in Intel NIC driver currently in Linux kernel that breaks when there is a wake-on-lan related activity on the network:
https://bugzilla.kernel.org/show_bug.cgi?id=216257

Removing PCI-e passthrough and moving to para-virtualized (VirtIO) interfaces fixed that problem for now.
Also you may disable WoL and power-saving features in BIOS and linux launch arguments:

GRUB_CMDLINE_LINUX_DEFAULT="... pcie_port_pm=off pcie_aspm.policy=performance"
3

Thank you for sharing this issue. I have been troubled by this problem for a very long time and have been unable to find a solution. I am using PVE as the host for CHR, with the I226V passed through to CHR as the WAN port. Do I only need to add these two items to the PVE startup parameters, or is there currently a better solution in CHR? I need to use CHR in passthrough mode. In my network, there is indeed a device that sends WOL signals to activate my host in special circumstances. I have now stopped using this device. Could this also resolve the issue? My English is not very good, and I look forward to your reply. Thank you again for sharing.

4

These two kernel parameters were the only thing I needed at that time (in 2023), could not remember the details, sorry. Moved to UniFi stack and could not be happier since.

5

Thank you for your reply. I tested my WOL device and it did not cause my WAN to freeze, so we may need to look for other reasons. I really appreciate it. My secondary router is also using UniFi, which is indeed excellent—very simple and intuitive, with excellent support as well.