CHR P10 - Latency creep to disconnect

RouterOS Virtualization
1

Hi,

I’m not a server administrator, so I’m not sure whether this is a server-side issue or something related to MikroTik CHR.

I have a SmokePing VM running to monitor latency to my CHR core gateway. From boot, the CHR shows about 100 µs latency (SmokePing → CHR, both hosted on the same Proxmox server). After a few hours, I see the latency gradually increase to 500 µs, and sometimes up to 1.2 ms.

After this gradual increase in latency, the CHR eventually appears to drop all traffic. I lose access to the router, and all traffic through it stops. The only way to recover is to reboot the CHR VM, after which everything works normally again and latency returns to ~100 µs.

Hardware and Software

  • Proxmox VE: 9.1.2

  • MikroTik CHR: 7.21.3 (P10 license)

  • CHR Resources:

    • 6 vCPUs

    • 8 GB RAM

    • ~11 GB disk

  • Host Server:

    • Dual socket

    • 12 cores / 24 threads per CPU (48 usable cores total)

    • 128 GB RAM

Proxmox VM Configuration (CHR – RAW disk)

balloon: 0
boot: order=scsi0
cores: 6
cpu: host
cpuunits: 2048
memory: 8192
name: core.con.gateway
net0: virtio=BC:24:11:48:00:A9,bridge=vmbr0,firewall=1,queues=6
net1: virtio=BC:24:11:CC:70:CF,bridge=chrbond,firewall=1,queues=6
numa: 0
ostype: l26
scsi0: local-lvm:vm-101-disk-0,iothread=1,size=10368M
scsihw: virtio-scsi-single
sockets: 1

Troubleshooting Already Done

I have tried extensively adjusting VM settings, including:

  • CPU units / weighting

  • VirtIO multiqueue (enabled and disabled)

  • Socket and core layouts

  • NUMA on/off

  • Memory ballooning

I have also:

  • Run CHR as both x86 install and RAW CHR image

  • Rebuilt the CHR multiple times

  • Purchased new P10 licenses to start fresh

Additional Context

This CHR is used as a core router, handling all internal routing and VPN connectivity to our sites. (average 22k pps in and also 22K pps out)

When I first deployed this setup about three months ago, everything worked perfectly. As far as I know, nothing significant has changed, but these issues started appearing around two weeks ago.

At this point, I have exhausted all available options on my side. My next step is to ask the community here and, if necessary, escalate to MikroTik support.

Any help or guidance would be greatly appreciated

2

image
image1493×743 108 KB

where it maxes on the graph - I have to reboot CHR, thats the drops you see then it starts climbing again

3

Interesting. I’ve never tried monitoring latency from two vms on the same box.

My CHR is setup similar, but I have not played with the cpuunits value and I don’t enable the firewall on my virtual network interfaces or change queue settings there.

Also I am just using a typical bridge on the two ports my chr has. It looks like you have some kind of bond running underneath as well.

If it were me I’d turn off the firewall on the virtio interfaces and see if that makes any difference first. If not I would look at maybe getting rid of the bond to simplify things next. It seems like those would be the most likely culprits in my mind not having any first hand experience with this particular issue.

4

ok, by bond, you mean the bridge on the interfaces?

That's just there to show what interfaces are LAN and what are WAN. It's not a real bond of multiple interfaces. Just named dumb with the person who set up the server at first.

I’ll check the firewall option when it dies again.

5

image
image718×330 31.5 KB

After starting a new CHR instance - the issue persists (Latency creep), but it has not dropped / failed yet.

you can see on the graph, the last climb is when I redid the CHR (27th on graph)

6

Failure happened again - made the changes and busy sending a support.rif to mik.

7

Hi all,

I was on a week’s leave, so I’m returning to this thread.

Support provided a few suggestions and I made the recommended changes. However, after a couple of days the VM crashed again with the same latency creep.

What we tried based on MikroTik’s feedback:

“From your provided supout with crash we see that the virtio_net driver has crashed due to an unexpected PCI hotplug event. RouterOS does not expect PCI hotplug events. Please disable ACPI hotplug for PCI and Network in your Proxmox VM settings. Also change the machine type from i440FX to q35.”

Unfortunately, this did not resolve the issue.

I have sent additional .rif files to support and will keep you updated.

8

Still no feedback yet from Mikrotik. Have had multiple crashes, and each time I haven’t sent new rif files.

9

Mikrotik has send feedback and looks like there is a problem with Virtio_Net drivers for CHR. which will be fixed in 7.23. I will update to 7.23 and test it.

10

So far, so good - 7.23 is a lot more stable on CHR environment. The spike at the beginning of the graph is 7.22, and once 7.23 was installed, the graph dropped, as you can see. It spiked but then normalised again almost immediately.

image
image176×201 4.69 KB

11

Survived ± 15Hrs with no problems - Latency graph looks good.

image
image445×284 52.9 KB

I will marked solved in a few days just to cover testing period.

12

not resolved. Happened again. rif back to mikrotik

13

not getting actual answers or guidance. I have to resort to using physical equipment because CHR is not stable.

14

What if it’s your hardware?

You stated earlier that the problem only started recently for you. It’s possible, maybe even likely that it’s something peculiar to your situation. It might be something related to your hardware or it might be something new in your environment.

I have not seen anyone else complain of this type of issue with CHR, so I wouldn’t necessarily expect easy answers. If it were me I’d try reproducing the issue on an entirely different Proxmox server. If nothing else it might rule out hardware or narrow down the range of possibilities.

We had a problem with one of our Proxmox servers almost 3 years ago now when the AMD CPU went bad. That was the first CPU I’ve ever had go bad. The server actually operated for some time with the bad CPU. We eventually decided we needed to reboot and the server never booted back up. That’s when we actually realized we had a hardware problem and the CPU was bad. AMD did eventually replace it for us, but we had to buy a new one anyway to get back up and running.

15

Hi Kevo,

Thanks for your reply.

I’ve tried everything I can think of at this point. Unfortunately, I’m unable to set up a new Proxmox server as this environment is cloud-hosted and not running on our own physical equipment. I’m also unable to log a ticket with Proxmox since our company does not have a paid license, so I’m somewhat stuck.

What’s puzzling is that we have two CHRs—a main gateway and a VPN concentrator. The VPN CHR appears to be running well, while the main CHR drops every couple of hours throughout the day. I’ve already deployed a new CHR and worked with MikroTik to verify the CHR image, configuration, and the Proxmox VM settings, but the issue persists. The two CHRs are identical - only difference is traffic.

At this stage, the only option I see for long-term stability is to colocate and move to physical (CCR) equipment to do our routing.

I’m currently handling all the fixes, maintenance, and troubleshooting on my own, in addition to my regular workload, which makes this quite challenging.

16

Yes, I can see how that would be challenging running on someone else’s hardware. Might be worth talking to the hosting provider and seeing if you can migrate your Proxmox to another physical machine. I wonder if they could tell you if there were any changes to the hosting environment around the time that the problem started.

17

I can try this yes, only thing that concerns me, is our P10 licenses. I doubt I will be allowed to do it and purchase two more for each CHR if we move physical machines.

18

I think migrating Proxmox to another machine at the provider shouldn’t affect the license. Maybe someone can confirm or you could run it by Mikrotik support, but I think that should be doable without any trouble.

19

I would have to find out from Mikrotik how that will work. Thanks for the advice. I’ll see what I can do once we have completed colocation.