Page 1 of 1

CHR kernel crash when heavy traffic

Posted: Sun May 06, 2018 6:41 am
by TigerHuang
Hi, I have a CHR running the latest version (v6.42.1) on VMware Esx 6.5 server

I found that if there is a heavy flow in a interface ( >5Gbps ) will cause CHR kernel crash.

on the console will showing ”No irq handler for vector“.

on the Esx server log showing "The CPU has been disabled by the guest operating system. Power off or reset the virtual machine."

Resources to the CHR are:
12 Cores of Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
Memory 2048GB
Network VMware VMXNET 3 x 10 (QLogic Corporation NetXtreme II BCM57810 10 Gigabit Ethernet)

How do I solve this problem ?

Re: CHR kernel crash when heavy traffic

Posted: Tue May 08, 2018 10:05 pm
by bbs2web
We have observed 6.42.1 locking up on KVM with VirtIO drivers as well. Our throughput is considerably less, 1.4Gbps with 4 x Intel 2640v4 cores.

6.41.4 was stable prior to this...

ie: Me too...

PS: I have not observed any messages on the VM console, screen doesn't wake from blanking...

Re: CHR kernel crash when heavy traffic

Posted: Sun May 13, 2018 2:38 am
by karwos
Disable conntrack

Re: CHR kernel crash when heavy traffic

Posted: Wed Oct 17, 2018 9:36 am
by total13
Hello I have also noticed that disabling connection tracking improves stability of CHR. However I need this feature since it runs WebProxy.

What is the root cause for this issue and can it be fixed?

Re: CHR kernel crash when heavy traffic

Posted: Thu Nov 08, 2018 10:54 pm
by Stevenhayes100
I have had 2 crash then automatic reboots on the same version using AHV (KVM)
Mine has been since creating an additional VRF. (only 2 in total!!)
Did you upgrade? Was the later relase more stable?

Thanks.

Re: CHR kernel crash when heavy traffic

Posted: Mon Dec 03, 2018 6:59 pm
by dmayan
I'm having the same problem on CHR 6.42.9

Image

1 gbps of traffic. Running on ESXI 6.7

Any words from Mikrotik?

Re: CHR kernel crash when heavy traffic

Posted: Tue Jan 08, 2019 11:08 pm
by Stevenhayes100
I have moved my CHRs back to ESXi 6.0 and they have been stable since.
Mikrotik support said they would help troubleshoot if I needed once back on ESXi.

Re: CHR kernel crash when heavy traffic

Posted: Mon May 20, 2019 6:41 pm
by abyss
You can have a look to your cpu readyness stats running:

esxtop

command...

If you have more than 20% of %ready, decrease your chr numbers of core to fit better your hardware CPU.
All our chr runs with 2-4 cores better than 10 or 20.

When CPU ready increase over 100%, it crash.

Image

Best regards,
Julien