Community discussions

 
TigerHuang
just joined
Topic Author
Posts: 2
Joined: Sun May 06, 2018 6:19 am

CHR kernel crash when heavy traffic

Sun May 06, 2018 6:41 am

Hi, I have a CHR running the latest version (v6.42.1) on VMware Esx 6.5 server

I found that if there is a heavy flow in a interface ( >5Gbps ) will cause CHR kernel crash.

on the console will showing ”No irq handler for vector“.

on the Esx server log showing "The CPU has been disabled by the guest operating system. Power off or reset the virtual machine."

Resources to the CHR are:
12 Cores of Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
Memory 2048GB
Network VMware VMXNET 3 x 10 (QLogic Corporation NetXtreme II BCM57810 10 Gigabit Ethernet)

How do I solve this problem ?
 
bbs2web
Member Candidate
Member Candidate
Posts: 198
Joined: Sun Apr 22, 2012 6:25 pm
Location: Johannesburg, South Africa
Contact:

Re: CHR kernel crash when heavy traffic

Tue May 08, 2018 10:05 pm

We have observed 6.42.1 locking up on KVM with VirtIO drivers as well. Our throughput is considerably less, 1.4Gbps with 4 x Intel 2640v4 cores.

6.41.4 was stable prior to this...

ie: Me too...

PS: I have not observed any messages on the VM console, screen doesn't wake from blanking...
 
karwos
Frequent Visitor
Frequent Visitor
Posts: 89
Joined: Thu Apr 02, 2015 7:28 pm
Location: Poland

Re: CHR kernel crash when heavy traffic

Sun May 13, 2018 2:38 am

Disable conntrack
 
total13
newbie
Posts: 34
Joined: Fri Jul 08, 2016 2:29 pm

Re: CHR kernel crash when heavy traffic

Wed Oct 17, 2018 9:36 am

Hello I have also noticed that disabling connection tracking improves stability of CHR. However I need this feature since it runs WebProxy.

What is the root cause for this issue and can it be fixed?
 
Stevenhayes100
just joined
Posts: 6
Joined: Sat May 26, 2018 4:22 pm

Re: CHR kernel crash when heavy traffic

Thu Nov 08, 2018 10:54 pm

I have had 2 crash then automatic reboots on the same version using AHV (KVM)
Mine has been since creating an additional VRF. (only 2 in total!!)
Did you upgrade? Was the later relase more stable?

Thanks.
 
dmayan
just joined
Posts: 21
Joined: Sun Nov 10, 2013 9:28 pm

Re: CHR kernel crash when heavy traffic

Mon Dec 03, 2018 6:59 pm

I'm having the same problem on CHR 6.42.9

Image

1 gbps of traffic. Running on ESXI 6.7

Any words from Mikrotik?
 
Stevenhayes100
just joined
Posts: 6
Joined: Sat May 26, 2018 4:22 pm

Re: CHR kernel crash when heavy traffic

Tue Jan 08, 2019 11:08 pm

I have moved my CHRs back to ESXi 6.0 and they have been stable since.
Mikrotik support said they would help troubleshoot if I needed once back on ESXi.
 
abyss
Frequent Visitor
Frequent Visitor
Posts: 67
Joined: Wed Sep 21, 2005 10:51 am

Re: CHR kernel crash when heavy traffic

Mon May 20, 2019 6:41 pm

You can have a look to your cpu readyness stats running:

esxtop

command...

If you have more than 20% of %ready, decrease your chr numbers of core to fit better your hardware CPU.
All our chr runs with 2-4 cores better than 10 or 20.

When CPU ready increase over 100%, it crash.

Image

Best regards,
Julien

Who is online

Users browsing this forum: No registered users and 4 guests