CCR1036 Kernel Failure (IBM and HP Servers Too)

drnicolau · August 14, 2013, 1:00am

Hi!

For months we are having a problem we can not yet solve. I use this forum because I can not resolve it with support@mikrotik.com they only answered “please use latest software version”

The problem is that the router stops working and needs to be restarted to continue working.
We observed that the error in the log says “System rebooted Because of kernel failure”.

We tested with the following equipment:

IBM x3250 server with routeros versions 5.11, 5.22, 5.25, and all the latest versions 6 to 6.22.
HP Proliant Server 1100m7 with routeros versions 5.11, 5.22, 5.25, and all the latest versions 6 to 6.22.
We buy these days one CloudCore ccr1036 because we think the aforementioned servers were small.

Fault:
In all cases the failure occurs randomly, operates two days, then run 1 day, then runs 8 days, then fails for 2 or 3 times per day.
In the latter case the ccr1036 CloudCore worked for a few hours and then had to reboot to continue working, and at the end of the day it did 2 times.

- Our configuration:
1200 pppoe tunnels LAN port
bandwidth control for PCQ
1 Wan (70mb)
Firewall Rules = 22 rules in the fordward + 1 chain with 36 rules
12 rules in the Input

Any suggestion?
Thanks.

jcr77 · October 9, 2013, 1:43pm

Y have the same problem.
My ccr sometimes restarts, and other times it is frozen, so I have to disconnecting the power to restart.

Anyone have any suggestion?

raz · October 9, 2013, 5:15pm

What kind of Config you’re using? I have too a CCR and it never got frozen.

drnicolau · October 9, 2013, 9:15pm

hi, i resolved it upgrading the routeros to v6.3 rc 1 and upgrade the bootloader.
it is working since 47 days without any problems…

regards.

Ramiro Nicolau

antenman · October 14, 2013, 12:41pm

We’re facing the same problem with CCR1036-12G-4S

The router was received with ROS 6.2 and we upgraded to 6.4 but the problem remains…

Sometimes only the power on/off helps…

The log line

system,error,critical System rebooted because of kernel failure

[admin@WAN_ROUTER] > system routerboard print
routerboard: yes
model: CCR1036-12G-4S
serial-number: 42A10274D602
current-firmware: 3.08
upgrade-firmware: 3.09

Console output (in the moment when rouer hangs)

MikroTik 6.4
WAN_ROUTER Login:
MikroTik 6.4
WAN_ROUTER Login:
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Pid: 0, comm: swapper/14, CPU: 14
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
r0 : 0xfffffff5000e096c r1 : 0xfffffff5000e0848 r2 : 0x000006cac6e97ccf
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
hrtimer: interrupt took 30948941 ns
Unable to handle kernel NULL pointer dereference
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
r3 : 0x0000000000000000 r4 : 0xfffffe007c385978 r5 : 0xfffffe00736cfc04
r6 : 0xfffffe00006e2540 r7 : 0x0000000000000000 r8 : 0x0000000000000000
r9 : 0x0000000000000000 r10: 0x0000000000000000 r11: 0xfffffff5000e0900
r12: 0x000001f4ffba0000 r13: 0x0000000000000000 r14: 0x0000000000000000
r15: 0x0000000000000000 r16: 0x666666666666679f r17: 0x0000000000000040
r18: 0x0000000000000080 r19: 0xfffffe00736cfa18 r20: 0xfffffe007c49fed0
r21: 0xfffffe007c49ff10 r22: 0xfffffe007c49fed8 r23: 0xfffffe007c49ff18
r24: 0xfffffe007c49fee0 r25: 0xfffffe007c49ff80 r26: 0xfffffe007c49ff40
r27: 0xfffffe007c49ffa0 r28: 0xfffffe007c49ffa8 r29: 0xfffffe007c49ffb0
r30: 0xfffffe0000540040 r31: 0xfffffe0000540080 r32: 0x0000000000000000
r33: 0x0000000000000000 r34: 0x0000000000000000 r35: 0x0000000000000000
r36: 0x0000000000000000 r37: 0x0000000000000000 r38: 0x0000000000000002
r39: 0xfffffff5000e0878 r40: 0x0000000000020000 r41: 0xfffffe007c49fea0
r42: 0xfffffff7000a8e30 r43: 0xfffffe007c49fec8 r44: 0x0000000000000000
r45: 0x000001f4ffba0000 r46: 0x0000000000000000 r47: 0xfffffe007c49ff30
r48: 0xfffffff5000e0848 r49: 0x000000000000000e r50: 0xfffffe007c49ff30
r51: 0xfffffe0000555ce8 r52: 0x0000000000000000 tp : 0x000001f4ffba0000
sp : 0xfffffe007c49ffb0 lr : 0xfffffff700029cd8
pc : 0xfffffff700029ce0 ex1: 1 faultnum: 18

Starting stack dump of tid 0, pid 0 (swapper/14) on cpu 14 at cycle
897213141846
8
frame 0: 0xfffffff700029ce0 cpu_idle+0x140/0x2b8 (sp 0xfffffe007c49ffb0)
frame 1: 0xfffffff700029cd8 cpu_idle+0x138/0x2b8 (sp 0xfffffe007c49ffb0)
Stack dump complete
—[ end trace 547095b6e05bca11 ]—
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8
Unable to handle kernel NULL pointer dereference
at virtual address 0x0000000000000000, pc cpu_idle+0x140/0x2b8

Pid: 0, comm: swapper/24, CPU: 24
r0 : 0xfffffff50018096c r1 : 0xfffffff500180848 r2 : 0x000006cacb11c314
r3 : 0x0000000000000000 r4 : 0xfffffe007c3892f8 r5 : 0xfffffe0073e2fc04
r6 : 0x0000000000000000 r7 : 0x0000000000000000 r8 : 0x0000000000000000
r9 : 0x0000000000000000 r10: 0x0000000000000000 r11: 0xfffffff500180900

Pid: 0, comm: swapper/13, CPU: 13
r0 : 0xfffffff5000d096c r1 : 0xfffffff5000d0848 r2 : 0x000006cac9e1df98
r3 : 0x0000000000000000 r4 : 0xfffffe007c3853b8 r5 : 0xfffffe0073e8fc04
r6 : 0x0000000000000000 r7 : 0x0000000000000000 r8 : 0x0000000000000000
r9 : 0x0000000000000000 r10: 0x0000000000000000 r11: 0xfffffff5000d0900
r12: 0x000001f4ffb90000 r13: 0x0000000000000000 r14: 0x0000000000000000
r15: 0x0000000000000000 r16: 0x666666666666679f r17: 0x0000000000000040
r18: 0x0000000000000080 r19: 0xfffffe0073e8fa18 r20: 0xfffffe007c48fed0
r21: 0xfffffe007c48ff10 r22: 0xfffffe007c48fed8 r23: 0xfffffe007c48ff18
r24: 0xfffffe007c48fee0 r25: 0xfffffe007c48ff80 r26: 0xfffffe007c48ff40
r27: 0xfffffe007c48ffa0 r28: 0xfffffe007c48ffa8 r29: 0xfffffe007c48ffb0
r30: 0xfffffe0000540040 r31: 0xfffffe0000540080 r32: 0x0000000000000000
r33: 0x0000000000000000 r34: 0x0000000000000000 r35: 0x0000000000000000
r36: 0x0000000000000000 r37: 0x0000000000000000 r38: 0xfffffe007c48fea8
r39: 0xfffffff5000d0848 r40: 0xfffffff5000d06c0 r41: 0xfffffff5000d0860
r42: 0x000000000000000d r43: 0xfffffff700082318 r44: 0xfffffe007c48fec8
r45: 0xfffffff5000d0848 r46: 0x0000000000000000 r47: 0xfffffe007c48ff30
r48: 0xfffffff5000d0848 r49: 0x000000000000000d r50: 0xfffffe007c48ff30
r51: 0xfffffe0000555ce8 r52: 0x0000000000000000 tp : 0x000001f4ffb90000
sp : 0xfffffe007c48ffb0 lr : 0xfffffff700029cd8
pc : 0xfffffff700029ce0 ex1: 1 faultnum: 18

Starting stack dump of tid 0, pid 0 (swapper/13) on cpu 13 at cycle
897293151005
8
frame 0: 0xfffffff700029ce0 cpu_idle+0x140/0x2b8 (sp 0xfffffe007c48ffb0)
frame 1: 0xfffffff700029cd8 cpu_idle+0x138/0x2b8 (sp 0xfffffe007c48ffb0)
Stack dump complete
—[ end trace 547095b6e05bca12 ]—
Kernel panic - not syncing: Kernel page fault running the idle task!

Starting stack dump of tid 0, pid 0 (swapper/13) on cpu 13 at cycle
897293160762
1
frame 0: 0xfffffff7004efe78 dump_stack+0x0/0x20 (sp 0xfffffe007c48fbf8)
frame 1: 0xfffffff7004e90b8 panic+0x168/0x3a8 (sp 0xfffffe007c48fbf8)
frame 2: 0xfffffff700038598 do_page_fault+0xa68/0xc78 (sp
0xfffffe007c48fca0)
frame 3: 0xfffffff7004f0128 handle_interrupt+0x1f0/0x208 (sp
0xfffffe007c48fd4
0)
<interrupt 18 while in kernel mode>
frame 4: 0xfffffff700029ce0 cpu_idle+0x140/0x2b8 (sp 0xfffffe007c48ffb0)
frame 5: 0xfffffff700029cd8 cpu_idle+0x138/0x2b8 (sp 0xfffffe007c48ffb0)
Stack dump complete

Pid: 0, comm: swapper/9, CPU: 9
r0 : 0xfffffff50009096c r1 : 0xfffffff500090848 r2 : 0x000006cacb13b0cd
r3 : 0x0000000000000000 r4 : 0xfffffe007c383cb8 r5 : 0xfffffe407dc3fc04
r6 : 0x0000000000000000 r7 : 0x0000000000000000 r8 : 0x0000000000000000
r9 : 0x0000000000000000 r10: 0x0000000000000000 r11: 0xfffffff500090900
r12: 0x000001f4ffb50000 r13: 0x0000000000000000 r14: 0x0000000000000000
r15: 0x0000000000000000 r16: 0x666666666666679f r17: 0x0000000000000040
r18: 0x0000000000000080 r19: 0xfffffe407dc3fa18 r20: 0xfffffe007c44fed0
r21: 0xfffffe007c44ff10 r22: 0xfffffe007c44fed8 r23: 0xfffffe007c44ff18
r24: 0xfffffe007c44fee0 r25: 0xfffffe007c44ff80 r26: 0xfffffe007c44ff40
r27: 0xfffffe007c44ffa0 r28: 0xfffffe007c44ffa8 r29: 0xfffffe007c44ffb0
r30: 0xfffffe0000540040 r31: 0xfffffe0000540080 r32: 0x0000000000000000
r33: 0x0000000000000000 r34: 0x0000000000000000 r35: 0x0000000000000000
r36: 0x0000000000000000 r37: 0x0000000000000000 r38: 0xfffffe007c44fea8
r39: 0xfffffff500090848 r40: 0xfffffff5000906c0 r41: 0xfffffff500090860
r42: 0x0000000000000009 r43: 0xfffffff700082318 r44: 0xfffffe007c44fec8
r45: 0xfffffff500090848 r46: 0x0000000000000000 r47: 0xfffffe007c44ff30
r48: 0xfffffff500090848 r49: 0x0000000000000009 r50: 0xfffffe007c44ff30
r51: 0xfffffe0000555ce8 r52: 0x0000000000000000 tp : 0x000001f4ffb50000
sp : 0xfffffe007c44ffb0 lr : 0xfffffff700029cd8
pc : 0xfffffff700029ce0 ex1: 1 faultnum: 18

panicSaver: dumping panic to flash
flash: prg 73
tile program flash offset:00073000 size:00001000
flash: prg 74
tile program flash offset:00074000 size:00001000

drnicolau · October 14, 2013, 1:58pm

Try to upgrade the bootloader.