Community discussions

MikroTik App
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

ax3-us kernel panic every few hours

Thu Jan 12, 2023 5:06 pm

I was lucky enough to receive two ax3's which I am using as AP's. Every few hours I experience both devices randomly restarting with "kernel failure in previous boot" message in syslog. message from autosupport.rif off both devices:
Device 1:
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
Mem abort info:
  ESR = 0x96000007
  EC = 0x25: DABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
Data abort info:
  ISV = 0, ISS = 0x00000007
  CM = 0, WnR = 0
user pgtable: 64k pages, 48-bit VAs, pgdp=000000004e570400
[0000000000000028] pgd=000000004fb40003, pud=000000004fb40003, pmd=000000004fb80003, pte=0000000000000000
Internal error: Oops: 96000007 [#1] SMP
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      5.6.3 #2
Hardware name: c53 ax (DT)
pstate: 80000005 (Nzcv daif -PAN -UAO)
pc : dp_tx_me_send_convert_ucast+0x2644/0x2f70 [wifi_3_0@0xffff800003ec0000]
lr : dp_tx_comp_handler+0x5b8/0x10a0 [wifi_3_0@0xffff800003ec0000]
sp : ffff80000800fc50
x29: ffff80000800fc50 x28: ffff00000e805d90 
x27: 00000000000000ab x26: ffff000034ecc000 
x25: ffff000035761f00 x24: 0000000000000013 
x23: 0000000000000000 x22: ffff80000800fd4c 
x21: ffff000034ecc000 x20: ffff000035760000 
x19: ffff00000e83cd00 x18: 0000000000000000 
x17: 0000000000000000 x16: 0000000000000000 
x15: 0000000000000000 x14: 0000000a00000000 
x13: 0000000000000000 x12: 0000000000000045 
x11: ffff80000800fbe0 x10: ffff800003f5b528 
x9 : 00000000ffffffd8 x8 : ffff80000800fbe0 
x7 : 00000000000d15c8 x6 : 0000000000000045 
x5 : 0000000000000000 x4 : ffff80000800fba0 
x3 : ffff00000e6ec400 x2 : ffff80000800fd4c 
x1 : 0000000000000000 x0 : 0000000000000000 
Call trace:
 dp_tx_me_send_convert_ucast+0x2644/0x2f70 [wifi_3_0@0xffff800003ec0000]
 dp_tx_comp_handler+0x5b8/0x10a0 [wifi_3_0@0xffff800003ec0000]
 dp_srng_deinit+0x308/0x670 [wifi_3_0@0xffff800003ec0000]
 hif_print_napi_stats+0x164/0x460 [qca_ol@0xffff8000040e0000]
 _stext+0x4350a4/0x51481c
 _stext+0x435380/0x51481c
 _stext+0x1a8/0x51481c
 _stext+0x1f0cc/0x51481c
 _stext+0x5619c/0x51481c
 _stext+0x212d38/0x51481c
 _stext+0x19f8/0x51481c
 _stext+0x5168/0x51481c
 _stext+0x3fde8/0x51481c
 _stext+0x3ff30/0x51481c
 _stext+0x513d98/0x51481c
 _sinittext+0x99c/0x23458
 _sinittext+0xd48/0x23458
Code: a8c47bfd d65f03c0 f9400660 f9400e83 (f9401401) 
---[ end trace fa0a006d41482683 ]---
Kernel panic - not syncing: Fatal exception in interrupt
SMP: stopping secondary CPUs
--- rebooted
--- 2
Device 2 (different day)
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
Mem abort info:
  ESR = 0x96000007
  EC = 0x25: DABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
Data abort info:
  ISV = 0, ISS = 0x00000007
  CM = 0, WnR = 0
user pgtable: 64k pages, 48-bit VAs, pgdp=000000004db41a00
[0000000000000028] pgd=000000004fa70003, pud=000000004fa70003, pmd=0000000041930003, pte=0000000000000000
Internal error: Oops: 96000007 [#1] SMP
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      5.6.3 #2
Hardware name: c53 ax (DT)
pstate: 80000005 (Nzcv daif -PAN -UAO)
pc : dp_tx_me_send_convert_ucast+0x2644/0x2f70 [wifi_3_0@0xffff800003db0000]
lr : dp_tx_comp_handler+0x5b8/0x10a0 [wifi_3_0@0xffff800003db0000]
sp : ffff80000800fc50
x29: ffff80000800fc50 x28: ffff00000efc5d90 
x27: 0000000000000102 x26: ffff000033288000 
x25: ffff000034f19f00 x24: 0000000000000017 
x23: 0000000000000000 x22: ffff80000800fd4c 
x21: ffff000033288000 x20: ffff000034f18000 
x19: ffff00000eff7000 x18: 0000000000000000 
x17: 0000000000000000 x16: 0000000000000000 
x15: 0000000000000000 x14: 0000000000000000 
x13: 0000000000000000 x12: 0000000000000045 
x11: ffff80000800fbe0 x10: ffff800003e4b528 
x9 : 00000000ffffffd8 x8 : ffff80000800fbe0 
x7 : 00000000000d1d48 x6 : 0000000000000045 
x5 : 0000000000000000 x4 : ffff80000800fba0 
x3 : ffff00000321a400 x2 : ffff80000800fd4c 
x1 : 0000000000000000 x0 : 0000000000000000 
Call trace:
 dp_tx_me_send_convert_ucast+0x2644/0x2f70 [wifi_3_0@0xffff800003db0000]
 dp_tx_comp_handler+0x5b8/0x10a0 [wifi_3_0@0xffff800003db0000]
 dp_srng_deinit+0x308/0x670 [wifi_3_0@0xffff800003db0000]
 hif_print_napi_stats+0x164/0x460 [qca_ol@0xffff800003fd0000]
 _stext+0x4350a4/0x51481c
 _stext+0x435380/0x51481c
 _stext+0x1a8/0x51481c
 _stext+0x1f0cc/0x51481c
 _stext+0x5619c/0x51481c
 _stext+0x212d38/0x51481c
 _stext+0x19f8/0x51481c
 _stext+0x5168/0x51481c
 _stext+0x3fde8/0x51481c
 _stext+0x3ff30/0x51481c
 _stext+0x513d98/0x51481c
 _sinittext+0x99c/0x23458
 _sinittext+0xd48/0x23458
Code: a8c47bfd d65f03c0 f9400660 f9400e83 (f9401401) 
---[ end trace fb5ab027f5b95338 ]---
Kernel panic - not syncing: Fatal exception in interrupt
SMP: stopping secondary CPUs
--- rebooted
I have since disabled wifi3 (which is a child of wifi1) interface to see if they stop but figured I'd post here incase anyone else experiences.
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: ax3-us kernel panic every few hours

Fri Jan 13, 2023 1:49 am

Few hours after applying 7.7 that was released, panic occurred again, this time w/o any virtual interfaces, since 7.7 is now released, I opened case SUP-104492 :
Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
Mem abort info:
  ESR = 0x96000007
  EC = 0x25: DABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
Data abort info:
  ISV = 0, ISS = 0x00000007
  CM = 0, WnR = 0
user pgtable: 64k pages, 48-bit VAs, pgdp=000000004eaa1e00
[0000000000000028] pgd=000000004ff90003, pud=000000004ff90003, pmd=000000004fff0003, pte=0000000000000000
Internal error: Oops: 96000007 [#1] SMP
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O      5.6.3 #2
Hardware name: c53 ax (DT)
pstate: 80000005 (Nzcv daif -PAN -UAO)
pc : dp_tx_me_send_convert_ucast+0x2644/0x2f70 [wifi_3_0@0xffff800003230000]
lr : dp_tx_comp_handler+0x5b8/0x10a0 [wifi_3_0@0xffff800003230000]
sp : ffff80000800fc50
x29: ffff80000800fc50 x28: ffff00000f1b5d90 
x27: 0000000000000015 x26: ffff000031b6c000 
x25: ffff000034759f00 x24: 000000000000003a 
x23: 0000000000000000 x22: ffff80000800fd4c 
x21: ffff000031b6c000 x20: ffff000034758000 
x19: ffff00000f1f6e80 x18: 0000000000000000 
x17: 0000000000000000 x16: 0000000000000000 
x15: 0000000000000000 x14: 08e70c0021dd0000 
x13: ffffffffffffff00 x12: 0000000000000045 
x11: ffff80000800fbe0 x10: ffff8000032cb530 
x9 : 00000000ffffffd8 x8 : ffff80000800fbe0 
x7 : 0000000001f22c92 x6 : 0000000000000045 
x5 : 0000000000000000 x4 : ffff80000800fba0 
x3 : ffff00000e546c00 x2 : ffff80000800fd4c 
x1 : 0000000000000000 x0 : 0000000000000000 
Call trace:
 dp_tx_me_send_convert_ucast+0x2644/0x2f70 [wifi_3_0@0xffff800003230000]
 dp_tx_comp_handler+0x5b8/0x10a0 [wifi_3_0@0xffff800003230000]
 dp_srng_deinit+0x308/0x670 [wifi_3_0@0xffff800003230000]
 hif_print_napi_stats+0x164/0x460 [qca_ol@0xffff800003970000]
 _stext+0x4350a4/0x51481c
 _stext+0x435380/0x51481c
 _stext+0x1a8/0x51481c
 _stext+0x1f0cc/0x51481c
 _stext+0x5619c/0x51481c
 _stext+0x212d38/0x51481c
 _stext+0x19f8/0x51481c
 _stext+0x5168/0x51481c
 _stext+0x3fde8/0x51481c
 _stext+0x3ff30/0x51481c
 _stext+0x513d98/0x51481c
 _sinittext+0x99c/0x23458
 _sinittext+0xd48/0x23458
Code: a8c47bfd d65f03c0 f9400660 f9400e83 (f9401401) 
---[ end trace 1c25c2d55eb8e38b ]---
Kernel panic - not syncing: Fatal exception in interrupt
SMP: stopping secondary CPUs
--- rebooted
--- 1
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: ax3-us kernel panic every few hours

Tue Jan 24, 2023 6:52 pm

7.8 also affected, Also tried removing wireless access list, best uptime I've had on an device was 14 hours so far, seems to mostly happen when endpoint associates to AP.
 
Elitecrewman
just joined
Posts: 1
Joined: Thu Feb 02, 2023 6:56 am

Re: ax3-us kernel panic every few hours

Thu Feb 02, 2023 6:57 am

I am having the same issue and have seen the same pattern (endpoint connecting to AP).
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: ax3-us kernel panic every few hours

Fri Feb 03, 2023 6:41 am

I got a response from support last weekend which provided me a new build and so far I've been stable. recommend sending an email to support with supout file from crash and refrence my case #.

Hopefully the fix i got makes it to next beta
 
aglabs
newbie
Topic Author
Posts: 39
Joined: Mon Dec 28, 2020 1:05 am

Re: ax3-us kernel panic every few hours

Tue Feb 07, 2023 5:43 am

Just an update, the firmware provided via support seems to greatly reduce the kernel panic's however it doesn't eliminate them. they still occur every few days. Still working with support to improve.

Who is online

Users browsing this forum: No registered users and 15 guests