[SOLVED] Kernel Oops when adding a comment to an interface

leonset · May 6, 2009, 9:20am

Hello,

I’ve seen a curious problem… When I change the comment of a Trunk interface, or to any of the VLAN or VRRP interfaces which depend on it, the whole trunk stops responding (no ping, no traffic, I can’t even get the MAC address of the iface). The serial console gives some more information:

[admin@ZIPI] /interface> Oops[#1]:
Cpu 0
$ 0   : 00000000 00000000 00010000 00010000
$ 4   : 00000000 c0a01eb0 c0730b80 c041d090
$ 8   : 00000003 00400000 00001d06 00008000
$12   : 00000000 c0379b80 c1cfcd48 00000001
$16   : c0731380 c0a01eb0 7fe6b5c0 80187802
$20   : 2abc3d80 2abbece4 00000000 7fe6b890
$24   : 00000000 c041c1d0
$28   : c0a00000 c0a01ea0 0afe0201 c041c3e4
Hi    : 000000e4
Lo    : e1c00000
epc   : c041c440 cleanup_module+0x434/0x9d4 [vrrp]
    Tainted: P
ra    : c041c3e4 cleanup_module+0x3d8/0x9d4 [vrrp]
Status: 1000f203    KERNEL EXL IE
Cause : 00800008
BadVA : 000001b8
PrId  : 00019374 (MIPS 24K)
Process vrrp (pid: 849, threadinfo=c0a00000, task=c1c1e2c0, tls=00000000)
Stack : c0251718 c0a01eb0 00000000 00000000 76727270 30000000 00000000 00000000
        00000000 00000000 7fe6b5c0 c0947700 7fe6b5c0 c01915a8 c14f6700 00000000
        c1c19dc0 00000001 c01803ec c01803ec c0947700 00000004 7fe6b5c0 c0191618
        c1c19dc0 c14f6700 00000000 c0181dec 00000000 00000028 7fe6b5c0 100024dc
        10002460 00000001 c010dcf0 c010dcf0 00000001 2ab25054 00000002 00000000
        ...
Call Trace:
[<c041c440>] cleanup_module+0x434/0x9d4 [vrrp]


Code: 00431024  10400008  8e040070 <c08201b8> 2442ffff  e08201b8  1040015b  00000000  0810712d
---[ end trace f5b7679711db6bdd ]---
vrrp: add vrrp0 25 2
vrrp: vrrp0 owns vlan10
vrrp: register_netdev failed
vrrp: open vrrp0
vrrp: slave lost
vrrp: add vrrp1 24 1
vrrp: vrrp1 owns vlan8
vrrp: register_netdev failed
vrrp: open vrrp1
vrrp: slave lost
vrrp: remove vrrp1 0 0
CPU 0 Unable to handle kernel paging request at virtual address 000001b8, epc == c041c440, ra == c041c3e4
Oops[#2]:
Cpu 0
$ 0   : 00000000 00000000 00010000 00010000
$ 4   : 00000000 c1a6deb0 c041d090 c041d090
$ 8   : 0000000a c041cae8 00000000 00000000
$12   : 10020000 00000000 c1f17000 00000000
$16   : c0730b80 c1a6deb0 7fa92000 80187802
$20   : 2abc3d80 2abbece4 00000000 7fa922d0
$24   : 00000000 c041c1d0
$28   : c1a6c000 c1a6dea0 0afe0101 c041c3e4
Hi    : 00000063
Lo    : f1454000
epc   : c041c440 cleanup_module+0x434/0x9d4 [vrrp]
    Tainted: P      D
ra    : c041c3e4 cleanup_module+0x3d8/0x9d4 [vrrp]
Status: 1000f203    KERNEL EXL IE
Cause : 00800008
BadVA : 000001b8
PrId  : 00019374 (MIPS 24K)
Process vrrp (pid: 1153, threadinfo=c1a6c000, task=c05dde68, tls=00000000)
Stack : c0251718 c1a6deb0 00000000 00000000 76727270 31000000 00000000 00000000
        00000000 00000000 7fa92000 c056c080 7fa92000 c01915a8 c0822e80 00000000
        c1fbb370 00000001 c01803ec c01803ec c056c080 00000004 7fa92000 c0191618
        c1fbb370 c0822e80 00000000 c0181dec 00000000 00fe0000 7fa92000 10009cdc
        10009c60 00000001 c010dcf0 c010dcf0 10004458 7fae53a8 7fae589c 00000000
        ...
Call Trace:
[<c041c440>] cleanup_module+0x434/0x9d4 [vrrp]


Code: 00431024  10400008  8e040070 <c08201b8> 2442ffff  e08201b8  1040015b  00000000  0810712d
---[ end trace f5b7679711db6bdd ]---
eth0: phy speed: 100, fdx: 1
vrrp: add vrrp0 25 2
vrrp: vrrp0 owns vlan10
vrrp: register_netdev failed
vrrp: open vrrp0
vrrp: slave lost
vrrp: add vrrp1 24 1
vrrp: vrrp1 owns vlan8
vrrp: register_netdev failed
vrrp: open vrrp1
vrrp: slave lost

Then, when I disable the trunk interface, I get this messages and the trunk is NOT disabled… I can’t issue any command related to interfaces (print, enable, etc…):

bonding: bond0: Removing slave eth0
bonding: bond0: Warning: the permanent HWaddr of eth0 - 00:0c:42:2e:b5:11 - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
bonding: bond0: releasing active interface eth0
eth0: stop
eth0: open
eth0: phy speed: 100, fdx: 1
bonding: bond0: Removing slave eth1
bonding: bond0: Removing an active aggregator
bonding: bond0: releasing active interface eth1
bonding: bond0: Warning: clearing HW address of bond0 while it still has VLANs.
bonding: bond0: When re-adding slaves, make sure the bond's HW address matches its VLANs'.
bonding: bond0 is being deleted...
unregister_netdevice: waiting for vlan10 to become free. Usage count = 2
unregister_netdevice: waiting for vlan10 to become free. Usage count = 2
unregister_netdevice: waiting for vlan10 to become free. Usage count = 2

Then, I’m forced to restart the whole router from the serial console. I’ve tried with 3.22 and 3.23 with the same results.

What’s going on?
Thank you!

leonset · May 7, 2009, 10:30am

After playing a little bit more with this, I get another dump if I just disable the trunk iface:

[admin@ZIPI] > Oops[#1]:
Cpu 0
$ 0   : 00000000 00000000 00010000 00010000
$ 4   : 00000000 c0b01eb0 c0c34380 c17c9090
$ 8   : 00002485 d184d710 00002485 d184d710
$12   : 00000000 c0379b80 c1c908f0 00000001
$16   : c0c34b80 c0b01eb0 7fadb390 80187802
$20   : 2abc3d80 2abbece4 00000000 7fadb660
$24   : 00000000 c17c81d0
$28   : c0b00000 c0b01ea0 00000000 c17c83e4
Hi    : 00000393
Lo    : 87000000
epc   : c17c8440 cleanup_module+0x434/0x9d4 [vrrp]
    Tainted: P
ra    : c17c83e4 cleanup_module+0x3d8/0x9d4 [vrrp]
Status: 1000f203    KERNEL EXL IE
Cause : 00800008
BadVA : 000001b8
PrId  : 00019374 (MIPS 24K)
Process vrrp (pid: 846, threadinfo=c0b00000, task=c1cff878, tls=00000000)
Stack : c0251718 c0b01eb0 00000000 00000000 76727270 30000000 00000000 00000000
        00000000 00000000 7fadb390 c0a9fb00 7fadb390 c01915a8 c0a9fc80 00000000
        c1c19e70 00000001 c01803ec c01803ec c0a9fb00 00000004 7fadb390 c0191618
        c1c19e70 c0a9fc80 00000000 c0181dec 00000000 00000028 7fadb390 100024dc
        10002460 00000001 c010dcf0 c010dcf0 00000001 2ab25054 00000002 00000000
        ...
Call Trace:
[<c17c8440>] cleanup_module+0x434/0x9d4 [vrrp]


Code: 00431024  10400008  8e040070 <c08201b8> 2442ffff  e08201b8  1040015b  00000000  085f212d
---[ end trace f5b7679711db6bdd ]---

I can reenable the iface, but I get no traffic at all through it and this dump is generated:

bonding: bond0 is being deleted...
bonding: bond0 is being created...
bonding: bond0: setting mode to 802.3ad (4).
bonding: bond0: Unable to set up delay as MII monitoring is disabled
bonding: bond0: Unable to set down delay as MII monitoring is disabled
bonding: bond0: Setting LACP rate to fast (1).
bonding: bond0: setting xmit hash policy to layer2 (0).
eth0: stop
bonding: bond0: Adding slave eth0.
eth0: open
eth0: phy speed: 100, fdx: 1
bonding: bond0: enslaving eth0 as a backup interface with an up link.
bonding: bond0: Adding slave eth1.
bonding: bond0: enslaving eth1 as a backup interface with an up link.
vrrp: add vrrp1 24 1
vrrp: vrrp1 owns vlan8
vrrp: register_netdev failed
vrrp: open vrrp1
vrrp: slave lost
vrrp: add vrrp0 25 2
vrrp: vrrp0 owns vlan10
vrrp: register_netdev failed
vrrp: open vrrp0
vrrp: slave lost

I must restart the router to get it working… but then, all the VRRP ifaces are shown in red, as if they had an error. To solve that, I have to disable the trunk and then reenable it again. Now I don’t get the error messages shown above and the trunk is disabled and reenabled correctly.

What’s going on with the trunking feature? Is it mature enough to be used in a production environment?

Should I open a support ticket?

Thanks

Letni · May 7, 2009, 11:38am

What hardware are you running it on? (X86, RBXXX)
If it is X86, do you have multi-cpu turned on?

-Louis

leonset · May 7, 2009, 12:03pm

I’m using RB450 and I have tried four diferent units… all show the same problem.

I should have specified that before… sorry!

janisk · May 7, 2009, 12:31pm

at this moment all of this information is completely useless because of few simple things:
1st - at start you did not provide what arch and hardware you are using
2nd - you do not state - what version you are using
3rd - what RouterBOOT version you are using
4th - on what version you have noticed the problem
5th - have you contacted support with supout.rif file attached from problematic routers. if not - upgrade to latest version of routers, see, if you still have the problem, create supout.rif file and contact support at mikrotik.com

leonset · May 7, 2009, 1:04pm

Hello!

I’m sorry for that… as I explained in a previous post, I’m using 4 different units of RouterBoard RB450

Yes, I did… RouterOS v3.22 and v3.23

Took a while to check that… I have RouterBOOT v2.15 on 2 units and v2.18 on the other two.

RouterOS v3.22 and 3.23. The unit with v3.23 has RouterBoot v2.15 and the one with v3.22 has RouterBoot 2.15
How may I upgrade RouterBoot? Should I do so?

I will write that email asap!

Thank you!

knects · May 7, 2009, 5:03pm

/system routerboard upgrade

leonset · May 8, 2009, 8:03am

Thanks for the hint… but that command doesn’t exist in my current Ros version.

Just in case someone needs it, to upgrade the RouterBoot firmware:

I downloaded the firmware from http://www.routerboard.com/comparison.html
Using a serial cable entered the “bios” and uploaded the file using XMODEM. All the process is menu driven and easy to follow.
Reboot the router…

BTW: I have exactly the same problem after upgrading RouterBoot to 2.19

I’m currently writing that email to Mikrotik’s support…

Thanks!

normis · May 8, 2009, 8:17am

you need to have Routerboard.npk package installed for that command to work. It works in all ROS versions

leonset · May 8, 2009, 1:08pm

Hello!

Mikrotik’s support team has been able to reproduce the problem easily, so they will look deeper into it and try to solve it for next release

Thanks everyone.

leonset · May 27, 2009, 12:35pm

I’ve just checked it and the problem has been solved in v3.24

Thanks Mikrotik!