CCR 1036 interface restart when passing > 460mbit/s

adyb76 · Fri Sep 20, 2013 1:51 pm

Hi,
I noticed some strange behaviour on one of our ccr last night.
The log entries are:

[admin@queue] > /log print 
sep/18 07:39:59 interface,info ether12 link down 
sep/18 07:40:00 interface,info sfp2 link up (speed 1000M, full duplex) 
sep/18 07:40:00 interface,info sfp1 link up (speed 1000M, full duplex) 
sep/18 07:40:02 interface,info ether12 link up (speed 1000M, full duplex) 
sep/18 16:45:00 interface,info sfp2 link down 
sep/18 16:45:00 interface,info sfp1 link down 
sep/18 16:45:00 interface,info ether12 link down 
sep/18 16:45:01 interface,info sfp2 link up (speed 1000M, full duplex) 
sep/18 16:45:01 interface,info sfp1 link up (speed 1000M, full duplex) 
sep/18 16:45:03 interface,info ether12 link up (speed 1000M, full duplex) 
sep/19 15:10:03 interface,info sfp2 link down 
sep/19 15:10:03 interface,info sfp1 link down 
sep/19 15:10:03 interface,info ether12 link down 
sep/19 15:10:04 interface,info sfp2 link up (speed 1000M, full duplex) 
sep/19 15:10:04 interface,info sfp1 link up (speed 1000M, full duplex) 
sep/19 15:10:06 interface,info ether12 link up (speed 1000M, full duplex) 
sep/19 19:06:18 interface,info sfp2 link down 
sep/19 19:06:18 interface,info sfp1 link down 
sep/19 19:06:18 interface,info ether12 link down

They only seem to do this when the total traffic exceeds 460 mbit/s - having said that it's just done it with traffic of only 200mbit/s. All interfaces drop and then come back up straight away. Even ether12 which is just used for management and only passes 200kbit/s maximum.

When it happens it stops passing traffic which then builds up to the previous level after a very short pause.

The sfp1 and 2 are TPLink devices with Wavesliptter chipset and are recognised by routeros, albeit programmed to 1gbps with autonegotiation turned off. sfp1 is from our upstream isp and sfp2 goes to another ccr with the exact same device and firmware versions.

Any ideas?

Thanks....

matlocktr · Sun Oct 06, 2013 3:31 pm

I Live same problem about 2 days.
Today the routeros disables the ports 6 times..
I dont know what can i do..
When i reboot the router, it works normal but after 1 hour it happens again.. there is no more than 100mbit traffic, today is Sunday....

CCR1036-12G-4S
ether1 - uplink
ether2 - tonetwork
using bridge these 2 interfaces...

Attaching screenshot..

paolopoz · Thu Oct 31, 2013 6:52 pm

Same here, with much less bandwidth than 460 Mbps.
However I see that the problem is tied with lot of traffic flowing.
To me it also happens on more than one interface at once:

16:35:32 system,info filter rule changed by admin 
16:35:32 system,info filter rule changed by admin 
16:35:45 system,info device changed by admin 
16:35:51 system,info device changed by admin 
16:52:43 interface,info ether1 link down 
16:52:45 interface,info ether1 link up (speed 100M, full duplex) 
16:52:55 interface,info ether1 link down 
16:52:55 interface,info ether2 link down 
16:52:55 interface,info ether3 link down 
16:52:55 interface,info ether4 link down 
16:52:55 interface,info ether5 link down 
16:52:55 interface,info ether6 link down 
16:52:55 interface,info ether7 link down 
16:52:55 interface,info ether10 link down 
16:52:56 interface,info ether1 link up (speed 100M, full duplex) 
16:52:56 interface,info ether3 link up (speed 100M, full duplex) 
16:52:57 interface,info ether4 link up (speed 100M, full duplex) 
16:52:57 interface,info ether5 link up (speed 100M, full duplex) 
16:52:58 interface,info ether6 link up (speed 1000M, full duplex) 
16:52:58 interface,info ether7 link up (speed 1000M, full duplex) 
16:53:11 interface,info ether2 link up (speed 100M, full duplex) 
16:53:12 interface,info ether10 link up (speed 100M, full duplex) 
16:59:22 interface,info ether1 link down 
16:59:23 interface,info ether1 link up (speed 100M, full duplex) 
17:02:35 system,info lcd config changed by admin 
17:09:34 interface,info ether1 link down 
17:09:36 interface,info ether1 link up (speed 100M, full duplex) 
17:10:16 interface,info ether1 link down 
17:10:18 interface,info ether1 link up (speed 100M, full duplex)

Fri Nov 01, 2013 9:08 am

Turn off LCD and SNMP and see if this makes any change. Also try latest v6.6rc1 pre-release build.

paolopoz · Mon Nov 04, 2013 1:52 pm

I upgraded my CCR-1036 from v6.5 to v6.6rc1 but now the router reboot continously shorly after the boot phase.
I will make other tests connecting in console while booting but by now it seems to me that the 6.6rc1 is bugged.

paolopoz · Mon Nov 04, 2013 3:09 pm

Here is what happen: booting CCR without any port attached works fine, as soon as I attach the ethernet cable from my notebook to the eth7 the router reboot.
Here is what I see on console:

 Pid: 0, comm:           swapper/34, CPU: 34
 r0 : 0xfffffe40751d0000 r1 : 0xfffffe00006e99d8 r2 : 0x0000000000002824
 r3 : 0x28242824ff00a8c0 r4 : 0x0000000000002824 r5 : 0x0000000000000100
 r6 : 0xfffffffd00009364 r7 : 0x0000000000000002 r8 : 0xfffffe4075200075
 r9 : 0xfffffe4075200076 r10: 0x28242824ff00a8c0 r11: 0x000000006000a8c0
 r12: 0xfffffe4077603060 r13: 0xfffffe40000000b8 r14: 0x00000002000003b7
 r15: 0x00000001ffffffff r16: 0x6000a8c011b41180 r17: 0x0000000000000043
 r18: 0xfffffe00734cb0ba r19: 0x0000000000000000 r20: 0xfffffe007c65f9b8
 r21: 0xfffffe007c65f9d8 r22: 0xfffffe007c65f9c0 r23: 0xfffffe007c65f9e0
 r24: 0xfffffe007c65f9c8 r25: 0xfffffe007c65f9e8 r26: 0xfffffe007c65f9d0
 r27: 0xfffffe007c65f9f0 r28: 0xfffffe007c65f9a8 r29: 0xfffffe007c65f9f8
 r30: 0xfffffe40751d0000 r31: 0xfffffe4077603064 r32: 0x0000000000000011
 r33: 0x0000000000000043 r34: 0x0000000000000000 r35: 0xfffffe00007bbc40
 r36: 0xfffffe00006e99d8 r37: 0xfffffe40000000a4 r38: 0xffffffffffffffff
 r39: 0xfffffe00006e90a8 r40: 0x0000000000000001 r41: 0xfffffe40751d0020
 r42: 0xfffffe40751d0058 r43: 0xfffffe40751d0060 r44: 0xfffffffcffffffff
 r45: 0xfffffe0000540040 r46: 0x0000000000000000 r47: 0x0000000000000000
 r48: 0x0000000000000000 r49: 0x0000000000000000 r50: 0x0000000000000000
 r51: 0x0000000000000000 r52: 0xfffffe007c65fc40 tp : 0x000001f4ffce0000
 sp : 0xfffffe007c65f9a0 lr : 0xfffffff7004460a8
 pc : 0xfffffff700496198 ex1: 1     faultnum: 17

Starting stack dump of tid 0, pid 0 (swapper/34) on cpu 34 at cycle 131714932257
  frame 0: 0xfffffff700496198 __udp4_lib_rcv+0xb48/0xc00 (sp 0xfffffe007c65f9a0)
  frame 1: 0xfffffff7004460a8 ip_local_deliver_finish+0x390/0x5d0 (sp 0xfffffe007c65f9f8)
  frame 2: 0xfffffff711e40da8 0xfffffff711e40da8 [imq@0xfffffff711e40000] (sp 0xfffffe007c65fa78)
  frame 3: 0xfffffff7003ffc88 dev_hard_start_xmit+0x750/0xaf8 (sp 0xfffffe007c65fac0)
  frame 4: 0xfffffff700425200 sch_direct_xmit+0xd8/0x390 (sp 0xfffffe007c65fb48)
  frame 5: 0xfffffff7004256d8 __qdisc_run+0x220/0x2b8 (sp 0xfffffe007c65fb88)
  frame 6: 0xfffffff7003f5780 net_tx_action+0x218/0x348 (sp 0xfffffe007c65fbf0)
  frame 7: 0xfffffff7000542a8 __do_softirq+0x208/0x388 (sp 0xfffffe007c65fc48)
  frame 8: 0xfffffff700054850 do_softirq+0xd0/0x140 (sp 0xfffffe007c65fcd8)
  frame 9: 0xfffffff700054de0 irq_exit+0xb0/0x1a0 (sp 0xfffffe007c65fcf8)
  frame 10: 0xfffffff700027000 tile_dev_intr+0x190/0x1f0 (sp 0xfffffe007c65fd08)
  frame 11: 0xfffffff7004f5060 handle_interrupt+0x1f0/0x210 (sp 0xfffffe007c65fd40)
  <interrupt 30 while in kernel mode>
  frame 12: 0xfffffff7004f4e58 _cpu_idle_nap+0x0/0x18 (sp 0xfffffe007c65ffb0)
  frame 13: 0xfffffff700029db8 cpu_idle+0x218/0x2b8 (sp 0xfffffe007c65ffb0)
Stack dump complete
---[ end trace 71169b09e1801d3a ]---
Kernel panic - not syncing: Aiee, killing interrupt handler!

Starting stack dump of tid 0, pid 0 (swapper/34) on cpu 34 at cycle 133066422360
  frame 0: 0xfffffff7004f4db0 dump_stack+0x0/0x20 (sp 0xfffffe007c65f530)
  frame 1: 0xfffffff7004edff0 panic+0x168/0x3a8 (sp 0xfffffe007c65f530)
  frame 2: 0xfffffff70004faf0 do_exit+0x1b0/0xdc8 (sp 0xfffffe007c65f5d8)
  frame 3: 0xfffffff70002dd70 do_trap+0x260/0x5e0 (sp 0xfffffe007c65f688)
  frame 4: 0xfffffff7004f5060 handle_interrupt+0x1f0/0x210 (sp 0xfffffe007c65f740)
  <interrupt 17 while in kernel mode>
  frame 5: 0xfffffff700496198 __udp4_lib_rcv+0xb48/0xc00 (sp 0xfffffe007c65f9a0)
  frame 6: 0xfffffff7004460a8 ip_local_deliver_finish+0x390/0x5d0 (sp 0xfffffe007c65f9f8)
  frame 7: 0xfffffff711e40da8 0xfffffff711e40da8 [imq@0xfffffff711e40000] (sp 0xfffffe007c65fa78)
  frame 8: 0xfffffff7003ffc88 dev_hard_start_xmit+0x750/0xaf8 (sp 0xfffffe007c65fac0)
  frame 9: 0xfffffff700425200 sch_direct_xmit+0xd8/0x390 (sp 0xfffffe007c65fb48)
  frame 10: 0xfffffff7004256d8 __qdisc_run+0x220/0x2b8 (sp 0xfffffe007c65fb88)
  frame 11: 0xfffffff7003f5780 net_tx_action+0x218/0x348 (sp 0xfffffe007c65fbf0)
  frame 12: 0xfffffff7000542a8 __do_softirq+0x208/0x388 (sp 0xfffffe007c65fc48)
  frame 13: 0xfffffff700054850 do_softirq+0xd0/0x140 (sp 0xfffffe007c65fcd8)
  frame 14: 0xfffffff700054de0 irq_exit+0xb0/0x1a0 (sp 0xfffffe007c65fcf8)
  frame 15: 0xfffffff700027000 tile_dev_intr+0x190/0x1f0 (sp 0xfffffe007c65fd08)
  frame 16: 0xfffffff7004f5060 handle_interrupt+0x1f0/0x210 (sp 0xfffffe007c65fd40)
  <interrupt 30 while in kernel mode>
  frame 17: 0xfffffff7004f4e58 _cpu_idle_nap+0x0/0x18 (sp 0xfffffe007c65ffb0)
  frame 18: 0xfffffff700029db8 cpu_idle+0x218/0x2b8 (sp 0xfffffe007c65ffb0)
Stack dump complete

panicSaver: dumping panic to flash
flash: prg 73
tile program flash offset:00073000 size:00001000
flash: prg 74
tile program flash offset:00074000 size:00001000
------------[ cut here ]------------
WARNING: at /home/build/6.6rc1/kernel/linux6/arch/tile/kernel/smp.c:239 smp_send_reschedule+0x70/0xb8()

Starting stack dump of tid 0, pid 0 (swapper/34) on cpu 34 at cycle 133495696619
  frame 0: 0xfffffff7004f4db0 dump_stack+0x0/0x20 (sp 0xfffffe007c65eff8)
  frame 1: 0xfffffff700048c08 warn_slowpath_common+0xb0/0xf8 (sp 0xfffffe007c65eff8)
  frame 2: 0xfffffff700032940 smp_send_reschedule+0x70/0xb8 (sp 0xfffffe007c65f030)
  frame 3: 0xfffffff7000904a8 try_to_wake_up+0x4b0/0x578 (sp 0xfffffe007c65f048)
  frame 4: 0xfffffff70007c328 autoremove_wake_function+0x28/0x90 (sp 0xfffffe007c65f0a8)
  frame 5: 0xfffffff7000886e8 __wake_up_common+0x88/0x130 (sp 0xfffffe007c65f0c0)
  frame 6: 0xfffffff700088ff0 __wake_up+0x70/0xb8 (sp 0xfffffe007c65f110)
  frame 7: 0xfffffff700060ec8 update_process_times+0x70/0xe0 (sp 0xfffffe007c65f150)
  frame 8: 0xfffffff7000aaa20 tick_sched_timer+0x110/0x170 (sp 0xfffffe007c65f178)
  frame 9: 0xfffffff700082928 __run_hrtimer.isra.18+0xc0/0x218 (sp 0xfffffe007c65f1a0)
  frame 10: 0xfffffff7000834b8 hrtimer_interrupt+0x218/0x498 (sp 0xfffffe007c65f1d0)
  frame 11: 0xfffffff70002da18 do_timer_interrupt+0xd0/0x110 (sp 0xfffffe007c65f258)
  frame 12: 0xfffffff7004f5060 handle_interrupt+0x1f0/0x210 (sp 0xfffffe007c65f280)
  <interrupt 22 while in kernel mode>
  frame 13: 0xfffffff7101006a0 flush_buffer+0xe8/0x130 [panics@0xfffffff710100000] (sp 0xfffffe007c65f4c8)
  frame 14: 0xfffffff7101007b8 saviourNotifier+0x28/0x40 [panics@0xfffffff710100000] (sp 0xfffffe007c65f4e8)
  frame 15: 0xfffffff7000851f0 notifier_call_chain+0x68/0x100 (sp 0xfffffe007c65f4f8)
  frame 16: 0xfffffff7004ee018 panic+0x190/0x3a8 (sp 0xfffffe007c65f530)
  frame 17: 0xfffffff70004faf0 do_exit+0x1b0/0xdc8 (sp 0xfffffe007c65f5d8)
  frame 18: 0xfffffff70002dd70 do_trap+0x260/0x5e0 (sp 0xfffffe007c65f688)
  frame 19: 0xfffffff7004f5060 handle_interrupt+0x1f0/0x210 (sp 0xfffffe007c65f740)
  <interrupt 17 while in kernel mode>
  frame 20: 0xfffffff700496198 __udp4_lib_rcv+0xb48/0xc00 (sp 0xfffffe007c65f9a0)
  frame 21: 0xfffffff7004460a8 ip_local_deliver_finish+0x390/0x5d0 (sp 0xfffffe007c65f9f8)
  frame 22: 0xfffffff711e40da8 0xfffffff711e40da8 [imq@0xfffffff711e40000] (sp 0xfffffe007c65fa78)
  frame 23: 0xfffffff7003ffc88 dev_hard_start_xmit+0x750/0xaf8 (sp 0xfffffe007c65fac0)
  frame 24: 0xfffffff700425200 sch_direct_xmit+0xd8/0x390 (sp 0xfffffe007c65fb48)
  frame 25: 0xfffffff7004256d8 __qdisc_run+0x220/0x2b8 (sp 0xfffffe007c65fb88)
  frame 26: 0xfffffff7003f5780 net_tx_action+0x218/0x348 (sp 0xfffffe007c65fbf0)
  frame 27: 0xfffffff7000542a8 __do_softirq+0x208/0x388 (sp 0xfffffe007c65fc48)
  frame 28: 0xfffffff700054850 do_softirq+0xd0/0x140 (sp 0xfffffe007c65fcd8)
  frame 29: 0xfffffff700054de0 irq_exit+0xb0/0x1a0 (sp 0xfffffe007c65fcf8)
  frame 30: 0xfffffff700027000 tile_dev_intr+0x190/0x1f0 (sp 0xfffffe007c65fd08)
  frame 31: 0xfffffff7004f5060 handle_interrupt+0x1f0/0x210 (sp 0xfffffe007c65fd40)
  <interrupt 30 while in kernel mode>
  frame 32: 0xfffffff7004f4e58 _cpu_idle_nap+0x0/0x18 (sp 0xfffffe007c65ffb0)
  frame 33: 0xfffffff700029db8 cpu_idle+0x218/0x2b8 (sp 0xfffffe007c65ffb0)
Stack dump complete
---[ end trace 71169b09e1801d3b ]---
Rebooting in 1 seconds..

There is surely something wrong with this. Please have a look.

mje · Wed Nov 06, 2013 1:27 pm

Seems I have similar woes... 1036CCR, 12G. 4SFP. V6.5

Set up a (copper only) bridge group and after adding a Cisco 7206 (100Mb copper),
noticed that every few seconds the link light went out for a second or two.
Unplugging the Cisco "sorted" the problem - but doesn't obviously help me.

Whilst doing initial configuration, I had an SFP / optic connection to an HP switch - and
it all just worked...
now I have this feeing things are not working quite so well, the SFP connection to
another cisco just does not work, link light is on - no packets seem to leave the interface.

Go back to V6.4 ???

mje · Thu Nov 07, 2013 11:47 pm

Followup...
Loaded 6.6

The Optic worked just fine once auto-negotiate was switched off.
ie Telling the Optic that can only run at 1Gb to run at 1Gb... hmmm

Got the same results - ports disable then return.
System got worse - and started rebooting every minute or so.

Left just the Optic plugged in - even that has since died (3 or 4 hours later)

The Optic was being given a full IPv4 and IPv6 BGP table. Looks like just one CPU does all the BGP???
Sometimes saw 3 CPU's at 100%.
There was otherwise no traffic - I can't get the CCR up long enough.

Was hoping to do diagnostics/reports from the comfort of home (Device is in Telehouse, Docklands, London)... but its dead.

I'm afraid I'm now looking for a Cisco NPE-G2 Engine!

CCR 1036 interface restart when passing > 460mbit/s

CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Re: CCR 1036 interface restart when passing > 460mbit/s

Who is online