Community discussions

MikroTik App
 
davidalain
just joined
Topic Author
Posts: 16
Joined: Thu Aug 03, 2017 3:24 am
Location: Brazil

OSPF disabling all ports on 'state change from Full to Down'

Wed Apr 01, 2020 7:28 am

Hi,

I am using a CCR1009-7G-1C-1S+ running RouterOS v6.45.8 and recently it disconnected all ports at the same time OSPF lost neighborhood.
This is an edge router where I'm running OSPFv2/v3, NAT and firewall. This is third time that router crashed in a couple of months and all my clients goes down with Internet until a reboot.

This time, the router have crashed and got back to life again with no reboot after about 30 minutes. I don't know neither why or how.
So, I got the graphs and log, and I saw some 'strange things': the router did not freezed, it disconnected all ports like all ports was manually unplugged at once.
The graph for CPU had lower usage (due to no traffic), and memory and disk usages had around the same values.

I think this can be an issue with OSPF because this happened at same time when OSPF got state change from Full to Down, as you can see in the attached log.
And always when OSPF neighboorhood is lost, it disconnect and reconnect all running ports after one second or two.
Thus, all my network goes down about two second very often.

Anyone can help me?

This is the log of CCR1009 filtered to get all messages that contains "link down", "link up", "OSPF", and "rebooted":
topics=route,ospf,info, .id=*A373, time=Feb/27/2020 02:04:20, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*A373, time=Feb/27/2020 02:04:20, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*A374, time=Feb/27/2020 02:04:20, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*A374, time=Feb/27/2020 02:04:20, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*A375, time=Feb/27/2020 02:04:20, message=ether7 link down
topics=interface,info, .id=*A376, time=Feb/27/2020 02:04:20, message=ether6 link down
topics=interface,info, .id=*A377, time=Feb/27/2020 02:04:20, message=ether4 link down
topics=interface,info, .id=*A378, time=Feb/27/2020 02:04:20, message=ether1 link down
topics=interface,info, .id=*A379, time=Feb/27/2020 02:04:20, message=sfp-combo1 link down
topics=interface,info, .id=*A37A, time=Feb/27/2020 02:04:20, message=sfp-sfpplus1 link down
topics=interface,info, .id=*A370, time=Feb/27/2020 02:04:21, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*A36C, time=Feb/27/2020 02:04:22, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*A36D, time=Feb/27/2020 02:04:22, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*A36A, time=Feb/27/2020 02:04:23, message=ether6 link up (speed 100M, full duplex)
topics=interface,info, .id=*A369, time=Feb/27/2020 02:04:25, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*A368, time=Feb/27/2020 02:04:26, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*9861, time=Mar/01/2020 09:02:15, message=ether6 link down
topics=interface,info, .id=*9860, time=Mar/01/2020 09:02:16, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*985F, time=Mar/01/2020 09:02:17, message=ether5 link down
topics=interface,info, .id=*985E, time=Mar/01/2020 09:02:18, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*985A, time=Mar/01/2020 09:03:40, message=ether6-gerencia-local link up (speed 1G, full duplex)
topics=interface,info, .id=*9851, time=Mar/01/2020 09:08:32, message=ether6-gerencia-local link down
topics=interface,info, .id=*9836, time=Mar/01/2020 10:16:30, message=ether6-gerencia-local link up (speed 1G, full duplex)
topics=interface,info, .id=*9820, time=Mar/01/2020 10:38:43, message=ether6-gerencia-local link down
topics=interface,info, .id=*981F, time=Mar/01/2020 10:44:14, message=ether6-gerencia-local link up (speed 1G, full duplex)
topics=interface,info, .id=*9819, time=Mar/01/2020 11:01:54, message=ether6-gerencia-local link down
topics=system,info, .id=*96EE, time=Mar/01/2020 16:09:32, message=OSPFv2 network added by XXXXXXXX
topics=system,info, .id=*96E9, time=Mar/01/2020 16:21:18, message=OSPFv2 network removed by XXXXXXXX
topics=route,ospf,info, .id=*9AF8, time=Mar/01/2020 17:06:58, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*9AF8, time=Mar/01/2020 17:06:58, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*9AF9, time=Mar/01/2020 17:06:58, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*9AF9, time=Mar/01/2020 17:06:58, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*9AFA, time=Mar/01/2020 17:06:58, message=ether4 link down
topics=interface,info, .id=*9AFB, time=Mar/01/2020 17:06:58, message=ether1 link down
topics=interface,info, .id=*9AFC, time=Mar/01/2020 17:06:58, message=sfp-combo1 link down
topics=interface,info, .id=*9AFD, time=Mar/01/2020 17:06:58, message=sfp-sfpplus1 link down
topics=interface,info, .id=*9AF6, time=Mar/01/2020 17:06:59, message=ether7 link down
topics=interface,info, .id=*9AF7, time=Mar/01/2020 17:06:59, message=ether6 link down
topics=interface,info, .id=*9AEF, time=Mar/01/2020 17:07:00, message=ether6 link up (speed 100M, full duplex)
topics=interface,info, .id=*9AF1, time=Mar/01/2020 17:07:00, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*9AF2, time=Mar/01/2020 17:07:00, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*9AED, time=Mar/01/2020 17:07:01, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*9AEE, time=Mar/01/2020 17:07:01, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*9AEC, time=Mar/01/2020 17:07:02, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*9A39, time=Mar/01/2020 20:53:00, message=ether6 link down
topics=interface,info, .id=*9A37, time=Mar/01/2020 20:56:16, message=ether6 link up (speed 100M, full duplex)
topics=interface,info, .id=*9A34, time=Mar/01/2020 21:12:52, message=ether6 link down
topics=interface,info, .id=*9A33, time=Mar/01/2020 21:13:17, message=ether6 link up (speed 100M, full duplex)
topics=route,ospf,info, .id=*46E6, time=Mar/02/2020 18:28:22, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*46E6, time=Mar/02/2020 18:28:22, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=system,info, .id=*4665, time=Mar/02/2020 18:54:14, message=OSPFv2 instance changed by XXXXXXXX
topics=system,info, .id=*4664, time=Mar/02/2020 18:54:16, message=OSPFv2 instance changed by XXXXXXXX
topics=system,info, .id=*465D, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*465E, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*465F, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4660, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4661, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4662, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4663, time=Mar/02/2020 18:54:20, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4656, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4657, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4658, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*4659, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*465A, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*465B, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=system,info, .id=*465C, time=Mar/02/2020 18:54:22, message=OSPFv2 network changed by XXXXXXXX
topics=route,ospf,info, .id=*4649, time=Mar/02/2020 19:04:36, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*4649, time=Mar/02/2020 19:04:36, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*464A, time=Mar/02/2020 19:04:36, message=ether7 link down
topics=interface,info, .id=*464B, time=Mar/02/2020 19:04:36, message=ether5 link down
topics=interface,info, .id=*464C, time=Mar/02/2020 19:04:36, message=ether4 link down
topics=interface,info, .id=*464D, time=Mar/02/2020 19:04:36, message=ether1 link down
topics=interface,info, .id=*464E, time=Mar/02/2020 19:04:36, message=sfp-combo1 link down
topics=interface,info, .id=*464F, time=Mar/02/2020 19:04:36, message=sfp-sfpplus1 link down
topics=interface,info, .id=*4645, time=Mar/02/2020 19:04:37, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*4648, time=Mar/02/2020 19:04:37, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*4648, time=Mar/02/2020 19:04:37, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*4641, time=Mar/02/2020 19:04:38, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*4642, time=Mar/02/2020 19:04:38, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*463F, time=Mar/02/2020 19:04:39, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*463E, time=Mar/02/2020 19:04:41, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*463D, time=Mar/02/2020 19:04:42, message=sfp-combo1 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*463A, time=Mar/02/2020 19:06:07, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*463A, time=Mar/02/2020 19:06:07, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*463B, time=Mar/02/2020 19:06:07, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*463B, time=Mar/02/2020 19:06:07, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*463C, time=Mar/02/2020 19:06:07, message=ether4 link down
topics=interface,info, .id=*4639, time=Mar/02/2020 19:06:16, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*4637, time=Mar/02/2020 19:06:31, message=ether4 link down
topics=interface,info, .id=*4636, time=Mar/02/2020 19:06:33, message=ether4 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*4601, time=Mar/02/2020 20:17:24, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Init
topics=route,ospf,info, .id=*4601, time=Mar/02/2020 20:17:24, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Init
topics=system,info, .id=*458B, time=Mar/02/2020 20:47:52, message=OSPFv2 instance changed by XXXXXXXX
topics=route,ospf,info, .id=*458A, time=Mar/02/2020 20:52:52, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Init
topics=route,ospf,info, .id=*458A, time=Mar/02/2020 20:52:52, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Init
topics=system,info, .id=*454A, time=Mar/02/2020 21:15:42, message=OSPFv2 instance changed by XXXXXXXX
topics=system,info, .id=*4549, time=Mar/02/2020 21:16:11, message=OSPFv2 instance changed by XXXXXXXX
topics=system,info, .id=*4548, time=Mar/02/2020 21:16:37, message=OSPFv2 instance changed by XXXXXXXX
topics=system,info, .id=*4547, time=Mar/02/2020 21:16:44, message=OSPFv2 instance changed by XXXXXXXX
topics=system,info, .id=*4505, time=Mar/02/2020 23:50:47, message=OSPFv3 instance changed by XXXXXXXX
topics=interface,info, .id=*43DB, time=Mar/03/2020 14:40:59, message=ether3 link up (speed 100M, full duplex)
topics=route,ospf,info, .id=*4090, time=Mar/04/2020 15:31:19, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*4090, time=Mar/04/2020 15:31:19, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*3F30, time=Mar/05/2020 19:37:48, message=ether7 link down
topics=interface,info, .id=*3F2F, time=Mar/05/2020 19:37:49, message=ether7 link up (speed 10M, full duplex)
topics=interface,info, .id=*3F2E, time=Mar/05/2020 19:37:50, message=ether7 link down
topics=interface,info, .id=*3F2D, time=Mar/05/2020 19:37:52, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3F2C, time=Mar/05/2020 19:38:08, message=ether7 link down
topics=interface,info, .id=*3F18, time=Mar/05/2020 20:48:01, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3F17, time=Mar/05/2020 20:48:18, message=ether7 link down
topics=interface,info, .id=*3F16, time=Mar/05/2020 20:49:53, message=ether7 link up (speed 10M, full duplex)
topics=interface,info, .id=*3F15, time=Mar/05/2020 20:49:54, message=ether7 link down
topics=interface,info, .id=*3F14, time=Mar/05/2020 20:49:56, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3F13, time=Mar/05/2020 20:50:21, message=ether7 link down
topics=interface,info, .id=*3F12, time=Mar/05/2020 20:50:25, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3F11, time=Mar/05/2020 20:50:31, message=ether7 link down
topics=interface,info, .id=*3F10, time=Mar/05/2020 20:55:54, message=ether7 link up (speed 10M, full duplex)
topics=interface,info, .id=*3F0F, time=Mar/05/2020 20:55:55, message=ether7 link down
topics=interface,info, .id=*3F0E, time=Mar/05/2020 20:55:57, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3F0D, time=Mar/05/2020 20:56:05, message=ether7 link down
topics=interface,info, .id=*3F0C, time=Mar/05/2020 20:56:10, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3F00, time=Mar/05/2020 21:16:27, message=ether7 link down
topics=interface,info, .id=*3EFF, time=Mar/05/2020 21:16:33, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EFE, time=Mar/05/2020 21:16:40, message=ether7 link down
topics=interface,info, .id=*3EFD, time=Mar/05/2020 21:16:45, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EFC, time=Mar/05/2020 21:21:34, message=ether7 link down
topics=interface,info, .id=*3EFB, time=Mar/05/2020 21:21:39, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EFA, time=Mar/05/2020 21:21:45, message=ether7 link down
topics=interface,info, .id=*3EF8, time=Mar/05/2020 21:24:19, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EF6, time=Mar/05/2020 21:28:36, message=ether7 link down
topics=interface,info, .id=*3EF5, time=Mar/05/2020 21:28:40, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EF4, time=Mar/05/2020 21:28:48, message=ether7 link down
topics=interface,info, .id=*3EF3, time=Mar/05/2020 21:33:31, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EE8, time=Mar/05/2020 21:45:05, message=ether7 link down
topics=interface,info, .id=*3EE7, time=Mar/05/2020 21:45:09, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*3EE6, time=Mar/05/2020 21:45:16, message=ether7 link down
topics=interface,info, .id=*3EE5, time=Mar/05/2020 21:45:22, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*398A, time=Mar/07/2020 08:14:22, message=sfp-sfpplus1 link down
topics=interface,info, .id=*3984, time=Mar/07/2020 11:06:26, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*3314, time=Mar/09/2020 12:32:22, message=sfp-combo1 link down
topics=interface,info, .id=*3308, time=Mar/09/2020 15:31:18, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*3307, time=Mar/09/2020 15:31:30, message=sfp-combo1 link down
topics=interface,info, .id=*3306, time=Mar/09/2020 15:33:14, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*3278, time=Mar/10/2020 11:49:43, message=ether5 link down
topics=interface,info, .id=*3277, time=Mar/10/2020 11:52:35, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*3276, time=Mar/10/2020 11:55:32, message=ether3 link down
topics=interface,info, .id=*3275, time=Mar/10/2020 11:55:54, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*3234, time=Mar/10/2020 12:39:36, message=ether5 link down
topics=interface,info, .id=*3233, time=Mar/10/2020 12:39:46, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*3230, time=Mar/10/2020 13:19:52, message=ether5 link down
topics=interface,info, .id=*322F, time=Mar/10/2020 13:19:58, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*3150, time=Mar/11/2020 02:02:31, message=ether3 link down
topics=interface,info, .id=*314F, time=Mar/11/2020 02:02:32, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*3102, time=Mar/11/2020 03:58:09, message=ether1 link down
topics=interface,info, .id=*3100, time=Mar/11/2020 03:58:11, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*30F8, time=Mar/11/2020 04:00:50, message=ether1 link down
topics=interface,info, .id=*30F5, time=Mar/11/2020 04:01:03, message=ether1 link up (speed 10M, half duplex)
topics=route,ospf,info, .id=*27AA, time=Mar/15/2020 16:16:43, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*27AA, time=Mar/15/2020 16:16:43, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*27AB, time=Mar/15/2020 16:16:43, message=ether7 link down
topics=interface,info, .id=*27AC, time=Mar/15/2020 16:16:43, message=ether5 link down
topics=route,ospf,info, .id=*27AE, time=Mar/15/2020 16:16:43, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*27AE, time=Mar/15/2020 16:16:43, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*27AF, time=Mar/15/2020 16:16:43, message=ether4 link down
topics=interface,info, .id=*27B0, time=Mar/15/2020 16:16:43, message=ether3 link down
topics=interface,info, .id=*27B1, time=Mar/15/2020 16:16:43, message=ether1 link down
topics=interface,info, .id=*27B2, time=Mar/15/2020 16:16:43, message=sfp-combo1 link down
topics=interface,info, .id=*27B3, time=Mar/15/2020 16:16:43, message=sfp-sfpplus1 link down
topics=interface,info, .id=*27A4, time=Mar/15/2020 16:16:44, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*27A6, time=Mar/15/2020 16:16:44, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*27A7, time=Mar/15/2020 16:16:44, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*27A2, time=Mar/15/2020 16:16:45, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*27A3, time=Mar/15/2020 16:16:45, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*27A1, time=Mar/15/2020 16:16:46, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*27A0, time=Mar/15/2020 16:16:47, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*16E9, time=Mar/22/2020 16:18:32, message=ether7 link down
topics=interface,info, .id=*16E8, time=Mar/22/2020 16:18:37, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*16E7, time=Mar/22/2020 16:18:46, message=ether7 link down
topics=interface,info, .id=*16E6, time=Mar/22/2020 16:18:50, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*1331, time=Mar/23/2020 21:12:52, message=ether5 link down
topics=interface,info, .id=*1330, time=Mar/23/2020 21:14:11, message=ether5 link up (speed 100M, full duplex)
topics=route,ospf,info, .id=*1251, time=Mar/24/2020 10:02:06, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*1251, time=Mar/24/2020 10:02:06, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*1252, time=Mar/24/2020 10:02:06, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*1252, time=Mar/24/2020 10:02:06, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*1253, time=Mar/24/2020 10:02:06, message=ether7 link down
topics=interface,info, .id=*1254, time=Mar/24/2020 10:02:06, message=ether5 link down
topics=interface,info, .id=*1256, time=Mar/24/2020 10:02:06, message=ether4 link down
topics=interface,info, .id=*1257, time=Mar/24/2020 10:02:06, message=ether3 link down
topics=interface,info, .id=*1258, time=Mar/24/2020 10:02:06, message=ether1 link down
topics=interface,info, .id=*1259, time=Mar/24/2020 10:02:06, message=sfp-combo1 link down
topics=interface,info, .id=*125A, time=Mar/24/2020 10:02:06, message=sfp-sfpplus1 link down
topics=interface,info, .id=*124B, time=Mar/24/2020 10:02:07, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*124C, time=Mar/24/2020 10:02:07, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*124D, time=Mar/24/2020 10:02:07, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*1247, time=Mar/24/2020 10:02:08, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*1248, time=Mar/24/2020 10:02:08, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*1249, time=Mar/24/2020 10:02:08, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*1246, time=Mar/24/2020 10:02:10, message=sfp-combo1 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*1233, time=Mar/24/2020 13:48:19, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*1233, time=Mar/24/2020 13:48:19, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*1234, time=Mar/24/2020 13:48:19, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*1234, time=Mar/24/2020 13:48:19, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*1236, time=Mar/24/2020 13:48:19, message=ether7 link down
topics=interface,info, .id=*1237, time=Mar/24/2020 13:48:19, message=ether5 link down
topics=interface,info, .id=*1239, time=Mar/24/2020 13:48:19, message=ether4 link down
topics=interface,info, .id=*123A, time=Mar/24/2020 13:48:19, message=ether3 link down
topics=interface,info, .id=*123B, time=Mar/24/2020 13:48:19, message=ether1 link down
topics=interface,info, .id=*123C, time=Mar/24/2020 13:48:19, message=sfp-combo1 link down
topics=interface,info, .id=*123D, time=Mar/24/2020 13:48:19, message=sfp-sfpplus1 link down
topics=interface,info, .id=*122D, time=Mar/24/2020 13:48:20, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*122E, time=Mar/24/2020 13:48:20, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*122F, time=Mar/24/2020 13:48:20, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*1230, time=Mar/24/2020 13:48:20, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*122B, time=Mar/24/2020 13:48:21, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*122C, time=Mar/24/2020 13:48:21, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*122A, time=Mar/24/2020 13:48:22, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*121C, time=Mar/24/2020 16:44:30, message=ether4 link down
topics=route,ospf,info, .id=*121A, time=Mar/24/2020 16:44:31, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*121A, time=Mar/24/2020 16:44:31, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*121B, time=Mar/24/2020 16:44:31, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*121B, time=Mar/24/2020 16:44:31, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*1210, time=Mar/24/2020 16:44:32, message=ether4 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*120C, time=Mar/24/2020 16:45:10, message=OSPFv3 neighbor 100.64.100.1: state change from 2-Way to Down
topics=route,ospf,info, .id=*120C, time=Mar/24/2020 16:45:10, message=OSPFv3 neighbor 100.64.100.1: state change from 2-Way to Down
topics=route,ospf,info, .id=*120D, time=Mar/24/2020 16:45:10, message=OSPFv2 neighbor 100.64.100.1: state change from 2-Way to Down
topics=route,ospf,info, .id=*120D, time=Mar/24/2020 16:45:10, message=OSPFv2 neighbor 100.64.100.1: state change from 2-Way to Down
topics=interface,info, .id=*120F, time=Mar/24/2020 16:45:10, message=ether4 link down
topics=interface,info, .id=*1203, time=Mar/24/2020 16:45:13, message=ether4 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*11FB, time=Mar/24/2020 16:46:15, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*11FB, time=Mar/24/2020 16:46:15, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*11FC, time=Mar/24/2020 16:46:15, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*11FC, time=Mar/24/2020 16:46:15, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*11D5, time=Mar/24/2020 16:46:19, message=ether4 link down
topics=interface,info, .id=*11B4, time=Mar/24/2020 16:46:22, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*10D4, time=Mar/24/2020 21:49:47, message=ether5 link down
topics=interface,info, .id=*10D2, time=Mar/24/2020 21:49:48, message=ether5 link up (speed 100M, full duplex)
topics=route,ospf,info, .id=*10C6, time=Mar/24/2020 21:50:02, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*10C6, time=Mar/24/2020 21:50:02, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*10C7, time=Mar/24/2020 21:50:02, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*10C7, time=Mar/24/2020 21:50:02, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*10C8, time=Mar/24/2020 21:50:02, message=ether5 link down
topics=interface,info, .id=*10C9, time=Mar/24/2020 21:50:02, message=ether4 link down
topics=interface,info, .id=*109A, time=Mar/24/2020 21:50:04, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*109B, time=Mar/24/2020 21:50:04, message=ether4 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*1073, time=Mar/24/2020 21:50:09, message=OSPFv2 neighbor 100.64.100.1: state change from Init to Down
topics=route,ospf,info, .id=*1073, time=Mar/24/2020 21:50:09, message=OSPFv2 neighbor 100.64.100.1: state change from Init to Down
topics=interface,info, .id=*1074, time=Mar/24/2020 21:50:09, message=ether5 link down
topics=interface,info, .id=*1075, time=Mar/24/2020 21:50:09, message=ether4 link down
topics=interface,info, .id=*1069, time=Mar/24/2020 21:50:11, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*106A, time=Mar/24/2020 21:50:11, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*A1B, time=Mar/27/2020 21:27:03, message=ether5 link down
topics=interface,info, .id=*A1A, time=Mar/27/2020 21:27:23, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*2B0, time=Mar/29/2020 02:15:03, message=ether5 link down
topics=system,error,critical, .id=*24683, time=Mar/29/2020 02:17:01, message=router was rebooted without proper shutdown
topics=interface,info, .id=*24684, time=Mar/29/2020 02:17:01, message=vlan113-sfp-sfpplus1 link up
topics=interface,info, .id=*24685, time=Mar/29/2020 02:17:01, message=vlan10-sfp-combo1 link up
topics=interface,info, .id=*24686, time=Mar/29/2020 02:17:02, message=vlan1000-ether4 link up
topics=interface,info, .id=*24687, time=Mar/29/2020 02:17:02, message=vlan10-ether4 link up
topics=interface,info, .id=*24688, time=Mar/29/2020 02:17:02, message=vlan2000-ether4 link up
topics=interface,info, .id=*24689, time=Mar/29/2020 02:17:02, message=vlan1001-sfp-combo1 link up
topics=system,error,critical, .id=*CC5C, time=Mar/29/2020 02:17:02, message=router was rebooted without proper shutdown
topics=interface,info, .id=*2468D, time=Mar/29/2020 02:17:03, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*2A4, time=Mar/29/2020 02:17:03, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*2A5, time=Mar/29/2020 02:17:03, message=vlan1001-sfp-combo1 link up
topics=interface,info, .id=*2A6, time=Mar/29/2020 02:17:03, message=vlan2000-ether4 link up
topics=interface,info, .id=*2A7, time=Mar/29/2020 02:17:03, message=vlan10-ether4 link up
topics=interface,info, .id=*2A8, time=Mar/29/2020 02:17:03, message=vlan1000-ether4 link up
topics=interface,info, .id=*2A9, time=Mar/29/2020 02:17:03, message=vlan10-sfp-combo1 link up
topics=interface,info, .id=*2AA, time=Mar/29/2020 02:17:03, message=vlan113-sfp-sfpplus1 link up
topics=interface,info, .id=*24691, time=Mar/29/2020 02:17:04, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*24692, time=Mar/29/2020 02:17:04, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*24693, time=Mar/29/2020 02:17:04, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*29F, time=Mar/29/2020 02:17:04, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*2A0, time=Mar/29/2020 02:17:04, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*24694, time=Mar/29/2020 02:17:05, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*24695, time=Mar/29/2020 02:17:05, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*29C, time=Mar/29/2020 02:17:05, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*29D, time=Mar/29/2020 02:17:05, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*29E, time=Mar/29/2020 02:17:05, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*24696, time=Mar/29/2020 02:27:32, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*2AE, time=Mar/29/2020 02:39:39, message=sfp-combo1 link down
topics=interface,info, .id=*291, time=Mar/29/2020 03:45:59, message=sfp-combo1 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*24A54, time=Mar/29/2020 05:45:50, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24A54, time=Mar/29/2020 05:45:50, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24A55, time=Mar/29/2020 05:45:50, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24A55, time=Mar/29/2020 05:45:50, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*24A56, time=Mar/29/2020 05:45:50, message=sfp-sfpplus1 link down
topics=interface,info, .id=*24A57, time=Mar/29/2020 05:45:50, message=sfp-combo1 link down
topics=interface,info, .id=*24A58, time=Mar/29/2020 05:45:50, message=ether1 link down
topics=interface,info, .id=*24A59, time=Mar/29/2020 05:45:50, message=ether3 link down
topics=interface,info, .id=*24A5A, time=Mar/29/2020 05:45:50, message=ether4 link down
topics=interface,info, .id=*24A5C, time=Mar/29/2020 05:45:51, message=ether5 link down
topics=interface,info, .id=*24A5D, time=Mar/29/2020 05:45:51, message=ether7 link down
topics=interface,info, .id=*24A60, time=Mar/29/2020 05:45:52, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*24A61, time=Mar/29/2020 05:45:52, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*24A63, time=Mar/29/2020 05:45:53, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*24A64, time=Mar/29/2020 05:45:53, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*24A65, time=Mar/29/2020 05:45:53, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*24A66, time=Mar/29/2020 05:45:54, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*24A67, time=Mar/29/2020 05:45:54, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B44, time=Mar/29/2020 10:07:43, message=sfp-sfpplus1 link down
topics=interface,info, .id=*24B45, time=Mar/29/2020 10:07:43, message=sfp-combo1 link down
topics=interface,info, .id=*24B46, time=Mar/29/2020 10:07:43, message=ether1 link down
topics=interface,info, .id=*24B47, time=Mar/29/2020 10:07:43, message=ether3 link down
topics=interface,info, .id=*24B48, time=Mar/29/2020 10:07:43, message=ether4 link down
topics=interface,info, .id=*24B49, time=Mar/29/2020 10:07:43, message=ether5 link down
topics=interface,info, .id=*24B4A, time=Mar/29/2020 10:07:43, message=ether7 link down
topics=route,ospf,info, .id=*24B4B, time=Mar/29/2020 10:07:43, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24B4B, time=Mar/29/2020 10:07:43, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24B4C, time=Mar/29/2020 10:07:43, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24B4C, time=Mar/29/2020 10:07:43, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*24B50, time=Mar/29/2020 10:07:44, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B53, time=Mar/29/2020 10:07:45, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*24B54, time=Mar/29/2020 10:07:45, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B56, time=Mar/29/2020 10:07:45, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*24B57, time=Mar/29/2020 10:07:46, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*24B58, time=Mar/29/2020 10:07:47, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B59, time=Mar/29/2020 10:07:48, message=sfp-combo1 link up (speed 1G, full duplex)
topics=route,ospf,info, .id=*24B71, time=Mar/29/2020 17:40:11, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24B71, time=Mar/29/2020 17:40:11, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24B72, time=Mar/29/2020 17:40:11, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=route,ospf,info, .id=*24B72, time=Mar/29/2020 17:40:11, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Down
topics=interface,info, .id=*24B73, time=Mar/29/2020 17:40:11, message=sfp-sfpplus1 link down
topics=interface,info, .id=*24B74, time=Mar/29/2020 17:40:11, message=sfp-combo1 link down
topics=interface,info, .id=*24B75, time=Mar/29/2020 17:40:11, message=ether1 link down
topics=interface,info, .id=*24B76, time=Mar/29/2020 17:40:11, message=ether3 link down
topics=interface,info, .id=*24B77, time=Mar/29/2020 17:40:11, message=ether4 link down
topics=interface,info, .id=*24B79, time=Mar/29/2020 17:40:12, message=ether5 link down
topics=interface,info, .id=*24B7A, time=Mar/29/2020 17:40:12, message=ether7 link down
topics=interface,info, .id=*24B7D, time=Mar/29/2020 17:40:13, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B7E, time=Mar/29/2020 17:40:13, message=ether5 link up (speed 100M, full duplex)
topics=interface,info, .id=*24B80, time=Mar/29/2020 17:40:14, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*24B81, time=Mar/29/2020 17:40:14, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B82, time=Mar/29/2020 17:40:15, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*24B83, time=Mar/29/2020 17:40:16, message=sfp-combo1 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B84, time=Mar/29/2020 17:40:16, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*24B8B, time=Mar/29/2020 18:23:41, message=ether5 link down
topics=interface,info, .id=*6, time=Mar/29/2020 19:07:24, message=ether5 link down
topics=interface,info, .id=*2512A, time=Mar/30/2020 23:07:27, message=ether1 link down
topics=interface,info, .id=*2512C, time=Mar/30/2020 23:07:29, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*25133, time=Mar/30/2020 23:11:35, message=ether1 link down
topics=interface,info, .id=*25138, time=Mar/30/2020 23:12:24, message=ether1 link up (speed 10M, half duplex)
topics=route,ospf,info, .id=*25155, time=Mar/30/2020 23:32:49, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Init
topics=route,ospf,info, .id=*25155, time=Mar/30/2020 23:32:49, message=OSPFv3 neighbor 100.64.100.1: state change from Full to Init
topics=route,ospf,info, .id=*25156, time=Mar/30/2020 23:32:49, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Init
topics=route,ospf,info, .id=*25156, time=Mar/30/2020 23:32:49, message=OSPFv2 neighbor 100.64.100.1: state change from Full to Init
topics=route,ospf,info, .id=*25159, time=Mar/30/2020 23:51:56, message=OSPFv2 neighbor 100.64.100.1: state change from Init to Down
topics=route,ospf,info, .id=*25159, time=Mar/30/2020 23:51:56, message=OSPFv2 neighbor 100.64.100.1: state change from Init to Down
topics=route,ospf,info, .id=*2515A, time=Mar/30/2020 23:51:56, message=OSPFv3 neighbor 100.64.100.1: state change from Init to Down
topics=route,ospf,info, .id=*2515A, time=Mar/30/2020 23:51:56, message=OSPFv3 neighbor 100.64.100.1: state change from Init to Down
topics=interface,info, .id=*2515B, time=Mar/30/2020 23:51:56, message=sfp-sfpplus1 link down
topics=interface,info, .id=*2515C, time=Mar/30/2020 23:51:56, message=sfp-combo1 link down
topics=interface,info, .id=*2515D, time=Mar/30/2020 23:51:56, message=ether1 link down
topics=interface,info, .id=*2515E, time=Mar/30/2020 23:51:56, message=ether3 link down
topics=interface,info, .id=*2515F, time=Mar/30/2020 23:51:56, message=ether4 link down
topics=interface,info, .id=*25161, time=Mar/30/2020 23:51:56, message=ether7 link down
topics=interface,info, .id=*25165, time=Mar/30/2020 23:51:57, message=sfp-sfpplus1 link up (speed 1G, full duplex)
topics=interface,info, .id=*25166, time=Mar/30/2020 23:51:58, message=ether1 link up (speed 10M, half duplex)
topics=interface,info, .id=*25167, time=Mar/30/2020 23:51:58, message=ether3 link up (speed 100M, full duplex)
topics=interface,info, .id=*25168, time=Mar/30/2020 23:51:58, message=ether4 link up (speed 1G, full duplex)
topics=interface,info, .id=*25169, time=Mar/30/2020 23:51:59, message=ether7 link up (speed 1G, full duplex)
topics=interface,info, .id=*2516A, time=Mar/30/2020 23:52:00, message=sfp-combo1 link up (speed 1G, full duplex)
Last edited by davidalain on Sun Apr 05, 2020 7:34 pm, edited 1 time in total.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7038
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: OSPF disabling all ports on 'state change from Full to Down'

Wed Apr 01, 2020 12:10 pm

OSPF goes down because physical links are flapping, it can be seen in your logs. Logs are written asynchronously, so when difference between events are in milliseconds interface flap may appear after OSPF message.
 
sid5632
Long time Member
Long time Member
Posts: 552
Joined: Fri Feb 17, 2017 6:05 pm

Re: OSPF disabling all ports on 'state change from Full to Down'

Thu Apr 02, 2020 3:29 pm

Logs are written asynchronously
Why? That makes problems a lot harder to diagnose and you can never be sure what is a cause and what is an effect.
Last edited by sid5632 on Mon Apr 06, 2020 2:55 am, edited 1 time in total.
 
davidalain
just joined
Topic Author
Posts: 16
Joined: Thu Aug 03, 2017 3:24 am
Location: Brazil

Re: OSPF disabling all ports on 'state change from Full to Down'

Sun Apr 05, 2020 8:07 am

OSPF goes down because physical links are flapping, it can be seen in your logs. Logs are written asynchronously, so when difference between events are in milliseconds interface flap may appear after OSPF message.
All connected physical links flapping at once is a very strange behavior. Is it a hardware issue?
 
davidalain
just joined
Topic Author
Posts: 16
Joined: Thu Aug 03, 2017 3:24 am
Location: Brazil

Re: OSPF disabling all ports on 'state change from Full to Down'

Sun Apr 05, 2020 6:05 pm

Today I looked at link-down and link-up informations and got another issue.

The information about date/time on link-down and link-up shown in console is different from Interface window into Winbox.
Also, in Winbox the time is about 3 days in future, as you can see in the attached image.

The system clock is synchonized by NTP servers. It is correct.

Image
 
davidalain
just joined
Topic Author
Posts: 16
Joined: Thu Aug 03, 2017 3:24 am
Location: Brazil

Re: OSPF disabling all ports on 'state change from Full to Down'

Sun Apr 26, 2020 10:59 pm

I believe there no issue in OSPF but I am pretty sure there are some incompatibilities in Ethernet ports with some other devices or another physical issue.
Another symthom of this issue is last link up or down time for those ports in Winbox is shown as happened in a future time.

I found many mentions here in MT forum about many ports with link down at the same time.
And most of them have solved that issue by changing ports or other devices, and also have not solved.

In my case, after do not using some some links, I have not faced this issue again.
I was using one port of CCR1009 connected to a 10 Mbps Half-duplex MGMT OLT port. This caused often link down/up.
I haved connected that OLT to a port of a RB1100AHx2 and its Ethernet port have no issues when connected to the MGMT OLT port.
OSPF goes down because physical links are flapping, it can be seen in your logs. Logs are written asynchronously, so when difference between events are in milliseconds interface flap may appear after OSPF message.
Thus, @mrz you are correct, the problem may not be in the OSPF itself.
But maybe there is an Ethernet ports issue that disconnects all ports at the same time.

A friend that uses a RB3011 have faced same issues with random 'link down' in many ports at the same time, but only in ports from same switch-chip (ports 6 to 10).
In a CCR1009-7G-1S-1S+ there are no switch chips. All ports are directly connected to CPU. So, I don't know what is the relation to switch-chip about this issue.

I think this issue is not caused by a last RouterOS version bug because there are many mentions about this issue since 2014 with many router/switch models and RouterOS versions.
Some of them are:

Jul 18, 2014:
viewtopic.php?t=87134

Feb 08, 2016:
viewtopic.php?t=104622

Feb 16, 2016:
viewtopic.php?t=104899

Jul 02, 2018:
viewtopic.php?t=136376

Apr 05, 2020:
viewtopic.php?f=1&t=159575

Apr 25, 2020:
viewtopic.php?f=13&t=160381


Please Mikrotik, take a look at this!!
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7038
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: OSPF disabling all ports on 'state change from Full to Down'

Mon Apr 27, 2020 10:14 am

CCR does not work correctly with half duplex links, it is hardware limitation. For interface not to hang completely it is occasionally flapped.

Regarding RB3011, if switch group is overloaded it will reset the switch group
One workaround is:
/interface ethernet switch set switch1,switch2 cpu-flow-control=no
It should prevent port resetting due to long packet delays to the CPU.
 
mhrivnatz
just joined
Posts: 1
Joined: Wed May 13, 2020 5:48 pm

Re: OSPF disabling all ports on 'state change from Full to Down'

Wed May 13, 2020 6:34 pm

Had this same issue -> normal stable operation on a CCR1016-12G before upgrade to 6.45.8. Upgraded from 6.43.14 to the latest (at the time) LT train of 6.45.8. Right afterwards started getting the same thing (all interfaces DOWN and then 1 second later -> all interfaces UP) which would trigger goofiness in OSPF. If router was a stub with just default route that would maybe be of little issue but the position of this router made for a sizable problem -> it was a hub with three separate rings and one stub hanging off it. The 1st two times it did it, I simply MAC telnet’ed to it and ran a CLI reboot command. It wasn’t until the 3rd time where I assumed some systemic problem and got into troubleshooting vs restoring service. It was the same issue as you described -> all ethernet interfaces down, then 1 second later -> all up, OSPF flaps and routes are then NOT distributed. For a work around, I placed a simple script that looked for a route that OSPF would have placed into the routing table (if it was working correctly) if missing it triggered a physical interface DISABLE for 45 seconds which would drop the OSPF neighbor session based on the 40s dead router timer. That worked OK, but sometimes it didn’t work on the 1st try and you had a “waiting period” for these routing checks . So, this script solution could “fix” the problem but would allow for a several minute outages randomly whenever the router decided to puke.
Right before reverting back ROS as the script solution didn’t seem “good enough” and the frequency it was happening at was more than I was willing to accept….-> Mikrotik released LT 6.45.9 with “system – improved system stability when forwarding traffic from switch chip to CPU”. I upgraded to 6.45.9 but still no change in behavior-> interfaces continued resets randomly -> sometimes every other day, sometimes every three days, but the longest it went in between interface pukes was about 4 days . But the average seemed to be about every other day while running 6.45.9. After the 2nd time it happened after the 6.45.9 upgrade, I decided to rollback down to the last running version of ROS on this device.

I reverted back down to 6.43.16 and have been running that clean for 2 days without issue now. Fingers crossed….

I am assuming the issue is probably some combination of this 6.45.X version with some HW version of some chip on some of the CCRs -> as I have 6.45.8 running on a few other CCRs with similar configs for the last 2-3 months and this one is the only one with this issue…

Good luck to all....
++++++++++++++
1 Day later update -> now on 3rd day after roll back to 6.43.16.
Still clean -> no ethernet interface resets and OSPF sessions have been ALIVE for 2Days 22Hours & 28 minutes
In one more day I am willing to call this "fixed"
Last edited by mhrivnatz on Thu May 14, 2020 5:09 pm, edited 1 time in total.
 
infectious
just joined
Posts: 6
Joined: Mon Apr 13, 2020 5:27 pm

Re: OSPF disabling all ports on 'state change from Full to Down'

Fri Jun 19, 2020 8:05 pm

mhrivnatz
I'm having the same problem with one of my CCR's (I have 6 others, with almost the same settings and 6.45.9 firmware).
Symptoms and recurrences are identical to yours.

By the way, I have 250 Routerboards of different models on the network (54 with ospf) and none of them are showing this problem.

Can you share your script?
 
infectious
just joined
Posts: 6
Joined: Mon Apr 13, 2020 5:27 pm

Re: OSPF disabling all ports on 'state change from Full to Down'

Mon Jun 29, 2020 11:58 pm

No solution yet.

Even downgrading to 6.45.8

anyone?!
 
darkorigins
just joined
Posts: 3
Joined: Thu Jul 02, 2015 9:28 am

Re: OSPF disabling all ports on 'state change from Full to Down'

Fri Jul 03, 2020 9:44 am

Hi All

I've also been seeing this all interfaces down/up flap on my CCR for some while now. Frequency was a little random, sometimes 2-3 times an hour, sometimes once every 2-3 hours. Have been through the theories of causes. OSPF, half-duplex, a problematic port pulling the rest down and hoping it wasn't the failing PSU theory!

Anyway ... just upgraded system/packages and system/routerboard to 6.47 ... early days but so far 12 hours in and not a single port flap.

Model CCR1009-8G-1S
Serial Number 4ABA02Fxxxxx
Factory Firmware 3.13
Current Firmware 6.47
Upgrade Firmware 6.47


Update 20 hours in - one all interface down/up. So better but not fixed :-(
2 days in - back to every few hours
 
gfrazier
just joined
Posts: 3
Joined: Wed Apr 16, 2014 5:42 pm

Re: OSPF disabling all ports on 'state change from Full to Down'

Fri Jan 22, 2021 5:17 pm

I am having a similar issue with OSPF on one RB3011, but am not seeing the port flapping errors. This is the only error I am getting. I have upgraded from 6.45.9 to 6.46.8, but there has been no change.
You do not have the required permissions to view the files attached to this post.
 
mikeeg02
Member Candidate
Member Candidate
Posts: 162
Joined: Fri Mar 30, 2018 2:28 am
Location: Pennsylvania

Re: OSPF disabling all ports on 'state change from Full to Down'

Sat Jan 23, 2021 6:42 am

No solution yet.

Even downgrading to 6.45.8

anyone?!
I've been running the 6.47 variant in my 3011s and ccr1009s, without port flapping issues. They have uptime since I upgraded to either 6.47.7 or 6.47.8, so 45-60 days ago. All are running ospf and mpls /vpls. I've migrated to the later versions of 6.47 for the arm stability issues fixes for the problems I've experienced with my 1100ah4s. 6.47 tree seems stable for these, including ccr1009s. My only issue with them has been interfacing to one Cisco router. If my port state ever went down (upgrade reboot) the only way I can restore the layer 1 connection to the Cisco is to admin the Cisco port down, then up. There is no other way I can get the link established. I've since swapped that out for a 3011 and that problem went away. That's my only ccr1009 issue to date.

Who is online

Users browsing this forum: galp and 15 guests