Hello folks, we are observing a condition that shows as all interfaces simply flapping physical link state. The only physical interface not affected is ether1 which we know is direct to the CPU. See logs:
Jan/10/2025 01:52:35 interface,info sfp28-1 link down
Jan/10/2025 01:52:36 interface,info sfp28-2 link down
Jan/10/2025 01:52:36 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.130%*a } destroyed
Jan/10/2025 01:52:36 interface,info sfp28-3 link down
Jan/10/2025 01:52:36 interface,info sfp28-4 link down
Jan/10/2025 01:52:36 interface,info sfp28-5 link down
Jan/10/2025 01:52:36 interface,info sfp28-6 link down
Jan/10/2025 01:52:36 interface,info sfp28-8 link down
Jan/10/2025 01:52:36 interface,info sfp28-9 link down
Jan/10/2025 01:52:36 interface,info sfp28-11 link down
Jan/10/2025 01:52:36 interface,info sfp28-12 link down
Jan/10/2025 01:52:36 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.134%*b } destroyed
Jan/10/2025 01:52:38 interface,info sfp28-3 link up (speed 10G, full duplex)
Jan/10/2025 01:52:38 interface,info sfp28-5 link up (speed 10G, full duplex)
Jan/10/2025 01:52:39 interface,info sfp28-4 link up (speed 10G, full duplex)
Jan/10/2025 01:52:39 interface,info sfp28-6 link up (speed 10G, full duplex)
Jan/10/2025 01:52:40 interface,info sfp28-1 link up (speed 25G, full duplex)
Jan/10/2025 01:52:40 interface,info sfp28-2 link up (speed 25G, full duplex)
Jan/10/2025 01:52:40 interface,info sfp28-8 link up (speed 1G, full duplex)
Jan/10/2025 01:52:40 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.130%sfp28-1 } created
Jan/10/2025 01:52:40 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.130%sfp28-1 } state change to Point-to-Point
Jan/10/2025 01:52:40 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.134%sfp28-2 } created
Jan/10/2025 01:52:40 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.134%sfp28-2 } state change to Point-to-Point
Jan/10/2025 01:52:41 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.134%sfp28-2 } neighbor { router-id: 10.50.255.1 state: Down } state change to Init
Jan/10/2025 01:52:41 route,ospf,info ospf-instance-1 { version: 2 router-id: 10.50.255.2 } ospf-area-1 { 0.0.0.0 } interface { p2p 10.255.255.130%sfp28-1 } neighbor { router-id: 10.50.255.1 state: Down } state change to Init
Jan/10/2025 01:52:41 interface,info sfp28-9 link up (speed 1G, full duplex)
Jan/10/2025 01:52:41 interface,info sfp28-10 link up (speed 1G, full duplex)
Jan/10/2025 01:52:41 interface,info sfp28-11 link up (speed 1G, full duplex)
Jan/10/2025 01:52:41 interface,info sfp28-12 link up (speed 1G, full duplex)
You will see some OSPF and other underlying log entries recorded that are associated with an interface drop but nothing else to indicate the root cause.
We have tried:
- disabling each interface port by port
- checking loop protect (always shows off)
- disabling bridging
- replacing optics from FS.COM to MikroTik
We have checked CPU memory, traffic patterns on 60 second PRTG polling. All comes back clean and steady normal state on the graphs.
The only thing that resolves this issue is rebooting the router. RouterOS version is 7.16 and we just upgraded to 7.16.2 tonight to try to resolve this. The issue usually manifests every 1-2 days but this last occurrence lasted over 10 days with no problem until now. These ports go to various different devices so it is not something in common with a singular far side device. I suspect a bug or problem on the switch chip itself but ability to drop down and debug at that level is pretty limited.
Does anyone have any thoughts on this or seen anything like this?