Flapping ethernet links on CRS312 - frequent & intermittent network disruptions

gerhard · Wed Jul 13, 2022 1:54 pm

I have been noticing various ethernet links going down and then back up again after a few seconds on CRS312.

This is what the looks like in the logs (all from today, July 13, 2022):

00:14:05 interface,info ether6-focker link down
00:14:08 interface,info ether6-focker link up (speed 100M, full duplex)
00:14:08 interface,info ether6-focker detect LAN
00:50:31 interface,info ether6-focker link down
00:50:37 interface,info ether6-focker link up (speed 10G, full duplex)
00:50:37 interface,info ether6-focker detect LAN
01:15:12 interface,info ether6-focker link down
01:15:15 interface,info ether6-focker link up (speed 100M, full duplex)
01:15:15 interface,info ether6-focker detect LAN
02:49:46 interface,info ether6-focker link down
02:49:52 interface,info ether6-focker link up (speed 10G, full duplex)
02:49:52 interface,info ether6-focker detect LAN
03:19:55 interface,info ether6-focker link down
03:19:58 interface,info ether6-focker link up (speed 100M, full duplex)
03:19:58 interface,info ether6-focker detect LAN
03:39:42 interface,info ether3-udmpro link down
03:39:45 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:39:45 interface,info ether3-udmpro detect LAN
03:39:48 interface,info ether3-udmpro link down
03:39:51 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:39:51 interface,info ether3-udmpro detect LAN
03:39:52 interface,info ether3-udmpro link down
03:39:55 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:39:55 interface,info ether3-udmpro detect LAN
03:39:56 interface,info ether3-udmpro link down
03:39:59 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:39:59 interface,info ether3-udmpro detect LAN
03:40:01 interface,info ether3-udmpro link down
03:40:04 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:04 interface,info ether3-udmpro detect LAN
03:40:08 interface,info ether3-udmpro link down
03:40:11 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:11 interface,info ether3-udmpro detect LAN
03:40:16 interface,info ether3-udmpro link down
03:40:19 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:19 interface,info ether3-udmpro detect LAN
03:40:27 interface,info ether3-udmpro link down
03:40:30 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:30 interface,info ether3-udmpro detect LAN
03:40:36 interface,info ether3-udmpro link down
03:40:39 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:39 interface,info ether3-udmpro detect LAN
03:40:40 interface,info ether3-udmpro link down
03:40:43 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:43 interface,info ether3-udmpro detect LAN
03:40:44 interface,info ether3-udmpro link down
03:40:47 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:47 interface,info ether3-udmpro detect LAN
03:40:48 interface,info ether3-udmpro link down
03:40:52 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:40:52 interface,info ether3-udmpro detect LAN
03:41:14 interface,info ether3-udmpro link down
03:41:18 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:41:18 interface,info ether3-udmpro detect LAN
03:41:19 interface,info ether3-udmpro link down
03:41:22 interface,info ether3-udmpro link up (speed 1G, full duplex)
03:41:22 interface,info ether3-udmpro detect LAN
04:44:45 interface,info ether5-x22 link down
04:44:59 interface,info ether5-x22 link up (speed 10G, full duplex)
04:44:59 interface,info ether5-x22 detect LAN
04:52:20 interface,info ether5-x22 link down
04:52:44 interface,info ether5-x22 link up (speed 10G, full duplex)
04:52:44 interface,info ether5-x22 detect LAN
05:19:56 interface,info ether6-focker link down
05:20:01 interface,info ether6-focker link up (speed 10G, full duplex)
05:20:01 interface,info ether6-focker detect LAN
05:20:49 interface,info ether6-focker link down
05:20:52 interface,info ether6-focker link up (speed 100M, full duplex)
05:20:52 interface,info ether6-focker detect LAN
07:20:50 interface,info ether6-focker link down
07:20:56 interface,info ether6-focker link up (speed 10G, full duplex)
07:20:56 interface,info ether6-focker detect LAN
07:50:59 interface,info ether6-focker link down
07:51:02 interface,info ether6-focker link up (speed 100M, full duplex)
07:51:02 interface,info ether6-focker detect LAN
08:00:03 interface,info ether6-focker link down
08:00:09 interface,info ether6-focker link up (speed 10G, full duplex)
08:00:09 interface,info ether6-focker detect LAN

These are all CRS312 interfaces:

[admin@MikrotTik.CRS312] > /interface print
Flags: D - dynamic, X - disabled, R - running, S - slave
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS
 0  XS combo1                              ether            1500  1584      10218 DC:2C:6E:90:54:60
 1  XS combo2                              ether            1500  1584      10218 DC:2C:6E:90:54:61
 2  XS combo3                              ether            1500  1584      10218 DC:2C:6E:90:54:62
 3  XS combo4                              ether            1500  1584      10218 DC:2C:6E:90:54:63
 4  RS ether1-rb5009                       ether            1500  1584      10218 DC:2C:6E:90:54:58
 5  XS ether2                              ether            1500  1584      10218 DC:2C:6E:90:54:59
 6  RS ether3-udmpro                       ether            1500  1584      10218 DC:2C:6E:90:54:5A
 7  XS ether4                              ether            1500  1584      10218 DC:2C:6E:90:54:5B
 8  RS ether5-x22                          ether            1500  1584      10218 DC:2C:6E:90:54:5C
 9  RS ether6-focker                       ether            1500  1584      10218 DC:2C:6E:90:54:5D
10  XS ether7                              ether            1500  1584      10218 DC:2C:6E:90:54:5E
11  XS ether8                              ether            1500  1584      10218 DC:2C:6E:90:54:5F
12  XS ether9                              ether            1500  1598       2028 DC:2C:6E:90:54:64
13  R  ;;; defconf
       bridge                              bridge           1500  1584            DC:2C:6E:90:54:60

While the device using ether6-focker goes into sleep mode and switches to 100M (runs at 10G otherwise), ether5-x22 & ether3-udmpro are always on.

I find it suspicious that ether1-rb5009 does not seem affected. It might be unrelated, but only non-MikroTik devices seem to be affected.

This is the relevant network topology:

MikroTik CRS312
|
|-> ether1-rb5009 - RB5009 (1Gbps) - terminates PPPoE WAN connections
|                                  - runs DHCP for all devices connecting into CRS312
|
|-> ether3-udmpro - UDMPro (1Gbps) - always on (all wireless devices affected when link goes down)
|
|-> ether5-x22    - Linux (10Gbps) - always on (affects pinger when link goes down)
|-> ether6-focker - Mac   (10Gbps) - sometimes sleeps (highly sensitive to link downs during the day)

This is how the link down affects https://github.com/nefilim/pinger which runs on x22:

And this is the RouterBOARD info:

[admin@MikrotTik.CRS312] > /system routerboard print
       routerboard: yes
             model: CRS312-4C+8XG
          revision: r2
     serial-number: D84B0F3F5165
     firmware-type: qca9531L
  factory-firmware: 6.47.10
  current-firmware: 6.48.6
  upgrade-firmware: 6.48.6

Has anyone seen this happen before? How do I continue troubleshooting this?

gerhard · Wed Jul 13, 2022 10:19 pm

Since my last post (8am), I had another 28 link down events:

12:41:12 interface,info ether3-udmpro link down
12:41:15 interface,info ether3-udmpro link up (speed 1G, full duplex)
12:41:15 interface,info ether3-udmpro detect LAN
12:41:20 interface,info ether3-udmpro link down
12:41:31 interface,info ether3-udmpro link up (speed 1G, full duplex)
12:41:31 interface,info ether3-udmpro detect LAN
12:45:12 interface,info ether3-udmpro link down
12:45:15 interface,info ether3-udmpro link up (speed 1G, full duplex)
12:45:15 interface,info ether3-udmpro detect LAN
12:56:24 interface,info ether6-focker link down
12:56:27 interface,info ether6-focker link up (speed 100M, full duplex)
12:56:27 interface,info ether6-focker detect LAN
13:12:20 interface,info ether6-focker link down
13:12:25 interface,info ether6-focker link up (speed 10G, full duplex)
13:12:25 interface,info ether6-focker detect LAN
13:14:32 interface,info ether6-focker link down
13:14:36 interface,info ether6-focker link up (speed 100M, full duplex)
13:14:36 interface,info ether6-focker detect LAN
13:50:20 interface,info ether6-focker link down
13:50:25 interface,info ether6-focker link up (speed 10G, full duplex)
13:50:25 interface,info ether6-focker detect LAN
14:15:57 interface,info ether3-udmpro link down
14:16:00 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:16:00 interface,info ether3-udmpro detect LAN
14:16:04 interface,info ether3-udmpro link down
14:16:07 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:16:07 interface,info ether3-udmpro detect LAN
14:16:14 interface,info ether3-udmpro link down
14:16:18 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:16:18 interface,info ether3-udmpro detect LAN
14:16:21 interface,info ether3-udmpro link down
14:16:24 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:16:24 interface,info ether3-udmpro detect LAN
14:16:25 interface,info ether3-udmpro link down
14:16:29 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:16:29 interface,info ether3-udmpro detect LAN
14:16:49 interface,info ether3-udmpro link down
14:16:52 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:16:52 interface,info ether3-udmpro detect LAN
14:16:57 interface,info ether3-udmpro link down
14:17:01 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:17:01 interface,info ether3-udmpro detect LAN
14:17:02 interface,info ether3-udmpro link down
14:17:05 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:17:05 interface,info ether3-udmpro detect LAN
14:17:12 interface,info ether3-udmpro link down
14:17:15 interface,info ether3-udmpro link up (speed 1G, full duplex)
14:17:15 interface,info ether3-udmpro detect LAN
14:20:29 interface,info ether6-focker link down
14:20:32 interface,info ether6-focker link up (speed 100M, full duplex)
14:20:32 interface,info ether6-focker detect LAN
15:17:28 interface,info ether3-udmpro link down
15:17:31 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:17:31 interface,info ether3-udmpro detect LAN
15:17:37 interface,info ether3-udmpro link down
15:17:40 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:17:40 interface,info ether3-udmpro detect LAN
15:18:01 interface,info ether3-udmpro link down
15:18:04 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:18:04 interface,info ether3-udmpro detect LAN
15:18:13 interface,info ether3-udmpro link down
15:18:16 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:18:16 interface,info ether3-udmpro detect LAN
15:18:21 interface,info ether3-udmpro link down
15:18:24 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:18:24 interface,info ether3-udmpro detect LAN
15:18:25 interface,info ether3-udmpro link down
15:18:28 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:18:28 interface,info ether3-udmpro detect LAN
15:18:59 interface,info ether3-udmpro link down
15:19:03 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:19:03 interface,info ether3-udmpro detect LAN
15:25:12 interface,info ether3-udmpro link down
15:25:25 interface,info ether3-udmpro link up (speed 1G, full duplex)
15:25:25 interface,info ether3-udmpro detect LAN
16:20:31 interface,info ether6-focker link down
16:20:36 interface,info ether6-focker link up (speed 10G, full duplex)
16:20:36 interface,info ether6-focker detect LAN
16:50:38 interface,info ether6-focker link down
16:50:42 interface,info ether6-focker link up (speed 100M, full duplex)
16:50:42 interface,info ether6-focker detect LAN
17:22:18 interface,info ether6-focker link down
17:22:23 interface,info ether6-focker link up (speed 10G, full duplex)
17:22:23 interface,info ether6-focker detect LAN

What is worse, packets going through the CRS312 (192.168.88.2 - trunk switch) to RB5009 (192.168.88.1 - WAN router) have brief periods of 20x increased latency (0.3ms to 6ms).

ICMP latency Linux x22 -> crs312 (192.168.88.2):

Same period, Linux x22 -> rb5009 (192.168.88.1):

Any ideas?

gerhard · Sat Jul 30, 2022 1:43 pm

TL;DR: Switching from a UTP to a SFP+ DAC patch cable made my intermittent network disruptions go away. I still don't understand why.

I was running a Vention Cat. 6A UTP 0.5m patch cable when these intermittent network disruptions were occurring. I have since switched to a 10G SFP+ DAC between the CRS312 & UDM Pro and I have not had a single disconnect since July 16, meaning the last 14 days.

The Linux host had a single disconnect in the same period. I continue to monitor it closely. The Linux host is connected via the same Vention Cat. 6A UTP 0.5m patch cable. The cable between the patch panel and wall data socket is an industrial Cat 6A SSTP that has been in constant use for 2 years with no issues. Data socket to physical host is connected via Vention Cat 7 SSTP. I have replaced both Ventions to rule out the cables.

- https://www.ventioncable.com/product/ca ... t-cable-2/
- https://www.ventioncable.com/product/ca ... slim-type/

For reference, this is a point-in-time picture of my current network cabinet setup. Notice the blue SSTP & black SFP+ DAC running from CRS312 to UDM Pro's two separate WAN interfaces. The SSTP proved unnecessary, my issues went away after I connected these two devices via SFP+ DAC.

The cables are nowhere near as tidy as they could be. Consider the above a work-in-progress view.

Flapping ethernet links on CRS312 - frequent & intermittent network disruptions [SOLVED]

Flapping ethernet links on CRS312 - frequent & intermittent network disruptions

Re: Flapping ethernet links on CRS312 - frequent & intermittent network disruptions

Re: Flapping ethernet links on CRS312 - frequent & intermittent network disruptions [SOLVED]

Who is online