Link up/down in

RouterOS General
1

Hello!

I have several MikroTiks connected by an IPoE tunnel.
On the router “kfood”, “uluga” and so on there is a tunnel to “kvp” and “kdk”. There is also a tunnel between “kvp” and “kdk”.
All tunnel interfaces are added to the bridge on each router. It turns out that all switches (sw01, sw02, sw03…) are combined together. RSTP is enabled. The root RSTP switch - D-Link DGS-3120-24SC is located next to sw01.
On all switches, the root port is “kvp” (except “kvp” itself).

In the logs of the routers on the left, I see a constant link up / link down associated with the “kdk” router. I think it has something to do with RSTP working, but I can’t figure out how to fix it.

I would appreciate any help, thanks.

Log “kfood”

08:40:23 system,info router rebooted
08:40:33 bridge,info hardware offloading activated on bridge “bridge” ports: ether2,ether3,ether4,ether5
08:40:36 interface,info ether1 link up (speed 1G, full duplex)
08:40:36 interface,info ether2 link up (speed 1G, full duplex)
08:40:37 interface,info kdk link up
08:40:37 interface,info kvp link up
08:40:40 system,info,account user admin logged in from 192.168.100.114 via winbox
08:41:29 system,info sntp change time Mar/01/2022 08:40:55 => Mar/01/2022 08:41:29
08:42:52 interface,info kdk link down
08:42:58 interface,info kdk link up
08:44:39 interface,info kdk link down
08:44:44 interface,info kdk link up
08:46:25 interface,info kdk link down
08:46:31 interface,info kdk link up
08:48:12 interface,info kdk link down
08:48:17 interface,info kdk link up
08:49:58 interface,info kdk link down
08:50:03 interface,info kdk link up
08:51:44 interface,info kdk link down
08:51:49 interface,info kdk link up
08:53:30 interface,info kdk link down
08:53:35 interface,info kdk link up
08:55:16 interface,info kdk link down
08:55:22 interface,info kdk link up
08:57:03 interface,info kdk link down
08:57:08 interface,info kdk link up
08:58:49 interface,info kdk link down
08:58:54 interface,info kdk link up
09:00:35 interface,info kdk link down
09:00:40 interface,info kdk link up
09:02:21 interface,info kdk link down
09:02:27 interface,info kdk link up
09:04:08 interface,info kdk link down
09:04:14 interface,info kdk link up
09:05:55 interface,info kdk link down
09:06:01 interface,info kdk link up
09:07:42 interface,info kdk link down
09:07:47 interface,info kdk link up
09:09:28 interface,info kdk link down
09:09:33 interface,info kdk link up
09:11:14 interface,info kdk link down
09:11:19 interface,info kdk link up
09:13:00 interface,info kdk link down

MikroTik IPoE.png

2

If you’ve left your priorities and path costs at their defaults, you may cause pointless root switch re-elections. Read the fine manual. You’d want that backup path to have a higher priority value (thus lower priority) than the main path to encourage it to elect kvp as the root.

It isn’t clear, but I think maybe you want the tunnel links set in point-to-point mode as well. If the nodes depicted as routers do indeed hide the MAC addresses of the devices behind them, then I believe that’s what you want. This shouldn’t affect your symptom, but it’ll improve convergence time.

If VLANs are involved, you may need to switch to MSTP.

Post the relevant parts of your configuration if tuning RSTP doesn’t fix it.

3

Also, I see no “IPoE” in the docs. Do you mean EoIP?

4

Forgive my carelessness, of course “EoIP”, not “IPoE”.

Increasing the cost for the “kdk” port does not change the situation. In general, this is not necessary, the RSTP root switch is always closer via the “kvp” port if all hardware is working.

Here is the configuration for routers (working in switch mode):

/interface bridge print
name="bridge" mtu=auto actual-mtu=1458 l2mtu=1598 arp=enabled arp-timeout=auto mac-address=48:8F:5A:61:D2:43 
     protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=no admin-mac=48:8F:5A:61:D2:43 ageing-time=5m 
     priority=0x8000 max-message-age=20s forward-delay=15s transmit-hold-count=6 vlan-filtering=no dhcp-snooping=no

/interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE                       BRIDGE                       HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0   H ;;; defconf
       ether2                          bridge                       yes    1     0x80         10                 10       none
 1 I H ;;; defconf
       ether3                          bridge                       yes    1     0x80         10                 10       none
 2 I H ;;; defconf
       ether4                          bridge                       yes    1     0x80         10                 10       none
 3 I H ;;; defconf
       ether5                          bridge                       yes    1     0x80         10                 10       none
 4 I   ;;; defconf
       wlan1                           bridge                              1     0x80         10                 10       none
 5 I   ;;; defconf
       wlan2                           bridge                              1     0x80         10                 10       none
 6     kvp                             bridge                              1     0x80         10                 10       none
 7     kdk                             bridge                              1     0x80         10                 10       none

/interface bridge port monitor 
numbers: 6
               interface: kvp
                  status: in-bridge
             port-number: 8
                    role: root-port
               edge-port: no
     edge-port-discovery: yes
     point-to-point-port: yes
            external-fdb: no
            sending-rstp: yes
                learning: yes
              forwarding: yes
          root-path-cost: 20020
       designated-bridge: 0x8000.48:8F:5A:61:D3:15
         designated-cost: 20010
  designated-port-number: 13

/interface bridge port monitor 
numbers: 7
            interface: kdk
               status: in-bridge
          port-number: 3
                 role: designated-port
            edge-port: no
  edge-port-discovery: yes
  point-to-point-port: yes
         external-fdb: no
         sending-rstp: yes
             learning: yes
           forwarding: yes
5

It did make me wonder what was different about “IP over Ethernet” in your configuration: that’s what we get without fancy tunnels!


Increasing the cost for the “kdk” port does not change the situation

I wrote “priorities” before “costs” on purpose: kvp should be declaring itself lower than “priority=0x8000” as on “kdk” to make itself the default path after [MR]STP settles into a stable state. Is it?

The only time packets should transit kdk is when kvp is down, if I understand the sense of your dotted lines.


routers (working in switch mode):

Then they’re not routers.

I make the distinction because it matters here: kvp and kdk don’t hide the MACs of the devices behind sw01 and sw02, so you should not enable point-to-point mode. A loop could be beyond those switches, after all.

What do you get for “/interface bridge monitor bridge” on kdk and kvp?

6

It looks like I didn’t state my problem clearly enough.
Once upon a time, all switches (sw01, sw02, sw03…) were connected by a cable into a ring. RSTP has been enabled. And everything worked great. Then they added equipment to protect communication channels. I cannot clean or adjust this equipment.

  1. MikroTiks are located in the address space 10.107.0.0/24.
  2. MikroTiks can only communicate with each other. Because of the nature of the hardware protection of this address space. In other words, do not configure routing.
  3. It is necessary to combine all network segments (sw01, sw02, sw03…) into a common L2 level network.

I solved this problem using EoIP tunnels. I imagined them as virtual cables. But for some reason, looped ports do not go into backup mode. Real ports with the same settings work properly.

 /interface eoip print
Flags: X - disabled, R - running 
 0  R name="kdk" mtu=auto actual-mtu=1458 l2mtu=65535 mac-address=02:54:EA:A8:C9:34 arp=enabled arp-timeout=auto 
      loop-protect=default loop-protect-status=off loop-protect-send-interval=5s loop-protect-disable-time=5m 
      local-address=10.107.0.54 remote-address=10.107.254.2 tunnel-id=206 keepalive=10s,10 dscp=inherit clamp-tcp-mss=yes 
      dont-fragment=no allow-fast-path=yes 

 1  R name="kvp" mtu=auto actual-mtu=1458 l2mtu=65535 mac-address=02:19:2F:DC:5D:C2 arp=enabled arp-timeout=auto 
      loop-protect=default loop-protect-status=off loop-protect-send-interval=5s loop-protect-disable-time=5m 
      local-address=10.107.0.54 remote-address=10.107.0.50 tunnel-id=106 keepalive=10s,10 dscp=inherit clamp-tcp-mss=yes 
      dont-fragment=no allow-fast-path=yes
7

I did some experiments and got the following results:
With default settings on “kfood” I get link up / down.
MikroTik RSTP cost 20.png
If you increase the priority on the bridge interface. (link does not blink anywhere)
MikroTik RSTP 9000.png
If you increase the cost of the “kvp” interface on MikroTik “kfood”, everything is fine too.
MikroTik RSTP 8000.png
I can’t figure out why I’m getting link problems with default settings.

8

I believe it’s because without the hints, both paths are equal, so RSTP flips a coin and goes with a random winner. Something happens to make it have another RSTP fight for dominance, it chooses differently, and you see the interface bounce briefly while it figures out how to break the loop you intentionally created.

With the hints, it keeps getting the same answer for which links to break and which to keep up each time, thus no bounces.

It might be educational to find out what keeps triggering those re-elections, but you could just as well declare success and go do something more productive with your day.

Do let us know if you learn anything.

9

Shouldn’t the STP protocol solve this problem? Finally, it must use the port number to calculate the path.

If this is a MikroTik bug, I would not want to get a similar result with an unplanned network rebuild.

But thanks anyway for your help!