RSTP - What the hell?

RouterOS General
1

Hello,

Long story short, I’ve got two DCs and since I can’t link them via Layer 2 networks, I am having two bridges. On each of the switches (4 in total), I’ve got two bridges:

  • one for the local DC traffic
    one for the inter-DC traffic - running VRRP with an IP on the VRRP interface

The local traffic bridge is running MSTP and it seems to be working fine. However, on the intra-dc bridge I’m running RSTP. The switches in each DC have a link that’s in this inter-dc bridge and each of the switches have a link to it’s counterpart, in the other dc. Something like:

Cisco ---- Cisco
 |          |
Sw3  ----  Sw4
 |          |
Sw1  ----  Sw2
 |
Cisco

Bridge priority on Sw03 is set to 4000 (hex). While Sw1, 2 and 3 all decided that Sw3 is the root bridge, Sw4 is struggling with this. It thinks that he’s the root. I thought that there’s an issue with Sw3 to Sw4 link, but that seems fine.
Inspecting the logs on Sw1 I see

P14-InterDC-to-Sw4: 0 learning
P15-InterDC-to-Sw4: 0 discarding
P14-InterDC-to-Sw4: 0 learning
P15-InterDC-to-Sw4: 0 discarding
P14-InterDC-to-Sw4: 0 learning
P15-InterDC-to-Sw4: 0 discarding

The funny thing is that this continues even after I disable the Sw4-Sw2 link.

In some really weird way from Sw3 I can ping Sw1 and 2, but I can’t ping Sw4, even if all the links are online.
From Sw1 and Sw2, I can ping all the switches…(WTF?)

If you guys have an idea, I would appreciate it. I’m sure it’s something small that I’m missing, as this used to work today, but I’ve made a thousand changes since then and I’ve got no clue why it’s behaving like this now.

Also, I would love to get rid of the inter-dc bridge, but the problem is that Sw3, Sw4 and Sw1 are connected to some Cisco switches which run PVST+ (Rapid PVST) and if I have a single bridge, it’s going to downgrade MSTP to RSTP (which means no VLAN) so the Cisco connected to Sw3/Sw4 will be in the same broadcast domain as the Cisco connected to Sw1, which is not good :frowning: If anyone has a clue on how I could resolve this without the need of the additional bridge, I’d appreciate it.

Thank you!

2

Isn’t it MSTP clashing with PVST+ protocol? Try to set same MSTP type everywhere.

3

please confirm which MikroTik device are you using, and RouterOS Version

4

Isn’t it MSTP clashing with PVST+ protocol? Try to set same MSTP type everywhere.

This used to work and since it’s a different bridge, I should be able to use something less resource-intensive, since the inter-dc bridge has no vlan filtering enabled.

Sw1,2,3 and 4 are CRS317-1G-16S+, running 7.15 [stable]

Just now, I disabled the inter-dc connections (sw3-sw1 and sw4-sw2) and while I would expect Sw3 to become the Root bridge and Sw4 recognize it as the root bridge, they’re both roots. It’s like the can’t communicate.
Since I disabled the inter-dc links, I was also able to disable STP since I was 100% sure there was no loop in that bridge. Sure enough, with STP offline, I can ping from Sw3 to Sw4 and vice-versa but clearly, leaving STP off is not a solution, since I also need the inter-dc links to be up.
Since a picture’s worth a thousand words:

To the left, Sw03, to the right, Sw04. As you cna see, interface P14 is online, on both switches but still, sw04, instead of picking Sw03 as the root (lower priority), it picks itself…

5

I would definitely start from pure MSTP for everything. Some time ago I tried to mix PVST + MSTP on old and new Netgear gears and there were problems so it ended with “greatest common divisor” :slight_smile:

6

While I appreciate the thought, it doesn’t help. This is not the bridge that’s communicating with the PVST+. Besides this, the PVST+ is running on the provider side, so there’s really nothing I can do there. Still, as I said, that’s a completely different bridge and, weirdly enough, that works perfectly. It’s the inter-dc bridge that’s all mikrotik, RSTP that’s being funny.

7

Since I had no other ideas, I rebooted Sw03 - not really in production yet.
It came up, it’s now the root for the entire bridge, I tried to disable / enable / disable interface again, maybe I can trick it, so far it works… (?)

If anyone has a clue what happened, I’m happy to learn.

Also, for the other issue. Does anyone know how can I avoid “linking” the two DCs with a single bridge? Right now, the connection from the DC goes into a “traffic-bridge” since that’s the Internet line, and the inter-dc links (sw03-sw01 for example) go into a different bridge. If I put them in the same bridge, the STP instance from DC1 can communicate with the STP instance of DC2 :frowning:

8

There are various potential pitfalls https://help.mikrotik.com/docs/display/ROS/Layer2+misconfiguration but impossible to say if you have hit any of these with seeing the configurations.

Also if you set edge=yes on the Mikrotik bridge ports connecting to the Cisco(s) they will ignore any received and not send any BPDUs (equivalent to PortFast) which may allow you to change to a single bridge. If you are using hardware-offloaded switching in the Mikrotiks it only applies to one bridge (except for CRS1xx/2xx devices).

9

This means that it will ignore the BPDUs that the Cisco’s send, turning the Cisco into a “dumb link”?
Also, will it forward those BPDUs that the Cisco is sending?

I know that a single bridge benefits from HW offloading, that’s why I want to ditch this 2 bridge configuration, as the performance is really poor (~400Mbps on a 10Gbps link - I know, it’s the CPU that’s limiting me).

Later edit: setting the ports to the Cisco will stop sending BPDUs alright, but the problem is that since you’ve got a link between the Mikrotiks, and the Ciscos are also connected between them, you’re looping like crazy, as expected.

Any other ideas to remove the 2nd bridge? I was reading about port isolation but I have no experience with it, maybe someone here does…?

10

Yes

Also, will it forward those BPDUs that the Cisco is sending?

No

Later edit: setting the ports to the Cisco will stop sending BPDUs alright, but the problem is that since you’ve got a link between the Mikrotiks, and the Ciscos are also connected between them, you’re looping like crazy, as expected.

Any other ideas to remove the 2nd bridge? I was reading about port isolation but I have no experience with it, maybe someone here does…?

Your original diagram isn’t very clear as to which links are L2, which are L3, and how you want failover to work.

If you cannot change the Cisco settings to use RSTP or MSTP instead of PVST+ the BPDUs on the links to/from them must be blocked as they are incompatible.

Bridge horizon / port isolation may work depending on exactly how the setup is expected to operate. Using horizon disables hardware offload so using port isolation https://help.mikrotik.com/docs/display/ROS/Switch+Chip+Features#SwitchChipFeatures-Portisolation or switch ACLs https://help.mikrotik.com/docs/display/ROS/CRS3xx%2C+CRS5xx%2C+CCR2116%2C+CCR2216+switch+chip+features#CRS3xx,CRS5xx,CCR2116,CCR2216switchchipfeatures-SwitchRules(ACL) is a better option.

11

Hello,

From the Cisco to the Mikrotiks, the links are Layer 2. Between the mikrotiks, I can do whatever I want.
As I said, I already tested stopping the BPDUs the Cisco is sending by setting the ports going to the Cisco with

edge=yes

, but that form a big fat loop. It’s somewhat normal, as the Cisco think that the ports are not interconnected, so it sends a broadcast to Sw3 and it gets it back via Sw4.

Still, from what I see and what I’ve tested, if the STP protocols are incompatible, they’ll downgrade until they are => my MSTP will downgrade to RSTP. I can already see this working, as the Cisco are the root bridge and my traffic bridge sees that, on both switches.

This is how everything’s set up right now.
The inputs from the Cisco are set with PVIDs: DC1 PVID=3, DC2 PVID=4, so the traffic from one should never reach the other.
I have two bridges:

  • Inter-DC-Bridge using ether14 and ether15 - with VRRP, routing the traffic between the two DCs
    Traffic-Bridge using all the remaining ports

While this approach works, using the CRS317 means that I can only HW offload a single bridge => the inter-dc is being handled by the CPU and that makes it really slow. So the question is: how do I end up with a single bridge, and without forming a single broadcast domain inter-dc?

Can I run an MLAG on each DC side and then add an IP address to the MLAG interface / run VRRP over the MLAG?
I do know that I can just assign IP addresses to ether14 and ether15 on all switches, and then route the traffic but I feel there’s a better way…?

12

If you can/have a choice, I’d say l3 interfaces with a routing protocol between the DC and a vlan l3 interface and routing protocol intra DC.

13

Thought of this but for some reason, I’ve got a really bad performance.
I’ve added a VLAN interface in each DC that allows the switches to communicate with each other.
On that VLAN I’ve added IP addresses.
On the inter-dc interfaces, I’ve added IP addresses.
I’ve added routes (sw03-dc1 routes to sw01-dc2 with a distance of 50, and sw03-dc1 routes to sw04-dc1 with a distance of 100) on each of the switches, so that each switch has a lower distance route to the other DC and (in case that link goes down) a higher distance route that points to the other switch in the same DC.

Unless I’m missing something, this is how it should work.
The only issue is that I’m getting 6-700Mbps on a 10Gbps link. I have already enabled L3 HW Offloading on the switches, as I was getting ~300Mbps without it.

Any ideas what am I missing?

Planning to change the manual routing once I get it working properly, but it was easier with manual routes right now.

14

Hmmm, looking at the specs for your devices, that’s a bit on the low side. When you did your try, did you see if the packets were going through the fasttrack/fast-path?

15

Found the culprit… VRRP disables l3 HW offloading. Would have been nice to have a warning or something, but after disabling VRRP, I went to > 3Gbps…