RouterOS bridge mysteries explained

[pcunite]

May I join this discussion as to why? From the very beginning, and even now, the bridge, bridge port is confusing. It will always be confusing because it is an insufficient abstraction for the people who use it. This is why @sindy can not make it any clearer. No one more capable and patient than him could be hired to do so, and yet, it is as murky as would be dark waters in the south pacific rim from falling volcanic ash.

Abstraction
Abstractions are very important in the software industry. Generally speaking, programmers create them (implemented from specifications created by software architects which maybe the same person). Then technical writers try to explain them. Marketers then try to sell them. If the abstractions are too close to the hardware, another layer of abstraction is created to make it more sensible, easier to reason about. In programming code, these abstractions are another class (object) that manages the inputs and outputs of the lower (closer to hardware) object. There maybe several. There maybe ten before anyone outside the company sees the final layer.

These layers mean something to different groups based on their needs and use cases. Consider the following: an ethernet frame gets assembled from a nic driver on a PC. A switch, reading in the series of electrical or optical impulses stores the bits in a buffer, a software object identifies the object as a frame, another object responsible for modifying frames inserts a VLAN tag, and so on. This abstraction is well below the needs of people who write programs for your computer. This abstraction is unnecessary and a distraction for them. So, another layer they use allows them to read into a memory buffer, and output file or screen data in a loop. They don’t need such verbose access to the wire.

How all these physical things get abstracted by programmers is something that may never be exposed to the network admin. In fact, a final abstraction, often termed the API, is what most generally gets exposed to customers and end uers. And that abstraction is what is the most important thing to MikroTik product administrators, forum commenters, explainers, and apologists. You and me. We care about the RouterOS API (syntax and Winbox).

The Missing Abstraction:
While at first I was thinking it was RouterOS’s fault, and indeed it could be a missed opportunity, the real issue, the real blame, the real source of confusion is the missing abstraction that we don’t have. Point your fingers at what does not yet exist! Therein lies the blame. Do not blame what we have, blame what we lack.

RouterOS, I theorize, is closely tied to Linux. It is a layer upon Linux and this layer is itself tied to the RouterOS syntax and how that relates to Winbox. Did you catch that? That is a line of dependencies. RouterOS, I suspect, is not an abstraction over a multi-vendor routing, switching, VLAN, whatever concept. Instead, it is a layer over Linux. This layer is tied to the way their GUI (Winbox) works which tries to be like RouterOS text syntax, which in turn tries to simplify the underlying Linux commands.

MikroTik could have created a syntax and GUI in isolation. However, they may not have separation of concerns in-house. The code is too intertwined. A point could be made that MikroTik should have started with an API concept, then separately picked a syntax, and then a vendor (Linux), to make the switching concept physically become a reality. But that is unfair. They’ve made a great product and we love it. As with all things, there is always something to improve. Looking back one can see more clearly. Maybe they should consider a new end user API. Start with the API, and work backwards from there, in fact. That would clear up all this confusion. But that is a business case for them and not for us to decide.

Conclusion:
So, step away from the MikroTik documentation, my own comments, and all the writing about MikroTik’s implementation for a moment. What is a switch and a router doing, conceptually speaking, when operating under the VLAN standard? When you have that diagrammed, demonstrated, understood, and explained, then you can show others how RouterOS API chose to implement this.

But here’s the deal. You may never be able to explain, however, why RouterOS does it this way. The RouterOS abstraction maybe based on a model that is not reflective of where their customer’s operate. It may not be where you and I actually live. What if RouterOS, the one that we know, is actually for programmers and not network admins? You and I fellow forum co-patriots, we are the ones who are assembling networks. RouterOS may have not been for us. It may have been for the programmers that designed it. They understood it. It made sense to them.

We have become the missing abstraction. We make RouterOS work with network concepts. This means the bridge port, lacking MikroTik’s explanation, is whatever @sindy says it is.

[bpwl]

I hope you add information about creating VLANs by new way… and how live with General vlan id:1 and others as Trunk.
For now I use always this howto: > https://mbum.pl/archive/vlan-po-nowemu.pdf > (Presentation from Ihor Hreskiv at Mikrotik Beer User Meating at Poland 2019, mbum.pl)
Pages: 34 - 50

I can take picture from that PDF and create a Photo-story here. I hope you can do that way edition of your post - this will be greate have it in English.

Deepl translation: if it helps
VLANs in the new way.txt|attachment (11.2 KB)

[Sob]

This confusing blending of multiple logical objects into a single one is not an invention of the Mikrotik team - this is how bridge is implemented in Linux kernel and represented in its configuration layer.

Maybe the right choice was to not expose the bridge itself as port. I hope that I didn’t miss something, but you don’t really need it, do you? Instead of using bridge as untagged port for VLAN X, you can add VLAN interface for VLAN X and use that. But now that it’s there and people already use it, it would be problematic to get rid of it.

What I don’t like much about current UI for bridge VLAN filtering is that it’s spread over different places. If I want some port as untagged access port for VLAN X, its PVID is defined on port, but then the rest of VLAN X is defined elsewhere. And I can even add conflicting config, saying that the port is untagged in completely different VLAN Y. I didn’t test what it does and if there’s perhaps some use for that, but it can be quite confusing.

Or the fact that I can define VLAN membership for ports that are not even part of the bridge, that can be confusing too. It makes sense when there’s support for dynamic interface lists. But it doesn’t make the simple config with static ports easier. If it wasn’t for dynamic interface lists, I’d prefer interface like this:

It would automatically list all member ports of bridge, and it would be immediatelly clear what belongs where. And if it would be linked with ports’ PVID, so that changes could be done from one place, it would be much easier to understand. And more efficient too, rather than adding ports to VLAN one by one. But I don’t know how to include handling of interface lists in this.

Another way how to make it easier could be built-in hints. Similar to how IPSec warns you about weak secret. So if I have VLAN interface for VLAN X on top of bridge, but bridge is not listed as tagged for VLAN X, there could be a warning about that. Or other way around, if bridge is listed as tagged for VLAN X, but there’s no inteface for VLAN X, there could be warning too. I’m sure it would help many people.

[Amm0]

What I don’t like much about current UI for bridge VLAN filtering is that it’s spread over different places. If I want some port as untagged access port for VLAN X, its PVID is defined on port, but then the rest of VLAN X is defined elsewhere. And I can even add conflicting config, saying that the port is untagged in completely different VLAN Y. I didn’t test what it does and if there’s perhaps some use for that, but it can be quite confusing.

Yup that’s much closer to the UI I’d expect – great mockup, seem Mikrotik-ish.

IMO the best UI for VLAN management is a grid (like Netonix, older UBNT ToughSwitch & NOT like Netgear which is only marginally better than MT’s brigde VLAN) – but a “grid” is pretty far from “winbox” style:

Maybe the axises filped and some grid of checkmarks? All I know is the drop-down is horrible way to do this. I generally use the CLI for stuff, but use winbox a fair share too. But I don’t even bother with winbox for Bridge VLANs – it’s confusing since you need a few windows open if you want confirm the PVID in an interface etc. But the CLI isn’t for everyone, and it’s VLAN configuration there isn’t that much better BUT you can at least see the PVID of the ports you’re working with on the same screen…

Another thing that I do think it would help here is if Mikrotik enabled vlan-filtering=yes in the default config, or at least in one of the QuickSet that create a bridge – in nearly all VLAN cases, you’d want to use - but applying later is where the trouble starts… Plus, some default config with vlan-filtering enabled more easily allowed applying the pcunite’s “VLAN’ing you network” kinda scenarios be really helpful. I mean QuickSet does have a checkbox for “Enable VPN”, so “Enable VLAN Filtering” doesn’t seem that far a stretch.

Idea being while Sob’s mock-up seem pretty good & reasonable… but getting to where you can use the “Sob VLAN UI” is also not that easy…

[Sob]

Problem with grid (which I otherwise like too) is space. It’s ok for switch with fixed ports 1-X. But number of ports in RouterOS is theoretically unlimited (you can add e.g. loads of EoIP interfaces). Configurable and possibly long interface names don’t help either. It’s not that it couldn’t be used, but if you end up having to scroll both horizontally and vertically, it’s not going to be great experience. And switching axes won’t help with this.

It also has the same problem as my idea, what to do with ports added using interface lists? One way would be listing all current ports that bridge already has, and additionally allow to add others, same way as it’s currenly done for all tagged and untagged, that could work.

[Amm0]

Problem with grid (which I otherwise like too) is space. It’s ok for switch with fixed ports 1-X. But number of ports in RouterOS is theoretically unlimited (you can add e.g. loads of EoIP interfaces).

Oh no, you’re idea certainly more realistic, I like it. I posted a grid based on Cunningham’s Law. And, even the current UI can run out of vertical space, but yeah dynamic interface & interface lists do make a tricky problem.

But Sob does have a point here: you kinda have to get through enable vlan-filtering, then getting through two-step setup to tag a new VLAN first to even grok the mysteries discussed here. But, some of these mysteries are created by the UI itself…

Perhaps a new “frame-type” on a Bridge>Port like “use as access port”? This would means it doesn’t need to be handle by Bridge>VLANs. But if you wanted a “hybrid” port, you use the existing “allow untagged and priority only” options – those would show up in Sob-like UI. A new “frame-type” wouldn’t break existing config, and remove clutter in Bridge>VLAN. e.g. the current drop-downs in Bridge>VLANs would have “less stuff” since these new “access port” are explicitly declared on the bridge port itself, nothing is needed Bridge>VLAN tab. The bridges dynamic assignment half covers this today, but you still always see “access ports” when trying to configure hybird/trunk ports.

None of this make the hybrid/“two faced” bridge interface/port easier, but do the think UI makes the whole topic even more difficult to explain.

[Guscht]

@Guscht, the “CPU port” is an oversimplification, based on an assumption that CPU is equivalent to the router process and that the bridge process runs somewhere else than on the CPU. If this simplification helps you understand the concept, stick with it, but actually the “port of the virtual switch” and “interface of the virtual router” form up the demarcation line between the bridge process and the router process, which both run at the CPU.

Hi sindy, late answer… Of course, both things run on the CPU. I borrowed the wording “CPU-Port” from the Switch-Menu, because this is the exact behaviour what the “virtual switch port facing the virtual router interface” does. It will become only a bit more simpler if you can think “it will go to the CPU where the other processes run”.

One BIG probelm is, that MT allows you so many invalid configurations. Where is documented, that the Bridge-[itself]-Interface works only with untagged frames? If this is somewhere written (I never found it), why is it not in big, red letters? Its a very(!!) important thing and will result in a lot frustration if you are not a “switching philosopher” with passion I read issues related to this every week on MT forums, groups.

Why do they not implement a warning/dialogue box if anyone tries to do that? I work with a lot other network-vendors and they all warn you about such things. MT says: FCK you, no help, our devices are cheap - you have to trial-and-error and learn it the hard way…

This toxic mix of poorly written documentations, poorly GUI-design and the way they allow invalid configs, makes it so hard. To be honest, no one of my colleagues is able to configure a ROS device in regards of VLAN. Not in the real world, not in a production enviroment.

And we speak only about the (simple) BRIDGE. MT has this incredible complexity in regards of their SWITCH-CHIPS. Like: this CRS allow this with this menu, this CRS allows this but with a totally different GUI because it contains another switch-chip, this works only from ROSvX.XX. Here we implement a GUI without functions. Here you need to create BRIDGE and add the ports for the SWITCH to function… And they let you mix the settings from the SWITCH and BRIDGE menu and dont tell you what takes precedence.

[tdw]

Where is documented, that the Bridge-[itself]-Interface works only with untagged frames? If this is somewhere written (I never found it), why is it not in big, red letters?

It isn’t because it does not. Just as with any physical interface the other CPU processes are expecting untagged IP packets, for tagged traffic you need an /interface vlan to remove the VLAN tag in the ‘parent interface to CPU process’ direction and add it in the other.

Take a simple example - a single physical ethernet port, ether5, providing the 192.168.1.0/24 subnet untagged plus 192.168.2.0/24 subnet tagged with VLAN ID 2:
/interface vlan
add interface=ether5 name=vlan2-ether5 vlan-id=2
/ip address
add address=192.168.1.1/24 interface=ether5
add address=192.168.2.1/24 interface=vlan2-ether5

If at some point you may wish have more than one port providing access the equivalent bridge with a single port as a member is:
/interface bridge
add name=mybridge pvid=1 vlan-filtering=yes
/interface bridge port
add bridge=mybridge ingress-filtering=yes interface=ether5 pvid=1
/interface bridge vlan
add bridge=mybridge untagged=mybridge,ether5 pvid=1
add bridge=mybridge tagged=mybridge,ether5 vlan-ids=2
/interface vlan
add interface=mybridge name=vlan2-mybridge vlan-id=2
/ip address
add address=192.168.1.1/24 interface=mybridge
add address=192.168.2.1/24 interface=vlan2-mybridge

The parent, subinterface and port members are highlighted by colour; default / dynamically created items are in italics and can be omitted.

Yes, the documentation is not concise, and it would be really helpful if the UI showed warnings for any /interface bridge vlan entries where the tagged= and untagged= lists do not have matching /interface bridge port entries.

[Zacharias]

and it would be really helpful if the UI showed warnings

That would be helpful for any configuration …

[mkx]

Where is documented, that the Bridge-[itself]-Interface works only with untagged frames?

It isn’t because it does not. Just as with any physical interface the other CPU processes are expecting untagged IP packets,…

Actually it’s not L2 ports (physical ports, bridge port) that would “hate” tagged frames. Rather it’s L3 interfaces (IP stack) that doesn’t handle VLAN headers.

Note the difference between port and interface … ROS distinguishes between the two as well.
The mental twist being that e.g. ether1 can be either port (member of bridge, either tagged, untagged or hybrid) or interface (carrying IP address or being anchor for VLAN interface) and one has to be careful to note which it is.
A basic error is to use same entity in both roles at the same time (e.g. making ether1 both member of bridge and attaching IP address directly to it). ROS doesn’t flag it as error and things kind of work, but not 100%.

[PackElend]

could you explain it like this as well?

any interface logical or physical has to have an IP to allow >=L3 services on it.
As the bridge interface ( “new” way of handling multiple VLANs ) must not have an IP, so you don’t use it anywhere, except handling L2 things for devices/ports connected to it

[anav]

So PackElend how do you distinguish the common use of bridge where there are no vlans and ports are attached to it and the IP address is assigned to the bridge.
Is it not an interface in this scenario?

[anav]

Yes, the documentation is not concise, and it would be really helpful if the UI showed warnings for any /interface bridge vlan entries where the tagged= and untagged= lists do not have matching /interface bridge port entries.

Hence, I always recommend fully documenting the untagged ports on /interface bridge vlans so as to enable clear cross-checking with /interface bridge ports.

[sindy]

any interface logical or physical has to have an IP to allow >=L3 services on it.
As the bridge interface ( “new” way of handling multiple VLANs ) must not have an IP, so you don’t use it anywhere, except handling L2 things for devices/ports connected to it

Sorry, I did not understand what you are asking.

In any case, “new” handling of multiple VLANs is just an add-on to the basic concept of the bridge, it doesn’t change it. Second, did you mean “a physical/virtual interface that has been made a member port of the bridge must not have an IP”? Because that’s the correct description of the requirement.

[PackElend]

sorry not all got pasted

when do you use the bridge interface at all?
Only for “new” way of handling multiple VLANs or in any other configuration as well?
I guess only first and the reason could be explained by:
any interface virtual or physical has to have an IP to allow >=L3 services on it.
As the bridge interface ( “new” way of handling multiple VLANs ) must not have an IP, so you don’t use it anywhere, except handling L2 things for devices/ports connected to it

[sindy]

You use a bridge interface whenever you want to connect multiple devices to the same IP subnet. Whether you use VLANs or not is a separate thing. You can use a dedicated bridge for each (V)LAN, you can use multiple bridges with multiple VLANs on each, or a single bridge for all your VLANs - it’s up to your actual needs. But the bridge still forwards frames between its ports up to their MAC addresses only, and if the Mikrotik device on which this bridge exists wants to talk IP to the external devices connected to the bridge, you have to attach an IP address to the appropriate interface of the router - to the bridge-facing interface of the router directly or to the /interface vlan attached to it, depending on the rest of the configuration.

So again in different words - the interface of the router that is “connected” to the matching port of the virtual switch must have an IP address if you want that the router had access to devices in the same (V)LAN which is configured as an access one on that port of the virtual switch. If vlan-filtering is off, what came in tagged stays tagged on egress and what came in tagless stays tagged on egress; if vlan-filtering is on, you can tag frames on ingress and untag them on ingress. But that has nothing to do with attaching or not attaching IP addresses to interfaces.

[chechito]

topic´s name is beautiful

[PackElend]

… the bridge still forwards frames between its ports up to their MAC addresses only, and if the Mikrotik device on which this bridge exists wants to talk IP to the external devices connected to the bridge, you have to attach an IP address to the appropriate interface of the router…

…the interface of the router that is “connected” to the matching port of the virtual switch must have an IP address if you want that the router had access to devices in the same (V)LAN …

thanks for the detailed answer

1. So Bridge does L2, being a mature L2-Switch, that is clear to me. If you active use IP-Firewall, it includes some L3 things.
2. but if you use (multiple) VLAN(s) in the same Bridge, the VLAN Interfaces are the door to L3?
3. but bridge-facing interface is the Bridge CPU port, if you don’t work with VLANs, so you put an IP on etc. on "bridge-… ", so it become the door to L3?

 > interface/print
 Flags: D - DYNAMIC; R - RUNNING; S - SLAVE
 Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
 #     NAME                           TYPE    ACTUAL-MTU  L2MTU  MAX-L2MTU  MAC-ADDRESS      
 0  R  combo1                         ether         1500   1580      10222  08:....
39  R  bridge-...                   bridge        1500   1580             08:55:31:31:24:88
....

[mkx]

Conceptually: physical port (e.g. ether1) is a port. When you add L3 configuration (e.g. IP address) to it, it becomes interface. If you want to work with tagged frames, coming in and leaving out of, you have to create appropriate VLAN ports, anchored to physical port. Adding IP addresses to those VLAN ports makes them VLAN interfaces (VLAN ports can be used as bridge ports when creating per-VLAN bridge in a pre-6.42 fashion).

Likewise: bridge (named e.g. bridge1) comes with CPU-facing port, named bridge1. If you add IP address to it, it becomes interface named bridge1 (and IP stack works with this interface, not with bridge member ports. I.e. in-interface is then bridge1 or out-interface is bridge1 … not some member port). So the same as with individual physical ports, if you want to work with VLANs, you have to create some VLAN ports and anchor them to port bridge1.
When doing VLANs with bridge, it’s necessary to configure the virtual switch … so you configure VLAN membership (and PVID) for all member ports, including CPU-facing port named bridge1.
So as you can see, port bridge1 is treated the same as other explicitly configured bridge ports (ether ports, SFP ports, wifi ports) except for implicit creation. Naming around ROS is slightly inconsistent (most ports are found under /interface … ethernet or wireless or …), but never the less they are ports prior to receiving L3 configuration.

[sindy]

• but if you use (multiple) VLAN(s) in the same Bridge, the VLAN Interfaces are the door to L3?
• but bridge-facing interface is the Bridge CPU port, if you don’t work with VLANs, so you put an IP on etc. on "bridge-… ", so it become the door to L3?

The items under /interface vlan are “featured pipes” whose one end (“tagged”) is attached to a carrier (underlying) interface/port, and the other end (“tagless”) behaves like an interface/port itself.

So when you attach an IP address to the tagless end, it becomes an IP interface.

When an IP stack sends a packet via an L2-capable interface, that packet gets encapsulated into a tagless ethernet frame (leaving wireless aside to simplify things). If the interface is the tagless end of the /interface vlan pipe, the frame gets an 802.1Q VLAN header as it passes through the pipe, so at the tagged end of the pipe, it is now a VLAN frame.

In the opposite direction, the pipes are “picky” - they only forward VLAN frames whose VID matches the one the pipe is configured to use, and untag them as they deliver them to their tagless end. They ignore frames with other VIDs.

You can attach the /interface vlan to an L2-capable interface directly, and you can also attach an IP address to it, but only if that L2-capable interface is not a member port of a bridge or a member link of a bond (exceptions exist but making use of them has side effects and causes incompatibility with less punky devices than Mikrotik is). At this step, we still think about the interface to which the IP address and/or /interface vlan pipes are attached as about the switch-facing interface of the router.

The next step is to see what happens to the frame as it enters the bridge via the router-facing port of the bridge. And here the difference between vlan-filtering=no and vlan-fltering=yes comes into play. With no, the bridge behaves like a “non-managed” (dumb) switch - it only cares about MAC addresses and nothing else. So if a frame came in tagless through one port, it will stay tagless on the bridge and as it egresses via another port(s). With yes, the bridge behaves like a “managed switch”, so each of its ports can be an “access”, “trunk”, or “hybrid” one, depending on the settings. An access port only accepts tagless frames on ingress, tags them with the configured VID, and ignores tagged ones; a trunk port only accepts tagged frames (and may only accept those that bear the permitted VIDs), and a hybrid port is a combination of both behaviours. “Inside” the bridge, all frames are tagged, but if a tagless frame comes in through to a port with a certain pvid and egresses via another port with the same pvid, it will be tagless also at egress, as it gets tagged on ingress, passes through the bridge tagged, and then gets untagged on egress.

The router-facing port of the bridge behaves exactly the same like any other port of that bridge. So if you attach an IP address “to the bridge itself” (in fact, to the bridge-facing interface of the router), it will end up being reachable via the VLAN which is configured as pvid “on the bridge itself” (in fact, the router-facing port of the bridge). If the “bridge” port is in pure trunk mode, such an address will not be reachable at all. If you attach an IP address to an /interface vlan that is attached to the bridge-facing interface of the router, that address will be reachable via the corresponding VLAN if that VLAN is permitted also on the “router-facing port of the bridge” (which means that the router-facing port of the bridge is listed in the tagged list for that VLAN under /interface bridge vlan).