Hi all,
I have a CCR2004 (which has two switch chips, 9 ports on chip 1 and 9 ports on chip 2, both Marvell 88E6191X). Can I create a single bridge containing ports of both switches or should they necessariliy be linked to different bridges? The aim would be to have ports with the same VLAN ID, member of the bridge but linked to different switches, would that work as if there was only on switch chip (if not, why not)?
Regards,
Kris.

Not a chip expert but wouldnt creating the same vlan to cross the two chips be self defeating as this then involves the CPU.
In concept, the idea is to maximize wire traffic between ports on the same chip and thus as you surmize, bridge the ports on one chip and the other ports on the other chip for another bridge.
In some case you may not be able to avoid inter vlan traffic crossing to the other chip.....???

Yes, you can have single bridge spanning both port groups. With potential performance hit mentioned by @anav.

There was a bug in how ROS configures VLAN offload to switch chips ... on devices with two switch chips it was necessary to add bridge port as tagged member of all VLANs which span both switch chips even if router doesn't communicate with it (useful when device is used as a switch), otherwise frames would not pass between ports on different switch chips. This bug was acknowledged by MT, but I'm not sure if it was fixed already.

Hi all,
I have a CCR2004 (which has two switch chips, 9 ports on chip 1 and 9 ports on chip 2, both Marvell 88E6191X).

Be careful, SFP+ ports are connected to CPU, not switch chips.

Yes, you can have single bridge spanning both port groups. With potential performance hit mentioned by @anav.

There was a bug in how ROS configures VLAN offload to switch chips ... on devices with two switch chips it was necessary to add bridge port as tagged member of all VLANs which span both switch chips even if router doesn't communicate with it (useful when device is used as a switch), otherwise frames would not pass between ports on different switch chips. This bug was acknowledged by MT, but I'm not sure if it was fixed already.

mkx was that across platforms or only applicable to the RB4011 ??

Two ASICs, means two bridges.

bridge1 for ports ether1-8, bridge2 for ether 9-16, this ensures both port groups are fully hardware offloaded to the correct ASIC.

For SFP1 and SFP2, both being independent paths towards the CPU, you could put them in bride3, but I wouldn't advise this, as you will likely want to make SFP1-2 an LACP bonding to your uplink, so the LACP bonding and the underlying physical ports are not members of any bridges at all. So 20G uplink via LACP bonding, independent of any bridges.

https://i.mt.lv/cdn/product_files/CCR20 ... 240151.png

Nice!!

mkx was that across platforms or only applicable to the RB4011 ??

As far as I understood MT staffer who chimed in (could be it was Normis, could be it was somebody else) was that the bug was in the way ROS configured the switch-CPU interconnect port of the switch. I.e. it was configured to pass only VLANs of which bridge (the CPU-facing) port was member. Which is fine for devices with single switch-chip and the switch-CPU interconnect is really used only for interaction between ROS and network. But this is not OK on devices with multiple switch chips where switch-CPU interconnects are used also for (indirectly) interconnecting different switch chips where switch-CPU ports must be configured to pass also VLANs present on other switch chips even if ROS doesn't interact with them.

So this bug is not RB4011 specific, but it seems it was first discovered there (perhaps because RB4011 is often used as router/switch combo for SOHO while CRS2004 is mostly used as a decent router).

Two ASICs, means two bridges.

bridge1 for ports ether1-8, bridge2 for ether 9-16, this ensures both port groups are fully hardware offloaded to the correct ASIC.

For SFP1 and SFP2, both being independent paths towards the CPU, you could put them in bride3

That's one way of doing it ... if two bridges come handy. But it doesn't have to be two bridges, one bridge spanning all ether ports will do just fine.

And there's nothing wrong with adding SFP+ port to a bridge. Surely it won't be HW offloaded, but the rest of bridge ports will be, just like when one adds wifi interface to a group of ether ports ... ether ports keep being HW offloaded, wireless isn't. One only has to keep in mind that SFP+ traffic will bog CPU, that's all.

But it doesn't have to be two bridges, one bridge spanning all ether ports will do just fine.

You could do that, by running a cable from ether8 to ether9, but why? This is a bandwidth poor approach.

... but why?

Using a CCR2004 in a switch manner is a sin to begin with. But it's up to device admin how he wants to use his device and I simply answered questions by @KrisVG. You, OTOH, are risking accusations about pushing your own ideas upon person asking for help (accusations seem to be fashionable these days).

Dont worry, Darknate has a thick skin, not concerned with wall flowers.........
But how insenstive of you mkx to imply the oP may have purchased the wrong product jajaja

... but why?

Using a CCR2004 in a switch manner is a sin to begin with. But it's up to device admin how he wants to use his device and I simply answered questions by @KrisVG. You, OTOH, are risking accusations about pushing your own ideas upon person asking for help (accusations seem to be fashionable these days).

Hi mkx, that's an interesting remark. I may indeed have purchased the wrong device (in my defence: I'm a system administrator, not a network administrator). I could use the CCR2004's ports only for routing/separating between interfaces but I thought using the switch chips in it would accomplish the same thing only on layer two. Furthermore, directly attached to the CCR2004 are a number of switches (different brands) that have a number of VLANs on them. How would I connect several VLANS (on different switches) if not by using a bridge on the CCR2004? I work for a school (thus (very) low budget) so I use the CCR2004 as a router with integrated core switch.

But it doesn't have to be two bridges, one bridge spanning all ether ports will do just fine.

You could do that, by running a cable from ether8 to ether9, but why? This is a bandwidth poor approach.

Hi DarkNate,
So if I want a single bridge I need to connect two ports (one of each switch chip) and probably configure them as trunk for all VLANs. Is that correct? If so, that would be a lot of unneccesary traffic going over that connection, is that what you mean by "bandwidth poor approach"?
FYI: I'm going for the two bridge approach and if a connected switch has VLANs of both bridges I'll just connect two ports of that switch with one port of each bridge.

Hi mkx, that's an interesting remark.

It was a more or less rethorical remark, directed at @DarkNate . Since you already have the device, you should use it as much as possible ("abuse" even). If using it as router/switch combo fits your needs, then just use it so. And if you're not after last bps of performance, then you don't have to bother with all the tricks @DarkNate mentioned (specially so as they come with their own price) which then allows you to apply a much more straight-forward config.

So if I want a single bridge I need to connect two ports (one of each switch chip) and probably configure them as trunk for all VLANs.

Wrong. Either you have single bridge and (implicitly) use internal interconnects to "glue" the two switch chips together. Or you have two bridges (with independent configuration) and use external interconnect to pass traffic between the two switches. Benefit of "two bridges" approach is that CPU will not be bothered by L2 traffic and will be free for routing duties (does your routing require the CPU power available?), the price is loss of two ethernet ports.

Hi DarkNate,
So if I want a single bridge I need to connect two ports (one of each switch chip) and probably configure them as trunk for all VLANs. Is that correct? If so, that would be a lot of unneccesary traffic going over that connection, is that what you mean by "bandwidth poor approach"?
FYI: I'm going for the two bridge approach and if a connected switch has VLANs of both bridges I'll just connect two ports of that switch with one port of each bridge.

Don't listen to @mkx, he's trying to sell you a piss poor implementation that itself doesn't match official MikroTik docs. He calls my approach as “tricks” even though official MikroTik agrees.

See this link:
https://help.mikrotik.com/docs/display/ ... switchchip

And read this fully:

For devices that have multiple switch chips (for example, RB2011, RB3011, RB1100), each switch chip is only able to switch VLAN traffic between ports that are on the same switch chip, VLAN filtering will not work on a hardware level between ports that are on different switch chips, this means you should not add all ports to a single bridge if you are intending to use VLAN filtering using the switch chip, VLANs between switch chips will not get filtered. You can connect a single cable between both switch chips to work around this hardware limitation, another option is to use Bridge VLAN Filtering, but it disables hardware offloading (and lowers the total throughput).

Using the cable works, offloading will work, but you're limited to just 1Gig for inter-ASIC forwarding.

@DarkNate, I could have misread and I don't even have 0,1% of your knowledge, the note you posted seems to be for "Other devices with a built-in switch chip" (VLANs configured on the switch).
I was one that reported bug on the RB4011 in v7.8 for devices with 2 switches and hardware offload, until v7.7 and from v7.10 devices (RB4011 + CCR2004) with multiple switches and a single bridge are working as expected.

@DarkNate, I could have misread and I don't even have 0,1% of your knowledge, the note you posted seems to be for "Other devices with a built-in switch chip" (VLANs configured on the switch).
I was one that reported bug on the RB4011 in v7.8 for devices with 2 switches and hardware offload, until v7.7 and from v7.10 devices (RB4011 + CCR2004) with multiple switches and a single bridge are working as expected.

CCR2004 fits under the "Other devices with a built-in switch chip" section, please check with official MikroTik support, you don't have to trust me blindly, verify.

Single bridge for all ports of both switch chips as members? With HW? Share your config sample, /int bridge export.

RB4011 is also under the "Other devices with a built-in switch chip" section:

This type of configuration should be used on RouterBOARD series devices, this includes RB4xx, RB9xx, RB2011, RB3011, hAP, hEX, cAP and other devices.

Yes sir. DarkNate,
I trust you more than me

This is an RB4011, currently I can't show you that ports were hardware offloaded due to testing dhcp/igmp snooping enabled; Support involved with SUP-141900 to give us an official statement.

Don't listen to @mkx, he's trying to sell you a piss poor implementation that itself doesn't match official MikroTik docs. He calls my approach as “tricks” even though official MikroTik agrees.

See this link:
https://help.mikrotik.com/docs/display/ ... switchchip

@DarkNate: since you're bringing my name out, I'll risk of getting a warning (or a ban; but I don't really care) ... but: you're often a smart ass. This time as well.

If you cared to check the specs of device we're talking about (CCR2004-16G-2S+), you'd see it features switch chip type Marvell 88E6191X. And devices with these switch chips fall into category CRS3xx, CRS5xx series switches, CCR2116, CCR2216 and RTL8367, 88E6393X, 88E6191X, 88E6190, MT7621 and MT7531 switch chips. And so is RB4011 which features two RTL8367 switch chips.

The text about bridges spanning multiple switch chips you quoted applies to devices, where bridge can not be offloaded to switch chip, instead switch chip can be configured directly.

So perhaps you should follow own advice and read the manual carefully.

mkx - believe you are wrong this time around. Every one of the devices cited in that section has 1 switch chip only. So, when there is 1 switch chip only, a 88E5191X would be configured accordingly. The disclaimer further down is specifically about devices with two switch chips. Therefore, one would efficiently configure a CCR2004 with 2 bridges, 8 ports per bridge. A CCR2216 or CCR2116 would be configured with 1 bridge only, because it only has one switch chip between the ports and the CPU. Can't be any clearer to me if you look at the block diagrams. YES, you could configure a CCR2004 with all ports on 1 switch, but you would lose the HW offloading on 8 of those ports.

mkx - believe you are wrong this time around. Every one of the devices cited in that section has 1 switch chip only. So, when there is 1 switch chip only, a 88E5191X would be configured accordingly. The disclaimer further down is specifically about devices with two switch chips. Therefore, one would efficiently configure a CCR2004 with 2 bridges, 8 ports per bridge. A CCR2216 or CCR2116 would be configured with 1 bridge only, because it only has one switch chip between the ports and the CPU. Can't be any clearer to me if you look at the block diagrams. YES, you could configure a CCR2004 with all ports on 1 switch, but you would lose the HW offloading on 8 of those ports.

Exactly. That's what the manual states. But some people are too dumb for their own good.

Yes sir. DarkNate,
I trust you more than me

This is an RB4011, currently I can't show you that ports were hardware offloaded due to testing dhcp/igmp snooping enabled;

Code: Select all

/interface bridge
add dhcp-snooping=yes frame-types=admit-only-vlan-tagged igmp-snooping=yes igmp-version=3 mld-version=2 name=Bridge protocol-mode=mstp vlan-filtering=yes
/interface bridge port
add bridge=Bridge edge=no-discover frame-types=admit-only-vlan-tagged interface=sfp-sfpplus1 point-to-point=yes
add bridge=Bridge edge=no-discover frame-types=admit-only-vlan-tagged interface=ether3 point-to-point=yes
add bridge=Bridge edge=no-discover frame-types=admit-only-vlan-tagged interface=ether10 point-to-point=yes
add bridge=Bridge edge=yes-discover fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether4 point-to-point=yes pvid=10
add bridge=Bridge edge=yes-discover fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether5 point-to-point=yes pvid=10
add bridge=Bridge edge=yes-discover fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether6 point-to-point=yes pvid=10
add bridge=Bridge edge=yes-discover fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether7 point-to-point=yes pvid=10
add bridge=Bridge edge=yes-discover fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether8 point-to-point=yes pvid=20
add bridge=Bridge edge=yes-discover fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether9 point-to-point=yes pvid=30

Support involved with SUP-141900 to give us an official statement.

I think only intra-asic ports are offloaded. There's no way inter-asic is offloaded without a cable because inter-asic physical path doesn't exist, and therefore would be punted to the CPU.

Let us know what official support says at least for RB4011.

mkx - believe you are wrong this time around. Every one of the devices cited in that section has 1 switch chip only. So, when there is 1 switch chip only, a 88E5191X would be configured accordingly. The disclaimer further down is specifically about devices with two switch chips. Therefore, one would efficiently configure a CCR2004 with 2 bridges, 8 ports per bridge. A CCR2216 or CCR2116 would be configured with 1 bridge only, because it only has one switch chip between the ports and the CPU. Can't be any clearer to me if you look at the block diagrams. YES, you could configure a CCR2004 with all ports on 1 switch, but you would lose the HW offloading on 8 of those ports.

No, there is nowhere in that 1st section that says one switch chip only. It also mostly lists the switch chips, not the devices. And as you can see in the the RouterBoard device tables on this page

https://help.mikrotik.com/docs/display/ ... troduction

There is no devices with only a single 88E6191X switch chip. Only the CCR2004 is listed there with two 88E6191X. So your assumption that the section only applies when there is a device with only one 88E6191X is completely wrong.

The disclaimer about devices with multiple swich chips in the other section only mentioned the RB2011, RB3011, RB1100. Not the CCR2004, because the CCR2004 falls under the previous section.

mkx - believe you are wrong this time around. Every one of the devices cited in that section has 1 switch chip only.

The document cites some devices and some switch chip types. Device which @IP uses, has one of cited switch chip types. And section doesn't consider multi-switch-chip devices at all. We can only guess why that is but reason being that it doesn't really matter is a probable one as well.
The section you're referring to is about "Other switch chip types". Now which one applies more to @OP's case?

I'm not arguing that two bridges can not be offloaded on devices with two switch chips. I'm arguing that single bridge can span two switch chips and while doing that, ports on both switch chips will be offloaded. Yes, I'm aware that in such setup traffic between ports on different switch chips will hit CPU and I already wrote that in my initial response. I'm also arguing that CPU in device that @OP uses is more than capable of bridging traffic at wire speed which is 10Gbps in this case (interconnect speed). And I already mentioned that this takes away some CPU power otherwise available for routing.

This thread became a prime example of how discussion deviates from what OP was asking to discussion because of discussion. Because some readers/posters don't care to read OP's questions and only react to posts late in the thread.

For someone with a CCR2004-16G-2S+ and a single bridge between both switch chips and one of the SFP+ ports, the hardware offloading does work as long as the same vlan to vlan traffic is on the same switch chip.

So the winner is mkx!

For someone with a CCR2004-16G-2S+ and a single bridge between both switch chips and one of the SFP+ ports, the hardware offloading does work as long as the same vlan to vlan traffic is on the same switch chip.

So the winner is mkx!

inter-asic traffic is punted via CPU, stop purporting fake information like mkx. The proper solution is to run a cable between both ASICs.
viewtopic.php?t=203659#p1051860

For someone with a CCR2004-16G-2S+ and a single bridge between both switch chips and one of the SFP+ ports, the hardware offloading does work as long as the same vlan to vlan traffic is on the same switch chip.

So the winner is mkx!

inter-asic traffic is punted via CPU, stop purporting fake information like mkx. The proper solution is to run a cable between both ASICs.
viewtopic.php?t=203659#p1051860

I guess you did not understand my statement - traffic on the same switch chip (via the same vlan) will be hardware offloaded. Anything outside of that one switch chip will go to the CPU. My statement was to say that the switch chip functionality still works even when using one bridge as long as the traffic does not need to leave the switch chip.

Maybe this is part of the miscommunication of the entire thread. I don't see any case where traffic could be offloaded once it leaves the switch chip.

Having one bridge is sufficient to HW offload each switch chip as long as the vlan and hosts remain on the same switch chip.

If you do need the traffic to cross the switch chips, then yes a trunk port on each port group would be needed OR it would need to go through the CPU.

I think this is what we've all been talking about but in different aspects, some of focused more on best practices versus if something can be done. I never meant to infer that you absolutely cannot use the router in different ways. It's like people who use CRS switches for routers in their home. Yes, it might work but that's not a best practice IMHO.

My thoughts were focused more on the most efficient way to use the router in order to eek out every bit of performance possible. Hitting the CPU isn't a problem if you have the CPU to spare and you're fine with that. My tests just now confirm the hardware offload with one bridge but also show a 10% CPU load doing a local speed test between devices connected to ports that are in the different switch chips. It's 0% if they are on the same chip. So it won't take many threads across the CPU before you would start seeing performance/stability issues. Running a cable between ports in the switch seems stupid since you'd be taking 7 possible ports at 1 gig each (7 gigs traffic) and limiting them to just a 1 gig trunk. Crazy talk.

inter-asic traffic is punted via CPU, stop purporting fake information like mkx.

Cable across two device's ports will allow for 1Gbps interconnect between switch chips and without bothering CPU at cost of dedicating 2 ports.
Configuring single bridge and thus using switch chip - CPU interconnect will allow for 10Gbps interconnect and will cost some CPU cycles.

Both possibilities are real and I don't know why you're insisting on giving out fake information.

I think this is what we've all been talking about but in different aspects, some of focused more on best practices versus if something can be done. I never meant to infer that you absolutely cannot use the router in different ways. It's like people who use CRS switches for routers in their home. Yes, it might work but that's not a best practice IMHO.

My thoughts we're focused more on the most efficient way to use the router in order to eek out every bit of performance possible. Hitting the CPU isn't a problem if you have the CPU to spare and you're fine with that. My tests just now confirm the hardware offload with one bridge but also show a 10% CPU load doing a local speed test between devices connected to ports that are in the different switch chips. It's 0% if they are on the same chip. So it won't take many threads across the CPU before you would start seeing performance/stability issues. Running a cable between ports in the switch seems stupid since you'd be taking 7 possible ports at 1 gig each (7 gigs traffic) and limiting them to just a 1 gig trunk. Crazy talk.

What is crazy is the fact MikroTik still makes new products with this weird double ASIC thing, which doesn't result in double performance at all in practice.

Why can't they just do a QFX5100 or something…

Probably all comes down to $$$ and the hardware they are able to acquire with that $$$ in order to meet the price point of their target market. That's my best guess.

At the end of the day, the story I've seen time and time again on this forum is that people buy things without researching the block diagram and understanding how the device was meant to be used versus how they want to use it or what their network needs are. For SOHO cases, I don't think some of the creative configurations are ever going to be problematic but for those pushing lots of data or WISPs or setting up building or apartment networks they hit hard walls. And I love how so many of then try to blame it on Mikrotik.

To be clear, OP did not do this and I commend him for doing some thinking on his own, understanding the scenario he was facing, and asking questions to better understand what could be done. Only he can determine if the one bridge at the expense of CPU or two bridges at the expense of 2 ether ports is acceptable. If not, he can always get a managed switch and go with that setup instead. This is the beauty of Mikrotik.

At the end of the day, the story I've seen time and time again on this forum is that people buy things without researching the block diagram and understanding how the device was meant to be used versus how they want to use it or what their network needs are.

I agree on this. Everyone's a network engineer, everyone's an expert, until shit hits the fan, then either one:
1. They blame vendor
2. They blame everyone else but themselves.

Let us know what official support says at least for RB4011.

Both methods are correct.

You can use a single bridge that spans across both switch chips. Packet forwarding within a single switch will use switch forwarding, but packet forwarding across both switches will use CPU. In case vlan-filtering is used and VLAN goes over both switches, you will need to add “bridge” interfaces (the switch CPU port) as a tagged VLAN member, otherwise VLAN will not be able to reach the opposite switch. A similar approach is needed when bridging HW and non-HW offloaded ports.

Or you can use two separate bridges, each HW offloaded.

Bottom-line, single bridge means packet is punted to CPU for inter-switch chip traffic… Don't know how I was wrong at all.

Bottom-line, single bridge means packet is punted to CPU for inter-switch chip traffic… Don't know how I was wrong at all.

Re-read post #10 above ... you claimed that single bridge means reduced throughput (you didn't go with CPU punting initially). And you claimed that one would have to use short patch cable (implicitly saying that switch-CPU interconnects could not be used at all).
And later you called me "spreading fud" due to saying that single bridge spanning both switch chips is a viable solution.

So while technically you may not have been wrong, your communication in this thread was very dismissive of other ideas (to put it very mildly). So you were wrong for denying other setups.