Also tried with just one single bridge and this is the partly working but mostly not working config.
Adding and trying bridge 2 does not work at all yet.
My posted config has both bridges with only attempting to have port 10 on bridge-2 which so far does not work at all.
I can just delete Bridge-2 and reassociate port 10 to Bridge-1 and I will be getting the same results (both ports 9 and 10).
So this is my current config..
VLAN33 is the upstream Internet and local LAN.
Works perfectly fine for anything connected to ports on EITHER switch CHIP on VLAN 33.
But does not work Across the switch chips.
So if I plug into ports 1,3 it works perfectly.
Same if I plug into ports 9,10 no problems.
Only testing/failing with VLAN33 (all other config works correctly and is outside of this problem).
VLAN33 happens to be where the router itself gets its Internet from and this works flawlessly no matter what port associated to VLAN33 is plugged into the upstream network.
My testing/failing testing is only in regard to testing between hosts connected to ports 1,3 ↔ 9,10
I’m trying to properly figure out how to pass L2 VLAN between all 4 ports both switch chips and have it work 
I’m not sure why it is sort of partly intermittently working..
It’s as if ARP gets a learned path and sticks to only that one path and denies any other path once it sees the first combination.
I am only speculating this but seems to be in alignment with this behaviour as I am only guessing.
/interface bridge
add name=BR-1
/interface vlan
add interface=BR-1 name=33-Internet vlan-id=33
add interface=BR-1 name=47-Tenant vlan-id=47
add interface=BR-1 name=70-video vlan-id=1
add interface=BR-1 name=99-MGMT vlan-id=99
/interface ethernet switch port
set 0 default-vlan-id=33 vlan-mode=secure
set 1 default-vlan-id=47 vlan-mode=secure
set 2 default-vlan-id=33 vlan-mode=secure
set 3 default-vlan-id=99 vlan-mode=secure
set 4 vlan-mode=secure
set 8 default-vlan-id=33 vlan-mode=secure
set 9 default-vlan-id=33 vlan-mode=secure
set 10 vlan-mode=secure
set 11 vlan-mode=secure
/ip pool
add name=dhcp_pool0 ranges=\
172.16.44.1-172.16.46.250,172.16.47.2-172.16.47.254
add name=dhcp_pool1 ranges=172.16.99.100-172.16.99.240
/ip dhcp-server
add address-pool=dhcp_pool0 interface=47-Tenant lease-time=2d30m name=dhcp1
add address-pool=dhcp_pool1 interface=99-MGMT lease-time=2d30m name=dhcp2
/port
set 0 name=serial0
/queue simple
add disabled=yes max-limit=20M/20M name=20-test target=47-Tenant
/interface bridge port
add bridge=BR-1 interface=ether2
add bridge=BR-1 interface=ether3
add bridge=BR-1 interface=ether4
add bridge=BR-1 interface=ether5
add bridge=BR-1 interface=ether1
add bridge=BR-1 interface=ether9
add bridge=BR-1 interface=ether10
/interface ethernet switch vlan
add independent-learning=yes ports=ether5,ether3,switch1-cpu,ether1 switch=\
switch1 vlan-id=33
add independent-learning=yes ports=ether5,ether2,switch1-cpu switch=switch1 \
vlan-id=47
add independent-learning=yes ports=ether5,ether4,switch1-cpu switch=switch1 \
vlan-id=99
add independent-learning=yes ports=ether9,ether10,switch2-cpu switch=switch2 \
vlan-id=33
/ip address
add address=172.16.47.1/22 interface=47-Tenant network=172.16.44.0
add address=10.73.73.2/24 interface=33-Internet network=10.73.73.0
add address=172.16.99.1/24 interface=99-MGMT network=172.16.99.0
add address=172.16.70.1/24 interface=70-video network=172.16.70.0
/ip dhcp-server network
add address=172.16.44.0/22 dns-server=10.73.73.31 gateway=172.16.47.1
add address=172.16.99.0/24 dns-server=10.73.73.31 gateway=172.16.99.1
/ip firewall filter
add action=fasttrack-connection chain=forward hw-offload=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=33-Internet
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=10.73.73.1 routing-table=main \
suppress-hw-offload=no
Also SFP port is not part of any switch chip.
Trying to figure out how to enable VLAN trunking on this and get access to existing VLAN arrangement that is setup and working on switch chips.
None of this is easy or simple at all if you are new to Mikrotik.
I’m having a lot of fun but am bumping into lack of know-how at every single turn.
For every item I figure out and make work I find the new things to have to learn how to figure out.
Yes- this is pretty ugly at the moment..
I have partly mastered Bridge/VLAN with VLAN filtering (works on newer stuff like CCR2004 I have).
And I have got to where I can do witch level VLAN on a single switch chip.
I was lalso able to make a separate bridge jsut for the SPF port and enable vlan filtering on this to get it to carry VLANS as a trunk..
However this can;t talk to the other bridge/Chip level VLAN - VLAN interfaces..
Only works if I move the VLAN interfaces to the new bridge for the SFP port but then I lose all L2 connectivity to BOTH switch chip ports.
It’s noted in the warning from MikroTik https://help.mikrotik.com/docs/display/ROS/Basic+VLAN+switching#BasicVLANswitching-Otherdeviceswithabuilt-inswitchchip that I posted above (repasted here, this time with different highlight 
):
For devices that have multiple switch chips (for example, RB2011, RB3011, RB1100), each switch chip is only able to switch VLAN traffic between ports that are on the same switch chip, VLAN filtering will not work on a hardware level between ports that are on different switch chips, this means you should not add all ports to a single bridge if you are intending to use VLAN filtering using the switch chip, > VLANs between switch chips will not get filtered> . You can > connect a single cable between both switch chips > to work around this hardware limitation, another option is to use > Bridge VLAN Filtering, but it disables hardware offloading > (and lowers the total throughput).
So, if you want ports on different switch chips to be part of the same VLAN (for instance ether9 and ether3 both parts of VLAN 33) you have two “solutions”:
- Either forget the whole switch chip configuration thingy, and use Bridge VLAN Filtering with a single bridge, with the downside of losing hardware offload and having high CPU usage (what you noticed at the start of this thread) because all processing is done by the main CPU.
- Or use the “trick” of plugging a physical cable between the two port groups, for instance between ether5 and ether6. With the huge downside of sacrificing two ethernet ports. Your RB3011 router becomes one with only 8 ethernet port and 1 SFP port.
If you chose to follow the later solution with the cable trick, you should set up your router like this:
- Setting up the VLANs on the first switch chip:
- Create a bridge with the ports that you want to use on the first switch chip.
- Create the /interface/vlan entries for the VLANs with Layer 3 access, with that bridge as parent interface of the VLAN interfaces.
- Configure the VLANs with /interface/ethernet/switch, with switch1-cpu as member port for the VLANs with Layer 3 access.
- Configure (with /interface/ethernet/switch) port ether5 to be the trunk port of all VLANs that will be shared between the two switch chips (your VLAN 33 falls into this).
- Setting up the shared VLANs on the second switch chip:
- Configure the VLANs with /interface/ethernet/switch. However, DON’T add switch2-cpu as member port for any of those VLANs at all. If Layer 3 access is needed to the VLANs, then the main CPU should access these VLANs solely through the first switch chip, not through both!
- Configure (with /interface/ethernet/switch) port ether6 to be the trunk port of all VLANs that will be shared between the two switch chips (your VLAN 33 falls into this).
- Plug an ethernet cable between port ether5 and ether6. At this moment, the ports on the 2nd chip that is part of this configuration act like an external managed switch (Layer 2 only) that you’ve plugged into port ether5 of the router.
- The following is optional, in case you have more VLANs that are only used on the second switch chip, then additional steps are needed:
- Create a bridge with at least the ports that you want to use for those VLANs on the second switch chip (or add all the ether6-ether10 ports if you want).
- Create the /interface/vlan entries for the VLANs with Layer 3 access, with the 2nd bridge as parent interface of the VLAN interfaces.
- Configure the VLANs with /interface/ethernet/switch, with switch2-cpu as member port for the VLANs with Layer 3 access.
A few random bits, hopefully they’ll help you to understand your device and setup:
- despite the urban legend of necessity to run two bridges on devices with two switch chips that’s not true. Indeed setup with using external patch cord connecting ports on different switch chips may help with performance (or rather: reduces CPU load), but from functional point of view it’s not necessary (plus it uses two RJ45 ports, if you can spare them then that’s fine).
So the most “straight forward” way of doing it is to use single bridge. - always use “dumb” bridge, i.e. don’t enable vlan-filtering on it
- configure switchX-cpu port (on both switch chips) as trunk port and add all necessary VLANs to it.
Necessary means: all VLANs which are present on both switch chips and all VLANs with which device will interact (e.g. route between VLANs). The dumb bridge will pass frames between both switch chips (unaltered) as well as make frames available on bridge port (where vlan interfaces will deal with them) - if you have to add any other interfaces to the mix (e.g. sfp port or wireless interface), then there are not many options. With legacy wireless drivers (and new wifi-qcom, non-ac) it’s possible to pass VLAN tag handling to driver.
In your particular case (sfp port, which is supposed to be trunk port), you can add sfp port to bridge. This way sfp port will be trunk port (or hybrid, depending on switchX-cpu port configuration) and you don’t have any control over which VLANs can pass that port.
If control over VLANs passing sfp port is absolutely required, then you can play with additional bridges, one bridge per VLAN. Something like the config example below.
Example of multi-bridge (one per VLAN) config (based on config shown in post #22 above):
/interface/bridge
add bridge_vlan47
add bridge_vlan90
/interface/vlan
add interface=sfp1 name=sfp1_vl47 vlan-id=47
add interface=sfp1 name=sfp1_vl90 vlan-id=90
/interface/bridge/port
add bridge=bridge_vlan47 interface=47-Tenant
add bridge=bridge_vlan47 interface=sfp1_vl47
add bridge=bridge_vlan99 interface=99-MGMT
add bridge=bridge_vlan99 interface=sfp1_vl99
# move IP address from VLAN interface to appropriate bridge
/ip address
set [ find address=172.16.47.1/22 ] interface=bridge_vlan47
set [ find address=172.16.99.1/24 ] interface=bridge_vlan99
etc. This way sfp1 port will only pass (and accept) VLANs where appropriate vlan interface is created (and made member of corresponding bridge). If you don’t care about VLAN security, then simply add sfp1 interface to bridge BR-1 and be done with it.
In any case, traffic via SFP port will hit on CPU. With the bridge-per-vlan mess illustrated above the CPU will be hit harder due to executing many vlan/bridge/… paths (as compared to single bridge path). And you can’t get away from it by using two bridges and external patch cable between a pair of ports …
Yeah, the SFP port is directly connected to the main CPU (and you lose 1Gbps link to the 2nd switch chip if you plug a module into the SFP port). I think the solutions for VLAN spanning across the SFP port and the rest of the ports might be:
-
Either to use Bridge VLAN Filtering, with one bridge containing all required ports. Again, this is a software only solution.
-
Or you can try this (this is only me talking from theory, I haven’t tried to see if it really works or is even configurable at all):
- Setup the VLANs on the switch chips as usual. For each VLAN ID that will also be used on the SFP port, the CPU port (switchX-cpu) should be added so that the CPU has access to the VLAN.
- Also for those VLANs (that should be available to the SFP port) interfaces under /interface/vlan should be added, with the bridge interface as parent. Let’s name them vlanXX.switch for instance (XX being the VLAN ID). But no Layer 3 configuration should be made on these interfaces (no /ip/address, no DHCP Client/Server)
- Under /interface/vlan, create the VLAN interface for the SFP port. These VLAN interfaces have sfp1 as parent interface. Let’s name them vlanXX.sfp. Again, no Layer 3 configurations on these interfaces.
- For each of those VLANs, create a separate new bridge interface. Let’s name them vlanXX.bridged.
- For each vlanXX.bridged, add vlanXX.switch and vlanXX.sfp as member ports.
- Configure Layer 3 features (IP addresses, DHCP, etc…) on vlanXX.bridged if needed.
With this config, traffic between the ethernet ports and sfp1 as well as broadcast/multicast traffic will require the CPU, however, hopefully, normal switching between the ethernet ports should still be hardware offloaded.
I will try this and see how it goes!
And will be back in a day to let you know.
What I had tried so far on my own was to create a separate bridge just for the SFP but have found no way yet that works to connect or pass traffic between the two bridges.
And the VLAN can;t be associated directly to two bridges, you can only choose one so you either get the SFP or the switch chip bridge.
And my only option for the additional bridge was of course VLAN filtering/software.
Which for now is OK I don;t care about the reduced performance or using the CPU I just want to get the experience to make it work on both the SFP and the Local switch chip ports.
Yeah, one bridge and VLAN filtering will probably work but then as you know and have pointed out I would lose what I alrady have working with the switch chip.
I also have been unable to get communication between the two Switch chips to work yet (same VLAN present on both chips (ok if one is software) but was trying to keep ONE still be hardware and then the other be able to link to it and communicate even if it has to be through software..
I got hat kinda working but mopstly not… it does that wired thing that seems to be ARP related where it “prefers” the first chip and one or two hots work temporarily through Chip #2 but in a weird way and not for very long.
I’m not exactly sure what that was doing or how to explain it but it was not working correctly nor was it completely dead and not passing any traffic whatsoever.
If you’re using RB3011 “for educational purposes”, then I suggest you to go with single bridge with vlan-filtering enabled. This kind of configuration works on all devices (regardless the underlying hardware) and on many devices (including all the modern switches and many modern routers) it can be HW-accelerated.
What this kind of config sucks at is performance on devices without HW-offload support. And that’s all the devices with Qualcomm switch chips (Atheros ). On those it’s possible to configure VLANs on switch chip, but then one often bumps into bridge port which is not run by switch chip and it’s hard to set VLAN parameters.
If such device is intended to be used in production environment, where wirespeed throughput is required, then either only use features which are wirespeed (in case of RB3011 only RJ45 ports with those gotchas) or get a more modern device which does things “the right way”. Educational effect of the exercises we’re now at is very questionable and don’t really end with a good outcome - some of traffic will hit CPU this way or another, just depends the scale of it. And personally I’m not a fan of workarounds since they tend to be complicated and hard to understand (after a while even for person who implemented them).
Cool, yes I started off with just VLAN filtering and a single swithc chip, that was super easy and was my introduction to VLANs on Mikrotik.
As soon as I stared “playing” around and trying to get hardware L2 on a single switch ship I needed a little help here and had to study/try things and now that works…
Problem with that now is NO way I have found to get the SFP port working with it (while at the same time) being able to access that VLAN with a switchchipport.
I can probably get to to work easily with an all software/bridge approach. (which would be needed anyway for the SFP port since it only goes to the CPU.
But I kinda wanted two ports to be able to talk directly on the same switch chip while at the same time getting a software bridge to work the to the SFP for the same VLAN.
I’m also running into crazy problems when trying to get communications between the two switch chips (software) (same VLAN) while at the same time having hardware L2 (SAME VLAN)
On the switchchips themselves..
So getting something like (L2 hardware)
THIS WORKS! VLAN 10: Ports 2 and 3 (2 and 3 can talk directly to each other on VLAN10 with no CPU)
THIS WORKS! VLAN 10: Ports 5 and 6 can talk directly to each other on VLAN10 with no CPU)
While all that is working:
What I am trying: (expected to use CPU for obvious reasons) Ports (2 or 3) cannot talk reliably to ports 5 or 6 (it “tries”) and sometime s works to certain hosts across this path but mostly does not work. I’ve made no good sense of thsi yet seems to be some kind of ARP path or first come first server thing, once it works the first time on one of the two paths it only works on that path.
Meaning fresh after reboot if you ping something on VLAN 10 that is connected to port 1 from port 5 it works!
But if something on port 2 has already pinged that host on port 1, something on port 5 now cannot ping that host on port 1.
Like it’s only allowing one path that it “learns” when it first happens..
Oddly any hosts than can be pinged from port 5 (that are connected to port 1) (and there always seems to be a couple that work while most do not- these can also always be pinged
from port 2 (which is on the same switch chip) This is the really strange part that is hard to determine what exactly is happening..
I’m at the point where I have determined “it mostly doesn’t work” as described above (that was really rough) but it kind of works a little.. I can randomly reach some hosts connected on port 1 from ports 5 or 6 very sporadically.
Haha..
This is the learning part for me..
When crazy stuff like this happens you have to try to break it down and simplify as much as possible.
This is completely outside of the SFP stuff, I added that as a separate task to try and lean how to deal with it.
Will mostly require software bridging only and will force me to LOSE any direct connects between ports on same switch chips.
Unless I am totally missing something cool.
There were some ideas presented here to try out and I will still have to try that..
All have access to VLAN10
------------
Hardware L2 : ports 3 and 4 can talk directly VLAN10
Port 3
Port 4
Port 5 tagged VLAN10 to external managed switch
CPU0 switch chip VLAN10 tagged
------------
-
-
-
CPU VLAN10 link
-
-
-
-------------
Hardware L2 ports 6 and 7 can talk directly
Port 6
Port 7
CPU1 switch chip VLAN10 tagged
--------------
All works except things on ports [6,7] cannot reach things on ports [3,4]
Expect and realize this would be via CPU while other things on each switch chip can be direct.
Also anything on ports 6,7 would be via CPU to tagged VLAN10 on other switch chip port5
It all works as wanted except being able to get from ports 3/4 and over to 5/6 across the CPU.
It is sorta working intermittently for a few hosts but mostly not working as described above.
The RB3011 was replaced with a CCR2004 at a production site and it is working fantastic with all of the hardware stuff+ bridge working perfectly.
I took home the RB3011 to learn more on.
I have it working perfectly using only one of the switch chips but am now just playing and learning to see if I can possibly span VLANS to the other switch chips while still having
hardware L2 VLAN on the second chip and software link between them.
But am running into that weird semi working (mostly not working) problem when I try that.
I will continue to study and try just for fun!
Some of this may apply and I will study and retry as I go!
I did get the SFP to work and carry the tagged VLANS (via CPU of course) with the first switch ship working perfectly.
All have access to VLAN10
Hardware L2 : ports 3 and 4 can talk directly VLAN10
Port 3
Port 4
Port 5 tagged VLAN10 to external managed switch
CPU0 switch chip VLAN10 tagged
CPU VLAN10 link
Hardware L2 ports 6 and 7 can talk directly
Port 6
Port 7
CPU1 switch chip VLAN10 tagged
All works except things on ports [6,7] cand reach things on ports [3,4]
Expect and realize this would be via CPU while other things on each switch chip can be direct.
Also anything on ports 6,7 would be via CPU to tagged VLAN10 on other switch chip port5
It all works as wanted except being able to get from ports 3/4 and over to 5/6 across the CPU.
It is sorta working intermittently for a few hosts but mostly not working as described above.
hi, CGGXANNX, I have a RB4011 device which has two switch chips too. I found the packet can flow between ports in different switch chips in default, no matter if vlan-filtering is enabled.
So I wonder in what condition each switch chip is only able to switch VLAN traffic between ports that are on the same switch chip?
The RB4011 has two RTL8367 switch chips, and for RTL8367 the recommended way to set up VLAN is Bridge VLAN Filtering, because hardware offload for Bridge VLAN Filtering has been supported since ROS 7.1 https://help.mikrotik.com/docs/spaces/ROS/pages/328068/Bridging+and+Switching#BridgingandSwitching-BridgeHardwareOffloading.
Which means you don’t configure the switch chips directly like with the RB3011 from the OP, but create bridge(s) and use VLAN filtering on the bridge(s). Now when you create only one bridge spanning the two switch chips then manipulations done by ROS in the background will ensure that things still work. Because the bridge (handled by software when you span two chips) knows which VLANs exist and are available on which ports, it can change the settings of both switch chips so that VLAN traffics between ports of the same chip are still hardware offloaded by the chip, and broadcast/multicast as well as frames for VLAN spanning the two chips will be passed to the CPU port of the switch chip too (and the CPU will forward them to the other switch chip). Which means the VLAN spanning the two chips will work normally, and will just use more CPU resources and use the link between the CPU and the switch chips when frames need to travel between the two chips.
For VLANs that reside only on one switch chip the CPU is not needed and VLAN filtering is hardware offloaded on the switch chip. Same for the case when traffics only need to travel between ports of the same chip.
Contrast this to the RB3011 where the only way to have hardware offloaded VLANs is to perform the configuration directly on the switch chips. But because each of the two switch chips of the RB3011 has no knowledge of the other one, there is no mechanism in the switch chip configuration to refer to external chips. To have hardware offload, the whole VLAN must reside on one switch chip, if the other switch chip needs to be part of the VLAN, it must be treated either as an external smart switch (with the additional cable), or you have to give up hardware offloading. On the RB3011, if you create a bridge spanning the two chips and enable Bridge VLAN Filtering then everything will be done in software, because ROS has no ability to automatically configure the switch chips in the background like with the RB4011 or other chips supporting hardware offloaded Bridge VLAN Filtering (from the table of the link above).
Thank you for your reply!
It sounds like the differences between the two products are due to design differences on the switch chips at the operating system software level.
Indeed. Only the claim about total loss of HW offload is not correct.
In case one configures VLANs on switch chip, bridge is still necessary, but no VLAN settings should be set, in particular vlan-filtering has to be disabled (bridge acts as a dumb switch but passes VLAN headers just fine). If one then configures same VLAN on both switches (and includes switchX-cpu port as trunk), then frames belonging to that VLAN will pass between both port groups (managed by different switch chips) just fine, only CPU will be doing some work.