Possible problem with VLAN

RouterOS General
1

Setup is next:

RB4011 is main router and from him there is a trunk port towards hEX S that acts as a switch. Trunk is transporting 3 VLANs: 100,150,200.

I added L009 and I decided to use SFP port as on L009 SFP port is connected to the switch chip.

VLANs are working on hEX S without a problem but what confused me is that when I create trunk between hEX S and L009 only management VLAN is accessible. When I tag bridge on hEX S to other 2 VLANs then on L009 all VLANs start to work.

Config:

L009

# 2024-02-02 18:50:48 by RouterOS 7.13.3
# software id = 
#
# model = L009UiGS
# serial number = 
/interface bridge
add name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=VLAN_100_MGMT vlan-id=100
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2 pvid=100
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether3 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether4 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether5 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether6 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether8 pvid=200
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=sfp1
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,sfp1 vlan-ids=100
add bridge=bridge1 tagged=sfp1 vlan-ids=150
add bridge=bridge1 tagged=sfp1 vlan-ids=200
/ip address
add address=172.16.0.1/24 comment=OFF_BRIDGE_MGMT interface=ether1 network=\
    172.16.0.0
/ip dhcp-client
add interface=VLAN_100_MGMT
/system clock
set time-zone-name=Europe/Zagreb
/system note
set show-at-login=no
/system routerboard settings
set enter-setup-on=delete-key

hEX S

# 2024-02-02 18:59:27 by RouterOS 7.13.3
# software id = 
#
# model = RB760iGS
# serial number = 
/interface bridge
add name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=VLAN100_MGMT vlan-id=100
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether1 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether3 pvid=200
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether4 pvid=200
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether5
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=sfp1
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,sfp1,ether5 vlan-ids=100
add bridge=bridge1 tagged=sfp1,ether5,bridge1 vlan-ids=150
add bridge=bridge1 tagged=sfp1,ether5,bridge1 vlan-ids=200
/ip dhcp-client
add interface=VLAN100_MGMT
/system clock
set time-zone-name=Europe/Zagreb
/system note
set show-at-login=no
2

It seems that your hEX S suffers from the same bug as devices with dual switch chips (e.g. RB4011). The bug being in the way bridge configures switch chip for HW offload. Normally the CPU-switch chip interconnect only has to pass VLANs of which bridge port is member. But in case where this interconnect is used to span ports of bridge (SFP port in your case, other switch chip in dual-switch chip device) switch should be configured to pass also VLANs present on “alien” ports. And the workaround you discovered makes sure that the switch chip-CPU interconnect is configured to pass required VLANs.

So it would be great if you’d open trouble ticket with support (create supout.rif file while bridge config is logically correct but doesn’t work like it should) and keep using workaround you already discovered.

3

Thanks for clarification, i will do that tomorrow. I lost about 2 hours today testing various configurations. I even netinstalled both L009 and hEX S…

Not an expert here but looking at the hEX S block diagram looks like SFP port is connected to the CPU so looks like if SFP is used as trunk towards other switches all traffic must go trough CPU ?

4

Yes, that’s right. The block diagram shows that if hEX S is used as a switch, then using SFP port cripples it quite severely: traffic to/ftom SFP has to pass CPU and CPU-switch chip interconnect is downgraded from 2Gbps to 1Gbps (which is not a huge problem since traffic passing interconnect will be only traffic to/ftom SFP which is limited to 1Gbps anyway).

5

In that case it’s much better solution to simply remove hex s from equation and use S-RJ01 on L009. But I will keep it for now and will create support ticket regarding this bug.

6

Update:

Opened support ticket: SUP-142759

7

Update:

Mikrotik support answered that they managed to reproduce a bug and they will be working on a fix for that in future ROS versions.

ETA unknown for now so workaround it is :smiley:

8

MKX loved your explanation para, but it was like being stuck in mud.
Can you state it in plainer english.

Take the hexS as an example.
ether1 is from ISP and ports 2-4 are bridge ports, a mix of one trunk port and 3 access ports and a trunk port on the last SFP port.
Using pcunites vlan methodology of setting up vlans.
Explain what should happen and what is happening ( the bug ) with respect to hardware offload. ( in basic terms, pretend I am 12, or 63 and partially senile :wink:

9

I’m not good in plain English, it’s alien to me (or is foreign correct word? :wink:)

Warning: config in this post is a hypothetical example and has nothing to do with actual config by @OP

Example: device is used as a switch, so there’s a bridge spanning ether1-5 and SFP. There are a few VLANs, e.g. 100, 200 and 300. SFP is trunk port and is thus tagged for all 3 VLANs, ether5 as well. VLAN 300 is management, ether1-4 are access ports of either VLAN 100 or 200.

Config might be something like this:

/interface bridge
add name=bridge vlan-filtering=yes
/interface bridge port
add bridge=bridge interface=ether1 pvid=100
add bridge=bridge interface=ether2 pvid=100
add bridge=bridge interface=ether3 pvid=200
add bridge=bridge interface=ether4 pvid=200
add bridge=bridge interface=ether5
add bridge=bridge interface=sfp1
/interface bridge vlan
add bridge=bridge tagged=sfp1,ether5 untagged=ether1,ether2 vlan-ids=100
add bridge=bridge tagged=sfp1,ether5 untagged=ether3,ether4 vlan-ids=200
add bridge=bridge tagged=sfp1,ether5,bridge vlan-ids=300
/interface vlan
add name=vlan300 interface=bridge vlan-id=300
/ip address
add address=192.168.88.1/24 interface=vlan300

So in principle management only works via SFP or ether5 if tagged with correct VID.

And in principle traffic should flow between ether1 and ether2 (HW offloaded to switch chip) or between ether3 and ether4 (again HW offloaded) or ether3 and ether5 (again HW offloaded).

Also in principle traffic should flow between ether1 and SFP (via CPU as SFP is not managed by switch chip) or between ether4 and SFP (via CPU).
Only it does not due to bug.

The bug is this: the bridge port and vlan configuration gets translated into switch chip config. I can’t write how it looks on MT switch chip (because manual switch chip config on it never existed), but I can translate it into config for Qualcomm switch chips:

/interface ethernet switch port
set ether1 default-vlan-id=100 vlan-header=always-strip vlan-mode=secure 
set ether2 default-vlan-id=100 vlan-header=always-strip vlan-mode=secure
set ether3 default-vlan-id=200 vlan-header=always-strip vlan-mode=secure
set ether4 default-vlan-id=200 vlan-header=always-strip vlan-mode=secure
set ether5 vlan-header=leave-as-is vlan-mode=secure
set switch1-cpu vlan-header=leave-as-is vlan-mode=secure
/interface ethernet switch vlan
add independent-learning=yes ports=ether1,ether2,ether5 switch=switch1 vlan-id=100
add independent-learning=yes ports=ether3,ether4,ether5 switch=switch1 vlan-id=200
add independent-learning=yes ports=switch1-cpu,ether5 switch=switch1 vlan-id=300

Note that “bridge port” in “bridge config” is essentially replaced by “switch1-cpu” interface. And SFP port doesn’t appear as it’s not run by switch chip.

Now, switch-cpu interconnect (that’s switch port named switch1-cpu) is correctly set as tagged member of VLAN 300 because bridge port is marked as tagged member of that VLAN. But what lacks is setting switch-cpu interconnect as tagged member of the rest of VLANs … indeed it doesn’t have to be as bridge port is not member of those VLANs, but interconnect has to be member because there are other bridge ports (SFP in case of hEX S) beyond the interconnect. And because of this, SFP port is cut off the rest of VLANs, traffic can only pass between SFP and ether5 on VID 300 (because switch chip only passes frames with that VID towards CPU).

Deploying the workaround (adding bridge as tagged port to the rest of VLANs) causes ROS to correctly configure the switch1-cpu port of switch chip … but causes one unwanted side effect (which might not be crucial, but one has to configure some other segments of ROS carefully): CPU becomes accessible via L2 from all VLANs.
E.g. MAC winbox connectivity is possible … and one has to block it by applying restrictive config elsewhere.


If device is used as router and it routes this way or another between all VLANs, then the bug won’t bite … because bridge port will be member of all VLANs and ROS will correctly configure the switch1-cpu port of switch chip.

10

Just to be clear, ether5 is from the router and SFP is to another smart device
OR
SFP is from router and ether5 is to another smart device.
Flows in, flows out, guess it doesnt really matter to the discussion.

vlans 100,200,300 have to come in on either ether5 or spf1, what goes out the other end (ether5, spf1) is determined by the OP, but somehow i doubt its just the management IP.
Suggest you slightly modify your config so it is viable. To distracting when its incomplete… :wink:

11

@Mesquite: my latest post is reply to request by @anav, it doesn’t relate to config by @OP in any way. Added a warning in nice large letters not to mislead any potential reader.

Alas: as I wrote, it can be used as complete config of a switch (but I’m not asserting any context).

12

Thanks for the clarification!