Hex PoE VLAN setup (are switch and bridge VLAN setups equal in terms of features?)

RouterOS Beginner Basics
1

Hi,

I want to setup my Mikrotik with secure and efficient VLANs. But there are many ways to setup VLANs. So I have some questions about that.

I suppose my device could best use the /interface bridge vlan method. Since it has hardware offloading. However, there is also /interface ethernet switch to set VLAN (options), but from what I understand from this presentation, that option is for hardware only setups.

The thing I want to setup is the following.
Management and replication: Only locally between the VLAN members (tagged)
Public: Goes to Internet, to host services on, will be using NAT for IPv4 and a subnet for IPv6 (tagged)
Native: Default VLAN that’s untagged and goes to my router and also has Internet, but is just for access, not hosting

So, I’ve setup my VLANs with this.

/interface vlan add interface=br0 name=management vlan-id=11
/interface vlan add interface=br0 name=replication vlan-id=12
/interface vlan add interface=br0 name=public vlan-id=13

/ip address add address=172.27.10.4/24 interface=br0
/ip address add address=172.27.11.1/24 interface=management
/ip address add address=172.27.12.1/24 interface=replication
/ip address add address=172.27.13.1/24 interface=public

/interface bridge vlan add bridge=br0 untagged=ether1,ether2,ether3,ether4,ether5 vlan-ids=10 comment=native
/interface bridge vlan add bridge=br0 tagged=ether2,ether3,ether4,ether5 vlan-ids=11 comment=management
/interface bridge vlan add bridge=br0 tagged=ether2,ether3,ether4,ether5 vlan-ids=12 comment=replication
/interface bridge vlan add bridge=br0 tagged=br0,ether2,ether3,ether4,ether5 vlan-ids=13 comment=public

With these port options:

 /interface/bridge/port/print detail 
Flags: X - disabled, I - inactive; D - dynamic; H - hw-offload 
 0     ;;; rp1
       interface=ether2 bridge=br0 priority=0x80 path-cost=10 internal-path-cost=10 edge=yes-discover 
       point-to-point=auto learn=auto horizon=none hw=no auto-isolate=no restricted-role=no restricted-tcn=no 
       pvid=10 frame-types=admit-all ingress-filtering=yes unknown-unicast-flood=yes unknown-multicast-flood=yes 
       broadcast-flood=yes tag-stacking=no bpdu-guard=yes trusted=no multicast-router=temporary-query fast-leave=no 

 1     ;;; rp2
       interface=ether3 bridge=br0 priority=0x80 path-cost=10 internal-path-cost=10 edge=yes-discover 
       point-to-point=auto learn=auto horizon=none hw=no auto-isolate=no restricted-role=no restricted-tcn=no 
       pvid=10 frame-types=admit-all ingress-filtering=yes unknown-unicast-flood=yes unknown-multicast-flood=yes 
       broadcast-flood=yes tag-stacking=no bpdu-guard=yes trusted=no multicast-router=temporary-query fast-leave=no 

 2     ;;; rp3
       interface=ether4 bridge=br0 priority=0x80 path-cost=10 internal-path-cost=10 edge=yes-discover 
       point-to-point=auto learn=auto horizon=none hw=no auto-isolate=no restricted-role=no restricted-tcn=no 
       pvid=10 frame-types=admit-all ingress-filtering=yes unknown-unicast-flood=yes unknown-multicast-flood=yes 
       broadcast-flood=yes tag-stacking=no bpdu-guard=yes trusted=no multicast-router=temporary-query fast-leave=no 

 3     ;;; rp4
       interface=ether5 bridge=br0 priority=0x80 path-cost=10 internal-path-cost=10 edge=yes-discover 
       point-to-point=auto learn=auto horizon=none hw=no auto-isolate=no restricted-role=no restricted-tcn=no 
       pvid=10 frame-types=admit-all ingress-filtering=yes unknown-unicast-flood=yes unknown-multicast-flood=yes 
       broadcast-flood=yes tag-stacking=no bpdu-guard=yes trusted=no multicast-router=temporary-query fast-leave=no 

 4     ;;; rt1
       interface=ether1 bridge=br0 priority=0x80 path-cost=10 internal-path-cost=10 edge=auto point-to-point=auto 
       learn=auto horizon=none hw=no auto-isolate=no restricted-role=no restricted-tcn=no pvid=10 
       frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes unknown-unicast-flood=yes 
       unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no bpdu-guard=no trusted=yes 
       multicast-router=temporary-query fast-leave=no

And these bridge settings:

/interface/bridge/print detail 
Flags: X - disabled, R - running 
 0 R name="br0" mtu=auto actual-mtu=1500 l2mtu=1598 arp=enabled arp-timeout=auto mac-address=CC:2D:E0:81:0A:BE protocol-mode=mstp fast-forward=yes igmp-snooping=no auto-mac=no admin-mac=CC:2D:E0:81:0A:BE ageing-time=5m priority=0x8000 
     max-message-age=20s forward-delay=15s transmit-hold-count=6 region-name="" region-revision=0 max-hops=20 vlan-filtering=yes ether-type=0x8100 pvid=10 frame-types=admit-all ingress-filtering=yes dhcp-snooping=yes 
     add-dhcp-option82=no

But would it make sense to also mix it with these /interface ethernet switch port settings? Like these to harden the VLAN configuration?

/interface ethernet switch port set ether1 vlan-mode=secure vlan-header=always-strip
/interface ethernet switch port set ether2 vlan-mode=secure
/interface ethernet switch port set ether3 vlan-mode=secure
/interface ethernet switch port set ether4 vlan-mode=secure
/interface ethernet switch port set ether5 vlan-mode=secure
/interface ethernet switch port set sw1-cpu vlan-mode=secure

Or can I just as well use /interface bridge vlan options for this that have a similar or exactly the same?

In any case, I have hardware offloading disabled on my Hex PoE because I’m using a combination of settings which don’t allow me to enable hardware offloading. So that makes me wonder, since hardware offloading is off, can I use /interface ethernet switch port?

So in summary, my questions come down to this.

  1. Can I mix /interface ethernet switch and /interface bridge vlan to set VLANs up? Also when hardware offloading is disabled (or will that just make things very slow?).
  2. Is one of these two superior?
  3. See code block below, I see more VLANs than I’ve configured, such as VLAN 1280, and 0 and 1. How can this be?
# tshark -r dump -e vlan.id -Tfields | sort -u 
Running as user "root" and group "root". This could be dangerous.

0
1
11
12
1280
13
2

I’m not reading all of the explanation, I’ll fast-forward to your questions:

.

  1. No, you should not mix settings in those two sections. Mixing can result in unpredictable behaviour.
  2. Bridge VLAN can not be offloaded to underlying hardware on hEX PoE. So performance wise the switch chip is much better.
  3. setting pvid=10 property on port means that port is actually untagged for VLAN ID 10 outside of routerboard device. ROS has VID 1 as default setting in many pkaces so if you don’t configure things entireky right, you might see frames with VID=1 flying around. You didn’t explain the tshark experiment so I can’t comment on the meaning of the results.

If you feel that we might give you some help, post complete configuration (/export file=anynameyouwish hide-sensitive), not just a few parts of it.

3

But in terms of features, are they very different? They seem to be very different when I check the command switches and what you could configure with them. Things like the “vlan-mode=secure vlan-header=add-if-missing” and such.

4

The difference is not big for this particular switch chip. Feature set of switch chip configuration is closely related to switch chip HW feature set. Some advanced switch chips feature very advanced features which are unique and not available on bridge. Bridge configuration is more generic and is the same regardless underlying hardware.
Surely syntax is quite different and it’s not easy to make a comparision.

5

I’ve been experimenting, and indeed using the switch chip natively increases the performance 5x (going from ~200Mbit/s when using just one interface to 1Gbit/s per each interface). So that’s a great win.

Security
I suppose that VLAN filtering in bridge mode, with some combinations of frame-type are the same as the VLAN mode and VLAN header settings in switch VLANs are doing the exact same thing? There is no gap in terms of security features there as far as I can understand these options. All the other fancy features I thought were nice in the bridge VLAN setup are not needed in the switch VLAN setup ( BPDU guard, trusted ports, and more bridge related options). So I’m not loosing anything, only gaining a simpler and faster setup that’s just as (or maybe more secure).

I do have some final questions though. Below is my config, which is very basic, just played with the switch VLAN stuff.

/export hide-sensitive
# jun/04/2022 14:25:18 by RouterOS 7.2.3
# software id = A1GI-TFVF
#
# model = 960PGS
# serial number = 89F90861A06A
/interface bridge
add admin-mac=CC:2D:E0:81:0A:BE auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether2 ] poe-out=forced-on
set [ find default-name=ether3 ] poe-out=forced-on
set [ find default-name=ether4 ] poe-out=forced-on
set [ find default-name=ether5 ] poe-out=forced-on
/interface ethernet switch port
set 0 default-vlan-id=10 vlan-header=always-strip vlan-mode=secure
set 1 default-vlan-id=10 vlan-mode=secure
set 2 default-vlan-id=10 vlan-mode=secure
set 3 default-vlan-id=10 vlan-mode=secure
set 4 default-vlan-id=10 vlan-mode=secure
set 5 default-vlan-id=10 vlan-mode=secure
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp disabled=yes interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface ethernet switch vlan
add comment=native independent-learning=yes ports=ether1,ether2,ether3,ether4,ether5,switch1-cpu switch=switch1 vlan-id=10
add comment=management independent-learning=yes ports=ether2,ether3,ether4,ether5 switch=switch1 vlan-id=11
add comment=replication independent-learning=yes ports=ether2,ether3,ether4,ether5 switch=switch1 vlan-id=12
add comment=public independent-learning=yes ports=ether2,ether3,ether4,ether5 switch=switch1 vlan-id=13
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" disabled=yes in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no dst-address=172.27.13.0/24 gateway=ether1 routing-table=main suppress-hw-offload=no
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/system clock
set time-zone-name=Europe/Amsterdam
/system identity
set name=rt2
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

As you can see I have set all ports to VLAN mode secure and the VLAN header to always-strip for ether1. This is because ether1 is not suppose to receive tagged frames. Maybe it will in the future, but for now it depends on the default VLAN tag. But no matter what VLAN header config I set for the ether1, it always works. Probably because of the default VLAN membership? What would then be the best setting for this port if I don’t expect tagged traffic?

The other ports have it set to leave-as-is. ether2 to ether5 are connected to Raspberry Pi’s that sent out tagged frames, but also depend on the default VLAN (untagged). So in a sense these are hybrid ports? I’m a bit confused about this. When I check the documentation, I suppose the ports should be set to add-if-missing. Because then you ensure the PVID is always set? For some reason all the VLANs work just fine in the current setting, but also when setting it to add-if-missing (trunk ports). When I remove a port member, the connection breaks between those VLAN members. So the VLAN setup does work as I intend it to. I just find these extra security options a bit hard to understand.

When I check this table it basically doesn’t seem to matter if I use add-if-missing or leave-as-is. Both ended being treated as “Tagged traffic is sent out, tag is already present on ingress port” and otherwise it’s dropped. Is there still some reason to chose for any of these modes?

As mentioned in the host table section of the documentation:
“Packets without VLAN tag are treated just like if they had a VLAN tag with port default-vlan-id.” This means that the secure mode works because I specified some VLAN tags, but also defined a default VLAN with number 10. And thus the following applies to my configuration and makes the secure mode work: “vlan-mode=check or secure” to be able to forward packets without VLAN tags you have to add a special entry to VLAN table with the same VLAN ID set according to default-vlan-id.

VLAN learning
I’ve enabled independent-learning, because it seems like it’s best to keep smaller VLAN linked tables rather than one large table. Does this have any benefit? Does this improve isolation and perhaps also performance? The documentation isn’t really clear about why you would turn this on or off and what you might gain or lose.

Spanning tree protocol
In the bridge VLAN setup I configured it with MSTP, which was recommended in the documentation when using a multi VLAN setup. It’s unclear to me which protocol I’m using now. Maybe this is a dumb question, because of the setup I’m using now (not bridge VLAN).

Firewall
With the bridge VLAN setup I could enable the IP firewall, for also VLAN traffic. This option is now not present in the switch VLAN setup. But maybe this is a dumb questions as well and I can just always use the firewall for this. But it just needs an extra layer for bridges and therefore it’s an extra option? But it’s enabled regardless for the switch VLAN setup? Just want to confirm. I can of course still use a layer 3 firewall and use input and output ports. So I guess it doesn’t matter.

Switch1 interface
In documentation it states that “All switch chips have a special port that is called switchX-cpu, this is the CPU port for a switch chip, it is meant to forward traffic from a switch chip to the CPU, such a port is required for management traffic and for routing features. By default the switch chip ensures that this special CPU port is not congested and sends out Pause Frames when link capacity is exceeded to make sure the port is not oversaturated, this feature is called CPU Flow Control. Without this feature packets that might be crucial for routing or management purposes might get dropped.”

However, this interface is also available for selecting VLAN membership. I already noticed that this interface is important for the default VLAN. Because when I remove switch1 from the default VLAN 10, I cannot connect to the Mikrotik device anymore. I can still reach the other devices connected to the Mikrotik. Acts this a bit the same as the bridge1 interface in the bridge? If so, can maybe someone explain how this exact works and in which situations I should include it in VLAN membership and when not. Maybe some other gotchas as well.

6

Security:

I’m not entirely sure but as far as I understand, switch chip section doesn’t have default setting of default-vlan-id (unlike pvid under bridge port). In this case tagless frames won’t get tagged on ingress and with vlan-mode=secure they will be dropped (with the rest of modes they will pass untagged). Thd vlan-header property is actually about egress frames and is largely ignored by switch chip QCA8337 (and AR8327) as explained by manual page.
My strategy is to set vlan-headers=leave-as-is for tagged-only ports (which in my setups includes switchX-cpu ports) and always-strip for untagged and hybrid ports (this setting only applies to frames with VID equal to default-vlan-id/pvid).


VLAN learning:

most of network implementations will mostly use same MAC address for all VLANs used by L3 host passing same physical interface of a NIC. And in this case this setting doesn’t matter. However some implementations allow setting different MAC addresses for different VLANs (but this still doesn’t make any change with regard to this setting). Which in turn allows for having same MAC in different VLANs on different switch ports. In which case independent-learning=yes makes a difference. Personally I’ve never run into such setup but one never knows. So either go with paranoia and enable this setting (or enable it just because it’s possible) or leave it disabled and be prepared to deal with it if “the worst thing happens” (it won’t be pretty do diagnose it, the odds of happening it are low though).


xSTP:

switch itself doesn’t do xSTP, bridge does. I’m not an expert on xSTP, but AFAIK in VKAN environment only MSPT actually does the right thing in all cases. RSTP is mostly fine as well. When I think of it, MSTP has to be used (rather than RSTP) exactly in cases where independent learning is a must. And in some other cases where different VLANs pass different paths between core switches (not something a typical SOHO or small business network admin would habitually do). Bridge default setting is to use RSTP.


Firewall:

by default firewall rules only act on packets passing L3 of router (i.e. routed traffic, also between different VLANs, from L3 perspective VLANs and stand-alone physical ports are not different at all). Options use-ip-firewall* make bridge to pass frames through normal IP firewall even if they are only bridged, but for that to happen frames have to be dealt with by CPU, not HW offloaded. In which case you automatically loose all the benefits of configuring VLANs on switch chip. Some advanced switch chip support ACLs, some firewall functions can be written by using those.


switchX-cpu:

hardware wise that’s the sixth port of switch chip, only that connects to CPU rather than physical ether ports. And when configuring switch chip you have to consider it as such. Traffic not passing from switch chip via this port can’t reach CPU. Traffic passing this port ends up on (or originate from) bridge interface. And config has to match. E.g. if switch chip config says switchX-cpu is tagged only, then bridge interface has to be configured as tagged only.