Help me visually understand routing

[ConradPino]

L3HW does’t apply to OP devices so I’m avoiding that hornet’s nest. Agreed L3HW routing should be the same but I’ve found multiple CRS326 settings that differ like L3HW doesn’t MAC learn.

IMO Packet Flow in RouterOS covers switch chip paths but doesn’t always explicitly differentiate them; sometimes you have to learn more elsewhere before the diagrams become fully recognized. The Packet Flow in RouterOS page uses “Bridge Forward” and “Switch Forward” terms so I follow suit even though inter-VLAN input and output have substantial merit as well.

By the time routing occurs, CPU works with untagged packets and agreed, nothing under “/interface bridge” affects routing,
Beneath “/interface bridge port” is where PVID values are set.
Beneath “/interface bridge vlan” is where VLAN Table is built.

[mkx]

But the docs do say which implies a “hidden untag” within the packet flow diagram:

Tagged packets might get decapsulated on the “BRIDGING DECISION” block, which means these packets will no longer match the mac-protocol=vlan and vlan-encap settings. Decapsulation can happen if the packet’s VLAN ID matches the outgoing port’s untagged VLAN membership

.

Not really. This case only describes some specifics of bridging (or switching) between bridge ports inside same VLAN … what the note says is that if egress port belongs to same VLAN as ingress port and ingress frame is not tagged, then VLAN header won’t be attached (or it will even be stripped on ingress port if ingress frame is tagged). And that’s only true when egress port is untagged.
Which might explain how bridge/switch might save a few CPU cycles on those particular frames, but doesn’t change the fact that inter-VLAN routing is done the way I explained. Additionally, this “saving a few cycles” adds confusion because, as note says, some frames might have different mac-protocol for no really apparent reason and thus making bridge filter rules correct quite more challenging.
Sometimes I wish developers didn’t get that smart with optimization and shortcuts because these can convert debugging into a nightmare.

@ConradPino: I think that MT is aiming to keep the basic logic even when employing HW offload (either L2 or L3). So it should not really matter where some function is executed, either in SW on CPU or in switc chip ASIC. So when studying packet flow, it should not be necessary to consider different underlying hardware. If there are differences in how the whole setup is applied to a particular packet, affecting the outcome (i.e. packet is not dropped if L3HW is enabled v.s. packet is dropped if dealt with in software) should clearly be documented. If things behave radically differently when toggling l3hw property and documentation doesn’t explain it, then this is clearly a bug and should be reported to MT.

[ConradPino]

@ConradPino: I think that MT is aiming to keep the basic logic even when employing HW offload (either L2 or L3). So it should not really matter where some function is executed, either in SW on CPU or in switc chip ASIC. So when studying packet flow, it should not be necessary to consider different underlying hardware. If there are differences in how the whole setup is applied to a particular packet, affecting the outcome (i.e. packet is not dropped if L3HW is enabled v.s. packet is dropped if dealt with in software) should clearly be documented. If things behave radically differently when toggling l3hw property and documentation doesn’t explain it, then this is clearly a bug and should be reported to MT.

MAC Learning implied disable occurs in Bridging and Switching # Fast Forward and MAC Learning implications are covered elsewhere; the documentation is where I made the discovery; IMO MikroTik knows. A different issue is how much repetition across related pages like the packet flow page actually helps or distracts which is target audience skill level specific.

[granjow]

Your comments were again really helpful, also the hint about IPv4 being skipped due to the 0x8100 Ethertype.

In the meantime I have spent some time on reading about iptables internals, which definitely helped a lot for my understanding of firewalls and routing.

Naturally, this does not prevent me from getting stuck again. I now have a router which delivers two VLANs over a trunk port (10 and 99), and I connect one trunk port to ether1 of the ax³ access point. Then I configure the

# Starting point is CAP mode, defcon (all interfaces bridged)

# Want ether4/5 to be access ports
/interface/bridge/port remove numbers=3,4
/interface/bridge/port add bridge=bridgeLocal interface=ether4 pvid=10
/interface/bridge/port add bridge=bridgeLocal interface=ether5 pvid=10

# Configure Bridge VLAN table: ether1-3 = TRUNK, ether4-5 = access to VLAN10
/interface/bridge/vlan/add bridge=bridgeLocal tagged=ether1,ether2,ether3 untagged=ether4,ether5,wlan1 vlan-ids=10
/interface/bridge/vlan/add bridge=bridgeLocal tagged=ether1,ether2,ether3 vlan-ids=99

# Create the L3 VLAN interfaces
/interface/vlan/add interface=bridgeLocal name=VLAN10 vlan-id=10
/interface/vlan/add interface=bridgeLocal name=VLAN99 vlan-id=99

# Need some static IPs (until now I can only use Winbox, and I don't like to dualboot to Windows)
/ip/address/add interface=bridgeLocal address=192.168.88.4/24
/ip/address/add interface=VLAN99 address=10.10.99.4/24

# Check if we can get a DHCP address (answer is: no)
/ip/dhcp-client/add interface=VLAN10 disabled=no
/ip/dhcp-client/add interface=VLAN99 disabled=no

# Enable VLAN Filtering
/interface/bridge/set bridgeLocal vlan-filtering=yes

Now almost everything works as it should. I can connect my laptop to a trunk port and (when I configure my laptop’s interface to use VLAN tags) I get a DHCP address from VLANs 10 and 99 from the router. I can connect it to one of the access ports and I get (without VLAN config on the laptop interface) a DHCP address in VLAN 10. I can also access the router over the 192.168.88.4/24 address from any port.

But I can not ping/access the router on 10.10.99.4/24 from the trunk port (with VLAN configured on the laptop). Also, the DHCP clients of the ax³ do not get an address.

I have no clue about the first point. This should work according to what I know so far. That should be exactly the INPUT case from the past few posts.

Regarding DHCP I assume the ax³ does not know where to get its DHCP address from because I have not configured a gateway. I actually also don’t know where I’d configure this correctly. A simple /ip/route/add dst-address=10.10.99.1/24 does not cut it.

[Amm0]

@granjow

But I can not ping/access the router on 10.10.99.4/24 from the trunk port (with VLAN configured on the laptop).

There are “interface list” in the default configurations. So if you add the [Layer3] VLAN interfaces as the “LAN” interface-list in /interface/list/members, inter-vlan routing should work. There is “drop !LAN” someplace in most firewall default/examples. Once it leave the bridge, without 0x8100 ether-type , the IPv4 hits the firewall filter rules. So this sounds like the firewall is blocking it…

@granjow

Also, the DHCP clients of the ax³ do not get an address.

Do you have an /ip/dhcp-server configured, or is there already one someplace else? If not, the “DHCP Setup” button in Winbox under IP>DHCP Server can create one on the [Layer3] VLAN interface and will generally do the right thing to create DHCP on a VLAN. It’s build-a-bear with the Mikrotik, so just creating a VLAN interface doesn’t enable DHCP on it.

By creating an IP address on the VLAN, that’s becomes the default gateway for the VLAN. And a “connected route” for the subnet is automatically added by virtue of the Mikrotik having an IP address on the interface – so shouldn’t need to mess with /ip/route for VLANs.

[granjow]

I first thought so too, but the CAP mode firewall is just empty except for one line which I added because I suspected the firewall blocking it, but this line should not even be necessary assuming the default policy of the input chain is ACCEPT.

/ip/firewall/export  
# jan/02/1970 00:04:56 by RouterOS 7.2.3
# software id = G5Z2-0M7F
#
# model = RBD53iG-5HacD2HnD
# serial number = …
# I added this line – did not change anything however
/ip firewall filter
add action=accept chain=input in-interface-list=all

Yes, I do have a DHCP server on the router, one for VLAN 10 and one for VLAN 99. I created it there on the VLANs.

/ip pool add name="pool_vlan10" ranges=10.10.10.160-10.10.10.199
/ip dhcp-server add name="dhcp_vlan10" interface=VLAN10 address-pool=pool_vlan10 disabled=no
/ip dhcp-server network add address=10.10.10.0/24 gateway=10.10.10.1

/ip pool add name="pool_base" ranges=10.10.99.160-10.10.99.199
/ip dhcp-server add name="dhcp_base" interface=BASE_VLAN address-pool=pool_base disabled=no
/ip dhcp-server network add address=10.10.99.0/24 gateway=10.10.99.1

This works fine when I connect my laptop to the trunk ports directly, I get an IP and internet access.

[anav]

Post your lastest router config and cap config and will have a look, i never look at snippets…

[granjow]

Thanks. So, here the router configuration I get with /export:

/interface bridge
add name=br-vlan protocol-mode=none vlan-filtering=yes
add admin-mac=64:D1:54:8D:B9:0F auto-mac=no comment=defconf name=bridge
/interface vlan
add interface=br-vlan name=BASE_VLAN vlan-id=99
add interface=br-vlan name=VLAN10 vlan-id=10
/caps-man configuration
add channel.band=2ghz-g/n .frequency=2412 country=switzerland datapath.bridge=br-vlan .local-forwarding=yes .vlan-id=10 .vlan-mode=no-tag installation=any mode=ap name=AP10 security.authentication-types=\
    wpa2-psk ssid=Game
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=BASE
add name=VLAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=pool_vlan10 ranges=10.10.10.160-10.10.10.199
add name=pool_base ranges=10.10.99.160-10.10.99.199
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
add address-pool=pool_vlan10 interface=VLAN10 name=dhcp_vlan10
add address-pool=pool_base interface=BASE_VLAN name=dhcp_base
/port
set 0 name=serial0
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled master-configuration=AP10
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=br-vlan frame-types=admit-only-vlan-tagged interface=ether3
add bridge=br-vlan frame-types=admit-only-vlan-tagged interface=ether4
add bridge=br-vlan frame-types=admit-only-vlan-tagged interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface bridge vlan
add bridge=br-vlan tagged=br-vlan,ether3,ether4,ether5 vlan-ids=10
add bridge=br-vlan tagged=br-vlan,ether3,ether4,ether5 vlan-ids=99
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=BASE_VLAN list=BASE
add interface=VLAN10 list=VLAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=10.10.99.1/24 interface=BASE_VLAN network=10.10.99.0
add address=10.10.10.1/24 interface=VLAN10 network=10.10.10.0
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server network
add address=10.10.10.0/24 gateway=10.10.10.1
add address=10.10.99.0/24 gateway=10.10.99.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="Allow management access" in-interface-list=BASE
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat out-interface-list=WAN
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/system clock
set time-zone-name=Europe/Zurich
/system identity
set name=hEX
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

And the ax³:

/interface bridge
add admin-mac=18:FD:74:39:29:1C auto-mac=no comment=defconf name=bridgeLocal vlan-filtering=yes
/interface wireless
# managed by CAPsMAN
set [ find default-name=wlan1 ] ssid=MikroTik
# managed by CAPsMAN
set [ find default-name=wlan2 ] ssid=MikroTik
/interface vlan
add interface=bridgeLocal name=VLAN10 vlan-id=10
add interface=bridgeLocal name=VLAN99 vlan-id=99
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridgeLocal comment=defconf interface=ether1
add bridge=bridgeLocal comment=defconf interface=ether2
add bridge=bridgeLocal comment=defconf interface=ether3
add bridge=bridgeLocal interface=ether4 pvid=10
add bridge=bridgeLocal interface=ether5 pvid=10
add bridge=bridgeLocal interface=wlan1 pvid=10
add bridge=bridgeLocal interface=wlan2 pvid=10
/interface bridge vlan
add bridge=bridgeLocal tagged=ether1,ether2,ether3 untagged=ether4,ether5,wlan1 vlan-ids=10
add bridge=bridgeLocal tagged=ether1,ether2,ether3 vlan-ids=99
/interface wireless cap
set bridge=bridgeLocal discovery-interfaces=bridgeLocal enabled=yes interfaces=wlan1,wlan2
/ip address
add address=10.10.99.4/24 interface=VLAN99 network=10.10.99.0
add address=192.168.88.4/24 interface=bridgeLocal network=192.168.88.0
/ip dhcp-client
add comment=defconf interface=bridgeLocal
add interface=VLAN10
add interface=VLAN99
/ip firewall filter
add action=accept chain=input in-interface-list=all
/ip route
add dst-address=10.10.99.1/24
/system identity
set name=ax3
/system logging
add topics=bridge,debug,firewall,interface,packet,raw,route

[anav]

R1

(1) Two bridges are for kids, ONE Bridge only!!!

/interface bridge
add name=br-vlan protocol-mode=none vlan-filtering=yes

(2) I dont mix apples and oranges, adds a layer of confusion. Just have the bridge bridge, no effing DHCP nonsense…

/interface vlan
add interface=br-vlan name=BASE_VLAN vlan-id=99
add interface=br-vlan name=VLAN10 vlan-id=10
add interfac=br-vlan name=vlanHOME vlan-id=5
/ip dhcp-server
add address-pool=default-dhcp interface=vlanHOME name=defconf
add address-pool=pool_vlan10 interface=VLAN10 name=dhcp_vlan10
add address-pool=pool_base interface=BASE_VLAN name=dhcp_base
/interface bridge port
add bridge=br-vlan frame-types=admit-priority-and-untagged ingress-filtering=yes interface=ether2 pvid=5
add bridge=br-vlan frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether3
add bridge=br-vlan frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether4
add bridge=br-vlan frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether5

/interface bridge vlan
add bridge=br-vlan tagged=br-vlan untagged=ether2 vlan-ids=5
add bridge=br-vlan tagged=br-vlan,ether3,ether4,ether5 vlan-ids=10
add bridge=br-vlan tagged=br-vlan,ether3,ether4,ether5 vlan-ids=99

/ip address
add address=192.168.88.1/24 comment=defconf interface=vlanHOME network=192.168.88.0
add address=10.10.99.1/24 interface=BASE_VLAN network=10.10.99.0
add address=10.10.10.1/24 interface=VLAN10 network=10.10.10.0

(3) Your interface does not require so many interfaces, simplify and apply BASE to correct entities…
/interface list member
add comment=defconf interface=ether1 list=WAN
add interface=vlanHOME list=LAN
add interface=VLAN10 list=LAN
add interface=BASE_VLAN list=LAN
add interface=BASE_VLAN list=BASE__
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=BASE
/ip neighbor discovery-settings
set discover-interface-list=BASE

/tool mac-server
set allowed-interface-list=NONE { not a secure method of access so set to NONE }
/tool mac-server mac-winbox
set allowed-interface-list=BASE

(4) Do you see the sillyness of this pairing…
add action=accept chain=input comment=“Allow management access” in-interface-list=BASE
add action=drop chain=input comment=“defconf: drop all not coming from LAN” in-interface-list=!LAN

WHY have the BASE as separate if you let every tom dick and harry access the router in the next line???
SHOULD BE…
add action=accept chain=input comment=“Allow management access” in-interface-list=BASE
add action=accept chain=input comment=“user access to services” in-interface-list=LAN dst-port=53 protocol=tcp
add action=accept chain=input comment=“user access to services” in-interface-list=LAN dst-port=53 protocol=udp
add action=drop chain=input comment="drop all else" { make sure this is the last rule implemented }

(5) Similarly modify this rule in the forward chain for clarity and better control…
add action=drop chain=forward comment=“defconf: drop all from WAN not DSTNATed” connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
TO:
add action=accept chain=forward comment=“internet traffic” in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment=“port forwarding” connection-nat-state=dstnat { disable if not required }
add action=drop chain=forward comment=“drop all else”

Note: If you need access to shared device on a subnet, or if you need access from one vlan to the other, then place an allow traffic rule prior to the drop rule.

(6) Why do you have duplicate rule, just remove the second one…
/ip firewall nat
add action=masquerade chain=srcnat comment=“defconf: masquerade” ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat out-interface-list=WAN

[anav]

ax3 - Assuming its acting as an AP / SWITCH and not a router!!!


(1) One only defines the Base VLAN.
add interface=bridgeLocal name=VLAN99 vlan-id=99

(2) Adjusted…
/interface bridge port
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-priority-and-untagged interface=ether4 pvid=10
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-priority-and-untagged interface=ether5 pvid=10
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-priority-and-untagged interface=wlan1 pvid=10
add bridge=bridgeLocal ingress-filtering=yes frame-types=admit-priority-and-untagged interface=wlan2 pvid=10

(3) Missing WLAN2 on first line!
/interface bridge vlan
add bridge=bridgeLocal tagged=ether1,ether2,ether3 untagged=ether4,ether5,wlan1,wlan2 vlan-ids=10
add bridge=bridgeLocal tagged=ether1,ether2,ether3 vlan-ids=99

(4) Remove address of Bridge… not relevant.
The fixed static IP address of the AXE on the BASE VLAN is necessary.

(5) REMOVE IP DHCP CLIENT ( all the entries are bogus), for two reasons, it has nothing to do with BRIDGE and is not required in this setup, NO ROUTER function, and its IP is fixed

(6) REMOVE the filter rule… ( no firewall rules required )

(7) ADD the following…

/interface list
add name=MANAGE

/interface list members
add interface=VLAN99 list=MANAGE

/ip neighbor discovery-settings
set discover-interface-list=MANAGE

/ip dns
set allow-remote-requests=yes servers=10.10.99.1 comment=“dns through trusted subnet gateway”

/ip route (change the one you have to this)
From:
add dst-address=10.10.99.1/24
TO:
add disabled=no dst-address=0.0.0.0/0 gateway=10.10.99.1 comment=“ensures route avail through trusted subnet gateway”

/tool mac-server mac-winbox
set allowed-interface-list=MANAGE

[granjow]

Thanks anav for all the valuable input, I’m learning a lot of good practices! Going through the configuration changes step by step, it becomes much cleaner and it makes more sense to me.

Yes, the ax³ should act as a switch, not as a router. I will go through that part of the config another day as it is already late here.

On the bridge, does it matter which protocol-mode I choose or can i leave it to the rstp default?

[anav]

The only thing I change on the bridge is when I turn on vlan filtering as a last step.

On my router I typically take one port and put if off the bridge for emergency access… and thus if I screw up on a bridge setting I am not affected…

[granjow]

Ah. So the emergency port directly has an IP address, e.g. /ip address add interface=ether2 address…?

I continued with the ac³ (not ax³ … unfortunately ), and when I got to CAPs mode, stuff stopped working. First, it created hidden SSIDs. I then enabled caps logging and found that:

 19:16:56 caps,debug CAP Select->Sulking
 19:17:01 caps,debug CAP Sulking->Discover
 19:17:01 caps,debug CAP discovery target list:
 19:17:03 caps,debug CAP discovery over, no results
 19:17:03 caps,debug CAP Discover->Select
 19:17:03 caps,debug CAP did not find suitable CAPsMAN

So I set the /interface/wireless/cap discovery-interface to vlanBase, and it found the CAPsMAN, but still had a hidden SSID.

Then I found out that I still had VLAN filtering disabled on the router. I enabled it, and there goes connectivity. When I connect to a ac³ trunk port, I can ping its address, but pinging the router has 90 % packet loss. When I connect to the router trunk port, pinging ac³ has huge packet loss.

If I add two switches (RB260GS) in-between, i.e. Router → Sw1 → Sw2 → ac³, and I get the same results. From Router, I can only work the router reliably. From Sw1, Sw2, or ac³, I can reach all devices except for the router reliably. When I disable vlan filtering, ping works fine again in both directions.

That’s confusing because it is not all-or-nothing. And also because I lose packets in the first place.

Here is the router configuration I am using now:

/interface bridge
add admin-mac=64:D1:54:8D:B9:0F auto-mac=no comment=defconf name=bridge vlan-filtering=yes

/interface vlan
add interface=bridge name=vlan10 vlan-id=10
add interface=bridge name=vlanBase vlan-id=99
add interface=bridge name=vlanHome vlan-id=5

/caps-man configuration
add channel.band=2ghz-g/n .frequency=2412 country=switzerland datapath.bridge=bridge .local-forwarding=yes .vlan-id=10 .vlan-mode=no-tag hide-ssid=no installation=indoor mode=ap name=AP10 \
    security.authentication-types=wpa2-psk ssid=Game

/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=BASE

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/ip hotspot profile
set [ find default=yes ] html-directory=hotspot

/ip pool
add name=pool_home ranges=192.168.88.160-192.168.88.254
add name=pool_vlan10 ranges=10.10.10.160-10.10.10.199
add name=pool_base ranges=10.10.99.160-10.10.99.199

/ip dhcp-server
add address-pool=pool_home interface=vlanHome name=dhcp_home
add address-pool=pool_vlan10 interface=vlan10 name=dhcp_vlan10
add address-pool=pool_base interface=vlanBase name=dhcp_base

/port
set 0 name=serial0

/caps-man manager
set enabled=yes

/caps-man provisioning
add action=create-dynamic-enabled master-configuration=AP10

/interface bridge port
add bridge=bridge comment=defconf frame-types=admit-only-untagged-and-priority-tagged interface=ether2 pvid=5
add bridge=bridge frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge frame-types=admit-only-vlan-tagged interface=ether4
add bridge=bridge frame-types=admit-only-vlan-tagged interface=ether5

/ip neighbor discovery-settings
set discover-interface-list=BASE

/interface bridge vlan
add bridge=bridge tagged=bridge,ether3,ether4,ether5 vlan-ids=10
add bridge=bridge tagged=bridge,ether3,ether4,ether5 vlan-ids=99
add bridge=bridge tagged=bridge untagged=ether2 vlan-ids=5

/interface list member
add comment=defconf interface=ether1 list=WAN
add interface=vlan10 list=LAN
add interface=vlanBase list=LAN
add interface=vlanHome list=LAN
add interface=vlanBase list=BASE

/ip address
add address=10.10.99.1/24 interface=vlanBase network=10.10.99.0
add address=10.10.10.1/24 interface=vlan10 network=10.10.10.0
add address=192.168.99.1/24 interface=vlanHome network=192.168.99.0

/ip dhcp-client
add comment=defconf interface=ether1

/ip dhcp-server network
add address=10.10.10.0/24 gateway=10.10.10.1
add address=10.10.99.0/24 gateway=10.10.99.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1

/ip dns
set allow-remote-requests=yes

/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="Allow management access" in-interface-list=BASE
add action=accept chain=input comment="Users: Services access" dst-port=53 in-interface-list=LAN protocol=tcp
add action=accept chain=input comment="Users: Services access" dst-port=53 in-interface-list=LAN protocol=udp
add action=drop chain=input comment="Drop all other traffic"
add action=accept chain=forward comment="internet traffic" in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat disabled=yes
add action=drop chain=forward comment="drop all else"

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN

/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6

/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN

/system clock
set time-zone-name=Europe/Zurich

/system identity
set name=hEX

/tool mac-server
set allowed-interface-list=none

/tool mac-server mac-winbox
set allowed-interface-list=BASE

And the ac³ configuration: (With ether2 as emergency port if I got it right)

/interface bridge
add admin-mac=18:FD:74:39:29:1D auto-mac=no comment=defconf name=bridge vlan-filtering=yes

/interface wireless
# managed by CAPsMAN
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX distance=indoors frequency=auto installation=indoor mode=ap-bridge ssid=MikroTik-392921 wireless-protocol=802.11
# managed by CAPsMAN
set [ find default-name=wlan2 ] band=5ghz-a/n/ac channel-width=20/40/80mhz-XXXX distance=indoors frequency=auto installation=indoor mode=ap-bridge ssid=MikroTik-392922 wireless-protocol=802.11

/interface vlan
add interface=bridge name=vlanBase vlan-id=99

/interface list
add comment=defconf name=LAN
add name=MANAGE

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridge frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge frame-types=admit-only-vlan-tagged interface=ether4
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=ether5 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=wlan1 pvid=10
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged interface=wlan2 pvid=10

/ip neighbor discovery-settings
set discover-interface-list=MANAGE

/interface bridge vlan
add bridge=bridge tagged=bridge,ether1,ether3,ether4 untagged=ether5,wlan1,wlan2 vlan-ids=10
add bridge=bridge tagged=bridge,ether1,ether3,ether4 vlan-ids=99

/interface list member
add comment=defconf interface=bridge list=LAN
add interface=vlanBase list=MANAGE

/interface wireless cap
#
set bridge=bridge discovery-interfaces=vlanBase enabled=yes interfaces=wlan1,wlan2

/ip address
add address=192.168.99.1/24 interface=ether2 network=192.168.99.0
add address=10.10.99.4/24 interface=vlanBase network=10.10.99.0

/ip dns
set allow-remote-requests=yes servers=10.10.99.1

/ip route
add comment="ensures route avail through trusted subnet gateway" disabled=no dst-address=0.0.0.0/0 gateway=10.10.99.1

/system clock
set time-zone-name=Europe/Zurich

/system identity
set name=ac3

/tool mac-server
set allowed-interface-list=none

/tool mac-server mac-winbox
set allowed-interface-list=MANAGE

What am I doing wrong? Why is the router dropping most of the packages?

[anav]

Probably because I dont use capsman in any of my configs and it requires something non-standard???
I detest capsman.

[Amm0]

It looks like there is no configuration for 5Ghz on the capsman side.

One important note is that the “local-forwarding=yes” is a pretty critical setting here & what you’d want I think you’d want – otherwise boy your diagram gets confusing with local-forwarding=no since that uses tunnels of the raw frames to be processed centrally (see @anav’s commentary above).

But I recommend disabling that capsman for now, and get the wifi working directly on the Wi-Fi interface first – see if your still getting lost packets. Then if you want to use capsman add the same local configuration to the capsman profile, and troubleshoot that separately.

@Anav -
__

I detest capsman.

Just waiting for your hAPax3 diatribe on wifiwave2 configuration…

[granjow]

Okay. I disabled CAPsMAN on the router, and the problem persists.
I disconnect ac³ and the problem persists.
I connect the router to Sw2 (which is just an unconfigured RB260Gs, unlike Sw1 where I have configured trunk ports), 2 packets get through, and then same problem.

The switches have IPs 10.10.99.2 and 10.10.99.3. When I check traffic on my PC, I see that ICMP requests with VLAN tag 99 are sent to 10.10.99.2/24, and just sometimes answered.

I disable all firewall rules, 2 packets get through, and again same problem.
I do /interface/bridge/port set frame-types=admit-all [ find where interface=ether4 ] (this goes to the switches), and all packets get through.

This gives me an idea. Is the switch maybe misconfigured so it sometimes responds to the ICMP with VLAN tag and sometimes without?
I connect my second laptop to ether5 on the router (all ports on admit-only-vlan-tagged again), it receives 10.10.99.198/24 from the router, and I can always ping it from my first laptop connected to ether3.
I give my second laptop a static IP 10.10.99.5/24 and I can also ping it.

However, I had the issue also without the switches.
I connect ac³ directly to the router on ether5 instead, and it is still unrechable.
I set ether5 to admit-all, and I can ping it.

So, I can ping my other laptop, but with the same settings I cannot ping another MikroTik device. The switches cannot be the issue because I see the same behaviour with and without them.
I set admit-all on the ac³ and admit-only-vlan-tagged on the router. ac³ is unreachable.
I set admit-only-vlan-tagged on ac³ and admit-all on the router. ac³ is rechable.
I connect the devices as follows: hEX → ac³ → Sw2 → Sw1. All are reachable when pinging from my laptop connected to the router on ether3. (note, still with admit-all on the router.)

What the??? What is wrong?

I’m on RouterOS 7.7 (firmware too) on both devices. The router is a hEX RB750Gr3.

(And, regarding CAPsMAN: Is there a better alternative when I have multiple APs?)

[granjow]

hEX] > /interface/bridge/host/print where !local
Flags: D - DYNAMIC; E - EXTERNAL
Columns: MAC-ADDRESS, VID, ON-INTERFACE, BRIDGE
#    MAC-ADDRESS        VID  ON-INTERFACE  BRIDGE
0 DE 18:FD:74:39:29:1C    1  ether5        bridge ← this is ether1 of ac³
1 DE 18:FD:74:39:29:1D   99  ether5        bridge ← this is ether2 of ac³ (which is not connected physically …) (edit: nevermind, probably a cached entry, had my other laptop connected on ac³s ether2  previously)

Why does VLAN learning learn that ac³ has VID 1? Or is this unrelated?

[granjow]

Using /tool/sniffer quick ip-protocol=icmp and running ping against 10.10.99.1-4, I see ICMP traffic between only 10.10.99.1 (the router) and my laptop. It is using a VLAN tag on bridge and ether4, and no tag von vlanBase, which makes sense. I cannot see any other ICMP traffic, even when I disable VLAN filtering and 10.10.99.2-4 are reachable too. Why?

[granjow]

New day, new luck.

I power on the network setup, and it behaves exactly the same way as yesterday. Finally, consistency. Thanks.

As I just found the configuration of the existing network setup (which is at a different place, but uses VLANs as well), I started comparing the lines. The only difference I found was that protocol-mode was set to none. So I
/interface/bridge set bridge protocol-mode=none
and shit works. With VLAN filtering enabled.

/interface/bridge/port set edge=yes [ find where interface=ether5 ]
has the same effect, as it disable STP learning on that specific port (that is where I have ac³ attached atm).

The L2 Misconfiguration Guide says to set protocol-mode=none when bridging VLAN interface and suggests to use VLAN filtering instead, but that’s what I’m doing here.

Well, it seems like I have to learn a bit about the STPs now. Because:

for larger networks properly configured STP is very crucial, leaving STP-related values to default may result in a completely unreachable network in case of an even single bridge failure. To achieve a proper loop-free and redundant topology, it is necessary to properly set bridge priorities, port path costs, and port priorities.

[Amm0]

If it touches other networks, understand something about STP is helpful for sure. But I’d almost say shouldn’t matter if you don’t have loops in your switched network, since M/R/STP only triggers to resolve loops.

But that’s not always true since some smart/managed switches (e.g. Cisco) sometimes default to “blocking” a port if it sees STP traffic on something defined as an “access port”. This feature is called “bpdu guard”, which RouterOS supports but on Mikrotik it defaults off. Other equipment sometimes has this feature enabled by default. I suspect this may be happening if setting =none helps and things work.