CAPsMAN on layer2 + vlans

nagylzs · June 17, 2021, 5:37pm

I’m trying to use CAPsMAN on a network that has vlans. In the first step, I wanted to do something very simple: add CAPsMAN and CAP on the same (main) router just to see if it works. I have used CAPsMAN before with success, but I have never used it on a vlan filtered bridge.

The caps-man is assigned to this BR1 bridge:

/interface bridge
add frame-types=admit-only-vlan-tagged ingress-filtering=yes name=BR1 vlan-filtering=yes
/caps-man manager interface
set [ find default=yes ] forbid=yes
add disabled=no interface=BR1

The same bridge is assigned to CAP:

/interface wireless cap
set bridge=BR1 certificate=request discovery-interfaces=BR1 enabled=yes interfaces=wlan1,wlan2

I want this to work at layer2, so caps-addresses is not given. Since this should work at layer 2, I believe that at least the CAP should find the CAPsMAN, regardless of the firewall filter rules.

But they can’t find each other. I see these in the logs repeating over and over:

19:09:24 caps,debug CAP Sulking->Discover
19:09:24 caps,debug CAP discovery target list:
19:09:27 caps,debug CAP discovery over, no results
19:09:27 caps,debug CAP Discover->Select
19:09:27 caps,debug CAP did not find suitable CAPsMAN
19:09:27 caps,debug CAP Select->Sulking
19:09:32 caps,debug CAP Sulking->Discover
19:09:32 caps,debug CAP discovery target list:
19:09:34 caps,debug CAP discovery over, no results
19:09:34 caps,debug CAP Discover->Select
19:09:34 caps,debug CAP did not find suitable CAPsMAN
19:09:34 caps,debug CAP Select->Sulking

I have read the https://wiki.mikrotik.com/wiki/Manual:CAPsMAN_with_VLANs wiki at least three times, but I can’t find the error.

In that wiki, there is a note:

In this example untagged traffic is going to be used to communicate between CAPs and CAPsMAN Router.

I suspect that the problem lies there. This note foreshadows that there might be a way to let them communicate with tagged traffic. But I don’t see how. I don’t see a way to configure /interface wireless cap to use a specific vlan tag, and also nothing about vlans under /caps-man manager interface menu

I’m also posting most of the config below, for reference.

# model = RBD52G-5HacD2HnD
/caps-man channel
add band=2ghz-onlyn extension-channel=XX frequency=2412,2432,2462 name=channels-2.4 secondary-frequency=\
    2412,2432,2462 tx-power=-10
add band=5ghz-onlyac extension-channel=XXXX frequency=5170,5190,5210,5230,5755,5775,5795 name=channels-5 \
    secondary-frequency=5170,5190,5210,5230,5755,5775,5795
/caps-man datapath
add local-forwarding=yes name=datapath-blue vlan-id=10 vlan-mode=use-tag
add local-forwarding=yes name=datapath-green vlan-id=20 vlan-mode=use-tag
add local-forwarding=yes name=datapath-green vlan-id=30 vlan-mode=use-tag
/interface bridge
add frame-types=admit-only-vlan-tagged ingress-filtering=yes name=BR1 vlan-filtering=yes
add name=ipsec protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] name=ether1-trunk
set [ find default-name=ether2 ] name=ether2-blue
set [ find default-name=ether3 ] name=ether3-blue
set [ find default-name=ether4 ] name=ether4-blue
set [ find default-name=ether5 ] name=ether5-wan
/interface wireless
# managed by CAPsMAN
set [ find default-name=wlan1 ] ssid=MikroTik
# managed by CAPsMAN
set [ find default-name=wlan2 ] ssid=MikroTik
/interface vlan
add interface=BR1 name=BASE_VLAN vlan-id=99
add interface=BR1 name=BLUE_VLAN vlan-id=10
add interface=BR1 name=GREEN_VLAN vlan-id=20
add interface=BR1 name=RED_VLAN vlan-id=30
/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm name=security-blue passphrase=********
add authentication-types=wpa2-psk encryption=aes-ccm name=security-green passphrase=********
add authentication-types=wpa2-psk encryption=aes-ccm name=security-red passphrase=********
/caps-man configuration
add channel.band=5ghz-onlyac channel.extension-channel=XXXX country=hungary installation=indoor name=caps-blue-5 \
    security=security-blue ssid=blue_fast
add channel=channels-2.4 channel.band=2ghz-onlyn channel.extension-channel=XX country=hungary datapath=datapath-blue \
    installation=indoor name=caps-blue-2.4 security=security-blue ssid=blue
add channel=channels-2.4 channel.band=2ghz-onlyn channel.extension-channel=XX country=hungary datapath=datapath-green \
    installation=indoor name=caps-green-2.4 security=security-green ssid=green
add channel.band=5ghz-onlyac channel.extension-channel=XXXX country=hungary installation=indoor name=caps-green-5 \
    security=security-green ssid=green_fast
/interface list
add name=WAN
add name=VLAN
add name=BASE
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=BLUE_POOL ranges=10.19.10.100-10.19.10.200
add name=GREEN_POOL ranges=10.19.20.100-10.19.20.200
add name=RED_POOL ranges=10.19.30.100-10.19.30.200
add name=BASE_POOL ranges=192.168.19.100-192.168.19.200
/ip dhcp-server
add address-pool=BLUE_POOL disabled=no interface=BLUE_VLAN lease-script=onDhcpLease name=BLUE_DHCP
add address-pool=GREEN_POOL disabled=no interface=GREEN_VLAN name=GREEN_DHCP
add address-pool=RED_POOL disabled=no interface=RED_VLAN name=RED_DHCP
add address-pool=BASE_POOL disabled=no interface=BASE_VLAN name=BASE_DHCP
/caps-man manager
set ca-certificate=auto certificate=auto enabled=yes upgrade-policy=suggest-same-version
/caps-man manager interface
set [ find default=yes ] forbid=yes
add disabled=no interface=BR1
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=ac master-configuration=caps-blue-5 name-format=identity \
    slave-configurations=caps-green-5
add action=create-dynamic-enabled master-configuration=caps-blue-2.4 name-format=identity slave-configurations=\
    caps-green-2.4
/interface bridge port
add bridge=BR1 frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether1-trunk
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether2-blue pvid=\
    10
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether3-blue pvid=\
    10
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether4-blue pvid=\
    10
add bridge=BR1 interface=wlan1
add bridge=BR1 interface=wlan2
/ip neighbor discovery-settings
set discover-interface-list=BASE
/interface bridge vlan
add bridge=BR1 tagged=BR1,ether1-trunk vlan-ids=20,30,99
add bridge=BR1 tagged=BR1,ether1-trunk untagged=ether2-blue,ether3-blue,ether4-blue vlan-ids=10
/interface list member
add interface=ether5-wan list=WAN
add interface=BASE_VLAN list=VLAN
add interface=BLUE_VLAN list=VLAN
add interface=GREEN_VLAN list=VLAN
add interface=RED_VLAN list=VLAN
add interface=BASE_VLAN list=BASE
/interface wireless cap
set bridge=BR1 certificate=request discovery-interfaces=BR1 enabled=yes interfaces=wlan1,wlan2
/ip address
add address=192.168.19.254/24 interface=BASE_VLAN network=192.168.19.0
add address=10.19.10.1/24 interface=BLUE_VLAN network=10.19.10.0
add address=10.19.20.1/24 interface=GREEN_VLAN network=10.19.20.0
add address=10.19.30.1/24 interface=RED_VLAN network=10.19.30.0
/ip dhcp-client
add disabled=no interface=ether5-wan use-peer-dns=no
/ip dhcp-server network
add address=10.19.10.0/24 dns-server=192.168.19.254 domain=magnet. gateway=10.19.10.1
add address=10.19.20.0/24 dns-server=192.168.19.254 domain=magnetpub. gateway=10.19.20.1
add address=10.19.30.0/24 dns-server=192.168.19.254 gateway=10.19.30.1
add address=192.168.19.0/24 dns-server=192.168.19.254 gateway=192.168.19.254
/ip dns
set allow-remote-requests=yes servers=1.1.1.3,1.0.0.3
/ip firewall nat
add action=masquerade chain=srcnat comment="Default masquerade" out-interface-list=WAN
/system logging
add topics=caps

nagylzs · June 17, 2021, 7:43pm

After removing all firewall rules, discovery succeeded:

19:24:14 caps,debug CAP Sulking->Discover
19:24:14 caps,debug CAP discovery target list:
19:24:16 caps,debug CAP discovery over, results:
19:24:16 caps,debug   router.magnet (::ffff:192.168.19.254:5246)
19:24:16 caps,debug CAP Discover->Select
19:24:16 caps,info CAP selected CAPsMAN router.magnet (::ffff:192.168.19.254:5246)
19:24:16 caps,debug CAP Select->Connect
19:24:16 caps,info CAP connected to router.magnet (::ffff:192.168.19.254:5246), CommonName 'CAPsMAN-085531E7F367'
19:24:16 caps,debug CAP Connect->Join
19:24:17 certificate,info generated certificate 1D15920EEDA97622:CAP-085531E7F367::::: key-size:2048 key-curve:0 usage:d valid:24854 for CA CAPsMAN-CA-085531E7F367
19:24:17 caps,info [::ffff:192.168.19.254:34887,IssueCert,CAP-085531E7F367] joined, provides radio(s): 08:55:31:E7:F3:6C,08:55:31:E7:F3:6D

But this can’t be good, because only IP based connection is established, not MAC based. Right?

biomesh · June 17, 2021, 8:08pm

Set the capsman discovery interface (on both the cap and capsman config) to the VLAN interface where you want the caps to capsman communication to happen.

If this was not on the same device, then the cap config might need to be configured differently, depending on your config (cap and switch)

nagylzs · June 17, 2021, 8:31pm

Yes, already tried this:

/caps-man manager interface
set [ find default=yes ] forbid=yes
add disabled=no interface=BASE_VLAN
/interface wireless cap
set bridge=BR1 certificate=request discovery-interfaces=BASE_VLAN enabled=yes interfaces=wlan1,wlan2

but it does not work either. Same problem: CAP cannot find CAPsMAN.

It is the same device. (I want to add more CAPs on other routers later, but I wanted to try this on the main router first.)

mkx · June 17, 2021, 8:33pm

As @biomesh wrote, the trick is to set discovery interface to some vlan interface. For example, I have VLAN 42 intended for usual LAN traffic and I allow CAP to CAPsMAN communication via that VLAN.

So on CAP device I have the following:

/interface bridge
add name=bridge vlan-filtering=yes
/interface vlan
add interface=bridge name=vlan-42 vlan-id=42
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether1
/interface bridge vlan
add bridge=bridge tagged=bridge,ether1 vlan-ids=42
add bridge=bridge tagged=ether1 vlan-ids=40,41
/interface wireless cap
set bridge=bridge discovery-interfaces=vlan-42 enabled=yes interfaces=wlan1  # it's a single-radio device

And on CAPsMAN (which is also router for all LANs) I have

/interface bridge
add name=bridge vlan-filtering=yes
/interface vlan
add interface=bridge name=vlan-40 vlan-id=40
add interface=bridge name=vlan-41 vlan-id=41
add interface=bridge name=vlan-42 vlan-id=42
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=ether5
/interface bridge vlan
add bridge=bridge  tagged=bridge,ether5 vlan-ids=40,41,42
/caps-man manager interface
add disabled=no interface=vlan-42

Of course CAPsMAN needs much more configuration in /caps-man configuration branch with all the provisioning etc. than shown above.

The point in configuring ether1 as trunk port for multiple VLAN IDs on CAP device while bridge is only tagged member of one of VLANs is that I’m using local forwarding set in capsman datapath hence VLANs for different SSIDs have to be configured on bridge on CAP device (remember: CAPsMAN only provisions wireless interfaces on CAP devices, nothing more). As CAP device won’t directly interact with those VLANs bridge interface doesn’t have to be member of those VLANs, CAPsMAN provisioned wireless devices (master and slaves) will become members of those VLANs when added to bridge.

biomesh · June 17, 2021, 8:53pm

Might want to take a look here:

https://wiki.mikrotik.com/wiki/Manual:Simple_CAPsMAN_setup#CAP_in_CAPsMAN

nagylzs · June 17, 2021, 9:09pm

All right, I’ll try this tomorrow. But it would be very surprising if that was the problem. Ip firewall can only block ip packets, right? Ip firewall rules should only matter when using ip based communication. Or maybe I’m not seeing something. Maybe layer 2 capsman-cap connection cannot be used when capsman and cap are on the same device?

nagylzs · June 18, 2021, 4:19am

After adding this rule:

/ip firewall filter
add action=accept chain=input dst-address-type=local src-address-type=local

It works! Here is the log:

06:06:37 caps,debug CAP None->Discover
06:06:37 caps,debug CAP discovery target list:
06:06:39 caps,debug CAP discovery over, results:
06:06:39 caps,debug   router.magnet (::ffff:192.168.19.254:5246)
06:06:39 caps,debug CAP Discover->Select
06:06:39 caps,info CAP selected CAPsMAN router.magnet (::ffff:192.168.19.254:5246)
06:06:39 caps,debug CAP Select->Connect
06:06:40 caps,info CAP connected to router.magnet (::ffff:192.168.19.254:5246), CommonName 'CAPsMAN-085531E7F367'
06:06:40 caps,debug CAP Connect->Join
06:06:40 caps,info [::ffff:192.168.19.254:35711,Join,CAP-085531E7F367] joined, provides radio(s): 08:55:31:E7:F3:6C,08:55:31:E7:F3:6D
06:06:40 caps,info CAP joined router.magnet (::ffff:192.168.19.254:5246)
06:06:40 caps,debug CAP Join->Joined

But I’m still missing something. There is this section in the docs: https://wiki.mikrotik.com/wiki/Manual:CAPsMAN#CAP_to_CAPsMAN_Connection it says the following:

CAP to CAPsMAN connection can be established using 2 transport protocols (via Layer 2 and Layer3).

MAC layer connection features:

no IP configuration necessary on CAP

CAP and CAPsMAN must be on the same Layer 2 segment - either physical or virtual (by means of L2 tunnels)
IP layer (UDP) connection features:

can traverse NAT if necessary

CAP must be able to reach CAPsMAN using IP protocol

if the CAP is not on the same L2 segment as CAPsMAN, it must be provisioned with the CAPsMAN IP address, because IP multicast based discovery does not work over Layer3

From this description, I had the idea that in L2 mode, no IP packets are used, the CAP and the CAPsMAN will find each other using layer 2 broadcast messages. It is specifically mentioned that IP layer connection uses UDP, one would assume that a so called L2 mode does not use IP packets.

But it is not the case. I can see that layer 3 ip firewall configuration is required even in L2 mode. I think that CAP to CAPsMAN connection is using IP packets. But, what protocol? Which port number? And why is it not documented anywhere? (Or maybe it is documented, just I can’t find it?)

It also means that I won’t be able to use a RouterOs device as a CAP, unless it has an IP address? This is also not obvious, and I did not see this requirement documented anywhere. (If the CAP-CAPsMAN connection is using IP packets, then there must be a source and a destination IP address? But then why is it called a layer 2 connection?)

Another thing that just adds to the confusion is the fact that there is a whole section ( https://wiki.mikrotik.com/wiki/Manual:Simple_CAPsMAN_setup#CAP_in_CAPsMAN ) for adding special ip firewall rules when the CAP and the CAPsMAN are on the same device. But I could not find anything about adding firewall rules when they are on different devices. Maybe we don’t need to add allow IP traffic between them in the later case? I doubt. But what kind of traffic? Again: protocol number, port number etc. are undocumented.

nagylzs · June 18, 2021, 5:47am

Okay, connection between CAP and CAPsMAN works on all devices now. But there is something wrong with the bridge vlan tables.

If I add vlan-mode=use-tag in the datapath:

/caps-man datapath
add local-forwarding=yes name=datapath-blue vlan-id=10 vlan-mode=use-tag
add local-forwarding=yes name=datapath-green vlan-id=20 vlan-mode=use-tag

Then this happens:

[gandalf@router.magnet] /interface bridge vlan> print
Flags: X - disabled, D - dynamic
 #   BRIDGE                       VLAN-IDS  CURRENT-TAGGED                       CURRENT-UNTAGGED
 0   BR1                          10        BR1
                                            ether1-trunk
                                            wlan2
                                            wlan1
 1   BR1                          20        BR1
                                            ether1-trunk
                                            wlan24
                                            wlan25
 2   BR1                          30        BR1
                                            ether1-trunk
 3   BR1                          99        BR1
                                            ether1-trunk

The wlan1, wlan2, wlan24, wlan25 devices are added under the correct vlan id, but they are added as tagged ports. I would like them to be untagged. (Otherwise dumb WiFi clients won’t be able to connect.)

Another interesting (confusing???) fact is that these wlan devices are added dynamically into a static vlan entry. So the entry is half-static half-dynamic, but this information cannot be obtained, unless you export the vlan table and compare that with the printout manually. (There is no “D” flag in the printout above.) There is a separate line for each interface+vlan pair, so there could be a separate D flag for them.

If I change the datapath to use “no-tag”:

/caps-man datapath
add local-forwarding=yes name=datapath-blue vlan-id=10 vlan-mode=no-tag
add local-forwarding=yes name=datapath-green vlan-id=20 vlan-mode=no-tag

Then this happens:

Flags: X - disabled, D - dynamic
 #   BRIDGE                       VLAN-IDS  CURRENT-TAGGED                       CURRENT-UNTAGGED
 0   BR1                          10        BR1
                                            ether1-trunk
 1   BR1                          20        BR1
                                            ether1-trunk
 2   BR1                          30        BR1
                                            ether1-trunk
 3   BR1                          99        BR1
                                            ether1-trunk
 4 D BR1                          1                                              wlan2
                                                                                 wlan26
                                                                                 wlan1
                                                                                 wlan27

They are now untagged, but they are also in the wrong vlan-id=1.

How can I make them untagged in the correct vlan? E.g. wlan1 and wlan2 as untagged with vlan-ids=10, wlan26 and wlan27 as untagged with vlan-ids=20.

mkx · June 18, 2021, 8:03am

That’s correct and won’t cause any problem … wlan interfaces are tagged from bridge point of view. However, as wlan interfaces are provisioned with vlan-id=XX vlan-mode=use-tag, wireless driver will do the tagging in direction wireless → bridge and untagging in the opposite direction.

If you would want to pass tagged frames via wireless to wireless client, then setup would be slightly different: wlan interface would still be tagged member of appropriate VLANs in bridge configuration, but wireless interface would be configured without VLAN configuration. This way wireless driver would not touch VLAN tags neither on ingress nor egress. I guess such configuration is not possible via CAPsMAN provisioning though.

This does seem slightly confusing indeed. Entries were configured by a ROS process (capsman client or whatever it’s called) which would otherwise mean they are dynamic. In fact they are dynamic because they are configured run-time and should not be saved (neither in system backup nor exported config).

nagylzs · June 18, 2021, 8:45am

Well, then there is something else wrong with my config because I can obtain IP addres on ether2-blue, but I can’t do it on blue ssid.

The bridge does the tagging/untagging for every interface in the vlan table - or so I tought. But why is it different for CAP wlan interfaces? It also means that if you see an interface in the “tagged” column, then you can’t tell if it is actually sending tagged or untagged egress packets. Because it may send out tagged on ethernet interfaces, it may send out untagged for CAP managed wlan interfaces, but for regular wlan interfaces maybe it won’t. Simply you can’t tell without looking at the interface type, and possibly the CAP provisioning rules. Basically, you can’t trust what you see in the bridge’s vlan table. This is very counter-intuitive!

Well yes, they are not exported. So if you compare the output of “export” and “print”, then you can find the dynamic entries. But then again: the “print” command has a “D” flag legend/description, but sometimes the flag it is missing from the output.

nagylzs · June 18, 2021, 12:01pm

Rebooted them and they started to work magically.

Everything is fine, I’m a happy camper now. I just need to digest these strangenesses in RouterOs.

Thank you for your help, again!

anav · June 18, 2021, 11:57pm

I avoid all that by simply configuring my capacs as normal access points. Also works great, didnt lose sleep time or hair and my wifi will not crash if capsman burps

mkx · June 19, 2021, 10:49am

The bridge does tagging/unragging for ports which are untagged members of VLANs. Bridge does nothing on trunk ports (ports that are tagged members of VLANs). With wlan interfaces they can either be tagged or untagged. When provisioning with VLANs via CAPsMAN tgey eill be tagged (and bridge won’t do anything about tags). When configuring wlan interfaces manually locally on devices themselves, you can configure wkan interfaces either way (so you can move tagging/untagging to bridge).

You can picture wireless interface as a sort of a ethernet switch, with multiple clients connected. That switch can be “dumb” and pases all traffic untagged - in this case bridge has to tag/untag traffic passing that port. Switch can also be managed and can do tagging/untagging on its own - in this case bridge doesn’t do any tagging/untagging, only needs to know which VLANs are allowed to pass that port. Likewise wireless driver can do tagging/untagging packets, either all of them (belonging to certain SSID) or selectively according to access list settings (you can set VLAN ID per client MAC).

nagylzs · June 20, 2021, 5:02pm

Okay, this makes sense now. So the wireless driver and the bridge can both do tagging/untagging. It is just an arbitrary decision of the CAPsMAN package to do this in the wireless driver.

I’m feeling smarter already.

mkx · June 20, 2021, 7:10pm

Actually it’s not an arbitrary decission … up till ROS version 6.41 bridge was not VLAN aware, hardware (or low level drivers) had to deal with VLAN tagging/untagging/filtering.