DHCP server on VLAN and 7.1beta3

leopardus2 · Thu Dec 31, 2020 12:33 am

Hello, I am trying to make DHCP server assign addresses to members of VLAN 166, but I am unable
to make it work. This is with OS 7.1beta3. My guess is I'm missing something obvious, so I would
appreciate if someone could have a look at my config below and tell me what I've done wrong...

Thanks in advance,
Rick

# dec/30/2020 23:26:07 by RouterOS 7.1beta3
# software id = G5ES-UNVF
#
# model = RB3011UiAS
# serial number = B8950C9801E6
/interface bridge
add admin-mac=00:00:00:00:00:01 auto-mac=no name=bridge1 vlan-filtering=yes
/interface wireguard
add listen-port=51820 mtu=1420 name=wireguard1
/interface vlan
add interface=bridge1 name=VLAN7 vlan-id=7
add interface=bridge1 name=VLAN166 vlan-id=166
/interface bonding
add mode=802.3ad name=bond1 slaves=ether5,ether9,ether10
/interface list
add name=WAN
add name=LAN
add name=VPN
/interface lte apn
set [ find default=yes ] apn=internet.it
add apn=internet.it ip-type=ipv4 name=windtre use-network-apn=no
/interface lte
set [ find ] allow-roaming=no apn-profiles=windtre name=lte1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool1 ranges=192.168.166.1-192.168.166.99
/ip dhcp-server
add address-pool=dhcp_pool1 authoritative=after-2sec-delay disabled=no interface=VLAN166 lease-time=23h name=dhcp1
/ip vrf
add list=all name=main
/port
set 1 name=usb2
/routing table
add fib name=vpn
/interface bridge port
add comment=defconf interface=ether1
add comment=defconf interface=ether2
add comment=defconf interface=ether3
add comment=defconf interface=ether4
add comment=defconf interface=ether5
add comment=defconf interface=ether6
add comment=defconf interface=ether7
add comment=defconf interface=ether8
add comment=defconf interface=ether9
add comment=defconf interface=ether10
add comment=defconf interface=sfp1
add bridge=bridge1 interface=bond1
/interface bridge vlan
add bridge=bridge1 tagged=bond1,bridge1 vlan-ids=1-4094
/interface list member
add interface=lte1 list=WAN
add comment=defconf interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add list=VPN
add interface=VLAN166 list=LAN
add interface=VLAN7 list=LAN
add interface=wireguard1 list=VPN
add list=WAN
/interface wireguard peers
add allowed-address=0.0.0.0/0 endpoint-address=1.2.3.4 endpoint-port=51820 interface=wireguard1 persistent-keepalive=25s public-key=\
    "redacted"
/ip address
add address=192.168.1.5/24 interface=ether1 network=192.168.1.0
add address=192.168.7.254/24 interface=VLAN7 network=192.168.7.0
add address=192.168.166.254/24 interface=VLAN166 network=192.168.166.0
add address=192.168.69.253/24 interface=wireguard1 network=192.168.69.0
/ip dhcp-client
add comment=defconf disabled=no
/ip dhcp-server network
add address=192.168.166.0/24 dns-server=8.8.8.8,8.8.8.4 gateway=192.168.166.254
/ip dns
set servers=8.8.8.8
/ip firewall address-list
add address=192.168.166.100-192.168.166.199 list=vpn
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="traffic from VLAN7" in-interface=VLAN7 protocol=tcp
add action=accept chain=forward comment="samba access" dst-address=192.168.7.13 dst-port=139,445 protocol=tcp src-address=192.168.166.243
add action=drop chain=forward comment="drop from VLAN166" in-interface=VLAN166 log=yes out-interface-list=LAN
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN log=yes
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=prerouting new-routing-mark=vpn passthrough=no src-address-list=vpn
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip route
add dst-address=0.0.0.0/0 gateway=wireguard1 routing-table=vpn

anav · Thu Dec 31, 2020 6:15 am

Not sure if this is legal to do with bridges????
add bridge=bridge1 interface=bond1
/interface bridge vlan
add bridge=bridge1 tagged=bond1

No idea what your are trying to accomplish by this??
vlan-ids=1-4094

leopardus2 · Thu Dec 31, 2020 9:18 am

Not sure if this is legal to do with bridges????
add bridge=bridge1 interface=bond1
/interface bridge vlan
add bridge=bridge1 tagged=bond1

This router has a trunk over bond1 to the switch that distributes VLANs around.
Anyway it is working as supposed, but not for dhcp.... VLAN 166 is coming to the RB3011
from the trunk.

No idea what your are trying to accomplish by this??
vlan-ids=1-4094

Maybe I need some advice here, as I never configured VLANs on RouterOS before.

So the scenario is, I have these VLANs reachable through the trunk bond1.
I need inter-vlan routing between them, and need VLAN 166 to get IP addresses assigned through DHCP.

Can you suggest the proper configuration for this?

Thanks in advance,
Rick

leopardus2 · Thu Dec 31, 2020 9:42 am

Perhaps my error is this:

add interface=bridge1 name=VLAN166 vlan-id=166

I should instead use:

add interface=bond1 name=VLAN166 vlan-id=166

leopardus2 · Thu Dec 31, 2020 1:39 pm

I have reviewed the config, this should be correct (I have checked documentation) but still DHCP does not work.
Can this be a 7.1beta3 bug?

Rick

# dec/31/2020 12:36:47 by RouterOS 7.1beta3
# software id = G5ES-UNVF
#
# model = RB3011UiAS
# serial number = B8950C9801E6
/interface bridge
add admin-mac=00:00:00:00:00:01 auto-mac=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=VLAN7 vlan-id=7
add interface=bridge1 name=VLAN166 vlan-id=166
/interface bonding
add mode=802.3ad name=bond1 slaves=ether5,ether9,ether10
/interface list
add name=WAN
add name=LAN
add name=VPN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool1 ranges=192.168.166.1-192.168.166.99
/ip dhcp-server
add address-pool=dhcp_pool1 authoritative=after-2sec-delay interface=VLAN166 lease-time=23h name=dhcp1
/ip vrf
add list=all name=main
/interface bridge port
add comment=defconf interface=ether1
add comment=defconf interface=ether2
add comment=defconf interface=ether3
add comment=defconf interface=ether4
add comment=defconf interface=ether6
add comment=defconf interface=ether7
add comment=defconf interface=ether8
add bridge=bridge1 interface=bond1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=166
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=7
/interface list member
add interface=lte1 list=WAN
add comment=defconf interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=ether10 list=LAN
add list=VPN
add list=LAN
add list=LAN
add interface=wireguard1 list=VPN
add list=WAN
add interface=VLAN166 list=LAN
add interface=VLAN7 list=LAN
/ip address
add address=192.168.1.5/24 interface=ether1 network=192.168.1.0
add address=192.168.69.253/24 interface=wireguard1 network=192.168.69.0
add address=192.168.166.254/24 interface=VLAN166 network=192.168.166.0
add address=192.168.7.254/24 interface=VLAN7 network=192.168.7.0
/ip dhcp-client
add comment=defconf disabled=no
/ip dhcp-server network
add address=192.168.166.0/24 dns-server=8.8.8.8,8.8.8.4 gateway=192.168.166.254
/ip dns
set servers=8.8.8.8
/ip firewall address-list
add address=192.168.166.100-192.168.166.199 list=vpn
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="traffic from VLAN7" in-interface=VLAN7 protocol=tcp
add action=accept chain=forward comment="samba access" dst-address=192.168.7.13 dst-port=139,445 protocol=tcp src-address=192.168.166.243
add action=drop chain=forward comment="drop from VLAN166" in-interface=VLAN166 log=yes out-interface-list=LAN
add action=accept chain=input comment=DHCP in-interface=bridge1 protocol=udp src-port=68
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN log=yes
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN

mkx · Thu Dec 31, 2020 10:55 pm

Bridge MAC address is not exactly correct, you should use proper locally administeted MAC address. Probably not a show stopper, but one never knows.

You may want to add firewall filter rule allowing UDP port 68 traffic from in-interface=VLAN166 ... probably not necessary, but the similar rule regarding in-interface=bridge surely doesn't help in this particular case.

sindy · Fri Jan 01, 2021 11:13 am

DHCP servers are disabled by default in RouterOS. As your export doesn't show disabled=no on the /ip dhcp-server row, it is administratively disabled. Some other configuration items are enabled by default, in yet other ones the name of the parameter is enabled instead of disabled, so it is quite easy to get confused.

leopardus2 · Fri Jan 01, 2021 12:53 pm

It was enabled, just not when I captured the config, so it’s not that unfortunately.... anything else that may come to mind? I also checked the firewall and there are no packets dropped by any rule.

sindy · Fri Jan 01, 2021 2:07 pm

I'd use logging and sniffing to see what's going on.

The bonding is implemented completely in software on the 3011, so try sniffing on all three Ethernet ports as well as the VLAN one:
tool/sniffer/quick interface=ether5,ether9,ether10,VLAN166 port=67,68

Logging:
/system/logging/add topics=dhcp
/log/print follow-only

Nothing in your configuration seems wrong to me. A similar configuration between two CHR running 7.1beta3 (bond in LACP mode runs over two EoIP links) works fine.

What kind of switch is connected at the other end of your LAG (= bond in 802.3ad mode)?

leopardus2 · Fri Jan 01, 2021 3:10 pm

A CSS610-8G-2S+IN 8p.Gig + 2p.SFP+

leopardus2 · Mon Jan 04, 2021 3:26 pm

The bonding is implemented completely in software on the 3011, so try sniffing on all three Ethernet ports as well as the VLAN one:
tool/sniffer/quick interface=ether5,ether9,ether10,VLAN166 port=67,68

Logging:
/system/logging/add topics=dhcp
/log/print follow-only

I enabled the tracing and power cycled an amazon echo, I see the DHCP request coming in, but it is received by bridge and not VLAN166...

  ether9   41.473  251  <-  F0:F0:A4:76:4E:FF  FF:FF:FF:FF:FF:FF  0.0.0.0:68 (bootpc)  255.255.255.255:67 (bootps)  ip:udp  356  0  no
  bridge1  41.473  247  <-  F0:F0:A4:76:4E:FF  FF:FF:FF:FF:FF:FF  0.0.0.0:68 (bootpc)  255.255.255.255:67 (bootps)  ip:udp  356  0  no
  bridge1  41.473  248  <-  F0:F0:A4:76:4E:FF  FF:FF:FF:FF:FF:FF  0.0.0.0:68 (bootpc)  255.255.255.255:67 (bootps)  ip:udp  356  1  no
  ether10  41.473  250  <-  F0:F0:A4:76:4E:FF  FF:FF:FF:FF:FF:FF  0.0.0.0:68 (bootpc)  255.255.255.255:67 (bootps)  ip:udp  356  0  no
  bridge1  41.473  249  <-  F0:F0:A4:76:4E:FF  FF:FF:FF:FF:FF:FF  0.0.0.0:68 (bootpc)  255.255.255.255:67 (bootps)  ip:udp  356  0  no

sindy · Mon Jan 04, 2021 3:48 pm

It means that the DHCPDISCOVER arrives tagless through the LAG, so concentrate at VLAN handling in the CSS610-xxx . I have no idea how the Amazon thing works, but e. g. most Windows network drivers strip the VLAN header from the received frames, so the networking works even if untagging is configured wrong on the switch in the egress direction. The ACL in the CSS610-xxx doesn't seem to allow specific handling of DHCPDISCOVER packets, but who knows.

Can you post screenshots of LAG, VLAN, VLANs, and ACL tabs from the CSS610, and indicate to which port the Amazon thing is connected and which ports are connected to the 3011 (and double check that the Amazon box is really plugged to the port to which you intended to plug it)?

leopardus2 · Mon Jan 04, 2021 4:03 pm

Actually, the DHCP server is meant to serve DHCP requests coming from the wifi clients.
Currently, I have a Mikrotik BaseBox2 which is used for IoT clients. It runs DHCP server for VLAN166.
I am trying to replace this DHCP server and use the RB3011 instead, hence this post.
The BaseBox2 is not directly connected to the CSS610, but rather to a CSS326 which in turn has a fiber connection to the CSS610.
The CSS610 then has the LACP trunk to the RB3011.
Trying to get some screenshots now.

sindy · Mon Jan 04, 2021 4:12 pm

So it seems the screenshots from the CSS610 alone are not sufficient, the network diagram all the way from the Amazon box to the 3011 is necessary, and those screenshots from both CSS. I won't be surprised if you find the issue while gathering these data 🙂

leopardus2 · Mon Jan 04, 2021 4:32 pm

Here are the screenshot from relevant tabs and a hand written diagram :-)

css326_vlan.jpg

css326_vlans.jpg

css326_link.jpg

CSS610_vlans.jpg

CSS610_vlan.jpg

CSS610_link.jpg

CSS610_lag.jpg

diagram.jpg

leopardus2 · Mon Jan 04, 2021 4:42 pm

Forgot to tell . I have no ACLs in place, that's why I did not post these screenshots!

sindy · Mon Jan 04, 2021 4:59 pm

Hm, so the last thing to come to my mind is the configuration of the Basebox. Both CSS have VLAN mode set to optional on all ports, which means that if something comes tagged with VID 1 from the Basebox, it will make it through the two CSS all the way to the 3011 will get untagged on the Interconnect link already at the CSS326.

As you say that everything else but the DHCP works fine, what happens if you sniff other traffic of the clients when their addresses are assigned by the Basebox? Does this "normal" (=non-DHCP) traffic arrive tagged to the 3011? Vice versa, if you connect a wired DHCP client to one of the access ports to VLAN 166 at the CSS326 and sniff on the 3011, does the DHCPDISCOVER come tagged or not?

leopardus2 · Mon Jan 04, 2021 5:20 pm

Hm, so the last thing to come to my mind is the configuration of the Basebox. Both CSS have VLAN mode set to optional on all ports, which means that if something comes tagged with VID 1 from the Basebox, it will make it through the two CSS all the way to the 3011 will get untagged on the Interconnect link already at the CSS326.

Interesting theory, however, I have no VLANs defined in config on the basebox, and also I have forced VLAN 166 on the CSS326 on that port (see screenshots).
Everything works as expected on the VLAN side anyways. Everything but DHCP...

Rick

leopardus2 · Mon Jan 04, 2021 5:35 pm

Sniffed on the basebox (Schifo AP) and this seems to indicate you were indeed right:

admin@schifo_AP] > /tool sniffer quick interface=bridge-local port 67
INTERFACE        TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP 
bridge-local    1.013      3 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    2.013      4 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    3.013      5 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    3.018      6 <-  48:8F:5A:47:0E:C7 FF:FF:FF:FF:FF:FF 1      192.168.88.1:67 (bootps)            255.255.255.255:68 (bootpc)         ip:udp      346   0 no 
bridge-local    4.013      7 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    4.017      8 <-  48:8F:5A:47:0E:C7 FF:FF:FF:FF:FF:FF 1      192.168.88.1:67 (bootps)            255.255.255.255:68 (bootpc)         ip:udp      346   0 no 
bridge-local    5.013      9 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    6.013     10 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    7.014     11 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    7.018     12 <-  48:8F:5A:47:0E:C7 FF:FF:FF:FF:FF:FF 1      192.168.88.1:67 (bootps)            255.255.255.255:68 (bootpc)         ip:udp      346   0 no 
bridge-local    8.013     13 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local    8.017     14 <-  48:8F:5A:47:0E:C7 FF:FF:FF:FF:FF:FF 1      192.168.88.1:67 (bootps)            255.255.255.255:68 (bootpc)         ip:udp      346   0 no 
bridge-local    9.013     15 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local   10.013     16 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local   11.013     17 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local   11.017     18 <-  48:8F:5A:47:0E:C7 FF:FF:FF:FF:FF:FF 1      192.168.88.1:67 (bootps)            255.255.255.255:68 (bootpc)         ip:udp      346   0 no 
bridge-local   12.013     19 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local   12.017     20 <-  48:8F:5A:47:0E:C7 FF:FF:FF:FF:FF:FF 1      192.168.88.1:67 (bootps)            255.255.255.255:68 (bootpc)         ip:udp      346   0 no 
bridge-local   13.013     21 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no 
bridge-local   14.013     22 <-  48:8F:5A:F9:02:96 FF:FF:FF:FF:FF:FF 1      0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      346   0 no

So it looks like VLAN ID 1 is actually used. How can I ensure there is no VLAN ID added on egress from the basebox? So that id 166 can be added by the CSS326?

sindy · Mon Jan 04, 2021 5:49 pm

Do you really expect me to advise anything without seeing the Basebox configuration export 🙂 ?

leopardus2 · Mon Jan 04, 2021 6:36 pm

Here you go ...

# jan/04/2021 17:32:34 by RouterOS 6.45.9
# software id = 0JQM-RNVG
#
# model = 912UAG-2HPnD
# serial number = 6EC5042C14C5
/interface bridge
add admin-mac=4C:5E:0C:1C:C6:C1 auto-mac=no fast-forward=no name=bridge-local
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce country=no_country_set disabled=no distance=indoors frequency=2422 frequency-mode=\
    manual-txpower mode=ap-bridge rate-set=configured rx-chains=0,1 ssid=schifo tx-chains=0,1 wireless-protocol=802.11 wps-mode=disabled
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa-psk,wpa2-psk mode=dynamic-keys supplicant-identity=MikroTik
add authentication-types=wpa-psk,wpa2-psk management-protection=allowed name=open supplicant-identity=basebox
add authentication-types=wpa-psk,wpa2-psk mode=dynamic-keys name=alternate supplicant-identity=basebox
/interface wireless
add disabled=no mac-address=4E:5E:0C:1C:C6:C2 master-interface=wlan1 name="alternate SSID" security-profile=alternate ssid=alternate wds-default-bridge=bridge-local \
    wps-mode=disabled
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=dhcp_pool1 ranges=192.168.166.1-192.168.166.99
/ip dhcp-server
add address-pool=dhcp_pool1 authoritative=after-2sec-delay interface=bridge-local lease-time=23h name=dhcp1
/interface bridge port
add bridge=bridge-local hw=no interface=ether1
add bridge=bridge-local interface=wlan1
add bridge=bridge-local interface="alternate SSID"
/ip settings
set arp-timeout=1h
/interface wireless access-list
add comment="Bedroom CPE" interface=wlan1 mac-address=4C:5E:0C:F9:51:11 vlan-mode=no-tag
add comment="C4 Touchscreen" disabled=yes interface=wlan1 mac-address=00:19:88:21:C6:BD vlan-mode=no-tag
add comment="LIFX Living Room Applique 1" interface=wlan1 mac-address=D0:73:D5:02:58:EC vlan-mode=no-tag
add comment="Living Room CPE" interface=wlan1 mac-address=E4:8D:8C:74:CC:4B vlan-mode=no-tag
add comment="Entrance Nest Cam" interface=wlan1 mac-address=18:B4:30:57:24:C6 vlan-mode=no-tag
add comment="Nest Thermostat" interface=wlan1 mac-address=18:B4:30:13:72:4D vlan-mode=no-tag
add comment="Withings Aura" interface=wlan1 mac-address=00:24:E4:20:37:56 vlan-mode=no-tag
add comment="Roku Media Player" interface=wlan1 mac-address=08:05:81:7B:13:71 vlan-mode=no-tag
add comment="LIFX Stairs" interface=wlan1 mac-address=D0:73:D5:01:77:F8 vlan-mode=no-tag
add comment="LIFX Living Room Spot 2" interface=wlan1 mac-address=D0:73:D5:11:DA:72
add comment="LIFX Living Room Sofa" interface=wlan1 mac-address=D0:73:D5:01:5A:50 vlan-mode=no-tag
add comment="Apple device" interface=wlan1 mac-address=1C:91:48:0F:92:36 vlan-mode=no-tag
add comment="Apple device" interface=wlan1 mac-address=F0:C1:F1:A1:E7:66 vlan-mode=no-tag
add comment="LIFX Living Room Applique 2" interface=wlan1 mac-address=D0:73:D5:02:B1:F7 vlan-mode=no-tag
add comment="LIFX Living Room Abat-Jour" interface=wlan1 mac-address=D0:73:D5:13:A1:50 vlan-mode=no-tag
add comment="LIFX Living Room Spot 1" interface=wlan1 mac-address=D0:73:D5:02:CF:F5 vlan-mode=no-tag
add comment="Amazon Echo Dot" interface=wlan1 mac-address=50:F5:DA:28:54:2D vlan-mode=no-tag
add comment=melange interface=wlan1 mac-address=D8:30:62:5D:3B:77 vlan-mode=no-tag
add comment="LIFX Living Room Window" interface=wlan1 mac-address=D0:73:D5:13:FE:45 vlan-mode=no-tag
add ap-tx-limit=4000000 comment="Wiwo Closet" interface=wlan1 mac-address=AC:CF:23:56:2B:FA vlan-mode=no-tag
add comment="LIFX Applique 1 (entrance)" interface=wlan1 mac-address=D0:73:D5:20:38:3C vlan-mode=no-tag
add comment="LIFX Bedroom Abat Jour" interface=wlan1 mac-address=D0:73:D5:20:61:F2
add comment="WAP CPE" interface=wlan1 mac-address=6C:3B:6B:6C:E3:9A vlan-mode=no-tag
/ip address
add address=192.168.88.1/24 comment="default configuration" disabled=yes interface=bridge-local network=192.168.88.0
add address=192.168.166.253/24 interface=wlan1 network=192.168.166.0
/ip dhcp-server network
add address=192.168.166.0/24 dns-server=8.8.8.8,8.8.8.4 gateway=192.168.166.254
/ip dns
set servers=8.8.8.8,8.8.8.4
/ip route
add distance=1 gateway=192.168.166.254
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=bridge-local type=internal
/system clock
set time-zone-name=Europe/Rome
/system identity
set name=schifo_AP
/system leds
set 0 interface=wlan1 leds=""
add interface=wlan1 leds=led1 type=wireless-status
add interface=wlan1 leds=led2 type=interface-activity
add interface=wlan1 leds=led3 type=interface-transmit
/system package update
set channel=long-term
/tool sniffer
set filter-interface=bridge-local filter-ip-protocol=udp

sindy · Mon Jan 04, 2021 6:58 pm

I don't get it, it looks as if the frames were coming tagged with VID 1 already over the radio. Can you remove the interface=bridge-local from the /tool sniffer parameters and try again?

leopardus2 · Mon Jan 04, 2021 7:12 pm

that doesn't seem to change anything...

I don't understand wht the CSS326 is not forcing VLAN 166 to all packets like it should according to https://wiki.mikrotik.com/wiki/SwOS/CSS ... _and_VLANs

sindy · Mon Jan 04, 2021 7:52 pm

that doesn't seem to change anything...

Wait, at least you should see each frame three times - on the "alternate SSID" interface, on the bridge-local interface, and on the ether1. Is it the case? Or literally nothing has changed any you can only see them on the bridge-local?

If you can see them on all three interfaces, and if it really appears on the wlan interface with VID 1, it should be possible to normalize things by the following:
/interface bridge vlan add vlan-ids=1 tagged="alternate SSID"
/interface bridge port set [find interface="alternate SSID"] pvid=3333 (actually anything between 2 and 4094 included)
/interface bridge set bridge-local vlan-filtering=yes
Another question is where the tag appears from, as the configuration doesn't explain that. Wherever I was using 6.45.9 with wireless, I was always running CAPsMAN, so if it is a "feature" of that release, I'm not aware of it.

I don't understand wht the CSS326 is not forcing VLAN 166 to all packets like it should according to https://wiki.mikrotik.com/wiki/SwOS/CSS ... _and_VLANs

Nor do I, but I've seen a live SwOS just once while passing by on board of Pendolino, so I'm not aware of any bugs that may be there. This functionality (rewriting of VID of already tagged ingress packets) is quite unusual, so some issue may have been missed during release testing.

leopardus2 · Mon Jan 04, 2021 9:16 pm

I tried the following -
1. assigned an IP address to bridge1
2. moved dhcp server to bridge1

Now DHCP seem to be working properly. Any drawbacks in doing this? Considering that DHCP bypasses firewall and VLANs anyway....

Only oddity is I am getting this warning in log:

bond1: bridge port received packet with own address as source address (48:8f:5a:8a:27:6a), probably loop

Not sure why.

Rick

leopardus2 · Mon Jan 04, 2021 9:34 pm

About that last oddity, here's the why

[admin@RB3011_GW] /ip/neighbor/discovery-settings> /interface/print                      
Flags: X - DISABLED, R - RUNNING; S - SLAVE
Columns: NAME, TYPE, ACTUAL-MTU, L2MTU, MAX-L2MTU, MAC-ADDRESS
   #      NAME        TYPE       ACTU  L2MT  MAX-  MAC-ADDRESS      
;;; do not enable, MAC stolen for bridge use
   0  X   ether1      ether      1500  1598  8156  48:8F:5A:8A:27:66
   1      ether2      ether      1500  1598  8156  48:8F:5A:8A:27:67
   2      ether3      ether      1500  1598  8156  48:8F:5A:8A:27:68
   3      ether4      ether      1500  1598  8156  48:8F:5A:8A:27:69
   4  RS  ether5      ether      1500  1598  8156  48:8F:5A:8A:27:6A
   5      ether6      ether      1500  1598  8156  48:8F:5A:8A:27:6C
   6      ether7      ether      1500  1598  8156  48:8F:5A:8A:27:6D
   7      ether8      ether      1500  1598  8156  48:8F:5A:8A:27:6E
   8  RS  ether9      ether      1500  1598  8156  48:8F:5A:8A:27:6F
   9  RS  ether10     ether      1500  1598  8156  48:8F:5A:8A:27:70
  10      sfp1        ether      1500  1600  8158  48:8F:5A:8A:27:6B
  11  R   VLAN7       vlan       1500  1594        48:8F:5A:8A:27:66
  12  R   VLAN88      vlan       1500  1594        48:8F:5A:8A:27:66
  13  R   VLAN166     vlan       1500  1594        48:8F:5A:8A:27:66
  14  RS  bond1       bond       1500  1598        48:8F:5A:8A:27:6A
  15  R   bridge1     bridge     1500  1598        48:8F:5A:8A:27:66
  16  R   lte1        lte        1500              72:86:25:E2:9E:AA
  17  R   wireguard1  wireguard  1420              00:00:00:00:00:00

sindy · Mon Jan 04, 2021 9:48 pm

I tried the following -
1. assigned an IP address to bridge1
2. moved dhcp server to bridge1

This way you gave up on the VLAN 166 and accommodated to the fact that the DHCP request arrives to the 3011 tagless. Doing so surely makes things work, but I don't know how far it is from your initial intention - presumably you had a reason to isolate the IoT traffic from the rest by using a dedicated VLAN and subnet for it.

Only oddity is I am getting this warning in log:
Code: Select all
bond1: bridge port received packet with own address as source address (48:8f:5a:8a:27:6a), probably loop
Not sure why.

Because something else is broken in the overall setup, causing a frame sent by the 3011 to come back. It may be caused by some external device, or it may be something in the interworking between the bonding in 802.3ad mode at the 3011 and the LAG on the CSS - it should be the same thing, but maybe it is not. I assume you don't have a plain wired L2 loop anywhere in the topology, but things like Ethernet interface bridged with a wireless one on two devices where one is a STA registered to the other one acting as an AP can cause this if protocol-mode=none on both those bridges. Other reasons are possible too. I'd start from keeping just a single member link of the bond up and seeing whether this log message stops coming or not.

But first of all I'd concentrate on understanding the VID 1 tag mystery on the Basebox. It may possibly cause also this loop.

Regarding your explanation of the oddity, I didn't get it at all. If ether1 is configured as a member port of a bridge, it can be enabled even if that bridge has inherited the MAC address from it with no negative effect.

leopardus2 · Mon Jan 04, 2021 10:17 pm

I tried the following -
1. assigned an IP address to bridge1
2. moved dhcp server to bridge1
This way you gave up on the VLAN 166 and accommodated to the fact that the DHCP request arrives to the 3011 tagless. Doing so surely makes things work, but I don't know how far it is from your initial intention - presumably you had a reason to isolate the IoT traffic from the rest by using a dedicated VLAN and subnet for it.

Well, for DHCP purposes then yes, but I do not have DHCP clients on the other VLANs.
But VLANs remain separated for other purposes still if I am not mistaken.

I can certainly disconnect 2 of the 3 cables of the LAG and see if error goes away...

sindy · Mon Jan 04, 2021 10:25 pm

But VLANs remain separated for other purposes still if I am not mistaken.

I can't see how the VID 1 tagging weirdness could selectively affect only DHCP and not the rest of the traffic.

That's why I've asked you to sniff the other traffic too.

leopardus2 · Mon Jan 04, 2021 10:33 pm

But VLANs remain separated for other purposes still if I am not mistaken.
I can't see how the VID 1 tagging weirdness could selectively affect only DHCP and not the rest of the traffic.

That's why I've asked you to sniff the other traffic too.

inter-vlan isolation remains. Also I verified that reducing the LAG to 1 channel makes the bridge loop error go away.
Any hint about what could be tweaked to make it happy? This all started when I assigned an IP address to bridge1.

I don't actually see a VLAN1 tag when I sniff on basebox AP. I was seeing some VLAN1 traffic coming from VLAN88 but I have segregated that with an ACL and now it is clean.
Any other theory about why dhcp packets land on bridge1 instead of vlan166? Traffic from vlan166 is properly tagged when I sniff on RB3011....

mkx · Mon Jan 04, 2021 10:40 pm

I'd suggest you to file the problem with MT support, could be some bug on CSS610 (it is a pretty new product after all).
I've had a marginal problem (possibly still exists) on hAP ac2 which, when configured with switch-chip VLANs, would fail to untag a particular PPPoE packet (PADI IIRC) causing PPPoE to fail. Using another device for tagging/untagging (e.g. RB951G or a 3rd party managed switch) makes setup work and using bridge-vlan on hAP ac2 is fine as well.

leopardus2 · Mon Jan 04, 2021 10:47 pm

Yeah, I already found other bugs in this unit. I checked for an update on Mikrotik site but there isn't one, which is odd.

leopardus2 · Tue Jan 05, 2021 12:21 am

I have reverted to the correct config, with no bridge IP address and with DHCP server on VLAN166.

While sniffing on the RB3011 I see some packets that come from VLAN166 that seem to be missing the tag.
Looking closely, they seem to be duplicate packets due to the bonded trunk.

See below. Is my suspicion correct? Is this normal?

  
  bridge1  5.052  3634  <-  00:05:CD:86:A6:81  01:00:5E:7F:FF:FA  166  192.168.166.200:47136  239.255.255.250:1900  ip:udp  669  0  no
  VLAN166  5.052  3646  <-  00:05:CD:86:A6:81  01:00:5E:7F:FF:FA       192.168.166.200:47136  239.255.255.250:1900  ip:udp  665  0  no
  VLAN166  5.052  3647  <-  00:05:CD:86:A6:81  01:00:5E:7F:FF:FA       192.168.166.200:47136  239.255.255.250:1900  ip:udp  665  1  no

sindy · Tue Jan 05, 2021 9:30 am

You've posted too few lines of the sniff output to answer simply yes or no. Normally, you should see the single actual frame once on the physical interface (etherX), then once on the bond interface, then once the bridge interface, all these with VID 166, and finally once on the VLAN166 interface without VID 166 (which is correct, the sniffer sniffs at the tagless end of the /interface vlan - no point in sniffing at the tagged one which is attached to some other interface, in this case the bridge one).

Having exactly the same frame twice on the VLAN166 interface without any gap in sequence numbering definitely looks suspicious, but if it was a duplicate due to LAG malfunction, you would have to see it twice also at the bridge and the bond, and once at each of the two Ethernets.

leopardus2 · Tue Jan 05, 2021 9:35 am

Good morning!
I have spent more time tracing packets and I have come to the conclusion that the packets arrive correctly tagged to the RB3011,
*except* the DHCP packets. So the issue must be on the RB3011.
I do know that DHCP is treated in a special way by RouterOS, perhaps that's part of the scenario.

Question, do I really need a bridge on bond1 in my setup?

Thanks
Rick

leopardus2 · Tue Jan 05, 2021 10:21 am

The DHCP packets land on bridge1 untagged, see below. That's why DHCP does not work on VLAN interface.
Any way to add a tag when the packets enter the bridge?

/interface bridge filter
add action=mark-packet chain=input dst-port=67 ip-protocol=udp mac-protocol=ip new-packet-mark=dhcp
add action=accept chain=input mac-protocol=vlan packet-mark=dhcp vlan-id=1
add action=accept chain=input mac-protocol=vlan packet-mark=dhcp vlan-id=166

/interface/bridge/filter> print stats       
Columns: CHAIN, ACTION, BYTES, PACKETS
  #  CHAIN  ACTION       BYTES  PA
  0  input  mark-packet  28632  87
  1  input  accept           0   0
  2  input  accept           0   0

leopardus2 · Tue Jan 05, 2021 10:55 am

Added pvid=166 to bridge port, to tag untagged packets with VLAN166 and that seemed to do the trick
(now DHCP packets are correctly tagged, and DHCP server answers):

/interface bridge port
add bridge=bridge1 interface=bond1 pvid=166

Still something is not proper, as the DHCP offer is not accepted it seems.

leopardus2 · Tue Jan 05, 2021 11:14 am

Funnily enough, the DHCP replies land on the basebox with a vlan encapsulation and vlan=1. Weird!

leopardus2 · Tue Jan 05, 2021 11:24 am

I have solved the issue by using an ACL on the CSS326 that replaces the VLAN tag 1 with tag 166 when packet comes from the DHCP VLAN interface!
Now everything works as expected.
Something really weird is going on with Mikrotik and DHCP packets handling, but it looks like I'm not the first one to say this!

So to recap:

- assigned pvid 166 to the bridge port bond1 so that packets arriving without a tag get tagged with VLAN166, this will fix DHCP requests from clients that come untagged
- used an ACL on CSS326 to map DHCP offers that are sent on management VLAN back to VLAN166 so that the access port of basebox will strip the VLAN tag on ingress

Thank you very much @sindy for your help with this!

Rick

leopardus2 · Tue Jan 05, 2021 11:45 am

Side note, with this setup, the message warning bond1: bridge port received packet with own address as source address (48:8f:5a:8a:27:66), probably loop reappeared,
although sporadically

sindy · Tue Jan 05, 2021 12:12 pm

Regarding the known specific handling of DHCP on Mikrotik, it only has to do with L3 processing, as DHCP traffic is handled before the firewall gets the packets. This is not RouterOS specific, it behaves the same on bare Linux.

On an interface with no L3 configuration attached (a dhcp client is also to be considered an "L3 configuration" in this sense), the frames carrying DHCP packets are normally not handled in a specific way, and definitely VLAN tagging of a frame should not depend on its payload IP packet.

Using an ACL to rewrite VID 1 to VID 166 on the 326 is a good way to work around the fact that the Force VLAN ID doesn't work, as it selectively addresses a particular, isolated issue. But why the tag with VID 1 is seen on the Basebox remains unclear - have you checked that this is the case only for DHCP requests from client, whereas frames from client carrying other packets are tagless as they should be?

Regarding the DHCP responses seen tagged as well - do they already come tagged from the CRS (so the VID 1 is visible already at the ether1)?

A blind shot, can you set fast-forward on the bridge-local on the Basebox to yes? Also, can you (in a separate step) set vlan-filtering to yes there? Both might affect the undesired tagging.

leopardus2 · Tue Jan 05, 2021 12:26 pm

Regarding the known specific handling of DHCP on Mikrotik, it only has to do with L3 processing, as DHCP traffic is handled before the firewall gets the packets. This is not RouterOS specific, it behaves the same on bare Linux.

On an interface with no L3 configuration attached (a dhcp client is also to be considered an "L3 configuration" in this sense), the frames carrying DHCP packets are normally not handled in a specific way, and definitely VLAN tagging of a frame should not depend on its payload IP packet.

Okay, but still the DHCP packets come untagged over the bridge, all other packets have proper tagging...

Using an ACL to rewrite VID 1 to VID 166 on the 326 is a good way to work around the fact that the Force VLAN ID doesn't work, as it selectively addresses a particular, isolated issue. But why the tag with VID 1 is seen on the Basebox remains unclear - have you checked that this is the case only for DHCP requests from client, whereas frames from client carrying other packets are tagless as they should be?

Force VLAN works on ingress (from basebox AP) but I needed to rewrite the tag on egress, i.e. on ingress from trunk (SFP)

I have sniffed on my macbook connected via wireless to the basebox and yes, only DHCP packets came encapsulated, all others were properly stripped - I believe this happens because the access port on the CSS326 has a default vlan ID of 166, so if packet has that tag it is stripped otherwise it is passed as-is.

Regarding the DHCP responses seen tagged as well - do they already come tagged from the CRS (so the VID 1 is visible already at the ether1)?

This I have not checked and don't think I can because I have a CSS and not a CRS (it runs switchos)

sindy · Tue Jan 05, 2021 1:04 pm

I have sniffed on my macbook connected via wireless to the basebox and yes, only DHCP packets came encapsulated, all others were properly stripped - I believe this happens because the access port on the CSS326 has a default vlan ID of 166, so if packet has that tag it is stripped otherwise it is passed as-is.

This would require that already the DHCP response was leaving the 3011 tagless or tagged with some other VID than 166. Is that the case?

Regarding the DHCP responses seen tagged as well - do they already come tagged from the CRS (so the VID 1 is visible already at the ether1)?
This I have not checked and don't think I can because I have a CSS and not a CRS (it runs switchos)

That's why I'm asking how it looks at ether1 of Basebox where sniffing is possible.

leopardus2 · Tue Jan 05, 2021 1:24 pm

Like I said, DHCP packets arrive on basebox encapsulated with VLAN id 1.
I have sniffed on a client connected through wireless. All other packets are tagless. DHCP does not work as a consequence.

Sniffing on RB3011 now shows DHCP is tagged with VLAN166 at bridge level:

/interface/bridge/filter> /tool sniffer quick interface=bridge1 ip-protocol=udp port=67                          
Columns: INTERFACE, TIME, NUM, DIR, SRC-MAC, DST-MAC, VLAN, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU, FP
  INTERFA  TIME   NU  DI  SRC-MAC            DST-MAC            VLA  SRC-ADDRESS                  DST-ADDRESS                  PROTOC  SIZ  C  FP
  bridge1  3.381   1  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  346  1  no
  bridge1  3.381   2  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  346  0  no
  bridge1  3.381   3  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  346  0  no
  bridge1  3.383   4  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.384   5  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.384   6  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.384   7  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.385   8  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.385   9  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.386  10  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.387  11  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bridge1  3.387  12  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no

But sniffing on bond1 shows different results, with tag missing:

/interface/bridge/filter> /tool sniffer quick interface=bond1 ip-protocol=udp port=67      
Columns: INTERFACE, TIME, NUM, DIR, SRC-MAC, DST-MAC, VLAN, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU, FP
  INTER  TIME    NU  DI  SRC-MAC            DST-MAC            VLA  SRC-ADDRESS                  DST-ADDRESS                  PROTOC  SIZ  C  FP
  bond1  3.303    1  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF       0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  342  1  no
  bond1  3.303    2  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF       0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  342  0  no
  bond1  3.303    3  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF       0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  342  0  no
  bond1  3.304    4  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bond1  3.305    5  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  bond1  3.306    6  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  bond1  3.306    7  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bond1  3.307    8  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  bond1  3.307    9  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  bond1  3.307   10  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  0  no
  bond1  3.308   11  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  bond1  3.308   12  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no

Looking at the single ethernets in the bond, you can see that DHCP request arrives untagged there. Offer has tags but not for each of them.
I guess it is normal to see 3 packets going in and out considering it is a 3 channel bond? But still it looks suspicious.

 /interface/bridge/filter> /tool sniffer quick interface=ether5,ether9,ether10 ip-protocol=udp port=67     
Columns: INTERFACE, TIME, NUM, DIR, SRC-MAC, DST-MAC, VLAN, SRC-ADDRESS, DST-ADDRESS, PROTOCOL, SIZE, CPU, FP
  INTERFA  TIME   NU  DI  SRC-MAC            DST-MAC            VLA  SRC-ADDRESS                  DST-ADDRESS                  PROTOC  SIZ  C  FP
  ether5   5.373   3  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF       0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  342  1  no
  ether9   5.373   2  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF       0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  342  0  no
  ether10  5.373   1  <-  8C:85:90:B9:C8:95  FF:FF:FF:FF:FF:FF       0.0.0.0:68 (bootpc)          255.255.255.255:67 (bootps)  ip:udp  342  0  no
  ether5   5.375   4  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  1  no
  ether9   5.376   6  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  ether5   5.376   7  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  1  no
  ether10  5.376   5  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  ether9   5.377   9  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  ether5   5.377  10  ->  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF  166  192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  346  1  no
  ether10  5.377   8  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  ether9   5.379  12  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no
  ether10  5.379  11  <-  48:8F:5A:8A:27:66  FF:FF:FF:FF:FF:FF       192.168.166.254:67 (bootps)  255.255.255.255:68 (bootpc)  ip:udp  342  0  no

EDIT: it looks like only ether5 (first interface in the bond) gets tagged. Perhaps it is normal, I don't know...

sindy · Wed Jan 06, 2021 10:31 am

At the first place it is not normal that the same frame is leaving through all three Ethernet interfaces which are grouped into a bond in 802.3ad (LACP) mode, even though that frame is a broadcast one. The hash of the source and destination addresses (MAC+ VID when you have the transmit-hash-policy set to the default layer-2) always chooses a single link to send a frame (out of the ones that are alive).

And on a CHR running 7.1beta3, it does work like that:

  INTERF  TIME    NU  DI  SRC-MAC            DST-MAC            VLA  SRC-ADDRESS                      DST-ADDRESS                       PROTOC  SIZ  C  FP
  ...
  eoip-2  46.178  19  ->  FE:1D:1C:3D:85:BC  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)              255.255.255.255:67 (bootps)       ip:udp  346  0  no
  eoip-2  47.17   20  ->  FE:1D:1C:3D:85:BC  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)              255.255.255.255:67 (bootps)       ip:udp  346  0  no
  eoip-2  49.392  21  ->  FE:1D:1C:3D:85:BC  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)              255.255.255.255:67 (bootps)       ip:udp  346  0  no
  eoip-2  52.465  22  ->  FE:1D:1C:3D:85:BC  FF:FF:FF:FF:FF:FF  166  0.0.0.0:68 (bootpc)              255.255.255.255:67 (bootps)       ip:udp  346  0  no

But the remote device is another CHR running 7.1beta3 in this case, not a CSS. Does anything change if you try to configure the LAG member ports on the CSS as active?

leopardus2 · Wed Jan 06, 2021 2:34 pm

So, does this look like a bug in switchos to you?
Should the DHCP discover arrive tagged over the LAG? It looks like it does in your setup?

I have also raised a support ticket one week ago, heard nothing from Miktotik as of yet.

sindy · Wed Jan 06, 2021 6:01 pm

I cannot tell you whether it is a bug in SwOS or a bug in 7.1beta3, because I don't know how RouterOS behaves if the remote party sends no LACP frames - you cannot choose between active and passive mode in RouterOS. But since the LACP did come up between the two CHRs, I assume the RouterOS is always in active mode.

Regarding support, I'm not sure whether they deal with issues related to beta releases at all.

You could sniff to file if files work in 7.1beta3 on ARM - on CHR they don't, the file gets created but always has a zero size.

/tool/sniffer/quick interface=ether5,ether9,ether10 mac-protocol=0x8809 will sniff only LACP, and opening the result in Wireshark should reveal whether the 3011 and the 610 understand each other. If both are passive, the LACP simply doesn't come up, so it works as three independent links, but if this happened, the consequences would probably be worse than just this.

If files don't work and remain with zero size also on the 3011, you can use /tool/sniffer/packet/print raw after stopping the sniffer, copy-paste the result to a text file on a PC, and then import that file to Wireshark from the text format. I can do that if that's over your head.

And yes, the DHCPDISCOVER should arrive tagged via a single interface (ether5 or 9 or 10), be seen still tagged on the bonding and on the bridge, and last untagged on VLAN166. The DHCPOFFER should take the same path in reverse order, except that it may leave through another ether5 or 9 or 10 (the sending side always decides autonomously).

leopardus2 · Wed Jan 06, 2021 7:20 pm

It looks like LAG works, RB3011 is active side and reports link up, mode is layer2+layer3 and all works fine except DHCP discover arrives untagged on RB3011 ethernets - since I cannot sniff on CSS610 I cannot investigate this further I’m afraid....

I can run that capture tomorrow but will leave it to you to understand if LACP is working fine or not.

Thanks
Rick

leopardus2 · Wed Jan 06, 2021 7:34 pm

If CSS610 has mirror feature I could use that and sniff using my macbook from another port maybe,
that will tell if DHCP packets leave the bond tagged or not

sindy · Wed Jan 06, 2021 7:36 pm

I'm still puzzled by the DHCP being treated so bad with regard to both VLAN tagging and bonding at both 7.1beta3 on arm and 6.45.9 on mipsbe. I've never seen such a weird behaviour, neither with VLAN tagging nor with bonding. Your export of the 3011 doesn't show the placement of the dhcp server, but even if it was attached to the bridge rather than the VLAN166, there'd still be no reason why the DHCPOFFER should be broadcast via all three member links of the bond.

As for the mirror feature, I have no clue how it plays with bonding, so I'd rather avoid that even if mirroring is available.

leopardus2 · Thu Jan 07, 2021 7:41 pm

I have sniffed the ingress traffic coming into the CSS610 from CSS326 (on SFP+ interconnect) and DHCP discover was NOT tagged!
To better explain, it was not 802.1Q encapsulated.
So it looks like the CSS326 is NOT encapsulating DHCP traffic even when "force vlan id" is set. It looks like a bug to me.

sindy · Thu Jan 07, 2021 8:24 pm

To me too, but I've got a bit lost in your gradual changes, so I'm not sure I understand properly your current setup. My understanding was that

the frames carrying the DHCPDISCOVER got tagged (with VID 1) in the Basebox (this was what the sniff on the Basebox itself was showing)
the CSS326 has not rewritten the VID in the 802.1Q tag from 1 to 166, which has caused the tag to be stripped on egress through the port facing to the CSS610, as its PVID (default VLAN ID) is 1 (which is what you can see when mirroring the link between the two CSS)
you have worked this CSS326 bug around by using an ACL rule which rewrites the VID from 1 to 166 on ingress, so the DHCPDISCOVER packets were coming to the 3011 in an 802.1Q tagged frames with VID 166
the DHCP server on the 3011 was attached to the /interface vlan VLAN166 (your configuration export from the 3011 doesn't show any dhcp-server at all)

But as you now say that the frame carrying the DHCPDISCOVER has no 802.1Q tag at all, it seems you either stopped using that workaround, or it never actually worked.

If the DHCPDISCOVER is coming to the 3011 tagless, the 3011 should not respond it at all unless the DHCP server is attached directly to the bridge, and thus reacts on packets coming within tagless frames; if there is a DHCP server listening directly on the bridge, there is no surprise that it also responds with a DHCPOFFER within a frame with no 802.1Q tag. If this is the case, the only remaining surprise related to the 3011 is why this answer is sent out via all links in the bond.

leopardus2 · Thu Jan 07, 2021 8:40 pm

the frames carrying the DHCPDISCOVER got tagged (with VID 1) in the Basebox (this was what the sniff on the Basebox itself was showing)

No, the basebox AP is not tagging anything, as it has no vlans configured.
The packets enter the CSS326 through an access port having "force vlan id = 166".

[*]the CSS326 has not rewritten the VID in the 802.1Q tag from 1 to 166, which has caused the tag to be stripped on egress through the port facing to the CSS610, as its PVID (default VLAN ID) is 1 (which is what you can see when mirroring the link between the two CSS)

No, the CSS326 fails to encapsulate DHCP discover packets.

[*]you have worked this CSS326 bug around by using an ACL rule which rewrites the VID from 1 to 166 on ingress, so the DHCPDISCOVER packets were coming to the 3011 in an 802.1Q tagged frames with VID 166

No, this is for the return path (DHCP offer), because the DHCP reply comes to the CSS326 tagged with VLAN=1

[*]the DHCP server on the 3011 was attached to the /interface vlan VLAN166 (your configuration export from the 3011 doesn't show any dhcp-server at all)[/list]

Yes, DHCP is configured on vlan 166 as per original (and correct) config.

But as you now say that the frame carrying the DHCPDISCOVER has no 802.1Q tag at all, it seems you either stopped using that workaround, or it never actually worked.

Workaround works fine. I kept investigating to help you and me (and forum visitors) understand where the issue may be.

If the DHCPDISCOVER is coming to the 3011 tagless, the 3011 should not respond it at all unless the DHCP server is attached directly to the bridge, and thus reacts on packets coming within tagless frames; if there is a DHCP server listening directly on the bridge, there is no surprise that it also responds with a DHCPOFFER within a frame with no 802.1Q tag. If this is the case, the only remaining surprise related to the 3011 is why this answer is sent out via all links in the bond.

Without the workaround, this is what was happening - no response from RB3011. To make it work, I assigned pvid 166 to the bond1 bridge port, so that tagless packets arriving there are tagged with the proper vlan.

leopardus2 · Thu Jan 07, 2021 8:44 pm

Reviewing the thread, I see where I was sniffing bridge on basebox and it showed VLAN=1, however these packets are not received tagged at CS326 (sniffed today using a mirror port, the packets are not 802.1q encapsulated so I guess that VLAN=1 is added by the sniffer tool.

sindy · Thu Jan 07, 2021 9:14 pm

To make it work, I assigned pvid 166 to the bond1 bridge port, so that tagless packets arriving there are tagged with the proper vlan.

OK, so at least one mystery solved. Have you done this systematically, i.e. not only set the pvid of the bond interface to 166 but also moved the bond to the untagged list on the /interface bridge vlan row with vlan-ids=166? If yes, this explains why you can see the response frames tagless on ether5, 9, 10 (but not why you can see them on all three).

Reviewing the thread, I see where I was sniffing bridge on basebox and it showed VLAN=1, however these packets are not received tagged at CS326 (sniffed today using a mirror port, the packets are not 802.1q encapsulated so I guess that VLAN=1 is added by the sniffer tool.

This would be weird too, I have never seen something like this on any device, including mipsbe ones (which is the CPU architecture of Basebox 2). So tell me more about how do you sniff on the mirror port. The thing is that most Windows network card drivers strip any VLAN tag before forwarding the frame further, so even the sniffing API already gets a modified packet. I have no idea how Apple does that, but some Linux drivers reportedly do the same.

leopardus2 · Thu Jan 07, 2021 9:18 pm

To make it work, I assigned pvid 166 to the bond1 bridge port, so that tagless packets arriving there are tagged with the proper vlan.
OK, so at least one mystery solved. Have you done this systematically, i.e. not only set the pvid of the bond interface to 166 but also moved the bond to the untagged list on the /interface bridge vlan row with vlan-ids=166? If yes, this explains why you can see the response frames tagless on ether5, 9, 10 (but not why you can see them on all three).

No, I didn't move bond1 to untagged list of vlan. I think it could have undesired effects.
This is what I have:

/interface bridge
add admin-mac=48:8F:5A:8A:27:66 auto-mac=no name=bridge1 vlan-filtering=yes
/interface bridge port
add comment=defconf interface=ether1
add comment=defconf interface=ether2
add comment=defconf interface=ether3
add comment=defconf interface=ether4
add comment=defconf interface=ether6
add comment=defconf interface=ether7
add comment=defconf interface=ether8
add bridge=bridge1 interface=bond1 pvid=166 trusted=yes
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=7
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=166
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=88

Reviewing the thread, I see where I was sniffing bridge on basebox and it showed VLAN=1, however these packets are not received tagged at CS326 (sniffed today using a mirror port, the packets are not 802.1q encapsulated so I guess that VLAN=1 is added by the sniffer tool.
This would be weird too, I have never seen something like this on any device, including mipsbe ones (which is the CPU architecture of Basebox 2). So tell me more about how do you sniff on the mirror port. The thing is that most Windows network card drivers strip any VLAN tag before forwarding the frame further, so even the sniffing API already gets a modified packet. I have no idea how Apple does that, but some Linux drivers reportedly do the same.

I have connected my macbook to the mirror port, then used tcpdump

sindy · Thu Jan 07, 2021 9:44 pm

Unfortunately,

/interface bridge port
...
add bridge=bridge1 interface=bond1 pvid=166 trusted=yes
/interface bridge vlan
...
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=166

is an undocumented configuration. Theoretically, it should mean that tagless frames get tagged with VID 166 on ingress whereas the tag is not stripped from frames with VID 166 on egress, but I've never tried whether it really works this way or whether the pvid setting wins also on egress.

I have connected my macbook to the mirror port, then used tcpdump

I have no clue how the macbook handles VLAN tags, and I don't have one handy. So I'd say feed it with some traffic which you know for sure to contain tagged frames and see whether the tcpdump shows the tags or not. Other than that, I assume you've watched the result using Wireshark, or that you know that you have to run tcpdump with -e to make it show the L2 information (MAC addresses, ethertype, VLAN ID) in the text output (the binary output saved to file using -w doesn't need this, it always contains all headers).

In general, it is hard for me to believe that there are so many mutually unrelated bugs causing all this mess. I know it's three different operating systems (ROS 6.45.9, ROS 7.1beta3, SwOS), but still...

leopardus2 · Thu Jan 07, 2021 9:54 pm

Unfortunately,
Code: Select all
/interface bridge port
...
add bridge=bridge1 interface=bond1 pvid=166 trusted=yes
/interface bridge vlan
...
add bridge=bridge1 tagged=bridge1,bond1 vlan-ids=166
is an undocumented configuration. Theoretically, it should mean that tagless frames get tagged with VID 166 on ingress whereas the tag is not stripped from frames with VID 166 on egress, but I've never tried whether it really works this way or whether the pvid setting wins also on egress.

I don't want to strip the tag on egress, why would I do that? I would lose access to vlan 166 from the bridge wouldn't I?
But I do want to tag untagged packets on ingress.

I have connected my macbook to the mirror port, then used tcpdump
I have no clue how the macbook handles VLAN tags, and I don't have one handy. So I'd say feed it with some traffic which you know for sure to contain tagged frames and see whether the tcpdump shows the tags or not. Other than that, I assume you've watched the result using Wireshark, or that you know that you have to run tcpdump with -e to make it show the L2 information (MAC addresses, ethertype, VLAN ID) in the text output (the binary output saved to file using -w doesn't need this, it always contains all headers).

A capture is a capture, macbook does the same as a linux box. I watched the results on screen (no need to use wireshark , and yes I used option -vvv -e to show the layer 2 details )

sindy · Thu Jan 07, 2021 10:08 pm

I don't want to strip the tag on egress, why would I do that? I would lose access to vlan 166 from the bridge wouldn't I?

Not necessarily, as you restore the tag at the far end (CSS326). If you can see some frames with VID 166 to egress through ether5, 9, 10, you can be sure the tag stripping does not happen in general and is limited to the DHCP server responses.

A capture is a capture, macbook does the same as a linux box. I watched the results on screen (no need to use wireshark , and yes I used option -vvv -e to show the layer 2 details )

I know that MacOS is a unix too, but I'm not sure about the network card driver. Some drivers on Linux reportedly (I haven't experienced that myself) strip the VLAN tags on the capture interface. So again - to be sure, you have to capture at least a single frame with the 802.1Q header this way.

leopardus2 · Thu Jan 07, 2021 10:34 pm

A capture is a capture, macbook does the same as a linux box. I watched the results on screen (no need to use wireshark , and yes I used option -vvv -e to show the layer 2 details )
I know that MacOS is a unix too, but I'm not sure about the network card driver. Some drivers on Linux reportedly (I haven't experienced that myself) strip the VLAN tags on the capture interface. So again - to be sure, you have to capture at least a single frame with the 802.1Q header this way.

Nothing is stripped. I do see 802.1q packets from other traffic when I sniff. That's why I'm sure the DHCP discover is sent without encapsulation.

sindy · Thu Jan 07, 2021 10:40 pm

Nothing is stripped. I do see 802.1q packets from other traffic when I sniff. That's why I'm sure the DHCP discover is sent without encapsulation.

Good. Now it would be fine to temporarily connect the cable from the Basebox directly to the Macbook and see what the Basebox actually sends. I'm not sure that the port mirroring on the CSS is fully transparent, and you suspect that the RouterOS sniffer running on the Basebox makes up the VLAN tags in the sniff.

leopardus2 · Thu Jan 07, 2021 10:52 pm

Nothing is stripped. I do see 802.1q packets from other traffic when I sniff. That's why I'm sure the DHCP discover is sent without encapsulation.
Good. Now it would be fine to temporarily connect the cable from the Basebox directly to the Macbook and see what the Basebox actually sends. I'm not sure that the port mirroring on the CSS is fully transparent, and you suspect that the RouterOS sniffer running on the Basebox makes up the VLAN tags in the sniff.

basebox in in use, and runs entire home's IoT so I can't do that unfortunately. every single light in this house works through wireless.

sindy · Fri Jan 08, 2021 12:02 am

basebox in in use, and runs entire home's IoT so I can't do that unfortunately. every single light in this house works through wireless.

If you can't do this for even two minutes, I'd be concerned what happens if the Basebox breaks for some reason 🙂 I can't expect you to have a plain old hub, but on a dumb switch, the DHCPDISCOVER will be broadcast to all ports so it should be sufficient too.

I'll replicate your Basebox setup on a mipsbe device with wireless once I find a free one which I'll be able to downgrade to 6.45.9, but it may take some days.

leopardus2 · Fri Jan 08, 2021 12:56 am

I think I know what happened. When I originally sniffed on basebox AP, I had another DHCP server running on VLAN88 (a VLAN I use to install Mikrotik devices).
The capture shows DHCP traffic coming from 192.168.88.1 which was an hAP I was installing. So the VLAN id 1 was very likely traffic originated there, arriving on basebox
tagged with 1 because the access port wouldn't allow the 88 tag so it becomes 1, is that a possibility?

I have temporarily re-enabled the DHCP server on basebox and re-sniffed on the basebox right now, and...

[admin@schifo_AP] > /tool sniffer quick interface=bridge-local port 67
INTERFACE        TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP 
bridge-local   68.128      1 <-  8C:85:90:B9:C8:95 FF:FF:FF:FF:FF:FF        0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      342   0 no 
bridge-local    68.13      2 <-  C4:AD:34:C0:32:87 FF:FF:FF:FF:FF:FF        192.168.166.246:67 (bootps)         255.255.255.255:68 (bootpc)         ip:udp      342   0 no 
bridge-local   68.373      3 <-  8C:85:90:B9:C8:95 FF:FF:FF:FF:FF:FF        0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      342   0 no 
bridge-local   68.908      4 ->  4C:5E:0C:1C:C6:C1 8C:85:90:B9:C8:95        192.168.166.253:67 (bootps)         192.168.166.5:68 (bootpc)           ip:udp      342   0 no 
bridge-local   70.013      5 <-  8C:85:90:B9:C8:95 FF:FF:FF:FF:FF:FF        0.0.0.0:68 (bootpc)                 255.255.255.255:67 (bootps)         ip:udp      342   0 no 
bridge-local   70.015      6 ->  4C:5E:0C:1C:C6:C1 8C:85:90:B9:C8:95        192.168.166.253:67 (bootps)         192.168.166.5:68 (bootpc)           ip:udp      342   0 no

... no VLAN ID...

sindy · Fri Jan 08, 2021 5:46 pm

The capture shows DHCP traffic coming from 192.168.88.1 which was an hAP I was installing. So the VLAN id 1 was very likely traffic originated there, arriving on basebox tagged with 1 because the access port wouldn't allow the 88 tag so it becomes 1, is that a possibility?

It doesn't sound likely unless there was yet another bug.

Since the port at CSS326 to which the Basebox is connected has PVID 166, whatever has other VID than 166 and leaves through that port will get tagged, including VID 1. The vlan-filtering is set to no at the basebox, hence no tagging and untagging takes place on egress and ingress, so what comes with any tag through a port remains tagged on the bridge.

So as you apparently haven't analyzed the MAC addresses during the previous sniffing at the Basebox, it is well possible that what we have seen there with dst-mac=ff:ff:ff:ff:ff:ff and tagged with VID 1 were actually the DHCP responses (DHCPOFFER) which the 3011 was sending tagless, hence they landed in VLAN 1 on the CSSs as 1 is the default VLAN on the 3011-facing ports of the CSS610 and on both ends of the link between the CSS, and the tag with VID 1 has been added at egress from CSS326 to Basebox.

Since you have added the ACL rule in the meantime, the tagless DHCPOFFER from the 3011 now gets VID 1 as it enters the CSS610, but that gets rewritten to 166 by the ACL (at the 610 or at the 326, no idea where you've set that ACL), so when it reaches the CSS326's Basebox-facing port, it gets untagged there on egress and lands untagged on the bridge in Basebox.

leopardus2 · Fri Jan 08, 2021 6:07 pm

I just meant to say that the DHCP discover (from client connected to basebox) is sent UNTAGGED and not with vlan=1, and that the former test at beginning of this thread
should be disregarded. So it is not worth spending your time retesting this....

sindy · Fri Jan 08, 2021 6:28 pm

I just meant to say that the DHCP discover (from client connected to basebox) is sent UNTAGGED and not with vlan=1, and that the former test at beginning of this thread should be disregarded.

Yeah, which means that there is nothing wrong with the Basebox, and the bugs are limited to the CSS and the 3011 running 7.1beta3.

leopardus2 · Fri Jan 08, 2021 7:15 pm

I just meant to say that the DHCP discover (from client connected to basebox) is sent UNTAGGED and not with vlan=1, and that the former test at beginning of this thread should be disregarded.
Yeah, which means that there is nothing wrong with the Basebox, and the bugs are limited to the CSS and the 3011 running 7.1beta3.

I think the bug is in the CSS326. Ingress DHCP packets should be tagged when force vlan id is set and they aren't.
RB3011 and ROS7 doesn't seem to be part of the problem at this point.

leopardus2 · Tue Jan 12, 2021 10:15 am

Just got a message from Mikrotik support that confirms this is a bug, not on the CSS326 but on the CSS610:

"When the "Add Information Option" option under the System menu is enabled, it will remove the VLAN header for the DHCP packets. A workaround is to disable this option. "

I have disabled that option, removed the ACL and everything works properly now.

Who is online