Client behind CRS switch unable to get VLAN DHCP from CCR Router

avggeek · Thu Nov 11, 2021 12:05 pm

Hello,

I've been looking at various threads on getting DHCP clients on a VLAN behind a CRS switch to work correctly and I'll admit that I'm a bit lost now. Here's an image that shows my network setup between a CRS-309 and a CCR-1009:

I had a lot of teething troubles getting basic connectivity working (which I posted on another thread) but that is resolved now. My current symptoms are a client PC connected via SFP-1 to the CRS is not able to get a DHCP address from a VLAN interface on the CCR-1009. Based on pcunite's thread here, I've configured the bridge on the CRS & CCR as follows:

CRS config:

/interface bridge
add name=bridge1 protocol-mode=mstp vlan-filtering=yes
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1 pvid=104
add bridge=bridge1 interface=sfp-sfpplus8
/interface bridge vlan
add bridge=bridge1 tagged=sfp-sfpplus8 vlan-ids=104
/ip address
add address=192.168.88.20/24 interface=bridge1 network=192.168.88.0
/ip route
add distance=1 gateway=192.168.88.3
/system routerboard settings
set boot-os=router-os

CCR config:

/interface bridge
add admin-mac=4C:5E:0C:03:20:22 auto-mac=no fast-forward=no name=\
    "Direct Clients Bridge" protocol-mode=mstp vlan-filtering=yes
/interface bridge port
add bridge="Direct Clients Bridge" interface="ether1-switch master"
add bridge="Direct Clients Bridge" interface=ether2
add bridge="Direct Clients Bridge" interface=ether3
add bridge="Direct Clients Bridge" interface=ether4
add bridge="Direct Clients Bridge" interface=sfp-sfpplus1
/interface bridge vlan
add bridge="Direct Clients Bridge" tagged="sfp-sfpplus1,Direct Clients Bridge,\
    ether1-switch master,ether2,ether3,ether4" vlan-ids=104
/ip neighbor discovery-settings
set discover-interface-list=!WAN-All
/interface detect-internet
set detect-interface-list=WAN-All internet-interface-list=WAN-Native \
    lan-interface-list=LAN wan-interface-list=WAN-All
/interface vrrp
add authentication=ah comment="VLAN 1 Network" interface=\
    "Home Network Trunk Ports" name=mgmt-net-vrrp priority=250 version=2 \
    vrid=48
/interface vrrp
add authentication=ah interface=dmz-net name=dmz-net-vrrp priority=250 \
    version=2 vrid=122
add authentication=ah interface=iot-net name=iot-net-vrrp priority=250 \
    version=2 vrid=50
add authentication=ah interface=lab-vlan name=lab-vlan-vrrp priority=250 \
    version=2 vrid=54
add authentication=ah interface=others-net name=others-net-vrrp priority=250 \
    version=2 vrid=75
add authentication=ah interface=server-net name=server-net-vrrp on-master=\
    " /system script run force-update-odns" priority=250 version=2 vrid=20
add authentication=ah interface=trusted-clients-net name=trusted-clients-vrrp \
    priority=250 version=2 vrid=104
/interface vlan
add comment="DMZ Network" interface="Home Network Trunk Ports" name=dmz-net \
    vlan-id=122
add comment="Guest network" disabled=yes interface="Home Network Trunk Ports" \
    name=guest-net vlan-id=90
add comment="IOT Devices Network" interface="Home Network Trunk Ports" name=\
    iot-net vlan-id=50
add comment="Lab Network" interface="Home Network Trunk Ports" name=lab-vlan \
    vlan-id=54
add comment="Untrusted Client(s) network" interface=\
    "Home Network Trunk Ports" name=others-net vlan-id=75
add comment="Server network" interface="Home Network Trunk Ports" name=\
    server-net vlan-id=20
add comment="\"Trusted\" clients network" interface=\
    "Home Network Trunk Ports" name=trusted-clients-net vlan-id=104
/interface list
add name=WAN-All
add name=LAN
add name=WAN-Native
/ip dhcp-server option
add code=15 name=labdomain value="'lab.domain.tld'"
add code=119 name=domainsearch value=\
    "s'srv.domain.tld,clients.domain.tld,lab.domain.tld'"
add code=15 name=clientdomain value="'clients.domain.tld'"
add code=15 name=srvdomain value="'srv.domain.tld'"
add code=15 name=mgmtdomain value="'mgmt.domain.tld'"
add code=15 name=iotdomain value="'iot.domain.tld'"
/ip dhcp-server option sets
add name=lab-dhcp-options options=labdomain,domainsearch
add name=client-dhcp-options options=clientdomain,domainsearch
add name=srv-dhcp-options options=srvdomain,domainsearch
add name=mgmt-dhcp-options options=mgmtdomain
/system logging action
add disk-file-count=4 disk-file-name=disk1/log disk-lines-per-file=2000 name=\
    sdcard target=disk
/interface list member
add interface=ether8-gateway list=WAN-All
add interface="Direct Clients Bridge" list=LAN
add interface="Home Network Trunk Ports" list=LAN
add interface=dmz-net list=LAN
add interface="ether1-switch master" list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=guest-net list=LAN
add interface=iot-net list=LAN
add interface=others-net list=LAN
add interface=server-net list=LAN
add interface=trusted-clients-net list=LAN
add interface=lab-vlan list=LAN
add interface=dmz-net-vrrp list=LAN
add interface=iot-net-vrrp list=LAN
add interface=mgmt-net-vrrp list=LAN
add interface=others-net-vrrp list=LAN
add interface=server-net-vrrp list=LAN
add interface=lab-vlan-vrrp list=LAN
add interface=trusted-clients-vrrp list=LAN
add interface=l2tp-evpn-IN list=WAN-All
add interface=l2tp-evpn-US list=WAN-All
add interface=ether8-gateway list=WAN-Native
/ip address
add address=192.168.88.3/24 comment="Address for directly attached clients" \
    interface="Direct Clients Bridge" network=192.168.88.0
add address=192.168.48.1/24 comment="Management network" interface=\
    "Home Network Trunk Ports" network=192.168.48.0
add address=192.168.54.1/24 comment="Lab Network" interface=lab-vlan network=\
    192.168.54.0
add address=192.168.20.1/24 comment="Server Network" interface=server-net \
    network=192.168.20.0
add address=192.168.50.1/24 comment="IOT Network" interface=iot-net network=\
    192.168.50.0
add address=192.168.75.1/24 comment="Untrusted Clients" interface=others-net \
    network=192.168.75.0
add address=192.168.90.1/24 comment="Guest Network (Inactive)" disabled=yes \
    interface=guest-net network=192.168.90.0
add address=192.168.104.1/24 comment="Trusted Clients" interface=\
    trusted-clients-net network=192.168.104.0
add address=192.168.122.1/24 comment="DMZ Network" interface=dmz-net network=\
    192.168.122.0
add address=192.168.54.254 interface=lab-vlan-vrrp network=192.168.54.254
add address=192.168.20.254 interface=server-net-vrrp network=192.168.20.254
add address=192.168.48.254 interface=mgmt-net-vrrp network=192.168.48.254
add address=192.168.50.254 interface=iot-net-vrrp network=192.168.50.254
add address=192.168.75.254 interface=others-net-vrrp network=192.168.75.254
add address=192.168.104.254 interface=trusted-clients-vrrp network=\
    192.168.104.254
add address=192.168.122.254 interface=dmz-net-vrrp network=192.168.122.254
add address=192.168.50.19 disabled=yes interface=iot-net network=\
    192.168.50.19
add address=192.168.75.19 disabled=yes interface=others-net network=\
    192.168.75.19
/ip arp
add address=192.168.48.255 comment="Broadcast MAC for WOL" interface=\
    "Home Network Trunk Ports" mac-address=FF:FF:FF:FF:FF:FF
add address=192.168.48.51 interface="Home Network Trunk Ports" mac-address=\
    FC:EC:DA:3A:96:66
add address=192.168.48.50 interface="Home Network Trunk Ports" mac-address=\
    FC:EC:DA:3A:9A:8B
/ip dhcp-client
add disabled=no interface=ether8-gateway use-peer-dns=no use-peer-ntp=no
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN-All
add action=drop chain=input comment="Block Winbox connections on WAN" \
    dst-port=8291 in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment=\
    "Block Mikrotik Bandwidth Test connections on WAN" dst-port=2000 \
    in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound TCP DNS" dst-port=53 \
    in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment=\
    "Drop all packets which does not have unicast source IP address" \
    src-address-type=!unicast
add action=drop chain=input comment="Drop all packets from public internet whi\
    ch should not exist in public network" in-interface-list=WAN-All \
    src-address-list=NotPublic
add action=fasttrack-connection chain=forward connection-state=\
    established,related dst-address-list=!External-Client src-address-list=\
    !External-Client
add action=drop chain=forward log-prefix=DMZ protocol=tcp src-address=\
    192.168.122.90 src-port=!443
add action=drop chain=forward comment="Drop all packets from public internet w\
    hich should not exist in public network" in-interface-list=WAN-All \
    src-address-list=NotPublic
add action=drop chain=forward comment="Drop all packets in local network which\
    \_does not have local network address" in-interface-list=LAN src-address=\
    !192.168.0.0/16
add action=drop chain=forward comment=\
    "Block IOT Traffic to critical LAN Segments" connection-state=\
    invalid,new,untracked dst-address-list=Sensitive-Internal in-interface=\
    iot-net log=yes log-prefix=iot-drop src-mac-address=!90:DD:5D:CA:59:A7
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="drop all from WAN" in-interface=\
    ether8-gateway
/ip firewall mangle
add action=mark-connection chain=prerouting in-interface=others-net-vrrp \
    new-connection-mark=non-trusted passthrough=yes
# guest-net not ready
add action=mark-connection chain=prerouting in-interface=guest-net \
    new-connection-mark=non-trusted passthrough=yes
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for fast.com (MyRepublic)" dst-port=53 layer7-protocol=\
    Netflix-Fast log-prefix=nflx-fast new-connection-mark=nflx-fast \
    passthrough=yes protocol=udp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Netflix Domains" dst-port=53 layer7-protocol=Netflix \
    log-prefix=nflx-conn new-connection-mark=nflx-conn passthrough=yes \
    protocol=udp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Netflix Domains" dst-port=53 layer7-protocol=Netflix \
    log-prefix=nflx-conn new-connection-mark=nflx-conn passthrough=yes \
    protocol=tcp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for IOT Devices" dst-address=!192.168.50.19 dst-port=53 \
    in-interface=iot-net-vrrp layer7-protocol=!Netflix log-prefix=iot-dns \
    new-connection-mark=iot-dns passthrough=yes protocol=udp src-address=\
    !192.168.50.19 src-address-list=""
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for IOT Devices" dst-address=!192.168.50.19 dst-port=53 \
    in-interface=iot-net-vrrp layer7-protocol=!Netflix new-connection-mark=\
    iot-dns passthrough=yes protocol=tcp src-address=!192.168.50.19 \
    src-address-list=""
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Lab Devices" dst-address=!192.168.54.230 dst-port=53 \
    in-interface=lab-vlan-vrrp log-prefix=lab-dns new-connection-mark=lab-dns \
    passthrough=yes protocol=udp src-address-list=!Lab-Whitelist
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    out-interface-list=WAN-All
add action=dst-nat chain=dstnat comment=Plex dst-port=32400 in-interface=\
    ether8-gateway protocol=tcp to-addresses=192.168.48.100 to-ports=32400
add action=dst-nat chain=dstnat comment="Forward to jumphost SSL Multiplexer" \
    dst-port=443 in-interface=ether8-gateway protocol=tcp to-addresses=\
    192.168.122.90 to-ports=443
add action=dst-nat chain=dstnat comment="Forward to jumphost UDP OpenVPN " \
    dst-port=1194 in-interface=ether8-gateway log-prefix=udp-vpn protocol=udp \
    to-addresses=192.168.122.91 to-ports=1194
add action=dst-nat chain=dstnat comment="Redirect IOT UDP DNS to PiHole" \
    connection-mark=iot-dns dst-port=53 log-prefix=iot-dns-nat protocol=udp \
    to-addresses=192.168.50.19 to-ports=53
add action=masquerade chain=srcnat comment="Redirect IOT UDP DNS to PiHole" \
    connection-mark=iot-dns dst-address=192.168.50.19 dst-port=53 log-prefix=\
    iot-dns-masq protocol=udp src-address=192.168.50.0/24
add action=dst-nat chain=dstnat comment="Redirect IOT TCP DNS to PiHole" \
    connection-mark=iot-dns dst-port=53 log-prefix=iot-dns-nat protocol=tcp \
    to-addresses=192.168.50.19 to-ports=53
add action=masquerade chain=srcnat comment="Redirect IOT TCP DNS to PiHole" \
    connection-mark=iot-dns dst-address=192.168.50.19 dst-port=53 log-prefix=\
    iot-dns-masq protocol=tcp src-address=192.168.50.0/24
add action=dst-nat chain=dstnat comment="Redirect Lab UDP DNS to PiHole" \
    connection-mark=lab-dns dst-port=53 log-prefix=lab-dns-nat protocol=udp \
    to-addresses=192.168.54.230 to-ports=53
add action=masquerade chain=srcnat comment="Redirect Lab UDP DNS to PiHole" \
    connection-mark=lab-dns dst-address=192.168.54.230 dst-port=53 \
    log-prefix=lab-dns-masq protocol=udp src-address=192.168.54.0/24
add action=dst-nat chain=dstnat comment=\
    "Redirect UDP DNS for Netflix to Express VPN Mediastreamer" \
    connection-mark=nflx-conn dst-port=53 protocol=udp to-addresses=\
    85.203.37.1 to-ports=53
add action=dst-nat chain=dstnat comment=\
    "Redirect TCP DNS for Netflix to Express VPN Mediastreamer" \
    connection-mark=nflx-conn dst-port=53 protocol=tcp to-addresses=\
    85.203.37.1 to-ports=53
add action=dst-nat chain=dstnat comment=Steam_Big-Rig dst-port=27015-27030 \
    in-interface=ether8-gateway port="" protocol=tcp to-addresses=\
    192.168.104.101 to-ports=27015-27030
add action=dst-nat chain=dstnat comment=Steam_Big-Rig dst-port=27000-27030 \
    in-interface=ether8-gateway protocol=udp to-addresses=192.168.104.101 \
    to-ports=27000-27030
add action=dst-nat chain=dstnat comment=Steam_Big-Rig dst-port=3478 \
    in-interface=ether8-gateway protocol=udp to-addresses=192.168.104.101 \
    to-ports=3478
/ip route
add comment="VPN TCP" distance=1 dst-address=192.168.126.0/24 gateway=\
    192.168.48.174
add comment="VPN UDP" distance=1 dst-address=192.168.166.0/24 gateway=\
    192.168.48.174

I've run Sniffer against the bridge on the CCR-1009 that has the SFP+ patch port from the CRS and in Wireshark I can see that the DHCP DISCOVER request reaches the Bridge but there is no return traffic. Screenshot of wireshark traffic:

Does anybody have any suggestions on what I'm doing wrong?

mkx · Thu Nov 11, 2021 1:58 pm

CCR config is far from trivial ... if you take VRRP out of picture, does DHCP for clients still fail? CRS config is simple enough not to allow for too many mistakes and by look of it it's fine.

avggeek · Thu Nov 11, 2021 4:28 pm

CCR config is far from trivial ... if you take VRRP out of picture, does DHCP for clients still fail? CRS config is simple enough not to allow for too many mistakes and by look of it it's fine.

I created a new VLAN interface (VLAN 200) under "Direct Clients Bridge" (which the CRS is patched into) with a new DHCP Server in the 192.168.200.0/24 range. When I changed the VLAN ID in the CRS Bridge & CCR Bridge to 200, I succeeded in getting an IP address on the client.

So is it an issue with my other VLAN(s) which are not directly on the bridge in the CCR that the CRS is attached to?

Edit: There's definitely something wrong with the config even in this setup. An iPerf test between the client device (connected with a 10Gig network card) and another machine (on a 1 Gig ethernet connection) is showing speeds of around 500 Mbps.

mkx · Thu Nov 11, 2021 6:34 pm

It seems that many Mikrotik switches don't like high throughput traffic between ports with different speed (e.g. 10G to 1G or 1G to 100M). There has been recently a debate about same problem recently on this forum.

anav · Thu Nov 11, 2021 7:45 pm

Is the Client vlan 104 the same vlan as our management vlan/trusted ????
It looks to me like the vlan104 is the management vlan with a subnet of 192.168.104.0/24
It looks to me like the vlan for the client is 192.168.88.0/24 which is none existtant on the CCR?? (the vlan that is)

Both your configs appear to be messed up..................

/interface bridge
add name=bridge1 protocol-mode=mstp vlan-filtering=yes { I think MSTP is best for vlan traffic between two different vendors, RTSP works fine between MT devices).
MISSING VLANs IDENTIFIED AND CONFIGURED TO BRIDGE
/interface vlan
add interface=bridge1 name=vlancontrol vlanid=104
add interface=bridge1 name=clientlan vlan-id=11

/interface list
add name=control
/interface list members
add interface=vlancontrol list=control

/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1 pvid=11 {access port to pc}
add bridge=bridge1 interface=sfp-sfpplus8 {trunk port from CCR)
/interface bridge vlan (missing tagging of bridge)
add bridge=bridge1 tagged=bridge1,sfp-sfpplus8 untagged=spf-sfpplus1 vlan-ids=11
add bridge=bridge1 tagged=bridge1,sfp-sfpplus8 vlan-ids=104
/ip address
add address=192.168.88.20/24 interface=bridge1 network=192.168.88.0 {nope not on bridge and not from clien lan}
192.168.104.xx/24 interface=vlancontrol network=192.168.104.0
/ip route
add distance=1 gateway=192.168.104.1
/tool mac-server mac-winbox
set allowed-interface-list=control

anav · Thu Nov 11, 2021 7:52 pm

On the CCR to matchup...........
Pretty sure I addressed this in another post.
AGAIN I dont see any vlans identified/created on the bridge ????

I dont see the management vlan 104
Ensure you create the client vlan11 and send it as a vlan to sfp-sfplus8 trunk port.

avggeek · Fri Nov 12, 2021 8:01 am

On the CCR to matchup...........
Pretty sure I addressed this in another post.
AGAIN I dont see any vlans identified/created on the bridge ????

I dont see the management vlan 104
Ensure you create the client vlan11 and send it as a vlan to sfp-sfplus8 trunk port.

So here is a visual representation of the network configuration of my CCR including the various VLAN's & VRRP devices:

My problem is that Client 1 cannot obtain a DHCP address in VLAN 104, whereas Client 2 can. Should I understand that the only way for this to work is that all the ports on the CCR need to be one bridge and the VLAN interfaces tied to that bridge? There are plenty of articles & forum threads about port-based VLAN in Mikrotik and it seems to work, so I'm definitely missing something.

It seems that many Mikrotik switches don't like high throughput traffic between ports with different speed (e.g. 10G to 1G or 1G to 100M). There has been recently a debate about same problem recently on this forum.

This is strange. What I noticed was that when I had client A with a non VLAN IP address (192.168.88.XX in the above diagram), an iperf3 test was showing speeds of 850-900 Mbps..
I take that back. Just retried now and even on VLAN1, it's showing around 500 Mbps. I'm pretty sure I achieved closer to wirespeed earlier, but given I've made several changes to the network config (and upgraded the ROS version on the CRS), I can't replicate this anymore.

mkx · Fri Nov 12, 2021 1:08 pm

If your diagram accurately reflects the configuration, then from L2 point of view (that's broadcast domain and DHCP client uses broadcasts to reach aout to DHCP server) ports on the left side (SFP, SFP+, Eth1, Eth2, Eth3 and Eth4) are completely separate from ports on the right side (Eth5, Eth6, Eth7 and Eth8). So clients (indirectly) connected to SFP+ port can only talk to servers in VLAN 104 if CCR routes packets between separate subnets. And broadcast packets are never routed (that's the one of main points of using routers).

If you want to add left-side ports to VLAN104, then you'll have to properly bridge things together. I think the most proper way would be to add LAGG interface as bridge port and move VLAN stuff over to (vlan-enabled) bridge. BTW, where's the VRRP on the diagram?

Between which two hosts are you testing throughput? If you're testing between Client A and Client B, then there's CCR routing between them. If you want to test throughput through CRS, you have to connect two clients locally to two bridged ports on CRS and run iperf3 between them.

avggeek · Fri Nov 12, 2021 1:26 pm

If your diagram accurately reflects the configuration, then from L2 point of view (that's broadcast domain and DHCP client uses broadcasts to reach aout to DHCP server) ports on the left side (SFP, SFP+, Eth1, Eth2, Eth3 and Eth4) are completely separate from ports on the right side (Eth5, Eth6, Eth7 and Eth8). So clients (indirectly) connected to SFP+ port can only talk to servers in VLAN 104 if CCR routes packets between separate subnets. And broadcast packets are never routed (that's the one of main points of using routers).

If you want to add left-side ports to VLAN104, then you'll have to properly bridge things together. I think the most proper way would be to add LAGG interface as bridge port and move VLAN stuff over to (vlan-enabled) bridge.

The visual is pretty accurate to how things are configured right now. Thanks for the confirmation on how DHCP broadcasts works. I was looking up where bridges operate in the OSI model and that gave me a suspicion that indeed this was an L2 problem. Moving the LAGG over to the bridge and then switching the VLANs over is gonna be a real pain as I'm worried clients connected to the Unifi switch are going to stop working. And that represents about 90% of my current clients on the network. Will definitely need to do this directly connected to the CCR and I guess with a fixed IP in the 192.168.48.0/24 range?

BTW, where's the VRRP on the diagram?

They currently hang off the VLAN interfaces (little diamonds in the visual). Since the VLAN interfaces will go away when I move the LAGG to the bridge, which interface should I associate them with now? The VLAN-enabled bridge?

Between which two hosts are you testing throughput? If you're testing between Client A and Client B, then there's CCR routing between them. If you want to test throughput through CRS, you have to connect two clients locally to two bridged ports on CRS and run iperf3 between them.

Between client A and client B yeah. Will the speeds improve once I move everything onto one bridge? I don't see the CCR CPU really working right now when I test so I'm not clear where the bottleneck is. The CRS isn't in "prod" yet, so I'm not really testing speeds on that device.

anav · Fri Nov 12, 2021 3:06 pm

Yup would make vlan1 into vlan11 and remove Bridge providing any dhcp etc......... All vlans, very clean.
whatever your management vlan is........... all smart devices get iP addresses on that vlan.

I know unifi APs are funny in that they like hybrid ports, in other words they will accept (default setup) the management vlan coming into the device untagged, while the rest of the data vlans are tagged.
Not sure if unifi switches are similarly stewpid???

What is not clear is how ether5 and ether6 on the ccr can both be two LAGG ports but also attached to the UNIFI on two ports?

mkx · Fri Nov 12, 2021 4:09 pm

BTW, where's the VRRP on the diagram?

They currently hang off the VLAN interfaces (little diamonds in the visual). Since the VLAN interfaces will go away when I move the LAGG to the bridge, which interface should I associate them with now? The VLAN-enabled bridge?

Basic idea about VLAN-aware bridge is that you keep using VLAN interfaces in exactly the same manner as you do now, the only difference is in the way VLAN interfaces get created.

Currently:

add comment="\"Trusted\" clients network" interface="Home Network Trunk Ports" name=trusted-clients-net vlan-id=104

With bridge:

add comment="\"Trusted\" clients network" interface="Direct Clients Bridge" name=trusted-clients-net vlan-id=104

Anyway, I'm somehow getting feeling that your L2 setup on CCR is overly complicated. I'm guessing VRRP is used to somehow add L3 (IP) setup to VLANs used in communication with Unifi while simply setting IP settings on VLAN interfaces (which are currently used by VRRP) would do the job just fine.

Between which two hosts are you testing throughput? If you're testing between Client A and Client B, then there's CCR routing between them. If you want to test throughput through CRS, you have to connect two clients locally to two bridged ports on CRS and run iperf3 between them.

Between client A and client B yeah. Will the speeds improve once I move everything onto one bridge? I don't see the CCR CPU really working right now when I test so I'm not clear where the bottleneck is. The CRS isn't in "prod" yet, so I'm not really testing speeds on that device.

CCR, depending on exact model (some models, e.g. CCR1009-8G-1S-1S+, have switch chip running ports ether1-ether4), can either switch or bridge ports. Even for bridging ports it should be able to do some Gbps (combined on all ports). So when bridging between two RJ45 1Gbps ports, it should be able to do it wire-speed. When bridging between SFP+ port and the rest it should be able to do it at around 1Gbps.

However, CCR is primarily router ... you really should use switch (CRS) for any heavy intra-subnet communications.

BTW, when doing TCP tests with iperf3, results can sometimes be heavily influenced (read: reduced) due some tiny timing issues. You might want to try UDP test, as test result always take stats from receiver's side (by default, receiver is iperf3 instance running in server mode). UDP as protocol doesn't have any feedback and sender always transmits as fast as it can ... only receiver can show real end-to-end throughput by accounting dropped packets.

avggeek · Sat Nov 13, 2021 1:56 pm

Yup would make vlan1 into vlan11 and remove Bridge providing any dhcp etc......... All vlans, very clean.
whatever your management vlan is........... all smart devices get iP addresses on that vlan.

At this point there is so much configuration tied to the VLAN1 subnet (192.168.48.0/24) that I'm really very reluctant to make such a wholesale change

I know unifi APs are funny in that they like hybrid ports, in other words they will accept (default setup) the management vlan coming into the device untagged, while the rest of the data vlans are tagged.
Not sure if unifi switches are similarly stewpid???

Yup, Switches and AP's work the same way and hence my reluctance to change things. Trying to get Unifi devices to re-adopt if they fall out of the network is extremely painful.

What is not clear is how ether5 and ether6 on the ccr can both be two LAGG ports but also attached to the UNIFI on two ports?

Maybe I'm using the term wrongly, but I have Ether 5 and Ether 6 configured as a Bond interface in the CCR and this is what I'm referring to as a single LAGG interface. They are connected to the SFP ports on the Unifi switch using SFP-RJ45 transceivers on the Unifi, with the SFP ports on the Unifi also configured as a bonded interface. Hopefully that makes things clearer.

avggeek · Sat Nov 13, 2021 2:33 pm

Basic idea about VLAN-aware bridge is that you keep using VLAN interfaces in exactly the same manner as you do now, the only difference is in the way VLAN interfaces get created.
[snip]
With bridge:
Code: Select all
add comment="\"Trusted\" clients network" interface="Direct Clients Bridge" name=trusted-clients-net vlan-id=104

So earlier today, I moved all the Ethernet ports on the CCR into a single bridge and assigned this bridge as the parent for all the VLAN's. I had a little bit of trouble getting the VLAN1 configuration to migrate over but eventually got that sorted out as well. After I made this change, Client A behind the CRS could get an IP in the 192.168.104.0/24 subnet just like Client B behind the Unifi switch - woo hoo! Thanks for the tips @mkx @anav!

Updated visual for reference:

Anyway, I'm somehow getting feeling that your L2 setup on CCR is overly complicated. I'm guessing VRRP is used to somehow add L3 (IP) setup to VLANs used in communication with Unifi while simply setting IP settings on VLAN interfaces (which are currently used by VRRP) would do the job just fine.

Perhaps I've misunderstood how VRRP interfaces work. The VRRP interfaces are used to provide WAN failover between the CCR and the RB750Gr3 (refer visual above) for all the clients behind the Unifi switch (and now the CRS as well). The VRRP interface has a very different configuration from the VLAN interface. Are you suggesting that the IP address assigned to the VRRP interface should be the same as that of the VLAN interface? I assume at minimum it does need an IP address.

BTW, when doing TCP tests with iperf3, results can sometimes be heavily influenced (read: reduced) due some tiny timing issues. You might want to try UDP test, as test result always take stats from receiver's side (by default, receiver is iperf3 instance running in server mode). UDP as protocol doesn't have any feedback and sender always transmits as fast as it can ... only receiver can show real end-to-end throughput by accounting dropped packets.

Thanks for the tips! Although I'm having a bigger issue just in terms of how the CRS is handling traffic..

However, CCR is primarily router ... you really should use switch (CRS) for any heavy intra-subnet communications.

Since I could get an IP address on Client A when connected to the CRS (see visual above), I tried to add another Client - call it Client C. I made sure that Client A and Client C are getting an IP in the same subnet (192.168.104.0/24) and indeed, I've got Client C configured to connect via a Bonded interface to the CRS. However, when I run a iPerf test from Client A to Client C, I can very clearly see that all the traffic is going onto the SFP+ port on the CCR. In other words, the CRS isn't doing anything and it's getting bottlenecked at the CCR!

I can post my updated config here, or I can start a new thread - what is the recommended way here?

tdw · Sat Nov 13, 2021 3:05 pm

Perhaps I've misunderstood how VRRP interfaces work. The VRRP interfaces are used to provide WAN failover between the CCR and the RB750Gr3 (refer visual above) for all the clients behind the Unifi switch (and now the CRS as well).

Your diagram doesn't show the physical connection to the RB750Gr3, does it have separate physical connection to the CRS & UniFi switch? If not it is pointless having the complexities and issues of VRRP as a failure of the CCR would make it unreachable.

Since I could get an IP address on Client A when connected to the CRS (see visual above), I tried to add another Client - call it Client C. I made sure that Client A and Client C are getting an IP in the same subnet (192.168.104.0/24) and indeed, I've got Client C configured to connect via a Bonded interface to the CRS. However, when I run a iPerf test from Client A to Client C, I can very clearly see that all the traffic is going onto the SFP+ port on the CCR. In other words, the CRS isn't doing anything and it's getting bottlenecked at the CCR!

I can post my updated config here, or I can start a new thread - what is the recommended way here?

Intra-subnet traffic (i.e. source and destination within the same subnet) should be switched by the CRS, inter-subnet traffic (i.e. source and destination in different subnets) will end up at the CCR to be routed between the subnets. As this is rather divergent from the original issue I'd suggest starting a new topic.

avggeek · Sat Nov 13, 2021 3:56 pm

Your diagram doesn't show the physical connection to the RB750Gr3, does it have separate physical connection to the CRS & UniFi switch? If not it is pointless having the complexities and issues of VRRP as a failure of the CCR would make it unreachable.

The RB750Gr3 does have a separate physical connection to the Unifi switch. Agree that there is still a problem for the CRS as it isn't physically connected to the RB750Gr3. But my current layout gives me reasonable WAN failover in case of ISP maintenance etc which is sufficient for now.

Intra-subnet traffic (i.e. source and destination within the same subnet) should be switched by the CRS, inter-subnet traffic (i.e. source and destination in different subnets) will end up at the CCR to be routed between the subnets. As this is rather divergent from the original issue I'd suggest starting a new topic.

Ok noted. I'll start a separate thread with the details.

tdw · Sat Nov 13, 2021 8:35 pm

Your diagram doesn't show the physical connection to the RB750Gr3, does it have separate physical connection to the CRS & UniFi switch? If not it is pointless having the complexities and issues of VRRP as a failure of the CCR would make it unreachable.

The RB750Gr3 does have a separate physical connection to the Unifi switch. Agree that there is still a problem for the CRS as it isn't physically connected to the RB750Gr3. But my current layout gives me reasonable WAN failover in case of ISP maintenance etc which is sufficient for now.

If both the CCR & RB750Gr3 are operating with the master VRRP interface presumably on the CCR, then if the WAN attached to that fails do you have scripts to change the priority to make the RB750Gr3 take over, or are there static routes between the two routers with traffic being tromboned back out of the VRRP master to the other Mikrotik. You also have to ensure the DHCP pools on the routers do not overlap as there is no sync of leases between the VRRP instances. If the main concern is a WAN failure then a simple dual-WAN configuration on the CCR would suffice.

avggeek · Sun Nov 14, 2021 11:56 am

If both the CCR & RB750Gr3 are operating with the master VRRP interface presumably on the CCR, then if the WAN attached to that fails do you have scripts to change the priority to make the RB750Gr3 take over

There is a script running via netwatch to disable the VLAN's on one device triggering the failover.

You also have to ensure the DHCP pools on the routers do not overlap as there is no sync of leases between the VRRP instances.

Sync'ing the leases and DNS entries was on the list of things to do but...

If the main concern is a WAN failure then a simple dual-WAN configuration on the CCR would suffice.

This is a very good point! Originally the RB750Gr3 was picked up when the CCR-1009 had a hardware failure and was in a physical location where I could not get the two WAN connections in the same place as the CCR-1009. However I've since reorganized things and indeed now I can connect both WAN to the CCR. I will need to do some extra cabling to ensure the CRS has connectivity to the RB750Gr3 but this should be possible. For now I'm just focused on getting the CRS to work as expected

Client behind CRS switch unable to get VLAN DHCP from CCR Router [SOLVED]

Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router [SOLVED]

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Re: Client behind CRS switch unable to get VLAN DHCP from CCR Router

Who is online