Community discussions

MikroTik App
  4. RouterOS
  5. General

Clients behind CRS on same subnet - but traffic is being switched by the Router?  [SOLVED]

Post Reply
 
avggeek
newbie
Topic Author
Posts: 48
Joined: Thu Jun 06, 2013 9:33 am

Clients behind CRS on same subnet - but traffic is being switched by the Router?

Sun Nov 14, 2021 11:43 am

So this is part 3 in a series of threads I've been posting to try and get a CRS-309 working as I'd expect and I'm hoping it is indeed the last one I need to post for a while - but thank you to folks like @mkx, @anav & @tdw for educating this noob. To begin with here is a visual that shows my overall network architecture, which allows clients behind the CRS-309 to obtain a IP from the CCR-1009 in a specific VLAN:

Image

Zooming into the CRS-309 itself, there are currently two clients connected to the switch as shown in the visual below:

Image


Here is the config export from the CRS-309 that is represented by the visual above:
Code: Select all
/interface bridge
add admin-mac=2C:C8:1B:20:06:C8 auto-mac=no comment="Clients Bridge" name=\
    bridge1 protocol-mode=mstp vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1592
set [ find default-name=sfp-sfpplus1 ] l2mtu=1592
set [ find default-name=sfp-sfpplus2 ] l2mtu=1592
set [ find default-name=sfp-sfpplus3 ] l2mtu=1592
set [ find default-name=sfp-sfpplus4 ] l2mtu=1592
set [ find default-name=sfp-sfpplus5 ] l2mtu=1592
set [ find default-name=sfp-sfpplus6 ] l2mtu=1592
set [ find default-name=sfp-sfpplus7 ] l2mtu=1592
set [ find default-name=sfp-sfpplus8 ] comment="MikroTik CCR1009 Uplink" \
    l2mtu=1592
/interface bonding
add mode=802.3ad name=bond-sfpp2_sfpp3 slaves=\
    sfp-sfpplus2,sfp-sfpplus3
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1 pvid=104
add bridge=bridge1 interface=sfp-sfpplus8
add bridge=bridge1 interface=bond-sfpp2_sfpp3 pvid=104
/interface bridge vlan
add bridge=bridge1 tagged=sfp-sfpplus8 vlan-ids=104
add bridge=bridge1 tagged=sfp-sfpplus8,bond-sfpp2_sfpp3 vlan-ids=50
add bridge=bridge1 tagged=sfp-sfpplus8,bond-sfpp2_sfpp3 vlan-ids=20
/ip address
add address=192.168.48.10/24 interface=bridge1 network=192.168.48.0
/ip dns
set servers=192.168.48.254
/ip route
add distance=1 gateway=192.168.48.254
/system clock
set time-zone-autodetect=no time-zone-name=Asia/Singapore
/system clock manual
set time-zone=+08:00
/system identity
set name="MikroTik CRS309-1G-8S+"
/system ntp client
set enabled=yes server-dns-names=0.sg.pool.ntp.org,1.sg.pool.ntp.org
/system routerboard settings
set boot-os=router-os

This is the config for the CCR-1009:
Code: Select all
/interface bridge
add admin-mac=4C:5E:0C:03:20:22 auto-mac=no fast-forward=no name=\
    "All Ports Bridge" protocol-mode=mstp vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] name="ether1-switch master" speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] comment="Unifi US-24 RJ45 SFP Uplink-1" \
    speed=100Mbps
set [ find default-name=ether6 ] comment="Unifi US-24 RJ45 SFP Uplink-2" \
    mac-address=4C:5E:0C:03:20:26 speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=ether8 ] advertise=100M-full,1000M-full comment=\
    "WAN Interface" name=ether8-gateway speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=\
    10M-full,100M-full,1000M-full,10000M-full comment=\
    "MikroTik CRS309-1G-8S+ Fibre SFP+ Uplink"
set [ find default-name=sfp1 ] advertise=10M-full,100M-full,1000M-full \
    mac-address=4C:5E:0C:03:20:22
/interface vrrp
add authentication=ah comment="VLAN 1 Network" interface="All Ports Bridge" \
    name=mgmt-net-vrrp priority=250 version=2 vrid=48
/interface vlan
add comment="DMZ network" interface="All Ports Bridge" name=dmz-net vlan-id=\
    122
add comment="Guest network" disabled=yes interface="All Ports Bridge" name=\
    guest-net vlan-id=90
add comment="IOT Devices network" interface="All Ports Bridge" name=iot-net \
    vlan-id=50
add comment="Lab network" interface="All Ports Bridge" name=lab-net vlan-id=\
    54
add comment="Untrusted Client(s) network" interface="All Ports Bridge" name=\
    others-net vlan-id=75
add comment="Server network" interface="All Ports Bridge" name=server-net \
    vlan-id=20
add comment="\"Trusted\" Clients network" interface="All Ports Bridge" name=\
    trusted-clients-net vlan-id=104
/interface bonding
add comment="Unifi US-24 Trunk Ports" lacp-rate=1sec mode=802.3ad name=\
    bond-ether5_ether6 slaves=ether5,ether6
/interface vrrp
add authentication=ah interface=dmz-net name=dmz-net-vrrp priority=250 \
    version=2 vrid=122
add authentication=ah interface=iot-net name=iot-net-vrrp priority=250 \
    version=2 vrid=50
add authentication=ah interface=lab-net name=lab-net-vrrp priority=250 \
    version=2 vrid=54
add authentication=ah interface=others-net name=others-net-vrrp priority=250 \
    version=2 vrid=75
add authentication=ah interface=server-net name=server-net-vrrp priority=250 version=2 vrid=20
add authentication=ah interface=trusted-clients-net name=trusted-clients-vrrp priority=250 version=2 vrid=104
/interface list
add name=WAN-All
add name=LAN
add name=WAN-Native
/ip dhcp-server option
add code=15 name=labdomain value="'lab.domain.tld'"
add code=119 name=domainsearch value=\
    "s'srv.domain.tld,clients.domain.tld,lab.domain.tld'"
add code=15 name=clientdomain value="'clients.domain.tld'"
add code=15 name=srvdomain value="'srv.domain.tld'"
add code=15 name=mgmtdomain value="'mgmt.domain.tld'"
add code=15 name=iotdomain value="'iot.domain.tld'"
/ip dhcp-server option sets
add name=lab-dhcp-options options=labdomain,domainsearch
add name=client-dhcp-options options=clientdomain,domainsearch
add name=srv-dhcp-options options=srvdomain,domainsearch
add name=mgmt-dhcp-options options=mgmtdomain
/ip pool
add comment="Management IP Range" name=mgmt-iprange ranges=\
    192.168.48.100-192.168.48.200
add comment="IP Range for Lab Network" name=lab-iprange ranges=\
    192.168.54.192-192.168.54.230
add comment="IP Range for direct attached clients" name=direct-iprange \
    ranges=192.168.88.10-192.168.88.20
add comment="IP Range for Servers" name=server-iprange ranges=\
    192.168.20.20-192.168.20.100
add comment="IP range for IOT network" name=iot-iprange ranges=\
    192.168.50.20-192.168.50.50
add comment="IP Range for untrusted clients" name=others-iprange ranges=\
    192.168.75.20-192.168.75.30
add comment="IP Range for Guest Network" name=guest-iprange ranges=\
    192.168.90.10-192.168.90.20
add comment="IP Range for \"trusted\" clients" name=trusted-iprange ranges=\
    192.168.104.100-192.168.104.200
add comment="IP Range for external facing hosts" name=dmz-iprange ranges=\
    192.168.122.90/31
/ip dhcp-server
add address-pool=mgmt-iprange disabled=no interface="All Ports Bridge" \
    lease-time=1d name=mgmt-dhcp
add address-pool=lab-iprange disabled=no interface=lab-net lease-time=1h name=lab-dhcp
add add-arp=yes address-pool=server-iprange disabled=no interface=server-net \
lease-time=3d name=server-dhcp
add address-pool=guest-iprange interface=guest-net lease-time=1h name=\
    guest-dhcp
add address-pool=iot-iprange disabled=no interface=iot-net lease-time=1d name=iot-dhcp
add address-pool=others-iprange disabled=no interface=others-net lease-time=\
    12h name=others-dhcp
add add-arp=yes address-pool=trusted-iprange disabled=no interface=\
    trusted-clients-net lease-time=1d name=trusted-dhcp
add address-pool=dmz-iprange disabled=no interface=dmz-net lease-time=1d \
    name=dmz-dhcp
/queue type
set 5 pcq-limit=1000KiB pcq-total-limit=1000KiB
set 6 pcq-limit=5000KiB pcq-total-limit=5000KiB
/queue simple
add burst-limit=5M/25M burst-threshold=5M/25M burst-time=5s/10s max-limit=\
    3M/20M name=others-net-queue queue=ethernet-default/ethernet-default \
    target=192.168.75.0/24 total-queue=ethernet-default
add burst-limit=2M/5M burst-threshold=2M/5M burst-time=5s/5s limit-at=256k/1M \
    max-limit=1M/3M name=guest-net-queue queue=\
    ethernet-default/ethernet-default target=guest-net
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/system logging action
add disk-file-count=4 disk-file-name=disk1/log disk-lines-per-file=2000 name=\
    sdcard target=disk
/interface bridge port
add bridge="All Ports Bridge" interface="ether1-switch master"
add bridge="All Ports Bridge" interface=ether2
add bridge="All Ports Bridge" interface=ether3 pvid=104
add bridge="All Ports Bridge" interface=ether4
add bridge="All Ports Bridge" comment="MikroTik CRS309-1G-8S+ Uplink" \
    interface=sfp-sfpplus1
add bridge="All Ports Bridge" interface=sfp1
add bridge="All Ports Bridge" interface=ether7
add bridge="All Ports Bridge" comment="Unifi US-24 Trunk Ports" interface=\
    bond-ether5_ether6
/ip neighbor discovery-settings
set discover-interface-list=!WAN-All
/interface bridge vlan
add bridge="All Ports Bridge" comment="\"Trusted\" Clients network" tagged="Al\
    l Ports Bridge,ether1-switch master,ether2,ether3,ether4,sfp1,sfp-sfpplus1\
    ,ether7,bond-ether5_ether6,trusted-clients-net" vlan-ids=104
add bridge="All Ports Bridge" comment="DMZ network" tagged="All Ports Bridge,e\
    ther1-switch master,ether2,ether3,ether4,sfp1,sfp-sfpplus1,ether7,bond-eth\
    er5_ether6,dmz-net" vlan-ids=122
add bridge="All Ports Bridge" comment="IOT Devices network" tagged="All Ports \
    Bridge,ether1-switch master,ether2,ether3,ether4,sfp1,sfp-sfpplus1,ether7,\
    bond-ether5_ether6,iot-net" vlan-ids=50
add bridge="All Ports Bridge" comment="Server network" tagged="All Ports Bridg\
    e,ether1-switch master,ether2,ether3,ether4,sfp1,sfp-sfpplus1,ether7,bond-\
    ether5_ether6,server-net" vlan-ids=20
add bridge="All Ports Bridge" comment="Lab network" tagged="All Ports Bridge,e\
    ther1-switch master,ether2,ether3,ether4,sfp1,sfp-sfpplus1,ether7,bond-eth\
    er5_ether6,lab-net" vlan-ids=54
add bridge="All Ports Bridge" comment="Untrusted Client(s) network" tagged="Al\
    l Ports Bridge,ether1-switch master,ether2,ether3,ether4,sfp1,sfp-sfpplus1\
    ,ether7,bond-ether5_ether6,others-net" vlan-ids=75
add bridge="All Ports Bridge" tagged=*2A vlan-ids=1
/interface detect-internet
set detect-interface-list=WAN-All internet-interface-list=WAN-Native \
    lan-interface-list=LAN wan-interface-list=WAN-All
/interface list member
add interface=ether8-gateway list=WAN-All
add interface="All Ports Bridge" list=LAN
add interface=bond-ether5_ether6 list=LAN
add interface=dmz-net list=LAN
add interface="ether1-switch master" list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=guest-net list=LAN
add interface=iot-net list=LAN
add interface=others-net list=LAN
add interface=server-net list=LAN
add interface=trusted-clients-net list=LAN
add interface=lab-net list=LAN
add interface=dmz-net-vrrp list=LAN
add interface=iot-net-vrrp list=LAN
add interface=mgmt-net-vrrp list=LAN
add interface=others-net-vrrp list=LAN
add interface=server-net-vrrp list=LAN
add interface=lab-net-vrrp list=LAN
add interface=trusted-clients-vrrp list=LAN
add interface=l2tp-evpn-IN list=WAN-All
add interface=l2tp-evpn-US list=WAN-All
add interface=ether8-gateway list=WAN-Native
/ip address
add address=192.168.48.1/24 comment="Management network" interface=\
    "All Ports Bridge" network=192.168.48.0
add address=192.168.54.1/24 comment="Lab Network" interface=lab-net network=\
    192.168.54.0
add address=192.168.20.1/24 comment="Server Network" interface=server-net \
    network=192.168.20.0
add address=192.168.50.1/24 comment="IOT Network" interface=iot-net network=\
    192.168.50.0
add address=192.168.75.1/24 comment="Untrusted Clients" interface=others-net \
    network=192.168.75.0
add address=192.168.90.1/24 comment="Guest Network (Inactive)" disabled=yes \
    interface=guest-net network=192.168.90.0
add address=192.168.104.1/24 comment="Trusted Clients" interface=\
    trusted-clients-net network=192.168.104.0
add address=192.168.122.1/24 comment="DMZ Network" interface=dmz-net network=\
    192.168.122.0
add address=192.168.54.254 interface=lab-net-vrrp network=192.168.54.254
add address=192.168.20.254 interface=server-net-vrrp network=192.168.20.254
add address=192.168.48.254 interface=mgmt-net-vrrp network=192.168.48.254
add address=192.168.50.254 interface=iot-net-vrrp network=192.168.50.254
add address=192.168.75.254 interface=others-net-vrrp network=192.168.75.254
add address=192.168.104.254 interface=trusted-clients-vrrp network=\
    192.168.104.254
add address=192.168.122.254 interface=dmz-net-vrrp network=192.168.122.254
add address=192.168.50.19 interface=iot-net network=192.168.50.19
add address=192.168.75.19 interface=others-net network=192.168.75.19
/ip arp
add address=192.168.48.255 comment="Broadcast MAC for WOL" interface=\
    bond-ether5_ether6 mac-address=FF:FF:FF:FF:FF:FF
add address=192.168.48.51 interface=bond-ether5_ether6 mac-address=\
    FC:EC:DA:3A:96:66
add address=192.168.48.50 interface=bond-ether5_ether6 mac-address=\
    FC:EC:DA:3A:9A:8B
/ip cloud
set update-time=no
/ip dhcp-client
add disabled=no interface=ether8-gateway use-peer-dns=no use-peer-ntp=no
add interface=sfp1
/ip dhcp-server lease
add address=192.168.48.10 comment=\
    "10Gbe Study Switch (Mikrotik CRS309-1G-8S+)" mac-address=\
    2C:C8:1B:20:06:C9
/ip dhcp-server network
add address=192.168.20.0/24 comment="Server VLAN IP Pool" dhcp-option-set=\
    srv-dhcp-options dns-server=192.168.20.254 domain=srv.domain.tld \
    gateway=192.168.20.254
add address=192.168.48.0/24 comment="Management IP Pool" dhcp-option-set=\
    mgmt-dhcp-options dns-server=192.168.48.254 domain=mgmt.domain.tld \
    gateway=192.168.48.254 netmask=24
add address=192.168.50.0/24 comment="IOT VLAN IP Pool" dhcp-option=iotdomain \
    dns-server=192.168.50.19 domain=iot.domain.tld gateway=192.168.50.254
add address=192.168.54.0/24 comment="Lab VLAN IP Pool" dhcp-option-set=\
    lab-dhcp-options dns-server=192.168.54.230 domain=lab.domain.tld \
    gateway=192.168.54.254
add address=192.168.75.0/24 comment="Others VLAN IP Pool" dns-server=\
    192.168.75.19 gateway=192.168.75.254
add address=192.168.90.0/24 comment="Guest VLAN IP Pool (Inactive)" \
    dns-server=8.8.8.8 gateway=192.168.90.1
add address=192.168.104.0/24 comment="\"Trusted\" VLAN IP Pool" \
    dhcp-option-set=client-dhcp-options dns-server=192.168.104.254 domain=\
    clients.domain.tld gateway=192.168.104.254
add address=192.168.122.0/24 comment="DMZ VLAN IP Pool" dns-none=yes gateway=\
    192.168.122.254
/ip dns
set allow-remote-requests=yes servers=9.9.9.9,208.67.220.222
/ip dns static
add address=192.168.88.1 name=router.lan ttl=1w3d
add address=192.168.20.1 name=ccr1009-router.srv.domain.tld ttl=1w3d
add address=192.168.50.1 name=ccr1009-router.iot.domain.tld ttl=1w3d
add address=192.168.54.1 name=ccr1009-router.lab.domain.tld ttl=1w3d
add address=192.168.75.1 name=ccr1009-router.others.domain.tld ttl=1w3d
add address=192.168.104.1 name=ccr1009-router.clients.domain.tld ttl=1w3d
add address=192.168.122.1 name=ccr1009-router.dmz.domain.tld ttl=1w3d
add address=192.168.20.254 name=router.srv.domain.tld ttl=1w3d
add address=192.168.50.254 name=router.iot.domain.tld ttl=1w3d
add address=192.168.54.254 name=router.lab.domain.tld ttl=1w3d
add address=192.168.75.254 name=router.others.domain.tld ttl=1w3d
add address=192.168.104.254 name=router.clients.domain.tld ttl=1w3d
add address=192.168.122.254 name=router.dmz.domain.tld ttl=1w3d
## Firewall rules need to be reworked!!
/ip firewall address-list
add address=192.168.88.2 comment="Exclude from PCC Example" disabled=yes \
    list="Exclude from PCC"
add address=0.0.0.0/8 comment=RFC6890 list=NotPublic
add address=10.0.0.0/8 comment=RFC6890 list=NotPublic
add address=100.64.0.0/10 comment=RFC6890 list=NotPublic
add address=127.0.0.0/8 comment=RFC6890 list=NotPublic
add address=169.254.0.0/16 comment=RFC6890 list=NotPublic
add address=172.16.0.0/12 comment=RFC6890 list=NotPublic
add address=192.0.0.0/24 comment=RFC6890 list=NotPublic
add address=192.0.2.0/24 comment=RFC6890 list=NotPublic
add address=192.168.0.0/16 comment=RFC6890 list=NotPublic
add address=192.88.99.0/24 comment=RFC3068 list=NotPublic
add address=198.18.0.0/15 comment=RFC6890 list=NotPublic
add address=198.51.100.0/24 comment=RFC6890 list=NotPublic
add address=203.0.113.0/24 comment=RFC6890 list=NotPublic
add address=224.0.0.0/4 comment=RFC4601 list=NotPublic
add address=240.0.0.0/4 comment=RFC6890 list=NotPublic
add address=192.168.48.0/24 comment="Critical LAN Network Ranges" list=\
    Sensitive-Internal
add address=192.168.20.0/24 comment="Critical LAN Network Ranges" list=\
    Sensitive-Internal
add address=192.168.122.0/24 comment="Critical LAN Network Ranges" list=\
    Sensitive-Internal
add address=192.168.50.21 comment="IOT Devices with DNS Whitelist" list=\
    IOT-Whitelist
add address=192.168.50.22 comment="IOT Devices with DNS Whitelist" list=\
    IOT-Whitelist
add address=192.168.54.226 comment="Lab DNS Whitelist" list=Lab-Whitelist
add address=192.168.104.0/24 comment="Client LAN Network Ranges" list=\
    Client-LAN
add address=192.168.50.0/24 comment="Client LAN Network Ranges" list=\
    Client-LAN
add address=192.168.75.0/24 comment="Non-default Client Networks" list=\
    External-Client
add address=192.168.90.0/24 comment="Non-default Client Networks" list=\
    External-Client
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN-All
add action=drop chain=input comment="Block Winbox connections on WAN" \
    dst-port=8291 in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment=\
    "Block Mikrotik Bandwidth Test connections on WAN" dst-port=2000 \
    in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound TCP DNS" dst-port=53 \
    in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound UDP DNS" disabled=yes \
    dst-port=53 in-interface=ether8-gateway protocol=udp
add action=drop chain=input comment=\
    "Drop all packets which does not have unicast source IP address" \
    src-address-type=!unicast
add action=drop chain=input comment="Drop all packets from public internet whi\
    ch should not exist in public network" in-interface-list=WAN-All \
    src-address-list=NotPublic
add action=fasttrack-connection chain=forward connection-state=\
    established,related dst-address-list=!External-Client src-address-list=\
    !External-Client
add action=drop chain=forward log-prefix=DMZ protocol=tcp src-address=\
    192.168.122.90 src-port=!443
add action=drop chain=forward comment="Drop all packets from public internet w\
    hich should not exist in public network" in-interface-list=WAN-All \
    src-address-list=NotPublic
add action=drop chain=forward comment="Drop all packets in local network which\
    \_does not have local network address" in-interface-list=LAN src-address=\
    !192.168.0.0/16
add action=drop chain=forward comment="Drop all packets from local network to \
    internet which should not exist in public network" disabled=yes \
    dst-address-list=NotPublic in-interface-list=LAN
add action=drop chain=forward comment=\
    "Block IOT Traffic to critical LAN Segments" connection-state=\
    invalid,new,untracked dst-address-list=Sensitive-Internal in-interface=\
    iot-net log=yes log-prefix=iot-drop src-mac-address=!90:DD:5D:CA:59:A7
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="drop all from WAN" in-interface=\
    ether8-gateway
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    out-interface-list=WAN-All
/ip route
add disabled=yes distance=1 dst-address=192.168.88.0/24 gateway=\
    "All Ports Bridge" routing-mark=wan1
add disabled=yes distance=1 dst-address=192.168.88.0/24 gateway=\
    "All Ports Bridge" routing-mark=wan2
add disabled=yes distance=1 gateway=ether7
add disabled=yes distance=1 gateway=ether8-gateway
add comment="VPN TCP" distance=1 dst-address=192.168.126.0/24 gateway=\
    192.168.48.174
add comment="VPN UDP" distance=1 dst-address=192.168.166.0/24 gateway=\
    192.168.48.174
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=192.168.0.0/16
set ssh address=192.168.0.0/16
set api address=192.168.0.0/16
set winbox address=192.168.0.0/16
set api-ssl address=192.168.0.0/16
/ip ssh
set always-allow-password-login=yes forwarding-enabled=remote strong-crypto=\
    yes
/ip upnp interfaces
add interface=ether8-gateway type=external
add interface="All Ports Bridge" type=internal
add interface=bond-ether5_ether6 type=internal
/lcd
set backlight-timeout=10m default-screen=stat-slideshow time-interval=hour
/lcd interface
set sfp1 disabled=yes
/routing filter
add chain=dynamic-in distance=3 set-routing-mark=wan1
add chain=dynamic-in distance=4 set-routing-mark=wan2
/routing igmp-proxy interface
add alternative-subnets=0.0.0.0/0 interface=trusted-clients-vrrp upstream=yes
add interface=iot-net-vrrp
/routing igmp-proxy mfc
add downstream-interfaces=trusted-clients-net group=239.255.255.250 source=\
    192.168.104.108 upstream-interface=iot-net
add downstream-interfaces=trusted-clients-net group=224.0.0.1 source=\
    192.168.104.108 upstream-interface=iot-net
add downstream-interfaces=trusted-clients-net group=224.0.0.251 source=\
    192.168.104.108 upstream-interface=iot-net
/system clock
set time-zone-name=Asia/Singapore
/system health
set cpu-overtemp-threshold=110C
/system identity
set name="MikroTik CCR1009 Router (Primary)"
/system leds
set 0 interface=sfp-sfpplus1
set 1 interface=sfp-sfpplus1
set 2 interface=sfp1
/system logging
set 0 action=sdcard
set 1 action=disk
set 2 action=disk
add action=sdcard topics=script
add action=disk topics=interface
add action=disk topics=critical
add disabled=yes topics=debug
/system ntp client
set enabled=yes server-dns-names=0.sg.pool.ntp.org,1.sg.pool.ntp.org
/tool graphing interface
add allow-address=192.168.0.0/16 interface=ether8-gateway
add allow-address=192.168.48.0/24 interface=bond-ether5_ether6
add allow-address=192.168.104.0/24 interface=bond-ether5_ether6
add allow-address=192.168.88.0/24 interface="All Ports Bridge"
/tool graphing resource
add allow-address=192.168.48.0/24
add allow-address=192.168.104.0/24
add allow-address=192.168.88.0/24
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool mac-server ping
set enabled=no


The behavior I'm seeing is as follows:
  • Both Client A and Client C (in the visual above) can get an DHCP IP in the 192.168.104.0/24 range
  • Both can route to the Internet (accessing other hosts in different VLAN's is an issue)
  • When I try to run a iperf3 test between Client A & Client C, I can see that the bandwidth is maxing out at 1 Gbps and all the traffic is at the SFP+ port on the CCR-1009, suggesting that even intra-subnet traffic is going to the Router instead of staying on the switch.

Screenshots:

IP address of Client A in 192.168.104.0/24 subnet:
Image

IP address of Client C in 192.168.104.0/24 subnet:
Image

Screenshot of iPerf3 command from Client A:
Image

Screenshot of CRS-309 Winbox during iPerf3 test - note the traffic is between SFP1 (Client A) and SFP8 (Uplink to CCR-1009):
Image

Screenshot of CCR-1009 during iPerf3 test - notice the traffic is all on SFP1 (Patch port for CRS-309):
Image

Screenshot of iPerf3 results from Client C running iPerf3 server:
Image

I also noticed that Client C is unable to ping any hosts outside of it's VLAN and indeed it cannot even ping the CCR-1009 on the router's VLAN address in the 192.168.104.0/24 subnet. Screenshot of ping failure:

Image

However, I found interesting that Wireshark shows that the router is indeed responding to the ping but it's not reaching the client:
Image

I'd appreciate if folks can provide me with some insights into why this happening.
Top
 
tdw
Forum Guru
Forum Guru
Posts: 1845
Joined: Sat May 05, 2018 11:55 am

Re: Clients behind CRS on same subnet - but traffic is being switched by the Router?  [SOLVED]

Sun Nov 14, 2021 5:43 pm

The CRS config appears fine, the I also noticed that Client C is unable to ping any hosts outside of it's VLAN and indeed it cannot even ping the CCR-1009 on the router's VLAN address in the 192.168.104.0/24 subnet comment suggests an issue with Client C - does it have multiple interfaces (as there are a couple of other VLANs tagged on the bond interface)? Whilst this can lead to asymmetric routes, and firewalls blocking traffic thus marked invalid, it shouldn't affect traffic within the same subnet. The Bridge > Hosts tab (use Show Columns to add the VID) and IP > ARP entries on the CRS and CCR may provide some insight.
Top
 
avggeek
newbie
Topic Author
Posts: 48
Joined: Thu Jun 06, 2013 9:33 am

Re: Clients behind CRS on same subnet - but traffic is being switched by the Router?

Mon Nov 15, 2021 9:13 am

The CRS config appears fine, the I also noticed that Client C is unable to ping any hosts outside of it's VLAN and indeed it cannot even ping the CCR-1009 on the router's VLAN address in the 192.168.104.0/24 subnet comment suggests an issue with Client C - does it have multiple interfaces (as there are a couple of other VLANs tagged on the bond interface)?

@tdw thank you for this tip! Indeed Client C had a VLAN interface in the same subnet on another bonded Ethernet interface. I changed Client C & Client A to both be on a subnet that did not overlap with any other interface on the device. Once I made this change, I was able to successfully run an iPerf test between the two devices at close to wirespeed - 9.5 Gbps.

Image

However, this doesn't seem to have entirely fixed the problem.

I find that on Client A, I can ping other hosts in the same subnet or Internet IP's:

Image

However on Client C, I still cannot ping any hosts within the same subnet or on the Internet. The only hosts it can ping are the CCR-1009 and Client A.

Image

Image

Image

This is the current ip route information for Client C:

Image
The Bridge > Hosts tab (use Show Columns to add the VID) and IP > ARP entries on the CRS and CCR may provide some insight.
I find that the IP>ARP entries on the CRS have only two entries - both for the CCR-1009 on VLAN1. However, Client C cannot ping these IP's either.
Top
 
tdw
Forum Guru
Forum Guru
Posts: 1845
Joined: Sat May 05, 2018 11:55 am

Re: Clients behind CRS on same subnet - but traffic is being switched by the Router?

Mon Nov 15, 2021 2:18 pm

It is pointing to all the issues being with Client C.

Normally you wouldn't have multiple default gateways on a host device unless you were intending to do policy-based routing. As there are differing metrics associated with the multiple default gateways any traffic not destined for the directly attached networks (192.168.20.0/24, 192.168.48.0/24, 192.168.50.0/24, 192.168.104.0/24) will be sent to 192.168.48.254, you should be able to see this with the packet sniffer on the CCR.

If you can't ping other devices on directly connected subnets it is either network misconfiguration or iptables rules/policies blocking the traffic, tcpdump or similar would provide you with more information.

The CRS ARP table entries sound correct, what are there on the CCR.
Top
 
avggeek
newbie
Topic Author
Posts: 48
Joined: Thu Jun 06, 2013 9:33 am

Re: Clients behind CRS on same subnet - but traffic is being switched by the Router?

Mon Nov 15, 2021 3:33 pm

If you can't ping other devices on directly connected subnets it is either network misconfiguration or iptables rules/policies blocking the traffic, tcpdump or similar would provide you with more information.
So this is very odd. I tested ping between Client C & Client B where Client B is on a different subnet. I also ensured that there is no overlap in subnets between Client C & Client B on any network interface on Client C. Client B has an address in the 192.168.50.0/24 subnet.

ip route information on Client C:
Code: Select all
# ip route
default via 192.168.48.254 dev bond0 proto dhcp src 192.168.48.100 metric 100
default via 192.168.104.254 dev vlan104 proto dhcp src 192.168.104.100 metric 200
default via 192.168.20.254 dev bond1 proto dhcp src 192.168.20.24 metric 300
192.168.20.0/24 dev bond1 proto kernel scope link src 192.168.20.24
192.168.20.254 dev bond1 proto dhcp scope link src 192.168.20.24 metric 300
192.168.48.0/24 dev bond0 proto kernel scope link src 192.168.48.100
192.168.48.254 dev bond0 proto dhcp scope link src 192.168.48.100 metric 100
192.168.104.0/24 dev vlan104 proto kernel scope link src 192.168.104.100
192.168.104.254 dev vlan104 proto dhcp scope link src 192.168.104.100 metric 200

Here is the output of ping from Client C:

Code: Select all
# ping -I bond1 192.168.50.24
PING 192.168.50.24 (192.168.50.24) from 192.168.20.24 bond1: 56(84) bytes of data.
^C
--- 192.168.50.24 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4087ms

Here is the output of tcpdump on Client C:

Code: Select all
# tcpdump -n -tttt -i bond1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond1, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-11-15 21:58:43.307063 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 1, length 64
2021-11-15 21:58:43.307583 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 1, length 64
2021-11-15 21:58:44.322507 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 2, length 64
2021-11-15 21:58:44.322970 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 2, length 64
2021-11-15 21:58:45.346390 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 3, length 64
2021-11-15 21:58:45.346807 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 3, length 64
2021-11-15 21:58:46.370404 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 4, length 64
2021-11-15 21:58:46.371005 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 4, length 64
2021-11-15 21:58:47.394385 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 5, length 64
2021-11-15 21:58:47.394840 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 5, length 64
Output of tcpdump from Client B which is on a different subnet (192.168.50.0/24)

Code: Select all
# tcpdump -n -tttt -i eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-11-15 13:58:43.296250 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 1, length 64
2021-11-15 13:58:43.296292 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 1, length 64
2021-11-15 13:58:44.311614 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 2, length 64
2021-11-15 13:58:44.311646 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 2, length 64
2021-11-15 13:58:45.335414 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 3, length 64
2021-11-15 13:58:45.335449 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 3, length 64
2021-11-15 13:58:46.359605 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 4, length 64
2021-11-15 13:58:46.359645 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 4, length 64
2021-11-15 13:58:47.383481 IP 192.168.20.24 > 192.168.50.24: ICMP echo request, id 14090, seq 5, length 64
2021-11-15 13:58:47.383512 IP 192.168.50.24 > 192.168.20.24: ICMP echo reply, id 14090, seq 5, length 64

Client B is on UTC compared to Client C which is on UTC+8.

So tcpdump output suggests that the packets are reaching Client C but ping doesn't recognize them?

iptables on Client C is empty btw:

Code: Select all
]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

The CRS ARP table entries sound correct, what are there on the CCR.
There are a lot of ARP entries on the CCR. What should I look for here?
Top
 
avggeek
newbie
Topic Author
Posts: 48
Joined: Thu Jun 06, 2013 9:33 am

Re: Clients behind CRS on same subnet - but traffic is being switched by the Router?

Tue Nov 16, 2021 1:06 pm

It is pointing to all the issues being with Client C.
I'm starting to get why some people really really hate systemd. I eventually figured out that although I had a systemctl.d override to disable reverse path filtering (rp_filter), some combination of systemctl files was overwriting my changes after a reboot. I eventually had to modify /etc/sysctl.conf directly - once I did this, the cross VLAN pings start working as expected.

I think I'm finally to the point where I can start adding new hosts to this switch and configuring them to be in "prod" via the CRS. I'll still need to figure out the changes needed to simplify the dual WAN failover and some of the firewall rule suggestions that @mkx had in another thread, but I think atleast this thread can be marked as solved.

Thanks @tdw!
Top
Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Bing [Bot], GoogleOther [Bot], LunaticRv, nl2024, scoobyn8, Semrush [Bot] and 40 guests
  4. RouterOS
  5. General