CRS309 Switch - cannot ping gateway or any other host

avggeek · Mon Nov 08, 2021 12:04 pm

Hello,

I've been working with Mikrotik devices for a while and thought I understood them, but clearly I'm still very much a noob. I've recently accquired a CRS309 switch and connected it to a CCR1009 router via SFP+ ports and Ethernet. I've been able to get the Ethernet port configured as a DHCP client for management purposes, but the SFP+ ports and bridge are just not working for me. From the switch, I cannot ping the upstream router or any other host in the same subnet.

The CRS is configured to boot in ROS mode and here's the current config with hide-sensitive turned on:

/interface bridge
add admin-mac=2C:C8:1B:20:06:C8 auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=sfp-sfpplus8 ] advertise=\
    10M-full,100M-full,1000M-full,10000M-full
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf disabled=yes interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/ip address
add address=192.168.48.10/24 comment=defconf interface=bridge network=\
    192.168.48.0
/ip dhcp-client
add add-default-route=no disabled=no interface=ether1 use-peer-dns=no \
    use-peer-ntp=no
/ip route
add check-gateway=ping distance=1 gateway=192.168.48.254
/system routerboard settings
set boot-os=router-os

The interfaces show up as connected in Interfaces as well as in the Bridge menu so I don't think that's the issue.

Can someone help me figure out what I'm doing wrong?

mkx · Mon Nov 08, 2021 3:06 pm

Could be xSTP is kicking in. So try to completely remove ether1 from bridge (simply disabling it as bridge port might not be enough).

anav · Mon Nov 08, 2021 5:53 pm

This should not be difficult!

You should only have one connection from the CCR1009 to the switch.
The connecting trunk (carrying all the vlans) should be the selected port joining the two units.

The switch should have an IP address on the trusted LAN or management LAN of the CCR1009

All the other ports should be on the bridge (save one ****)
All vlans should be identified as belonging to the bridge.

The interface list member should include "control"
interface=trusted vlan list=control
The tool macserver winmacserver should be "control"

One route rule dest 0.0.0.0./0 gw= gateway IP of trusted vlan.

THATS IT DONE (no other rules on the switch required)!!

Obviously you have to decide which other ports on the switch may be trunk ports and the rest access ports (minus the emerg access port),
and set the /interface bridge port and /interface bridge vlan rules correspondingly.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

******* Keep one port for emergency access.
name etherX-emerg

remove etherX-emerg from the bridge
add interface etherX-emerg list=control
add ip address = 192.168.66.2 network 192.168.66.0

Now if the bridge ever burps on you, simply plug in the laptop to etherX-emerg port and set the laptop to IP address 192.168.66.5 or .10 for example.

avggeek · Tue Nov 09, 2021 6:51 am

@mkx / @anav thanks for the suggestions.

@anav there are no VLAN's enabled yet. This is all on PVID 1 and the 192.168.48.0/24 range is the IP range for the PVID1 VLAN.

I have tried resetting the configuration of the switch again and what I noticed is that while the switch is assigned the IP address of 192.168.88.1, things seem to work. Clients connected to the switch can atleast get an IP address, although they cannot route to Internet. However, once I change the IP address to the 192.168.48.0/24 subnet, things stop working.

I'm not sure if this is because the DHCP server for the 192.168.48.0/24 range listens on a different interface on my router and that interface does not include the SFP+ port on the router that the CRS309 is connected to.

Edit: I take back the statement about things working when the switch is using an IP address of 192.168.88.1. Although clients can get an IP address, the switch itself cannot ping the router on the 192.168.48.0/24 subnet or even the router's own 192.168.88.X address.

Here's the current config - it's pretty much bog-standard with the exception of removing ether1 from the bridge:

/interface bridge
add admin-mac=2C:C8:1B:20:06:C8 auto-mac=no comment=defconf name=bridge
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add add-default-route=no disabled=no interface=ether1
/system routerboard settings
set boot-os=router-os

mkx · Tue Nov 09, 2021 8:30 am

What does

/interface bridge port print

show?

avggeek · Tue Nov 09, 2021 8:51 am

The issue seems to be with how the interfaces are configured in my CCR-1009 router. If I add the SFP+ port on my router to a Bridge which has other ports on it, and has an IP address of the router assigned to it, the switch is able to ping my router and Internet IP's.

However, assigning the SFP+ port to a Bridge on the CCR-1009 seems like a terrible idea - I can see that the SFP+ port is not hardware accelarated, which means all the traffic is gonna go via the CPU. Parts of the CCR-1009 router config are below as it has lots of other configurations that might not be very relevant:

/interface bridge
add admin-mac=4C:5E:0C:03:20:22 auto-mac=no fast-forward=no name=\
    "Direct Clients Bridge"
/interface ethernet
set [ find default-name=ether1 ] name="ether1-switch master" speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] comment="Unifi Switch Copper SFP upstream-1" \
    speed=100Mbps
set [ find default-name=ether6 ] comment="Unifi Switch Copper SFP upstream-2" \
    mac-address=4C:5E:0C:03:20:26 speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=ether8 ] advertise=100M-full,1000M-full comment=\
    "WAN Interface" name=ether8-gateway speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=\
    10M-full,100M-full,1000M-full,10000M-full
 /interface bonding
add lacp-rate=1sec mode=802.3ad name="Home Network Trunk Ports" slaves=\
    ether5,ether6
/interface vrrp
add authentication=ah comment="VLAN 1 Network" interface=\
    "Home Network Trunk Ports" name=mgmt-net-vrrp priority=250 version=2 \
    vrid=48
/ip pool
add comment="Management IP Range" name=mgmt-iprange ranges=\
    192.168.48.100-192.168.48.200
add comment="IP Range for direct attached clients" name=direct-iprange \
    ranges=192.168.88.10-192.168.88.20
/ip dhcp-server
add address-pool=direct-iprange disabled=no interface="Direct Clients Bridge" \
    lease-time=1d name=direct-dhcp
add address-pool=mgmt-iprange disabled=no interface=\
    "Home Network Trunk Ports" lease-time=1d name=mgmt-dhcp
 /ip address
add address=192.168.88.3/24 comment="Address for directly attached clients" \
    interface="Direct Clients Bridge" network=192.168.88.0
add address=192.168.48.1/24 comment="Management network" interface=\
    "Home Network Trunk Ports" network=192.168.48.0
add address=192.168.48.254 interface=mgmt-net-vrrp network=192.168.48.254

avggeek · Tue Nov 09, 2021 8:54 am

What does
Code: Select all
/interface bridge port print
show?

On the CRS-309:

[admin@MikroTik] > /interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE           BRIDGE           HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0 I H ;;; defconf
       sfp-sfpplus1        bridge           yes    1     0x80         10                 10       none
 1 I H ;;; defconf
       sfp-sfpplus2        bridge           yes    1     0x80         10                 10       none
 2 I H ;;; defconf
       sfp-sfpplus3        bridge           yes    1     0x80         10                 10       none
 3 I H ;;; defconf
       sfp-sfpplus4        bridge           yes    1     0x80         10                 10       none
 4 I H ;;; defconf
       sfp-sfpplus5        bridge           yes    1     0x80         10                 10       none
 5 I H ;;; defconf
       sfp-sfpplus6        bridge           yes    1     0x80         10                 10       none
 6 I H ;;; defconf
       sfp-sfpplus7        bridge           yes    1     0x80         10                 10       none
 7   H ;;; defconf
       sfp-sfpplus8        bridge           yes    1     0x80         10                 10       none

On the CCR-1009:

[admin@MikroTik CCR1009 Router (Primary)] > /interface bridge port print 
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE                                                                     BRIDGE                                                                    HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0 I H ether1-switch master                                                          Direct Clients Bridge                                                     yes    1     0x80         10                 10       none
 1 I H ether2                                                                        Direct Clients Bridge                                                     yes    1     0x80         10                 10       none
 2 I H ether3                                                                        Direct Clients Bridge                                                     yes    1     0x80         10                 10       none
 3 I H ether4                                                                        Direct Clients Bridge                                                     yes    1     0x80         10                 10       none
 4     sfp-sfpplus1                                                                  Direct Clients Bridge                                                     yes    1     0x80         10                 10       none

mkx · Tue Nov 09, 2021 9:02 am

However, assigning the SFP+ port to a Bridge on the CCR-1009 seems like a terrible idea - I can see that the SFP+ port is not hardware accelarated, which means all the traffic is gonna go via the CPU.

Without any special configuration, CCR1009 should still be able to bridge a few Gbps between SFP+ interface and the rest of ether ports (using CPU). So in principle things should work if configured properly. HW layout of your CCR doesn't explain the fact things don't work at all if you connect device through CRS309. So it's got to be configuration, either on CRS or on CCR.

But yes, routers (and CCR1009 is a proper router) are generally not really great at switching/bridging. Think of SFP+ port on router as a great interface to be used when CCR is configured as pure router - i.e. every interface carries a separate subnet (or multiple if VLANs are in use). Exempt form this rule are routers which have switch chip built-in and your CCR seems to be one of those.

avggeek · Tue Nov 09, 2021 9:28 am

Without any special configuration, CCR1009 should still be able to bridge a few Gbps between SFP+ interface and the rest of ether ports (using CPU). So in principle things should work if configured properly. HW layout of your CCR doesn't explain the fact things don't work at all if you connect device through CRS309. So it's got to be configuration, either on CRS or on CCR.

But yes, routers (and CCR1009 is a proper router) are generally not really great at switching/bridging. Think of SFP+ port on router as a great interface to be used when CCR is configured as pure router - i.e. every interface carries a separate subnet (or multiple if VLANs are in use). Exempt form this rule are routers which have switch chip built-in and your CCR seems to be one of those.

@mkx I have posted most of my CCR-1009 configuration earlier in the thread. Looking at it, do you see anything that might be the cause of the CRS309 not working correctly?

anav · Tue Nov 09, 2021 2:35 pm

Everything is muddled here.
A. What port is the CCR1009 coming in on for the Switch - I think ether1
B. Is the traffic coming in on a vlan - I think no
C. Is all the traffic on the switch supposed to be using the same subnet that is coming in on ether1 - I assume yes
D. Is that subnet 192.168.48.0/24 - I think yes
E. Is there only one link between the router and the switch - I think yes via ether1
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

Then: This should work!, NOTE the IP address of the switch is 192.168.48.10

/interface bridge
add admin-mac=2C:C8:1B:20:06:C8 auto-mac=no comment=defconf name=bridge
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/ip address
add address=192.168.48.10/24 comment=defconf interface=bridge network=\
    192.168.48.0
/ip route
add check-gateway=ping distance=1 gateway=192.168.48.254
/system routerboard settings
set boot-os=router-os
/interface list
add control
/interface list member
add interface=bridge list=control
/tool mac-server mac-winbox
set allowed-interface-list=control

anav · Tue Nov 09, 2021 2:39 pm

We need to see the full latest config of the CCR1009
/export hide-sensitive file=anynameyouwish

Also you need to provide a network diagram for the CCR1009.
It is not clear at all what is attached to etc. both upstream and downstream

avggeek · Tue Nov 09, 2021 5:08 pm

We need to see the full latest config of the CCR1009
/export hide-sensitive file=anynameyouwish

Also you need to provide a network diagram for the CCR1009.
It is not clear at all what is attached to etc. both upstream and downstream

@anav here is a stripped down network diagram that shows how the CRS309 is connected to the CCR-1009:

To answer your questions

A. What port is the CCR1009 coming in on for the Switch - Switch comes into CCR1009 on SFP8 and Ether 1. Ether 1 via another Switch and SFP8 directly to CCR1009.
B. Is the traffic coming in on a vlan - Not at the moment.
C. Is all the traffic on the switch supposed to be using the same subnet that is coming in on ether1 - No. Traffic will come in on other subnets.
D. Is that subnet 192.168.48.0/24 - Subnet for ether 1 is 192.168.104.0.
E. Is there only one link between the router and the switch - No there are two.

Here is the full config for CCR-1009:

# model = CCR1009-8G-1S-1S+
/interface bridge
add admin-mac=4C:5E:0C:03:20:22 auto-mac=no fast-forward=no name=\
    "Direct Clients Bridge"
/interface ethernet
set [ find default-name=ether1 ] name="ether1-switch master" speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] comment="Unifi Switch Copper SFP upstream-1" \
    speed=100Mbps
set [ find default-name=ether6 ] comment="Unifi Switch Copper SFP upstream-2" \
    mac-address=4C:5E:0C:03:20:26 speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=ether8 ] advertise=100M-full,1000M-full comment=\
    "WAN Interface" name=ether8-gateway speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=\
    10M-full,100M-full,1000M-full,10000M-full
set [ find default-name=sfp1 ] advertise=10M-full,100M-full,1000M-full \
    mac-address=4C:5E:0C:03:20:22
/interface bonding
add lacp-rate=1sec mode=802.3ad name="Home Network Trunk Ports" slaves=\
    ether5,ether6
/interface vrrp
add authentication=ah comment="VLAN 1 Network" interface=\
    "Home Network Trunk Ports" name=mgmt-net-vrrp priority=250 version=2 \
    vrid=48
/interface vlan
add comment="DMZ Network" interface="Home Network Trunk Ports" name=dmz-net \
    vlan-id=122
add comment="Guest network" disabled=yes interface="Home Network Trunk Ports" \
    name=guest-net vlan-id=90
add comment="IOT Devices Network" interface="Home Network Trunk Ports" name=\
    iot-net vlan-id=50
add comment="Lab Network" interface="Home Network Trunk Ports" name=lab-vlan \
    vlan-id=54
add comment="Untrusted Client(s) network" interface=\
    "Home Network Trunk Ports" name=others-net vlan-id=75
add comment="Server network" interface="Home Network Trunk Ports" name=\
    server-net vlan-id=20
add comment="\"Trusted\" clients network" interface=\
    "Home Network Trunk Ports" name=trusted-clients-net vlan-id=104
/interface vrrp
add authentication=ah interface=dmz-net name=dmz-net-vrrp priority=250 \
    version=2 vrid=122
add authentication=ah interface=iot-net name=iot-net-vrrp priority=250 \
    version=2 vrid=50
add authentication=ah interface=lab-vlan name=lab-vlan-vrrp priority=250 \
    version=2 vrid=54
add authentication=ah interface=others-net name=others-net-vrrp priority=250 \
    version=2 vrid=75
add authentication=ah interface=server-net name=server-net-vrrp on-master=\
    " /system script run force-update-odns" priority=250 version=2 vrid=20
add authentication=ah interface=trusted-clients-net name=trusted-clients-vrrp \
    on-master="/tool e-mail send to=avggeek@domain.tld subject=\"Primary Ro\
    uter Failover Triggered\" body=\"Primary Router is now VRRP Master\"" \
    priority=250 version=2 vrid=104
/interface list
add name=WAN-All
add name=LAN
add name=Native-WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server option
add code=15 name=labdomain value="'lab.domain.tld'"
add code=119 name=domainsearch value=\
    "s'srv.domain.tld,clients.domain.tld,lab.domain.tld'"
add code=15 name=clientdomain value="'clients.domain.tld'"
add code=15 name=srvdomain value="'srv.domain.tld'"
add code=15 name=mgmtdomain value="'mgmt.domain.tld'"
add code=15 name=iotdomain value="'iot.domain.tld'"
/ip dhcp-server option sets
add name=lab-dhcp-options options=labdomain,domainsearch
add name=client-dhcp-options options=clientdomain,domainsearch
add name=srv-dhcp-options options=srvdomain,domainsearch
add name=mgmt-dhcp-options options=mgmtdomain
/ip pool
add comment="Management IP Range" name=mgmt-iprange ranges=\
    192.168.48.100-192.168.48.200
add comment="IP Range for Lab Network" name=lab-iprange ranges=\
    192.168.54.192-192.168.54.230
add comment="IP Range for direct attached clients" name=direct-iprange \
    ranges=192.168.88.10-192.168.88.20
add comment="IP Range for Servers" name=server-iprange ranges=\
    192.168.20.20-192.168.20.100
add comment="IP range for IOT network" name=iot-iprange ranges=\
    192.168.50.20-192.168.50.50
add comment="IP Range for untrusted clients" name=others-iprange ranges=\
    192.168.75.20-192.168.75.30
add comment="IP Range for Guest Network" name=guest-iprange ranges=\
    192.168.90.10-192.168.90.20
add comment="IP Range for \"trusted\" clients" name=trusted-iprange ranges=\
    192.168.104.100-192.168.104.200
add comment="IP Range for external facing hosts" name=dmz-iprange ranges=\
    192.168.122.90/31
/ip dhcp-server
add address-pool=direct-iprange disabled=no interface="Direct Clients Bridge" \
    lease-time=1d name=direct-dhcp
add address-pool=mgmt-iprange disabled=no interface=\
    "Home Network Trunk Ports" lease-time=1d name=mgmt-dhcp
add address-pool=lab-iprange disabled=no interface=lab-vlan-vrrp \
    lease-script=":local scriptName \"dhcp2dns\"\r\
    \n:do {\r\
    \n  :local scriptSrc [ /system script get [ find name=\$scriptName  ] sour\
    ce ]\r\
    \n  :local scriptObj [ :parse \$scriptSrc ]\r\
    \n  \$scriptObj leaseBound=\$leaseBound  leaseServerName=\$leaseServerName\
    \_leaseActIP=\$leaseActIP leaseActMAC=\$leaseActMAC\r\
    \n} on-error={ :log warning \"DHCP server '\$leaseServerName' lease script\
    \_error\" }\r\
    \n" lease-time=1h name=lab-dhcp
add add-arp=yes address-pool=server-iprange disabled=no interface=server-net \
    lease-script=":local scriptName \"dhcp2dns\"\r\
    \n:do {\r\
    \n  :local scriptSrc [ /system script get [ find name=\$scriptName  ] sour\
    ce ]\r\
    \n  :local scriptObj [ :parse \$scriptSrc ]\r\
    \n  \$scriptObj leaseBound=\$leaseBound  leaseServerName=\$leaseServerName\
    \_leaseActIP=\$leaseActIP leaseActMAC=\$leaseActMAC\r\
    \n} on-error={ :log warning \"DHCP server '\$leaseServerName' lease script\
    \_error\" }\r\
    \n" lease-time=3d name=server-dhcp
add address-pool=guest-iprange interface=guest-net lease-time=1h name=\
    guest-dhcp
add address-pool=iot-iprange disabled=no interface=iot-net lease-script=":loca\
    l scriptName \"dhcp2dns\"\r\
    \n:do {\r\
    \n  :local scriptSrc [ /system script get [ find name=\$scriptName  ] sour\
    ce ]\r\
    \n  :local scriptObj [ :parse \$scriptSrc ]\r\
    \n  \$scriptObj leaseBound=\$leaseBound  leaseServerName=\$leaseServerName\
    \_leaseActIP=\$leaseActIP leaseActMAC=\$leaseActMAC\r\
    \n} on-error={ :log warning \"DHCP server '\$leaseServerName' lease script\
    \_error\" }\r\
    \n" lease-time=1d name=iot-dhcp
add address-pool=others-iprange disabled=no interface=others-net-vrrp \
    lease-time=12h name=others-dhcp
add add-arp=yes address-pool=trusted-iprange disabled=no interface=\
    trusted-clients-net lease-script=":local scriptName \"dhcp2dns\"\r\
    \n:do {\r\
    \n  :local scriptSrc [ /system script get [ find name=\$scriptName  ] sour\
    ce ]\r\
    \n  :local scriptObj [ :parse \$scriptSrc ]\r\
    \n  \$scriptObj leaseBound=\$leaseBound  leaseServerName=\$leaseServerName\
    \_leaseActIP=\$leaseActIP leaseActMAC=\$leaseActMAC\r\
    \n} on-error={ :log warning \"DHCP server '\$leaseServerName' lease script\
    \_error\" }\r\
    \n" lease-time=1d name=trusted-dhcp
add address-pool=dmz-iprange disabled=no interface=dmz-net lease-time=1d \
    name=dmz-dhcp
/queue type
set 5 pcq-limit=1000KiB pcq-total-limit=1000KiB
set 6 pcq-limit=5000KiB pcq-total-limit=5000KiB
/queue simple
add burst-limit=5M/25M burst-threshold=5M/25M burst-time=5s/10s max-limit=\
    3M/20M name=others-net-queue queue=ethernet-default/ethernet-default \
    target=192.168.75.0/24 total-queue=ethernet-default
add burst-limit=2M/5M burst-threshold=2M/5M burst-time=5s/5s limit-at=256k/1M \
    max-limit=1M/3M name=guest-net-queue queue=\
    ethernet-default/ethernet-default target=guest-net
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/system logging action
add disk-file-count=4 disk-file-name=disk1/log disk-lines-per-file=2000 name=\
    sdcard target=disk
/interface bridge port
add bridge="Direct Clients Bridge" interface="ether1-switch master"
add bridge="Direct Clients Bridge" interface=ether2
add bridge="Direct Clients Bridge" interface=ether3
add bridge="Direct Clients Bridge" interface=ether4
add bridge="Direct Clients Bridge" interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=!WAN-All
/interface bridge vlan
add tagged=lab-vlan untagged="Direct Clients Bridge" vlan-ids=5
/interface detect-internet
set detect-interface-list=Native-WAN internet-interface-list=WAN-All \
    lan-interface-list=LAN wan-interface-list=WAN-All
/interface list member
add interface=ether8-gateway list=WAN-All
add interface="Direct Clients Bridge" list=LAN
add interface="Home Network Trunk Ports" list=LAN
add interface=dmz-net list=LAN
add interface="ether1-switch master" list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=guest-net list=LAN
add interface=iot-net list=LAN
add interface=others-net list=LAN
add interface=server-net list=LAN
add interface=trusted-clients-net list=LAN
add interface=lab-vlan list=LAN
add interface=dmz-net-vrrp list=LAN
add interface=iot-net-vrrp list=LAN
add interface=mgmt-net-vrrp list=LAN
add interface=others-net-vrrp list=LAN
add interface=server-net-vrrp list=LAN
add interface=lab-vlan-vrrp list=LAN
add interface=trusted-clients-vrrp list=LAN
add interface=l2tp-evpn-IN list=WAN-All
add interface=l2tp-evpn-US list=WAN-All
add interface=ether8-gateway list=Native-WAN
/ip address
add address=192.168.88.3/24 comment="Address for directly attached clients" \
    interface="Direct Clients Bridge" network=192.168.88.0
add address=192.168.48.1/24 comment="Management network" interface=\
    "Home Network Trunk Ports" network=192.168.48.0
add address=192.168.54.1/24 comment="Lab Network" interface=lab-vlan network=\
    192.168.54.0
add address=192.168.20.1/24 comment="Server Network" interface=server-net \
    network=192.168.20.0
add address=192.168.50.1/24 comment="IOT Network" interface=iot-net network=\
    192.168.50.0
add address=192.168.75.1/24 comment="Untrusted Clients" interface=others-net \
    network=192.168.75.0
add address=192.168.90.1/24 comment="Guest Network (Inactive)" disabled=yes \
    interface=guest-net network=192.168.90.0
add address=192.168.104.1/24 comment="Trusted Clients" interface=\
    trusted-clients-net network=192.168.104.0
add address=192.168.122.1/24 comment="DMZ Network" interface=dmz-net network=\
    192.168.122.0
add address=192.168.54.254 interface=lab-vlan-vrrp network=192.168.54.254
add address=192.168.20.254 interface=server-net-vrrp network=192.168.20.254
add address=192.168.48.254 interface=mgmt-net-vrrp network=192.168.48.254
add address=192.168.50.254 interface=iot-net-vrrp network=192.168.50.254
add address=192.168.75.254 interface=others-net-vrrp network=192.168.75.254
add address=192.168.104.254 interface=trusted-clients-vrrp network=\
    192.168.104.254
add address=192.168.122.254 interface=dmz-net-vrrp network=192.168.122.254
add address=192.168.50.19 interface=iot-net network=192.168.50.19
add address=192.168.75.19 interface=others-net network=192.168.75.19
/ip arp
add address=192.168.48.255 comment="Broadcast MAC for WOL" interface=\
    "Home Network Trunk Ports" mac-address=FF:FF:FF:FF:FF:FF
add address=192.168.48.51 interface="Home Network Trunk Ports" mac-address=\
    FC:EC:DA:3A:96:66
add address=192.168.48.50 interface="Home Network Trunk Ports" mac-address=\
    FC:EC:DA:3A:9A:8B
/ip cloud
set update-time=no
/ip dhcp-client
add disabled=no interface=ether8-gateway use-peer-dns=no use-peer-ntp=no
add interface=sfp1
/ip dhcp-server lease
add address=192.168.104.11 comment="Mikrotik CRS309-1G-8S+ Management" \
    mac-address=2C:C8:1B:20:06:CA server=trusted-dhcp
add address=192.168.48.10 comment=\
    "10Gbe Switch (Mikrotik CRS309-1G-8S+)" disabled=yes mac-address=\
    2C:C8:1B:20:06:C8
/ip dhcp-server network
add address=192.168.20.0/24 comment="Server VLAN IP Pool" dhcp-option-set=\
    srv-dhcp-options dns-server=192.168.20.254 domain=srv.domain.tld \
    gateway=192.168.20.254
add address=192.168.48.0/24 comment="Management IP Pool" dhcp-option-set=\
    mgmt-dhcp-options dns-server=192.168.48.254 domain=mgmt.domain.tld \
    gateway=192.168.48.254 netmask=24
add address=192.168.50.0/24 comment="IOT VLAN IP Pool" dhcp-option=iotdomain \
    dns-server=192.168.50.19 domain=iot.domain.tld gateway=192.168.50.254
add address=192.168.54.0/24 comment="Lab VLAN IP Pool" dhcp-option-set=\
    lab-dhcp-options dns-server=192.168.54.230 domain=lab.domain.tld \
    gateway=192.168.54.254
add address=192.168.75.0/24 comment="Others VLAN IP Pool" dns-server=\
    192.168.75.19 gateway=192.168.75.254
add address=192.168.88.0/24 comment="Direct Attached Clients IP Pool" \
    dhcp-option-set=client-dhcp-options dns-server=192.168.88.254 domain=\
    clients.domain.tld gateway=192.168.88.254
add address=192.168.90.0/24 comment="Guest VLAN IP Pool (Inactive)" \
    dns-server=8.8.8.8 gateway=192.168.90.1
add address=192.168.104.0/24 comment="\"Trusted\" VLAN IP Pool" \
    dhcp-option-set=client-dhcp-options dns-server=192.168.104.254 domain=\
    clients.domain.tld gateway=192.168.104.254
add address=192.168.122.0/24 comment="DMZ VLAN IP Pool" dns-none=yes gateway=\
    192.168.122.254
/ip dns
set allow-remote-requests=yes servers=9.9.9.9,208.67.220.222
/ip dns static
add address=192.168.88.1 name=router.lan ttl=1w3d
add address=192.168.20.1 name=ccr1009-router.srv.domain.tld ttl=1w3d
add address=192.168.50.1 name=ccr1009-router.iot.domain.tld ttl=1w3d
add address=192.168.54.1 name=ccr1009-router.lab.domain.tld ttl=1w3d
add address=192.168.75.1 name=ccr1009-router.others.domain.tld ttl=1w3d
add address=192.168.104.1 name=ccr1009-router.clients.domain.tld ttl=1w3d
add address=192.168.122.1 name=ccr1009-router.dmz.domain.tld ttl=1w3d
add address=192.168.20.254 name=router.srv.domain.tld ttl=1w3d
add address=192.168.50.254 name=router.iot.domain.tld ttl=1w3d
add address=192.168.54.254 name=router.lab.domain.tld ttl=1w3d
add address=192.168.75.254 name=router.others.domain.tld ttl=1w3d
add address=192.168.104.254 name=router.clients.domain.tld ttl=1w3d
add address=192.168.122.254 name=router.dmz.domain.tld ttl=1w3d
/ip firewall address-list
add address=192.168.88.2 comment="Exclude from PCC Example" disabled=yes \
    list="Exclude from PCC"
add address=0.0.0.0/8 comment=RFC6890 list=NotPublic
add address=10.0.0.0/8 comment=RFC6890 list=NotPublic
add address=100.64.0.0/10 comment=RFC6890 list=NotPublic
add address=127.0.0.0/8 comment=RFC6890 list=NotPublic
add address=169.254.0.0/16 comment=RFC6890 list=NotPublic
add address=172.16.0.0/12 comment=RFC6890 list=NotPublic
add address=192.0.0.0/24 comment=RFC6890 list=NotPublic
add address=192.0.2.0/24 comment=RFC6890 list=NotPublic
add address=192.168.0.0/16 comment=RFC6890 list=NotPublic
add address=192.88.99.0/24 comment=RFC3068 list=NotPublic
add address=198.18.0.0/15 comment=RFC6890 list=NotPublic
add address=198.51.100.0/24 comment=RFC6890 list=NotPublic
add address=203.0.113.0/24 comment=RFC6890 list=NotPublic
add address=224.0.0.0/4 comment=RFC4601 list=NotPublic
add address=240.0.0.0/4 comment=RFC6890 list=NotPublic
add address=192.168.48.0/24 comment="Critical LAN Network Ranges" list=\
    Sensitive-Internal
add address=192.168.20.0/24 comment="Critical LAN Network Ranges" list=\
    Sensitive-Internal
add address=192.168.122.0/24 comment="Critical LAN Network Ranges" list=\
    Sensitive-Internal
add address=192.168.50.21 comment="IOT Devices with DNS Whitelist" list=\
    IOT-Whitelist
add address=192.168.50.22 comment="IOT Devices with DNS Whitelist" list=\
    IOT-Whitelist
add address=192.168.54.226 comment="Lab DNS Whitelist" list=Lab-Whitelist
add address=192.168.104.0/24 comment="Client LAN Network Ranges" list=\
    Client-LAN
add address=192.168.50.0/24 comment="Client LAN Network Ranges" list=\
    Client-LAN
add address=192.168.75.0/24 comment="Non-default Client Networks" list=\
    External-Client
add address=192.168.90.0/24 comment="Non-default Client Networks" list=\
    External-Client
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN-All
add action=drop chain=input comment="Block Winbox connections on WAN" \
    dst-port=8291 in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment=\
    "Block Mikrotik Bandwidth Test connections on WAN" dst-port=2000 \
    in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound TCP DNS" dst-port=53 \
    in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound UDP DNS" disabled=yes \
    dst-port=53 in-interface=ether8-gateway protocol=udp
add action=drop chain=input comment=\
    "Drop all packets which does not have unicast source IP address" \
    src-address-type=!unicast
add action=drop chain=input comment="Drop all packets from public internet whi\
    ch should not exist in public network" in-interface-list=WAN-All \
    src-address-list=NotPublic
add action=fasttrack-connection chain=forward connection-state=\
    established,related dst-address-list=!External-Client src-address-list=\
    !External-Client
add action=drop chain=forward log-prefix=DMZ protocol=tcp src-address=\
    192.168.122.90 src-port=!443
add action=drop chain=forward comment="Drop all packets from public internet w\
    hich should not exist in public network" in-interface-list=WAN-All \
    src-address-list=NotPublic
add action=drop chain=forward comment="Drop all packets in local network which\
    \_does not have local network address" in-interface-list=LAN src-address=\
    !192.168.0.0/16
add action=drop chain=forward comment="Drop all packets from local network to \
    internet which should not exist in public network" disabled=yes \
    dst-address-list=NotPublic in-interface-list=LAN
add action=drop chain=forward comment=\
    "Block IOT Traffic to critical LAN Segments" connection-state=\
    invalid,new,untracked dst-address-list=Sensitive-Internal in-interface=\
    iot-net log=yes log-prefix=iot-drop src-mac-address=!90:DD:5D:CA:59:A7
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="drop all from WAN" in-interface=\
    ether8-gateway
/ip firewall mangle
add action=mark-connection chain=prerouting in-interface=others-net-vrrp \
    new-connection-mark=non-trusted passthrough=yes
# guest-net not ready
add action=mark-connection chain=prerouting in-interface=guest-net \
    new-connection-mark=non-trusted passthrough=yes
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for fast.com" dst-port=53 layer7-protocol=\
    Netflix-Fast log-prefix=nflx-fast new-connection-mark=nflx-fast \
    passthrough=yes protocol=udp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Netflix Domains" dst-port=53 layer7-protocol=Netflix \
    log-prefix=nflx-conn new-connection-mark=nflx-conn passthrough=yes \
    protocol=udp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Netflix Domains" dst-port=53 layer7-protocol=Netflix \
    log-prefix=nflx-conn new-connection-mark=nflx-conn passthrough=yes \
    protocol=tcp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Disney+ Domains" disabled=yes dst-port=53 layer7-protocol=\
    Disney+ log-prefix=dplus-conn new-connection-mark=dplus-conn passthrough=\
    yes protocol=udp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Disney+ Domains" disabled=yes dst-port=53 layer7-protocol=\
    Disney+ log-prefix=dplus-conn new-connection-mark=dplus-conn passthrough=\
    yes protocol=tcp src-address-list=Client-LAN
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for IOT Devices" dst-address=!192.168.50.19 dst-port=53 \
    in-interface=iot-net-vrrp layer7-protocol=!Netflix log-prefix=iot-dns \
    new-connection-mark=iot-dns passthrough=yes protocol=udp src-address=\
    !192.168.50.19 src-address-list=""
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for IOT Devices" dst-address=!192.168.50.19 dst-port=53 \
    in-interface=iot-net-vrrp layer7-protocol=!Netflix new-connection-mark=\
    iot-dns passthrough=yes protocol=tcp src-address=!192.168.50.19 \
    src-address-list=""
add action=mark-connection chain=prerouting comment=\
    "DNS Mark for Lab Devices" dst-address=!192.168.54.230 dst-port=53 \
    in-interface=lab-vlan-vrrp log-prefix=lab-dns new-connection-mark=lab-dns \
    passthrough=yes protocol=udp src-address-list=!Lab-Whitelist
add action=mark-packet chain=prerouting disabled=yes layer7-protocol=Netflix \
    log=yes new-packet-mark=nflx-pkt passthrough=yes src-address=\
    192.168.104.0/24
add action=mark-connection chain=prerouting comment=\
    "DNS Reroute for VQTV Box" disabled=yes dst-port=53 new-connection-mark=\
    evpn-dns passthrough=yes protocol=tcp src-address=192.168.50.22
add action=mark-connection chain=prerouting comment=\
    "DNS Reroute for VQTV Box" disabled=yes dst-port=53 new-connection-mark=\
    evpn-dns passthrough=yes protocol=udp src-address=192.168.50.22
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    out-interface-list=WAN-All
add action=dst-nat chain=dstnat comment=Plex dst-port=32400 in-interface=\
    ether8-gateway protocol=tcp to-addresses=192.168.48.100 to-ports=32400
add action=dst-nat chain=dstnat comment="Forward to jumphost SSL Multiplexer" \
    dst-port=443 in-interface=ether8-gateway protocol=tcp to-addresses=\
    192.168.122.90 to-ports=443
add action=dst-nat chain=dstnat comment="Forward to jumphost UDP OpenVPN " \
    dst-port=1194 in-interface=ether8-gateway log-prefix=udp-vpn protocol=udp \
    to-addresses=192.168.122.91 to-ports=1194
add action=dst-nat chain=dstnat comment="Redirect IOT UDP DNS to PiHole" \
    connection-mark=iot-dns dst-port=53 log-prefix=iot-dns-nat protocol=udp \
    to-addresses=192.168.50.19 to-ports=53
add action=masquerade chain=srcnat comment="Redirect IOT UDP DNS to PiHole" \
    connection-mark=iot-dns dst-address=192.168.50.19 dst-port=53 log-prefix=\
    iot-dns-masq protocol=udp src-address=192.168.50.0/24
add action=dst-nat chain=dstnat comment="Redirect IOT TCP DNS to PiHole" \
    connection-mark=iot-dns dst-port=53 log-prefix=iot-dns-nat protocol=tcp \
    to-addresses=192.168.50.19 to-ports=53
add action=masquerade chain=srcnat comment="Redirect IOT TCP DNS to PiHole" \
    connection-mark=iot-dns dst-address=192.168.50.19 dst-port=53 log-prefix=\
    iot-dns-masq protocol=tcp src-address=192.168.50.0/24
add action=dst-nat chain=dstnat comment="Redirect Lab UDP DNS to PiHole" \
    connection-mark=lab-dns dst-port=53 log-prefix=lab-dns-nat protocol=udp \
    to-addresses=192.168.54.230 to-ports=53
add action=masquerade chain=srcnat comment="Redirect Lab UDP DNS to PiHole" \
    connection-mark=lab-dns dst-address=192.168.54.230 dst-port=53 \
    log-prefix=lab-dns-masq protocol=udp src-address=192.168.54.0/24
add action=dst-nat chain=dstnat comment=\
    "Redirect UDP DNS for Netflix to Express VPN Mediastreamer" \
    connection-mark=nflx-conn dst-port=53 protocol=udp to-addresses=\
    85.203.37.1 to-ports=53
add action=masquerade chain=srcnat comment=\
    "Redirect UDP DNS via Express VPN Mediastreamer" connection-mark=\
    nflx-conn disabled=yes dst-address=192.168.104.254 dst-port=53 \
    log-prefix=nflx-dns-masq protocol=udp src-address=192.168.104.0/24
add action=dst-nat chain=dstnat comment=\
    "Redirect TCP DNS for Netflix to Express VPN Mediastreamer" \
    connection-mark=nflx-conn dst-port=53 protocol=tcp to-addresses=\
    85.203.37.1 to-ports=53
add action=masquerade chain=srcnat comment=\
    "Redirect TCP DNS via Express VPN Mediastreamer" connection-mark=\
    nflx-conn disabled=yes dst-address=192.168.104.254 dst-port=53 \
    log-prefix=nflx-dns-masq protocol=tcp src-address=192.168.104.0/24
add action=dst-nat chain=dstnat comment=\
    "Redirect UDP DNS for Disney+ to Express VPN Mediastreamer" \
    connection-mark=dplus-conn disabled=yes dst-port=53 protocol=udp \
    to-addresses=85.203.37.1 to-ports=53
add action=dst-nat chain=dstnat comment=\
    "Redirect TCP DNS for Disney+ to Express VPN Mediastreamer" \
    connection-mark=dplus-conn disabled=yes dst-port=53 protocol=tcp \
    to-addresses=85.203.37.1 to-ports=53
/ip route
add disabled=yes distance=1 dst-address=192.168.88.0/24 gateway=\
    "Direct Clients Bridge" routing-mark=wan1
add disabled=yes distance=1 dst-address=192.168.88.0/24 gateway=\
    "Direct Clients Bridge" routing-mark=wan2
add disabled=yes distance=1 gateway=ether7
add disabled=yes distance=1 gateway=ether8-gateway
add comment="VPN TCP" distance=1 dst-address=192.168.126.0/24 gateway=\
    192.168.48.174
add comment="VPN UDP" distance=1 dst-address=192.168.166.0/24 gateway=\
    192.168.48.174
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=192.168.0.0/16
set ssh address=192.168.0.0/16
set api address=192.168.0.0/16
set winbox address=192.168.0.0/16
set api-ssl address=192.168.0.0/16
/ip ssh
set always-allow-password-login=yes forwarding-enabled=remote strong-crypto=\
    yes
/ip upnp interfaces
add interface=ether8-gateway type=external
add interface="Direct Clients Bridge" type=internal
add interface="Home Network Trunk Ports" type=internal
/lcd
set backlight-timeout=10m default-screen=stat-slideshow time-interval=hour
/lcd interface
set sfp-sfpplus1 disabled=yes
set sfp1 disabled=yes
/routing filter
add chain=dynamic-in distance=3 set-routing-mark=wan1
add chain=dynamic-in distance=4 set-routing-mark=wan2
/routing igmp-proxy interface
add alternative-subnets=0.0.0.0/0 interface=trusted-clients-vrrp upstream=yes
add interface=iot-net-vrrp
/routing igmp-proxy mfc
add downstream-interfaces=trusted-clients-net group=239.255.255.250 source=\
    192.168.104.108 upstream-interface=iot-net
add downstream-interfaces=trusted-clients-net group=224.0.0.1 source=\
    192.168.104.108 upstream-interface=iot-net
add downstream-interfaces=trusted-clients-net group=224.0.0.251 source=\
    192.168.104.108 upstream-interface=iot-net
/system clock
set time-zone-name=Asia/Singapore
/system health
set cpu-overtemp-threshold=110C
/system identity
set name="MikroTik CCR1009 Router (Primary)"
/system leds
set 0 interface=sfp-sfpplus1
set 1 interface=sfp-sfpplus1
set 2 interface=sfp1
/system logging
set 0 action=sdcard
set 1 action=disk
set 2 action=disk
add action=sdcard topics=script
add action=disk topics=interface
add action=disk topics=critical
/system ntp client
set enabled=yes server-dns-names=0.sg.pool.ntp.org,1.sg.pool.ntp.org
/system scheduler
add comment="Update DNS-O-Matic IP on reboot" name=dynip-update on-event=\
    force-update-odns policy=ftp,read,write,policy,test start-time=startup
add comment="Check for Dynamic IP updates 15 minutes past midnight every day" \
    interval=1d name=dynip-24hupdate on-event=check-update-odns policy=\
    ftp,read,write,policy,test start-date=sep/12/2020 start-time=00:15:00
add comment="Disable WAN Netwatch on Reboot for Primary Router" name=\
    netwatch-disable on-event=disable-wan-netwatch policy=\
    reboot,read,write,policy,test start-time=startup
add comment="Enable WAN Netwatch on Reboot for Primary Router" name=\
    netwatch-enable on-event=enable-wan-netwatch policy=\
    reboot,read,write,policy,test start-time=startup
add dont-require-permissions=no name=vlan-down owner=admin policy=\
    reboot,read,write,test source=":local i value=0;\r\
    \n:while ((\$i < 20) && ([/ping address=8.8.8.8 interval=3 count=1]=0)) do\
    ={\r\
    \n    :set i value=(\$i+1)\r\
    \n    :delay 3s;\r\
    \n    } ;\r\
    \n:if (\$i=20) do={\r\
    \n:log info message=\"Warning: 10 unsuccessful pings to 8.8.8.8\";\r\
    \n# Disable VLAN Interfaces based on VLAN ID's\r\
    \n/interface vlan disable [/interface vlan find vlan-id=20]\r\
    \n/interface vlan disable [/interface vlan find vlan-id=50]\r\
    \n/interface vlan disable [/interface vlan find vlan-id=54]\r\
    \n/interface vlan disable [/interface vlan find vlan-id=75]\r\
    \n/interface vlan disable [/interface vlan find vlan-id=104]\r\
    \n/interface vlan disable [/interface vlan find vlan-id=122]\r\
    \n\r\
    \n# VRRP for Management IP's is not tied to a VLAN so we must disable the \
    VRRP\r\
    \n/interface vrrp disable [/interface vrrp find vrid=48]\r\
    \n};"
add dont-require-permissions=no name=vlan-up owner=admin policy=\
    reboot,read,write,test source="# Enable VLAN Interfaces based on VLAN ID's\
    \r\
    \n/interface vlan enable [/interface vlan find vlan-id=20]\r\
    \n/interface vlan enable [/interface vlan find vlan-id=50]\r\
    \n/interface vlan enable [/interface vlan find vlan-id=54]\r\
    \n/interface vlan enable [/interface vlan find vlan-id=75]\r\
    \n/interface vlan enable [/interface vlan find vlan-id=104]\r\
    \n/interface vlan enable [/interface vlan find vlan-id=122]\r\
    \n\r\
    \n# VRRP for Management IP's is not tied to a VLAN so we must enable the V\
    RRP\r\
    \n/interface vrrp enable [/interface vrrp find vrid=48]"
add dont-require-permissions=no name=dhcp2dns owner=admin policy=\
    reboot,read,write,test source="# DNS TTL to set for DNS entries\r\
    \n:local dnsttl \"00:15:00\";\r\
    \n\r\
    \n###\r\
    \n# Script entry point\r\
    \n#\r\
    \n# Expected environment variables:\r\
    \n# leaseBound         1 = lease bound, 0 = lease removed\r\
    \n# leaseServerName    Name of DHCP server\r\
    \n# leaseActIP         IP address of DHCP client\r\
    \n#leaseActMAC      MAC address of DHCP client\r\
    \n###\r\
    \n\r\
    \n# \"a.b.c.d\" -> \"a-b-c-d\" for IP addresses used as replacement for mi\
    ssing host names\r\
    \n:local ip2Host do=\\\r\
    \n{\r\
    \n  :local outStr\r\
    \n  :for i from=0 to=([:len \$inStr] - 1) do=\\\r\
    \n  {\r\
    \n    :local tmp [:pick \$inStr \$i];\r\
    \n    :if (\$tmp =\".\") do=\\\r\
    \n    {\r\
    \n      :set tmp \"-\"\r\
    \n    }\r\
    \n    :set outStr (\$outStr . \$tmp)\r\
    \n  }\r\
    \n  :return \$outStr\r\
    \n}\r\
    \n\r\
    \n:local mapHostName do={\r\
    \n# param: name\r\
    \n# max length = 63\r\
    \n# allowed chars a-z,0-9,-\r\
    \n  :local allowedChars \"abcdefghijklmnopqrstuvwxyz0123456789-\";\r\
    \n  :local numChars [:len \$name];\r\
    \n  :if (\$numChars > 63) do={:set numChars 63};\r\
    \n  :local result \"\";\r\
    \n\r\
    \n  :for i from=0 to=(\$numChars - 1) do={\r\
    \n    :local char [:pick \$name \$i];\r\
    \n    :if ([:find \$allowedChars \$char] < 0) do={:set char \"-\"};\r\
    \n    :set result (\$result . \$char);\r\
    \n  }\r\
    \n  :return \$result;\r\
    \n}\r\
    \n\r\
    \n:local lowerCase do={\r\
    \n# param: entry\r\
    \n  :local lower \"abcdefghijklmnopqrstuvwxyz\";\r\
    \n  :local upper \"ABCDEFGHIJKLMNOPQRSTUVWXYZ\";\r\
    \n  :local result \"\";\r\
    \n  :for i from=0 to=([:len \$entry] - 1) do={\r\
    \n    :local char [:pick \$entry \$i];\r\
    \n    :local pos [:find \$upper \$char];\r\
    \n    :if (\$pos > -1) do={:set char [:pick \$lower \$pos]};\r\
    \n    :set result (\$result . \$char);\r\
    \n  }\r\
    \n  :return \$result;\r\
    \n}\r\
    \n\r\
    \n:local token \"\$leaseServerName-\$leaseActMAC\";\r\
    \n:local LogPrefix \"DHCP2DNS (\$leaseServerName)\"\r\
    \n\r\
    \n:if ( [ :len \$leaseActIP ] <= 0 ) do=\\\r\
    \n{\r\
    \n  :log error \"\$LogPrefix: empty lease address\"\r\
    \n  :error \"empty lease address\"\r\
    \n}\r\
    \n\r\
    \n:if ( \$leaseBound = 1 ) do=\\\r\
    \n{\r\
    \n  # new DHCP lease added\r\
    \n  /ip dhcp-server\r\
    \n  #:local dnsttl [ get [ find name=\$leaseServerName ] lease-time ]\r\
    \n  network\r\
    \n  :local domain [ get [ find \$leaseActIP in address ] domain ]\r\
    \n  #:log info \"\$LogPrefix: DNS domain is \$domain\"\r\
    \n\r\
    \n  :local hostname [/ip dhcp-server lease get [:pick [find mac-address=\$\
    leaseActMAC and server=\$leaseServerName] 0] value-name=host-name]\r\
    \n  #:log info \"\$LogPrefix: DHCP hostname is \$hostname\"\r\
    \n\r\
    \n #Hostname cleanup\r\
    \n  :if ( [ :len \$hostname ] <= 0 ) do=\\\r\
    \n  {\r\
    \n    :set hostname [ \$ip2Host inStr=\$leaseActIP ]\r\
    \n    :log info \"\$LogPrefix: Empty hostname for '\$leaseActIP', using ge\
    nerated host name '\$hostname'\"\r\
    \n  }\r\
    \n  :set hostname [\$lowerCase entry=\$hostname]\r\
    \n  :set hostname [\$mapHostName name=\$hostname]\r\
    \n  #:log info \"\$LogPrefix: Clean hostname for FQDN is \$hostname\";\r\
    \n\r\
    \n  :if ( [ :len \$domain ] <= 0 ) do=\\\r\
    \n  {\r\
    \n    :log warning \"\$LogPrefix: Empty domainname for '\$leaseActIP', can\
    not create static DNS name\"\r\
    \n    :error \"Empty domainname for '\$leaseActIP'\"\r\
    \n  }\r\
    \n\r\
    \n  :local fqdn (\$hostname . \".\" .  \$domain)\r\
    \n  #:log info \"\$LogPrefix: FQDN for DNS is \$fqdn\"\r\
    \n\r\
    \n    :if ([/ip dhcp-server lease get [:pick [find mac-address=\$leaseActM\
    AC and server=\$leaseServerName] 0] ]) do={\r\
    \n      # :log info message=\"\$LogPrefix: \$leaseActMAC -> \$hostname\"\r\
    \n      :do {\r\
    \n        /ip dns static add address=\$leaseActIP name=\$fqdn ttl=\$dnsttl\
    \_comment=\$token;\r\
    \n      } on-error={:log error message=\"\$LogPrefix: Failure during dns r\
    egistration of \$fqdn with \$leaseActIP\"}\r\
    \n    }\r\
    \n\r\
    \n} else={\r\
    \n# DHCP lease removed\r\
    \n  /ip dns static remove [find comment=\$token];\r\
    \n}"
add dont-require-permissions=no name=testscript owner=admin policy=\
    read,write,policy,test source=\
    "/tool netwatch enable [/tool netwatch find comment=\"WAN\"]"
add dont-require-permissions=no name=disable-wan-netwatch owner=admin policy=\
    reboot,read,write,policy,test source=\
    "/tool netwatch disable [find where host=\"8.8.8.8\"]"
add dont-require-permissions=no name=enable-wan-netwatch owner=admin policy=\
    reboot,read,write,policy,test source=":log info message=\"Disabing externa\
    l netwatch via startup script\";\r\
    \n/tool netwatch disable [find where host=\"8.8.8.8\"]\r\
    \n\r\
    \n:while ([ :len [ / interface detect-internet state find where state=inte\
    rnet ] ] = 0) do={\r\
    \n  :delay 2000ms;\r\
    \n}\r\
    \n\r\
    \n:log info message=\"Enabling external netwatch via startup script\";\r\
    \n/tool netwatch enable [find where host=\"8.8.8.8\"]"
/tool bandwidth-server
set enabled=no
/tool graphing interface
add allow-address=192.168.0.0/16 interface=ether8-gateway
add allow-address=192.168.48.0/24 interface="Home Network Trunk Ports"
add allow-address=192.168.104.0/24 interface="Home Network Trunk Ports"
add allow-address=192.168.88.0/24 interface="Direct Clients Bridge"
/tool graphing resource
add allow-address=192.168.48.0/24
add allow-address=192.168.104.0/24
add allow-address=192.168.88.0/24
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool mac-server ping
set enabled=no
/tool netwatch
add comment=WAN down-script=":log info message=\"External Netwatch failed. Exe\
    cuting down script now\";\r\
    \nvlan-down" host=8.8.8.8 interval=30s timeout=2s up-script=":log info mes\
    sage=\"External Netwatch succeeded. Executing up script now\";\r\
    \nvlan-up"
add comment="Secondary Router Monitor" down-script="/tool e-mail send to=me@th\
    eaveragegeek.com subject=\"Secondary Router Down\" body=\"Netwatch Test fr\
    om Primary Router Failed\"" host=192.168.20.2 interval=30s up-script="/too\
    l e-mail send to=me@theaveragegeek.com subject=\"Secondary Router Availabl\
    e\" body=\"Netwatch Test from Primary Router Suceeded\""
/tool romon port
add

biomesh · Tue Nov 09, 2021 5:34 pm

The ccr1009 can route around 7-8 Gbps with or without a bridge involved. It really does not matter with the tests I have done in the past.

anav · Tue Nov 09, 2021 7:34 pm

What is the purpose of the connection between the UNIFI switch and the CRS309??
This could lead to issues?
The only advantage I could see is if there was much traffic on one VLAN that involved like servers/users where they were split up on both switches. The traffic should be able to travel between the switches without having to go through the router I suppose???

So in this case, Trunk port from Router to MT switch, Trunk port from Router to Unifi switch. Trunk port connecting MT switch to Unifi switch.
On both MT devices ensure you set bridge STP to MSTP!

mkx · Tue Nov 09, 2021 9:34 pm

I mentioned possible loops in one of my previous posts and it seems you have it. So until you are sure your SFP-related config is right, disconnect ether1. Or use ether1 to directly connect management PC, just don't make any loop in your network.

anav · Tue Nov 09, 2021 10:54 pm

Starting with the CCR1009, the main change is not to mix vlans and not vlans.
Best/simple/clear to use all vlans...... (by that I mean not vlan1 for anything other than default bridge settings).

So Convert/Create........
Create vlan11 with Ip address 192.168.48.0/24, as well as IP pool, Dhcp server, DHCP Server network.
This appears to be your management VLAN!!
All smart devices should have an IP address from this subnet!!

I see that VLAN104 is also going to the switch.

Conclusion: You only have two subnets going to the CR switch. Then why do you need to connect them to the UNIFI switch?
If there is no vlan104 on the unifi switch (no possible direct traffic switch to switch) then what is the point? Any vlan X, to vlanY traffic will have to go through the router anyway?

In general, all the vlan interfaces should be created with interface being the ONE bridge port, with appropriate /interface bridge port settings and appropriate /interface bridge vlan settings.
(I see sfp+8 is not even on the bridge port settings)

Unless this actually serves some purpose would set all to NONE!
/interface detect-internet
set detect-interface-list=Native-WAN internet-interface-list=WAN-All \
lan-interface-list=LAN wan-interface-list=WAN-All

Dislike your interface list naming convention, NOT clear and is confusing.
Stick to norms.
/interface list
add name=WAN-All
add name=LAN
add name=Native-WAN

Try
WAN (for WAN connections)
LAN (for all vlans)
control (for management vlan)
Internet (for vlans allowed internet access) (not required if ALL VLANS are allowed internet)

Rule of thumb: if you have two or more subnets (but not all of them) that are allowed or should have access to something create an interface list!!

For example these are all better setup as 4 interface lists (vice source-address lists).
SENSITIVE
IOT-WL
CLIENT-LAN
EXTERNAL-CLIENT

Final comment on the topic, firewall address lists are best used when you have a subset of IPs from a subnet involved, any mix of IPs across subnets, or any mix of IPs and whole subnets.

In terms of interface list members.
for LAN, ensure you name all vlans (bridge not required).
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

FIREWALL Rules
INPUT CHAIN
These are kind of not all that useful. Remember the rule above already blocked all WAN traffic (not dst-natted).

add action=drop chain=input comment="Block Winbox connections on WAN" \
dst-port=8291 in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment=\
"Block Mikrotik Bandwidth Test connections on WAN" dst-port=2000 \
in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound TCP DNS" dst-port=53 \
in-interface=ether8-gateway protocol=tcp
add action=drop chain=input comment="Drop inbound UDP DNS" disabled=yes \
dst-port=53 in-interface=ether8-gateway protocol=udp
add action=drop chain=input comment=\
"Drop all packets which does not have unicast source IP address" \
src-address-type=!unicast
add action=drop chain=input comment="Drop all packets from public internet whi\
ch should not exist in public network" in-interface-list=WAN-All \
src-address-list=NotPublic

In any case, they can all be replaced by three simple rules.
1 - only allow admin to router FROM the LAN (no LAN users require full access to the router only the admin)
2- allow required services traffic from LAN users to the router (DNS.NTP etc.)
3. block all else

(1) Looks like
add action=accept chain=input in-interface-list=Control (or in-interface=vlan11) {optional: src-address-list=adminaccess}
where firewall address list "adminaccess" is comprised of static dhp leases
add address=IP of admin desktop list=adminaccess
add address=IP of admin laptop list=adminaccess
add address=IP of admin smartphone list=adminaccess
etc.....

(2) allow DNS queries (if the router provides ntp that as well should be added here udp port 123l)
add action=accept chain=input comment="Allow LAN DNS queries-UDP" \
connection-state=new dst-port=53,123 in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN DNS queries - TCP" \
connection-state=new dst-port=53 in-interface-list=LAN protocol=tcp

(3) add action=drop chain=input comment="drop all else"

FORWARD CHAIN
Stick to the defaults as provided, getting too fancy is best left to the necessary admin entered rules where you delineate what traffic is ALLOWED.
Just like the input chain, we use a drop everything else rule at the end very much simplifying your rules!!

So recommend
add action=accept chain=forward comment="defconf: accept in ipsec policy" \ (if not doing ipsec these 2 rules can be disabled/removed)
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid log=yes log-prefix=invalid
******************************************************************************
add action=drop chain=forward comment="drop all else"

******* This is where you need to put what traffic IS ALLOWED *******************
Typically allow LAN to WAN for internet
Typically allow a shared device across vlans (like a printer)
Typically allow admin access to all Vlans
Typically allow dst-nat (if doing any port forwarding)

Now I see you have some specific concerns that touch upon
a. bad traffic leaving the LAN Towards the internet
c. One IP on one vlan (dmz) has a specific requirement

SO you have to decide what you need to block before the following more general allow rule for LAN to Internet traffic!!!
Note that by default ALL vlan to vlan traffic is already blocked, so do not have to worry about that
Note that by default all LAN to WAN traffic is blocked
Note by default all WAN to LAN traffic is blocked.

For example these rules............. What are you actually trying to achieve??

1. add action=drop chain=forward log-prefix=DMZ protocol=tcp src-address=\
192.168.122.90 src-port=!443
It seems this is an internet access attempt to block all http traffic to the internet (but you are missing out-interface-list=WAN and protocol tcp )

Since the rest of the DMZ it appears needs full access to the internet, we need to match a rule for this particular Device JUST BEFORE the ALLOW RULE for lan to wan traffic.
add action=drop chain=forward in-interface=vlanDMZ source-address=192.168.122.90 dst-port=!443 protocol=tcp out-interface-list=WAN

That is the simple and most direct way.
The other is to create an Internet Interface list that the DMZ does not belong too. So the general rule does not apply to the DMZ.
Then create two rules and order in relation to the general rule does not matter.
add chain=forward action=accept in-interface=vlandmz source-address=!192.168.122.90 out-interface-list=WAN ( allow all traffic from rest of DMZ)
add chain=forward action=accept in-interface=vlandmz source-address=192.168.122.90 dst-port=443 out-interface-list=WAN (allow only HTTPS traffic from specific dmz IP)

2. add action=drop chain=forward comment="Drop all packets from public internet w\
hich should not exist in public network" in-interface-list=WAN-All \
src-address-list=NotPublic
This looks to ensure, that no internet traffic that is not public should enter the router?
NOT REQUIRED: All WAN to LAN traffic is blocked by default!!

3. add action=drop chain=forward comment="Drop all packets in local network which\
\_does not have local network address" in-interface-list=LAN src-address=\
!192.168.0.0/16
NOT REQUIRED: All vlan to vlan traffic is blocked by default.

4. add action=drop chain=forward comment="Drop all packets from local network to \
internet which should not exist in public network" disabled=yes \
dst-address-list=NotPublic in-interface-list=LAN out-interface-list=WAN
This rule wants to ensure that no bad internet (dst) addressess are sent out from the router.
Not a bad rule but need to add out interface.
Better (courtesy of sindy) is to create black hole routes for such addresses.

5. add action=drop chain=forward comment=\
"Block IOT Traffic to critical LAN Segments" connection-state=\
invalid,new,untracked dst-address-list=Sensitive-Internal in-interface=\
iot-net log=yes log-prefix=iot-drop src-mac-address=!90:DD:5D:CA:59:A7
add action=accept chain=forward comment=\
NOT REQUIRED, all vlan to vlan traffic is blocked by default.

Last comment. clean up these settings for security
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=192.168.0.0/16
set ssh address=192.168.0.0/16
set api address=192.168.0.0/16
set winbox address=192.168.0.0/16
set api-ssl address=192.168.0.0/16

Only enabled ones should be winbox and perhaps SSH, unless you have specific reasons/needsd.
certainly WWW is a bad idea and API.
The winbox should be matched to your control subnet!

avggeek · Wed Nov 10, 2021 1:25 pm

@anav I will post a separate reply to your comments on the config. But I will just say that with the exception of the firewall rules (which I don't fully understand the proposed changes although I'm sure you are correct), most of the other config items have a specific purpose and are not just randomly thrown in there.

I've made some progress on my original problem, by making the following changes:

On the CRS-309, removed eth1 from the bridge and removed any IP addresses assigned to eth1 (as @mkx suggested)

On the CCR-1009, added the SFP+ port to the default bridge ("Direct Clients Bridge") which has an IP address 192.168.88.3/24 (as suggested earlier)

On the CRS-309, set the IP address of the Bridge to 192.168.88.20/24 (matching with the IP range of the bridge on the CCR-1009)

On the CRS-309, added a default route for 0.0.0.0/0 pointing to the CCR-1009 IP - 192.168.88.3/24

With this configuration, I'm now able to ping hosts in the network from the CRS-309 switch across multiple VLAN's and clients behind the CRS-309 can also connect to other devices in other VLAN's (with exactly one exception). There is one particular host on my network which for some reason cannot be reached via IP addresses it has in the 192.168.104.0/24 or 192.168.20.0/24 range. It can only be reached via an IP address in the 192.168.48.0/24 range (VLAN1). Other multi-homed devices on the network are reachable so of the 30-odd devices on the network, 29 work fine with the above config - I have to admit I'm really baffled by this now.

I realize also that one of my mistakes was that I was trying to have multiple IP's in the same subnet on diff. interfaces on my CCR-1009, which is clearly highlighted in the wiki as a mistake.

avggeek · Thu Nov 11, 2021 10:48 am

There is one particular host on my network which for some reason cannot be reached via IP addresses it has in the 192.168.104.0/24 or 192.168.20.0/24 range. It can only be reached via an IP address in the 192.168.48.0/24 range (VLAN1).

Turns out I needed to enable rp_filter on those interfaces in the host itself and once I did that, the host was reachable from the CRS. I'm now struggling with VLAN's on the CRS but will create a separate thread for that.

CRS309 Switch - cannot ping gateway or any other host

CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Re: CRS309 Switch - cannot ping gateway or any other host

Who is online