Multi WAN PBR problem

RouterOS General
1

Hi guys!

First of all, hats off to you for this amazing community!

I volunteer for an NGO that deals with rehabilitating schools. One school I was assigned to got a donation of one CCR2004-1G-12S+2XS, a CRS309 8 port SFP+ switch and a CRS328 24 port PoE switch. This school is in a remote rural setting and has been the focus of multiple NGOs for developing it into a nice summer camp setting.

I am not all that familiar to Mikrotik. They plan on having up to 6 WAN connections due to multiple factors (fiber goes down from time to time, weather affects the backup 4G line etc) and need some policies in place in order to route every connection to the right gateway. If this was a (pf/opn)sense scenario, I wouldn’t be here. :slight_smile:

Requirements:

  1. up to 6 WAN connections of multiple types.
    a. WAN1 is a 2.5G fiber connection from a neighbouring factory that has its Internet connection God-knows-where-from. It gets a public IP via DHCP (not RFC1918) if connected to VLAN35.
    b. WAN2 is a 1G PPPoE fiber connection from the only ISP that managed to get their services to that remote area but is very unreliable due to (yet) unknown reasons.
    c. WAN3 will be a starlink connection in the near future. Max 250MB/s
    d. WAN4 is for future use. A possible wireless uplink to a nearby village (WISP scenario?). I don’t know when this will happen or if it will a public or a private IP leased from that end.
    e. WAN5 & WAN6 will be 4G/5G backup lines. Which means DHCP leased private IPs, nost likely.

  2. Multiple VLANs, each with its own policy.
    a. The main network (VLAN1) should use WAN2 as the primary internet connection with the rest of the Internet connections as failover. Easily done with distance.
    b. VLAN 173: uses WAN2 as the primary and WAN1 as failover.
    c. VLAN 187: uses WAN1 as the primary and WAN2 as failover
    d. VLAN 199: management VLAN. WAN1 as the primary, the rest of the WAN connections as backup. This VLAN is the most important in terms of internet connectivity.
    e. VLAN 773: load balancing WAN1, WAN2 and WAN3 (take into account the weights based on bandwidth).
    f. VLAN 783: load balancing WAN1, WAN2 and WAN3 (take into account the weights based on bandwidth). I would absolutely implement some queue system as well. This will be the generally available student network.
    g. VLAN 373: security cameras. Did not yet come up with a strategy for this network. Suggestions are welcome, ofc.
    h. VLAN 271: IoT VLAN. Same as 373, no strategy yet.

  3. Other considerations:
    a. Some destinations (hostnames or IP addresses) need to be reached via a certain WAN interface due to IP whitelisting.
    b. Same logic needs to be applied to certain protocols (Eg: all FTP traffic should be routed through WAN1).
    c. For some IoT devices, I’ll have to configure a Wireguard Tunnel to a NordVPN UK server in order to set up. TL;DR of it is: some IoT devices are location locked to UK so they need this in order to properly install and configure them.
    d. A DMZ VLAN will be added and some services will be exposed to the internet (ports 80, 443, 21 etc) on WAN1 and WAN2 public IP addresses. Both IPs will be an A record for some subdomains (using DDNS for WAN2)
    e. The admin of the school asked if I could install tailscale and expose some routes in order to expose a subnet from this installation to another subnet from a remote highschool and viceversa. I don’t have the exact details of this yet.

That’s it for now. A pretty complex installation.


Where I am now:

I added the WAN interfaces and some VLANs. I tried to configure 2b and 2c using PBR but it’s a mess. When WAN1 and WAN2 are both up the local computers go absolutely bananas. By that I mean that no website is loading correctly. My suspicion is that the connections are not properly marked so they create noise rather then get in and get out as they should. Eg: if loading facebook, it seems that only 1/3 of the assets (JS, img etc) are loading. This happens for both VLAN 173 and 187.

If this were for personal use I would absolutely spend more time and energy with it, seeing as it’s an amazing learning opportunity. Unfortunately the summer camps are due soon (3 weeks?) hence the urgency.
And they don’t want me installing a pfSense server. :roll_eyes:

Thank you all so much for the help! :folded_hands:

# 2025-05-28 09:21:49 by RouterOS 7.19.1
# software id = YWAP-ZWRK
#
# model = CCR2004-1G-12S+2XS
/interface bridge
add comment=defconf name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] auto-negotiation=no speed=2.5G-baseX
set [ find default-name=sfp-sfpplus2 ] name=wan2-fiber
set [ find default-name=sfp-sfpplus3 ] name=wan3
set [ find default-name=sfp-sfpplus4 ] name=wan4
set [ find default-name=sfp-sfpplus5 ] name=wan5
set [ find default-name=sfp-sfpplus6 ] name=wan6
/interface pppoe-client
add disabled=no interface=wan2-fiber name=wan2 user=thisismyuser password=thisismypass
/interface vlan
add comment=MANAGEMENT interface=bridge name=mgmt-vlan vlan-id=199
add interface=bridge name=orange-vlan vlan-id=187
add interface=bridge name=rds-vlan vlan-id=173
add interface=sfp-sfpplus1 name=wan1 vlan-id=35
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=dhcp_main_network ranges=10.137.39.50-10.137.39.200
add name=dhcp_pool_rds_clients ranges=192.168.173.101-192.168.173.200
add name=dhcp_pool_orange_clients ranges=192.168.187.101-192.168.187.200
add name=dhcp_pool_mgmt ranges=10.199.199.101-10.199.199.200
/ip dhcp-server
add address-pool=dhcp_main_network interface=bridge name=dhcp_main_network
add address-pool=dhcp_pool_rds_clients interface=rds-vlan lease-time=1h name=dhcp_rds_clients
add address-pool=dhcp_pool_orange_clients interface=orange-vlan lease-time=1h name=dhcp_orange_clients
add address-pool=dhcp_pool_mgmt comment="Management DHCP Server" interface=mgmt-vlan name=dhcp_mbmt_cleints
/port
set 0 name=serial0
/routing table
add disabled=no fib name=wan1-rt
add disabled=no fib name=wan2-rt
add disabled=no fib name=wan3-rt
add disabled=no fib name=wan4-rt
add disabled=no fib name=wan5-rt
add disabled=no fib name=wan6-rt
add disabled=no fib name=wan1-first-rt
add disabled=no fib name=wan2-first-rt
/certificate settings
set builtin-trust-anchors=not-trusted
/interface bridge port
add bridge=bridge comment=defconf frame-types=admit-only-untagged-and-priority-tagged interface=sfp-sfpplus7 pvid=173
add bridge=bridge comment=defconf frame-types=admit-only-untagged-and-priority-tagged interface=sfp-sfpplus8 pvid=173
add bridge=bridge comment=defconf interface=sfp-sfpplus9 pvid=187
add bridge=bridge comment=defconf interface=sfp-sfpplus10 pvid=187
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp28-1
add bridge=bridge comment=defconf interface=sfp28-2
add bridge=bridge interface=ether1
/ip firewall connection tracking
set enabled=yes
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus11,sfp-sfpplus12,sfp28-1,sfp28-2 vlan-ids=187
add bridge=bridge tagged=bridge,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-sfpplus12,sfp28-1,sfp28-2 vlan-ids=173
add bridge=bridge tagged=bridge,sfp-sfpplus11,sfp-sfpplus12 vlan-ids=199
/interface list member
add comment=defconf interface=bridge list=LAN
add interface=wan1 list=WAN
add interface=wan2 list=WAN
add interface=wan3 list=WAN
add interface=wan4 list=WAN
add interface=wan5 list=WAN
add interface=wan6 list=WAN
add interface=orange-vlan list=LAN
add interface=rds-vlan list=LAN
add interface=mgmt-vlan list=LAN
/ip address
add address=10.137.36.1/22 comment=defconf interface=bridge network=10.137.36.0
add address=192.168.173.1/24 comment="RDS Clients Gateway" interface=rds-vlan network=192.168.173.0
add address=192.168.187.1/24 comment="Orange Clients Gateway" interface=orange-vlan network=192.168.187.0
add address=10.199.199.1/24 comment="Management VLAN" interface=mgmt-vlan network=10.199.199.0
/ip dhcp-client
add comment="Orange static DHCP lease" default-route-distance=11 default-route-tables=main !dhcp-options interface=wan1 use-peer-dns=no
# Interface not active
add comment=defconf default-route-distance=31 interface=wan3
# Interface not active
add comment=defconf default-route-distance=41 interface=wan4
# Interface not active
add comment=defconf default-route-distance=51 interface=wan5
# Interface not active
add comment=defconf default-route-distance=61 interface=wan6
/ip dhcp-server network
add address=10.137.36.0/22 comment="Main network" dns-server=10.137.36.1 gateway=10.137.36.1 netmask=22
add address=10.199.199.0/24 comment="Management Network" dns-server=10.199.199.1 gateway=10.199.199.1
add address=192.168.173.0/24 comment="RDS Clients Network" dns-server=192.168.173.1 gateway=192.168.173.1
add address=192.168.187.0/24 comment="Orange Clients Network" dns-server=192.168.187.1 gateway=192.168.187.1
/ip dns
set allow-remote-requests=yes servers=9.9.9.9,1.1.1.1
/ip firewall address-list
add address=192.168.173.0/24 list=wan2-first
add address=192.168.187.0/24 list=wan1-first
add address=192.168.173.0/24 list=local-nets
add address=192.168.187.0/24 list=local-nets
add address=10.199.199.0/24 list=local-nets
add address=10.137.36.0/22 list=local-nets
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="allow IPsec NAT" dst-port=4500 protocol=udp
add action=accept chain=input comment="allow IKE" dst-port=500 protocol=udp
add action=accept chain=input comment="allow l2tp" dst-port=1701 protocol=udp
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!local-nets dst-address-type=!local new-connection-mark=wan1-first-mark src-address-list=wan1-first
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!local-nets dst-address-type=!local new-connection-mark=wan2-first-mark src-address-list=wan2-first
add action=mark-routing chain=prerouting connection-mark=wan1-first-mark in-interface-list=LAN new-routing-mark=wan1-first-rt
add action=mark-routing chain=prerouting connection-mark=wan2-first-mark in-interface-list=LAN new-routing-mark=wan2-first-rt 
add action=mark-routing chain=output connection-mark=wan1-first-mark new-routing-mark=wan1-first-rt
add action=mark-routing chain=output connection-mark=wan2-first-mark new-routing-mark=wan2-first-rt
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=wan1 new-connection-mark=wan1-first-mark
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=wan2 new-connection-mark=wan2-first-mark
add action=mark-routing chain=prerouting in-interface-list=WAN new-routing-mark=main routing-mark=wan1-first-rt
add action=mark-routing chain=prerouting in-interface-list=WAN new-routing-mark=main routing-mark=wan2-first-rt
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip route
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=wan3 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=4 dst-address=0.0.0.0/0 gateway=wan4 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=5 dst-address=0.0.0.0/0 gateway=wan5 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=6 dst-address=0.0.0.0/0 gateway=wan6 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=109.166.202.220/32 gateway=<wan1-gateway> routing-table=main scope=10 suppress-hw-offload=no
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=main suppress-hw-offload=no target-scope=11
add disabled=no distance=2 dst-address=1.0.0.1/32 gateway=<wan2-gateway> routing-table=main scope=10 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=99 dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=wan2-first-rt scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=wan2-first-rt scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=wan1-first-rt scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=99 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=wan1-first-rt scope=10 suppress-hw-offload=no target-scope=11
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" dst-port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/routing rule
add action=lookup-only-in-table comment="wan2 first" disabled=no routing-mark=wan2-first-rt table=wan2-first-rt
add action=lookup-only-in-table comment="wan1 first" disabled=no routing-mark=wan1-first-rt table=wan1-first-rt
Configuration Parser Spreadsheet
Configuration Parser Spreadsheet
2

At first sight, your configuration is very neat and ordered (good :slight_smile: ) but I believe you have not correct scope/target-scope on the routes.
See starting from here:
http://forum.mikrotik.com/t/primary-gateway-with-static-ip-address-not-activating/183380/1

The simplified approach:
http://forum.mikrotik.com/t/primary-gateway-with-static-ip-address-not-activating/183380/1
make use of 10, 11 and 12, but you have only 10 and 11.

Also (but let’s wait some of the more expert members’ comments) it seems to me that some issues may be related to mangle connection-marks/routing rules. :confused:

3

That’s a start. Nice catch, thank you!

4

Also, seen in my (newish, experimental :open_mouth: ) parser (shameless plug :wink: ) :
http://forum.mikrotik.com/t/configuration-parser-spreadsheet/183922/1
it seems like you have some empty fields in routes settings.
route_check.jpg

5

ok, so in this section here

/ip route
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=wan3 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=4 dst-address=0.0.0.0/0 gateway=wan4 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=5 dst-address=0.0.0.0/0 gateway=wan5 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=6 dst-address=0.0.0.0/0 gateway=wan6 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=109.166.202.220/32 gateway=<wan1-gateway> routing-table=main scope=10 suppress-hw-offload=no
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=main suppress-hw-offload=no target-scope=11
add disabled=no distance=2 dst-address=1.0.0.1/32 gateway=<wan2-gateway> routing-table=main scope=10 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=main scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=99 dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=wan2-first-rt scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=wan2-first-rt scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=wan1-first-rt scope=10 suppress-hw-offload=no target-scope=11
add disabled=no distance=99 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=wan1-first-rt scope=10 suppress-hw-offload=no target-scope=11

I fall short in understanding where I should change the target scope. I was lead to believe that, seeing as 1.0.0.1 and 109.166.202.220 (the DNS servers I use for recursive routing) are used 3 times but in different routing tables, the target scope should be +1 form scope.
Worth noting that I have yet to fully understand the scope/target scope mechanism.

6

You absolutely rock for creating that spereadsheet! :astonished_face:

Maybe a UI bug but that route seems to have the distance set.

7

Which makes two (an even, round number) of us. :slight_smile:

Hence the simplified approach, quoting myself:

  1. Both routes must have scope=10
  2. The “narrow” (/32) route to the canary must have target-scope=11
  3. The “wide” (0.0.0.0/0) route must have target scope=12

so, one of them (example) should be (IMHO):

add disabled=no distance=2 dst-address=1.0.0.1/32 gateway= routing-table=main scope=10 suppress-hw-offload=no target-scope=> 10 > > 11
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=1.0.0.1 routing-table=main scope=10 suppress-hw-offload=no target-scope=> 11 > > 12

Ahh, and I believe you need a different canary for each recursive route.

The route is missing distance also in the original config you posted:

add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=109.166.202.220 routing-table=main suppress-hw-offload=no target-scope=11

maybe it is some sort of false positive/rendering issue in the UI?

But is that webfig?
(if you can, avoid webfig and use Winbox, possibly the 3.x version)

8

Hmm, did some changes in the target scopes but still get noise instead of routed connections. Let’s see what the mangle gods have to say.
Yes, webfig. I’m on macOS like a proper hipster, hence no winbox. SSH and HTTP only, ftm.

Thanks for your help and patience, jaclaz!

9

Try Winbox 4.

It should work, besides with the crappy Windows we (good?) ol’ time dinosaurs use (Windows XP, and SP2 while we are at it, in my case, and yes, I know) also with (hard core) Linux and (hipster) Mac, though being still in the works it has still some corners that needs be rounded.

Winbox 3 should be working through Wine.

The advanced users normally use terminal (via Winbox or ssh) anyway, so possible bugs in the rendering of Webfig are usually the last ones to be found/reported and later fixed by the good Mikrotik guys.

In order of reliability:

  1. terminal
  2. winbox (3)
  3. winbox (4)
  4. webfig
    the last two could be viceversa.
10

Confirm please if you get a private IP on WAN1 ( you seem to indicate its actually terminated at another site).
( I do see vlan35 in the config, so should we assume that the other site simply forwarding the ISP to you over vlan35 and thus you do get a public IP?)
TO BE CLEAR… is vlan35 from the ISP, or is the other site simply using vlan35 to forward the WAN traffic between the two devices ???

Network diagram will help as well.
Be advised, that mangling takes precedence over routing rules if there is any overlap!
DO NOT mix bridge with DHCP once you go vlans, simply create another vlan if you need another subnet!!!
which ports was the main network (now vlan137) being supplied on ???

You do not clearly state which ports are trunk ports to the two switches…guessing sfp11 and sfp12 ??
What are on the other sfp ports… 7,8,9,10, 28-1 and 28-2 ???
( it would appear 7,8 going to dumb red devices, and 9,10 going to dumb orange devices, all four being access ports ???)
( no clue as to 28-1 and 28-2 which seem to needing both vlans 173 and 187 and thus trunk ports ( aka going to smart devices ??)

11

ADDITIONS/CHANGES ONLY
Added working off ether1 as a safe place to make all bridge and vlan changes ..................saves much frustration during initial configuration, also recommended for all 3XX switches.

model = CCR2004-1G-12S+2XS

/interface ethernet
set [ find default-name=ether1 ] name=OffBridge
/interface vlan
and interface=bridge name=vlan-br vlan-id=137
/interface list
add name=MGMT
/ip dhcp-server
add address-pool=dhcp_main_network interface=vlan-br name=dhcp_main_network
/interface list member
add comment=defconf interface**=vlan-br** list=LAN
add interface=mgmt-vlan list=MGMT
add interface=OffBridge list=MGMT
/ip address
add address=10.137.36.1/22 comment=defconf interface=vlan-br network=10.137.36.0
add address=192.168.77.1/30 interface=OffBridge network=192.168.77.0 comment="Offbridge and Emerg access"
/ip dhcp-server network
add address=10.137.36.0/22 comment="Main network" dns-server=10.137.36.1 gateway=10.137.36.1 ( removed netmask )
/ip firewall address-list
add address=IOT Device IP1 list=iot-uk
add address=IOT Device IP2 list=iot-uk
add address=IOT Device IP3 list=iot-uk
etc...
/ip firewall filter
REMOVE THIS LINE
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
ADD REPLACE WITH
add action=accept chain=input comment="admin access only" in-interface-list=MGMT
add action=accept chain=input comment="users to services" in-interface-list=LAN dst-port=53 protocol=udp
add action=accept chain=input comment="users to services" in-interface-list=LAN dst-port=53 protocol=tcp
add action=drop chain=input comment="drop all else" {add this rule in this spot but last of all rules entered }
++++++++++++++++++++++++++++++++++++++++++++++++
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes connection-mark=no-mark
REMOVE THIS LINE
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
ADD REPLACE WITH
add action=accept chain=forward comment="internet traffic" in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="UK wireguard traffic" in-interface=vlan-iot src-address-list=iot-uk out-interface=wireguard1
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat
add action=drop chain=forward comment="drop all else"
/ip firewall nat
add action=masquerade chain=srcnat comment="To 3rdParty WG" out-interface=wireguard1
/ip neighbor discovery-settings
set discover-interface-list=MGMT

/tool mac-server
set allowed-interface-list**=none**
/tool mac-server mac-winbox
set allowed-interface-list=MGMT

Disable IPV6 if not being used and remove associated noise of firewall rules and address lists.
There is no point attempting to do ROUTES or MANGLES or ROUTING RULES until all requirements are clearly established as a piecemeal approach will create rework and potential redesigns.

12

Would appear thus far you need.

  1. VLANS 773/783 to PCC load balance wan1,2,3 ( where WAN1 is 2.5x greater than WAN2 and 10x greater than WAN1 in terms of throughput )

  2. VLANS 137/173 (main/red) use WAN2 as primary and WAN1 as Secondary and 3-6 tertiary

  3. VLANS 187/199 (orange/mgmt) use WAN1 as primary and WAN2 as Secondary and 3-6 tertiary

  4. VLANS 271/373 in limbo.

  5. Need to create destination address lists based on destination public IP addresses ( need to define lists and which WAN they are supposed to use)
    public address1 → out WANX
    address2 → out WANX
    etc.
    firewall-address-list=DG1-via-WANX

public address3 → out WANY
address4 → out WANY
etc.
firewall-address-list=DG2-via-WANY

  1. Need same concept for groups of ports…
    (here simply list of which destination ports go through which WAN??

  2. VLAN for Servers… vlan-dmz
    Need to identify, the IPs on this vlan specific to which WAN they should be accessed by.
    IP address1 thru WANA
    IP address1 thru WANA
    etc.
    firewall-address-list=SG-1-WANA

IP address3 thru WANB
IP address4 thru WANB
etc
firewall-address-list=SG-2-WANB

13

Oh wow anav!

Thanks for your input, it’s much more then I expected. If you don’t have a buymeacoffee.com then maybe you should create a profile there.
I’ll answer all your questions and try to apply some changes based on your input first thing tomorrow morning.

With sincere gratitude,
A

14

Vlan35 is from the IP. It’s just a FTTH installation using a simple ONU stick. The upstream ISP provides Internet connectivity on VLAN35 (they have some other vlans for voice and tv)

Network diagram will help as well.

I’ll try to make one but this is still “a developing story”

Be advised, that mangling takes precedence over routing rules if there is any overlap!

I wasn not aware of that, thank you!

DO NOT mix bridge with DHCP once you go vlans, simply create another vlan if you need another subnet!!!

I’m not sure I understand what you mean.

which ports was the main network (now vlan137) being supplied on ???

You do not clearly state which ports are trunk ports to the two switches…guessing sfp11 and sfp12 ??

Correct. sfp11&12 are trunk ports.

What are on the other sfp ports… 7,8,9,10, 28-1 and 28-2 ???
( it would appear 7,8 going to dumb red devices, and 9,10 going to dumb orange devices, all four being access ports ???)
( no clue as to 28-1 and 28-2 which seem to needing both vlans 173 and 187 and thus trunk ports ( aka going to smart devices ??)

You’re read of the config is correct. sfpplus7&8 are access ports for VLAN173 , sfpplus9&10 are access ports for VLAN187. 11&12 are trunk and sfp28-1 and sfp28-2 are not used.

The port designation is temporary for testing purposes.

15

Hence a diagram is needed so one can config with context to as close to final implementation as possible.
I wouldnt touch mangles or routes until I know the requirements.

16

That’s a very fair point. I’ll get back to it after I finalise the requirements.