Multi WAN policy routing

Schime85 · July 31, 2023, 1:27pm

Hi guys!

Thank you anav for this https://forum.mikrotik.com/viewtopic.php?t=182373
I have problems communicating between interfaces when using policy routing the “new” ros7 way
The goal here is to route vlan5 and vlan40 over WAN2 (ether6 with gateway Y.Y.Y.Y) and retaining communication between interfaces if wanted.

/interface vlan
add interface=bridge1 name=vlan1 vlan-id=1
add interface=bridge1 name=vlan5 vlan-id=5
add interface=bridge1 name=vlan10 vlan-id=10
add interface=bridge1 name=vlan30 vlan-id=30
add interface=bridge1 name=vlan40 vlan-id=40
add interface=bridge1 name=vlan50 vlan-id=50
/interface list
add name=WAN
add name=LAN
add name=GUESTS
add name=BACKUP
/interface list member
add interface=ether1 list=WAN
add interface=ether6 list=WAN
add interface=vlan10 list=GUESTS
add interface=vlan30 list=GUESTS
add interface=vlan50 list=GUESTS
add interface=vlan40 list=BACKUP
add interface=vlan5 list=BACKUP
/routing table
add disabled=no fib name=backup
/ip firewall address-list
add address=192.168.5.0/24 list=BACKUP
add address=192.168.40.0/24 list=BACKUP
add address=172.16.1.0/24 list=connectedSubnets
add address=192.168.5.0/24 list=connectedSubnets
add address=192.168.40.0/24 list=connectedSubnets

Simple routes with failover:

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    X.X.X.X pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=Y.Y.Y.Y \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=\
    Y.Y.Y.Y pref-src="" routing-table=backup scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=20 dst-address=0.0.0.0/0 gateway=X.X.X.X \
    routing-table=backup suppress-hw-offload=no

I wanted to implement policy routing this way but I’m unable to get a connection between the interfaces then …

/routing rule
add action=lookup disabled=no interface=vlan5 table=backup
add action=lookup disabled=no interface=vlan40 table=backup

The following mangle rules are working (ros6 variant):

/ip firewall mangle
add action=accept chain=prerouting dst-address-list=connectedSubnets \
    src-address-list=connectedSubnets
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" in-interface=ether6 new-connection-mark=wan-backup \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type="" new-connection-mark=wan-backup passthrough=yes \
    src-address-list=BACKUP
add action=mark-routing chain=prerouting dst-address-type="" \
    in-interface-list=BACKUP new-routing-mark=backup passthrough=yes
add action=mark-routing chain=output connection-mark=wan-backup \
    new-routing-mark=backup passthrough=yes

The whole magic does the line where I exclude connectedSubnets to connectedSubnets from being mangled.
Is there a better/leaner/nicer way using the ros7 variant without mangle rules?

anav · July 31, 2023, 4:40pm

Well there are two ways to route 2 out of a bunch of vlans out a specific WAN.
The way I would be likely to approach this is the use of routing rules and not mangling.
Also there is no need for distance in created tables. By the way, I typically use 5,10 in my routes so that I can always fit other routes before, between, or after..

The problem you experienced or noted needs more discussion. Its not only the vlans reaching the other interfaces its also the vlans returning queries from other interfaces.
Thus you need to ensure that the traffic is not routed out the WAN.
This is easily accomplished by using additional routing rules in the correct order!

Fiinal point dont use vlan1 for data, its a default vlan in the background not to be messed with.

++++++++++++++++++++++++++++++++++++++++
add check-gateway=ping dst-address=0.0.0.0/0 gateway=X.X.X.X routing-table=main distance=5 comment=WAN1
add dst-address=0.0.0.0/0 gateway=Y.Y.Y.Y routing-table=main distance=10 comment=WAN2
add dst-address=0.0.0.0/0 gateway=Y.Y.Y.Y routing-table=backup comment=WAN2
/routing rule
add action=lookup-only-in-table dst-address=VLANX table=main comment=LAN access VLANX
add action=lookup-only-in-table dst-address=VLANY table=main comment=LAN access VLANY
etc
add action=lookup src-address=10.10.5.0/24 table=backup comment=VLAN5 to WAN2
add action=lookup src-address=10.10.40.0/24 table=backup comment=VLAN40 to WAN2

Schime85 · July 31, 2023, 11:39pm

Thanks anav for looking into it!
The routing rules you posted don’t give me interconnection access.
I tried with this…

/routing rule
add action=lookup-only-in-table dst-address=10.10.50.0/24 table=main comment=LAN access VLANX
add action=lookup-only-in-table dst-address=10.10.40.0/24 table=main comment=LAN access VLANY
add action=lookup src-address=10.10.5.0/24 table=backup comment=VLAN5 to WAN2
add action=lookup src-address=10.10.40.0/24 table=backup comment=VLAN40 to WAN2

ah don’t worry about vlan1 … I use bridge vlan-filtering and vlan1 is my native vlan

anav · August 1, 2023, 1:34am

I do worry because there is no requirement to define, get rid of the definition.
Provide the complete config to understand why its n ot working.

Schime85 · August 1, 2023, 7:02am

Here is the config …

# 2023-08-01 08:38:23 by RouterOS 7.10.2
# software id = 8F5U-AUEI
#
# model = CCR1009-7G-1C-1S+

/interface bridge
add admin-mac=64:D1:54:D0:5A:5B auto-mac=no ingress-filtering=no name=bridge1 \
    vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan1-mgmt vlan-id=1
add interface=bridge1 name=vlan5-rez vlan-id=5
add interface=bridge1 name=vlan10-guest vlan-id=10
add interface=bridge1 name=vlan30-fewo vlan-id=30
add interface=bridge1 name=vlan40-gastro vlan-id=40
add interface=bridge1 name=vlan50-business vlan-id=50
/interface list
add name=WAN
add name=LAN
add name=GUESTS
add name=BACKUP
/ip hotspot profile
add dns-name=hotspot.custom hotspot-address=10.1.0.1 html-directory=\
    hotspot2023 install-hotspot-queue=no login-by=http-chap,mac-cookie name=\
    hsprof_custom
/ip hotspot
add addresses-per-mac=unlimited disabled=no interface=vlan10-guest name=hs \
    profile=hsprof_custom
/ip hotspot user profile
set [ find default=yes ] keepalive-timeout=1d shared-users=2000
/ip pool
add comment=gastro name=dhcp_pool1 ranges=192.168.11.100-192.168.11.199
add comment=fewo name=dhcp_pool2 ranges=192.168.10.20-192.168.10.249
add comment=mgmt name=dhcp_pool3 ranges=172.16.1.10-172.16.1.199
add comment=hotspot name=dhcp_pool4 ranges=10.1.0.20-10.1.7.254
add comment=business name=dhcp_pool5 ranges=192.168.50.1-192.168.50.253
/ip dhcp-server
add address-pool=dhcp_pool1 interface=vlan40gastro lease-time=3d name=gastro
add address-pool=dhcp_pool2 interface=vlan30-fewo lease-time=1d name=fewo
add address-pool=dhcp_pool3 interface=vlan1-mgmt lease-time=1d name=mgmt
add address-pool=dhcp_pool4 interface=vlan10-guest lease-time=3h name=\
    hotspot.custom
add address-pool=dhcp_pool5 interface=vlan50-business lease-time=3d name=\
    business
/routing table
add disabled=no fib name=backup
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    ingress-filtering=no interface=ether2 pvid=5
add bridge=bridge1 ingress-filtering=no interface=ether7
add bridge=bridge1 ingress-filtering=no interface=ether5
add bridge=bridge1 ingress-filtering=no interface=ether4
add bridge=bridge1 ingress-filtering=no interface=combo1
add bridge=bridge1 ingress-filtering=no interface=ether3
add bridge=bridge1 ingress-filtering=no interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 vlan-ids=1
add bridge=bridge1 tagged=\
    bridge1,combo1,ether4,ether5,ether7,sfp-sfpplus1,ether3 vlan-ids=5
add bridge=bridge1 tagged=\
    ether4,ether5,ether7,sfp-sfpplus1,bridge1,combo1,ether3 vlan-ids=10
add bridge=bridge1 tagged=\
    bridge1,combo1,ether4,ether5,ether7,sfp-sfpplus1,ether3 vlan-ids=30
add bridge=bridge1 tagged=\
    bridge1,combo1,ether4,ether5,ether7,sfp-sfpplus1,ether3 vlan-ids=40
add bridge=bridge1 tagged=\
    ether4,ether5,ether7,sfp-sfpplus1,bridge1,combo1,ether3 vlan-ids=50
/interface list member
add interface=ether1 list=WAN
add interface=vlan10-guest list=GUESTS
add interface=vlan30-fewo list=GUESTS
add interface=vlan50-business list=GUESTS
add interface=ether6 list=WAN
add interface=vlan40gastro list=BACKUP
add interface=vlan5-rez list=BACKUP
/ip address
add address=X.X.X.186/30 interface=ether1 network=X.X.X.184
add address=192.168.1.1/24 interface=vlan5-rez network=192.168.1.0
add address=192.168.11.254/24 interface=vlan40gastro network=192.168.11.0
add address=172.16.1.254/24 interface=vlan1-mgmt network=172.16.1.0
add address=192.168.10.1/24 interface=vlan30-fewo network=192.168.10.0
add address=10.1.0.1/21 interface=vlan10-guest network=10.1.0.0
add address=192.168.50.254/24 interface=vlan50-business network=192.168.50.0
add address=Y.Y.Y.30/30 interface=ether6 network=Y.Y.Y.28
/ip dhcp-server network
add address=10.1.0.0/21 comment=hotspot.custom dns-server=10.1.0.1 gateway=\
    10.1.0.1
add address=172.16.1.0/24 dns-server=172.16.1.254 gateway=172.16.1.254
add address=192.168.10.0/24 dns-server=192.168.10.1 gateway=192.168.10.1
add address=192.168.11.0/24 comment=gastro dns-server=192.168.11.254 gateway=\
    192.168.11.254
add address=192.168.50.0/24 dns-server=192.168.50.254 gateway=192.168.50.254
/ip dns
set allow-remote-requests=yes cache-max-ttl=1h cache-size=4096KiB \
    max-concurrent-queries=1000 max-concurrent-tcp-sessions=40 servers=\
    1.1.1.1,8.8.8.8
/ip firewall address-list
add address=192.168.1.0/24 list=BACKUP
add address=192.168.11.0/24 list=BACKUP
add address=172.16.1.0/24 list=connectedSubnets
add address=192.168.1.0/24 list=connectedSubnets
add address=192.168.11.0/24 list=connectedSubnets
/ip firewall filter
add action=passthrough chain=unused-hs-chain comment=\
    "place hotspot rules here" disabled=yes
add action=drop chain=input connection-state=invalid log-prefix=INV
add action=accept chain=input log-prefix=PING protocol=icmp
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input in-interface-list=WAN
add action=drop chain=forward comment="drop guests except to wan" \
    in-interface-list=GUESTS out-interface-list=!WAN
add action=drop chain=forward connection-state=invalid
add action=fasttrack-connection chain=forward connection-state=\
    established,related,untracked disabled=yes hw-offload=yes
add action=accept chain=forward connection-state=\
    established,related,untracked
add action=drop chain=forward connection-nat-state=!dstnat connection-state=\
    new in-interface=ether1
/ip firewall mangle
add action=accept chain=prerouting dst-address-list=connectedSubnets \
    src-address-list=connectedSubnets
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state="" in-interface=ether6 new-connection-mark=wan-backup \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type="" new-connection-mark=wan-backup passthrough=yes \
    src-address-list=BACKUP
add action=mark-routing chain=prerouting dst-address-type="" \
    in-interface-list=BACKUP new-routing-mark=backup passthrough=yes
add action=mark-routing chain=output connection-mark=wan-backup \
    new-routing-mark=backup passthrough=yes
/ip firewall nat
add action=passthrough chain=unused-hs-chain comment=\
    "place hotspot rules here" disabled=yes
add action=dst-nat chain=dstnat comment="guest DNS redirect" dst-port=53 \
    protocol=udp src-address=10.1.0.0/21 to-addresses=10.1.0.1 to-ports=53
add action=masquerade chain=srcnat out-interface-list=WAN
/ip hotspot user
add name=guest
/ip route
add check-gateway=ping disabled=no distance=5 dst-address=0.0.0.0/0 gateway=\
    X.X.X.185 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=Y.Y.Y.29 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=\
    Y.Y.Y.29 pref-src="" routing-table=backup scope=30 \
    suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=192.168.1.0/24,172.16.1.0/24
set api disabled=yes
set api-ssl disabled=yes
/routing rule
add action=lookup-only-in-table disabled=yes dst-address=192.168.1.0/24 \
    table=main
add action=lookup-only-in-table disabled=yes dst-address=192.168.11.0/24 \
    table=main
add action=lookup disabled=yes src-address=192.168.1.0/24 table=backup
add action=lookup disabled=yes src-address=192.168.11.0/24 table=backup

With the mangle rules everything is working fine, just wondering why it’s not possible with routing rules

anav · August 1, 2023, 12:03pm

It would appear that etherports: ether3,ether4,ether5,ether7,sfp=sfpplus1 are all trunk ports going to smart devices.
Ingress filtering should be YES, and frame types should be frame-types=admit-only-vlan-tagged.

It would appear that etherport 2 is going to a dumb device and is part of vlan5-rez.

Just remove this crap from /interface bridge vlans…
add bridge=bridge1 tagged=bridge1 vlan-ids=1

should look like.
_/interface bridge vlan
add bridge=bridge1 tagged=bridge1,combo1,ether4,ether5,ether7,sfp-sfpplus1,ether3
untagged=ether2 vlan-ids=5
add bridge=bridge1 tagged=
ether4,ether5,ether7,sfp-sfpplus1,bridge1,combo1,ether3 vlan-ids=11,10,30,40,50
add bridge=bridge1 tagged=_

Vlan 11 is the management VLAN and it is what all smart devices on the network get their IP address from.
This tells me that you are missing the POOL for vlan5-rez, you are missing the dhcp-server for vlan5-rez, you are missing dhcp-server network for vlan5-rez!

What device is on 10.1.0.1?

Remove check-gateway ping from EXTRA ROUTE (third route) , not required.

Fixed this
/interface list member
add interface=ether1 list=WAN
add interface=ether6 list=WAN
add interface=vlan5-rez list=LAN
add interface=vlan11-mgmt list=LAN
add interface=vlan10-guest list=LAN
add interface=vlan30-fewo list=LAN
add interface=vlan40gastro list=LAN
add interface=vlan50-business list=LAN
add interface=vlan10-guest list=GUESTS
add interface=vlan30-fewo list=GUESTS
add interface=vlan50-business list=GUESTS
add interface=vlan5-rez list=BACKUP
add interface=vlan40gastro list=BACKUP
add interface=vlan11-mgmt list=MANAGE

/ip neighbor discovery-settings
set discover-interface-list**=MANAGE**

Why did you give vlan5-rez access to router via SSH??

ip firewall filter
add action=passthrough chain=unused-hs-chain comment=
“place hotspot rules here” disabled=yes
add action=drop chain=input connection-state=invalid log-prefix=INV
add action=accept chain=input log-prefix=PING protocol=icmp
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input in-interface-list=WAN
add action=drop chain=forward comment=“drop guests except to wan”
in-interface-list=GUESTS out-interface-list=!WAN
add action=drop chain=forward connection-state=invalid
add action=fasttrack-connection chain=forward connection-state=
established,related,untracked disabled=yes hw-offload=yes
add action=accept chain=forward connection-state=
established,related,untracked
add action=drop chain=forward connection-nat-state=!dstnat connection-state=
new in-interface=ether1

/ip firewall filter
{Input Chain}
(default rules)
add action=accept chain=input comment=“defconf: accept established,related,untracked” connection-state=established,related,untracked
add action=drop chain=input comment=“defconf: drop invalid” connection-state=invalid
add action=accept chain=input comment=“defconf: accept ICMP” protocol=icmp
add action=accept chain=input comment=“defconf: accept to local loopback (for CAPsMAN)” dst-address=127.0.0.1
(admin rules)
add action=accept chain=input src-address-list=Authorized comment=“admin access to router”
add action=accept chain=input comment=“Allow LAN DNS queries-UDP” dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=input comment=“Allow LAN DNS queries-TCP” dst-port=53 in-interface-list=LAN protocol=tcp
add action=drop chain=input comment=“drop all else”
{forward chain}
(default rules)
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related
add action=accept chain=forward comment=“defconf: accept established,related, untracked” connection-state=established,related,untracked
add action=drop chain=forward comment=“defconf: drop invalid” connection-state=invalid
(admin rules)
add action=accept chain=forward comment=“allow internet traffic” in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment= port forwarding)" connection-nat-state=dstnat { disable if not required }
add action=drop chain=forward comment=“drop all else”

All traffic between vlans are blocked. If you need any intervlan traffic ensure you add it ABOVE the drop all rule in the forward chain!
If you need access to a common device like a printer, same same…
If you want the management vlan to access all the other vlans same same etc etc…

Lastly try this for your Routing Rules ( and disable all mangling )

/routing rule
add action=lookup-only-in-table dst-address=192.168.0.0/18 table=main
add action=lookup-only-in-table dst-address=172.16.1.0/24 table=main
add action=lookup src-address=192.168.1.0/24 table=backup
add action=lookup src-address=192.168.11.0/24 table=backup

Note:1 192.168.0.0/18 covers local interactions with all 192.168 vlans…
Note:2 172.16.1.0/24 covers local interactions with management vlan…

Schime85 · August 1, 2023, 6:54pm

Thank you sir, but I have to point out that there is some config in place with a good reason behind it.
Yes every port except wan or access ports like ether2 are trunk ports AND they have a native vlan1 which is my management network.
Yes vlan5 doesn’t have a dhcp config because there is a server on the network which does the job.
10.1.0.1 is the CCR’s IP address of the hotspot network
Yes vlan5 has ssh access to the router and I want it that way, not a big deal.
Firewall rules are fine and I don’t see a reason to change it because I know what they do and thats ok…

So now we are getting to the topic where I have no clue how it’s done

Oh yes that’s a bingo! Now I can ping from 192.168.1.x to 192.168.11.y but I have no clue why it’s working with routing-rule dst-addr 192.168.0.0/16 and not with the /24 net? I don’t care, happy it’s working that way

Thanks anav again!

anav · August 1, 2023, 8:40pm

Regardless, you dont need that vlan1 identified Line, your config will work just fine without it.
I can say that no one else has done that… so if you are trying to be unique you have succeeded LOL.