Forcing specified VLAN's through failover ISP with mangle rules

RouterOS General
1

Hi Mikrotik members,

We have a customer with 2 ISP’s, configured in a simple fail-over using the check-gateway function. We want to force internet traffic out through the secondary ISP for some specific VLAN’s, these are in a interface list. We’re using mangle rules to mark packets, then connections coming in from this interface list (while excluding non-internet IP’s), then mark routing these connections to the table with only one default route to ISP2’s gateway.
For some reason completely unknown to me, this isn’t working, I have done this before successfully using similar methods, but no luck. We’re running ROS7.10.1.

Here’s a snippet of the relevant configuration:

/ip route export
add check-gateway=ping comment="Default WAN-ISP1" disabled=no distance=1 \
    dst-address=0.0.0.0/0 gateway=ISP1.IP pref-src="" routing-table=main \
    scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment=ISP2 disabled=no distance=5 dst-address=0.0.0.0/0 \
    gateway=ISP2.IP pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add comment="Default WAN-ISP1" disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=ISP1.IP pref-src="" routing-table=ISP1 scope=30 \
    suppress-hw-offload=no target-scope=10
add comment=ISP2 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ISP2.IP \
    pref-src="" routing-table=ISP2 scope=30 suppress-hw-offload=no target-scope=10




    /ip firewall mangle export
add action=mark-packet chain=prerouting comment=\
    "Mark packets from CamLANs to internet" dst-address-list=!not_in_internet \
    in-interface-list=CamLANs new-packet-mark=CamLANs-IN passthrough=yes
add action=mark-connection chain=prerouting comment=\
    "Mark connection from CamLANs to internet" new-connection-mark=CamLANs-IN \
    packet-mark=CamLANs-IN passthrough=yes
add action=mark-routing chain=prerouting comment=\
    "Mark routing for ISP2 for packets with CamLANS-in connection mark" \
    connection-mark=CamLANs-IN new-routing-mark=ISP2 passthrough=yes

When packetsniffing the WAN2 port, we’re not seeing any traffic coming through. Even when changing the mark routing to the ISP1 table, nothing. Only when I configure the mark-routing for the main table is traffic forwarded out, to ISP1.

What am I overseeing?

2

How many vlans are we talking.
Much easier to use routing rules for whole subnets and no mangling required.

3

Hi Anav,

We’re talking 2-3 VLAN’s. And obviously only traffic destined for the internet should be routed over the seperate tables, not internal (inter VLAN/VPN traffic).

4

I’m happy to report back that I’ve found the issue, and it’s working correctly now.

Apparently in the last mangle rule I have to specify the IN interface as the local VLAN(s) that I want to mark the routing of outgoing packets to the internet. Incoming packets (presumably returning) should not be marked with the route mark, otherwise they’d be returned to the ISP router instead of the internal host (presumably).

Anyway, here’s the correct configuration:

/ip firewall mangle
add action=mark-packet chain=prerouting comment="Mark packets from CamLANs to internet" \
    dst-address-list=!not_in_internet in-interface-list=CamLANs new-packet-mark=CamLANs-IN \
    passthrough=yes
add action=mark-connection chain=prerouting comment="Mark connection from CamLANs to internet" \
    new-connection-mark=CamLANs-IN packet-mark=CamLANs-IN passthrough=yes
add action=mark-routing chain=prerouting comment=\
    "Mark routing for ISP2 for packets with CamLANS-in connection mark" connection-mark=CamLANs-IN \
    in-interface-list=CamLANs new-routing-mark=ISP2 passthrough=yes
5

Much easier to do this, still need extra table and the the route with that Table for WAN2 selected of course.
No mangling required.

/ip routing rule
add src-address=vlan1 action=lookup table=useWAN2
add src-address=vlan2 action=lookup table=useWAN2
add src-address=vlan2 action=lookup table=useWAN2

Note1: If you want the vlans TO NEVER BE ABLE to access WAN1, no failover, then simply use the action
action=lookup-only-in-table.

Note2: If you need the vlans to access local LANS then you need to add such rules before the ones forcing all traffic out WAN2.

/ip routing rule
add dst-address=SUBNETA action=lookup-only-in-table table=main
add src-address=vlan1 action=lookup table=useWAN2
add src-address=vlan2 action=lookup table=useWAN2
add src-address=vlan2 action=lookup table=useWAN2

6

Limitations with this method is that if you’re port forwarding through one of the ISP’s, which is not the primary outbound for that VLAN packets will be routed incorrectly, so mangle rules are still required in such an environment, correct?

7

Well, that is a different requirement. If you mangle so fully then fastrack rule advantages start getting lost…

To ensure traffic coming in a wan goes out the same wan, is less mangling and more fastrack available for the rest of the traffic.
{forward chain}
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related connection-mark=no-mark

as for mangling rules… basically need
/ip firewall mangle
{ handle traffic originating outside router to ensure return traffic goes out same WAN }
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=viaISP1
passthrough=yes in-interface=ether1-ISP1
add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=viaISP2
passthrough=yes in-interface=ether2-ISP2
add action=mark-routing chain=output connection-mark=viaISP1
new-routing-mark=preferISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=viaISP2
new-routing-mark=preferISP2 passthrough=yes

In your IP routes you will need an IP route for each WAN using the routing-table=preferISP1 and routing-table=preferISP2 respectively.

8

Hi everyone,
MrSerio I encounter the same issue. In my scenario, I have a LAN manage network and two vlans (private and public). I use failover between my two ISPs but I want also when everything works fine the private vlan to use ISP1 and public vlan to use ISP2. When ISP1 or ISP2 fails, the traffic should be redirected to the ISP that is active. Following this thread, everything works perfect except the forwards (dstnat) that are made to vlans.

Is there a right way to do it or a workaround to this?

9

YES, as described above.

10 
# 2023-09-12 18:44:53 by RouterOS 7.11.2
# software id = KSL6-4515
#
# model = RB2011UiAS-2HnD
/interface bridge
add admin-mac=4C:5E:0C:2C:95:42 auto-mac=no comment=defconf name=bridge
add name=bridge.vlan16
add name=bridge.vlan200
/interface vlan
add interface=ether2 name=vlan16_ether2 vlan-id=16
add interface=ether4 name=vlan16_ether4 vlan-id=16
add interface=ether5 name=vlan16_ether5 vlan-id=16
add interface=ether6 name=vlan16_ether6 vlan-id=16
add interface=ether7 name=vlan16_ether7 vlan-id=16
add interface=ether8 name=vlan16_ether8 vlan-id=16
add interface=ether9 name=vlan16_ether9 vlan-id=16
add interface=ether10 name=vlan16_ether10 vlan-id=16
add interface=ether2 name=vlan200_ether2 vlan-id=200
add interface=ether4 name=vlan200_ether4 vlan-id=200
add interface=ether5 name=vlan200_ether5 vlan-id=200
add interface=ether6 name=vlan200_ether6 vlan-id=200
add interface=ether7 name=vlan200_ether7 vlan-id=200
add interface=ether8 name=vlan200_ether8 vlan-id=200
add interface=ether9 name=vlan200_ether9 vlan-id=200
add interface=ether10 name=vlan200_ether10 vlan-id=200
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa-psk,wpa2-psk mode=\
    dynamic-keys supplicant-identity=MikroTik
add authentication-types=wpa-psk,wpa2-psk eap-methods="" mode=dynamic-keys \
    name=SecurityProfile.Private supplicant-identity=""
add authentication-types=wpa-psk,wpa2-psk eap-methods="" mode=dynamic-keys \
    name=SecurityProfile.Public supplicant-identity=""
add authentication-types=wpa-psk,wpa2-psk eap-methods="" mode=dynamic-keys \
    name=SecurityProfile.Pbx supplicant-identity=""
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX \
    disabled=no distance=indoors frequency=auto installation=indoor mode=\
    ap-bridge name=WiFi1.Private security-profile=SecurityProfile.Private \
    ssid="test123123 Private" wireless-protocol=802.11
add keepalive-frames=disabled mac-address=4E:5E:0C:2C:95:4B master-interface=\
    WiFi1.Private multicast-buffering=disabled name=WiFi3.Public \
    security-profile=SecurityProfile.Public ssid="test123123 Public" \
    wds-cost-range=0 wds-default-cost=0 wps-mode=disabled
add keepalive-frames=disabled mac-address=4E:5E:0C:2C:95:4C master-interface=\
    WiFi1.Private multicast-buffering=disabled name=WiFi5.Pbx \
    security-profile=SecurityProfile.Pbx ssid="test123123 PBX" \
    wds-cost-range=0 wds-default-cost=0 wps-mode=disabled
/interface vlan
add interface=WiFi3.Public name=vlan16_WiFi3 vlan-id=16
add interface=WiFi5.Pbx name=vlan200_WiFi5 vlan-id=200
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=pool.vlan16 ranges=172.16.0.2-172.16.0.254
add name=pool.vlan200 ranges=10.200.0.11-10.200.0.220
/ip dhcp-server
add address-pool=default-dhcp interface=bridge lease-time=10m name=defconf
add address-pool=pool.vlan16 interface=bridge.vlan16 name=dhcp.vlan16
add address-pool=pool.vlan200 interface=bridge.vlan200 name=dhcp.vlan200
/port
set 0 name=serial0
/routing table
add fib name=to-wan1
add fib name=to-wan2
/system logging action
set 0 memory-lines=10000
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge.vlan16 comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether9
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=sfp1
add bridge=bridge comment=defconf interface=WiFi1.Private
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether2
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether4
add bridge=bridge.vlan16 comment=customconf disabled=yes interface=\
    vlan16_ether5
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether6
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether7
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether8
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether9
add bridge=bridge.vlan16 comment=customconf interface=vlan16_ether10
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether2
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether4
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether5
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether6
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether7
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether8
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether9
add bridge=bridge.vlan200 comment=customconf interface=vlan200_ether10
add bridge=bridge.vlan16 comment=customconf interface=WiFi3.Public
add bridge=bridge.vlan200 comment=customconf interface=WiFi5.Pbx
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=WAN internet-interface-list=all lan-interface-list=\
    LAN wan-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add comment=wan2 interface=ether3 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
add address=172.16.0.1/24 interface=bridge.vlan16 network=172.16.0.0
add address=10.200.0.1/24 interface=bridge.vlan200 network=10.200.0.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=5m
/ip dhcp-client
add comment=defconf interface=ether1
add default-route-distance=2 interface=ether3
/ip dhcp-server lease
add address=172.16.0.253 client-id=1:30:5a:3a:3:13:b3 mac-address=\
    30:5A:3A:03:13:B3 server=dhcp.vlan16
/ip dhcp-server network
add address=10.200.0.0/24 dns-server=1.1.1.1,8.8.8.8 gateway=10.200.0.1
add address=172.16.0.0/24 dns-server=1.1.1.1,8.8.8.8 gateway=172.16.0.1
add address=192.168.88.0/24 comment=defconf dns-server=1.1.1.1,8.8.8.8 \
    gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.8.8
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment=Custom.MikroTik dst-port=8921 \
    protocol=tcp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-mark=no-mark connection-state=established,related hw-offload=\
    yes
add action=drop chain=forward in-interface=bridge out-interface=bridge.vlan16
add action=drop chain=forward in-interface=bridge out-interface=\
    bridge.vlan200
add action=drop chain=forward in-interface=bridge.vlan16 out-interface=bridge
add action=drop chain=forward in-interface=bridge.vlan16 out-interface=\
    bridge.vlan200
add action=drop chain=forward in-interface=bridge.vlan200 out-interface=\
    bridge
add action=drop chain=forward in-interface=bridge.vlan200 out-interface=\
    bridge.vlan16
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=ether1 new-connection-mark=wan1-conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=ether3 new-connection-mark=wan2-conn passthrough=yes
add action=mark-routing chain=output connection-mark=wan1-conn \
    new-routing-mark=to-wan1 passthrough=yes
add action=mark-routing chain=output connection-mark=wan2-conn \
    new-routing-mark=to-wan2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat src-address=172.16.0.0/24
add action=masquerade chain=srcnat src-address=10.200.0.0/24
add action=dst-nat chain=dstnat dst-port=44044 protocol=tcp to-addresses=\
    172.16.0.253 to-ports=3389
/ip route
add disabled=no distance=1 dst-address=1.1.1.1/32 gateway=192.168.11.1 \
    pref-src="" routing-table=main scope=10 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=8.8.8.8/32 gateway=192.168.10.1 \
    pref-src="" routing-table=main scope=10 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping distance=1 gateway=1.1.1.1 routing-table=to-wan1 \
    target-scope=11
add check-gateway=ping distance=2 gateway=8.8.8.8 routing-table=to-wan1 \
    target-scope=11
add check-gateway=ping distance=1 gateway=8.8.8.8 routing-table=to-wan2 \
    target-scope=11
add check-gateway=ping distance=2 gateway=1.1.1.1 routing-table=to-wan2 \
    target-scope=11
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/lcd interface pages
set 0 interfaces="sfp1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8\
    ,ether9,ether10"
/routing rule
add action=lookup-only-in-table disabled=no dst-address=192.168.88.0/24 \
    table=main
add action=lookup-only-in-table disabled=no dst-address=172.16.0.0/24 table=\
    main
add action=lookup disabled=no src-address=192.168.88.0/24 table=to-wan1
add action=lookup disabled=no src-address=172.16.0.0/24 table=to-wan2
/system clock
set time-zone-name=Europe/Athens
/system identity
set name=test123123
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Hi anav,
I tried to follow your steps but something is wrong with my configuration. In my example script, I’m trying to remote connect to 172.16.0.253 (vlan16) that uses ISP2. Even though packets are arriving on NAT (under dstnted rule - the rdp port) remote connection will not work, unless I change the routing rule of 172.16.0.0/24 to table=to-wan1 (or just disable the rule). Any ideas?

Thank you in advance.

11

Yes your configuration of vlans is out to lunch… I am surprized anything works …

For me to assist I would need to know…
Which VLANs go to which ports?
Which Ports are only for the 192.1.68.88 subnet ?
Which ports are going to smart devices that can read either tagged vlans or a combination of one untagged vlans and multiple vlans?

THis is the model to be followed in general…
http://forum.mikrotik.com/t/using-routeros-to-vlan-your-network/126489/1

12

So, ETH1 is ISP1 (GW 192.168.10.1), ETH3 is ISP2 (GW 192.168.11.1) and all other ports are on same bridge (192.168.88.0/24) expect ETH5 that is on bridge.vlan16 (172.16.0.0/24). I will use few smart switches which will be on 192.168.88.0/24 (untag) and they will transfer all vlans and use them separately.