Multi WAN Recursive route issue with VPNs

cleangreen · August 13, 2023, 4:24pm

Hi all,
Hoping to get to the bottom of a problem with my PCC load balance using recursive routes, I have it set up as a 5x WAN load balancer which really well apart from 1 small issue which keeps coming up with some sites we supply connectivity too.

I am using mangle rules to in the prerouting/output/prerouting chain as per all of the guides I have seen, running rOS7, RB4011. Port 1-5 WANs and rest for LAN

VPNs do not always work, The issue must be return traffic as depending on the type some connect and then do not work, others such as Watchguard VPN app will not establish a connection at all, I confirmed the VPN’s all work by removing the LB out of the network and they connect and work as expected.

My network topo is
WAN 1-5 (not all always in use mainly use 2x) which I adjust the mangle pcc rules for each time.
NAT being handled on this LB for the network below
LAN out to another RB4011 acting as a network core to a wider network.

Below is the config I am using, any pointers what might be causing VPNs not to work would be amazing.

I have some additional mangle rules in there to force subnets to a certain WAN too. LAN2 is also on there but not used.

Thanks in advance for your help

/interface bridge
add name="LAN 1"
add name="LAN 2"
add name=l2tp-bridge

/interface list
add name=WAN

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

/ip pool
add name=dhcp_pool1 ranges=192.168.48.50-192.168.51.254
add name=dhcp_pool2 ranges=192.168.52.50-192.168.55.254

/ip dhcp-server
add address-pool=dhcp_pool1 interface="LAN 1" lease-time=1d name=dhcp1
add address-pool=dhcp_pool2 interface="LAN 2" lease-time=1d name=dhcp2

/port
set 0 name=serial0
set 1 name=serial1

/routing table
add disabled=no fib name=ISP1
add disabled=no fib name=ISP2
add disabled=no fib name=ISP3
add disabled=no fib name=ISP4
add disabled=no fib name=ISP5

/interface bridge port
add bridge="LAN 1" interface=ether10
add bridge="LAN 1" interface=ether9
add bridge="LAN 1" interface=ether8
add bridge="LAN 1" interface=ether6
add bridge="LAN 1" interface=ether7

/ip neighbor discovery-settings
set discover-interface-list=!dynamic

/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=ether3 list=WAN
add interface=ether4 list=WAN
add interface=ether5 list=WAN

/ip address
add address=192.168.48.1/22 interface="LAN 1" network=192.168.48.0
add address=10.250.0.1 interface=l2tp-bridge network=10.250.0.1
add address=192.168.52.1/22 interface="LAN 2" network=192.168.52.0

/ip dhcp-client
add add-default-route=no interface=ether1 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP1\"] gateway=\$\"gateway-addr\
    ess\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP4_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP5_conn\"]\r\
    \n" use-peer-dns=no use-peer-ntp=no
add add-default-route=no interface=ether2 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP2\"] gateway=\$\"gateway-addr\
    ess\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP4_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP5_conn\"]" \
    use-peer-dns=no use-peer-ntp=no
add add-default-route=no interface=ether3 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP3\"] gateway=\$\"gateway-addr\
    ess\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP4_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP5_conn\"]" \
    use-peer-dns=no use-peer-ntp=no
add add-default-route=no interface=ether4 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP4\"] gateway=\$\"gateway-addr\
    ess\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP4_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP5_conn\"]" \
    use-peer-dns=no use-peer-ntp=no
add add-default-route=no interface=ether5 script=":if (\$bound=1) do={\r\
    \n    /ip/route/set [find where comment=\"ISP5\"] gateway=\$\"gateway-addr\
    ess\"\r\
    \n}\r\
    \n\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP1_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP2_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP3_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP4_conn\"]\r\
    \n/ip/firewall/connection/remove [find connection-mark=\"ISP5_conn\"]" \
    use-peer-dns=no use-peer-ntp=no

/ip dhcp-server lease
add address=192.168.50.252 client-id=1:c4:ad:34:55:65:22 mac-address=\
    C4:AD:34:55:65:22 server=dhcp1

/ip dhcp-server network
add address=192.168.48.0/22 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.48.1
add address=192.168.52.0/22 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.52.1

/ip dns
set servers=8.8.8.8,8.8.4.4

/ip firewall address-list
add address=10.0.64.0/20 disabled=yes list=FORCE-WAN1
add address=10.0.48.0/20 list=FORCE-WAN1
add address=10.0.0.0/20 list=FORCE-WAN2
add address=10.0.15.253 disabled=yes list=FORCE-WAN3
add address=10.0.15.253 disabled=yes list=FORCE-WAN4
add address=10.0.15.253 disabled=yes list=FORCE-WAN5
add address=192.168.48.0/22 list=LAN1
add address=192.168.52.0/22 list=LAN2
add address=10.0.16.0/20 list=FORCE-WAN2
add address=10.0.32.0/20 list=FORCE-WAN2

/ip firewall filter
add action=accept chain=input connection-mark=ISP1_conn disabled=yes \
    in-interface="LAN 1" protocol=ipsec-esp
add action=accept chain=input connection-mark=ISP1_conn disabled=yes \
    dst-port=500,1701,4500 in-interface="LAN 1" protocol=udp
add action=drop chain=input comment="Block LAN 2 from Accessing LAN 1" \
    dst-address-list=LAN1 src-address-list=LAN2

/ip firewall mangle
add action=mark-connection chain=prerouting comment="FORCE WAN 1" \
    connection-mark=no-mark connection-state=new in-interface="LAN 1" \
    new-connection-mark=ISP1_conn passthrough=yes src-address-list=FORCE-WAN1
add action=mark-connection chain=prerouting comment="FORCE WAN 2" \
    connection-mark=no-mark connection-state=new in-interface="LAN 1" \
    new-connection-mark=ISP2_conn passthrough=yes src-address-list=FORCE-WAN2
add action=mark-connection chain=prerouting comment="FORCE WAN 3" \
    connection-mark=no-mark connection-state=new in-interface="LAN 1" \
    new-connection-mark=ISP3_conn passthrough=yes src-address-list=FORCE-WAN3
add action=mark-connection chain=prerouting comment="FORCE WAN 4" \
    connection-mark=no-mark connection-state=new in-interface="LAN 1" \
    new-connection-mark=ISP4_conn passthrough=yes src-address-list=FORCE-WAN4
add action=mark-connection chain=prerouting comment="FORCE WAN 5" \
    connection-mark=no-mark connection-state=new in-interface="LAN 1" \
    new-connection-mark=ISP5_conn passthrough=yes src-address-list=FORCE-WAN5
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new in-interface=ether1 new-connection-mark=ISP1_conn \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new in-interface=ether2 new-connection-mark=ISP2_conn \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new in-interface=ether3 new-connection-mark=ISP3_conn \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new in-interface=ether4 new-connection-mark=ISP4_conn \
    passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=new in-interface=ether5 new-connection-mark=ISP5_conn \
    passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1_conn \
    new-routing-mark=ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn \
    new-routing-mark=ISP2 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP3_conn \
    new-routing-mark=ISP3 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP4_conn \
    new-routing-mark=ISP4 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP5_conn \
    new-routing-mark=ISP5 passthrough=yes
add action=mark-connection chain=prerouting comment="PCC on these - WAN1" \
    connection-mark=no-mark dst-address-type=!local in-interface="LAN 1" \
    new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:2/0
add action=mark-connection chain=prerouting comment="PCC on these - WAN1" \
    connection-mark=no-mark disabled=yes dst-address-type=!local \
    in-interface="LAN 1" new-connection-mark=ISP1_conn passthrough=yes \
    per-connection-classifier=src-address-and-port:3/1
add action=mark-connection chain=prerouting comment="PCC on these - WAN2" \
    connection-mark=no-mark dst-address-type=!local in-interface="LAN 1" \
    new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:2/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN 1" new-connection-mark=\
    ISP3_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:5/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN 1" new-connection-mark=\
    ISP4_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:5/4
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN 1" new-connection-mark=\
    ISP5_conn passthrough=yes per-connection-classifier=\
    src-address-and-port:5/5
add action=mark-routing chain=prerouting connection-mark=ISP1_conn \
    in-interface="LAN 1" new-routing-mark=ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2_conn \
    in-interface="LAN 1" new-routing-mark=ISP2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP3_conn \
    in-interface="LAN 1" new-routing-mark=ISP3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP4_conn \
    in-interface="LAN 1" new-routing-mark=ISP4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP5_conn \
    in-interface="LAN 1" new-routing-mark=ISP5 passthrough=yes
add action=mark-routing chain=prerouting comment="test for vpn" \
    connection-mark=ISP1_conn disabled=yes new-routing-mark=ISP1 passthrough=\
    yes

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=masquerade chain=srcnat comment=\
    "NAT Rule when Using SRE CORE - Must Disable NAT MAS Rules on CORE" \
    out-interface-list=WAN src-address=172.18.0.0/22
add action=masquerade chain=srcnat comment=\
    "NAT Rule when Using SRE CORE - Must Disable NAT MAS Rules on CORE" \
    out-interface-list=WAN src-address=172.17.0.0/22
add action=masquerade chain=srcnat comment=\
    "NAT Rule when Using SRE CORE - Must Disable NAT MAS Rules on CORE" \
    out-interface-list=WAN src-address=172.16.0.0/22
add action=masquerade chain=srcnat out-interface=ether2
add action=masquerade chain=srcnat out-interface=ether3
add action=masquerade chain=srcnat out-interface=ether4
add action=masquerade chain=srcnat out-interface=ether5

/ip route
add check-gateway=ping disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=\
    192.168.80.1 pref-src="" routing-table=ISP1 scope=30 suppress-hw-offload=\
    no target-scope=10
add check-gateway=ping disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=\
    192.168.1.1 pref-src="" routing-table=ISP2 scope=30 suppress-hw-offload=\
    no target-scope=10
add check-gateway=ping disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=\
    192.168.2.1 pref-src="" routing-table=ISP3 scope=30 suppress-hw-offload=\
    no target-scope=10
add check-gateway=ping disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=\
    192.168.80.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=\
    no target-scope=10
add check-gateway=ping disabled=yes distance=2 dst-address=0.0.0.0/0 gateway=\
    192.168.1.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=\
    no target-scope=10
add comment=ISP1 distance=1 dst-address=9.9.9.9/32 gateway=192.168.1.1 scope=\
    10 target-scope=10
add comment=ISP2 distance=1 dst-address=8.26.56.26/32 gateway=192.168.88.1 \
    scope=10 target-scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=9.9.9.9 \
    scope=10 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=8.26.56.26 \
    scope=10 target-scope=11
add comment=ISP1 dst-address=64.6.64.6/32 gateway=192.168.1.1 scope=10
add comment=ISP1 dst-address=208.67.220.220/32 gateway=192.168.1.1 scope=10
add comment=ISP2 dst-address=208.67.222.222/32 gateway=192.168.88.1 scope=10
add comment=ISP2 dst-address=64.6.65.6/32 gateway=192.168.88.1 scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=64.6.64.6 \
    routing-table=ISP1 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=64.6.65.6 \
    routing-table=ISP1 scope=10 target-scope=11
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=\
    208.67.222.222 routing-table=ISP2 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=\
    208.67.220.220 routing-table=ISP2 scope=10 target-scope=11
add comment=ISP3 distance=1 dst-address=185.73.44.147/32 gateway=192.168.1.1 \
    scope=10 target-scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=185.73.44.147 \
    scope=10 target-scope=11
add comment=ISP3 dst-address=76.76.2.0/32 gateway=192.168.1.1 scope=10
add comment=ISP3 dst-address=76.76.10.0/32 gateway=192.168.1.1 scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=76.76.2.0 \
    routing-table=ISP3 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=76.76.2.0 \
    routing-table=ISP3 scope=10 target-scope=11
add comment=ISP4 distance=1 dst-address=185.228.168.9/32 gateway=192.168.80.1 \
    scope=10 target-scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=185.228.168.9 \
    scope=10 target-scope=11
add comment=ISP4 dst-address=185.228.169.9/32 gateway=192.168.80.1 scope=10
add comment=ISP4 dst-address=76.76.19.19/32 gateway=192.168.80.1 scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=185.228.169.9 \
    routing-table=ISP4 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=76.76.19.19 \
    routing-table=ISP4 scope=10 target-scope=11
add comment=ISP5 distance=1 dst-address=76.223.122.150/32 gateway=192.168.1.1 \
    scope=10 target-scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=\
    76.223.122.150 scope=10 target-scope=11
add comment=ISP5 dst-address=94.140.14.14/32 gateway=192.168.1.1 scope=10
add comment=ISP5 dst-address=94.140.15.15/32 gateway=192.168.1.1 scope=10
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=94.140.14.14 \
    routing-table=ISP5 scope=10 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=94.140.15.15 \
    routing-table=ISP5 scope=10 target-scope=11
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    172.16.0.0/22 gateway=192.168.50.252 pref-src="" routing-table=main \
    scope=30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    172.17.0.0/16 gateway=192.168.50.252 pref-src="" routing-table=main \
    scope=30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.0.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.16.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    172.18.0.0/16 gateway=192.168.50.252 pref-src="" routing-table=main \
    scope=30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.32.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.48.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.64.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.80.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.96.0/20 gateway=192.168.50.252 pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add comment="Route back to CORE Networks" disabled=no distance=1 dst-address=\
    10.0.112.0/20 gateway=192.168.50.252 pref-src="" routing-table=main \
    scope=30 suppress-hw-offload=no target-scope=10

/routing rule
add action=lookup-only-in-table disabled=yes src-address=10.250.0.1/32 table=\
    ISP5

Guscht · August 13, 2023, 5:35pm

You give no real details, are the VPN seen as to the Router to does the Router connect to some other server?

Its hard to build your network (given by the less details and a config-file) in the imagination. So, my only advice is, exclude all VPN-traffic from LB or force all VPN-traffic to some dedicated ISP.

anav · August 13, 2023, 5:44pm

A network diagram will help.
Describe more fully the VPN, is the router acting as a server aka wireguard or something else.
Are external users coming in via VPN to do what.
Are local users using VPN to go out?

cleangreen · August 14, 2023, 4:06pm

Thanks for replies.

No the RB’s are not acting as a VPN server these are just clients connected to wifi access points who cannot access thier VPNs. I have no control of these users. The issue was with a watchguard VPN for a corp. I think the issue is the packet marking not either tracking correctly or marking correctly when it comes back.

This is purely client users using VPN clients to access their own corp networks/home/or whatever people do on VPN’s these days!

I will put together a network diagram as I don’t have one drawn up right now.

I have tried forcing a client to use WAN1 but they still have the same problem with the VPNs. I am doing that with additional mangle rules

Essentially I have;
Load balancer which has a DHCP LAN connected to another Router, which has around 50 additional nodes behind it with MPLS/VPLS tunnels which bridge back to the ‘core’ router, there are 10 networks which are all handled by the core router. These are all segregated and accessed via VLANs on SSIDs. That part works perfectly and VPN’s work fine when I take the WAN directly in to that side. My issue happens when I add the load balancer between the core router and the WANs.

Thanks

anav · August 14, 2023, 6:24pm

So in summary…

Vpn users use their own client software to get reach their office.
The traffic is mostly transparent to the router.
The traffic causing the issues is local outbound traffic from those users.
You do not host VPN servers for these clients.

You have 5 WANS and apply PCC load balancing.
It would appear a user assigned a WAN does not necessarily work for vpn traffic outbound and return success ??
++++++++++++++++++++++

Dont understand your network in the least missing interface list members for LAN
Where is LAN2, which ports

Why do you have three bridges. Use one bridge and 3 vlans for example.

No firewall rules except a few that dont belong there anyway???

What is going over WANS 1-5 traffic from ???

Assuming the l2tp bridge has nothing to do with WAN traffic and the client traffic of which you speak.

Seems like you have 30,000 to many mangle rules

FIRST THING I Would do is get rid of the gazillion script rules in IP DHCP client. What is their purpose.

WHat kind of connections do you have for these 5 WANS. dynamic assigned WANIP or static fixed WANIP??