Mangle route for WAN2 causing loop

Hoping one of you wizards can tell me where I f*cked up.

  • RB5009, ROS 7.12.1
  • WAN1, DHCP Cable
  • WAN2, PPPoE VDSL

I have simple failover setup but I also want a “guest” network to be pushed out of WAN2.

I created a routing table “wan2” with the route for wan2. I use mangle to connmark the packets I want to go out of wan2, and routemark those marked packets to “wan2”.

Problem is I then find the return packets get sent out of wan2 as well, even though they are destined for the internal address.

Relevant configurations:

PPPoE interface

/interface pppoe-client
add add-default-route=yes default-route-distance=2 disabled=no interface=eth2_wan2 name=eth2_wan2__pppoe_zen user=user@domain

Routes

/ip route
add comment="TAG: eth1_wan1 ROUTE GOOGLE" distance=1 dst-address=0.0.0.0/0 gateway=8.8.4.4 scope=10 target-scope=12
add comment="TAG: eth1_wan1 PING GOOGLE" distance=1 dst-address=8.8.4.4 gateway=82.10.x.x scope=10 target-scope=11
add comment="TAG: eth1_wan1 ROUTE CLOUDFLARE" distance=1 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12
add comment="TAG: eth1_wan1 PING CLOUDFLARE" distance=1 dst-address=1.0.0.1 gateway=82.10.x.x scope=10 target-scope=11
add comment=ROUTE_WAN2 disabled=no distance=2 dst-address=0.0.0.0/0 gateway=eth2_wan2__pppoe_zen pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add comment=ROUTE_WAN2_MARKED disabled=no distance=2 dst-address=0.0.0.0/0 gateway=eth2_wan2__pppoe_zen pref-src="" routing-table=wan2 scope=30 suppress-hw-offload=no target-scope=10

VLANs

/interface vlan
add interface=br1 name=vlan10_core vlan-id=10
add interface=br1 name=vlan25_guest vlan-id=25

Address lists

/ip firewall address-list
add address=172.16.0.0/12 list=NET_PRIVATE
add address=10.0.0.0/8 list=NET_PRIVATE
add address=192.168.0.0/16 list=NET_PRIVATE

Firewall NAT

/ip firewall nat
add action=masquerade chain=srcnat comment="srcnat || masquerade > wan" ipsec-policy=out,none out-interface-list=WAN

Firewall mangle

/ip firewall mangle
add action=mark-connection chain=prerouting comment="connmark || mark incoming connections from wan2" connection-state=new in-interface=eth2_wan2__pppoe_zen new-connection-mark=connmark_wan2 passthrough=no
add action=mark-connection chain=prerouting comment="connmark || mark vlan20_guest connections for wan2" connection-mark=no-mark dst-address-list=!NET_PRIVATE in-interface=vlan25_guest new-connection-mark=connmark_wan2 passthrough=yes
add action=mark-routing chain=prerouting comment="routemark || route connmark'd connections to wan2" connection-mark=connmark_wan2 dst-address-list=!NET_PRIVATE log=yes new-routing-mark=wan2 passthrough=no

Essentially, as soon as I enabled the mangle routemark rule, the guest network loses internet. Using packet sniffer I see the packets destined for the internal network sent back out of the WAN link.

Packet sniffer

Let me know if any further info would be useful. And thanks :slight_smile:

  1. You have two sets of recursive going on, aka check google and if google is possibly not available, then check cloudflare.
    You should differentiate the two by distance like so…
    add comment=“TAG: eth1_wan1 ROUTE GOOGLE” distance=1 dst-address=0.0.0.0/0 gateway=8.8.4.4 scope=10 target-scope=12
    add comment=“TAG: eth1_wan1 PING GOOGLE” distance=1 dst-address=8.8.4.4 gateway=82.10.x.x scope=10 target-scope=11
    add comment=“TAG: eth1_wan1 ROUTE CLOUDFLARE” distance=2 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12
    add comment=“TAG: eth1_wan1 PING CLOUDFLARE” distance=2 dst-address=1.0.0.1 gateway=82.10.x.x scope=10 target-scope=11

Also note that since the gateways are dynamic, the gateway IPs may not get automatically corrected to new ones.
In this case you will need scripts to replace gateway IPs in routing rules as required.

  1. No need to put distance anything on the table for WAN2, its only one entry in the table!

  2. Your mangle rules seem off!! Lets deal with one part at a time.
    /ip firewall mangle
    add action=mark-connection chain=prerouting connection-state=new in-interface=eth2_wan2__pppoe_zen new-connection-mark=connmark_wan2 passthrough=no
    add action=mark-connection chain=prerouting comment=“connmark || mark vlan20_guest connections for wan2” connection-mark=no-mark dst-address-list=!NET_PRIVATE in-interface=vlan25_guest new-connection-mark=connmark_wan2 passthrough=yes
    add action=mark-routing chain=prerouting comment= dst-address-list=!NET_PRIVATE log=yes new-routing-mark=wan2 passthrough=no

A. Ensuring traffic entering WAN2 leaves from WAN2, and by this I mean traffic intended for the router itself and arriving on WAN2.
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=eth2_wan2__pppoe_zen new-connection-mark=connmark_wan2 passthrough=yes
add action=mark-routing chain=output connection-mark=connmark_wan2 new-routing-mark=wan2 passthrough=no

Now to be clear if you DO NOT have traffic such as VPN hitting the router for WAN2, you do not need the above two rules at all.

B. In fact to ensure vlan20 goes out vlan20 you dont need any mangling rules, just routing rule and table and perhaps another routing rule and the existing ip route.

add fib name=wan2
/routing rule add **dst-**address=192.168.25.0/24 action=lookup-only-in-table table=main comment=“permit local traffic”
/routing rule add src-address=192.168.10.0/24 action=lookup table=wan2 comment=“force traffic to WAN2”

======================
The last two mangle rules were suspect anyway.

Thanks for the response and config check.

Re: routing distance
Logic was that seeing as it’s the same gateway, it doesn’t matter the order or priority. Current setup works well.

Re: dynamic gateway
There are scripts that update it, hence the unique comment names used to identify the rules

Re: Mangle vs routing rules
There are other (currently disabled) rules that won’t work with routing rules (protocol, port identifiers)

Re: income wan2 mangle
Without it, the router won’t reply to anything on wan2. Like pings used for uptime monitoring, or winbox requests from trusted sources.

Any mangle users?

Hi,
The rule setting the routing table likely needs to use incoming interfaces. Not destination IP addresses.
The Natted destination address is not yet known, it will still be the external IP address of the router.
(You could add this to your address list I guess)

That was it! Thank you @rplant

Of course the mangle pre-routing table is applied prior to NAT, so the destination address is still the WAN IP. All is clear once you have it explained to you haha :slight_smile:

I have an “INTERNAL” interface list that includes all the internal non-wan interfaces, and replaced the dst-address with in-interface-list

New mangle rule

add action=mark-routing chain=prerouting comment=“routemark || route connmark’d connections to wan2” connection-mark=connmark_wan2 > in-interface-list=INTERNAL > log=yes new-routing-mark=wan2 passthrough=no

Working great, thanks again!

Hello!

I’m with allmost same problem, seems mark-conn and mark-route are going into a loop.

Mangle rules:
chain=prerouting action=mark-connection new-connection-mark=con-wan2 passthrough=yes protocol=icmp connection-mark=no-mark in-interface=pppoe-out-P4 log=yes log-prefix=“MARKCONN”
chain=prerouting action=mark-routing new-routing-mark=rt_p4 passthrough=yes connection-mark=con-wan2 log=yes log-prefix=“MARKROUTE”
chain=prerouting action=accept protocol=icmp src-address=15.235.84.82 log=yes log-prefix=“PING”

Then, when i ping this connection from 15.235.84.82, on the log i saw markconnection, markroute, ping, and then starts a loop, main question why source ip and destination are the same..
.
.:
Captura de tela 2024-09-20 093853.png
Have some idea what’s wrong?