Routing rule VS mangle mark routing

vovan700i · Tue Dec 05, 2023 3:39 pm

Hi,

According to the docs, one can use any or both of the following methods for policy routing:

routing rule, e.g.

/routing rule 
add action=lookup dst-address=9.9.9.9/32 src-address=192.168.2.0/24 table=vrf-wan2

mangle mark routing, e.g.

/ip firewall mangle
add action=mark-routing chain=prerouting dst-address=9.9.9.9/32 new-routing-mark=vrf-wan2 src-address=192.168.2.0/24

In my experience it is routing rules that work on v7.12.1 while mangle rules won't. Routing rules don't accept address lists and generally provide very few parameters for traffic filtering. Thus, I would like to use mangle rules instead, but can't make them work the way routing rules do.

Does anybody know what the actual difference is between these 2 methods? Do I miss any more mangle rules that routing rules have under the hood?

aoakeley · Tue Dec 05, 2023 3:48 pm

.
Do I miss any more mangle rules that routing rules have under the hood?

Maybe. but we can't see what you have missed if you do not post more of your configuration.
Post the rest of your config when you are trying to use mangle rules so we can see where there might be an error.

anav · Tue Dec 05, 2023 4:35 pm

Also the requirement should be expressed in terms of user traffic required.
Mangling and routing rules are simply tools to use, for a purpose, and that purpose has not been communicated........

vovan700i · Tue Dec 05, 2023 5:46 pm

Maybe. but we can't see what you have missed if you do not post more of your configuration.
Post the rest of your config when you are trying to use mangle rules so we can see where there might be an error.

Fair enough. Below is a list of other relevant commands for simplicity. All drop/reject filter rules disabled, no other nat/mangle/raw rules enabled.

/ip vrf
add interfaces=wan2 name=vrf-wan2

/interface list member
add interface=wan2 list=WAN

/ip address
add address=10.16.51.242/24 interface=wan2 network=10.16.51.0

/ip route
add disabled=no distance=100 dst-address=0.0.0.0/0 gateway=10.16.51.1@vrf-wan2 routing-table=vrf-wan2 suppress-hw-offload=no
add disabled=no distance=100 dst-address=192.168.2.0/24 gateway=lan routing-table=vrf-wan2 suppress-hw-offload=no

/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN

Also the requirement should be expressed in terms of user traffic required.
Mangling and routing rules are simply tools to use, for a purpose, and that purpose has not been communicated........

What I would like to achieve is a domain-based VPN: some domains are resolved, their addresses put into lists, these lists used by mangle rules to route traffic through VRF.

UPD: fixed the config excerpt above to be consistent with a full config posted below.

vovan700i · Tue Dec 05, 2023 9:38 pm

Created an empty CHR and tested the same config. Sadly, the problem persists: route rule works, mangle mark routing doesn't work.

/interface bridge
add ingress-filtering=no name=bridge vlan-filtering=yes

/interface bridge port
add bridge=bridge interface=ether1 pvid=17

/interface bridge vlan
add bridge=bridge tagged=bridge untagged=ether1 vlan-ids=17

/interface vlan
add interface=bridge name=wan1 vlan-id=17

/interface vxlan
add mtu=1370 name=lan port=8472 vni=1000 vrf=main vteps-ip-version=ipv4
add mtu=1370 name=wan2 port=8472 vni=4147 vrf=main vteps-ip-version=ipv4

/interface vxlan vteps
add interface=lan remote-ip=10.19.16.1
add interface=wan2 remote-ip=10.80.16.1

/interface list
add name=WAN

/interface list member
add interface=wan1 list=WAN
add interface=wan2 list=WAN

/ip vrf
add interfaces=wan2 name=vrf-wan2

/ip address
add address=10.16.51.242/24 interface=wan2 network=10.16.51.0
add address=192.168.2.1/24 interface=lan network=192.168.2.0

/ip dhcp-client
add default-route-distance=100 interface=wan1 use-peer-dns=no use-peer-ntp=no

/ip dns
set servers=1.1.1.1,1.0.0.1

/ip firewall filter
add action=accept chain=input
add action=accept chain=forward

/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN

/ip route
add disabled=no distance=100 dst-address=0.0.0.0/0 gateway=10.16.51.1@vrf-wan2 routing-table=vrf-wan2 suppress-hw-offload=no
add disabled=no distance=100 dst-address=192.168.2.0/24 gateway=lan routing-table=vrf-wan2 suppress-hw-offload=no

# THIS IS WHAT DOESN'T WORK: if enabled, icmp traceroute from 192.168.2.2 to 9.9.9.9 never goes beyond 192.168.2.1
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address=9.9.9.9/32 new-routing-mark=vrf-wan2 passthrough=yes src-address=192.168.2.0/24

# THIS IS WHAT ACTUALLY WORKS: if enabled, icmp traceroute from 192.168.2.2 to 9.9.9.9 goes through 192.168.2.1 and 10.16.51.1 (wan2 gw)
/routing rule
add action=lookup disabled=yes dst-address=9.9.9.9/32 interface=lan src-address=192.168.2.0/24 table=vrf-wan2

# IT IS ALSO FINE WITH BOTH RULES DISABLED: icmp traceroute from 192.168.2.2 to 9.9.9.9 goes through 192.168.2.1 and wan1 gw

sas2k · Tue Dec 05, 2023 10:30 pm

What I would like to achieve is a domain-based VPN: some domains are resolved, their addresses put into lists, these lists used by mangle rules to route traffic through VRF.

I use domain based vpn with mangle (mark routing).
Additionally you have to add ip-route rule to route (what you marked with mangle) to vpn gateway.
Example:

/ip firewall mangle:
add action=mark-routing chain=prerouting dst-address=!192.168.2.0/24 \
dst-address-list=!LIST new-routing-mark=ipsec passthrough=yes src-address=\
192.168.2.0/24

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=l2tp-out1 pref-src=\
"" routing-table=ipsec scope=30 suppress-hw-offload=yes target-scope=10

LIST - local country ip list.

If you want I can post here step by step config.
My setup uses local country internet directly. And everything that is not local country with vpn .
My vpn is remote vps with libreswan ipsec
Mikrotik is an ipsec client .

I tried to use remote vps\vpn as chr mikrotik but I failed to get speeds more than 40-50 mbps.
Simple ubuntu with libreswan 1 cpu 512 ram can be set up with scripts from github in 5 min, produce up to 185 mbps thru ipsec.

Same easily can be done thru wireguard.
You should be awared that chr withiut licence is limited to 1mbps.

rplant · Wed Dec 06, 2023 7:33 am

Hi,
I built an approximation to this, but with no vxlans (just used another bridge, and ether2 as wan2)

It seemed to mostly work, but the vrf-wan2 being a vrf made it quite painful.
I seemed to need to reboot whenever I had done more than a couple of changes.

I was also unable to traceroute from the router using src-address=192.168.2.1
My laptop on 192.168.2.220 was fine, (though this is somewhat expected).

I turned vrf-wan2 into just routing table (in fib) entry and adjusted for that and it seemed to work quite a bit better and easier.
(You do lose some isolation, but you don't seem to really need any in this instance)

Some (my) notes on routing, seem to be how it is currently working. (not really checked for vrf's)

First:
If the packet has a routing mark on it, and there is a matching entry in the routing table.
(IP range, routing table matches routing mark) it WILL use that routing entry.
This means if there is an item like
add disabled=no distance=100 dst-address=192.168.2.0/24 gateway=lan pref-src="" routing-table=\
vrf-wan2 scope=30 suppress-hw-offload=no target-scope=10
It seems it will (attempt to) send the marked packet out the lan interface, even if it is going to 192.168.2.1

Next:
It will then use the Routing Rules table, (in order) and grab the first match, and do what it says.

Next2:
It will then process the packet through the routing table again, potentially with a different routing mark obtained from the rules table.
(But it seems perhaps a bit more lenient this time)

You can (and it works well) use the routing rules table with routing marks.

mangle:
mark selected packets with routing mark eg. rule-wan2

routing rule
action=lookup table=main dst-address=192.168.2.0/24
action=lookup table=vrf-wan2 routing-mark=rule-wan2

vovan700i · Wed Dec 06, 2023 10:06 am

I use domain based vpn with mangle (mark routing).
Additionally you have to add ip-route rule to route (what you marked with mangle) to vpn gateway.
Example:
Code: Select all
/ip firewall mangle:
add action=mark-routing chain=prerouting dst-address=!192.168.2.0/24 \
dst-address-list=!LIST new-routing-mark=ipsec passthrough=yes src-address=\
192.168.2.0/24

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=l2tp-out1 pref-src=\
"" routing-table=ipsec scope=30 suppress-hw-offload=yes target-scope=10
LIST - local country ip list.

Thank you for a detailed reply. This is roughly what I have, I just simplified the config to a single target address, but I have the same problem if I use lists in mangle rules instead. And I certainly have a default gateway in vrf-wan2. Is your "ipsec" routing table a VRF or a simple routing table created manually?

If you want I can post here step by step config.

Do you have anything else in your config that is relevant? E.g. firewall rules, ip routes, routing rules, etc.

I tried to use remote vps\vpn as chr mikrotik but I failed to get speeds more than 40-50 mbps.
Simple ubuntu with libreswan 1 cpu 512 ram can be set up with scripts from github in 5 min, produce up to 185 mbps thru ipsec.

These speed issues are probably due to ipsec. I use wireguard and it easily handles 200-300 mbps between 2 CHRs or CHR and RB5009, at least as shown by the built-in bandwidth test tool. I think, it can even do faster, but it is VPS bandwidth that seems to be a limitation for me in terms of speed.

vovan700i · Wed Dec 06, 2023 10:38 am

I built an approximation to this, but with no vxlans (just used another bridge, and ether2 as wan2)

First of all, thank you for your effort.

It seemed to mostly work, but the vrf-wan2 being a vrf made it quite painful.
I seemed to need to reboot whenever I had done more than a couple of changes.

I was also unable to traceroute from the router using src-address=192.168.2.1
My laptop on 192.168.2.220 was fine, (though this is somewhat expected).

I turned vrf-wan2 into just routing table (in fib) entry and adjusted for that and it seemed to work quite a bit better and easier.
(You do lose some isolation, but you don't seem to really need any in this instance)

It's a good idea to replace VRF with a simple routing table in FIB. Unfortunately, it barely changed anything for me. Tested on an empty CHR, changes to the full config posted above are the following:

/ip vrf
add interfaces=none name=vrf-wan2

/routing table
add disabled=no fib name=rtab-wan2

/ip route
add distance=100 dst-address=0.0.0.0/0 gateway=10.16.51.1 routing-table=rtab-wan2
add distance=100 dst-address=192.168.2.0/24 gateway=lan routing-table=rtab-wan2

# IT DOESN'T WORK: icmp traceroute from 192.168.2.2 to 9.9.9.9 never goes beyond 192.168.2.1
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=yes dst-address=9.9.9.9/32 new-routing-mark=rtab-wan2 src-address=192.168.2.0/24

# IT WORKS: icmp traceroute from 192.168.2.2 to 9.9.9.9 goes via 192.168.2.1 and 10.16.51.1 (wan2 gw)
/routing rule
add action=lookup disabled=yes dst-address=9.9.9.9/32 interface=lan src-address=192.168.2.0/24 table=rtab-wan2

# IT ALSO WORKS WITH BOTH RULES DISABLED: icmp traceroute from 192.168.2.2 to 9.9.9.9 goes via 192.168.2.1 and wan1 gw

You can (and it works well) use the routing rules table with routing marks.

mangle:
mark selected packets with routing mark eg. rule-wan2

routing rule
action=lookup table=main dst-address=192.168.2.0/24
action=lookup table=vrf-wan2 routing-mark=rule-wan2

That's another good idea I tested myself. In fact, it equals to enabling both mangle and routing rules (because mangle mark routing only allows setting routing tables as marks). It doesn't work for me either. But I'm not sure whether it should, since it is discouraged to combine the two methods of policy routing according to the docs.

vovan700i · Wed Dec 06, 2023 10:56 am

but with no vxlans

That is a key, thanks! I replaced vxlan with eoip and mangle mark routing now works.

So, it seems to be a vxlan bug, will report to MikroTik support soon.

UPD: SUP-136716 + reply on 27 December 2023:

We have managed to reproduce the issue locally in our labs and look forward to fixing it on upcoming RouterOS versions, unfortunately, I cannot provide a release date now.

otgooneo · Fri Jul 19, 2024 11:56 am

you guys guess worked well on this case and it's useful for others like me. Thank you.

Routing rule VS mangle mark routing

Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Re: Routing rule VS mangle mark routing

Who is online