In my experience it is routing rules that work on v7.12.1 while mangle rules won’t. Routing rules don’t accept address lists and generally provide very few parameters for traffic filtering. Thus, I would like to use mangle rules instead, but can’t make them work the way routing rules do.
Does anybody know what the actual difference is between these 2 methods? Do I miss any more mangle rules that routing rules have under the hood?
Maybe. but we can’t see what you have missed if you do not post more of your configuration.
Post the rest of your config when you are trying to use mangle rules so we can see where there might be an error.
Also the requirement should be expressed in terms of user traffic required.
Mangling and routing rules are simply tools to use, for a purpose, and that purpose has not been communicated…
What I would like to achieve is a domain-based VPN: some domains are resolved, their addresses put into lists, these lists used by mangle rules to route traffic through VRF.
UPD: fixed the config excerpt above to be consistent with a full config posted below.
I use domain based vpn with mangle (mark routing).
Additionally you have to add ip-route rule to route (what you marked with mangle) to vpn gateway.
Example:
If you want I can post here step by step config.
My setup uses local country internet directly. And everything that is not local country with vpn .
My vpn is remote vps with libreswan ipsec
Mikrotik is an ipsec client .
I tried to use remote vps\vpn as chr mikrotik but I failed to get speeds more than 40-50 mbps.
Simple ubuntu with libreswan 1 cpu 512 ram can be set up with scripts from github in 5 min, produce up to 185 mbps thru ipsec.
Same easily can be done thru wireguard.
You should be awared that chr withiut licence is limited to 1mbps.
Hi,
I built an approximation to this, but with no vxlans (just used another bridge, and ether2 as wan2)
It seemed to mostly work, but the vrf-wan2 being a vrf made it quite painful.
I seemed to need to reboot whenever I had done more than a couple of changes.
I was also unable to traceroute from the router using src-address=192.168.2.1
My laptop on 192.168.2.220 was fine, (though this is somewhat expected).
I turned vrf-wan2 into just routing table (in fib) entry and adjusted for that and it seemed to work quite a bit better and easier.
(You do lose some isolation, but you don’t seem to really need any in this instance)
Some (my) notes on routing, seem to be how it is currently working. (not really checked for vrf’s)
First:
If the packet has a routing mark on it, and there is a matching entry in the routing table.
(IP range, routing table matches routing mark) it WILL use that routing entry.
This means if there is an item like
add disabled=no distance=100 dst-address=192.168.2.0/24 gateway=lan pref-src=“” routing-table=
vrf-wan2 scope=30 suppress-hw-offload=no target-scope=10
It seems it will (attempt to) send the marked packet out the lan interface, even if it is going to 192.168.2.1
Next:
It will then use the Routing Rules table, (in order) and grab the first match, and do what it says.
Next2:
It will then process the packet through the routing table again, potentially with a different routing mark obtained from the rules table.
(But it seems perhaps a bit more lenient this time)
You can (and it works well) use the routing rules table with routing marks.
mangle:
mark selected packets with routing mark eg. rule-wan2
Thank you for a detailed reply. This is roughly what I have, I just simplified the config to a single target address, but I have the same problem if I use lists in mangle rules instead. And I certainly have a default gateway in vrf-wan2. Is your “ipsec” routing table a VRF or a simple routing table created manually?
Do you have anything else in your config that is relevant? E.g. firewall rules, ip routes, routing rules, etc.
These speed issues are probably due to ipsec. I use wireguard and it easily handles 200-300 mbps between 2 CHRs or CHR and RB5009, at least as shown by the built-in bandwidth test tool. I think, it can even do faster, but it is VPS bandwidth that seems to be a limitation for me in terms of speed.
It’s a good idea to replace VRF with a simple routing table in FIB. Unfortunately, it barely changed anything for me. Tested on an empty CHR, changes to the full config posted above are the following:
/ip vrf
add interfaces=none name=vrf-wan2
/routing table
add disabled=no fib name=rtab-wan2
/ip route
add distance=100 dst-address=0.0.0.0/0 gateway=10.16.51.1 routing-table=rtab-wan2
add distance=100 dst-address=192.168.2.0/24 gateway=lan routing-table=rtab-wan2
# IT DOESN'T WORK: icmp traceroute from 192.168.2.2 to 9.9.9.9 never goes beyond 192.168.2.1
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=yes dst-address=9.9.9.9/32 new-routing-mark=rtab-wan2 src-address=192.168.2.0/24
# IT WORKS: icmp traceroute from 192.168.2.2 to 9.9.9.9 goes via 192.168.2.1 and 10.16.51.1 (wan2 gw)
/routing rule
add action=lookup disabled=yes dst-address=9.9.9.9/32 interface=lan src-address=192.168.2.0/24 table=rtab-wan2
# IT ALSO WORKS WITH BOTH RULES DISABLED: icmp traceroute from 192.168.2.2 to 9.9.9.9 goes via 192.168.2.1 and wan1 gw
That’s another good idea I tested myself. In fact, it equals to enabling both mangle and routing rules (because mangle mark routing only allows setting routing tables as marks). It doesn’t work for me either. But I’m not sure whether it should, since it is discouraged to combine the two methods of policy routing according to the docs.
That is a key, thanks! I replaced vxlan with eoip and mangle mark routing now works.
So, it seems to be a vxlan bug, will report to MikroTik support soon.
UPD: SUP-136716 + reply on 27 December 2023:
We have managed to reproduce the issue locally in our labs and look forward to fixing it on upcoming RouterOS versions, unfortunately, I cannot provide a release date now.
Tested several configuration, and there are some approach to address this issues with its own pros and cons.
Totally using Tables Rule.
Pros: simple when dealing with few vlan/subnet, auto-block vlan/subnet i.e. can not ping each other, etc.
Cons: more vlan more headache, and need to work out to make some ip/subnet (i.e. local server) accessible from other vlan/subnet.
Combining Tables Rule with Mangle>Mark Routing.
Pros: more flexible, and if it configured properly (with nat and ip>route) then no need to worry about access vlan/port/subnet on same bridge.
Cons: I guess it takes more resources (cpu, memory), but I’m not sure.
Look at this line:
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=!local_subnet new-routing-mark=rtab-WAN1
It is substantial when we only want to mark connection to internet. So local connection not affected by Table Rule.
No! dst-address-type local is not what you think. local in this case means all the individual IP addresses of the router itself, and it includes any WAN public IP addresses of the router too. It doesn’t contain the subnets of the LAN interfaces!
I might be wrong, lets find out the truth. https://wiki.mikrotik.com/Manual:IP/Firewall/Mangle dst-address-type (unicast | local | broadcast | multicast; Default: )
local - if dst-address is assigned to one of router’s interfaces
So my guess is that ip-to-interface assignments located : IP->Adresses
I suppose that all the interface-to-address assignments are included into “local” alias.
As well as defconf bridge like this:
/ip address
add address=192.168.0.1/24 comment=defconf interface=bridge network=192.168.0.0
There is normally another defconf:
/interface list member
add comment=defconf interface=bridge list=LAN
In your example dst-address-type=local will match 192.168.0.1 (that single /32 address) but not 192.168.0.2, 192.168.0.3, etc… So, you can’t use it to replace dst-address=192.168.0.0**/24**! Similarly, dst-address-type=!local excludes 192.168.0.1 but will happily match 192.168.0.40, which means it’s not a replacement for dst-address=!192.168.0.1/24.
with this definition of local_subnet from @Lokamaya’s post:
where 91.82.73.64 is a WAN IP addresses provided by the ISP, and for instance 192.168.10.1/24 and 192.168.20.1/24 were assigned to two interfaces of the routers.