Mangle Rules with Multi WAN

insanex3 · June 30, 2024, 12:28pm

Hello,
I am very new to using Mikrotik. I use a CHR in my homelab.
I have two ISPs, from each of them I get a dynamic IPv4 address.

I want to route some of the connections via ISP2 based on the destination port. As I understood, I have to use Mangle Rules to accomplish that. I did set that up according to the mikrotik wiki.

To test the configuration, I created a new mangle rule, which matches ICMP traffic to 1.1.1.1 - just to see, if this would send my packets via ISP2, but this seems not to work.

Also i don’t quite understand, what I have to do, with the 2 new routing tables. Currently all my routes are in “main”. I tried adding new routes in the 2 other tables, but I can’t statically enter the dst. ip address since my ISP provides me with a dynamic IPv4.
Also I am unsure, when to use passthrough on the mangle rules and when not.

mangle:

[admin@MikroTik] /ip/firewall/mangle> print
Flags: X - disabled, I - invalid; D - dynamic 
 0    chain=prerouting action=mark-connection new-connection-mark=ISP1_Conn passthrough=yes connection-mark=no-mark in-interface=WAN-DREI log=no log-prefix="" 

 1    chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn passthrough=yes connection-mark=no-mark in-interface=WAN-INNONET log=no log-prefix="" 

 2    chain=prerouting action=mark-connection new-connection-mark=ISP1_Conn passthrough=yes psd=21,3s,3,1 dst-address-type=!local connection-mark=no-mark in-interface=bridge1 limit=1,5:packet dst-limit=1,5,dst-address/1m40s 
      time=0s-1d,sun,mon,tue,wed,thu,fri,sat per-connection-classifier=both-addresses:2/0 log=no log-prefix="" 

 3    chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn passthrough=yes psd=21,3s,3,1 dst-address-type=!local connection-mark=no-mark in-interface=bridge1 limit=1,5:packet dst-limit=1,5,dst-address/1m40s 
      time=0s-1d,sun,mon,tue,wed,thu,fri,sat per-connection-classifier=both-addresses:2/1 log=no log-prefix="" 

 4    chain=prerouting action=mark-routing new-routing-mark=to_ISP1 passthrough=yes connection-mark=ISP1_Conn in-interface=bridge1 log=no log-prefix="" 

 5    chain=prerouting action=mark-routing new-routing-mark=to_ISP2 passthrough=yes connection-mark=ISP2_Conn in-interface=bridge1 log=no log-prefix="" 

 6    chain=output action=mark-routing new-routing-mark=to_ISP1 passthrough=yes connection-mark=ISP1_Conn log=no log-prefix="" 

 7    chain=output action=mark-routing new-routing-mark=to_ISP2 passthrough=yes connection-mark=ISP2_Conn log=no log-prefix="" 

 8    chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn passthrough=yes protocol=icmp dst-address=1.1.1.1 in-interface=bridge1 log=no log-prefix=""

/ip/route:

[admin@MikroTik] /ip/route> print
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
    DST-ADDRESS       GATEWAY         DISTANCE
DAd 0.0.0.0/0         178.xxx.xxx.xxx        1
DAc 149.xxx.xxx.xxx/23    WAN-INNONET            0
DAc 172.27.63.0/24    VLAN-IOT               0
DAc 178.xxx.xxx.xxx/27  WAN-DREI               0
DAc 192.168.1.0/24    bridge1                0

rplant · July 2, 2024, 11:47am

Hi,

I have a dhcp-client script that changes some of the routes.

/ip dhcp-client
add default-route-distance=70 interface=ether2 script="/ip route\r\
    \n:if (\$bound = 1) do={\r\
    \n  set [find where comment~\"altgw\"]  gateway=\$\"gateway-address\"\r\
    \n}"

I also find it better to go via routing rules.
The mangle rules specify routing marks that live in the routing rules, rather than directly specifying routing tables in ip route.

/ip firewall mangle
add action=mark-routing chain=prerouting comment=via-alt src-address-list=USEALT new-routing-mark=Rmaybe-alt passthrough=yes

First entry in routing rule assumes all known non default routes use main table.

/routing rule
add action=lookup comment="min-prefix=0, all except 0.0.0.0/0" disabled=no min-prefix=0 table=main
add action=lookup disabled=no dst-address=0.0.0.0/0 routing-mark=Rmaybe-alt table=maybe-alt

in ip route I have routes some with comments that include the word “altgw” where the gateway is updated by the above script when the dhcp-client
gets a new/updated lease.

/ip route
add comment="altgw via-alt" disabled=no distance=50 dst-address=0.0.0.0/0 gateway=192.168.114.157 pref-src=0.0.0.0 \
  routing-table=maybe-alt scope=30 suppress-hw-offload=no target-scope=11

Note also:
Packets that need to go via the secondary gateway (with higher distance) must not go via fasttrack.
Can put an accept rule in ip firewall filter, forward chain prior to the fasttrack rule for these specific packets,

/ip firewall filter
add action=accept chain=forward comment="accept ether2 traffic, established related" in-interface=ether2 \
 connection-state=established,related
add action=accept chain=forward comment="accept ether2 traffic, established related" out-interface=ether2 \
 connection-state=established,related
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes

llamajaja · July 2, 2024, 6:13pm

So the single and only special requirement for your configuration is that ANY user (not single user, group of users or one subnet of users if have multiple subnets) with destination of port ABC, must use WAN2?

I’m assuming WAN1 is primary and WAN2 is secondary (failover).
What you haven stated is WHAT happens when WAN2 is down and not available ???

/ip route
add dst-address=0.0.0.0/0 gateway=ISP1 routing-table=main check-gateway=ping distance=2
add dst-address=0.0.0.0/0 gateway=ISP2 routing-table=main check-gateway=ping distance=4
add dst-address=0.0.0.0/0 gateway=ISP2 routing-table=to-WAN2

Then you need to mangle traffic for that port
add chain=prerouting action=mark-routing [ ****** Put in applicable Source**** ] dst-port=ABC
protocol=tcp new-routing-mark=to-WAN2 passthrough=no

****** src-address=Subnet, OR src-address-list=list of users, OR in-interface=, OR in-interface-list= etc…

/routing table
add fib name=to-WAN2

As alluded to by the previous poster, the difficulty is that the gateway of the ISPs, being dynamic will change so you need a mechanism to recognize when that change occurs and then modify the existing routes. To do so the easiest technique is to identify by unique comments…

add dst-address=0.0.0.0/0 gateway=ISP1 routing-table=main check-gateway=ping distance=2 comment=ID-WAN1
add dst-address=0.0.0.0/0 gateway=ISP2 routing-table=main check-gateway=ping distance=4 comment=ID-WAN2
add dst-address=0.0.0.0/0 gateway=ISP2 routing-table=to-WAN2 comment=ID-WAN2

llamajaja · July 2, 2024, 6:17pm

I am not sure why rplant is suggesting routing rules because that assumes you are working with IP addresses only.
This is only true for SOURCE but not for DESTINATION. The only requirement as you have stated is basically any IP address with dst port ABC needs to go through WAN2.
Routing RULES do NOT distinguish between ports and thus are not applicable here. You will capture all traffic from the users, not just the port in question.

Since you have mangled already and now simply need to add the route, there is no point in adding a redundant routing rule on top. Makes no sense to me, but perhaps I am missing something.

Regardless, in general, whether one mangles or uses routing rules,

you need a table
you need a route referencing the table.

llamajaja · July 2, 2024, 6:30pm

The point about fastrack is very important as you want to keep fastrack for the rest of your traffic.
Since we have clearly identified a source address structure for this traffic we can really nail it down.

Using rplants excellent work for maintaining fastrackk (for dynamic gateway changes as well)

/ip firewall filter
add action=accept chain=forward comment=“Avoid fastrack outgoing” connection-state=established,related [ ****** Put in applicable Source ]
add action=accept chain=forward comment=“Avoid fastrack incoming” connection-state=established,related [ [ ****** Put in applicable Destination ]
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related hw-offload=yes

**** src-address=local-subnet, OR src-address-list=local-address-list etc.
**** dst-adress=local-subnet OR dst-address-list=local-addess-list etc.

what is not clear to me and would be beneficial, to limit to port traffic as well…
is this possible, probably not, since we do not know what the reply port would be necessarily…?
add action=accept chain=forward comment=“Avoid fastrack outgoing” connection-state=established,related [ ****** Put in applicable Source ] dst-port=ABC protocol=tcp
add action=accept chain=forward comment=“Avoid fastrack incoming” connection-state=established,related [ [ ****** Put in applicable Destination ] dst-port=ABC protocol=tcp
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related hw-offload=yes

insanex3 · July 5, 2024, 10:38am

Thanks everybody for your replies!
I used the script from rplant in my DHCP Client and it does set the Gateway in the routing table “to_ISP2”. Also I cleaned up my mangle rules a little bit. I figured, I dont need all that mangle rules for my ISP1 connection, since everything goes over this by default anyway.
Do these look correct to you guys and is the “passthrough” checkmark set correctly, where it needs to be set?

[admin@MikroTik] /ip/firewall/mangle> print
Flags: X - disabled, I - invalid; D - dynamic 
 0    chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn 
      passthrough=yes connection-mark=no-mark in-interface=WAN-INNONET log=no 
      log-prefix="" 

 1    chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn 
      passthrough=yes psd=21,3s,3,1 dst-address-type=!local 
      connection-mark=no-mark in-interface=bridge1 limit=1,5:packet 
      dst-limit=1,5,dst-address/1m40s time=0s-1d,sun,mon,tue,wed,thu,fri,sat 
      per-connection-classifier=both-addresses:2/1 log=no log-prefix="" 

 2    ;;; Routing via INNONET
      chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn 
      passthrough=yes protocol=tcp in-interface=bridge1 dst-port=1119,3724,6012 
      log=no log-prefix="" 

 3    ;;;  Routing via INNONET
      chain=prerouting action=mark-connection new-connection-mark=ISP2_Conn 
      passthrough=yes protocol=udp in-interface=bridge1 dst-port=1119,3724,6012 
      log=no log-prefix="" 

 4    chain=prerouting action=mark-routing new-routing-mark=to_ISP2 
      passthrough=yes connection-mark=ISP2_Conn in-interface=bridge1 log=no

Also could someone explain me, what the setting “per-connection-classifier=both-addresses:2/1” is doing? I added this, because it was stated in the mikrotik wiki.

Some of you were talking about fasttrack, ist that working now with my setup or do I need to add the filter rules llamajaja supplied?

Thanks for your help and sorry for the load of questions.

anav · July 5, 2024, 2:06pm

Ask Mikrotik, we didnt write the Wiki, nor was adding PCC load balancing ever part of your request. AKa am moving on.