PCC balance of two WAN (lte) and failover

export.txt.rsc (7.09 KB)
On RB3011 I connected two identical lte routers one to port eth1 and the other to port eth10 and assigned static ip to both.

Inserting these rules that I copied from a previous configuration the balance works, but if I disconnect one of the two wan and try to navigate I see the main page of the router lte
and the vpn gets stuck

/ip route
add gateway=192.168.8.1 routing-mark=wan1
add gateway=192.168.9.1 routing-mark=wan2

/ip firewall mangle
add action=accept chain=prerouting dst-address=192.168.8.0/24
add action=accept chain=prerouting dst-address=192.168.9.0/24
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=wan1conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=wan2conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-routing chain=prerouting connection-mark=wan1conn new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan2conn new-routing-mark=wan2 passthrough=no

Someone can help me, maybe I forgot some important rules in the firewall , or rules routes.

the balance works quite well, I noticed that the traceroute, ping and updates on the RB3011 no longer works and the vpn does not connect, I read many posts but I do not understand what to add in the firewall.
I added in /ip route
add distance=1 gateway=192.168.8.1
vpn works and pinging from tik works, but internet browsing becomes very slow.

Use of routing marks is incompatible with use of fasttracking. If you disable the action=fasttrack-connection rule in chain forward of /ip firewall filter, newly established connections will be handled with normal speed.

Depending on your uplink bandwidth, the CPU power may be sufficient fo fully use it even with fasttracking disabled; if it is not, you may want to use fasttracking selectively.

Hi, Sindy,
I changed again the firewall rules and routes because I wanted to have a failover with the recursive method, I disabled the fastrack as you suggested. Now they work . ping direct from Tik, vpn and failover
Balancing does not work, internet browsing only takes place on gateway 192.168.8.1 .
Here are the firewall files and the routes where I am wrong?
firewall.txt.rsc (5.14 KB)
iproute.txt.rsc (1011 Bytes)

The only marked route I can see there is the one via VPN. So load balancing cannot work as it assigns routing marks for which no routes exist.

So you’d have to add at least
add distance=1 gateway=10.2.2.2 routing-mark=to_WAN2
add distance=2 gateway=10.1.1.1 routing-mark=to_WAN2

the balance works perfectly!! but the mangle rule that routes in sip traffic on the vpn no longer works

add action=mark-routing chain=prerouting comment=\
    "redirect sip to VPN pc frank" new-routing-mark=vpn passthrough=yes \
    protocol=udp src-address=10.0.0.152 src-port=5060

I deleted the rule and rewritten it in the same position, now it works.
Mysteries!
Schermata del 2020-06-06 18.51.23.png

I’m noticing, however, that while browsing I lose the ip on the pages. For example, if I log into the mikrotik forum I lose the login.
How you could solve?

In your specific case where you use per-connection-classifier to assign the connection-mark, you can set the values-to-hash to dst-address-and-port, so all connections to the same remote server will match the same rule no matter which port wil be chosen on client side.

If other strategies of load balancing are used, you can “statically” override the generic balancing rules for sites which are known to check client’s IP address:

/ip firewall address-list add list=use-WAN1 address=forum.mikrotik.com

/ip firewall mangle add … dst-address-list=use-WAN1 action=mark-connection new-connection-mark=WAN1_conn
(placed before the other action=mark-connection rules).

And, lastly, instead of placing the remote servers to address lists statically, you can populate these lists automatically - whenever you send a packet to a given remote IP via a given WAN, you add that IP to the address list for that WAN with some expiration time, so any new connection to the same address will be established via the same WAN. Only connections to remote IPs which are not on any address list yet will be distributed “freely”.

Interesting .
I figured out how to make a static list of addresses.
Can you explain how to make a dynamic list with an expiry time for accumulated addresses?

/ip firewall mangle
add chain=postrouting out-interface=WAN1 action=add-dst-to-address-list address-list=use-WAN1 address-list-timeout=1h
add chain=postrouting out-interface=WAN2 action=add-dst-to-address-list address-list=use-WAN2 address-list-timeout=1h

Sindy, that’s what I did.

/ip firewall mangle
add action=passthrough chain=postrouting comment="test_speedtest.net (server 88.149.202.248)" dst-address-list=speedtest protocol=tcp routing-mark=to_WAN1
add action=passthrough chain=postrouting dst-address-list=speedtest protocol=tcp routing-mark=to_WAN2

add action=add-dst-to-address-list address-list=use-WAN1 address-list-timeout=1h chain=postrouting comment="list connection balance" out-interface=ether1-WAN1
add action=add-dst-to-address-list address-list=use-WAN2 address-list-timeout=1h chain=postrouting out-interface=ether10-WAN2

add action=mark-connection chain=prerouting comment="Allocate specific WAN connection to host" connection-mark=no-mark disabled=yes dst-address-type=!local \
    in-interface=bridge new-connection-mark=WAN1_conn passthrough=yes src-address=10.0.0.202
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn \
    passthrough=yes src-address=10.0.0.152
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn passthrough=yes \
    src-address=10.0.0.109
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn passthrough=yes \
    src-address=10.0.0.191
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn \
    passthrough=yes src-address=10.0.0.30
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn passthrough=yes \
    src-address=10.0.0.70
add action=mark-connection chain=input comment="PCC for LTE" connection-mark=no-mark in-interface=ether1-WAN1 new-connection-mark=WAN1_conn
add action=mark-connection chain=input connection-mark=no-mark in-interface=ether10-WAN2 new-connection-mark=WAN1_conn
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=to_WAN1
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=to_WAN2
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN2_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface=bridge new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface=bridge new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting comment="redirect sip pcfrank, to VPN " new-routing-mark=vpn passthrough=yes protocol=udp src-address=10.0.0.152 src-port=\
    5060

but does not create lists

That’s weird because the same rule on my machine does (I’ve substituted just the interface name and timeout, otherwise copy-pasted from your post).

It seems to work, I’ve put the mangle rules in this way for both WAN1 and WAN2.

/ip firewall mangle

add action=add-dst-to-address-list address-list=use-WAN1 address-list-timeout=1h chain=postrouting comment="list connection balance" out-interface=ether1-WAN1
add action=add-dst-to-address-list address-list=use-WAN2 address-list-timeout=1h chain=postrouting out-interface=ether10-WAN2
add action=mark-connection chain=prerouting dst-address-list=use-WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=prerouting dst-address-list=use-WAN2 new-connection-mark=WAN2_conn passthrough=yes

Schermata del 2020-06-06 23.06.53.png
I’ll try to reboot the router tonight, I’ll let you know in the morning if it’s working properly.
Thanks, Sindy.

sindy, thanks again,
everything works properly, practically the traffic is sorted a bit on one gateway a bit on the other keeping the routes for an hour.
The only thing I lost with the latest changes are the multiple connections and the sum of the total bandwidth, but I think it’s inevitable.
I’ll be adding a copy of my configuration shortly to help those who will find it as difficult as I do.

If you mean that speedtest always uses a single uplink for all test sessions, then yes, this is inevitable, unless you’d turn the approach upside down and instead of manually configuring remote IPs which require use of the same WAN, you’d manually configure IPs which don’t.

:open_mouth: I don’t understand exactly how you can do it and what you get is very complicated.
I enclose here below my final working configuration.
mybackup.rsc (12.5 KB)

Actually the config is a bit confusing, some questions!

(1) You have at SIX duplicate set of this rule for specific source addresses. 10.0.0.202, 10.0.0.152, 10.0.0.109, 10.0.0.191, 10.0.0.30, 10.0.0.70 !

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=
yes dst-address-type=!local in-interface=bridge new-connection-mark=
WAN1_conn passthrough=yes src-address=10.0.0.70



PLUS the usual expected rule for PCC!!

add action=mark-connection chain=prerouting connection-mark=no-mark
dst-address-type=!local in-interface=bridge new-connection-mark=WAN1_conn
passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark
dst-address-type=!local in-interface=bridge new-connection-mark=WAN2_conn
passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1

What is the purpose of the six rules you have that assign new connection marks to specific IP addressess on the bridge??
I am assuming its so these IPs never get assigned WAN2, but only WAN1?

(2) The difficulty with dischers PCC is that he assumes fixed/static WANIPs, and I was looking for a way around his config for dyanmic IPs…
But I see you are using fixed WANIPs so this example doesnt provide any illumination on the issue.

What I found interesting though, is your use of these rules (dischers first two rules) OR LACK THEREOF,
Nowhere to be found are these ACCEPT RULES…
ip firewall mangle
add action=accept chain=prerouting disabled=no dst-address=fixedIP-WAN1
add action=accept chain=prerouting disabled=no dst-address=fixedIP-WAN2

Instead you have six rules I have no clue what you are doing…???

/ip firewall mangle
add action=passthrough chain=postrouting comment=\
    "test_speedtest.net (server 88.149.202.248)" dst-address-list=speedtest \
    protocol=tcp routing-mark=to_WAN1

add action=passthrough chain=postrouting dst-address-list=speedtest protocol=\
    tcp routing-mark=to_WAN2

add action=add-dst-to-address-list address-list=use-WAN1 \
    address-list-timeout=1h chain=postrouting comment=\
    "list connection balance" out-interface=ether1-WAN1

add action=add-dst-to-address-list address-list=use-WAN2 \
    address-list-timeout=1h chain=postrouting out-interface=ether10-WAN2

add action=mark-connection chain=prerouting dst-address-list=use-WAN1 \
    new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=prerouting dst-address-list=use-WAN2 \
    new-connection-mark=WAN2_conn passthrough=yes

@anav,

  1. those rules (some are disabled) are pecedent to those of PCC and are used to direct the traffic of some devices with fixed ip (managed by dhcp server) exclusively to a specific WAN.
  2. I didn’t understand your problem with dynamic ip’s.
    ---- In dhcp server I reserved static addresses for some devices .

---- the first two rules you see make sure that some services (grouped in list-address) such as speed test , are processed in postrouting and thus do not suffer from any slowdowns .
---- the third and fourth regenerate lists to instard traffic , so that some sites (such as banks ) are always connected from the same WAN
---- the fifth and sixth is prerouting the WANs .