load-balancing don't work

plisken · March 2, 2019, 4:27am

Hello,
I have searched on the forum but nothing seems to work because they are outdated.
The wiki did not yield anything either. That’s why my new topic

I want to configure an 8x WAN load balancing and that did not work.
It also did not work when i testing a 2xWAN load-balancing configuration.

It seemed that the distance failed because when the two internet gateways were connected, gateway 2 was always used.
After a while search, I have downgraded the RouterOs to
6.37.4 bug fix on an old Mikrotik router.
Then the load balancing worked properly.
But I want to make this configuration with the newest 6.44.
This does not seem to be possible after the RouterOS 6.37.4

For clarity this code is made with RouterOS 6.44 on an RB760iGS
What am I doing wrong with this configuration?

/interface bridge
add name=bridge-local
/interface ethernet
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
set [ find default-name=ether3 ] name=ether3_lan
set [ find default-name=ether4 ] name=ether4_lan
set [ find default-name=ether5 ] name=ether5_trunk
/ip pool
add name=dhcp_pool0 ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=bridge-local name=dhcp1
/interface bridge port
add bridge=bridge-local interface=ether3_lan
add bridge=bridge-local interface=ether4_lan
/ip address
add address=192.168.100.1/24 interface=bridge-local network=192.168.100.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=WAN1 use-peer-dns=no
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=WAN2 use-peer-dns=no
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.1.1 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.1.2.2
/ip dns static
add address=192.168.100.1 name=ns1.uwbedrijf.net
/ip firewall mangle
add action=accept chain=prerouting dst-address=192.168.1.0/24 in-interface=bridge-local
add action=accept chain=prerouting dst-address=10.98.213.0/24 in-interface=bridge-local
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface=bridge-local new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface=bridge-local new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=to_WAN2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
/ip route
add check-gateway=ping distance=1 gateway=192.168.1.1 routing-mark=to_WAN1
add check-gateway=ping distance=2 gateway=10.98.213.20 routing-mark=to_WAN2

Who can help me with a 2x WAN load balancing so that I can finish the 8x WAN configuration

anav · March 2, 2019, 2:46pm

Get rid of the static DNS setting.
You dont have eth5 on a bridge port???
Next to peruse mangle rules…

I’m no expert and thus recommend this MUM presentation…
https://mum.mikrotik.com/presentations/LB19/presentation_6360_1548749570.pdf

I note it uses in-interface=LAN for its two pre-routing accept rules where you have use in-interface=bridge?? Not sure if that makes a difference?

The final slide says
RECAP
•New connections inbound on each WAN get marked
•Outbound connections with that mark get a routing mark
•LAN traffic heading outbound gets load balanced with the same routing marks
•Routing marks match default gateway routes and head out that specified interface
•Verification of the Load balancing setup

plisken · March 2, 2019, 4:11pm

Thanks for your reply, ether5 must become a trunk for vlan.
I try this link out and let you know if it works.
Thanks

plisken · March 7, 2019, 2:55am

This configuration don’t work.
I have not a commulatieve bandwidth and ISP2 become the main ISP in place off ISP1.

I want to set ISP1 to the main ISP because this is the fasted ISP.
WHo can help me.

plisken · March 7, 2019, 5:16am

This is the code from the 8x failover and load-balancing configuration that not working correct.

/interface bridge
add fast-forward=no name=LAN
/interface ethernet
set [ find default-name=ether9 ] name=LAN9
set [ find default-name=ether10 ] name=LAN10
set [ find default-name=ether11 ] name=LAN11
set [ find default-name=ether12 ] name=LAN12
set [ find default-name=ether13 ] name=LAN13
set [ find default-name=ether14 ] name=LAN14
set [ find default-name=ether15 ] name=LAN15
set [ find default-name=ether16 ] name=LAN16
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
set [ find default-name=ether3 ] name=WAN3
set [ find default-name=ether4 ] name=WAN4
set [ find default-name=ether5 ] name=WAN5
set [ find default-name=ether6 ] name=WAN6
set [ find default-name=ether7 ] name=WAN7
set [ find default-name=ether8 ] name=WAN8
/ip pool
add name=dhcp_pool0 ranges=192.168.100.20-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=LAN name=dhcp1
/interface bridge port
add bridge=LAN interface=LAN9
add bridge=LAN interface=LAN10
add bridge=LAN interface=LAN11
add bridge=LAN interface=LAN12
add bridge=LAN interface=LAN13
add bridge=LAN interface=LAN14
add bridge=LAN interface=LAN15
add bridge=LAN interface=LAN16
/ip address
add address=192.168.100.1/24 interface=LAN network=192.168.100.0
add address=192.168.1.2/24 interface=WAN1 network=192.168.1.0
add address=192.168.2.2/24 interface=WAN1 network=192.168.2.0
add address=192.168.3.2/24 interface=WAN1 network=192.168.3.0
add address=192.168.4.2/24 interface=WAN1 network=192.168.4.0
add address=192.168.5.2/24 interface=WAN1 network=192.168.5.0
add address=192.168.6.2/24 interface=WAN1 network=192.168.6.0
add address=192.168.7.2/24 interface=WAN1 network=192.168.7.0
add address=192.168.8.2/24 interface=WAN1 network=192.168.8.0
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.2.2 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.1.2.2
/ip dns static
add address=192.168.100.1 name=ns1
/ip firewall mangle
add action=mark-connection chain=input in-interface=WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN3 new-connection-mark=WAN3_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN4 new-connection-mark=WAN4_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN5 new-connection-mark=WAN5_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN6 new-connection-mark=WAN6_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN7 new-connection-mark=WAN7_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN8 new-connection-mark=WAN8_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN3_conn new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN4_conn new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN5_conn new-routing-mark=to_WAN5 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN6_conn new-routing-mark=to_WAN6 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN7_conn new-routing-mark=to_WAN7 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN8_conn new-routing-mark=to_WAN8 passthrough=yes
add action=accept chain=prerouting dst-address=192.168.1.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.2.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.3.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.4.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.5.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.6.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.7.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.8.0/24 in-interface=LAN
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:8/0
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:8/1
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN3_conn passthrough=yes per-connection-classifier=both-addresses:8/2
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN4_conn passthrough=yes per-connection-classifier=both-addresses:8/3
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN5_conn passthrough=yes per-connection-classifier=both-addresses:8/4
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN6_conn passthrough=yes per-connection-classifier=both-addresses:8/5
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN7_conn passthrough=yes per-connection-classifier=both-addresses:8/6
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN8_conn passthrough=yes per-connection-classifier=both-addresses:8/7
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface=LAN new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface=LAN new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn in-interface=LAN new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN4_conn in-interface=LAN new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN5_conn in-interface=LAN new-routing-mark=to_WAN5 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN6_conn in-interface=LAN new-routing-mark=to_WAN6 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN7_conn in-interface=LAN new-routing-mark=to_WAN7 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN8_conn in-interface=LAN new-routing-mark=to_WAN8 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
add action=masquerade chain=srcnat out-interface=WAN3
add action=masquerade chain=srcnat out-interface=WAN4
add action=masquerade chain=srcnat out-interface=WAN5
add action=masquerade chain=srcnat out-interface=WAN6
add action=masquerade chain=srcnat out-interface=WAN7
add action=masquerade chain=srcnat out-interface=WAN8
/ip route
add check-gateway=ping distance=1 gateway=192.168.1.1 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=192.168.2.1 routing-mark=to_WAN2
add check-gateway=ping distance=1 gateway=192.168.3.1 routing-mark=to_WAN3
add check-gateway=ping distance=1 gateway=192.168.4.1 routing-mark=to_WAN4
add check-gateway=ping distance=1 gateway=192.168.5.1 routing-mark=to_WAN5
add check-gateway=ping distance=1 gateway=192.168.6.1 routing-mark=to_WAN6
add check-gateway=ping distance=1 gateway=192.168.7.1 routing-mark=to_WAN7
add check-gateway=ping distance=1 gateway=192.168.8.1 routing-mark=to_WAN8
add check-gateway=ping distance=1 gateway=192.168.1.1
add check-gateway=ping distance=2 gateway=192.168.2.1
add check-gateway=ping distance=3 gateway=192.168.3.1
add check-gateway=ping distance=4 gateway=192.168.4.1
add check-gateway=ping distance=5 gateway=192.168.5.1
add check-gateway=ping distance=6 gateway=192.168.6.1
add check-gateway=ping distance=7 gateway=192.168.7.1
add check-gateway=ping distance=8 gateway=192.168.8.1

See the step by step video a made. In dutch but follow the test configuration.
https://www.youtube.com/watch?v=KO_9UsXb2_w

What do I want to achieve:
Load-balancing starting to WAN1, WAN2 etc.
I want to combine the 8 WAN bandwidths to create one larger bandwidth on 1 LAN infrastructure
A correct failover.

anav · March 7, 2019, 12:07pm

from
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.2.2 gateway=192.168.100.1
to
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1 gateway=192.168.100.1

remove this
/ip dns static
add address=192.168.100.1 name=ns1

Not sure but thinking that for fixed WANIPs…
these should all be source nat rules not masquerade rules.

/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
add action=masquerade chain=srcnat out-interface=WAN3
add action=masquerade chain=srcnat out-interface=WAN4
add action=masquerade chain=srcnat out-interface=WAN5
add action=masquerade chain=srcnat out-interface=WAN6
add action=masquerade chain=srcnat out-interface=WAN7
add action=masquerade chain=srcnat out-interface=WAN8

Will look at mangles and routing later

solar77 · March 7, 2019, 1:34pm

I have not used PCC but compare your config with https://wiki.mikrotik.com/wiki/Manual:PCC
the example does not use passthrough=yes,
without understand all your config, i suspect some of the traffic got marked twice because of this. then the result would be only WAN2_conn is left and that’s why all traffic is going to WAN2

I’d suggest remove all passthrough=yes and enable log on the mangle rules to do with WAN1 and watch what is going through and what is not.

Let us know.

anav · March 7, 2019, 3:01pm

Solar77, that is an interesting observation.
This begs the question for me, as to the purpose and use of passthrough=yes and passthrough=no!
Everytime I see that used I get confused.
I always thought that passthrough=no means there is no more rules for this data flow in this ‘bounded ruleset’
I always thought that passthrough=yes means that there are more rules for this data flow in this “bounded rule set”
In other words, if there is no more ACTION of the same type required for this data flow, then ignore the rest of the same ACTIONs that follow??

okay doing some research, the Wiki tells me didly squat, too vague and does not address when passthrough is set to NO??
WIKI: passthrough - if packet is matched by the rule, increase counter and go to next rule (useful for statistics).

Okay one gem I understand LOL
(1) Passthrough = Yes Packet continues down the list of rules after this rule
Passthough = No Packet leaves the Mangle after this rule and is not affected by any of the following mangle rules

Other tidbits seen.
(2) For all “mark-packet” mangle rules it doesn’t make sense to set passthrough=yes.
If packet is already marked then simply do not send to next mangle rules, to save CPU resources. ****

(3) Dont market packets, mark connections … much more efficient.

Conclusion: Use passthrough=yes if you want any of the rest of the rules in that section to apply to that particular packet flow
So based on the above I would conclude that for the OPs config shown above:

A. first set rules - wan input mark connection rules should have passthrough=no. These packets will enter the router without any further mangle processing job done!
(purpose is to ensure packets coming in one WAN go out the same WAN for routing purposes)
B. second set rules - wan output mark routing rules should have passthrough=no These packets are leaving the router no further mangle processing job done!
(purpose working with A. is to ensure connections coming from wanx are routed back out wanx and so on)
C. third set rules - This is in three parts
i. basic assigning of mangle rules to the available destination addresses (wanIPs) for later pcc processing [ no option here for passthrough so not to worry about ]
ii. assign pcc classifier to traffic from lan heading to wan, to an unspecified WANIP but we will mark the connection with wanx aka the specific connection markings we used in A.
(Im assuming we are saying for destination traffic not headed locally, assign this pcc classifier and mark the connections with those we marked in A - tieing things together slowly) and to let the actual routing be handled by ip routes)
KEY HERE IS THAT THESE PACKETS THEN NEED TO HIT THE NEXT RULE SO passthrough=YES
iii. mark the connections from the previous pcc rule with a routing mark, so that ip routes knows where they need to go…
PASSTHROUGH=NO (no subsequent rules apply)

Based on this logic, the only place passthrough is required is during the PCC rules is to allow Marked Packets (via connection marking) to be Marked Routed!!!

SERIOUS
I also want to point out that the OPs configuration is completely 100% in line with the MUM presentation I linked to, from Lebanon 2019. https://mum.mikrotik.com/presentations/LB19/presentation_6360_1548749570.pdf
If that presentation is in error, then I would like to take MT to the court of ridicule and embarrassment for endorsing and allowing a presentation that is wrong and in direct conflict with the WIKI. If my supposition is true, if nothing else, someone at the next MUM should get Normis so drunk that he doesn’t realize he is being given a spanking, and it would make a great viral MT video!!
(pssst.. Vienna attendees do it regardless)

plisken · March 7, 2019, 3:34pm

Hello thanks for the reply’s, i don’t know what’s going wrong but i made a video with the log file. I hope someone can help me correct this error.
Also the people from Mikrotik.

https://youtu.be/6u7dQB_rhAE
I made the video for more clarity.
I’ve set up a log for every mangle rule.
In this situation I can only use 2 WAN

anav · March 7, 2019, 3:49pm

@plisken your config is not wrong but may benefit from some tweaking.
(as you read on later, the first thing to try is change wan interface incoming rules to PRE-ROUTING chain and NOT input chains.
(try this change first before changing any passthrough settings).

Try changing all passthrough rules to NO, except for these ones as they should remain YES…
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:8/0
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:8/1
etc…

@solar77
This is the config from the wiki , it is not correct from my understanding. The use of passthrough = yes ( my additions in red ) applies to the sequence of marking connections (Green text) on a packet flow and then that connection needs to be marked with routing marks (Blue text) and thus passthrough=no does NOT apply. After that connection flow has been route marked so to speak it can leave the mangle rules until mangle rules need to be reviewed again by the router.

/ ip firewall mangle
add chain=prerouting dst-address=10.111.0.0/24 action=accept in-interface=LAN
add chain=prerouting dst-address=10.112.0.0/24 action=accept in-interface=LAN
add chain=prerouting in-interface=ISP1 connection-mark=no-mark action=mark-connection
new-connection-mark=ISP1_conn
add chain=prerouting in-interface=ISP2 connection-mark=no-mark action=mark-connection
new-connection-mark=ISP2_conn
add chain=prerouting in-interface=LAN connection-mark=no-mark dst-address-type=!local
per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=ISP1_conn passthrough=yes
add chain=prerouting in-interface=LAN connection-mark=no-mark dst-address-type=!local
per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=ISP2_conn passthrough=yes
add chain=prerouting connection-mark=ISP1_conn in-interface=LAN action=mark-routing
new-routing-mark=to_ISP1
add chain=prerouting connection-mark=ISP2_conn in-interface=LAN action=mark-routing
new-routing-mark=to_ISP2
add chain=output connection-mark=ISP1_conn action=mark-routing new-routing-mark=to_ISP1
add chain=output connection-mark=ISP2_conn action=mark-routing new-routing-mark=to_ISP2

solar77 · March 7, 2019, 4:54pm

nice presentation, well you making me doubt myself.

however, it’s not complete identical. check page 22, the route section,
the presentation has 4 rules, the 1st two has identical distance, both been set o 1, and using routing mark. This is part of the load balancing route.
then the next two rules, I think, is the fail-over rule to the 1s two.
if any of the first two rules fails (gateway ping fails ), then Rule 3 will kick in, if it happens to be the gateway 192.168.0.1 that was failed, then rule 4 will kick in (that’s why Rule 4 has distance of 2) , ensure the traffic go to 192.168.0.1

regarding the passthrough=yes.
I think that this does, is
when it is yes,router will go through the rest of the rule to check if any of them applies, in this case, the connection will be checked by the rules below it
when it is NO, the mangle part is done and traffic will not be checked by the rules below it

edit:
@anav, you are probably right on the passthrough thing, once traffic is marked by connection, it still need the routing mark so the only two rules don’t need passthrough is the final two
reading more into it, mark connection will have passthrough=yes by default. so the manual does have them even it’s not in the command.
also the passthrough=no is only needed once the routing mark is done. as per the last example here:
https://wiki.mikrotik.com/wiki/Manual:IP/Firewall/Mangle

and it also say " whether to let the packet to pass further (like action passthrough) into firewall or not (property only valid some actions)." so yes we have to pass it through.
then It’s more the routing side to check. OP if you can chagne the routing as per the pdf presentation and watch the the traffic counter,

NetWorker · March 7, 2019, 5:03pm

I admittedly haven’t read through all the code because I don’t have a lot of time right now. But load balancing is all about mangle marking.

First mark connections that are NOT to be load balanced. You can either do action=accept or use some other mark. For example DNS requests or specific addresses that are present on one WAN but not on the other.

Then add one connection marking rule and one routing mark rule for each WAN. Connection marking can employ different selection criterias but I use NTH. Remember to set the “no-mark” flag in connection mark and passthrough=yes, otherwise it won’t reach the routing rule. You can use passthrough=no on the routing mark rules.

Finally add the static routes with the routing mark.

Note that this setup does NOT provide failover. If a WAN goes down, all traffic marked for that WAN will be passed on to the default gateway (if check gateway is enabled on your static routes) or timeout. It’s always good practice to set the distance for default gateways in order of preference (i.e. 1 for your preferred WAN, 2 for the second and so on). In case you don’t want traffic to go to the default gateway, you’ll have to script it.

anav · March 7, 2019, 5:32pm

More information: • The “accept” action causes the packet to leave the mangle chain, thereby
not marking it and allowing that traffic to use the main routing table
/ ip firewall mangle
add chain=prerouting dst-address=10.111.0.0/24 action=accept in-interface=LAN
add chain=prerouting dst-address=10.112.0.0/24 action=accept in-interface=LAN

DONT FORGET ON THE MANGLE PCC RULES TO ADD
connection-mark=no-mark before the new-connection-mark=WAN1_conn

I would do this for the initial input chain rule when first assigning new-connection-mark.

anav · March 7, 2019, 5:46pm

@solar77
I found Steve Dischers PCC presentation and its very similar to the MUM one where passthrough=yes is used everywhere… arggg different from the WIKI.

But what I found interesting in that in Steves presentation and the WIKI they both OMITTED the input chain rules that the MUM presentation has (and the OP).
/ip firewall mangle
add action=mark-connection chain=input in-interface=WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
etc…

(by the way I think there should be a connection-mark=no-connection-mark added to the above rules if these rules are even valid/useful…)

What is the purpose of these rules, if THEY ARE NOT USED by either the wiki or Steve Discher???
It has been nagging on my why we care about traffic headed to the router and not across the router???
Can we remove these rules and are they getting in the way???

OKAY Steve has these rules and I see them now on the WIKI, my eyes are glazed over - which make more sense to me PREROUTING RULES.
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no
in-interface=ether1 new-connection-mark=WAN1 passthrough=yes

SO what is possibly dead wrong from the MUM2019 presentation is the use of the input chain, vice the use of the prerouting chain..
I still need it explained to me why we need to mark this traffic what is the use case???

solar77 · March 7, 2019, 6:57pm

I had the input chain question this morning, ha ha !
what I did was create a rule on my own router, on prerouting chian, in-interface is WAN, which I thought won’t have any traffic. but it did,
as all traffic heading to my WAN ip, it get translated (NATed) into my LAN IP. which then get marked with the correct WAN connection mark so the reply to the internet gets out through the correct gateway.

I don’t know why they don’t just use the input chain. suspect if NAT is involved it has to go through routing decision and it involves prerouting chain. but will check the packet flow again tomorrow .

enough time been spent on reading up things which is great but I want to do a lap test , somehow, to see for my self.

back to the OP’s original issue, highly suspect it’s the routing part needs to be changed.

anav · March 7, 2019, 7:02pm

I think its the wan incoming interface rules…
Using steve dischers MUM presentation on same subject.

a. he matches up with the wiki in that the input chain is NOT used, its properly the PRE-ROUTING chain.
b. he uses passthrough=yes up the ying yang.

So for the OP suggest change all wan interface rule chains from input to pre-routing.
keep your current passthrough rules in effect
See what happens.

plisken · March 7, 2019, 8:09pm

I have try the tutorial from Steve but it’s don’t work also.
Thank you to all to find a solution.
I will keep you informed

solar77 · March 8, 2019, 1:33pm

Right, I’ve done it myself. Its working and here is what I did:

my set up is not a real multiple WAN but 3 virtual wlan interface ( AP Bridge modem, broadcasting 3 SSID) from Mikrotik hAP, wihch has Internet access.
Each SSID has it’s own network: 10.10.1.0/24, 10.10.2.0/24, 10.10.3.0/24
Tested each SSID has internet access.

then use another Mikrotik, with 3 virtual wlan interface (Station), connecting to each of these 3 SSID
I did not use DHCP client but added IP to each interface:

/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=10.10.1.250/24 interface=wlan2 network=10.10.1.0
add address=10.10.2.250/24 interface=wlan3 network=10.10.2.0
add address=10.10.3.250/24 interface=wlan4 network=10.10.3.0

then the mangle part I have:

/ip firewall mangle
add action=accept chain=prerouting disabled=yes dst-address=10.10.1.0/24 in-interface=bridge
add action=accept chain=prerouting disabled=yes dst-address=10.10.2.0/24 in-interface=bridge
add action=accept chain=prerouting disabled=yes dst-address=10.10.3.0/24 in-interface=bridge
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan2 new-connection-mark=wan1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan3 new-connection-mark=wan2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan4 log=yes new-connection-mark=wan3_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=wan1_conn passthrough=yes per-connection-classifier=both-addresses:3/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=wan3_conn passthrough=yes per-connection-classifier=both-addresses:3/2
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=wan2_conn passthrough=yes per-connection-classifier=both-addresses:3/1
add action=mark-routing chain=prerouting connection-mark=wan1_conn in-interface=bridge new-routing-mark=wan1_mark passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan2_conn in-interface=bridge new-routing-mark=wan2_mark passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan3_conn in-interface=bridge new-routing-mark=wan3_mark passthrough=no
add action=mark-routing chain=output connection-mark=wan1_conn disabled=yes new-routing-mark=wan1_mark passthrough=yes
add action=mark-routing chain=output connection-mark=wan2_conn disabled=yes new-routing-mark=wan2_mark passthrough=yes
add action=mark-routing chain=output connection-mark=wan3_conn disabled=yes new-routing-mark=wan3_mark passthrough=yes

I had all the connection_mark rules on prerouting chain, in-inerface-wan1 etc and the output chain but for testing, I’ve disabled them for now and it still works so far. I think they are for traffic intitialised from the Internet and also for DNS out of the router so will play with them later on.

NAT part, nothing special here

/ip firewall nat
add action=masquerade chain=srcnat out-interface=wlan2
add action=masquerade chain=srcnat out-interface=wlan3
add action=masquerade chain=srcnat out-interface=wlan4

Route:

/ip route
add distance=1 gateway=10.10.1.1 routing-mark=wan1_mark
add distance=1 gateway=10.10.2.1 routing-mark=wan2_mark
add distance=1 gateway=10.10.3.1 routing-mark=wan3_mark
add check-gateway=ping comment="fail-over route" disabled=yes distance=1 gateway=10.10.1.1
add comment="EMCP test" disabled=yes distance=1 gateway=10.10.3.1,10.10.1.1,10.10.2.1

Nnotes and tests:

Note 1 the disabled rule for ECMP test, I setup ECMP first to make sure all three gateway works and you can indeed load balance between them.
Note 2: I have to enable the last fail-over rule otherwise the router itself won’t have Internet access. so it’s outgoing traffic is not marked. I will have to test more on that.
Note 3, I used passthrough=no on the mark routing rules. I think there is no need to pass it on anyway.
However, so far all three connection works, all mangle rules counters goes up, I can see connections with each mark in IP - firewall - connections
further tested watching youtube from two computers each grabed one connection.
It’s been few hours and traffic status on each wan connection is increasing nicely. so I think it’s working for me.
Note 4, if I change the connection marking rules to passthrough=no, I have no internet access. I think this is because the mark routing did not get applied to these connection-marked traffic .
Note 5: It was not working initially because I missed the in-interface=bridge option in the connection marking rules

add action=mark-routing chain=prerouting connection-mark=wan1_conn in-interface=bridge new-routing-mark=wan1_mark passthrough=no

I guess this re-marks the return traffic from wan interface and it just won’t reach the originating IP. that’s only a guess.

There are few people on the forum struggling with PCC at the moment so hope this helps .

solar77 · March 8, 2019, 1:42pm

Note 6, when winbox into the uplink router (Router 1 broadcasting 3 SSID), it used wan2, when I open 2nd winbox session, it still use the same wan2, so the bandwidth doubles,
then I opened 3rd, 4th winbox, they all use the same wan2, with increasing bandwidth flowing.

this shows the both address option in the PCC works, as long as the source IP and dst-ip are the same, it uses the same wan gateway.

anav · March 8, 2019, 3:14pm

SOLAR, both the WIKI and Steve Discher show that rule as a prerouting chain BUT IN_INTERFACE=WAN ???
The MUM2019 presentation shows that as input chain BUT IN_INTERFACE=WAN.

Can i surmize that you are using bridge because you are simply simulating wan input???

I like your logic. ALL rules with marking connections are marked so that something happens to them later in mangling (ie routing) and thus require passthrough=yes, whereas once route marked there is no further mangling involved.