Load-Balancing (PCC) with two PPPoE Clients (Two different home ISPs) not working

RouterOS Beginner Basics
1

So
pppoe-out1 = ISP1, dynamic IP
pppoe-out2 = ISP2, dynamic IP
Both are part of the “WAN” interface

LAN=bridge, single subnet.

Failover is working 100% without any issues.

I followed a combination of the guide here: https://help.mikrotik.com/docs/display/ROS/Firewall+Marking#heading-Example3PCC
And here (for TCP, HTTPS): https://www.qualityology.com/tech/easy-dual-wan-load-balancing-with-mikrotik-routeros-pcc/

So, what happens is, the connections simply continue to use pppoe-out1 without ever using pppoe-out2 unless pppoe-out1 failover to pppoe-out2. Basically, load balancing isn’t working at all.

I think I figured out the problem, inside IP>Route, the route to pppoe-out2 is not active when pppoe-out1 is active/working. This is only logical, of course.
pppoe-out1 has route distance = 1
pppoe-out2 has route distance = 2

But it simply won’t push traffic through ISP2 unless ISP1 is down. Any ideas?

This is what I tried:

/ip firewall mangle
add action=accept chain=prerouting dst-address-list=!not_in_internet in-interface=bridge

/ip firewall mangle
add chain=prerouting in-interface=pppoe-out1 connection-mark=no-mark action=mark-connection new-connection-mark=ISP1_conn
add chain=prerouting in-interface=pppoe-out2 connection-mark=no-mark action=mark-connection new-connection-mark=ISP2_conn

/ip firewall mangle
#ForHTTPS
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet protocol=tcp dst-port=22,80,443 per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=ISP1_conn
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet protocol=tcp dst-port=22,80,443 per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=ISP2_conn

add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet per-connection-classifier=both-addresses-and-ports:2/0 action=mark-connection new-connection-mark=ISP1_conn
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet per-connection-classifier=both-addresses-and-ports:2/1 action=mark-connection new-connection-mark=ISP2_conn

/ip firewall mangle
add chain=prerouting connection-mark=ISP1_conn in-interface=bridge action=mark-routing new-routing-mark=to_ISP1
add chain=prerouting connection-mark=ISP2_conn in-interface=bridge action=mark-routing new-routing-mark=to_ISP2
add chain=output connection-mark=ISP1_conn action=mark-routing new-routing-mark=to_ISP1
add chain=output connection-mark=ISP2_conn action=mark-routing new-routing-mark=to_ISP2

/ip route
/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out2 distance=2 check-gateway=ping

add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_ISP1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-mark=to_ISP2 check-gateway=ping

My config dump

/interface bridge
add admin-mac=###sensitive### auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether5 ] poe-out=off
/interface pppoe-client
add disabled=no interface=ether1 keepalive-timeout=5 name=pppoe-out1 user=###sensitive###
add allow=pap disabled=no interface=ether2 keepalive-timeout=5 name=pppoe-out2 user=###sensitive###
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add comment=ONT name=ExternalLAN
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge lease-time=1d name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set accept-source-route=yes
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=ONT interface=ether1 list=ExternalLAN
add interface=pppoe-out1 list=WAN
add interface=pppoe-out2 list=WAN
add comment=ONU interface=ether2 list=ExternalLAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
/ip dhcp-client
add add-default-route=no comment=defconf disabled=no interface=ether1 use-peer-dns=no use-peer-ntp=no
add add-default-route=no disabled=no interface=ether2 use-peer-dns=no use-peer-ntp=no
/ip dhcp-server lease
add address=192.168.88.103 client-id=###sensitive### mac-address=###sensitive### server=defconf
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes query-server-timeout=100ms query-total-timeout=5s servers=\
    176.103.130.130,206.189.142.179,8.8.8.8,1.1.1.1,9.9.9.10,2a00:5a60::ad1:ff,2a00:5a60::ad2:ff,2001:4860:4860::8888,2001:4860:4860::8844,2606:4700:4700::1111,2606:4700:4700::1001 \
    use-doh-server=https://dns.google/dns-query verify-doh-cert=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
add address=8.8.8.8 name=dns.google type=A
add address=8.8.4.4 name=dns.google type=A
add address=2001:4860:4860::8888 name=dns.google type=AAAA
add address=2001:4860:4860::8844 name=dns.google type=AAAA
/ip firewall address-list
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=not_in_internet
add address=1.1.1.1 comment=DoH list=DoH
add address=1.0.0.1 comment=DoH list=DoH
add address=8.8.8.8 comment=DoH list=DoH
add address=8.8.4.4 comment=DoH list=DoH
add address=192.168.88.0/24 list=allowed_to_router
add address=192.168.1.0/24 list=allowed_to_router
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="Accept access to router" src-address-list=allowed_to_router
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface=!bridge in-interface-list=!ExternalLAN log-prefix=AllNotFromLAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related disabled=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=accept chain=forward comment="Xbox NAT" connection-nat-state=dstnat in-interface-list=WAN
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="Drop tries to reach not public addresses from LAN" dst-address-list=not_in_internet in-interface=bridge out-interface-list=WAN
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=jump chain=forward comment="jump to ICMP filters" jump-target=icmp protocol=icmp
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface-list=WAN src-address-list=not_in_internet
add action=drop chain=forward comment="Drop packets from LAN that do not have LAN IP" in-interface=bridge src-address=!192.168.88.0/24
add action=accept chain=icmp comment="echo reply" icmp-options=0:0 protocol=icmp
add action=accept chain=icmp comment="net unreachable" icmp-options=3:0 protocol=icmp
add action=accept chain=icmp comment="host unreachable" icmp-options=3:1 protocol=icmp
add action=accept chain=icmp comment="host unreachable fragmentation required" icmp-options=3:4 protocol=icmp
add action=accept chain=icmp comment="allow echo request" icmp-options=8:0 protocol=icmp
add action=accept chain=icmp comment="allow time exceed" icmp-options=11:0 protocol=icmp
add action=accept chain=icmp comment="allow parameter bad" icmp-options=12:0 protocol=icmp
add action=drop chain=icmp comment="deny all other types"
add action=jump chain=forward connection-state=new in-interface-list=WAN jump-target=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddos-target address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddos-attackers address-list-timeout=10m chain=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s protocol=tcp tcp-flags=syn,ack
/ip firewall mangle
###I removed the PCC rules and the custom routes to clear up this config dump###
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=pppoe-out1
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=pppoe-out2
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface-list=ExternalLAN
/ip firewall raw
add action=drop chain=prerouting dst-address-list=dddos-targets src-address-list=ddos-attackers
/ip route
add check-gateway=ping distance=1 gateway=pppoe-out1
add check-gateway=ping distance=2 gateway=pppoe-out2
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=bridge type=internal
add interface=pppoe-out1 type=external
add disabled=yes type=internal
add interface=pppoe-out2 type=external
2

The main routing table and associated route distances apply to any packets which do not have a routing mark applied. Check the counters on
/ip firewall mangle
add chain=prerouting connection-mark=ISP1_conn in-interface=bridge action=mark-routing new-routing-mark=to_ISP1
add chain=prerouting connection-mark=ISP2_conn in-interface=bridge action=mark-routing new-routing-mark=to_ISP2

I suspect you will find they are not incrementing as your LAN traffic does not match in-interface=bridge.

3

Hi, turns out, it was something weird, I deleted this mangle rule and PCC is working as expected.

/ip firewall mangle
add action=accept chain=prerouting dst-address-list=!not_in_internet in-interface=bridge

HTTPS/SSL do not break, getting aggregated bandwidth in speedtests. And yes, LAN=bridge in my config.

Any ideas why that specific ruled created a problem? Even though I followed the official guide?

4

Time Bump

I still don’t how the first two rules as per the guides would work with dynamic IPs via PPPoE, they are for static IPs since the chain is prerouting.

Does anybody know a solution?

/ip firewall mangle
add chain=prerouting dst-address=###dynamicIP###  action=accept in-interface=bridge
add chain=prerouting dst-address=###dynamicIP###  action=accept in-interface=bridge

Source: https://help.mikrotik.com/docs/display/ROS/Firewall+Marking#heading-Mangle

Also in traceroutes with my PCC setup, the router simply “times out” in the first hop and never shows up. When I disable the PCC mangle rules, it shows up again, any ideas?