2 connections On RB5009

RouterOS General
1

Hello,

I have a MikroTik RB5009 running RouterOS 7, and I am trying to combine two PPPoE connections using PCC load balancing to achieve combined download and upload speeds.

I previously used the exact same scenario on an RB951Ui with RouterOS 6.xx, and it worked perfectly.
Now with RouterOS 7, the configuration does not work even though I kept the same logic.

Below is my current configuration.

/interface pppoe-client print detail
0 R name="pppoe-out1" interface=ether1 user="xx" password="xx"
add-default-route=no use-peer-dns=yes allow=pap,chap,mschap1,mschap2

1 R name="pppoe-out2" interface=ether2 user="xx" password="xx"
add-default-route=no use-peer-dns=yes allow=pap,chap,mschap1,mschap2

/ip firewall mangle print detail
0 D chain=prerouting action=passthrough
1 D chain=forward action=passthrough
2 D chain=postrouting action=passthrough

4 chain=prerouting action=mark-connection new-connection-mark=WAN1_conn
dst-address-type=!local connection-mark=no-mark in-interface=bridge
per-connection-classifier=both-addresses:2/0

5 chain=prerouting action=mark-connection new-connection-mark=WAN2_conn
dst-address-type=!local connection-mark=no-mark in-interface=bridge
per-connection-classifier=both-addresses:2/1

6 chain=prerouting action=mark-routing new-routing-mark=TO_WAN1
connection-mark=WAN1_conn in-interface=bridge

7 chain=prerouting action=mark-routing new-routing-mark=TO_WAN2
connection-mark=WAN2_conn in-interface=bridge

/ip route print detail

1 As dst-address=0.0.0.0/0 routing-table=main gateway=pppoe-out1 distance=1
2 s dst-address=0.0.0.0/0 routing-table=main gateway=pppoe-out1 distance=2

/ip firewall nat print detail
0 chain=srcnat action=masquerade src-address=192.168.88.0/24 out-interface=pppoe-out1
1 chain=srcnat action=masquerade src-address=192.168.88.0/24 out-interface=pppoe-out2

With RouterOS 7, this configuration no longer provides combined download/upload speeds.
It appears that PCC routing is not working as it did in RouterOS 6, but I’m not sure whether:

  • My configuration is wrong, or

  • RouterOS 7 no longer supports this method

Would appreciate guidance from anyone who successfully implemented PCC with dual PPPoE on RouterOS 7, or confirmation if this feature behaves differently now..

Thank You

2

At first sight you are applying the mark/routing connections with mangle Wan1_conn and Wan2_conn and TO_WAN1/2, but then you have no mechanism to distinguish the marked packets and use the appropriate routes.
In 6 the routing marks were used in /ip route, in 7 separate routing tables (that need to be declared explicitly as fib) are used instead.
Example:

Additionally in 7 It Is often better/easier to use when possible routing rules instead of mangle.

3

Hi,

A configuration export (with appropriate redactions) would be much better.

My current GUESSES:

You don't have any routing entries for the TO_WAN1 and TO_WAN2 routing marks.

You may not have disabled fasttrack forwarding of your marked packets.
(Simplest short term option is to disable it completely)

4

Yup, we need the complete picture to answer in an accurate and thorough way.

/export file=anynameyouwish ( minus router serial number, any publicWAN IP information, keys, dhcp lease lists

Without at least that, I rarely entertain replies that are guessing.

5

here my final updated configuration please red it carefully.

Note im using 2 pppoe connection from the same ISP no static ips.

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=accept chain=prerouting comment=Accept
add action=accept chain=prerouting
add action=accept chain=prerouting
add action=mark-connection chain=input comment=Input in-interface=ether1 new-connection-mark=ISP1
add action=mark-connection chain=input in-interface=ether2 new-connection-mark=ISP2
add action=mark-connection chain=prerouting comment=Mark-con connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1
add action=mark-routing chain=prerouting comment=Mark-Route connection-mark=ISP1 in-interface=bridge new-routing-mark=TO-ISP1
add action=mark-routing chain=prerouting connection-mark=ISP2 in-interface=bridge new-routing-mark=TO-ISP2
add action=mark-connection chain=prerouting comment=PCC connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=ISP1 per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=local in-interface=bridge new-connection-mark=ISP2 per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2
add action=mark-routing chain=output comment=Output connection-mark=ISP1 new-routing-mark=TO-ISP1
add action=mark-routing chain=output connection-mark=ISP2 new-routing-mark=TO-ISP2
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=pppoe-out1 src-address=192.168.88.0/24
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=pppoe-out2 src-address=192.168.88.0/24
add action=masquerade chain=srcnat disabled=yes

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-table=TO-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-table=TO-ISP2 scope=30 suppress-hw-offload=no target-scope=10

/interface pppoe-client
add disabled=no interface=ether1 name=pppoe-out1 service-name=xxx use-peer-dns=yes user=xxx
add disabled=no interface=ether2 name=pppoe-out2 service-name=xxx use-peer-dns=yes user=xxx
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=sfp-sfpplus1
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-out1 list=WAN
add interface=pppoe-out2 list=WAN
add comment=defconf interface=ether2 list=WAN
/interface ovpn-server server
add mac-address=FE:D0:70:81:70:86 name=ovpn-server1
6

I recommend using src-address-list=LAN instead or src-address=... in case you add VLANs or other LAN-like connections and for constancy with default configuration.

Since you're using PCC, you want to use failover in the main routing table, so the 2nd entry should use a distance=1. Otherwise, you're creating an ECMP route in main, which could have some side-effects, so easier if your using PCC to not mix it with ECMP (which the alternative way to load balance using routing table/rules).

8

will it works with ECMP method ? can you please provide a detailed steps for it ?

10

@gardaerk's concept is right, except the example has wrong attribute names for V7, this is invalid syntax:

For background, ECMP is simpler, since you can mostly* skip all firewall mangle rules, and possibly* do not need any route tables either. ECMP is simple since any route with same distance= value automatically gets load balanced.

In fact, by just by adding the two /interface/pppoe-clients, by default, you will have ECMP route (since default is use-default-gateway=yes and distance=1). So all of the mangle rules, routing tables, and routes are not need for ECMP. So it's just NAT rules you need, one for each pppoe-client interface.

One important distinction is ECMP is always equal (the E in ECMP), so traffic is 50%/50% with two connections, while in PCC you can define any kinda traffic splits you want so 75%/25% is possible with PCC. Your current config is using 2/1 and 2/0 for PCC, so it's 50/50%.

Another difference is ECMP load balancing is based only and src/dst address (by default), thus similar to PCC's both-address but OP's PCC approach uses both-addresses-and-ports — so that is a important difference between PCC and ECMP, since ECMP traffic will not be "as balanced" without using PCC "with ports". Now in recent RouterOS, you can use /ip/settings/set ipv4-multipath-hash-policy=l4 to make ECMP use -and-ports similar to PCC:

IPv4 Hash policy used for ECMP routing in /ip/settings menu
l3 -- layer-3 hashing of src IP, dst IP
l3-inner -- layer-3 hashing or inner layer-3 hashing if available
l4 -- layer-4 hashing of src IP, dst IP, IP protocol, src port, dst port

@gardlaerk adds what's called "recursive routing" (see Failover (WAN Backup) - RouterOS - MikroTik Documentation), which has the effect of checking the internet is actually up. But this is kinda separate from ECMP, and could be applied to the PCC approach too. But OP not currently doing this in PCC, so add confusion when explaining ECMP, since recursive routing is separate. And there is middle ground to check the ISP far-end gateway of PPPoE (while recursive routes can check any host on interface to "test" WAN connection is up). This approach adds the check-gateway-ping to the pppoe-client interface dynamically when the connection comes up using a "dynamic-in routing filter rule":

/routing settings set dynamic-in-chain=setcheckgw
/routing filter rule add chain=setcheckgw disabled=no rule="if ( afi ipv4 ) { set gw-check icmp; accept }"

This avoid needing to know about target-scope but is a less robust check on WAN available than recursive route, but it's automatic for any DHCP client or PPPoE. For example, if you added a third PPPoE interface it actually be "checked" by above rule by only adding a 3rd pppoe-client

*Finally I use "mostly" and "possibly" about route tables, since they too have a use with ECMP (and required for PCC), and I'd recommend keeping routing tables, one for each WAN, since if you do want to steer particular IPs/subnet to a particular WAN, you can use /routing/rule if you have the tables setup (see Policy Routing - RouterOS - MikroTik Documentation) or still use firewall rules to mark-routing to a table still, even with ECMP active in main. Also with ECMP you can run into a situation where chain=input traffic (WAN to router) might go out the other WAN than it came in, so often it's smart to add input rules to force traffic to always go out the WAN it came in on & you do need routing tables for this...

P.S. @gardlaerk, the length of this clarification is why focusing too much on "target scope clarity" can be bit too myopic in topic of multiwan. Whether "TS" or correctly target-scope is 11 or 12 or 15 really a few levels down from the overall approach someone want to take to multiwan. But agree with ya that the OP like should use "basic" check-gateway or "full" recursive routing if more advanced WAN failure detection is needed.