RB750Gr3 3xPPPoE PCC Load Balancing breaking webpages

RouterOS General
1

I am fairly new towards the world of RouterOS. Recently I bought a RB750Gr3 to be used as a load-balancer across 3x 100M PPPoE lines. Following the guide in Mikrothik’s wiki page, I came up with the following configurations:
/ip firewall mangle
add chain=input in-interface=pppoe-out_1 action=mark-connection new-connection-mark=ISP1_conn
add chain=input in-interface=pppoe-out_2 action=mark-connection new-connection-mark=ISP2_conn
add chain=input in-interface=pppoe-out_3 action=mark-connection new-connection-mark=ISP3_conn

add chain=output connection-mark=ISP1_conn action=mark-routing new-routing-mark=to_ISP1
add chain=output connection-mark=ISP2_conn action=mark-routing new-routing-mark=to_ISP2
add chain=output connection-mark=ISP3_conn action=mark-routing new-routing-mark=to_ISP3

add chain=prerouting dst-address=pppoe-out_1 action=accept in-interface=ether4
add chain=prerouting dst-address=pppoe-out_2 action=accept in-interface=ether4
add chain=prerouting dst-address=pppoe-out_3 action=accept in-interface=ether4

add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/0 action=mark-connection new-connection-mark=ISP1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/1 action=mark-connection new-connection-mark=ISP2_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/2 action=mark-connection new-connection-mark=ISP3_conn passthrough=yes
add chain=prerouting connection-mark=ISP1_conn in-interface=ether4 action=mark-routing new-routing-mark=to_ISP1
add chain=prerouting connection-mark=ISP2_conn in-interface=ether4 action=mark-routing new-routing-mark=to_ISP2
add chain=prerouting connection-mark=ISP3_conn in-interface=ether4 action=mark-routing new-routing-mark=to_ISP3

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out_1 routing-mark=to_ISP1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out_2 routing-mark=to_ISP2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out_3 routing-mark=to_ISP3 check-gateway=ping

add dst-address=0.0.0.0/0 gateway=pppoe-out_1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out_2 distance=2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out_3 distance=3 check-gateway=ping

/ip firewall nat
add chain=srcnat out-interface=pppoe-out_1 action=masquerade
add chain=srcnat out-interface=pppoe-out_2 action=masquerade
add chain=srcnat out-interface=pppoe-out_3 action=masqueradeBasically ether1, 2 and 3 is configured for PPPoE and ether4 and 5 for LAN with ether5 slave to ether4. Both ether4 and 5 is serving DHCP addresses of 10.1.0.0/24. This RB750Gr3 do not handle all the client routes as it’s sole purpose is to load balance between 3 ISPs. Connecting to RB750Gr3’s ether4 and 5 is a Fortigate 100D firewall. The firewall is responsible of all the clients DHCP, routing and policies. Firewall has 2 WAN ports which obtains 2 addresses from RB750Gr3 dynamically.

With all of these configured, suddenly all my client’s networks slows down to a crawl. DNS takes minutes to resolve, webpages takes extremely longer than usual to load and sometimes wouldn’t load at all. Yet, all of these issues are solved when there is only 1 active ISP but when all 3 of them running, everything breaks. What am I doing wrong here? Image below shows roughly how and where RB750Gr3 belongs.

2

Please state also which firmware your using in the RB750Gr3.

3

Looks like you’re re-marking your connections.

add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/0 action=mark-connection new-connection-mark=ISP1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/1 action=mark-connection new-connection-mark=ISP2_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/2 action=mark-connection new-connection-mark=ISP3_conn passthrough=yes

Suggestion: Add connection-mark=no-mark to these entries

4

The firmware version would be 6.39.1



I am not sure on what do you mean by I’ve remarked my connections but base on your suggestions, do you mean something like this?

add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/0 action=mark-connection new-connection-mark=no-mark passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/1 action=mark-connection new-connection-mark=no-mark passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/2 action=mark-connection new-connection-mark=no-mark passthrough=yes
5

Your rule potentially overwrites the connection mark (e.g. incoming connection from isp2 gets connection mark isp2, but PCC could change it into isp3).

Add connection-mark=no-mark to these entries means:

Old value:

add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/0 action=mark-connection new-connection-mark=ISP1_conn passthrough=yes

New value:

add chain=prerouting dst-address-type=!local in-interface=ether4 per-connection-classifier=both-addresses-and-ports:3/0 action=mark-connection new-connection-mark=ISP1_conn passthrough=yes connection-mark=no-mark
6

I’ve applied this script on site and it works okay:

/ip firewall address-list
add address=www.example.com list=via-ISP1
add address=192.168.1.0/24 list=via-ISP1
add address=192.168.2.0/24 list=via-ISP2
add address=192.168.3.0/24 list=via-ISP3
/ip firewall mangle
add action=mark-connection chain=input comment="IF1 -> CM1" in-interface=\
    pppoe-out_1 new-connection-mark=ISP1_conn passthrough=yes
add action=mark-connection chain=input comment="IF2 -> CM2" in-interface=\
    pppoe-out_2 new-connection-mark=ISP2_conn passthrough=yes
add action=mark-connection chain=input comment="IF3 -> CM3" in-interface=\
    pppoe-out_3 new-connection-mark=ISP3_conn passthrough=yes
add action=mark-connection chain=output comment="AL1 -> CM1" connection-mark=\
    no-mark dst-address-list=via-ISP1 new-connection-mark=ISP1_conn \
    passthrough=yes
add action=mark-connection chain=output comment="AL2 -> CM2" connection-mark=\
    no-mark dst-address-list=via-ISP2 new-connection-mark=ISP2_conn \
    passthrough=yes
add action=mark-connection chain=output comment="AL3 -> CM3" connection-mark=\
    no-mark dst-address-list=via-ISP3 new-connection-mark=ISP3_conn \
    passthrough=yes
add action=mark-routing chain=output comment="CM1 -> RM1" connection-mark=\
    ISP1_conn new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=output comment="CM2 -> RM2" connection-mark=\
    ISP2_conn new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=output comment="CM3 -> RM3" connection-mark=\
    ISP3_conn new-routing-mark=to_ISP3 passthrough=no
add action=mark-connection chain=prerouting comment="AL1 -> CM1" \
    connection-mark=no-mark dst-address-list=via-ISP1 new-connection-mark=\
    ISP1_conn passthrough=yes
add action=mark-connection chain=prerouting comment="AL2 -> CM2" \
    connection-mark=no-mark dst-address-list=via-ISP2 new-connection-mark=\
    ISP2_conn passthrough=yes
add action=mark-connection chain=prerouting comment="AL3 -> CM3" \
    connection-mark=no-mark dst-address-list=via-ISP3 new-connection-mark=\
    ISP3_conn passthrough=yes
add action=mark-connection chain=prerouting comment="PCC0 -> CM1" \
    connection-mark=no-mark dst-address-type=!local in-interface=ether4 \
    new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=\
    both-addresses-and-ports:3/0
add action=mark-connection chain=prerouting comment="PCC1 -> CM2" \
    connection-mark=no-mark dst-address-type=!local in-interface=ether4 \
    new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=\
    both-addresses-and-ports:3/1
add action=mark-connection chain=prerouting comment="PCC2 -> CM3" \
    connection-mark=no-mark dst-address-type=!local in-interface=ether4 \
    new-connection-mark=ISP3_conn passthrough=yes per-connection-classifier=\
    both-addresses-and-ports:3/2
add action=mark-routing chain=prerouting comment="CM1 -> RM1" connection-mark=\
    ISP1_conn in-interface=ether4 new-routing-mark=to_ISP1 passthrough=\
    no
add action=mark-routing chain=prerouting comment="CM2 -> RM2" connection-mark=\
    ISP2_conn in-interface=ether4 new-routing-mark=to_ISP2 passthrough=\
    no
add action=mark-routing chain=prerouting comment="CM3 -> RM3" connection-mark=\
    ISP3_conn in-interface=ether4 new-routing-mark=to_ISP3 passthrough=\
    no
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out_1
add action=masquerade chain=srcnat out-interface=pppoe-out_2
add action=masquerade chain=srcnat out-interface=pppoe-out_3
/ip route
add distance=1 gateway=192.168.1.254 routing-mark=to_ISP1
add distance=1 gateway=192.168.2.254 routing-mark=to_ISP2
add distance=1 gateway=192.168.3.254 routing-mark=to_ISP3

Remarks:

  • I use address lists (via-ISP1/2/3) to redirect the pppoe-out_1/2/3 address ranges + some specific sites (requiring specific ip) to a particular interface
  • I’ve disabled the check gateway setting - I don’t like pinging the direct gateway for network failure detection and assume I get more false positives than actual disruptions so I’d rather disable it
  • I’ve added add-default-route on the uplink - so no need to define default gateways in main routing table
  • Note that traffic from the router (output chain) is not pcc’d - will always leave default interface (or isp2/3 if defined in address list)
  • I’ve added comments (A)ddress(L)ist, (R)outing(Mark), (C)onnection(Mark), PCC for easier identification
  • The passthrough=no on the (C)onnection(M)ark entries suddenly made traffic flow, not sure why
7

You’re the man my friend. I’ll give it a try. Can’t afford to remote now just in case something goes wrong. Once again, many thanks for sharing your precious suggestions and knowledge.

8

Hey there my friend, I’ve tried all your suggestions and still the problem persist. I’ve ran out of ways within my knowledge on how am I supposed to code for PCC load balancing. Or is it because I have a firewall after RB750Gr3? Will that be the issue? Because at this moment only 2 IP addresses has been leased from RB750Gr3 to Fortigate’s WAN1 and WAN2.

9

You could rule that out by connecting a laptop directly to the MT. Not sure why this could be a problem if ether4 and ether5 are bridged, but worth the effort.

10

Yeap. You’re right. It has nothing to do with it. I went on to the detailed logging in firewall and traffic control. It seems that it had problem with DNS resolving and from which connection the result should be returned to. I’m using Mr. G’s DNS by the way.

11

You could provider-specific DNS servers to their associated address list:

/ip firewall address-list
add address=6.7.8.9 list=via-ISP1
add address=1.2.3.4 list=via-ISP2
12

@kennyyap92 I have the same. May you help me?