Dual WAN (PPPoE, dynamic IP) PCC load balancing to bridge interface

RouterOS Beginner Basics
1

Hello, I have just got to the Mikrotik party and am getting used to the drinks (RB2011UiAS). My knowledge about networking is bits and pieces I’ve picked up over the years, I’d be grateful for your corrections. What I want to achieve is:

  • share two ISP links concurrently, both are dynamic IPs but I can make them static if it simplifies things
  • allow incoming DNAT to a couple of hosts on the LAN through either ISP link
  • VPN access into the LAN

Where I am currently:

  • created a PPPoE interface for each ISP, act-ge1 and airtel-fe9 below. Connections established, can reach WAN IPs through both links from the router.
  • one of the PPPoE clients sets a default route, the other does not
  • created a bridge with all the other interfaces in it
  • created two interface lists LAN (with the bridge), and WAN (with physical ge1 and fe9 ifaces, not the PPPoE interfaces act-ge1 and airtel-fe9)
  • /ip firewall
    nat add action=masquerade chain=srcnat comment=“defconf: masquerade”
    ipsec-policy=out,none out-interface-list=WAN

With this setup, traffic seems to be flowing through one LAN and things look ok. Now for PCC load-balancing. After reading a few forum posts, I came up with the following configuration:

/ip firewall mangle
add action=accept chain=prerouting disabled=no in-interface=act-ge1
add action=accept chain=prerouting disabled=no in-interface=airtel-fe9
add chain=prerouting in-interface=act-ge1 connection-mark=no-mark action=mark-connection new-connection-mark=act
add chain=prerouting in-interface=airtel-fe9 connection-mark=no-mark action=mark-connection new-connection-mark=airtel

/ip firewall mangle
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet protocol=tcp dst-port=22,80,443 per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=act
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet protocol=tcp dst-port=22,80,443 per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=airtel
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet per-connection-classifier=both-addresses-and-ports:2/0 action=mark-connection new-connection-mark=act
add chain=prerouting in-interface=bridge connection-mark=no-mark dst-address-list=!not_in_internet per-connection-classifier=both-addresses-and-ports:2/1 action=mark-connection new-connection-mark=airtel

/ip firewall mangle
add chain=prerouting connection-mark=act in-interface=bridge action=mark-routing new-routing-mark=to_act
add chain=prerouting connection-mark=airtel in-interface=bridge action=mark-routing new-routing-mark=to_airtel
add chain=output connection-mark=act action=mark-routing new-routing-mark=to_act
add chain=output connection-mark=airtel action=mark-routing new-routing-mark=to_airtel

/ip route
add dst-address=0.0.0.0/0 gateway=ether1-act distance=1 routing-mark=to_act check-gateway=ping
add dst-address=0.0.0.0/0 gateway=ether9-airtel distance=2 routing-mark=to_airtel check-gateway=ping

The moment I enter the first route add dst-address=0.0.0.0/0 gateway=ether1-act distance=1 routing-mark=to_act check-gateway=ping something terrible happens to the router and I have to reset all configuration via the nifty LCD display to get back to the router. Just plan reboots seem to do nothing, the router does not provide a DHCP lease and setting up a static IP on my laptop plugged directly into one of the bridge interfaces cannot estabish connectibity. Clearly I’m doing something heinous here but I can’t figure out what.

I also have a couple of questions:

  • Most of the posts start with adding LAN addresses to the WAN interfaces. I am assuming that I don’t need to do this as the WAN interfaces are configured by PPPoE.
  • Are the PPPoE interfaces first class interfaces? Can I use it as I have done above?
  • Can I use the WAN interface list in the mangle rules? Something like: /ip firewall add action=accept chain=prerouting disabled=no in-interface-list=WAN
2

You’re Indian, I’m Indian, so I can be of more help in the context of Indian ISPs.

Did you bridge ACT’s ONT/Router if any?
Did you bridge Airtel’s ONT/Router if any?
Is your Tik handling both ISPs completely? That is you have two PPPoE clients?

If above is yes then you got it 99% done. There are some issues with the re-branded ONTs Indian ISPs give you. They can’t auto-negotiate MTU of their ethernet interfaces. More information this here (applicable to any ISPs in India for the most part): https://www.reddit.com/r/bsnl/comments/ht37q4/guide_for_bsnl_ftth/

For the load balancing config, it’s simple:

#First add RFC6890 reserved private subnets to exclude from PCC/Load balancing
/ip firewall address
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=not_in_internet
add address=255.255.255.255 comment=RFC6890 list=not_in_internet

/ip firewall mangle
#Marks incoming connections to their correct WAN interface
#Passthrough = No to save CPU cycles
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no

#Split traffic 50% to each ISP

#HTTPS uses PCC both addresses 
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses:2/0 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses:2/1 port=80,443 protocol=tcp

#QUIC uses PCC both addresses
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses:2/0 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses:2/1 port=80,443 protocol=udp

#Send other traffic to PCC's both addresses and ports to get aggregated bandwidth
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0 
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1

#Sends marked connections to routing table
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no

#Sends incoming WAN traffic to destined WAN interface
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 out-interface=pppoe-out1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 out-interface=pppoe-out2 passthrough=no

#Finally add routes for marked traffic
add check-gateway=ping comment="Load Balancing Route to ISP 1" distance=1 gateway=pppoe-out1 routing-mark=to_ISP1
add check-gateway=ping comment="Load Balancing Route to ISP 2" distance=1 gateway=pppoe-out1 routing-mark=to_ISP2
3

I’m suprised you get any internet connectivity - the WAN interfaces are the PPPoE interfaces, not the ethernet ports which are merely transporting PPPoE packets. Update the WAN interface list accordingly, and reference the PPPoE interfaces instead of the ethernet interfaces in your rules.

You do not have to add IP addresses to the PPPoE interfaces as they are negotiated during the PPP setup. As you say many examples add addresses, but they are typically using IPoE.

4

Tdw stole my thunder. Best to put all four interfaces (wan affiliated interfaces - the pppoe ones are the ‘active’ ones) for the interface-list=WAN.

5

Thanks for the warm welcome. This morning when I posted I thought I would never wrap my head around this simple concept and this Mikrotik purchase was a mistake. But I ran into this sensational presentation by Tomas Kirnak at https://www.youtube.com/watch?v=67Dna_ffCvc and things just clicked. I gotta say that it is a pleasure to work with this RB after years of “open-source” firmware on janky hardware, I finally have the home network I deserve.

I implemented bandwidth based load balancing that Tomas described in this talk using Traffic Monitoring. This seemed a better fit than PCC as my links are 1Gbps fibre and 16Mbps vDSL. Incoming worked out of the box. I still have some small niggles to work out but this looks promising and provides various knobs to adjust the utilisation and load.

6

Kewl, i always toyed with the idea of frigging with the numbers on PCC for that type of load balancing but this seems more intuitive.

7

I noticed that “tiktube” is mentioned often in talks but tiktube.com is not reachable. Were those videos moved to YouTube or something? TikTube on YouTube is not what I expected.

@Darknate, I am planning on using the SFP interface to eliminate the ACT ONT. The ONT is labelled Acton 1000W2A(SM-10) and some googling suggests that this is a single-mode 1310nm device. I was unable to find any documentation that says so definitely. Do you have any advice on where to buy such a module? I did find sLIFE (Cisco,MIKROTIK,ALCATEL Compatible) 1.25G SFP Single Mode Dual Fiber TRANSCEIVERS, 1310nm, LC, DDM-20Km but I haven’t ordered it yet.

8

PON is “Passive Optical Network” which means it has an extra layer of encryption and TDM etc. PON will ALWAYS need an ONT/ONU for said purposes. Not sure where people get this idea that SFP works with PON on the user’s termination point.

9

It depends on the optical delivery and type of SFP. For GPON a dumb SFP will not work, but an active SFP which contains ONT functionality and presents a 1000Base-X electrical interface should. There are also some ISPs who use point-to-point 1000Base-LX or 1000Base-BX optics rather than GPON.

10

A true SFP/SFP+ module by definition and design is not going to work with PON. Anything else that does work is an ONT in the form of an SFP, not an actual SFP in the truest sense.

The OP is Indian, down here we use EPON (LOID Tables) and GPON (VLAN tagging) actively.

11

Thanks tdw.

12

#Sends incoming WAN traffic to destined WAN interface
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 out-interface=pppoe-out1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 out-interface=pppoe-out2 passthrough=no

is this ture ? in output chain put out interface ??
output not count except one when put out interface in rules

13

Yes, it is “true”.

http://forum.mikrotik.com/t/need-some-explanation-regarding-pcc-load-balancing-mangle-rules/143448/33