Have searched and debugged this one for the last few hours, so pretty sure i’m at the stage of missing something dead obvious and too deep to see it.
Quick rundown of the setup, Two wans come in on a QinQ VLAN, these are both terminated to their own PPPoE connections (Dynamic)
These are “Spark” and “VF”.
The LAN interface is “bridge”, obviously being a bridge interface.
Individually, these work perfect. Upstream transparent shaping also works perfectly.
Unfortunately, load balancing doesn’t seem as keen to just fall into place.
Currently what seems to happen is, the Routing mask isn’t being applied correctly.
I’ve confirmed this by disabling failover routes, instantly there is no data transfer at all on new connections.
The connections are being marked correctly, so i suspect i’ve done something wrong in the ip route or mark routing.
# dec/30/2018 02:09:07 by RouterOS 6.43.8
# software id =
#
# model = RouterBOARD 3011UiAS
# serial number =
/interface bridge
add admin-mac= auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=ether2 ] name=ether2-master speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] speed=100Mbps
set [ find default-name=ether6 ] name=ether6-master speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=ether8 ] speed=100Mbps
set [ find default-name=ether9 ] speed=100Mbps
set [ find default-name=ether10 ] speed=100Mbps
set [ find default-name=sfp1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full
/interface vlan
add interface=ether4 name=vlan110 use-service-tag=yes vlan-id=110
add interface=ether4 name=vlan210 use-service-tag=yes vlan-id=210
/interface pppoe-client
add add-default-route=yes interface=vlan210 name=Spark password=password user=user@spark.co.nz
/interface vlan
add interface=vlan110 name=vlan10-modem1 vlan-id=10
add disabled=yes interface=vlan210 name=vlan10-modem2 vlan-id=10
/interface pppoe-client
add add-default-route=yes interface=vlan10-modem1 name=VF password=password user=user@spark.co.nz
/interface list
add exclude=dynamic name=discover
add name=mactel
add name=mac-winbox
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=10.3.59.10-10.3.59.200
/ip dhcp-server
add address-pool=dhcp authoritative=after-2sec-delay disabled=no interface=bridge name=defconf
/interface list member
add interface=ether2-master list=discover
add interface=ether3 list=discover
add interface=ether4 list=discover
add interface=ether5 list=discover
add interface=sfp1 list=discover
add interface=ether6-master list=discover
add interface=ether7 list=discover
add interface=ether8 list=discover
add interface=ether9 list=discover
add interface=ether10 list=discover
add interface=bridge list=discover
add list=discover
add list=discover
add interface=bridge list=mactel
add interface=bridge list=mac-winbox
add interface=ether1 list=WAN
/ip address
add address=10.3.59.1/24 comment=defconf interface=ether2-master network=10.3.59.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.3.59.0/24 comment=defconf gateway=10.3.59.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8
/ip dns static
add address=192.168.88.1 name=router
/ip firewall filter
add action=accept chain=input dst-port=8291 protocol=tcp
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept established,related" connection-state=established,related
add action=drop chain=input comment="defconf: drop all from WAN" in-interface=Spark
add action=drop chain=input in-interface=VF
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related" connection-state=established,related
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface=Spark
add action=drop chain=forward in-interface=VF
/ip firewall mangle
add action=accept chain=prerouting in-interface=bridge
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=Spark new-connection-mark=SparkWAN passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=VF new-connection-mark=VFWAN passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=SparkWAN passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=VFWAN passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=SparkWAN in-interface=bridge new-routing-mark=SparkConnectionRoute passthrough=yes
add action=mark-routing chain=prerouting connection-mark=VFWAN in-interface=bridge new-routing-mark=VFConnectionRoute passthrough=yes
add action=mark-routing chain=output connection-mark=SparkWAN new-routing-mark=SparkConnectionRoute
add action=mark-routing chain=output connection-mark=VF new-routing-mark=VFConnectionRoute
/ip firewall nat
add action=masquerade chain=srcnat out-interface=Spark
add action=masquerade chain=srcnat out-interface=VF
/ip route
add check-gateway=ping distance=1 gateway=VF routing-mark=VFConnectionRoute
add check-gateway=ping distance=1 gateway=Spark routing-mark=SparkConnectionRoute
add check-gateway=ping distance=2 gateway=Spark
add check-gateway=ping distance=2 gateway=VF
/system clock
set time-zone-name=Pacific/Auckland
Obviously there are a few security holes in this configuration, I’ve left firewall to do last as to not complicate the setup further than it already is.