Thank you - unfortunately the gateway is assigned by DHCP from the PPPoE and after speaking to the provider, the gateway changes periodically, so reluctant to hard code.
Maybe the only option is to have two MikroTik routers, each with a single PPPoE connection and default gateway. Then create an additional default route from the first router to the second with a higher “distance”?
You can try to play with routing filters (distance 7 serves as unique selector):
/ip dhcp-client
add default-route-distance=7 interface=test2
/routing filter
add chain=dynamic-in distance=7 set-distance=1 set-in-nexthop-direct=test3
Thank you, I will try this in the next maintenance window. So I replace “test2” with “pppoe-out1”, what is “test3” referencing?
I have included some more information below, when both WANs are PPPoE with the same provider and connect and obtain their static IP and routing information.
If both are enabled, only the primary interface appears functional, input or dst-nat to the secondary WAN does not work. Ping from the router itself to 8.8.8.8 works only IF the src-address is set to the secondary interface IP.
I do not need load balancing, only dst-nat to function inbound for both IPs, so I can spread inbound services across the two connections. Also if the connections could failover based on their distance - that would be great too, but I think that part works already - as I set the distance on the pppoe connection.
/interface print
Flags: D - dynamic, X - disabled, R - running, S - slave
# NAME TYPE ACTUAL-MTU L2MTU MAX-L2MTU MAC-ADDRESS
0 R ;;; WAN1
ether1 ether 1500 1598 8156
1 R ;;; WAN2
ether2 ether 1500 1598 8156
2 R ;;; LAN Trunk
ether3 ether 1500 1598 8156
3 ether4 ether 1500 1598 8156
4 ;;; Config
ether5 ether 1500 1598 8156
5 ether6 ether 1500 1598 8156
6 ether7 ether 1500 1598 8156
7 ether8 ether 1500 1598 8156
8 ether9 ether 1500 1598 8156
9 ether10 ether 1500 1598 8156
10 sfp1 ether 1500 1600 8158
11 R ;;; WAN1
pppoe-out1 pppoe-out 1492
12 R ;;; WAN2
pppoe-out2 pppoe-out 1492
13 R ;;; Main
vlan2 vlan 1500 1594
14 R ;;; Staff
vlan3 vlan 1500 1594
15 R ;;; Guest
vlan4 vlan 1500 1594
16 R ;;; Hosted
vlan5 vlan 1500 1594
/ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADS 0.0.0.0/0 62.3.80.17 1
1 DS 0.0.0.0/0 62.3.80.17 2
2 ADC 62.3.80.17/32 88.xxx.xxx.151 pppoe-out1 0
3 ADC 62.3.80.17/32 88.xxx.xxx.182 pppoe-out2 0
4 ADC 192.168.1.0/24 192.168.1.1 vlan3 0
5 ADC 192.168.2.0/24 192.168.2.1 vlan4 0
6 ADC 192.168.50.0/24 192.168.50.1 vlan2 0
7 ADC 192.168.70.0/24 192.168.70.1 vlan5 0
8 A S 192.168.71.0/24 192.168.70.10 1
9 DC 192.168.88.0/24 192.168.88.1 ether5 255
“test2” and “test3” are my interfaces. Unfortunately, this is most likely wrong. I still think that something can be done using route filters, but I’ll have to do few experiments to see if I can come up with working solution.
You can help by sharing your PPPoE config, specifically I’m looking for any hint why you have two routes for 62.3.80.17/32, but I get only one with two interfaces (I’m a little new to PPPoE):
4 ADC 62.3.80.17/32 88.0.0.182 pppoe-out2 0
pppoe-out1
Hi again, you caught me out that time, correct - when they match it shows as one line, I however doctored the “route print” to show with the same gateway, sometimes it comes up with different gateways for each, but often they are the same - just not that day.
If the gateways are the same or different it still will not work - despite all the mangles I have tried.
I don’t see a way how to do what I wanted to with route filters (it doesn’t necessarily mean that there isn’t one, I might have overlooked something). But if I understand it correctly, PPPoE should not care about gateways very much and should work with routes that have interface as gateway. I tried it with connections to router and it works:
/ip route
add distance=1 gateway=pppoe-out1 routing-mark=wan1
add distance=1 gateway=pppoe-out2 routing-mark=wan2
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out1 \
new-connection-mark=wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out2 \
new-connection-mark=wan2 passthrough=yes
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1 \
passthrough=yes
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2 \
passthrough=yes
To use it with internal services (dstnat), add either just these rules:
/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> \
new-routing-mark=wan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> \
new-routing-mark=wan2 passthrough=yes
Or you can drop in-interface= to make it work with all interfaces and also add this:
/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main
Hi
I’ve already done this 100 times over, just to get a PING to the router from outside, it works only when one interface is enabled. I even forced all ICMP traffic to reply through the interface I was pinging, but when using the TORCH I could see it was replying out of the other interface. The only way I could get it to stick to the interface was make a route rule based on destination address.
/interface pppoe-client
add add-default-route=yes comment=WAN1 default-route-distance=1 disabled=no interface=ether1 keepalive-timeout=disabled max-mru=1492 max-mtu=1492 name=\
pppoe-out1 password=xxx user=acc1
add add-default-route=yes comment=WAN2 default-route-distance=2 disabled=no interface=ether2 keepalive-timeout=disabled max-mru=1492 max-mtu=1492 name=\
pppoe-out2 password=xxx user=acc2
It may be time to post your config (routes, firewall rules, …), because marking routing is something that usually works without any problem. You may have some tiny mistake somewhere…
Below is the configuration, currently pppoe-out1 is disconnected to allow function on pppoe-out2 as normal.
This is the clean config without any mangles or route marking.
/ip route
add distance=1 dst-address=192.168.71.0/24 gateway=192.168.70.10
/ip firewall address-list
add address=224.0.0.0/4 list=Bogon
add address=203.0.113.0/24 list=Bogon
add address=198.51.100.0/24 list=Bogon
add address=198.18.0.0/15 list=Bogon
add address=192.88.99.0/24 list=Bogon
add address=192.0.2.0/24 list=Bogon
add address=192.168.0.0/16 list=Bogon
add address=172.16.0.0/12 list=Bogon
add address=169.254.0.0/16 list=Bogon
add address=127.0.0.0/16 list=Bogon
add address=10.0.0.0/8 list=Bogon
add address=0.0.0.0/8 list=Bogon
/ip firewall filter
add action=drop chain=input comment=Invalid connection-state=invalid
add action=accept chain=input comment=Config ipsec-policy=in,ipsec
# pppoe-out1 not ready
add action=drop chain=input comment="Bogon WAN1" in-interface=pppoe-out1 src-address-list=Bogon
add action=drop chain=input comment="Bogon WAN2" in-interface=pppoe-out2 src-address-list=Bogon
add action=accept chain=input comment=Site-to-Site in-interface=pppoe-out2 protocol=ipsec-esp
add action=accept chain=input comment="Established, Related" connection-state=established,related
add action=accept chain=input comment=DNS dst-port=53 in-interface=all-vlan protocol=udp
add action=accept chain=input comment=DNS dst-port=53 in-interface=all-vlan protocol=tcp
add action=accept chain=input comment=Config in-interface=vlan2
add action=accept chain=input comment=Config in-interface=vlan5
add action=accept chain=input comment=Config in-interface=ether5
add action=accept chain=input comment=ICMP in-interface=all-vlan protocol=icmp
add action=drop chain=input comment=Drop
add action=drop chain=forward comment=Invalid connection-state=invalid
add action=accept chain=forward comment=Site-to-Site ipsec-policy=in,ipsec
add action=accept chain=forward comment="Main VLAN to Hosted VLAN" in-interface=vlan2 out-interface=vlan5
add action=accept chain=forward comment="Hosted VLAN to Main VLAN" in-interface=vlan5 out-interface=vlan2
# pppoe-out1 not ready
add action=drop chain=forward comment="Bogon WAN1" in-interface=pppoe-out1 src-address-list=Bogon
add action=drop chain=forward comment="Bogon WAN2" in-interface=pppoe-out2 src-address-list=Bogon
add action=accept chain=forward comment="Established, Related" connection-state=established,related
# pppoe-out1 not ready
add action=accept chain=forward comment="VLAN to WAN1" in-interface=all-vlan out-interface=pppoe-out1
add action=accept chain=forward comment="VLAN to WAN2" in-interface=all-vlan out-interface=pppoe-out2
# pppoe-out1 not ready
add action=accept chain=forward comment="Config to WAN1" in-interface=ether5 out-interface=pppoe-out1
add action=accept chain=forward comment="Config to WAN2" in-interface=ether5 out-interface=pppoe-out2
add action=accept chain=forward comment="HTTPS: srv-remote" dst-address=192.168.50.12 dst-port=443 protocol=tcp
add action=accept chain=forward comment="HTTP: srv-lamp" dst-address=192.168.50.14 dst-port=80 protocol=tcp
add action=accept chain=forward comment="OpenVPN: srv-vpn" dst-address=192.168.70.10 dst-port=1194 protocol=udp
add action=accept chain=forward comment="SSH: srv-host1" dst-address=192.168.50.5 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-back" dst-address=192.168.50.30 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-arch" dst-address=192.168.50.31 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-lamp" dst-address=192.168.50.14 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-vpn" dst-address=192.168.70.10 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-qb" dst-address=192.168.50.13 dst-port=22 protocol=tcp src-address=82.145.45.154
add action=accept chain=forward comment="Camera: camera" dst-address=192.168.50.81 dst-port=8000 protocol=tcp
add action=drop chain=forward comment=Drop
/ip firewall nat
add action=masquerade chain=srcnat comment=Loopback dst-address=192.168.50.0/24 src-address=192.168.50.0/24
add action=masquerade chain=srcnat comment="Loopback (Hosted)" dst-address=192.168.70.0/24 src-address=192.168.70.0/24
add action=accept chain=srcnat comment=Site-to-Site dst-address=192.168.0.0/16 src-address=192.168.50.0/24
add action=accept chain=srcnat comment="Site-to-Site (Hosted)" dst-address=192.168.0.0/16 src-address=192.168.70.0/24
add action=accept chain=srcnat comment="Site-to-Site (VPN)" dst-address=192.168.0.0/16 src-address=192.168.71.0/24
# pppoe-out1 not ready
add action=masquerade chain=srcnat comment=WAN1 out-interface=pppoe-out1
add action=masquerade chain=srcnat comment=WAN2 out-interface=pppoe-out2
add action=dst-nat chain=dstnat comment="HTTPS: srv-remote" dst-address=88.xx.xx.182 dst-port=443 protocol=tcp to-addresses=192.168.50.12
add action=dst-nat chain=dstnat comment="HTTP: srv-lamp" dst-address=88.xx.xx.182 dst-port=80 protocol=tcp to-addresses=192.168.50.14
add action=dst-nat chain=dstnat comment="OpenVPN: srv-vpn" dst-address=88.xx.xx.182 dst-port=1194 protocol=udp to-addresses=192.168.70.10
add action=dst-nat chain=dstnat comment="SSH: srv-host1 (WAN1)" dst-address=88.xx.xx.151 dst-port=22 protocol=tcp to-addresses=192.168.50.5
add action=dst-nat chain=dstnat comment="SSH: srv-host1 (WAN2)" dst-address=88.xx.xx.182 dst-port=22 protocol=tcp to-addresses=192.168.50.5
add action=dst-nat chain=dstnat comment="SSH: srv-back (WAN1)" dst-address=88.xx.xx.151 dst-port=222 protocol=tcp to-addresses=192.168.50.30 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-back (WAN2)" dst-address=88.xx.xx.182 dst-port=222 protocol=tcp to-addresses=192.168.50.30 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-arch (WAN1)" dst-address=88.xx.xx.151 dst-port=223 protocol=tcp to-addresses=192.168.50.31 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-arch (WAN2)" dst-address=88.xx.xx.182 dst-port=223 protocol=tcp to-addresses=192.168.50.31 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-lamp" dst-address=88.xx.xx.182 dst-port=224 protocol=tcp to-addresses=192.168.50.14 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-vpn" dst-address=88.xx.xx.182 dst-port=225 protocol=tcp to-addresses=192.168.70.10 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-qb" dst-address=88.xx.xx.182 dst-port=230 protocol=tcp to-addresses=192.168.50.13 to-ports=22
add action=dst-nat chain=dstnat comment="Camera: camera" dst-address=88.xx.xx.182 dst-port=8000 protocol=tcp to-addresses=192.168.50.81
And if you add what I wrote (routes and mangle rules), then what exactly happens? When you ping the router from internet (both addresses), do you see increasing counters on those rules?
I cannot do it right now, as I am not on site, and I do not want to risk locking myself out.
When I did it before the counters do increase yes - it does hit the rules in the mangle.
It must work, it’s just three simple steps:
What could go wrong there? You can easily verify first two steps. There are counters and if you have doubts, you can enable logging for rules and make sure they really catch the right packets. And once the packet gets routing mark, it can’t choose wrong route, because there’s only one in given routing table, which points to right interface. I don’t have any explanation, except perhaps a typo in routing table name. 
I know exactly what you mean, I have tried from scratch several times (by reverting to clean backup) but still no dice, I will try again soon when I am onsite - it is infuriating, and your help is greatly appreciated. Thank you
Living dangerously - I’ve done it remotely - same result 
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 A S 0.0.0.0/0 pppoe-out1 1
1 A S 0.0.0.0/0 pppoe-out2 1
2 ADS 0.0.0.0/0 62.3.80.17 1
3 DS 0.0.0.0/0 62.3.80.17 2
4 ADC 62.3.80.17/32 88.xx.xx.182 pppoe-out2 0
pppoe-out1
5 ADC 192.168.1.0/24 192.168.1.1 vlan3 0
6 ADC 192.168.2.0/24 192.168.2.1 vlan4 0
7 ADC 192.168.50.0/24 192.168.50.1 vlan2 0
8 ADC 192.168.70.0/24 192.168.70.1 vlan5 0
9 A S 192.168.71.0/24 192.168.70.10 1
10 DC 192.168.88.0/24 192.168.88.1 ether5 255
/ip route
add distance=1 gateway=pppoe-out1 routing-mark=wan1-route
add distance=1 gateway=pppoe-out2 routing-mark=wan2-route
add distance=1 dst-address=192.168.71.0/24 gateway=192.168.70.10
And your exact mangle rules are? Just as a last check, before you go find a shaman or similar non-technical help? 
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out1 new-connection-mark=wan1-conn passthrough=yes
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out2 new-connection-mark=wan2-conn passthrough=yes
add action=mark-routing chain=output connection-mark=wan1-conn new-routing-mark=wan1-route passthrough=yes
add action=mark-routing chain=output connection-mark=wan2-conn new-routing-mark=wan2-route passthrough=yes
/ping 8.8.8.8 interface=pppoe-out1
SEQ HOST SIZE TTL TIME STATUS
0 8.8.8.8 56 61 36ms
1 8.8.8.8 56 61 18ms
2 8.8.8.8 56 61 18ms
sent=3 received=3 packet-loss=0% min-rtt=18ms avg-rtt=24ms max-rtt=36ms
/ping 8.8.8.8 interface=pppoe-out2
SEQ HOST SIZE TTL TIME STATUS
0 8.8.8.8 timeout
1 8.8.8.8 timeout
2 8.8.8.8 timeout
sent=3 received=0 packet-loss=100%
/ip route nexthop
0 address=62.3.80.17 gw-state=reachable forwarding-nexthop="" interface="" scope=10 check-gateway=none
1 address=192.168.70.10 gw-state=reachable forwarding-nexthop="" interface="" scope=10 check-gateway=none
Try a different test, ping both public addresses from outside. Because when you initiate connection from router to internet, it won’t get marked, because first packet coming back won’t match connection-state=new.
I tried that first.. same as before - only the lowest distance interface responds.