Multiple WAN, PPPoE and routing challenge

RouterOS Beginner Basics
1

I have an RB5009 ROS v7.12.1 which I intend to use for up to 3 WAN connections and with client/guest network on the LAN side.
I will use PPPoE connections to keep track of the data usage per carrier.

ether 1-3 configured as WAN sources with DHCP client configured for each, connected a source on ether1 with IP 192.168.100.1
ether 4 configured as a hotspot bridge (service hotspot+ppp service), configured to use our shore Radius server
ether 5 is physically connected to ether4 with eth cable, pppoe client named pppoe-client is configured establishes via ether5
ether 6 is configured for the client network bridge (10.1.10.0/24)

WAN1 (ether1) is the priority link which will be used for the client data, if WAN1 is down the client data will be switched to WAN2 or WAN3

I’m only able to get the data to flow if I mangle the client traffic and route the traffic directly via the WAN1 gateway, if I try to set the gateway to the pppoe-client interface there is no traffic flow.

In my test setup:
WAN1 gw is 192.168.100.1
PPPoE gw is 192.168.99.1 (pppoe-client establishes with 192.168.99.254)
Business network 10.1.10.0/24

/ip firewall mangle
add action=mark-routing chain=prerouting in-interface=pppoe-business
new-routing-mark=wan1 passthrough=yes
add action=mark-routing chain=prerouting in-interface=business
new-routing-mark=business_traffic passthrough=yes src-address=10.1.10.0/24

/routing rule
add action=lookup-only-in-table disabled=no routing-mark=business_traffic table=business_traffic
add action=lookup-only-in-table disabled=no routing-mark=wan1 table=wan1

/ip route
add comment=“business client” disabled=no distance=1 dst-address=0.0.0.0/0 gateway=pppoe-vsat pref-src=“”
routing-table=business_traffic scope=30 suppress-hw-offload=no target-scope=10
add comment=wan1 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.100.1 pref-src=“” routing-table=wan1
scope=30 suppress-hw-offload=no target-scope=1

Any pointers would be great.

2

Draw a diagram and nothing is clear.
Do you have three pppoe connections, and if so why are you using IP DHCP CLIENT?
What type of WAN connections.
Is each give you a private or public IP…
Is each from a different provider.
Do you have any vpns coming into the router
Do you have any servers on the LAN.
Which is the first backup WAN2 or WAN3

3

This is for use in a maritime setup , the PPPoE server is on shore and is used for data usage tracking and forms a part of our billing platform.
WAN1 is a low latency satellite link (40ms), private IP received
WAN2 is a high latency satellite link (650+ms), private IP received
WAN3 is a mobile/4G link (low latency), private IP received
All links are from different providers

RADIUS server is on a remote(shore) servers so we’re authenticating through the available WAN sources, we’re establishing one pppoe-client per WAN, we have two user sources which is business and crew, business traffic should be allowed on all WAN’s, priority order is WAN1, WAN2, WAN3. Crew traffic should only be able to go over WAN1, if it is not available this network would not get internet access. On the crew networks we’re running hotspot for data allocation.

The key things is to be able to route the traffic via the PPPoE connections so the traffic is accounted for and we can report on the usage.

Running Wireguard for our remote connection only, there may be servers on the LAN side (customer equipment) but we have no control over that.
lite_setup.png

4

So can I conclude that
-both sets of users (business and private) are to use WAN1 as their primary source of internet?
-if WAN1 is not available, then WAN2 should be used next by business users ONLY
-if WAN2 is not available, then WAN3 should be used next by business users ONLY.

In other words there is no fail over for private users.

+++++++++++++++++++++++++++++
When you say private IPs.
All you get is a private Gateway IP and a private IP,
Does the gateway change or is it fixed.

5

That’s correct Mesquite

6

First things is that we need to make all three WANS available on the Main Table. I have no clue about your pppoe connections and dont care, going to focus on the WAN connections.
Dont think any mangles required ??

/ip route
add check-gateway=ping distance=1 dst-address=0.0.0.0 gateway=current-private-gateway-ISP1 routing-table=main comment=WAN1-Update
add check-gateway=ping distance=2 dst-address=0.0.0.0 gateway=current-private-gateway-ISP2 routing-table=main comment=WAN2-Update
add check-gateway=ping distance=3 dst-address=0.0.0.0 gateway=current-private-gateway-ISP3 routing-table=main comment=WAN3-Update

Now we have established as all three being available with WAN1 as primary. I dont see any IP DHCP client settings.
You should have them indicated for each WAN and set so that the settings are, and that the selection for add default route = NO,

However, if the private IP and more importantly, the ISP private gateway you get can change, then you need to do more.
You need to go back to IP DHCP client for each WAN and add default route=yes and set the distance to 255 and add this script to the script area in advanced tab.

:if ($bound=1) do={
:local gw [ /ip dhcp-client get $“gateway-address” ]
/ip route set [ find comment=“WANX-Update” gateway!=$gw ] gateway=$gw
}

In this manner, when the gateway IP changes, your manual route will also be updated.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

So far we have established routes and a failover so that all can access 1, then 2 and then 3
However we only want business users to access WAN2 and WAN3 in the failover scenario (where WAN1 is not working).

My mind is going blank and the only thing I can think of is NOT to mangle but simply to add a routing rules to force those users to wan1 only…
In this manner, the traffic will simply be dropped.
This should not be a problem assuming there are no local addresses ( other local subnets the non business users need to access ).

/routing table add fib name=StickToWAN1
/routing rule
add src-address=10.7.70.0/24 action=lookup-only-in-table table=StickToWAN1
add src-address=10.8.80.0/24 action=lookup-only-in-table table=StickToWAN1

/ip route
add check-gateway=ping distance=1 dst-address=0.0.0.0 gateway=current-private-gateway-ISP1 routing-table=main comment=WAN1-Update
add check-gateway=ping distance=2 dst-address=0.0.0.0 gateway=current-private-gateway-ISP2 routing-table=main comment=WAN2-Update
add check-gateway=ping distance=3 dst-address=0.0.0.0 gateway=current-private-gateway-ISP3 routing-table=main comment=WAN3-Update
add distance=1 dst-address=0.0.0.0 gateway=current-private-gateway-ISP1 routing-table=StickToWAN1 comment=WAN1-Update

7

Thanks Mesquite, adding the routes and that routing is ok, but the main challenge remains to force the traffic to go via the pppoe so the traffic is recorded. Working on a solution using VRF’s to achieve this, hopefully get over the finish line soon.

Thanks for the script to bind the gw-ip.