Routed Multi-WAN without NAT

Hi all
I’m going to start this with a disclosure (and I totally understand if you stop reading immediately afterward): I haven’t deployed a BGP setup in many years. The last time I did this was on a Cisco 2601 with a pair of T-1 WAN modules for a whopping available bandwidth of 3.088mbps.

I’ve googled and searched this forum and I’ve hit that point where I keep googling different terms that keep sending me to the same few articles that tell me that what I need to do can’t be done in current versions of the RouterOS. I’m looking for confirmation or not that this is the case. If It can be done, I’d be extra bonus credit appreciative if someone could tell me how to do it!

On to the actual question: I’ve been tasked with adding a second 10Gb WAN link to a customer’s CCR1036 router. The customer has a /21 ARIN assigned IP block that is currently being statically routed by their upstream ISP. The setup looks like this:
0/0 ↔ ISP1 ↔ router interface with ISP assigned /30 ↔ router interfaces with IPs on different subnets of the /21

The router has a number of interfaces with subnets of the /21 block. All are routed, there is no NAT of any of the /21 address space on the edge router.

The new setup adds a second ISP WAN link, with an additional /30 address and gateway. I need the /21 logically behind the router to be advertised by both upstream ISPs. That appears to be the easy part - set up a pair of BGP peers in the router and turn down the static advertisement on ISP1. That part is done, the ISPs are ready to turn up BGP and start advertising the routes as soon as I say go.

The hard part seems to be in implementing the load balancing on the outbound side, which I believe requires ECMP, which I (mistakenly?) thought would be pretty trivial to do in a Mikrotik.
That assumption appears to be a mistaken one?

So, to summarize: I need to add a second, routed, Internet connection, over which all of the subnets of the /21 are available and that provides load balancing and failover for the subnets behind it.
Here’s what it will look like when it’s all set up:
Multiwan with BGP & ECMP.png
Thanks in advance for any light you can shed, even if it’s flames telling me I’m in over my head… Over my head or not I have to get this working :slight_smile:

Hi,

If you just need basic ecmp you can set up same destination (default) routes from each interface with the same distance.

You can set up recursive routing on the 2 links, so if some link specific upstream target becomes unavailable, that link
is regarded as dead.

If you need bgp to do clever things, sorry I can’t help you.

First, thanks very much for your reply! That sounds like what I need (the dual outbound routes). In the back of my head I’ve been wondering if I actually needed ECMP or not.
Can you tell me a little more about the recursive routing/ failover? One of the primary use cases for this set up is for when one of the upstream connections goes down, and I need the router to be able to detect that and automatically send all traffic over the remaining circuit.
Thanks again :slight_smile:

Some (maybe) helpful stuff

https://help.mikrotik.com/docs/pages/viewpage.action?pageId=26476608
http://forum.mikrotik.com/t/recursive-routes/147430/1

Mostly they seem to like route marking but if you have some IP addresses that you don’t mind if only accessible via
one route.

Perhaps Something like this

/ip/route/
add dst-address=8.8.8.8 scope=10 gateway=1.2.3.4
add blackhole comment=“blackhole 8.8.8.8 except via 1.2.3.4” distance=80 dst-address=8.8.8.8

add dst-address=0.0.0.0/0 gateway=8.8.8.8 check-gateway=ping target-scope=11

add dst-address=8.8.4.4 scope=10 gateway=5.6.7.8
add blackhole comment=“blackhole 8.8.4.4 except via 5.6.7.8” distance=80 dst-address=8.8.4.4

add dst-address=0.0.0.0/0 gateway=8.8.4.4 check-gateway=ping target-scope=11

You will need to play with it to see how/if it works for you.

A simpler option that might (or might not) work

/ip/route/
add dst-address=8.8.8.8 scope=10 gateway=1.2.3.6
add dst-address=8.8.8.8 scope=10 gateway=5.6.7.10

add dst-address=0.0.0.0/0 gateway=8.8.8.8 check-gateway=ping target-scope=11

This seems likely to work

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=8.8.8.8/32 gateway=1.2.3.6%ether1 pref-src="" routing-table=\
    main scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=8.8.8.8/32 gateway=5.6.7.10%ether2 pref-src="" routing-table=\
    main scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=8.8.8.8%ether1 pref-src="" routing-table=main \
    scope=10 target-scope=11
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=8.8.8.8%ether2 pref-src="" routing-table=main \
    scope=10 target-scope=11

Thanks for all this. It seems like I may have been overthinking the problem, and that I don’t actually need full ECMP but just dual default routes.

I know when I’ve done this using IPROUTE2 on Linux, I could enable ‘stickiness’ for outgoing sessions so the whole session would go over the same outbound route - is there anything like that in the Mikrotik RouterOS?

Thanks again for all the help, I’m going to try to light this up this week.

I gain the impression that sticky is the only way the Mikrotik will do it.

https://wiki.mikrotik.com/wiki/Manual:IP/Route#Multipath_(ECMP)_routes

Stickiness is due to connection tracking. The tracking happens regardless of NAT.
So in theory you can disable connection tracking to avoid, but I can’t speak to the side-effects.