Setup Multiple ISPs

JordanReich · December 6, 2019, 11:25pm

I have done a number of advanced things with MikroTik routers in the past but I have never had any experience in this area. Really looking for a conceptual place to start and perhaps some knowledge documents or examples of other people doing something similar that can I can follow.

We are not looking for a load balanced concept. Looking to divert specific ISP traffic to specific places.

The Scenario:
We have two internet service providers. An LTE provider and a DSL provider.

The Want:
Want to have ISP [LTE] assigned to ETH1
Want to have ISP [DSL] assigned to ETH2
Want all traffic on ETH3 to go through the LTE connection to the WAN
Want all traffic on ETH4 to go through the DSL connection to the WAN
ETH5 will be used to support an EoIP tunnel for other needs

Advanced:
The long term goal is to find a way to measure bandwidth usage and swap ETH3 over to DSL and back to LTE depending upon data usage. But for now the setup above will work to get everything moving forward.

Any general thoughts on this process would be appreciated.

Thank you!

ingdaka · December 7, 2019, 10:40am

Just use mangle routing-mark to mark you traffic coming from port 3 and 4 with 2 different marks and them assign them to default route rule!

SiB · December 9, 2019, 10:50pm

In 2 minutes I have your setup in GNS3 … simple and static solution with Route Rule and interface as source of traffic..

[admin@JR] > interface ethernet print 
Flags: X - disabled, R - running, S - slave 
 #    NAME
 0 R  ether1-lte
 1 R  ether2-dsl
 2 R  ether3-via-lte
 3 R  ether4-via-dsl
 4 R  ether5
 
/ip route
add comment=LTE_own_route_table distance=1 gateway=10.2.3.1 routing-mark=RouteTable--LTE
/ip route rule
add interface=ether4-via-dsl table=main
add interface=ether3-via-lte table=RouteTable--LTE

[admin@JR] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  ;;; LTE_own_route_table
        dst-address=0.0.0.0/0 gateway=10.2.3.1 gateway-status=10.2.3.1 reachable via  ether1-lte distance=1 scope=30 target-scope=10 routing-mark=RouteTable--LTE 

 1 ADS  dst-address=0.0.0.0/0 gateway=10.3.4.1 gateway-status=10.3.4.1 reachable via  ether2-dsl distance=1 scope=30 target-scope=10 vrf-interface=ether2-dsl 

 2  DS  dst-address=0.0.0.0/0 gateway=10.2.3.1 gateway-status=10.2.3.1 reachable via  ether1-lte distance=2 scope=30 target-scope=10 vrf-interface=ether1-lte 

 3 ADC  dst-address=10.2.3.0/24 pref-src=10.2.3.254 gateway=ether1-lte gateway-status=ether1-lte reachable distance=0 scope=10 

 4 ADC  dst-address=10.3.4.0/24 pref-src=10.3.4.254 gateway=ether2-dsl gateway-status=ether2-dsl reachable distance=0 scope=10

JordanReich · December 21, 2019, 1:14am

Finally was able to get a test lab setup at home where I could work on this configuration. I attempted to follow the general information provided to the best of my ability. I do seem to be able to get DHCP ranges set appropriately whether I am on the LTE or DSL network. But I still do not have any internet connectivity. Any help you can provide would be appreciated!

[admin@MikroTik] > /export hide-sensitive 
# dec/20/2019 20:11:41 by RouterOS 6.45.2
# software id = WD8P-ZQPL
#
# model = RB750Gr3
# serial number = 8B000A2ABF57
/interface ethernet
set [ find default-name=ether1 ] name=ether1-lte
set [ find default-name=ether2 ] name=ether2-dsl
set [ find default-name=ether3 ] name=ether3-via-lte
set [ find default-name=ether4 ] name=ether4-via-dsl
set [ find default-name=ether5 ] disabled=yes
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp-lte ranges=10.0.1.2-10.0.1.200
add name=dhcp-dsl ranges=10.0.3.2-10.0.3.200
/ip dhcp-server
add address-pool=dhcp-lte disabled=no interface=ether3-via-lte name=dhcp-lte
add address-pool=dhcp-dsl disabled=no interface=ether4-via-dsl name=dhcp-dsl
/interface list member
add interface=ether1-lte list=WAN
add interface=ether2-dsl list=WAN
add interface=ether3-via-lte list=LAN
add interface=ether4-via-dsl list=LAN
/ip address
add address=10.0.1.1/24 interface=ether3-via-lte network=10.0.1.0
add address=10.0.3.1/24 interface=ether4-via-dsl network=10.0.3.0
/ip dhcp-client
add dhcp-options=clientid,hostname disabled=no interface=ether1-lte \
    use-peer-dns=no
add dhcp-options=clientid,hostname disabled=no interface=ether2-dsl \
    use-peer-dns=no
/ip dns
set servers=8.8.8.8
/ip firewall filter
add action=accept chain=input comment=\
    "DEFAULT: Accept established, related, and untracked traffic." \
    connection-state=established,related,untracked
add action=drop chain=input comment="DEFAULT: Drop invalid traffic." \
    connection-state=invalid
add action=accept chain=input comment="DEFAULT: Accept ICMP traffic." protocol=\
    icmp
add action=drop chain=input comment=\
    "DEFAULT: Drop all other traffic not coming from LAN." in-interface-list=\
    !LAN
add action=accept chain=forward comment="DEFAULT: Accept In IPsec policy." \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="DEFAULT: Accept Out IPsec policy." \
    ipsec-policy=out,ipsec
add action=accept chain=forward comment=\
    "DEFAULT: Accept established, related, and untracked traffic." \
    connection-state=established,related,untracked
add action=drop chain=forward comment="DEFAULT: Drop invalid traffic." \
    connection-state=invalid
add action=drop chain=forward comment=\
    "DEFAULT: Drop all other traffic from WAN that is not DSTNATed." \
    connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip route
add comment=LTE_own_route_table distance=1 gateway=10.0.0.1 routing-mark=\
    RouteTable--DSL
add distance=1 gateway=192.168.1.1 routing-mark=RouteTable--DSL
/ip route rule
add interface=ether4-via-dsl table=RouteTable--DSL
add interface=ether3-via-lte table=RouteTable--LTE
/system clock
set time-zone-name=America/New_York
[admin@MikroTik] >

SiB · December 21, 2019, 9:36am

You not ask about that in your Question Post#1 but for inspiration please read, analyze, apply this one:
.

Bandwidth-based load-balancing with failover. This presentation also covers Mangle.
This was presented at the MUM (MikroTik User Meeting) in New Orelans, USA.
Tomas Kirnak - YouTube: > https://www.youtube.com/watch?v=67Dna_ffCvc&t=1s
http://mum.mikrotik.com/presentations/US12/tomas.pdf

This is very good way to setup “MultiWAN” but it’s very very good to learn about traffic on multi interface’s. For me it’s next step for you to do.
Even if your LTE haven’t a PublicIP and you not receive traffic from LTE even do still the way of setup cover many aspects.
The way to send traffic to internet can by done by Route>Rules; Firewall>Mangle rule via script’s…; PCC; hand change; Route Distance… even this is just way to send traffic into web and upper material give more.
.

But I still do not have any internet connectivity. Any help you can provide would be appreciated!

I miss read this part. I will import your setup and edit this post. Please wait. I still your project and I must add tho VM as client-s of both your LANs.
See this:

About your config, this part is bad:

/ip dhcp-client
add dhcp-options=clientid,hostname disabled=no interface=ether1-lte use-peer-dns=no
add dhcp-options=clientid,hostname disabled=no interface=ether2-dsl use-peer-dns=no

Means both have distans default=1 and no one is Active, you should on one of them add this:
default-route-distance=2
means:

/ip dhcp-client
add default-route-distance=2 dhcp-options=hostname,clientid disabled=no interface=ether1-lte
add dhcp-options=hostname,clientid disabled=no interface=ether2-dsl

.
You configure DHCP Server by hand, not from WinBox? I not see the dhcp network entry.
DHCP-Server Wizzard work from CLI too: /ip dhcp-server setup

/ip dhcp-server network> add address=10.0.1.0/24 gateway=10.0.1.1 dns-server=1.1.1.1
/ip dhcp-server network> add address=10.0.3.0/24 gateway=10.0.3.1 dns-server=1.1.1.1

.
Next, you have PC behind your Router, you need NAT, means SNAT to hide your PC in internet and share internet to them.

/ip firewall nat add out-interface-list=WAN chain=srcnat action=masquerade

.
After this correction of your setup my GNS3 VPCS’s have access to internet and one by LTE and second one by DSL - checked by tracert.
.

Finally was able to get a test lab setup at home where I could work on this configuration.

I recommend you to install GNS3 VM via my HowTo:
https://gns3.com/community/discussion/gns3-with-workstation-player-15- → By Marcin Przysowa OCTOBER 19TH, 2019
On All MikroTik in GNS3 I can logon via WinBox by using RoMON - very helpful.

I hope I help you with your setup.
Good Luck.

JordanReich · December 30, 2019, 5:14pm

Thank you the above steps worked perfectly and solves the original problem. I do have one complication you may be able to help with.

I have been able to split the internet between DSL and LTE, no problem. I can even swap the DSL and LTE adapters in the route table and I can flip what side of the house gets one or the other. But there are scenarios where - for example - if DSL went out completely where I’d want to swap the whole house back over to LTE.

I figured this would be easy to do and simply set both routing tables the DSL and LTE routing tables both to the LTE adapter. That immediately caused the internet to fail. I could swap them so they were on either side but I could not roll the house over completely into one connection.

I was able to do so but had to disable the route table rules the DHCP server for the internet not being used. Then the whole house picked up the LTE service instead of the DSL. But this was a 5-8 step process. And this is something I would like to potentially script in the future.

Can you think of a faster way to move the internet from split to one system without doing all of the above steps? Appreciate your help once again!

SiB · January 1, 2020, 11:54pm

IP>Routes check nexthop every 10s and after 20s is the longest time to change route.
If your connection is not lte1 interface menas gateway is IP Address then you can use simple way like:
https://wiki.mikrotik.com/wiki/Advanced_Routing_Failover_without_Scripting
but remembet that every checking function base on icmp 1 time failure (when somebody do speedtest on your LAN) can do changing your topology.

Gold way is use a few way checking script who do action after few testing.

I hope I add you some information.
Your post have some info and question and I don’t know what is a big one question.
BR Marcin

JordanReich · January 25, 2020, 10:05pm

You have been amazing thank you again for all your help. I need your services for one additional effort and then I think I have all of this wrapped up.

Right now … this is my current routing table for the residence …

The multiple ISP setup works perfectly. The problem I am now facing is that I have an L2TP/IPSEC connection to a central hub. These addresses are set in the route list above as 10.0.0.1 and 10.0.2.1 and are different physical house locations. I can ping them through the terminal on the router and get a response. However when I ping them from one of the computer devices between the multiple ISP I do not get a response.

I am pretty sure this is due to the fact that ALL DATA from ETH3/ETH4 are now being directed to a specific RouteTable and ignoring the other locations. How do I add those routes into consideration so the multiple ISP still works as well as the addresses I’ve mentioned above?

Thanks again!

SiB · January 26, 2020, 10:06pm

JordanReich

True. RB can ping them but why?

Because RB use a main route table and the firewall filter chain=ouput.
The src.address are from MikroTik itself. What IP?

.
In Ping you can do some additional parameters like:
ping 1.1.1.1 src-address=(One’s of IP who you see in pref.Source column)
ping 1.1.1.1 routing-table=(RouteTable–DSL or RouteTable–LTE)
or with src-address and routing-table together. Play with them and reach next level of understanding this.
.

However when I ping them from one of the computer devices between the multiple ISP I do not get a response.
I am pretty sure this is due to the fact that ALL DATA from ETH3/ETH4 are now being directed to a specific RouteTable and ignoring the other locations. How do I add those routes into consideration so the multiple ISP still works as well as the addresses I’ve mentioned above?

This can be done on few ways and for you the better is do a simple workaround.
Example.

Add route to 10.0.0.0/24 and 10.0.2.0/24 into additional RouteTable–[LTE or/and DSL] = RT’s.
This way each RT can access traffic by duplication static route into other RT’s who is a workaround and on production with many VPN’s it’s horrible idea but for you is the best :D.
WinBox have got perfect button with name “Copy” and that way you just select RT and press OK and repeat it few times.
Do exclusion in current IP > Route > Rules on traffic from ether3/4 to “10.0.0.0/24 and/or 10.0.2.0/24” are not enter those additional RT but stuck / be / stay in RT=MAIN. And exclusion must be upper then your MultiWan rule.Few rules are necessery. In production this is rare config because a Firewall Mangle are better in bigger network’s but for you it’s perfect

Chose one and play with this.
Good Luck.

JordanReich · January 27, 2020, 5:45pm

Thank you! This resolved my problem completely and everything is now functioning as it should be. Your help has been very much appreciated.

Final Result:

SiB · January 27, 2020, 10:12pm

Until you want use Public IP at LTE and do dual DNAT via both ISP.
Until you not want send session’s via both ISP together that way of configuration is 3S = simple + static + stable.

Remember that next level of MultiWAN is in

Bandwidth-based load-balancing with failover. This presentation also covers Mangle.
This was presented at the MUM (MikroTik User Meeting) in New Orelans, USA.
Tomas Kirnak - YouTube: > https://www.youtube.com/watch?v=67Dna_ffCvc&t=1s
http://mum.mikrotik.com/presentations/US12/tomas.pdf

BR.
Marcin Przysowa

JordanReich · January 30, 2020, 3:20pm

I do have one additional curiosity that I am having a problem figuring out. It is ongoing!

All traffic from the site with all the route tables above is working perfectly. I can contact anything anywhere no problems.

However from the HUB that hosts the L2TP server I cannot ping any devices within 10.0.3.0/24 from the router. Although I can ping devices at that address from some of my subnets. For example if I am on 10.0.0.0/24 within one of my physical computers I can get a ping response no problem from 10.0.3.185. But the router consistently returns a timeout.

This is my routing table on the HUB which has always worked in the past and does work for all of my other sites.

But I am curious if the problem is with the complex routing table or something I need to do on the HUB?
I need my HUB router to be able to ping devices inside the 10.0.3.0/24 subnet range.

Hopefully this makes sense.

SiB · January 31, 2020, 1:17pm

You not do a homework with:

In Ping you can do some additional parameters like:
ping 1.1.1.1 src-address=(One’s of IP who you see in pref.Source column)
ping 1.1.1.1 routing-table=(RouteTable–DSL or RouteTable–LTE)
or with src-address and routing-table together. Play with them and reach next level of understanding this.

Means from MikroTik CLI you must provide the SOURCE who will be like your computer’s in LAN who use the Route>Rules.
PC’s in LAN use R>Rules who give them separately RouteTable–A/B and in internal PING from MikroTik you must provide it too, to simulate your PC traffic.

ping 10.0.3.1 src-address=10.0.0.1 routing-table=“RouteTable–DSL”
This is answer or not, because you write big one question

JordanReich · February 4, 2020, 8:58pm

Brilliant! Source IP solved the problem, thank you.

The questions are never ending …

When the L2TP/IPSEC connection occurs I noticed on my primary hub that the multi-site router is connecting to me via DSL.

How can I force the VPN connection to use LTE as its connection ISP rather then using DSL?

SiB · February 5, 2020, 9:35am

Check answer #9 and picture on it about Pref.Source.
One of many solution is to create in MAIN route table, the new classic route and say

# When you have only one PublicIP from your ISP then just do route to your vpn server via proper gateway (ISP).
/ip route add dst-address=1.1.1.1 gateway=[192.168.8.1|lte1] disabled=yes;
# When you have FEW/MANY PublicIP from your ISP then Pref.Source can choose which one will be used. At DSL you can receive few of them.
/ip route add dst-address=1.1.1.1 gateway=[192.168.8.1|lte1] pref-src=192.168.8.100 disabled=yes;

where:

1.1.1.1 is IP of your main router who is your vpn gateway/server/host/hq/dc/vps
192.168.8.1 or lte1 is gateway of LTE devices
192.168.8.100 is your IP what you receive and see in /ip address in lte interface

Other way is to use Route Rules for MikroTik itself like: to 1.1.1.1 use RouteTable–LTE
Other way is to use Firewall Mangle on output and do to 1.1.1.1 action mark-route to RouteTable–LTE.
Other way is … hmm, no idea :). Joke. You always can have 2xWAN in every branch and 2xWAN in HQ and 2xWAN in AdditionalDataCenter and create a vpn mesh with ospf on it

Hope those methods will be good for you. Play with first one, classic route

JordanReich · February 6, 2020, 9:35pm

You’re solving all my problems! I went with a route rule and that worked perfectly. Every time I think I have it all wrapped up I run into another one.

Hopefully this will be the last one…

So I have an external DNS name lets call it johndoe.com. I am calling from an external IP address will call it 1.1.1.1.
It hits my primary router 2.2.2.2 which is the L2TP/IPSEC server and than NATs that traffic over port 8083 to an address at this secondary hub in this case 10.0.3.40.

I have confirmed via Wireshark that the packets are getting all the way to the server. If I am inside my network lets say on 10.0.0.53 and not external this loads without any problems. But all the external traffic results in a 404 timeout.

However I have traced my attempted external connection and it does in fact reach 10.0.3.40 as I get a ton of TCP transmissions followed by a connection reset. What seems to be happening is that traffic being sent back is not being sent back over the line that was requesting the page in the first place but it is attempting to dial that external address 1.1.1.1 directly. I am guessing this is because the route rules say that anything on the LTE ethernet port goes through the LTE ISP connection.

How would I route return traffic back over the VPN instead of dialing back out to the internet? Assuming my assumption is correct.

Thanks! Hopefully that makes sense.

SiB · February 6, 2020, 10:35pm

You should create a graf because it is so hard understand what you have a problem this time.
.

I am calling from an external IP address will call it 1.1.1.1. It hits my primary router 2.2.2.2 which is the L2TP/IPSEC server and than NATs that traffic over port 8083 to an address at this secondary hub in this case 10.0.3.40.

Realy, I always ignore that sentence
You write about 1.1.1.1 and next 2.2.2.2 but in network we speak From/To or Src/Dst and via this Host/Port… .
.

I have confirmed via Wireshark that the packets are getting all the way to the server. If I am inside my network lets say on 10.0.0.53 and not external this loads without any problems. But all the external traffic results in a 404 timeout.

If we speak that local PC cannot open webpage from the webserwer who is in the same LAN network then you should setup HairPinNat rule.
If we speak that traffic from internet to your VPN server reach your local host then of course his answer will be send by @main route table to internet directly - not comeback via vpn. This must be fixed via next rules.
.

However I have traced my attempted external connection and it does in fact reach 10.0.3.40 as I get a ton of TCP transmissions followed by a connection reset. What seems to be happening is that traffic being sent back is not being sent back over the line that was requesting the page in the first place but it is attempting to dial that external address 1.1.1.1 directly. I am guessing this is because the route rules say that anything on the LTE ethernet port goes through the LTE ISP connection.

How would I route return traffic back over the VPN instead of dialing back out to the internet? Assuming my assumption is correct.

Bingo. Only this is a real question.

You must just set proper color on packet incomming via vpn and set rule that specific color package must go comeback the same path.
Coloring means Marking - it’s adding label like in painting.

step) Do paint and color=mark some traffic, all session incomming via vpn. Sessions we say a Connection who are combain from packages.
You know… woman speak in Connections and Men in packages or maybe opposite - it’s so hard to say :D. We must select only package about beer - this is our target.

/ip firewall mangle add chain=prerouting protocol=tcp in-interface=l2tp-some-vpn1 connection-mark=no-mark dst-address=10.20.30.1 dst-port=22,8291 action=mark-connection new-connection-mark="MyFavoriteIncommingTraffic" passthrough=yes
# incomming tcp via vpn interface who is not mark jet and go to my IP at VPNServer on port ssh/winbox will be mark in connection way as MyFavoriteIncommingTraffic

.
2. step) When router receive answer=comeback connections/package from local host WHO have our label=mark=color on it THEN we send that traffic on proper RouteTable–VPN Yes, next one

/ip firewall mangle add chain=prerouting connection-mark="MyFavoriteIncommingTraffic" in-interface="bridge1-servers-with-dnat_DMZ" action=mark-routing new-routing-mark=RouteTable--VPN passthrough=yes
# Traffic from LAN with marking MyFavoriteIncommingTraffic I will be send via separated RouteTable--VPN and this way package comeback from where are

I hope this is proper answer.

JordanReich · February 13, 2020, 4:55pm

Thank you for that … So I appear to be having an issue with the prerouting passthrough.

On a random computer on the internet if I go to https://johndoe.com:8083 … That hits my MikroTik primary router. Then a NAT pushes that port to 10.0.3.40 which is located at another site location that is through the VPN.

On the site location with 10.0.3.40 I setup the following on the router …

But the bytes/packets remain 0 when I dial the address on the internet external to the network. Not sure if I set it up incorrectly?

SiB · February 13, 2020, 5:43pm

For what? On what device witch how many WAN/VPN interface? You have some devices and I have not idea on what one you set what configuration.
.

On a random computer on the internet if I go to > https://johndoe.com:8083 > …

Traffic from internet.
.

That hits my MikroTik primary router.

Traffic from internet hit Main router.
.

Then a NAT pushes that port to 10.0.3.40 which is located at another site location that is through the VPN.

Traffic from internet hit Main router and it’s DNAT to next router 10.0.3.40 via VPN who connect them.
.

On the site location with 10.0.3.40 I setup the following on the router …
But the bytes/packets remain 0 when I dial the address on the internet external to the network. Not sure if I set it up incorrectly?

I have no idea what you setup at this “next router” and how many he have got the WAN interface. All previouse information are proper when you input/enter trafic via main ISP and any other interface (ISP/VPN) have not any incomming traffic.

Then:

First you should check the Tool>Torch at vpn interface and check if traffic is incomming.
Next you should check the Firewall>Connections and if this 8083 port have tcp communication then you should see one of TCP handshake status who give you idea if package is only receive or send back. Check here the 4 collumn with Reply Src/Dst Address to who is good to recheck if your SNAT not change IP and what status this rule have.
Next is Tool/Sniffer with filter on 8083 port and track of interface’s with Rx/Tx who give you info on what interface is incomming and on what is outgoing. In this place is you have more ISP/VPN then one you should play with this tool.

Finally, maybe you count 0 package because you put to match a catchers/filters option and/or you should try input chain and/or #17 is answer you miss.

In normal scenatio you should mark traffic at chain=input and send this marked traffic to Route-Table–VPN via Mangle chain=output as I write at #17.
I don’t know why you try mark this traffic and what do. How many you have routes. Just like you not read my prev. answer and aks the same question.

SiB · February 14, 2020, 1:54am

@JordanReich - Remember that you can write me message to Telegram - this is secure and anonymous (you and I not know the phone numbers!). This way will be easy to speak about that problem.