Multi-WAN Load Balancing Starlink issue

[jaysen]

I had just added -wanN to the end of the default names but willing to rename them to simplify things if necessary.

Depending on the order of creating the DHCP client, renaming the interface it is attached to, and possibly rebooting the router in the past the result may be different. The lease script uses the interface name to find, add, and modify the routes, and on my test router it was remembering the previous name. So forcing the DHCP client to renew an address after adding the script using /interface/ethernet/disable sfp28-N-wanN ; delay 2s ; /interface/ethernet/enable sfp28-N-wanN rather than /ip/dhcp-client/release [find where interface=sfp28-N-wanN] may be a better choice after renaming the interface or if you are not certain.

Good news about the satellite visibility … OneWeb can’t even begin to compete …

Frankly speaking the simulation at starlink.sx has scared me a lot as it is showing long (as in tens of minutes) outages in coverage for Unalakleet, so I hope they are using outdated/inaccurate data. LEO satellites are superior to geostationary ones for both delay and throughput per square mile of coverage. Far lower distance to travel so far lower delay, far lower attenuation to compensate, and far bigger area to cover by a single antenna with a given throughput. But in these areas, polar trajectories seem to be mandatory so it’s more or less a separate (from the main one covering the belt from 52°S to 52°N) Starlink constellation. And OneWeb probably didn’t get that far (yet) so geostationary is their only choice for those areas.

I did have a question about the failover section and while I did set the distances in the routes. I am starting to wonder whether I even need to do that since all the gateways are the same on most of the terminals so if it goes down then it seems like it would be down for every connection that uses the same gateway?

From my experience with other Starlink installations in “bypass” mode, the gateway is probably redundant - if you look into the ARP table, you will probably see the MAC addresses of all (currently “both”) the gateway IPs to begin with 00:00:5E:00:. So it is more likely that all your terminals lose connection than that a gateway in just one of the subnets becomes unreachable.

So I maintain my previouos position that we have to deal with issues we have to capacity to deal with, i.e. a breakdown of a single terminal. The only satellite within reach gone bonkers does not fit into this category.

As for route distances - these determine the priority among routes whose dst-address and routing-table parameters are identical. If multiple such routes are eligible for being active, those with the lowest value of distance are actually made active, and if there are multiple such ones, they are used in a round-robin manner (ECMP). So as @anav probably wrote earlier - in order that failover and load distribution worked in accord, you need to have one routing table per uplink that is used for traffic that should prefer that uplink but can send the traffic via other uplinks if the preferred one becomes unusable. In the simplest to configure case, you define just one backup uplink for each preferred one, so if the preferred one dies, the backup one has to bear its full load on top of its own one. In the optimized case, the load of the failed link is evenly distributed among all the remaining ones. So much more lines of configuration with much more room for mistakes but potentially less impact on customers if a terminal eventually fails. Choose your poison.

So for starters, let me give you an example for the L2TP, which is the most wanted functionality right now I gather.

You’ll add a routing table named for-l2tp with two routes, one with distance=1 and the other one with distance=2, using the two WANs for which you have modified the DHCP client behavior. Let’s say you’ve chosen WAN 9 and WAN 10:

/routing/table/add name=for-l2tp fib
/ip/route/add routing-table=for-l2tp gateway=1.1.1.1%sfp28-10-wan10 distance=1
/ip/route/add routing-table=for-l2tp gateway=1.1.1.1%sfp28-9-wan9 distance=2

By forcing a release followed by a re-lease (pun intended) of the DHCP address as described above, you’ll trigger the lease script that should replace the 1.1.1.1 by the correct addresses of the respective gateways in these routes.

There are multiple possible ways to make the L2TP connection use this table; since the own WAN addresses of the Mikrotik are dynamic and since you identify the VPN server by its fqdn, so I assume its IP address may also change, all these ways require that the routing table to be used was assigned using mangle rules in chain output.

So let’s make any connection from the router itself to your VPN server use this routing table. To make sure it won’t break once the server migrates to another IP address, we’ll add an address list to track the fqdn:
/ip/firewall/address-list/add list=re-vpn address=vpn.richesineng.com
It should create a dynamic item in the same address list but with the actual IP address as address. If it doesn’t, something is wrong with the DNS setting.

Then, you add a single rule to the very top of chain output in mangle (which is easy as that chain is totally empty now):
/ip/firewall/mangle/add chain=output dst-address-list=re-vpn action=mark-routing new-routing-mark=for-l2tp passthrough=no

It seems this should be it; there is an additional issue, though. A packet sent by the router itself must first be routed using routing table main before it can get to mangle chain output, so it gets assigned some source address depending on the out-interface chosen by main. If mangle assigns a routing-mark to it, the routing is done again, but the source address remains the same. So a src-nat (or masquerade) rule must replace it by the address of the out-interface. This is OK until the uplink connected to that out-interface stops working without the interface going physically down. The connection tracking only removes the connection from its inventory if it has been src-nated using a masquerade rule and if the reply-dst-address assigned by the masquerade rule goes missing. Starlink seems to lease the addresses for 5 minutes (but that’s for CGNAT ones, the public ones may be treated in a different way); if you are happy with the L2TP connection being re-established some 5 minutes after the preferred uplink goes down, nothing else needs to be done. If not, it requires a housekeeping script that removes the address as soon as it detects the failure of the uplink. But again, step by step. Right now I’ll be glad if you make the L2TP work without fancy stuff.

If these settings won’t get the L2TP going, I’ll have to see it online.

Hello Sindy,

I am pleased to say that our VPN connection is working again and I have verified that I can access all local devices in Unalakleet remotely. I am interested in the housekeeping script even though 5 minutes isn’t a huge deal since it’s just for remote access. That’s one problem solved.

Now I just need to get our Sonar Billing instance re-established as it controls the DHCP server and customer account packages. Sonar is integrated with the MikroTik using the secure API on it’s default port of 8729. Connections from sonar come from a source IP of 52.158.209.86 which is static. This IP is standard across all Sonar instances in the USA per documentation. On the Sonar side under the networking settings. The only settings needed for the MikroTik are: WAN IP, Username, Password and Port Number as shown in attached screenshot. This was all working prior just like the VPN so just need some guidance on what to change on the MikroTik. The difference is this is a connection coming into the MikroTik.

Currently I have no filter rules that would block incoming connections, Firewall is wide open until I get it working then I will work on the security.

I assume I will need separate routing table and address list so those I have already created. I would like to use the same interfaces 9 and 10 to keep things simple if possible.

Thank You!

Jaysen

[sindy]

Currently I have no filter rules that would block incoming connections, Firewall is wide open until I get it working then I will work on the security.

A big fat NO for this. Firewall is the first thing to deal with when you connect something directly to the internet, always, no exceptions. The filth from the net is incredibly fast to squat in. Since the VPN works, you should be able to only allow connection to the router itself via the VPN and via LAN as backup and block everything else except what you know needs to be open.

I am pleased to say that our VPN connection is working again and I have verified that I can access all local devices in Unalakleet remotely.

Great.

Now I just need to get our Sonar Billing instance re-established…
The only settings needed for the MikroTik are: WAN IP, Username, Password and Port Number as shown in attached screenshot. This was all working prior just like the VPN so just need some guidance on what to change on the MikroTik. The difference is this is a connection coming into the MikroTik.

I assume I will need separate routing table and address list so those I have already created. I would like to use the same interfaces 9 and 10 to keep things simple if possible.

The router always answers incoming connections from the same address to which they have arrived, but you need to make sure it will also send them via the correct interface because it does not choose a routing table up to a source address automatically. We need to let it use a routing table that contains a single route through the correct interface or at least has such a “correct” route as the most preferred one. If the WAN addresses were static, you could use routing rules for that, but that’s not possible here (unless we would make the lease script update them as well) so we need mangle rules again. The generic way to do this is to use connection marking where we save the information about the in-interface of the initial packet of the incoming connection to the context of that connection maintained by the connection tracking module and use it to assign a routing mark to the response packets. So add a rule to the top of chain prerouting of mangle, and keep the rest of rules in that chain disabled for now so that none of them would overwrite the connection mark assigned by this one.

/ip firewall mangle print chain=prerouting where !dynamic
/ip/firewall/mangle/add chain=prerouting place-before=0 in-interface=sfp28-10-wan10 connection-state=new action=mark-connection new-connection-mark=use-wan10

Then, add a rule to chain output of mangle that will assign an appropriate routing mark based on this connection mark:

/ip/firewall/mangle add chain=output connection-mark=use-wan10 action=mark-routing new-routing-mark=for-l2tp passthrough=no

Now the router should respond via wan10 any connections that arrive to the IP address assigned to wan10.

It is just a quick solution because I am almost asleep; tomorrow I’ll give you some additional information regarding possible redundancy scenarios.

[jaysen]

Thanks again for the help Today and I do agree with the points you made regarding the firewall. Now that VPN is working again, I will do that. I am going to work on fixing the Sonar connection and then setup firewall like I have with all my Oregon sites. Have a good rest. I’ll look for any responses here Tomorrow.

[sindy]

A couple of questions regarding Sonar:

• can it use an FQDN of the router it manages rather than an IPv4 number?
• does it require a continuous connection or it is not an issue if it loses contact for minutes?

[jaysen]

A couple of questions regarding Sonar:

• can it use an FQDN of the router it manages rather than an IPv4 number?
• does it require a continuous connection or it is not an issue if it loses contact for minutes?

Hello Sindy,

Sonar is working again as of last night. Unfortunately it does not have the capability to use FQDN and not sure if they will add that in future updates. I would like that option though. I don’t think a few minutes would be too big of an issue but it’s preferred that it be continuous as it controls the DHCP server leases and plan speeds. When it’s offline I have to do everything manually on the MIkroTik when adding/removing customers.

[gotsprings]

I would look at BigLeaf.

Let them Bond and Distribute Across the feeds back to their VPS. Then on to the internet.

[jaysen]

I would look at BigLeaf.

Let them Bond and Distribute Across the feeds back to their VPS. Then on to the internet.

Thank you for the suggestion. I will look into them. In the meantime, I hope to get the rest of these routes working today and the rest of the terminals into my NMS monitor. I do appreciate everyone’s patience and assistance with all of these issues but progress is being made and client is happy. I have approx 14+ more villages in Alaska coming up so this is quite a learning experience. Unalakleet is the first 2.5 Ghz broadband installation in the state of Alaska (So I’ve been told) so they are going to be the example that we showcase to the rest of the villages. This is huge and people’s lives are going to be positively impacted due to all this work. This will enable access to distance learning opportunities, Access to TeleMedicine and of course the occasional Call of Duty match.

[anav]

gotsprings looks like your trying to put mikrotik wan solutions under the bus LOL. Here I am trying to figure out optimal failover WAN approaches and it turns out I just need to use BigLeaf…
Please send $$$$

[sindy]

Unfortunately it does not have the capability to use FQDN and not sure if they will add that in future updates.

The reason for the question was that the router could update a DNS record in your company DNS if the latter has an API for that, or using the Mikrotik “ip cloud” service, or using some 3rd party dynamic DNS service that has a simple enough API. The Mikrotik service has a limitation of a single IP address per device, a fixed hostname generated from the serial number of the device (so to stay safe when using the FQDN to reach that device from too many other devices or if a VPN connection established towards that FQDN is the only way how those other devices can be reached, you still need another DNS with a CNAME pointing to that fixed FQDN based on device serial number), and I have seen it to be down for days in the past which apparently wasn’t Mikrotik’s fault but for some it was a really tough time.

The use case here would be that if the chosen terminal would fail, the router would use a backup one to update the dynamic DNS with the public address of the public one, and the Sonar could reconnect.

I don’t think a few minutes would be too big of an issue but it’s preferred that it be continuous as it controls the DHCP server leases and plan speeds. When it’s offline I have to do everything manually on the MIkroTik when adding/removing customers.

Well, my question was whether it was a provisioning tool (which your explanation seems to confirm) or whether it directly controlled the traffic (in terms of e.g. cutting off a client if they run off their quota, I have no clue what your business offer is up there). A provisioning tool only has to work when you actively use it, so you can manually change the address if the currently configured one changes or dies; a traffic policing tool needs a constant connection.

What you can do to make VPN traffic switch to a backup WAN far sooner than the VPN client detects an outage and re-establishes the connection via the backup WAN is to establish two VPN tunnels, one using strictly the preferred WAN and the other one using strictly the backup one, and let one of these VPN tunnels be a backup for the other one. This approach doesn’t suffer from the issue of src-nated connection surviving an outage of the uplink until the lease is lost as mentioned earlier, but it requires a compatible setup at the VPN server side.

With such a setup, you can also set up port forwarding for Sonar on one of your machines in a datacenter with a static public address to the CCR in Unalakleet via this pair of VPN tunnels. But it is still only a protection against failure of a terminal, it cannot handle an absence of a functional satellite within reach or a failure of the gateway machine.

Totally unrelated, you have mentioned that people keep getting security warnings. Some web sites like to handle requests within the same application session by different servers at their end, but check whether all the requests come from the same public address at client side and either reject them or at least issue security warnings if not. To avoid this, the hash in the per-connection-classifier matcher must be calculated solely from the src-address. This means that all outgoing connections of a given LAN address will always get mapped to the same WAN address (unless it fails of course), so the traffic will not be distributed as evenly as when you hash both addresses and ports, but it may be bearable (and also controllable to some extent, you can change the addresses of the clients that generate most traffic to evenly distribute them over the WANs manually).

[jaysen]

Unfortunately it does not have the capability to use FQDN and not sure if they will add that in future updates.

The reason for the question was that the router could update a DNS record in your company DNS if the latter has an API for that, or using the Mikrotik “ip cloud” service, or using some 3rd party dynamic DNS service that has a simple enough API. The Mikrotik service has a limitation of a single IP address per device, a fixed hostname generated from the serial number of the device (so to stay safe when using the FQDN to reach that device from too many other devices or if a VPN connection established towards that FQDN is the only way how those other devices can be reached, you still need another DNS with a CNAME pointing to that fixed FQDN based on device serial number), and I have seen it to be down for days in the past which apparently wasn’t Mikrotik’s fault but for some it was a really tough time.

The use case here would be that if the chosen terminal would fail, the router would use a backup one to update the dynamic DNS with the public address of the public one, and the Sonar could reconnect.

Sonar Support has said that everything that I see in Sonar can be controlled via webhooks so I am wondering if there is some way to automate updating the IP address via a webhook should it change. I don’t have much experience with that but I will reach out to Sonar to see if that is possible.

I don’t think a few minutes would be too big of an issue but it’s preferred that it be continuous as it controls the DHCP server leases and plan speeds. When it’s offline I have to do everything manually on the MIkroTik when adding/removing customers.

Well, my question was whether it was a provisioning tool (which your explanation seems to confirm) or whether it directly controlled the traffic (in terms of e.g. cutting off a client if they run off their quota, I have no clue what your business offer is up there). A provisioning tool only has to work when you actively use it, so you can manually change the address if the currently configured one changes or dies; a traffic policing tool needs a constant connection.

Correct, Sonar is an all-in-one solution that handles provisioning and it ties in with the Mikrotik and a Preseem appliance to apply shaping to keep customers on their provision package speeds as well as cut off delinquent customers. The only thing Preseem does not do is throttle down a customer when they exceed their data cap. However, I am working with Sonar and have a set of scripts that work with address lists and queues to do the Throttling which I will be implementing at a later time once all the rest of this is setup.

The way I currently handle any IP change is by creating multiple inline device and dhcp server entries in sonar. I would add all the terminal IP addresses in there and leave all disabled except for the current active one. Then all I need to do should a change happen is switch between them. Sonar says this is fine to do but to not have more than one enabled at a time as that could cause problems with having duplicate leases. I’ve only had to do it once when we still had OneWeb and a backup Starlink set as a failover. The OneWeb terminal failed and the Mikrotik failed over to the Starlink as expected. I then disabled the OneWeb and enabled the Starlink in Sonar and all was good.

What you can do to make VPN traffic switch to a backup WAN far sooner than the VPN client detects an outage and re-establishes the connection via the backup WAN is to establish two VPN tunnels, one using strictly the preferred WAN and the other one using strictly the backup one, and let one of these VPN tunnels be a backup for the other one. This approach doesn’t suffer from the issue of src-nated connection surviving an outage of the uplink until the lease is lost as mentioned earlier, but it requires a compatible setup at the VPN server side.

With such a setup, you can also set up port forwarding for Sonar on one of your machines in a datacenter with a static public address to the CCR in Unalakleet via this pair of VPN tunnels. But it is still only a protection against failure of a terminal, it cannot handle an absence of a functional satellite within reach or a failure of the gateway machine.

We have been exploring this option of providing a static via VPN tunnels and do own several blocks of IP addresses that we could use but we are not quite ready to set that up yet. We are in the process of building a new NOC in Anchorage that will house monitoring systems and the ability to provide these static addresses. With so many projects going on in multiple states, that process has been slowed down.

Totally unrelated, you have mentioned that people keep getting security warnings. Some web sites like to handle requests within the same application session by different servers at their end, but check whether all the requests come from the same public address at client side and either reject them or at least issue security warnings if not. To avoid this, the hash in the > per-connection-classifier > matcher must be calculated solely from the > src-address> . This means that all outgoing connections of a given LAN address will always get mapped to the same WAN address (unless it fails of course), so the traffic will not be distributed as evenly as when you hash both addresses and ports, but it may be bearable (and also controllable to some extent, you can change the addresses of the clients that generate most traffic to evenly distribute them over the WANs manually).

Could you provide some assistance with getting this going? I have read a bit about PCC and started working on it with the help of anov who was the first to respond to my post. I do have many mangle rules and static routes created but they are currently disabled until I am certain they are configured correctly. I think my most recent config that I provided will show them but if not I can provide it again.

Thank You!

[sindy]

Could you provide some assistance with getting this going? I have read a bit about PCC and started working on it with the help of anov who was the first to respond to my post. I do have many mangle rules and static routes created but they are currently disabled until I am certain they are configured correctly. I think my most recent config that I provided will show them but if not I can provide it again.

I can assist, just bear in mind the time shift. But your last config posted should work if you enable the mangle rules and the routes and redo the QoS-related mangle rules in chain forward so that they would not use connection marks, because they overwrite those assigned in prerouting.

To do so, take all the match conditions of the mangle forward rule that assigns the connection mark, add them to the mangle forward rule that currently translates that connection mark to a packet mark, remove the match on connection mark from the latter rule, and remove the former rule completely. Do this 4 times and that’s it.

If no route is active in a routing table indicated by the routing mark, the system uses routing table main instead (unless a routing rule explicitly prohibits that). So the existing configuration does contain a backup for the case that a single Starlink terminal stops working.

As I wrote earlier, it is better to replace both-addresses by src-address in the per-connection-classifier.

[jaysen]

Could you provide some assistance with getting this going? I have read a bit about PCC and started working on it with the help of anov who was the first to respond to my post. I do have many mangle rules and static routes created but they are currently disabled until I am certain they are configured correctly. I think my most recent config that I provided will show them but if not I can provide it again.

I can assist, just bear in mind the time shift. But your last config posted should work if you enable the mangle rules and the routes and redo the QoS-related mangle rules in chain forward so that they would not use connection marks, because they overwrite those assigned in prerouting.

To do so, take all the match conditions of the mangle forward rule that assigns the connection mark, add them to the mangle forward rule that currently translates that connection mark to a packet mark, remove the match on connection mark from the latter rule, and remove the former rule completely. Do this 4 times and that’s it.

If no route is active in a routing table indicated by the routing mark, the system uses routing table main instead (unless a routing rule explicitly prohibits that). So the existing configuration does contain a backup for the case that a single Starlink terminal stops working.

As I wrote earlier, it is better to replace both-addresses by src-address in the per-connection-classifier.

Hi Sindy,

It might be best to start fresh with the mangle rules. The ones I had originally created were based on another posters advice which is different. While I appreciate their help. What you and I have done up to this point, things are beginning to work. Since they are all disabled, It might be easier for me to follow what you are saying if I start fresh. I’ve been pulled in all different directions the last few days with multiple clients so my train of thought on this is a little off. I am thinking we get rid of all the disabled ones in the screenshot or at least the ones that are not needed. Then make corrections to the ones I do need based on what you are saying. It’ll make more sense to me that way. I appreciate your understanding as this is new to me.

Thank You

[gotsprings]

gotsprings looks like your trying to put mikrotik wan solutions under the bus LOL. Here I am trying to figure out optimal failover WAN approaches and it turns out I just need to use BigLeaf…
Please send $$$$

Speedify has a lab that ran that at one point.

But Big Leaf seams a bit more business oriented.

[sindy]

It might be best to start fresh with the mangle rules.

Let’s go that way if you like, but I’d prefer a more interactive communication channel than the forum. This kind of “share the wisdom” sites is great to describe typical setups and principles so that others could follow them, but there are so many topics dealing with load distribution&backup here that I can’t see any point in documenting the process here for the 500th time, so only the waste of time remains. So please consider following this post.

[holvoetn]

FWIW the PCC youtube vid made by MT is quite good.
https://www.youtube.com/watch?v=nlb7XAv57tw

Used it again to clean up an AC3 LTE setup for PCC sharing across VDSL and LTE.
Only, THIS time I disabled the subtitles which all of a sudden made me see a couple of important things I missed the previous time I saw that video.
And now it works stable and reliably

Just saying …

[jaysen]

Hello Sindy,

I did more work on it last night. I updated and activated all the mangle rules plus added a few more. I also added in all the masquerade rules for each interface. So far things appear to be working great. All interfaces are responding to ping and SNMP queries so I am now monitoring them in our NMS. Sonar and VPN connections are still working and no complaints so far from the village that internet isn’t working. I’m out of the office most of the morning but plan to work more on it later Today.

I did setup some basic firewall rules using the section on that in the Wiki but it still needs a little more work which I also plan to complete Today. I’ll post an updated config later tonight and you can make any suggestions on how to improve things.

A question I have on that lease script. I assume that is meant to be placed inside the script section in each DHCP client? It makes sense to me that it would be appropriate to put it there but figured I would ask.

I do have a remote windows box at the tower as a backup way into the network in case VPN fails for any reason. I am going to play around with some DDNS clients then work with Sonar to see if they have a backend way of making things work with an FQDN.

Thanks again for all the help

[jaysen]

FWIW the PCC youtube vid made by MT is quite good.
https://www.youtube.com/watch?v=nlb7XAv57tw

Used it again to clean up an AC3 LTE setup for PCC sharing across VDSL and LTE.
Only, THIS time I disabled the subtitles which all of a sudden made me see a couple of important things I missed the previous time I saw that video.
And now it works stable and reliably

Just saying …

Awesome, I’ll take a look at it. Thanks for sharing

[anav]

FWIW the PCC youtube vid made by MT is quite good.
https://www.youtube.com/watch?v=nlb7XAv57tw

Used it again to clean up an AC3 LTE setup for PCC sharing across VDSL and LTE.
Only, THIS time I disabled the subtitles which all of a sudden made me see a couple of important things I missed the previous time I saw that video.
And now it works stable and reliably

Just saying …

Actually, I just viewed this video and I think it has some serious flaws LOL, and no I am not just saying that to contradict holvoe, as much as fun as that is. .

[sindy]

A question I have on that lease script. I assume that is meant to be placed inside the script section in each DHCP client? It makes sense to me that it would be appropriate to put it there but figured I would ask.

To be precise, it is meant to be placed just once into the /system script section, and its name to be placed to the script item of each DHCP client. You could put the complete script to the script item of each DHCP client (that item is interpreted in a contextual way, i.e. if the contents is a single word, it is interpreted as a script name, otherwise it is interpreted as a script code), but that would be a maintenance nightmare.

Referring to the script this way still allows you to create a modified instace for some special purpose or just for testing new features and let just one of the DHCP clients use it.

[holvoetn]

Ok Anav.
Eagerly awaiting your version then …
Let’s see if it is better.