Multi-WAN Load Balancing Starlink issue

jaysen · Tue Nov 14, 2023 1:52 am

Hello,

I am a Network Engineer managing a WISP setup in a remote Alaskan tribal village. Last week I was on-site and replaced our middle-mile Internet connection from OneWeb which was very costly and underperforming with 10 high performance Starlink terminals. I have set the IP policy to public IP on all the terminals and using the DHCP client on each of the 10 interfaces on our MikroTik router and configured to automatically add the default routes since Starlink does not offer true static IP addresses and we have no other ISP options available in this very remote area. I see it automatically setup ECMP load balancing and that appears to be working as I can see traffic being distributed across all 10 WAN interfaces and data usage also appears almost equal on each terminal in the Starlink portal.

The problem we are now facing is this setup broke the secure connection between the MikroTik and our Sonar billing instance which controls the LAN DHCP server. Our L2TP/IPSEC VPN no longer works. I figure this is because how traffic is routed out one WAN interface and comes back through a different WAN interface therefore the source IP is keeps changing.

I've done a bit of research on this and also watched a few YouTube videos of MikroTik engineers setting up MultiWAN scenarios using PCC with routing marks and separate routing tables for each WAN interface. The difference is in the videos, only 2-3 WAN connections were used and the ISPs were different. However, I have not been as successful with getting this to work as all of the IP addresses Starlink as assigned are in the same subnet and have the same gateway. I know using 10 Starlinks might seem a bit extreme but that was not my decision.

I would like to ask for some guidance in getting this setup to work and fix our secure connections so the village can begin billing customers. Our company is also willing to consider hiring a MikroTik expert to work with us on a consultancy basis to get this to work if there is someone on here available and has proven experience with this scenario. I do consider myself knowledgeable with MikroTik and have formal education in Networking but will admit this is my first attempt at setting up something of this magnitude. I am a quick learner and love all things MikroTik and Networking related so I am willing to listen and learn from anyone on here who knows more about this than I do.

I am attaching a copy of our config with sensitive info removed as well as screenshots of our System > Resources and IP > Routes for reference.

Thank you all for your time and expertise.

Jaysen

anav · Tue Nov 14, 2023 5:27 am

Okay you have other issues.
Your way of separating users and use of vlans and bridge is very confusing.
You have only one pool but then many addresses, whats going on.

Looks like you should have 3 vlans, subscribers group1, subscribers group2 and servers.
One bridge-lan is fine, assign the three vlans (2,4,6) to the bridge, etc.
3 pools, 3 ip dhcp servers and 3 ip dhcp server networks to go along with three IP addresses with interface being the applicable vlan!!!

All are a trunk port on sfp28-12 to the switch

/interface bridge port
add bridge=bridge-lan ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=sfp28-12
/interface bridge vlan
add bridge=bridge-lan tagged=bridge-lan,spf28-12 vlan-ids=2,4,6

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I think what you need to do is for all the default routes display them as such.....

add dst-address=0.0.0.0/0 gateway=98.97.96.1%spf28-1-wan1 routing-table=main
add dst-address=0.0.0.0/0 gateway=98.97.96.1%spf28-2-wan2 routing-table=main
add dst-address=0.0.0.0/0 gateway=98.97.96.1%spf28-3-wan3 routing-table=main
etc.........

Then MANGLE rules, have no clue what your mangle rules are doing???
AT a minimum for PCC you will need
-10 rules to mark packets coming from the LAN sources that require PCC.
-10 rules to mark connections for those packets

Example:
Create one PCC mangle rule for each WAN connection
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface-list=NEED_PCC new-connection-mark=to_wan1 passthrough=yes per-connection-classifier=both-addresses:10/0
....
....
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface-list=NEED_PCC new-connection-mark=to_wan10 passthrough=yes per-connection-classifier=both-addresses:10/9

Then additional 10 routes for PCC marked traffic!
add dst-address=0.0.0.0/0 gateway=98.97.96.1%spf28-1-wan1 routing-table=to_wan1
...
...
add dst-address=0.0.0.0/0 gateway=98.97.96.1%spf28-10-wan10 routing-table=to_wan10

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Thats very basic, if you have traffic externally arriving on a WAN then it needs to go out the same WAN and we will need additional mangle rules for that.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

If you have some traffic originating on the router that should not PCCs but needs to go out a specific WAN that needs to be communicated and accounted for.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Finally you have to ask yourself the question, what happens when one of the WAN connections is NOT available.
THe associated LAN traffic assigned PCC to that route will be screwed and will not pass.

You have two options.................. Assign additional routes
FOR WAN1 - If 1 fails go to 2, then 3, then 4, then 5
FOR WAN2 - if 2 fails go to 1, then 2, then3, then4 etc....

Within each TABLE, the the sub routes are differentiated by distance.....

Example take the to_wan1 rule, give it a distance 2 and then add 9 more rules, so the first and last would look like
add dst-address=0.0.0.0/0 gateway=98.97.96.%spf28-1-wan1 routing-table=to_wan1 distance=2
....
....
add dst-address=0.0.0.0/0 gateway=98.97.96.1%spf28-10-wan10 routing-table=to_wan1 distance=11

THe problem with this approach is that you dont spread the load out to the other WANS you pass a failure onto another TABLE and thus WAN.
The positive side is that this is much easier to config then option B and thats 100 route lines easy and manageable LOL.

Option B, consists of spreading out the PCCs to all the other WANs and its a exploding matrix with considering starting with 10 wans LOL

What we are doing is taking the 1/10 approach alloted for each WAN and dividing its responsibility by 9 other WANs. Each table/each PCC gets 1/90 of the traffic so when one fails we divy up the 1/10 (any singular WAN gets) by giving the remaining wans 1/90th each.. 9/90 = 1/10

10 wans x 9 alternative paths = 90 PCCs
AthenB PC 90/0
AthenC
AthenD
AthenE
AthenF
AthenG
AthenH
AthenI
AthenJ PCC 90/8

TO the last one.
JThenA PCC 90/72
JthenB
JthenC
JtheD
JthenE
JthenF
JthenG
JthenH
JthenI PCC 90/89

Thus considering each one has two routes thats what, approx 180 routes total.......

add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp1-WAN1 routing-table=useAthenB distance=2
add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp2-WAN2 routing-table=useAthenB distance=3

add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp1-WAN1 routing-table=useAthenC distance=2
add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp3-WAN3 routing-table=useAthenC distance=3

....
....
add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp1-WAN1 routing-table=useAthenJ distance=2
add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp10-WAN10 routing-table=useAthenJ distance=3

....
....

add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp10-WAN10 routing-table=useJthenH distance=2
add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp8-WAN8 routing-table=useJthenH distance=3

add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp10-WAN10 routing-table=useJthenI distance=2
add dst-address=0.0.0.0/0 gateway=gatewayIP%sfp9-WAN9 routing-table=useJthenI distance=3

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Note that just the routes, dont forget the 90 mangle rules one for each table

jaysen · Tue Nov 14, 2023 10:04 am

Hello Anav,

Thank you very much for responding. I really do appreciate it. I admit I could have done better at explaining the network setup which I will attempt to do now and I apologize in advance for the lengthy post. I tend to over explain and add extra detail in hopes of others better understanding our issues.

Our company took over management of this network about a year ago. The internal network was designed by another network engineer and I do agree and want to change the VLANs. When we first took over, an Adtran NetVanta router was in place but I switched it out for the MikroTik due to our billing platform (Sonar) having some features we needed which required the MikroTik to implement as it's the only router platform that Sonar integrates with. Not to mention that I love MikroTik and Adtran GUI is crap.

We originally only had one WAN connection to a Satellite middle-mile provider (OneWeb) which is garbage for the price they were charging the village. Over $36,000/month for a 100Mbps connection and trying to share that with approx 100+ customers who connect via Cambium cnRanger LTE platform and many who thought they could stream Netflix and also game on that connection. Latency was very high (about 500-600ms consistently) and the link would constantly drop. The speeds we were getting when the connection was up made dial-up internet seem like broadband. I couldn't even send a single picture via iMessage on my phone over WiFi without it dropping. Needless to say, it was terrible.

The Tribal Council decided to dump OneWeb and purchase 10 high priority Starlink subscriptions and 10 terminals on their own thinking they would get 10x the speed by bonding them all together but later found out that it wouldn't work that way. They were also told by Starlink they would get static IP addresses which they also learned that wasn't true. Now we have the situation we are in with 10 WANs and 10 dynamic IP addresses which Starlink has said are DHCP reserved and won't change unless they have major network maintenance where they would need to failover to different ground stations.

For the LAN network. The previous engineer designed it as follows:

VLAN 1: 10.130.1.0/24 network assigned statically to Core infrastructure (Servers, Router, Switch, Monitoring system)

VLAN 2: Subscriber network consisting of the two /23 subnets below:
The bridge interface is assigned the sfp28-12 port which handles both of the VLANs and trunks to a port on an Adtran Netvanta switch that connects to all the servers and Cambium LTE equipment to the customers. The DHCP server is also running on this bridge interface and handles only the 10.130.2.0/23 subnet. All of that is working with customers getting leases and able to browse the internet but again I agree it could be setup better.

- 10.130.2.0/23 is for customer routers inside their homes and is assigned via DHCP Pool setup in Sonar which communicates with the Mikrotik to assign static leases.

- 10.130.4.0/23 assigned statically to customer subscriber modules (dishes) that are mounted on their homes. I chose static addresses due to limitations with the Cambium cnRanger platform. There is no option to configure external DHCP server per Cambium Support and no future plans to add that so addresses are assigned statically to the LTE SIM cards. There is another way to set this up and I have documentation from Sonar to do this which I am working on. Our village techs are not networking professionals so my goal is to simplify it so all they need to do is install the equipment, assign the equipment to the customer account in Sonar and it does everything else automatically. The less phone calls I get from the techs, the better since this is the 1st of 14 villages we are working on.

The current mangle rules were setup using Sonar documentation. These rules also use queues and address lists for the data packages that are currently offered and they work with Sonar and a Preseem QoE appliance to control speeds and data caps so I do understand why those are confusing. Those were setup back when we still had only one WAN connection and no load balancing.

The network itself has been working but now with the 10 Starlinks. It sounds like I need to change my default routes and add the mangle rules that you described for PCC to work. A question I have is should I disable the setting in each WAN DHCP client that automatically adds the default routes or leave it enabled and still add the default routes that you described? My gut is telling me to disable it but I figured I would ask anyway to be safe since I am roughly 2000 miles away and any mess up is a potential flight to the village during the winter. Safe mode is nice but sometimes I forget to turn it on.

Regarding the failover issue. I'm going to need to think about that. Before changing to the 10 starlinks. I did have one starlink acting as a failover for the OneWeb and it was setup recursively using a video I found on YouTube from a MikroTik expert that calls himself "The Network Berg". This worked surprisingly well and easy to implement with only 2 WANs. Only problem with that was the secondary connection stayed down until a failover occurred which is not what I want in this new setup.

It's looking like I have lots of work to do and again I thank you very much for pointing me in the right direction. I think I have a better understanding but I'm going to review all of this with our chief engineer and begin implementing it. I'll let you know how it goes. If after reading this novel of a post you find more that needs to be done. Do let me know.

Jaysen

anav · Tue Nov 14, 2023 2:45 pm

my apologies jaysen, the information I provided is at the edge of my scope of expertise, the rest is a tad over my head.

jaysen · Tue Nov 14, 2023 2:56 pm

No worries Anav, What you provided is a giant step in the right direction. I built out all the default routes and mangle rules but kept them disabled until I can have a village tech at our tower site to assist remotely should anything go wrong. I do plan to test this today to see if it all works. Another question I have that maybe you could answer. The NEED_PCC interface list that is referenced in the mangle rule. I created that list and need to know if I put all the WAN interfaces in that list? It makes sense that I would but wanted to get your thoughts.

Thank you for the assistance.

Jaysen

anav · Tue Nov 14, 2023 3:33 pm

Hi Jaysen, fair question as I had not defined what that was anywhere.
Since it was not clear to me which LAN side entities were getting PCCd so to speak I left it as an new interface list. It has nothing to do with the WANS....

So once you define what subnets will be included in the PCC you can add them to that interface list.
There may be some LAN subnets going out a totallly different WAN (LTE etc) or you want them to go out a specific WAN regardless (starlink 5 for example) and these would NOT be included in the interface list.
If you have some users that are in a subnet that is getting PCCd but you dont want them PCC then you will need to also create a firewall address list of those users.
If you have some users that are not in a subnet that is getting PCCd but you want them PCCd then you will need to also create a firewall address list.

In other words, put all subnets that will be shared amongst the WANs in one interface list and ensure you create firewall address lists for user exceptions.

Feel free to share your mangle rules and routes for viewing ....... use code block to keep it short. ( the black square with white square brackets on the same line as Bold and Underline etc.)

sindy · Wed Nov 15, 2023 12:32 am

So I finally got to reading through it, and I am trying to put the bits together.

You wrote that the public IP addresses assigned by Starlink with the high priority subscription were changing but now they are not, but if I get it right, the L2TP/IPsec tunnel still keeps disconnecting (or is not connecting at all)?

Second, I cannot see any L2TP configuration in the export nor a dst-nat rule for IPsec ports, so I figure the L2TP/IPsec client is running in Alaska on some device connected to the 2216's LAN and connecting to a server at your HQ location, is that correct?

ECMP used to be working on RouterOS 6 where a routing cache existed, so if a connection started using a particular gateway, it continued using it until the routing cache was flushed. I'm not sure whether ECMP can be used on RouterOS 7, which uses a kernel that does not support routing cache any more, when src-nat is engaged, because I don't know how a Starlink terminal treats packets with a wrong source address. But since you encounter problems, I would first dig in this direction. The thing is that the firewall rules assign the "external" address of a src-nated connection (officialy, the reply-dst-address) once for good while handling the initial packet of the connection. So if ECMP sends a subsequent packets belonging to the same connection via another gateway, that packets will leave with an address that does not belong to the one of the out-interface, so if the Starlink network is picky about it, it will not reach the destination.

If my assumptions above regarding the L2TP/IPsec topology, Starlink behavior, and ECMP behavior in ROS 7 are correct, you need to make sure that the L2TP/IPsec connection keeps using the same WAN all the time; to do that, you need to make it use a dedicated routing table that prefers a particular WAN and only uses another one if the primary one fails, so that the tunnel could re-establish.

Before proceeding, please confirm or correct my understanding.

jaysen · Wed Nov 15, 2023 12:41 am

Hello Anav,

I have built the routes and the mangle rules as you explained in your first reply. I've left them disabled for now until I know they are correct. I am putting them in code blocks below for you to take a look at and make any suggestions.

I created the 10 mangle rules for connection marks below. You had said I also need 10 mangle rules to mark the packets. Would you be able to provide some guidance on how to setup the packet marks? I didn't see that in your post. Thank you kindly. The other mangle rules in there are for controlling speeds in Sonar and were created using their documentation.

[b]Mangle Rules:[/b]

/ip firewall mangle
add action=mark-connection chain=prerouting comment="For PCC" \
    connection-mark=no-mark disabled=yes dst-address-type=!local \
    in-interface-list=NEED_PCC new-connection-mark=to_wan1 passthrough=yes \
    per-connection-classifier=both-addresses:10/0
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan2 passthrough=yes per-connection-classifier=\
    both-addresses:10/1
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan3 passthrough=yes per-connection-classifier=\
    both-addresses:10/2
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan4 passthrough=yes per-connection-classifier=\
    both-addresses:10/3
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan5 passthrough=yes per-connection-classifier=\
    both-addresses:10/4
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan6 passthrough=yes per-connection-classifier=\
    both-addresses:10/5
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan7 passthrough=yes per-connection-classifier=\
    both-addresses:10/6
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan8 passthrough=yes per-connection-classifier=\
    both-addresses:10/7
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan9 passthrough=yes per-connection-classifier=\
    both-addresses:10/8
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan10 passthrough=yes per-connection-classifier=\
    both-addresses:10/9
add action=mark-connection chain=forward comment="For Sonar" \
    dst-address-list="Standard Internet" new-connection-mark=\
    Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list=\
    "Standard Internet"
add action=mark-connection chain=forward dst-address-list=\
    "High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=High_Capacity_Traffic_Up passthrough=yes \
    src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" connection-mark=\
    Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=\
    yes
add action=mark-packet chain=forward connection-mark=\
    High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up \
    passthrough=yes

[b]Routes:[/b]

/ip route
add disabled=no dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=no dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-1-wan1 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-2-wan2 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-3-wan3 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-4-wan4 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-5-wan5 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-6-wan6 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-7-wan7 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-8-wan8 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-9-wan9 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-10-wan10 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-1-wan1 \
    routing-table=to_wan1 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-2-wan2 \
    routing-table=to_wan2 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-3-wan3 \
    routing-table=to_wan3 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-4-wan4 \
    routing-table=to_wan4 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-5-wan5 \
    routing-table=to_wan5 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-6-wan6 \
    routing-table=to_wan6 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-7-wan7 \
    routing-table=to_wan7 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-8-wan8 \
    routing-table=to_wan8 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-9-wan9 \
    routing-table=to_wan9 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-10-wan10 \
    routing-table=to_wan10 suppress-hw-offload=no
    
[b]Routing Tables:[/b]
    
/routing table
add fib name=to_wan1
add fib name=to_wan2
add fib name=to_wan3
add fib name=to_wan4
add fib name=to_wan5
add fib name=to_wan6
add fib name=to_wan7
add fib name=to_wan8
add fib name=to_wan9
add fib name=to_wan10

I came across this post that looks like it might have some useful information and wanted to get your thoughts on it. viewtopic.php?t=86071

The poster suggests changing passthough=yes to passthrough=no. I also currently have only one srcnat masquerade rule to a WAN interface list that has all my WAN interfaces in. Will that work or should I have a separate masquerade rule for each WAN interface?

Back to my question about the NEED_PCC list. It's my understanding that PCC will solve my issue with secure connections including our L2TP/IPSEC VPN not working. Customers are currently not able to visit HTTPS sites without getting privacy warnings as they have reported to me and our VPN client on the MikroTik router is unable to connect to office VPN server in Oregon when it used to work when we only had 1 WAN connection. Am I correct in my thinking that all my LAN subnets will need PCC and therefore I should add them all to that list? I would think all customers should get PCCd since pretty much everything we do online these days is secure and most modern browsers default to it. Thoughts on this?

Jaysen

Amm0 · Wed Nov 15, 2023 12:46 am

ECMP used to be working on RouterOS 6 where a routing cache existed, so if a connection started using a particular gateway, it continued using it until the routing cache was flushed. I'm not sure whether ECMP can be used on RouterOS 7, which uses a kernel that does not support routing cache any more, when src-nat is engaged

ECMP works fine V7 – it relies on connection tracking to store the routing decision for future packets. I'm not sure that's the problem here. I suspect it just the variable the speed of starlink sats moving is going to be same across all 10 terminals (e.g. one is slow, all will likely be slow)

The LTE being 600ms would imply the cell networks backhaul is using GEO sat, which at a full transponder would be ~50-100Mb capacity range.

jaysen · Wed Nov 15, 2023 1:14 am

So I finally got to reading through it, and I am trying to put the bits together.

You wrote that the public IP addresses assigned by Starlink with the high priority subscription were changing but now they are not, but if I get it right, the L2TP/IPsec tunnel still keeps disconnecting (or is not connecting at all)?

Second, I cannot see any L2TP configuration in the export nor a dst-nat rule for IPsec ports, so I figure the L2TP/IPsec client is running in Alaska on some device connected to the 2216's LAN and connecting to a server at your HQ location, is that correct?

ECMP used to be working on RouterOS 6 where a routing cache existed, so if a connection started using a particular gateway, it continued using it until the routing cache was flushed. I'm not sure whether ECMP can be used on RouterOS 7, which uses a kernel that does not support routing cache any more, when src-nat is engaged, because I don't know how a Starlink terminal treats packets with a wrong source address. But since you encounter problems, I would first dig in this direction. The thing is that the firewall rules assign the "external" address of a src-nated connection (officialy, the reply-dst-address) once for good while handling the initial packet of the connection. So if ECMP sends a subsequent packets belonging to the same connection via another gateway, that packets will leave with an address that does not belong to the one of the out-interface, so if the Starlink network is picky about it, it will not reach the destination.

If my assumptions above regarding the L2TP/IPsec topology, Starlink behavior, and ECMP behavior in ROS 7 are correct, you need to make sure that the L2TP/IPsec connection keeps using the same WAN all the time; to do that, you need to make it use a dedicated routing table that prefers a particular WAN and only uses another one if the primary one fails, so that the tunnel could re-establish.

Before proceeding, please confirm or correct my understanding.

Hello Sindy,

Regarding the Starlink public IP addresses. We were told by Starlink that they do not offer static IP addresses. The public addresses are assigned by DHCP but they said they are static leases or "sticky" as they called it. The only time they could change is if they had to failover our connections to another ground station for whatever reason. Otherwise, they said the IP addresses won't change. I hope that clears that up. Sorry if it was confusing when I first explained it.

The L2TP client is running directly on the Mikrotik router in Alaska and it connects to another Mikrotik router at our HQ in Oregon running the L2TP server. That was working great prior to changing over to the 10 starlinks. It worked with our previous OneWeb connection and also with just 1 starlink after we got rid of OneWeb. We have been operating on 1 starlink for the past few months temporarily until last week we made the switch to 10 priority dishes which broke it. The logs are now showing it trying to connect but it always fails, then retries, then fails again. It never does successfully connect

You are correct, we have no dst-nat rule for L2TP setup and currently no filter rules at this time that would block any incoming connections. I decided to keep it open for now while I am trying to get this load balancing and PCC working. Then I will go back and work on securing it. I do have a few dst-nat rules that point to some servers on the local server subnet but I would like to get rid of those once the VPN is working again.

Thank You,

Jaysen

sindy · Wed Nov 15, 2023 1:35 am

ECMP works fine V7 – it relies on connection tracking to store the routing decision for future packets.

Thanks. I was wondering about this for some time already.

The LTE being 600ms would imply the cell networks backhaul is using GEO sat, which at a full transponder would be ~50-100Mb capacity range.

That was the previous satellite operator, and I can easily imagine they haven't got any LEO satellites that high to the north. Starlink has a few satellites on polar orbits specially to cover these areas.

sindy · Wed Nov 15, 2023 1:43 am

The L2TP client is running directly on the Mikrotik router in Alaska and it connects to another Mikrotik router at our HQ in Oregon running the L2TP server.

Then something must have gone wrong in the process of posting the configuration, because I can see no /interface l2tp-client section there.

There are also multiple routing tables but just two static routes in the configuration.

we have no dst-nat rule for L2TP setup

If the L2TP client is running directly on the 2216, no dst-nat rule is necessary, it's just that there absence of the client configuration in the export made me think it is running on another device so I was thinking of possible topologies (an L2TP server in LAN in Alaska would require a dst-nat rule).

Sorry, I ran out of today here, so I'll be back online in 8-10 hours from now.

jaysen · Wed Nov 15, 2023 2:01 am

The L2TP client is running directly on the Mikrotik router in Alaska and it connects to another Mikrotik router at our HQ in Oregon running the L2TP server.
Then something must have gone wrong in the process of posting the configuration, because I can see no /interface l2tp-client section there.

There are also multiple routing tables but just two static routes in the configuration.

we have no dst-nat rule for L2TP setup
If the L2TP client is running directly on the 2216, no dst-nat rule is necessary, it's just that there absence of the client configuration in the export made me think it is running on another device so I was thinking of possible topologies (an L2TP server in LAN in Alaska would require a dst-nat rule).

Sorry, I ran out of today here, so I'll be back online in 8-10 hours from now.

I had removed some sensitive data in the config before posting it. That's likely why it wasn't in there. The original config contained complete address lists with customer info in them so I removed all that. Here is the most recent config with address list data removed.

# 2023-11-14 14:51:49 by RouterOS 7.12
# software id = DVNU-PTAC
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HE808NY9TF8
/interface bridge
add admin-mac=48:A9:8A:85:65:FB auto-mac=no comment="Link to NV1560 Port 1" \
    name=bridge-lan
add name=bridge-oob
/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=qsfp28-1-1 ] disabled=yes
set [ find default-name=qsfp28-1-2 ] disabled=yes
set [ find default-name=qsfp28-1-3 ] disabled=yes
set [ find default-name=qsfp28-1-4 ] disabled=yes
set [ find default-name=qsfp28-2-1 ] disabled=yes
set [ find default-name=qsfp28-2-2 ] disabled=yes
set [ find default-name=qsfp28-2-3 ] disabled=yes
set [ find default-name=qsfp28-2-4 ] disabled=yes
set [ find default-name=sfp28-1 ] comment="Link to Starlink Terminal 1" name=\
    sfp28-1-wan1
set [ find default-name=sfp28-2 ] comment="Link to Starlink Terminal 2" name=\
    sfp28-2-wan2
set [ find default-name=sfp28-3 ] comment="Link to Starlink Terminal 3" name=\
    sfp28-3-wan3
set [ find default-name=sfp28-4 ] comment="Link to Starlink Terminal 4" name=\
    sfp28-4-wan4
set [ find default-name=sfp28-5 ] comment="Link to Starlink Terminal 5" name=\
    sfp28-5-wan5
set [ find default-name=sfp28-6 ] comment="Link to Starlink Terminal 6" name=\
    sfp28-6-wan6
set [ find default-name=sfp28-7 ] comment="Link to Starlink Terminal 7" name=\
    sfp28-7-wan7
set [ find default-name=sfp28-8 ] comment="Link to Starlink Terminal 8" name=\
    sfp28-8-wan8
set [ find default-name=sfp28-9 ] comment="Link to Starlink Terminal 9" name=\
    sfp28-9-wan9
set [ find default-name=sfp28-10 ] comment="Link to Starlink Terminal 10" \
    name=sfp28-10-wan10
set [ find default-name=sfp28-11 ] comment="Link to OOB PC" name=sfp28-11-oob
set [ find default-name=sfp28-12 ] comment="Link to Adtran Switch Port 1" \
    name=sfp28-12-lan
[b] /interface l2tp-client
add connect-to=vpn.richesineng.com disabled=no name=RE use-ipsec=yes user=\
    unalakleet [/b]
/interface vlan
add comment=Customers interface=sfp28-12-lan name=vlan2 vlan-id=2
/interface list
add comment="contains WAN interfaces" name=WAN
add comment="contains LAN interfaces" name=LAN
add comment="Contains VPN interfaces" name=VPN
add name=WAN1
add name=NEED_PCC
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add authoritative=after-2sec-delay interface=bridge-lan lease-time=5m name=\
    vlan2-dhcp
/ip pool
add name=vlan2-pool ranges=10.130.2.10-10.130.3.254
add name=oob-pool ranges=10.130.0.2-10.130.0.254
/ip dhcp-server
add address-pool=oob-pool interface=bridge-oob name=oob-dhcp
/port
set 0 name=serial0
/queue type
add kind=pcq name=MaxDownloadSpeed pcq-classifier=src-address pcq-rate=10M
add kind=pcq name=MaxUploadSpeed pcq-classifier=dst-address pcq-rate=1M
add kind=pcq name=Standard_Download_Speed pcq-burst-rate=30M pcq-burst-time=\
    15s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=25M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=Standard_Upload_Speed pcq-burst-rate=3600k pcq-burst-time=\
    15s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=3M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Download_Speed pcq-burst-rate=30M \
    pcq-burst-time=15s pcq-classifier=dst-address pcq-dst-address6-mask=64 \
    pcq-rate=25M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Upload_Speed pcq-burst-rate=3600k \
    pcq-burst-time=15s pcq-classifier=src-address pcq-dst-address6-mask=64 \
    pcq-rate=3M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
/queue tree
add max-limit=300M name=Standard_Download_Speed packet-mark=\
    Standard_Traffic_Down parent=bridge-lan priority=7 queue=\
    Standard_Download_Speed
add max-limit=50M name=Standard_Upload_Speed packet-mark=Standard_Traffic_Up \
    parent=bridge-lan priority=7 queue=Standard_Upload_Speed
add max-limit=300M name=High_Capacity_Download_Speed packet-mark=\
    High_Capacity_Traffic_Down parent=bridge-lan priority=7 queue=\
    High_Capacity_Download_Speed
add max-limit=50M name=High_Capacity_Upload_Speed packet-mark=\
    High_Capacity_Traffic_Up parent=bridge-lan priority=7 queue=\
    High_Capacity_Upload_Speed
/routing table
add fib name=to_wan1
add fib name=to_wan2
add fib name=to_wan3
add fib name=to_wan4
add fib name=to_wan5
add fib name=to_wan6
add fib name=to_wan7
add fib name=to_wan8
add fib name=to_wan9
add fib name=to_wan10
/snmp community
set [ find default=yes ] disabled=yes
add addresses=::/0 name=atbunal
/interface bridge port
add bridge=bridge-lan interface=vlan2
add bridge=bridge-oob interface=sfp28-11-oob
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=bridge-lan list=LAN
add interface=sfp28-1-wan1 list=WAN
add interface=sfp28-2-wan2 list=WAN
add comment="RE VPN" interface=RE list=VPN
add interface=sfp28-3-wan3 list=WAN
add interface=sfp28-4-wan4 list=WAN
add interface=sfp28-5-wan5 list=WAN
add interface=sfp28-6-wan6 list=WAN
add interface=sfp28-7-wan7 list=WAN
add interface=sfp28-8-wan8 list=WAN
add interface=sfp28-9-wan9 list=WAN
add interface=sfp28-10-wan10 list=WAN
/ip address
add address=10.130.1.1/24 comment="Servers - vlan1" interface=sfp28-12-lan \
    network=10.130.1.0
add address=10.130.2.1/23 comment="Subscribers - CPE" interface=vlan2 \
    network=10.130.2.0
add address=10.130.4.1/23 comment="Subscribers - SM" interface=vlan2 network=\
    10.130.4.0
add address=10.130.0.1/24 interface=bridge-oob network=10.130.0.0
/ip dhcp-client
add interface=sfp28-1-wan1 use-peer-dns=no
add interface=sfp28-2-wan2 use-peer-dns=no
add interface=sfp28-3-wan3 use-peer-dns=no
add interface=sfp28-4-wan4 use-peer-dns=no
add interface=sfp28-5-wan5 use-peer-dns=no
add interface=sfp28-6-wan6 use-peer-dns=no
add interface=sfp28-7-wan7 use-peer-dns=no
add interface=sfp28-8-wan8 use-peer-dns=no
add interface=sfp28-9-wan9 use-peer-dns=no
add interface=sfp28-10-wan10 use-peer-dns=no
/ip dhcp-server alert
add comment="Rogue DHCP Detect" disabled=no interface=bridge-lan on-alert=\
    ":log error message=\"Rogue DHCP Server Discovered\"" valid-server=\
    48:A9:8A:85:65:FB
/ip dhcp-server lease
add address=10.130.3.3 mac-address=BC:A9:93:46:C5:D9 server=vlan2-dhcp
add address=10.130.2.10 mac-address=BC:A9:93:47:A2:D9 server=vlan2-dhcp
add address=10.130.2.11 mac-address=BC:A9:93:47:2A:C1 server=vlan2-dhcp
add address=10.130.2.12 mac-address=BC:A9:93:46:94:39 server=vlan2-dhcp
add address=10.130.2.13 mac-address=BC:A9:93:46:7A:59 server=vlan2-dhcp
add address=10.130.2.14 mac-address=BC:A9:93:46:90:99 server=vlan2-dhcp
add address=10.130.3.29 mac-address=BC:A9:93:47:58:11 server=vlan2-dhcp
add address=10.130.2.251 mac-address=BC:A9:93:47:A2:E9 server=vlan2-dhcp
add address=10.130.2.147 mac-address=BC:A9:93:46:85:F9 server=vlan2-dhcp
add address=10.130.2.52 mac-address=BC:A9:93:46:83:71 server=vlan2-dhcp
add address=10.130.2.16 mac-address=BC:A9:93:46:70:21 server=vlan2-dhcp
add address=10.130.2.17 mac-address=BC:A9:93:47:2B:B1 server=vlan2-dhcp
add address=10.130.2.20 mac-address=BC:A9:93:47:99:09 server=vlan2-dhcp
add address=10.130.2.21 mac-address=BC:A9:93:46:7A:61 server=vlan2-dhcp
add address=10.130.2.22 mac-address=BC:A9:93:47:9B:B9 server=vlan2-dhcp
add address=10.130.2.23 mac-address=BC:A9:93:47:2B:A9 server=vlan2-dhcp
add address=10.130.2.24 mac-address=BC:A9:93:46:BC:89 server=vlan2-dhcp
add address=10.130.2.26 mac-address=BC:A9:93:46:94:29 server=vlan2-dhcp
add address=10.130.2.27 mac-address=BC:A9:93:46:9E:D1 server=vlan2-dhcp
add address=10.130.2.28 mac-address=BC:A9:93:46:88:61 server=vlan2-dhcp
add address=10.130.2.29 mac-address=BC:A9:93:46:EF:89 server=vlan2-dhcp
add address=10.130.2.30 mac-address=BC:A9:93:47:10:B9 server=vlan2-dhcp
add address=10.130.2.31 mac-address=BC:A9:93:46:6D:19 server=vlan2-dhcp
add address=10.130.2.32 mac-address=BC:A9:93:47:2C:99 server=vlan2-dhcp
add address=10.130.2.33 mac-address=BC:A9:93:47:9B:41 server=vlan2-dhcp
add address=10.130.2.35 mac-address=BC:A9:93:46:C6:91 server=vlan2-dhcp
add address=10.130.2.37 mac-address=BC:A9:93:47:1A:81 server=vlan2-dhcp
add address=10.130.2.38 mac-address=BC:A9:93:46:71:41 server=vlan2-dhcp
add address=10.130.2.39 mac-address=BC:A9:93:46:A7:09 server=vlan2-dhcp
add address=10.130.2.40 mac-address=BC:A9:93:46:A7:69 server=vlan2-dhcp
add address=10.130.2.41 mac-address=BC:A9:93:46:84:B1 server=vlan2-dhcp
add address=10.130.2.42 mac-address=BC:A9:93:46:90:49 server=vlan2-dhcp
add address=10.130.2.43 mac-address=BC:A9:93:47:1E:01 server=vlan2-dhcp
add address=10.130.2.44 mac-address=BC:A9:93:47:A3:49 server=vlan2-dhcp
add address=10.130.2.45 mac-address=BC:A9:93:46:6F:29 server=vlan2-dhcp
add address=10.130.2.46 mac-address=BC:A9:93:46:90:01 server=vlan2-dhcp
add address=10.130.2.50 mac-address=BC:A9:93:46:8D:99 server=vlan2-dhcp
add address=10.130.2.51 mac-address=BC:A9:93:46:EC:B1 server=vlan2-dhcp
add address=10.130.2.53 mac-address=BC:A9:93:46:86:61 server=vlan2-dhcp
add address=10.130.2.54 mac-address=BC:A9:93:46:86:71 server=vlan2-dhcp
add address=10.130.2.55 mac-address=BC:A9:93:46:C5:89 server=vlan2-dhcp
add address=10.130.2.56 mac-address=BC:A9:93:46:90:51 server=vlan2-dhcp
add address=10.130.2.57 mac-address=BC:A9:93:47:2D:B9 server=vlan2-dhcp
add address=10.130.2.59 mac-address=BC:A9:93:46:C6:D9 server=vlan2-dhcp
add address=10.130.2.60 mac-address=BC:A9:93:46:87:89 server=vlan2-dhcp
add address=10.130.2.61 mac-address=BC:A9:93:46:C6:81 server=vlan2-dhcp
add address=10.130.2.62 mac-address=BC:A9:93:46:A6:31 server=vlan2-dhcp
add address=10.130.2.64 mac-address=BC:A9:93:46:C5:F9 server=vlan2-dhcp
add address=10.130.2.66 mac-address=BC:A9:93:46:C6:E9 server=vlan2-dhcp
add address=10.130.2.67 mac-address=BC:A9:93:46:ED:29 server=vlan2-dhcp
add address=10.130.2.68 mac-address=BC:A9:93:46:DB:49 server=vlan2-dhcp
add address=10.130.2.69 mac-address=BC:A9:93:46:97:21 server=vlan2-dhcp
add address=10.130.2.71 mac-address=BC:A9:93:46:CF:71 server=vlan2-dhcp
add address=10.130.2.74 mac-address=BC:A9:93:47:5E:11 server=vlan2-dhcp
add address=10.130.2.76 mac-address=BC:A9:93:47:2B:21 server=vlan2-dhcp
add address=10.130.2.77 mac-address=BC:A9:93:46:BC:11 server=vlan2-dhcp
add address=10.130.2.78 mac-address=BC:A9:93:46:84:C9 server=vlan2-dhcp
add address=10.130.2.79 mac-address=BC:A9:93:46:6F:39 server=vlan2-dhcp
add address=10.130.2.81 mac-address=BC:A9:93:46:84:D9 server=vlan2-dhcp
add address=10.130.2.86 mac-address=BC:A9:93:47:29:61 server=vlan2-dhcp
add address=10.130.2.72 mac-address=BC:A9:93:46:ED:39 server=vlan2-dhcp
add address=10.130.2.73 mac-address=BC:A9:93:46:B2:69 server=vlan2-dhcp
add address=10.130.2.80 mac-address=BC:A9:93:46:C9:79 server=vlan2-dhcp
add address=10.130.2.87 mac-address=BC:A9:93:47:1E:09 server=vlan2-dhcp
add address=10.130.2.88 mac-address=BC:A9:93:47:9F:E9 server=vlan2-dhcp
add address=10.130.2.89 mac-address=BC:A9:93:47:9F:D9 server=vlan2-dhcp
add address=10.130.2.90 mac-address=BC:A9:93:46:87:59 server=vlan2-dhcp
add address=10.130.2.92 mac-address=BC:A9:93:47:A2:F9 server=vlan2-dhcp
add address=10.130.2.93 mac-address=BC:A9:93:47:2B:D1 server=vlan2-dhcp
add address=10.130.2.19 mac-address=BC:A9:93:46:C5:69 server=vlan2-dhcp
add address=10.130.2.94 mac-address=BC:A9:93:47:A3:E9 server=vlan2-dhcp
add address=10.130.2.96 mac-address=BC:A9:93:46:F4:51 server=vlan2-dhcp
add address=10.130.2.58 mac-address=BC:A9:93:46:F4:41 server=vlan2-dhcp
add address=10.130.2.98 mac-address=BC:A9:93:47:1D:D9 server=vlan2-dhcp
add address=10.130.2.99 mac-address=BC:A9:93:46:AE:49 server=vlan2-dhcp
add address=10.130.2.82 mac-address=BC:A9:93:46:ED:31 server=vlan2-dhcp
add address=10.130.2.100 mac-address=BC:A9:93:47:2B:C9 server=vlan2-dhcp
add address=10.130.2.95 mac-address=BC:A9:93:47:6A:19 server=vlan2-dhcp
add address=10.130.2.252 mac-address=BC:A9:93:46:D0:39 server=vlan2-dhcp
add address=10.130.2.25 mac-address=BC:A9:93:47:A1:F9 server=vlan2-dhcp
add address=10.130.2.91 mac-address=BC:A9:93:47:86:01 server=vlan2-dhcp
add address=10.130.2.34 mac-address=BC:A9:93:46:C9:71 server=vlan2-dhcp
/ip dhcp-server network
add address=10.130.0.0/24 comment=OOB gateway=10.130.0.1
add address=10.130.1.0/24 comment=Servers gateway=10.130.1.1
add address=10.130.2.0/23 comment="Subscribers - CPE" dns-server=\
    8.8.8.8,1.1.1.1 gateway=10.130.2.1
add address=10.130.4.0/23 comment="Subscribers - SM" dns-server=\
    8.8.8.8,1.1.1.1 gateway=10.130.4.1
/ip dns
set servers=1.1.1.1,8.8.8.8
/ip firewall mangle
add action=mark-connection chain=prerouting comment="For PCC" \
    connection-mark=no-mark disabled=yes dst-address-type=!local \
    in-interface-list=NEED_PCC new-connection-mark=to_wan1 passthrough=yes \
    per-connection-classifier=both-addresses:10/0
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan2 passthrough=yes per-connection-classifier=\
    both-addresses:10/1
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan3 passthrough=yes per-connection-classifier=\
    both-addresses:10/2
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan4 passthrough=yes per-connection-classifier=\
    both-addresses:10/3
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan5 passthrough=yes per-connection-classifier=\
    both-addresses:10/4
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan6 passthrough=yes per-connection-classifier=\
    both-addresses:10/5
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan7 passthrough=yes per-connection-classifier=\
    both-addresses:10/6
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan8 passthrough=yes per-connection-classifier=\
    both-addresses:10/7
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan9 passthrough=yes per-connection-classifier=\
    both-addresses:10/8
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan10 passthrough=yes per-connection-classifier=\
    both-addresses:10/9
add action=mark-connection chain=forward comment="For Sonar" \
    dst-address-list="Standard Internet" new-connection-mark=\
    Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list=\
    "Standard Internet"
add action=mark-connection chain=forward dst-address-list=\
    "High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=High_Capacity_Traffic_Up passthrough=yes \
    src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" connection-mark=\
    Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=\
    yes
add action=mark-packet chain=forward connection-mark=\
    High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up \
    passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat comment="Cacti HTTP" dst-port=180 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=80
add action=dst-nat chain=dstnat comment="Cacti HTTPS" dst-port=1443 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=443
add action=dst-nat chain=dstnat comment="Preseem SSH" dst-port=22222 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=22
add action=dst-nat chain=dstnat comment="Preseem Cockpit" dst-port=9090 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=9090
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
/ip route
add disabled=no dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=no dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-1-wan1 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-2-wan2 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-3-wan3 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-4-wan4 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-5-wan5 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-6-wan6 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-7-wan7 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-8-wan8 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-9-wan9 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-10-wan10 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-1-wan1 \
    routing-table=to_wan1 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-2-wan2 \
    routing-table=to_wan2 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-3-wan3 \
    routing-table=to_wan3 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-4-wan4 \
    routing-table=to_wan4 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-5-wan5 \
    routing-table=to_wan5 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-6-wan6 \
    routing-table=to_wan6 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-7-wan7 \
    routing-table=to_wan7 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-8-wan8 \
    routing-table=to_wan8 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-9-wan9 \
    routing-table=to_wan9 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-10-wan10 \
    routing-table=to_wan10 suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=myCa disabled=no port=8443
set api-ssl certificate=myCa
/ip ssh
set strong-crypto=yes
/snmp
set contact="Richesin Engineering <jaysen@richesinengineering.com>" enabled=\
    yes location="Communications Shelter, Unalakleet, AK [63.89890275724056, -\
    160.7788028412428]" trap-community=atbunal
/system clock
set time-zone-autodetect=no time-zone-name=America/Anchorage
/system identity
set name=UnalakleetRouter
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.pool.ntp.org
add address=1.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/system scheduler
add disabled=yes interval=5m name=QueueTreeUsageCheck on-event=":log info \"Qu\
    eueTreeUsageCheck script started\"\r\
    \nQueueTreeUsageCheck\r\
    \n:log info \"QueueTreeUsageCheck script finished\"" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-03 start-time=11:04:00
add disabled=yes interval=4w2d name=datareset on-event=\
    "/queue tree reset-counters numbers=[find]" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-11-01 start-time=09:33:17
add disabled=yes interval=5m name=runThrottle on-event=\
    "/system script run throttle;" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-23 start-time=00:00:00
/system script
add dont-require-permissions=no name=rogue-dhcp-scan owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    log info \"Found rogue DHCP server at \$[/system clock get date] \$[/syste\
    m clock get time] on interface bridge-trunk]\"\r\
    \n#Email\r\
    \n/tool e-mail send to=\"jaysen@richesinengineering.com\" subject=\"Found \
    rogue DHCP server at \$[/system clock get date] \$[/system clock get time]\
    \_on interface bridge-trunk\""
add dont-require-permissions=no name=throttle owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local thresholdBytes 161061273600; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"Standard_Download_Sp\
    eed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"Standard_Upload_Speed\
    \"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}\r\
    \n\r\
    \n:local thresholdBytes 268435456000; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"High_Capacity_Downlo\
    ad_Speed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"High_Capacity_Upload_S\
    peed\"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}"
/tool bandwidth-server
set enabled=no
/tool e-mail
set from="Unalakleet Router" port=2525 server=mail.smtp2go.com tls=yes user=\
    alerts@richesinengineering.com
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
/tool traffic-monitor
add interface=sfp28-3-wan3 name=tmon1
# 2023-11-14 14:51:49 by RouterOS 7.12
# software id = DVNU-PTAC
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HE808NY9TF8
/interface bridge
add admin-mac=48:A9:8A:85:65:FB auto-mac=no comment="Link to NV1560 Port 1" \
    name=bridge-lan
add name=bridge-oob
/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=qsfp28-1-1 ] disabled=yes
set [ find default-name=qsfp28-1-2 ] disabled=yes
set [ find default-name=qsfp28-1-3 ] disabled=yes
set [ find default-name=qsfp28-1-4 ] disabled=yes
set [ find default-name=qsfp28-2-1 ] disabled=yes
set [ find default-name=qsfp28-2-2 ] disabled=yes
set [ find default-name=qsfp28-2-3 ] disabled=yes
set [ find default-name=qsfp28-2-4 ] disabled=yes
set [ find default-name=sfp28-1 ] comment="Link to Starlink Terminal 1" name=\
    sfp28-1-wan1
set [ find default-name=sfp28-2 ] comment="Link to Starlink Terminal 2" name=\
    sfp28-2-wan2
set [ find default-name=sfp28-3 ] comment="Link to Starlink Terminal 3" name=\
    sfp28-3-wan3
set [ find default-name=sfp28-4 ] comment="Link to Starlink Terminal 4" name=\
    sfp28-4-wan4
set [ find default-name=sfp28-5 ] comment="Link to Starlink Terminal 5" name=\
    sfp28-5-wan5
set [ find default-name=sfp28-6 ] comment="Link to Starlink Terminal 6" name=\
    sfp28-6-wan6
set [ find default-name=sfp28-7 ] comment="Link to Starlink Terminal 7" name=\
    sfp28-7-wan7
set [ find default-name=sfp28-8 ] comment="Link to Starlink Terminal 8" name=\
    sfp28-8-wan8
set [ find default-name=sfp28-9 ] comment="Link to Starlink Terminal 9" name=\
    sfp28-9-wan9
set [ find default-name=sfp28-10 ] comment="Link to Starlink Terminal 10" \
    name=sfp28-10-wan10
set [ find default-name=sfp28-11 ] comment="Link to OOB PC" name=sfp28-11-oob
set [ find default-name=sfp28-12 ] comment="Link to Adtran Switch Port 1" \
    name=sfp28-12-lan
/interface l2tp-client
add connect-to=vpn.richesineng.com disabled=no name=RE use-ipsec=yes user=\
    unalakleet
/interface vlan
add comment=Customers interface=sfp28-12-lan name=vlan2 vlan-id=2
/interface list
add comment="contains WAN interfaces" name=WAN
add comment="contains LAN interfaces" name=LAN
add comment="Contains VPN interfaces" name=VPN
add name=WAN1
add name=NEED_PCC
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add authoritative=after-2sec-delay interface=bridge-lan lease-time=5m name=\
    vlan2-dhcp
/ip pool
add name=vlan2-pool ranges=10.130.2.10-10.130.3.254
add name=oob-pool ranges=10.130.0.2-10.130.0.254
/ip dhcp-server
add address-pool=oob-pool interface=bridge-oob name=oob-dhcp
/port
set 0 name=serial0
/queue type
add kind=pcq name=MaxDownloadSpeed pcq-classifier=src-address pcq-rate=10M
add kind=pcq name=MaxUploadSpeed pcq-classifier=dst-address pcq-rate=1M
add kind=pcq name=Standard_Download_Speed pcq-burst-rate=30M pcq-burst-time=\
    15s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=25M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=Standard_Upload_Speed pcq-burst-rate=3600k pcq-burst-time=\
    15s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=3M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Download_Speed pcq-burst-rate=30M \
    pcq-burst-time=15s pcq-classifier=dst-address pcq-dst-address6-mask=64 \
    pcq-rate=25M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Upload_Speed pcq-burst-rate=3600k \
    pcq-burst-time=15s pcq-classifier=src-address pcq-dst-address6-mask=64 \
    pcq-rate=3M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
/queue tree
add max-limit=300M name=Standard_Download_Speed packet-mark=\
    Standard_Traffic_Down parent=bridge-lan priority=7 queue=\
    Standard_Download_Speed
add max-limit=50M name=Standard_Upload_Speed packet-mark=Standard_Traffic_Up \
    parent=bridge-lan priority=7 queue=Standard_Upload_Speed
add max-limit=300M name=High_Capacity_Download_Speed packet-mark=\
    High_Capacity_Traffic_Down parent=bridge-lan priority=7 queue=\
    High_Capacity_Download_Speed
add max-limit=50M name=High_Capacity_Upload_Speed packet-mark=\
    High_Capacity_Traffic_Up parent=bridge-lan priority=7 queue=\
    High_Capacity_Upload_Speed
/routing table
add fib name=to_wan1
add fib name=to_wan2
add fib name=to_wan3
add fib name=to_wan4
add fib name=to_wan5
add fib name=to_wan6
add fib name=to_wan7
add fib name=to_wan8
add fib name=to_wan9
add fib name=to_wan10
/snmp community
set [ find default=yes ] disabled=yes
add addresses=::/0 name=atbunal
/interface bridge port
add bridge=bridge-lan interface=vlan2
add bridge=bridge-oob interface=sfp28-11-oob
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=bridge-lan list=LAN
add interface=sfp28-1-wan1 list=WAN
add interface=sfp28-2-wan2 list=WAN
add comment="RE VPN" interface=RE list=VPN
add interface=sfp28-3-wan3 list=WAN
add interface=sfp28-4-wan4 list=WAN
add interface=sfp28-5-wan5 list=WAN
add interface=sfp28-6-wan6 list=WAN
add interface=sfp28-7-wan7 list=WAN
add interface=sfp28-8-wan8 list=WAN
add interface=sfp28-9-wan9 list=WAN
add interface=sfp28-10-wan10 list=WAN
/ip address
add address=10.130.1.1/24 comment="Servers - vlan1" interface=sfp28-12-lan \
    network=10.130.1.0
add address=10.130.2.1/23 comment="Subscribers - CPE" interface=vlan2 \
    network=10.130.2.0
add address=10.130.4.1/23 comment="Subscribers - SM" interface=vlan2 network=\
    10.130.4.0
add address=10.130.0.1/24 interface=bridge-oob network=10.130.0.0
/ip dhcp-client
add interface=sfp28-1-wan1 use-peer-dns=no
add interface=sfp28-2-wan2 use-peer-dns=no
add interface=sfp28-3-wan3 use-peer-dns=no
add interface=sfp28-4-wan4 use-peer-dns=no
add interface=sfp28-5-wan5 use-peer-dns=no
add interface=sfp28-6-wan6 use-peer-dns=no
add interface=sfp28-7-wan7 use-peer-dns=no
add interface=sfp28-8-wan8 use-peer-dns=no
add interface=sfp28-9-wan9 use-peer-dns=no
add interface=sfp28-10-wan10 use-peer-dns=no
/ip dhcp-server alert
add comment="Rogue DHCP Detect" disabled=no interface=bridge-lan on-alert=\
    ":log error message=\"Rogue DHCP Server Discovered\"" valid-server=\
    48:A9:8A:85:65:FB
/ip dhcp-server lease
add address=10.130.3.3 mac-address=BC:A9:93:46:C5:D9 server=vlan2-dhcp
add address=10.130.2.10 mac-address=BC:A9:93:47:A2:D9 server=vlan2-dhcp
add address=10.130.2.11 mac-address=BC:A9:93:47:2A:C1 server=vlan2-dhcp
add address=10.130.2.12 mac-address=BC:A9:93:46:94:39 server=vlan2-dhcp
add address=10.130.2.13 mac-address=BC:A9:93:46:7A:59 server=vlan2-dhcp
add address=10.130.2.14 mac-address=BC:A9:93:46:90:99 server=vlan2-dhcp
add address=10.130.3.29 mac-address=BC:A9:93:47:58:11 server=vlan2-dhcp
add address=10.130.2.251 mac-address=BC:A9:93:47:A2:E9 server=vlan2-dhcp
add address=10.130.2.147 mac-address=BC:A9:93:46:85:F9 server=vlan2-dhcp
add address=10.130.2.52 mac-address=BC:A9:93:46:83:71 server=vlan2-dhcp
add address=10.130.2.16 mac-address=BC:A9:93:46:70:21 server=vlan2-dhcp
add address=10.130.2.17 mac-address=BC:A9:93:47:2B:B1 server=vlan2-dhcp
add address=10.130.2.20 mac-address=BC:A9:93:47:99:09 server=vlan2-dhcp
add address=10.130.2.21 mac-address=BC:A9:93:46:7A:61 server=vlan2-dhcp
add address=10.130.2.22 mac-address=BC:A9:93:47:9B:B9 server=vlan2-dhcp
add address=10.130.2.23 mac-address=BC:A9:93:47:2B:A9 server=vlan2-dhcp
add address=10.130.2.24 mac-address=BC:A9:93:46:BC:89 server=vlan2-dhcp
add address=10.130.2.26 mac-address=BC:A9:93:46:94:29 server=vlan2-dhcp
add address=10.130.2.27 mac-address=BC:A9:93:46:9E:D1 server=vlan2-dhcp
add address=10.130.2.28 mac-address=BC:A9:93:46:88:61 server=vlan2-dhcp
add address=10.130.2.29 mac-address=BC:A9:93:46:EF:89 server=vlan2-dhcp
add address=10.130.2.30 mac-address=BC:A9:93:47:10:B9 server=vlan2-dhcp
add address=10.130.2.31 mac-address=BC:A9:93:46:6D:19 server=vlan2-dhcp
add address=10.130.2.32 mac-address=BC:A9:93:47:2C:99 server=vlan2-dhcp
add address=10.130.2.33 mac-address=BC:A9:93:47:9B:41 server=vlan2-dhcp
add address=10.130.2.35 mac-address=BC:A9:93:46:C6:91 server=vlan2-dhcp
add address=10.130.2.37 mac-address=BC:A9:93:47:1A:81 server=vlan2-dhcp
add address=10.130.2.38 mac-address=BC:A9:93:46:71:41 server=vlan2-dhcp
add address=10.130.2.39 mac-address=BC:A9:93:46:A7:09 server=vlan2-dhcp
add address=10.130.2.40 mac-address=BC:A9:93:46:A7:69 server=vlan2-dhcp
add address=10.130.2.41 mac-address=BC:A9:93:46:84:B1 server=vlan2-dhcp
add address=10.130.2.42 mac-address=BC:A9:93:46:90:49 server=vlan2-dhcp
add address=10.130.2.43 mac-address=BC:A9:93:47:1E:01 server=vlan2-dhcp
add address=10.130.2.44 mac-address=BC:A9:93:47:A3:49 server=vlan2-dhcp
add address=10.130.2.45 mac-address=BC:A9:93:46:6F:29 server=vlan2-dhcp
add address=10.130.2.46 mac-address=BC:A9:93:46:90:01 server=vlan2-dhcp
add address=10.130.2.50 mac-address=BC:A9:93:46:8D:99 server=vlan2-dhcp
add address=10.130.2.51 mac-address=BC:A9:93:46:EC:B1 server=vlan2-dhcp
add address=10.130.2.53 mac-address=BC:A9:93:46:86:61 server=vlan2-dhcp
add address=10.130.2.54 mac-address=BC:A9:93:46:86:71 server=vlan2-dhcp
add address=10.130.2.55 mac-address=BC:A9:93:46:C5:89 server=vlan2-dhcp
add address=10.130.2.56 mac-address=BC:A9:93:46:90:51 server=vlan2-dhcp
add address=10.130.2.57 mac-address=BC:A9:93:47:2D:B9 server=vlan2-dhcp
add address=10.130.2.59 mac-address=BC:A9:93:46:C6:D9 server=vlan2-dhcp
add address=10.130.2.60 mac-address=BC:A9:93:46:87:89 server=vlan2-dhcp
add address=10.130.2.61 mac-address=BC:A9:93:46:C6:81 server=vlan2-dhcp
add address=10.130.2.62 mac-address=BC:A9:93:46:A6:31 server=vlan2-dhcp
add address=10.130.2.64 mac-address=BC:A9:93:46:C5:F9 server=vlan2-dhcp
add address=10.130.2.66 mac-address=BC:A9:93:46:C6:E9 server=vlan2-dhcp
add address=10.130.2.67 mac-address=BC:A9:93:46:ED:29 server=vlan2-dhcp
add address=10.130.2.68 mac-address=BC:A9:93:46:DB:49 server=vlan2-dhcp
add address=10.130.2.69 mac-address=BC:A9:93:46:97:21 server=vlan2-dhcp
add address=10.130.2.71 mac-address=BC:A9:93:46:CF:71 server=vlan2-dhcp
add address=10.130.2.74 mac-address=BC:A9:93:47:5E:11 server=vlan2-dhcp
add address=10.130.2.76 mac-address=BC:A9:93:47:2B:21 server=vlan2-dhcp
add address=10.130.2.77 mac-address=BC:A9:93:46:BC:11 server=vlan2-dhcp
add address=10.130.2.78 mac-address=BC:A9:93:46:84:C9 server=vlan2-dhcp
add address=10.130.2.79 mac-address=BC:A9:93:46:6F:39 server=vlan2-dhcp
add address=10.130.2.81 mac-address=BC:A9:93:46:84:D9 server=vlan2-dhcp
add address=10.130.2.86 mac-address=BC:A9:93:47:29:61 server=vlan2-dhcp
add address=10.130.2.72 mac-address=BC:A9:93:46:ED:39 server=vlan2-dhcp
add address=10.130.2.73 mac-address=BC:A9:93:46:B2:69 server=vlan2-dhcp
add address=10.130.2.80 mac-address=BC:A9:93:46:C9:79 server=vlan2-dhcp
add address=10.130.2.87 mac-address=BC:A9:93:47:1E:09 server=vlan2-dhcp
add address=10.130.2.88 mac-address=BC:A9:93:47:9F:E9 server=vlan2-dhcp
add address=10.130.2.89 mac-address=BC:A9:93:47:9F:D9 server=vlan2-dhcp
add address=10.130.2.90 mac-address=BC:A9:93:46:87:59 server=vlan2-dhcp
add address=10.130.2.92 mac-address=BC:A9:93:47:A2:F9 server=vlan2-dhcp
add address=10.130.2.93 mac-address=BC:A9:93:47:2B:D1 server=vlan2-dhcp
add address=10.130.2.19 mac-address=BC:A9:93:46:C5:69 server=vlan2-dhcp
add address=10.130.2.94 mac-address=BC:A9:93:47:A3:E9 server=vlan2-dhcp
add address=10.130.2.96 mac-address=BC:A9:93:46:F4:51 server=vlan2-dhcp
add address=10.130.2.58 mac-address=BC:A9:93:46:F4:41 server=vlan2-dhcp
add address=10.130.2.98 mac-address=BC:A9:93:47:1D:D9 server=vlan2-dhcp
add address=10.130.2.99 mac-address=BC:A9:93:46:AE:49 server=vlan2-dhcp
add address=10.130.2.82 mac-address=BC:A9:93:46:ED:31 server=vlan2-dhcp
add address=10.130.2.100 mac-address=BC:A9:93:47:2B:C9 server=vlan2-dhcp
add address=10.130.2.95 mac-address=BC:A9:93:47:6A:19 server=vlan2-dhcp
add address=10.130.2.252 mac-address=BC:A9:93:46:D0:39 server=vlan2-dhcp
add address=10.130.2.25 mac-address=BC:A9:93:47:A1:F9 server=vlan2-dhcp
add address=10.130.2.91 mac-address=BC:A9:93:47:86:01 server=vlan2-dhcp
add address=10.130.2.34 mac-address=BC:A9:93:46:C9:71 server=vlan2-dhcp
/ip dhcp-server network
add address=10.130.0.0/24 comment=OOB gateway=10.130.0.1
add address=10.130.1.0/24 comment=Servers gateway=10.130.1.1
add address=10.130.2.0/23 comment="Subscribers - CPE" dns-server=\
    8.8.8.8,1.1.1.1 gateway=10.130.2.1
add address=10.130.4.0/23 comment="Subscribers - SM" dns-server=\
    8.8.8.8,1.1.1.1 gateway=10.130.4.1
/ip dns
set servers=1.1.1.1,8.8.8.8
/ip firewall mangle
add action=mark-connection chain=prerouting comment="For PCC" \
    connection-mark=no-mark disabled=yes dst-address-type=!local \
    in-interface-list=NEED_PCC new-connection-mark=to_wan1 passthrough=yes \
    per-connection-classifier=both-addresses:10/0
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan2 passthrough=yes per-connection-classifier=\
    both-addresses:10/1
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan3 passthrough=yes per-connection-classifier=\
    both-addresses:10/2
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan4 passthrough=yes per-connection-classifier=\
    both-addresses:10/3
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan5 passthrough=yes per-connection-classifier=\
    both-addresses:10/4
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan6 passthrough=yes per-connection-classifier=\
    both-addresses:10/5
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan7 passthrough=yes per-connection-classifier=\
    both-addresses:10/6
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan8 passthrough=yes per-connection-classifier=\
    both-addresses:10/7
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan9 passthrough=yes per-connection-classifier=\
    both-addresses:10/8
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface-list=NEED_PCC \
    new-connection-mark=to_wan10 passthrough=yes per-connection-classifier=\
    both-addresses:10/9
add action=mark-connection chain=forward comment="For Sonar" \
    dst-address-list="Standard Internet" new-connection-mark=\
    Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list=\
    "Standard Internet"
add action=mark-connection chain=forward dst-address-list=\
    "High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=High_Capacity_Traffic_Up passthrough=yes \
    src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" connection-mark=\
    Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=\
    yes
add action=mark-packet chain=forward connection-mark=\
    High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up \
    passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat comment="Cacti HTTP" dst-port=180 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=80
add action=dst-nat chain=dstnat comment="Cacti HTTPS" dst-port=1443 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=443
add action=dst-nat chain=dstnat comment="Preseem SSH" dst-port=22222 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=22
add action=dst-nat chain=dstnat comment="Preseem Cockpit" dst-port=9090 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=9090
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
/ip route
add disabled=no dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=no dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-1-wan1 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-2-wan2 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-3-wan3 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-4-wan4 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-5-wan5 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-6-wan6 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-7-wan7 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-8-wan8 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-9-wan9 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-10-wan10 \
    routing-table=main suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-1-wan1 \
    routing-table=to_wan1 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-2-wan2 \
    routing-table=to_wan2 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-3-wan3 \
    routing-table=to_wan3 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-4-wan4 \
    routing-table=to_wan4 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-5-wan5 \
    routing-table=to_wan5 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp28-6-wan6 \
    routing-table=to_wan6 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-7-wan7 \
    routing-table=to_wan7 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-8-wan8 \
    routing-table=to_wan8 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-9-wan9 \
    routing-table=to_wan9 suppress-hw-offload=no
add disabled=yes dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp28-10-wan10 \
    routing-table=to_wan10 suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=myCa disabled=no port=8443
set api-ssl certificate=myCa
/ip ssh
set strong-crypto=yes
/snmp
set contact="Richesin Engineering <jaysen@richesinengineering.com>" enabled=\
    yes location="Communications Shelter, Unalakleet, AK [63.89890275724056, -\
    160.7788028412428]" trap-community=atbunal
/system clock
set time-zone-autodetect=no time-zone-name=America/Anchorage
/system identity
set name=UnalakleetRouter
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.pool.ntp.org
add address=1.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/system scheduler
add disabled=yes interval=5m name=QueueTreeUsageCheck on-event=":log info \"Qu\
    eueTreeUsageCheck script started\"\r\
    \nQueueTreeUsageCheck\r\
    \n:log info \"QueueTreeUsageCheck script finished\"" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-03 start-time=11:04:00
add disabled=yes interval=4w2d name=datareset on-event=\
    "/queue tree reset-counters numbers=[find]" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-11-01 start-time=09:33:17
add disabled=yes interval=5m name=runThrottle on-event=\
    "/system script run throttle;" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-23 start-time=00:00:00
/system script
add dont-require-permissions=no name=rogue-dhcp-scan owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    log info \"Found rogue DHCP server at \$[/system clock get date] \$[/syste\
    m clock get time] on interface bridge-trunk]\"\r\
    \n#Email\r\
    \n/tool e-mail send to=\"jaysen@richesinengineering.com\" subject=\"Found \
    rogue DHCP server at \$[/system clock get date] \$[/system clock get time]\
    \_on interface bridge-trunk\""
add dont-require-permissions=no name=throttle owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local thresholdBytes 161061273600; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"Standard_Download_Sp\
    eed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"Standard_Upload_Speed\
    \"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}\r\
    \n\r\
    \n:local thresholdBytes 268435456000; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"High_Capacity_Downlo\
    ad_Speed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"High_Capacity_Upload_S\
    peed\"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}"
/tool bandwidth-server
set enabled=no
/tool e-mail
set from="Unalakleet Router" port=2525 server=mail.smtp2go.com tls=yes user=\
    alerts@richesinengineering.com
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
/tool traffic-monitor
add interface=sfp28-3-wan3 name=tmon1

sindy · Wed Nov 15, 2023 5:55 pm

I had removed some sensitive data in the config before posting it. That's likely why it wasn't in there.

Clear.

OK. Now let me clarify some points that may not be obvious.

First a disclaimer - I am aware that you have inherited most of the current configuration from the previous administrator, so if I occasionally say "you use", it is just an abbreviation of "I can see in the configuration".

this point may seem a cosmetic one, but it needs to be clarified to avoid issues in future and to explain the background of some suggestions. @anav has given you a template configuration that uses connection marks as a basis for assignment of routing marks; the "pre-anav" configuration uses connection marks as a basis to assign packet marks to be further used to choose QoS queues. Unfortunately, Mikrotik's wrapper to netfilter allows at most one connection mark per connection; luckily, the way the connection marks are used for QoS purposes in the pre-anav configuration is clearly a consequence of copy-pasting some template without really understanding it and it can be redone do work the same without use of the connection marks. Another positive point is that in your particular setup, you probably don't need to use the connection marks for routing either.
the reason why you have to set the gateway parameters of the routes in the form ip.add.re.ss%interface-name is that even if the Starlink network accepts packets sent with a "wrong" source address through a particular terminal, I suppose the bandwidth limitations apply per terminal, not per source address (but only X can confirm this). Since all leases from the same /21 use the same address of the gateway, setting the gateway parameter to IP address alone (which is what the DHCP client normally does) is not sufficient for the routing to choose a particular interface.
However, the screenshot you have posted shows that your WANs get own addresses, and therefore also gateways, from at least two different subnets. So although the leases are "long-term stable", we cannot risk that the gateway addresses in the manually configured routes would stop matching the real gateway addresses once the leases change, so we need to use a lease script to update the route configuration, not only in the dedicated routing tables but once at it, also in the default routing table (called main). It's RouterOS 7 so I'll have to do some tests before giving you a script to apply - I am well aware of those 2000 miles.

anav · Wed Nov 15, 2023 6:59 pm

Those are dogsled miles

Your second point lost its train of thought........
the reason why..... if that even if........................... ??????

sindy · Wed Nov 15, 2023 7:41 pm

So here's how to make the DHCP clients add/modify the routes the necessary way.

First, copy-paste the following script to the command line window of the router. The exported form looks awful, but it is not invoked until you change other things in the configuration, so you can paste the creation script from and then open the created script named lease-script in the GUI or command line editor to see it in a more user friendly way.

/system/script/add name=lease-script source=":if (\$bound=1) do={\
    \n  /ip route {\
    \n    :if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n      add gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n    }\
    \n    :foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole] do={\
    \n      set \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n    }\
    \n  }\
    \n}\
    \n"

Once the script is added and you make yourself comfortable with what it does, you can test the following steps for one of the DHCP clients (choose the N in the command as required):
/ip/dhcp-client/set [find where interface=sfp28-N-wanN] script=lease-script add-default-route=no
/ip/dhcp-client/release [find where interface=sfp28-N-wanN]

In the last configuration you've posted, the statically configured routes are already present even in table main, so they will get only updated, not added (unless you remove them before). But as the dynamically added one will be removed due to the change of add-default-route to no, you'll have to enable the disabled static route in main.

There may be a catch - you have renamed the sfp28-N interfaces to sfp-28-N-wanN, and I had some fun with that when testing it here. I have created and assigned the dhcp script while the interface had a custom name, but then decided to reset the custom name back to the default one. Nevertheless, the script kept using the old custom one; deleting and recreating the dhcp client was not enough to sort that out, but disabling and re-enabling the inteface did. Any questions that might arise are not to me, thank you

If you can see the expected outcome, which is to have a route via sfp28-N-wanN with the correct gateway address in the configuration as a static one, you can do the same for one more DHCP client. My plan is to let the L2TP client use those two before eventually extending the approach to all of them.

All the above is still just an intermediary step. My feeling is that with all uplinks served by the same satellite constellation, most of the outages will be affected by poor satellite visibility and will thus affect all of them, yet still we'll have to add means to notice and stop using a broken terminal, but that's for later.

Amm0 · Wed Nov 15, 2023 8:35 pm

All the above is still just an intermediary step. My feeling is that with all uplinks served by the same satellite constellation, most of the outages will be affected by poor satellite visibility and will thus affect all of them, yet still we'll have to add means to notice and stop using a broken terminal, but that's for later.

Maybe consider adding some monitoring of the starlink performance — which is kinda annoying since it use gRPC, not SNMP. I know there is starlink plugin for Prometheus, but if you have some other NMS somewhere... imagine there are plugins for starlink. The terminal's gRPC data includes stuff like max speed, # sats, etc. if you know that it be easier to find where fault lies down the road.

sindy · Wed Nov 15, 2023 8:55 pm

Maybe consider adding some monitoring of the starlink performance — which is kinda annoying since it use gRPC, not SNMP. I know there is starlink plugin for Prometheus, but if you have some other NMS somewhere... imagine there are plugins for starlink. The terminal's gRPC data includes stuff like max speed, # sats, etc. if you know that it be easier to find where fault lies down the road.

All the Starlinks I happen to deal with are remote ones so I never delved into monitoring of the terminal, assuming that the owner's app shows you something. What I had in mind was the "good old" monitoring of uplink transparency all the way to internet, preventing the situation where the WAN interface is physically up so the router keeps using it although the packets sent through that interface get nowhere due to an outage further in the ISP network.

But some weeks ago I was wondering whether the location data could be retrieved from the Starlink and haven't come across gRPC when googling. Once the 7.13 (with its capabilities to interwork between json and variables) becomes stable, I guess I'll give it a try.

jaysen · Wed Nov 15, 2023 8:57 pm

So here's how to make the DHCP clients add/modify the routes the necessary way.

First, copy-paste the following script to the command line window of the router. The exported form looks awful, but it is not invoked until you change other things in the configuration, so you can paste the creation script from and then open the created script named lease-script in the GUI or command line editor to see it in a more user friendly way.
Code: Select all
/system/script/add name=lease-script source=":if (\$bound=1) do={\
    \n  /ip route {\
    \n    :if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n      add gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n    }\
    \n    :foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole] do={\
    \n      set \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n    }\
    \n  }\
    \n}\
    \n"
Once the script is added and you make yourself comfortable with what it does, you can test the following steps for one of the DHCP clients (choose the N in the command as required):
/ip/dhcp-client/set [find where interface=sfp28-N-wanN] script=lease-script add-default-route=no
/ip/dhcp-client/release [find where interface=sfp28-N-wanN]

In the last configuration you've posted, the statically configured routes are already present even in table main, so they will get only updated, not added (unless you remove them before). But as the dynamically added one will be removed due to the change of add-default-route to no, you'll have to enable the disabled static route in main.

There may be a catch - you have renamed the sfp28-N interfaces to sfp-28-N-wanN, and I had some fun with that when testing it here. I have created and assigned the dhcp script while the interface had a custom name, but then decided to reset the custom name back to the default one. Nevertheless, the script kept using the old custom one; deleting and recreating the dhcp client was not enough to sort that out, but disabling and re-enabling the inteface did. Any questions that might arise are not to me, thank you

If you can see the expected outcome, which is to have a route via sfp28-N-wanN with the correct gateway address in the configuration as a static one, you can do the same for one more DHCP client. My plan is to let the L2TP client use those two before eventually extending the approach to all of them.

All the above is still just an intermediary step. My feeling is that with all uplinks served by the same satellite constellation, most of the outages will be affected by poor satellite visibility and will thus affect all of them, yet still we'll have to add means to notice and stop using a broken terminal, but that's for later.

Thank you Sindy for the script. I was starting to wonder how I was going to handle changing the routes in the event our IP addresses change. I didn't want to do it manually especially if any change occurred in the middle of the night or otherwise working with other clients and out of reach so this is very helpful. I am going to create that now.

I do apologize about the interface names. I had just added -wanN to the end of the default names but willing to rename them to simplify things if necessary. I am very flexible with any changes as is our client. They've lived with terrible or no internet most of their lives so they are very understanding and know I am working to make things better for them.

Good news about the satellite visibility is this village is in an area of Alaska that is high up and visibility should be of no concern as we were told by Starlink. The site is wide open with little to no obstructions and so far performance has been very impressive. OneWeb can't even begin to compete and the village has made sure I knew that. Even with the fraction of a percent obstruction that Starlink dashboard is showing on a few of the terminals. Single strength is very strong and latency is low.

I did rework some of mangle rules late last night prior to seeing your post this morning. I found this guide on the MikroTik site that seems very informative. https://mum.mikrotik.com/presentations/ ... ancing.pdf Attached is my latest config although it sounds like I may not need all the connection marks as you mentioned so we can get rid of anything that is not needed. I did have a question about the failover section and while I did set the distances in the routes. I am starting to wonder whether I even need to do that since all the gateways are the same on most of the terminals so if it goes down then it seems like it would be down for every connection that uses the same gateway?

To your point of copying-pasting configs without fully understanding. While it made me laugh, it's a very valid point and I do this quite a bit when I am playing around to see how things will work. I do always make sure to back up my original configs so I can recover in the case I mess up and that has gotten me in the past.

Again I appreciate you and everyone else for being patient and working with me on this.


# 2023-11-14 19:10:35 by RouterOS 7.12
# software id = DVNU-PTAC
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HE808NY9TF8
/interface bridge
add admin-mac=48:A9:8A:85:65:FB auto-mac=no comment="Link to NV1560 Port 1" \
    name=bridge-lan
/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=qsfp28-1-1 ] disabled=yes
set [ find default-name=qsfp28-1-2 ] disabled=yes
set [ find default-name=qsfp28-1-3 ] disabled=yes
set [ find default-name=qsfp28-1-4 ] disabled=yes
set [ find default-name=qsfp28-2-1 ] disabled=yes
set [ find default-name=qsfp28-2-2 ] disabled=yes
set [ find default-name=qsfp28-2-3 ] disabled=yes
set [ find default-name=qsfp28-2-4 ] disabled=yes
set [ find default-name=sfp28-1 ] comment="Link to Starlink Terminal 1" name=\
    sfp28-1-wan1
set [ find default-name=sfp28-2 ] comment="Link to Starlink Terminal 2" name=\
    sfp28-2-wan2
set [ find default-name=sfp28-3 ] comment="Link to Starlink Terminal 3" name=\
    sfp28-3-wan3
set [ find default-name=sfp28-4 ] comment="Link to Starlink Terminal 4" name=\
    sfp28-4-wan4
set [ find default-name=sfp28-5 ] comment="Link to Starlink Terminal 5" name=\
    sfp28-5-wan5
set [ find default-name=sfp28-6 ] comment="Link to Starlink Terminal 6" name=\
    sfp28-6-wan6
set [ find default-name=sfp28-7 ] comment="Link to Starlink Terminal 7" name=\
    sfp28-7-wan7
set [ find default-name=sfp28-8 ] comment="Link to Starlink Terminal 8" name=\
    sfp28-8-wan8
set [ find default-name=sfp28-9 ] comment="Link to Starlink Terminal 9" name=\
    sfp28-9-wan9
set [ find default-name=sfp28-10 ] comment="Link to Starlink Terminal 10" \
    name=sfp28-10-wan10
set [ find default-name=sfp28-11 ] comment="Link to OOB PC" name=sfp28-11-oob
set [ find default-name=sfp28-12 ] comment="Link to Adtran Switch Port 1" \
    name=sfp28-12-lan
/interface l2tp-client
add connect-to=vpn.richesineng.com disabled=no name=RE use-ipsec=yes user=\
    unalakleet
/interface vlan
add comment=Customers interface=sfp28-12-lan name=vlan2 vlan-id=2
/interface list
add comment="contains WAN interfaces" name=WAN
add comment="contains LAN interfaces" name=LAN
add comment="Contains VPN interfaces" name=VPN
add name=WAN1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add authoritative=after-2sec-delay interface=bridge-lan lease-time=5m name=\
    vlan2-dhcp
/ip pool
add name=vlan2-pool ranges=10.130.2.10-10.130.3.254
/port
set 0 name=serial0
/queue type
add kind=pcq name=MaxDownloadSpeed pcq-classifier=src-address pcq-rate=10M
add kind=pcq name=MaxUploadSpeed pcq-classifier=dst-address pcq-rate=1M
add kind=pcq name=Standard_Download_Speed pcq-burst-rate=30M pcq-burst-time=\
    15s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=25M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=Standard_Upload_Speed pcq-burst-rate=3600k pcq-burst-time=\
    15s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=3M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Download_Speed pcq-burst-rate=30M \
    pcq-burst-time=15s pcq-classifier=dst-address pcq-dst-address6-mask=64 \
    pcq-rate=25M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Upload_Speed pcq-burst-rate=3600k \
    pcq-burst-time=15s pcq-classifier=src-address pcq-dst-address6-mask=64 \
    pcq-rate=3M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
/queue tree
add max-limit=300M name=Standard_Download_Speed packet-mark=\
    Standard_Traffic_Down parent=bridge-lan priority=7 queue=\
    Standard_Download_Speed
add max-limit=50M name=Standard_Upload_Speed packet-mark=Standard_Traffic_Up \
    parent=bridge-lan priority=7 queue=Standard_Upload_Speed
add max-limit=300M name=High_Capacity_Download_Speed packet-mark=\
    High_Capacity_Traffic_Down parent=bridge-lan priority=7 queue=\
    High_Capacity_Download_Speed
add max-limit=50M name=High_Capacity_Upload_Speed packet-mark=\
    High_Capacity_Traffic_Up parent=bridge-lan priority=7 queue=\
    High_Capacity_Upload_Speed
/routing table
add fib name=to_wan1
add fib name=to_wan2
add fib name=to_wan3
add fib name=to_wan4
add fib name=to_wan5
add fib name=to_wan6
add fib name=to_wan7
add fib name=to_wan8
add fib name=to_wan9
add fib name=to_wan10
/snmp community
set [ find default=yes ] disabled=yes
add addresses=::/0 name=atbunal
/interface bridge port
add bridge=bridge-lan interface=vlan2
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=bridge-lan list=LAN
add interface=sfp28-1-wan1 list=WAN
add interface=sfp28-2-wan2 list=WAN
add comment="RE VPN" interface=RE list=VPN
add interface=sfp28-3-wan3 list=WAN
add interface=sfp28-4-wan4 list=WAN
add interface=sfp28-5-wan5 list=WAN
add interface=sfp28-6-wan6 list=WAN
add interface=sfp28-7-wan7 list=WAN
add interface=sfp28-8-wan8 list=WAN
add interface=sfp28-9-wan9 list=WAN
add interface=sfp28-10-wan10 list=WAN
/ip address
add address=10.130.1.1/24 comment="Servers - vlan1" interface=sfp28-12-lan \
    network=10.130.1.0
add address=10.130.2.1/23 comment="Subscribers - CPE" interface=vlan2 \
    network=10.130.2.0
add address=10.130.4.1/23 comment="Subscribers - SM" interface=vlan2 network=\
    10.130.4.0
/ip dhcp-client
add interface=sfp28-1-wan1 use-peer-dns=no
add interface=sfp28-2-wan2 use-peer-dns=no
add interface=sfp28-3-wan3 use-peer-dns=no
add interface=sfp28-4-wan4 use-peer-dns=no
add interface=sfp28-5-wan5 use-peer-dns=no
add interface=sfp28-6-wan6 use-peer-dns=no
add interface=sfp28-7-wan7 use-peer-dns=no
add interface=sfp28-8-wan8 use-peer-dns=no
add interface=sfp28-9-wan9 use-peer-dns=no
add interface=sfp28-10-wan10 use-peer-dns=no
/ip dhcp-server alert
add comment="Rogue DHCP Detect" disabled=no interface=bridge-lan on-alert=\
    ":log error message=\"Rogue DHCP Server Discovered\"" valid-server=\
    48:A9:8A:85:65:FB
/ip dhcp-server lease
add address=10.130.3.3 mac-address=BC:A9:93:46:C5:D9 server=vlan2-dhcp
add address=10.130.2.10 mac-address=BC:A9:93:47:A2:D9 server=vlan2-dhcp
add address=10.130.2.11 mac-address=BC:A9:93:47:2A:C1 server=vlan2-dhcp
add address=10.130.2.12 mac-address=BC:A9:93:46:94:39 server=vlan2-dhcp
add address=10.130.2.13 mac-address=BC:A9:93:46:7A:59 server=vlan2-dhcp
add address=10.130.2.14 mac-address=BC:A9:93:46:90:99 server=vlan2-dhcp
add address=10.130.3.29 mac-address=BC:A9:93:47:58:11 server=vlan2-dhcp
add address=10.130.2.251 mac-address=BC:A9:93:47:A2:E9 server=vlan2-dhcp
add address=10.130.2.147 mac-address=BC:A9:93:46:85:F9 server=vlan2-dhcp
add address=10.130.2.52 mac-address=BC:A9:93:46:83:71 server=vlan2-dhcp
add address=10.130.2.16 mac-address=BC:A9:93:46:70:21 server=vlan2-dhcp
add address=10.130.2.17 mac-address=BC:A9:93:47:2B:B1 server=vlan2-dhcp
add address=10.130.2.20 mac-address=BC:A9:93:47:99:09 server=vlan2-dhcp
add address=10.130.2.21 mac-address=BC:A9:93:46:7A:61 server=vlan2-dhcp
add address=10.130.2.22 mac-address=BC:A9:93:47:9B:B9 server=vlan2-dhcp
add address=10.130.2.23 mac-address=BC:A9:93:47:2B:A9 server=vlan2-dhcp
add address=10.130.2.24 mac-address=BC:A9:93:46:BC:89 server=vlan2-dhcp
add address=10.130.2.26 mac-address=BC:A9:93:46:94:29 server=vlan2-dhcp
add address=10.130.2.27 mac-address=BC:A9:93:46:9E:D1 server=vlan2-dhcp
add address=10.130.2.28 mac-address=BC:A9:93:46:88:61 server=vlan2-dhcp
add address=10.130.2.29 mac-address=BC:A9:93:46:EF:89 server=vlan2-dhcp
add address=10.130.2.30 mac-address=BC:A9:93:47:10:B9 server=vlan2-dhcp
add address=10.130.2.31 mac-address=BC:A9:93:46:6D:19 server=vlan2-dhcp
add address=10.130.2.32 mac-address=BC:A9:93:47:2C:99 server=vlan2-dhcp
add address=10.130.2.33 mac-address=BC:A9:93:47:9B:41 server=vlan2-dhcp
add address=10.130.2.35 mac-address=BC:A9:93:46:C6:91 server=vlan2-dhcp
add address=10.130.2.37 mac-address=BC:A9:93:47:1A:81 server=vlan2-dhcp
add address=10.130.2.38 mac-address=BC:A9:93:46:71:41 server=vlan2-dhcp
add address=10.130.2.39 mac-address=BC:A9:93:46:A7:09 server=vlan2-dhcp
add address=10.130.2.40 mac-address=BC:A9:93:46:A7:69 server=vlan2-dhcp
add address=10.130.2.41 mac-address=BC:A9:93:46:84:B1 server=vlan2-dhcp
add address=10.130.2.42 mac-address=BC:A9:93:46:90:49 server=vlan2-dhcp
add address=10.130.2.43 mac-address=BC:A9:93:47:1E:01 server=vlan2-dhcp
add address=10.130.2.44 mac-address=BC:A9:93:47:A3:49 server=vlan2-dhcp
add address=10.130.2.45 mac-address=BC:A9:93:46:6F:29 server=vlan2-dhcp
add address=10.130.2.46 mac-address=BC:A9:93:46:90:01 server=vlan2-dhcp
add address=10.130.2.50 mac-address=BC:A9:93:46:8D:99 server=vlan2-dhcp
add address=10.130.2.51 mac-address=BC:A9:93:46:EC:B1 server=vlan2-dhcp
add address=10.130.2.53 mac-address=BC:A9:93:46:86:61 server=vlan2-dhcp
add address=10.130.2.54 mac-address=BC:A9:93:46:86:71 server=vlan2-dhcp
add address=10.130.2.55 mac-address=BC:A9:93:46:C5:89 server=vlan2-dhcp
add address=10.130.2.56 mac-address=BC:A9:93:46:90:51 server=vlan2-dhcp
add address=10.130.2.57 mac-address=BC:A9:93:47:2D:B9 server=vlan2-dhcp
add address=10.130.2.59 mac-address=BC:A9:93:46:C6:D9 server=vlan2-dhcp
add address=10.130.2.60 mac-address=BC:A9:93:46:87:89 server=vlan2-dhcp
add address=10.130.2.61 mac-address=BC:A9:93:46:C6:81 server=vlan2-dhcp
add address=10.130.2.62 mac-address=BC:A9:93:46:A6:31 server=vlan2-dhcp
add address=10.130.2.64 mac-address=BC:A9:93:46:C5:F9 server=vlan2-dhcp
add address=10.130.2.66 mac-address=BC:A9:93:46:C6:E9 server=vlan2-dhcp
add address=10.130.2.67 mac-address=BC:A9:93:46:ED:29 server=vlan2-dhcp
add address=10.130.2.68 mac-address=BC:A9:93:46:DB:49 server=vlan2-dhcp
add address=10.130.2.69 mac-address=BC:A9:93:46:97:21 server=vlan2-dhcp
add address=10.130.2.71 mac-address=BC:A9:93:46:CF:71 server=vlan2-dhcp
add address=10.130.2.74 mac-address=BC:A9:93:47:5E:11 server=vlan2-dhcp
add address=10.130.2.76 mac-address=BC:A9:93:47:2B:21 server=vlan2-dhcp
add address=10.130.2.77 mac-address=BC:A9:93:46:BC:11 server=vlan2-dhcp
add address=10.130.2.78 mac-address=BC:A9:93:46:84:C9 server=vlan2-dhcp
add address=10.130.2.79 mac-address=BC:A9:93:46:6F:39 server=vlan2-dhcp
add address=10.130.2.81 mac-address=BC:A9:93:46:84:D9 server=vlan2-dhcp
add address=10.130.2.86 mac-address=BC:A9:93:47:29:61 server=vlan2-dhcp
add address=10.130.2.72 mac-address=BC:A9:93:46:ED:39 server=vlan2-dhcp
add address=10.130.2.73 mac-address=BC:A9:93:46:B2:69 server=vlan2-dhcp
add address=10.130.2.80 mac-address=BC:A9:93:46:C9:79 server=vlan2-dhcp
add address=10.130.2.87 mac-address=BC:A9:93:47:1E:09 server=vlan2-dhcp
add address=10.130.2.88 mac-address=BC:A9:93:47:9F:E9 server=vlan2-dhcp
add address=10.130.2.89 mac-address=BC:A9:93:47:9F:D9 server=vlan2-dhcp
add address=10.130.2.90 mac-address=BC:A9:93:46:87:59 server=vlan2-dhcp
add address=10.130.2.92 mac-address=BC:A9:93:47:A2:F9 server=vlan2-dhcp
add address=10.130.2.93 mac-address=BC:A9:93:47:2B:D1 server=vlan2-dhcp
add address=10.130.2.19 mac-address=BC:A9:93:46:C5:69 server=vlan2-dhcp
add address=10.130.2.94 mac-address=BC:A9:93:47:A3:E9 server=vlan2-dhcp
add address=10.130.2.96 mac-address=BC:A9:93:46:F4:51 server=vlan2-dhcp
add address=10.130.2.58 mac-address=BC:A9:93:46:F4:41 server=vlan2-dhcp
add address=10.130.2.98 mac-address=BC:A9:93:47:1D:D9 server=vlan2-dhcp
add address=10.130.2.99 mac-address=BC:A9:93:46:AE:49 server=vlan2-dhcp
add address=10.130.2.82 mac-address=BC:A9:93:46:ED:31 server=vlan2-dhcp
add address=10.130.2.100 mac-address=BC:A9:93:47:2B:C9 server=vlan2-dhcp
add address=10.130.2.95 mac-address=BC:A9:93:47:6A:19 server=vlan2-dhcp
add address=10.130.2.252 mac-address=BC:A9:93:46:D0:39 server=vlan2-dhcp
add address=10.130.2.25 mac-address=BC:A9:93:47:A1:F9 server=vlan2-dhcp
add address=10.130.2.91 mac-address=BC:A9:93:47:86:01 server=vlan2-dhcp
add address=10.130.2.34 mac-address=BC:A9:93:46:C9:71 server=vlan2-dhcp
/ip dhcp-server network
add address=10.130.1.0/24 comment=Servers gateway=10.130.1.1
add address=10.130.2.0/23 comment="Subscribers - CPE" dns-server=\
    8.8.8.8,1.1.1.1 gateway=10.130.2.1
add address=10.130.4.0/23 comment="Subscribers - SM" dns-server=\
    8.8.8.8,1.1.1.1 gateway=10.130.4.1
/ip dns
set servers=1.1.1.1,8.8.8.8

/ip firewall mangle
add action=accept chain=prerouting disabled=yes dst-address=98.97.64.0/21 \
    in-interface=bridge-lan
add action=accept chain=prerouting disabled=yes dst-address=98.97.96.0/21 \
    in-interface=bridge-lan
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-1-wan1 new-connection-mark=to_wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-2-wan2 new-connection-mark=to_wan2 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-3-wan3 new-connection-mark=to_wan3 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-4-wan4 new-connection-mark=to_wan4 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-5-wan5 new-connection-mark=to_wan5 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-6-wan6 new-connection-mark=to_wan6 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-7-wan7 new-connection-mark=to_wan7 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-8-wan8 new-connection-mark=to_wan8 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-9-wan9 new-connection-mark=to_wan9 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes in-interface=sfp28-10-wan10 new-connection-mark=to_wan10 passthrough=\
    yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan1 passthrough=yes per-connection-classifier=both-addresses:10/0
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan2 passthrough=yes per-connection-classifier=both-addresses:10/1
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan3 passthrough=yes per-connection-classifier=both-addresses:10/2
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan4 passthrough=yes per-connection-classifier=both-addresses:10/3
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan5 passthrough=yes per-connection-classifier=both-addresses:10/4
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan6 passthrough=yes per-connection-classifier=both-addresses:10/5
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan7 passthrough=yes per-connection-classifier=both-addresses:10/6
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan8 passthrough=yes per-connection-classifier=both-addresses:10/7
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan9 passthrough=yes per-connection-classifier=both-addresses:10/8
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=\
    yes dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan10 passthrough=yes per-connection-classifier=both-addresses:10/9
add action=mark-routing chain=prerouting connection-mark=to_wan1 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan2 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan3 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan4 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan5 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan5 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan6 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan6 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan7 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan7 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan8 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan8 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan9 disabled=yes \
    in-interface=bridge-lan new-routing-mark=to_wan9 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=to_wan10 disabled=\
    yes in-interface=bridge-lan new-routing-mark=to_wan10 passthrough=yes
add action=mark-connection chain=forward comment="For Sonar" \
    dst-address-list="Standard Internet" new-connection-mark=\
    Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list=\
    "Standard Internet"
add action=mark-connection chain=forward dst-address-list=\
    "High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=High_Capacity_Traffic_Up passthrough=yes \
    src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" connection-mark=\
    Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=\
    yes
add action=mark-packet chain=forward connection-mark=\
    High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up \
    passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-1-wan1
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-2-wan2
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-3-wan3
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-4-wan4
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-5-wan5
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-6-wan6
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-7-wan7
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-8-wan8
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-9-wan9
add action=masquerade chain=srcnat disabled=yes out-interface=sfp28-10-wan10
add action=masquerade chain=srcnat out-interface-list=WAN
add action=dst-nat chain=dstnat comment="Cacti HTTP" dst-port=180 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=80
add action=dst-nat chain=dstnat comment="Cacti HTTPS" dst-port=1443 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=443
add action=dst-nat chain=dstnat comment="Preseem SSH" dst-port=22222 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=22
add action=dst-nat chain=dstnat comment="Preseem Cockpit" dst-port=9090 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=9090
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
/ip route
add disabled=no dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=no dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-1-wan1 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=2 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-2-wan2 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=3 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-3-wan3 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=4 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-4-wan4 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=5 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-5-wan5 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=6 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-6-wan6 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=7 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-7-wan7 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=8 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-8-wan8 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=9 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-9-wan9 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=10 dst-address=0.0.0.0/0 \
    gateway=98.97.96.1%sfp28-10-wan10 routing-table=main suppress-hw-offload=\
    no
add check-gateway=ping disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-1-wan1 routing-table=to_wan1 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=2 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-2-wan2 routing-table=to_wan2 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=3 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-3-wan3 routing-table=to_wan3 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=4 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-4-wan4 routing-table=to_wan4 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=5 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-5-wan5 routing-table=to_wan5 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=6 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-6-wan6 routing-table=to_wan6 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=7 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-7-wan7 routing-table=to_wan7 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=8 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-8-wan8 routing-table=to_wan8 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=9 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-9-wan9 routing-table=to_wan9 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=10 dst-address=0.0.0.0/0 \
    gateway=98.97.96.1%sfp28-10-wan10 routing-table=to_wan10 \
    suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=myCa disabled=no port=8443
set api disabled=yes
set api-ssl certificate=myCa
/ip ssh
set strong-crypto=yes
/snmp
set contact="Richesin Engineering <jaysen@richesinengineering.com>" enabled=\
    yes location="Communications Shelter, Unalakleet, AK [63.89890275724056, -\
    160.7788028412428]" trap-community=atbunal
/system clock
set time-zone-autodetect=no time-zone-name=America/Anchorage
/system identity
set name=UnalakleetRouter
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.pool.ntp.org
add address=1.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/system scheduler
add disabled=yes interval=5m name=QueueTreeUsageCheck on-event=":log info \"Qu\
    eueTreeUsageCheck script started\"\r\
    \nQueueTreeUsageCheck\r\
    \n:log info \"QueueTreeUsageCheck script finished\"" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-03 start-time=11:04:00
add disabled=yes interval=4w2d name=datareset on-event=\
    "/queue tree reset-counters numbers=[find]" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-11-01 start-time=09:33:17
add disabled=yes interval=5m name=runThrottle on-event=\
    "/system script run throttle;" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-23 start-time=00:00:00
/system script
add dont-require-permissions=no name=rogue-dhcp-scan owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    log info \"Found rogue DHCP server at \$[/system clock get date] \$[/syste\
    m clock get time] on interface bridge-trunk]\"\r\
    \n#Email\r\
    \n/tool e-mail send to=\"jaysen@richesinengineering.com\" subject=\"Found \
    rogue DHCP server at \$[/system clock get date] \$[/system clock get time]\
    \_on interface bridge-trunk\""
add dont-require-permissions=no name=throttle owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local thresholdBytes 161061273600; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"Standard_Download_Sp\
    eed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"Standard_Upload_Speed\
    \"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}\r\
    \n\r\
    \n:local thresholdBytes 268435456000; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"High_Capacity_Downlo\
    ad_Speed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"High_Capacity_Upload_S\
    peed\"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}"
/tool bandwidth-server
set enabled=no
/tool e-mail
set from="Unalakleet Router" port=2525 server=mail.smtp2go.com tls=yes user=\
    alerts@richesinengineering.com
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
/tool traffic-monitor
add interface=sfp28-3-wan3 name=tmon1

sindy · Wed Nov 15, 2023 10:58 pm

I had just added -wanN to the end of the default names but willing to rename them to simplify things if necessary.

Depending on the order of creating the DHCP client, renaming the interface it is attached to, and possibly rebooting the router in the past the result may be different. The lease script uses the interface name to find, add, and modify the routes, and on my test router it was remembering the previous name. So forcing the DHCP client to renew an address after adding the script using /interface/ethernet/disable sfp28-N-wanN ; delay 2s ; /interface/ethernet/enable sfp28-N-wanN rather than /ip/dhcp-client/release [find where interface=sfp28-N-wanN] may be a better choice after renaming the interface or if you are not certain.

Good news about the satellite visibility ... OneWeb can't even begin to compete ...

Frankly speaking the simulation at starlink.sx has scared me a lot as it is showing long (as in tens of minutes) outages in coverage for Unalakleet, so I hope they are using outdated/inaccurate data. LEO satellites are superior to geostationary ones for both delay and throughput per square mile of coverage. Far lower distance to travel so far lower delay, far lower attenuation to compensate, and far bigger area to cover by a single antenna with a given throughput. But in these areas, polar trajectories seem to be mandatory so it's more or less a separate (from the main one covering the belt from 52°S to 52°N) Starlink constellation. And OneWeb probably didn't get that far (yet) so geostationary is their only choice for those areas.

I did have a question about the failover section and while I did set the distances in the routes. I am starting to wonder whether I even need to do that since all the gateways are the same on most of the terminals so if it goes down then it seems like it would be down for every connection that uses the same gateway?

From my experience with other Starlink installations in "bypass" mode, the gateway is probably redundant - if you look into the ARP table, you will probably see the MAC addresses of all (currently "both") the gateway IPs to begin with 00:00:5E:00:. So it is more likely that all your terminals lose connection than that a gateway in just one of the subnets becomes unreachable.

So I maintain my previouos position that we have to deal with issues we have to capacity to deal with, i.e. a breakdown of a single terminal. The only satellite within reach gone bonkers does not fit into this category.

As for route distances - these determine the priority among routes whose dst-address and routing-table parameters are identical. If multiple such routes are eligible for being active, those with the lowest value of distance are actually made active, and if there are multiple such ones, they are used in a round-robin manner (ECMP). So as @anav probably wrote earlier - in order that failover and load distribution worked in accord, you need to have one routing table per uplink that is used for traffic that should prefer that uplink but can send the traffic via other uplinks if the preferred one becomes unusable. In the simplest to configure case, you define just one backup uplink for each preferred one, so if the preferred one dies, the backup one has to bear its full load on top of its own one. In the optimized case, the load of the failed link is evenly distributed among all the remaining ones. So much more lines of configuration with much more room for mistakes but potentially less impact on customers if a terminal eventually fails. Choose your poison.

So for starters, let me give you an example for the L2TP, which is the most wanted functionality right now I gather.

You'll add a routing table named for-l2tp with two routes, one with distance=1 and the other one with distance=2, using the two WANs for which you have modified the DHCP client behavior. Let's say you've chosen WAN 9 and WAN 10:

/routing/table/add name=for-l2tp fib
/ip/route/add routing-table=for-l2tp gateway=1.1.1.1%sfp28-10-wan10 distance=1
/ip/route/add routing-table=for-l2tp gateway=1.1.1.1%sfp28-9-wan9 distance=2

By forcing a release followed by a re-lease (pun intended) of the DHCP address as described above, you'll trigger the lease script that should replace the 1.1.1.1 by the correct addresses of the respective gateways in these routes.

There are multiple possible ways to make the L2TP connection use this table; since the own WAN addresses of the Mikrotik are dynamic and since you identify the VPN server by its fqdn, so I assume its IP address may also change, all these ways require that the routing table to be used was assigned using mangle rules in chain output.

So let's make any connection from the router itself to your VPN server use this routing table. To make sure it won't break once the server migrates to another IP address, we'll add an address list to track the fqdn:
/ip/firewall/address-list/add list=re-vpn address=vpn.richesineng.com
It should create a dynamic item in the same address list but with the actual IP address as address. If it doesn't, something is wrong with the DNS setting.

Then, you add a single rule to the very top of chain output in mangle (which is easy as that chain is totally empty now):
/ip/firewall/mangle/add chain=output dst-address-list=re-vpn action=mark-routing new-routing-mark=for-l2tp passthrough=no

It seems this should be it; there is an additional issue, though. A packet sent by the router itself must first be routed using routing table main before it can get to mangle chain output, so it gets assigned some source address depending on the out-interface chosen by main. If mangle assigns a routing-mark to it, the routing is done again, but the source address remains the same. So a src-nat (or masquerade) rule must replace it by the address of the out-interface. This is OK until the uplink connected to that out-interface stops working without the interface going physically down. The connection tracking only removes the connection from its inventory if it has been src-nated using a masquerade rule and if the reply-dst-address assigned by the masquerade rule goes missing. Starlink seems to lease the addresses for 5 minutes (but that's for CGNAT ones, the public ones may be treated in a different way); if you are happy with the L2TP connection being re-established some 5 minutes after the preferred uplink goes down, nothing else needs to be done. If not, it requires a housekeeping script that removes the address as soon as it detects the failure of the uplink. But again, step by step. Right now I'll be glad if you make the L2TP work without fancy stuff.

If these settings won't get the L2TP going, I'll have to see it online.

jaysen · Thu Nov 16, 2023 1:08 am

I had just added -wanN to the end of the default names but willing to rename them to simplify things if necessary.
Depending on the order of creating the DHCP client, renaming the interface it is attached to, and possibly rebooting the router in the past the result may be different. The lease script uses the interface name to find, add, and modify the routes, and on my test router it was remembering the previous name. So forcing the DHCP client to renew an address after adding the script using /interface/ethernet/disable sfp28-N-wanN ; delay 2s ; /interface/ethernet/enable sfp28-N-wanN rather than /ip/dhcp-client/release [find where interface=sfp28-N-wanN] may be a better choice after renaming the interface or if you are not certain.

Good news about the satellite visibility ... OneWeb can't even begin to compete ...
Frankly speaking the simulation at starlink.sx has scared me a lot as it is showing long (as in tens of minutes) outages in coverage for Unalakleet, so I hope they are using outdated/inaccurate data. LEO satellites are superior to geostationary ones for both delay and throughput per square mile of coverage. Far lower distance to travel so far lower delay, far lower attenuation to compensate, and far bigger area to cover by a single antenna with a given throughput. But in these areas, polar trajectories seem to be mandatory so it's more or less a separate (from the main one covering the belt from 52°S to 52°N) Starlink constellation. And OneWeb probably didn't get that far (yet) so geostationary is their only choice for those areas.

I did have a question about the failover section and while I did set the distances in the routes. I am starting to wonder whether I even need to do that since all the gateways are the same on most of the terminals so if it goes down then it seems like it would be down for every connection that uses the same gateway?
From my experience with other Starlink installations in "bypass" mode, the gateway is probably redundant - if you look into the ARP table, you will probably see the MAC addresses of all (currently "both") the gateway IPs to begin with 00:00:5E:00:. So it is more likely that all your terminals lose connection than that a gateway in just one of the subnets becomes unreachable.

So I maintain my previouos position that we have to deal with issues we have to capacity to deal with, i.e. a breakdown of a single terminal. The only satellite within reach gone bonkers does not fit into this category.

As for route distances - these determine the priority among routes whose dst-address and routing-table parameters are identical. If multiple such routes are eligible for being active, those with the lowest value of distance are actually made active, and if there are multiple such ones, they are used in a round-robin manner (ECMP). So as @anav probably wrote earlier - in order that failover and load distribution worked in accord, you need to have one routing table per uplink that is used for traffic that should prefer that uplink but can send the traffic via other uplinks if the preferred one becomes unusable. In the simplest to configure case, you define just one backup uplink for each preferred one, so if the preferred one dies, the backup one has to bear its full load on top of its own one. In the optimized case, the load of the failed link is evenly distributed among all the remaining ones. So much more lines of configuration with much more room for mistakes but potentially less impact on customers if a terminal eventually fails. Choose your poison.

So for starters, let me give you an example for the L2TP, which is the most wanted functionality right now I gather.

You'll add a routing table named for-l2tp with two routes, one with distance=1 and the other one with distance=2, using the two WANs for which you have modified the DHCP client behavior. Let's say you've chosen WAN 9 and WAN 10:
Code: Select all
/routing/table/add name=for-l2tp fib
/ip/route/add routing-table=for-l2tp gateway=1.1.1.1%sfp28-10-wan10 distance=1
/ip/route/add routing-table=for-l2tp gateway=1.1.1.1%sfp28-9-wan9 distance=2
By forcing a release followed by a re-lease (pun intended) of the DHCP address as described above, you'll trigger the lease script that should replace the 1.1.1.1 by the correct addresses of the respective gateways in these routes.

There are multiple possible ways to make the L2TP connection use this table; since the own WAN addresses of the Mikrotik are dynamic and since you identify the VPN server by its fqdn, so I assume its IP address may also change, all these ways require that the routing table to be used was assigned using mangle rules in chain output.

So let's make any connection from the router itself to your VPN server use this routing table. To make sure it won't break once the server migrates to another IP address, we'll add an address list to track the fqdn:
/ip/firewall/address-list/add list=re-vpn address=vpn.richesineng.com
It should create a dynamic item in the same address list but with the actual IP address as address. If it doesn't, something is wrong with the DNS setting.

Then, you add a single rule to the very top of chain output in mangle (which is easy as that chain is totally empty now):
/ip/firewall/mangle/add chain=output dst-address-list=re-vpn action=mark-routing new-routing-mark=for-l2tp passthrough=no

It seems this should be it; there is an additional issue, though. A packet sent by the router itself must first be routed using routing table main before it can get to mangle chain output, so it gets assigned some source address depending on the out-interface chosen by main. If mangle assigns a routing-mark to it, the routing is done again, but the source address remains the same. So a src-nat (or masquerade) rule must replace it by the address of the out-interface. This is OK until the uplink connected to that out-interface stops working without the interface going physically down. The connection tracking only removes the connection from its inventory if it has been src-nated using a masquerade rule and if the reply-dst-address assigned by the masquerade rule goes missing. Starlink seems to lease the addresses for 5 minutes (but that's for CGNAT ones, the public ones may be treated in a different way); if you are happy with the L2TP connection being re-established some 5 minutes after the preferred uplink goes down, nothing else needs to be done. If not, it requires a housekeeping script that removes the address as soon as it detects the failure of the uplink. But again, step by step. Right now I'll be glad if you make the L2TP work without fancy stuff.

If these settings won't get the L2TP going, I'll have to see it online.

Hello Sindy,

I am pleased to say that our VPN connection is working again and I have verified that I can access all local devices in Unalakleet remotely. I am interested in the housekeeping script even though 5 minutes isn't a huge deal since it's just for remote access. That's one problem solved.

Now I just need to get our Sonar Billing instance re-established as it controls the DHCP server and customer account packages. Sonar is integrated with the MikroTik using the secure API on it's default port of 8729. Connections from sonar come from a source IP of 52.158.209.86 which is static. This IP is standard across all Sonar instances in the USA per documentation. On the Sonar side under the networking settings. The only settings needed for the MikroTik are: WAN IP, Username, Password and Port Number as shown in attached screenshot. This was all working prior just like the VPN so just need some guidance on what to change on the MikroTik. The difference is this is a connection coming into the MikroTik.

Currently I have no filter rules that would block incoming connections, Firewall is wide open until I get it working then I will work on the security.

I assume I will need separate routing table and address list so those I have already created. I would like to use the same interfaces 9 and 10 to keep things simple if possible.

Thank You!

Jaysen

sindy · Thu Nov 16, 2023 2:30 am

Currently I have no filter rules that would block incoming connections, Firewall is wide open until I get it working then I will work on the security.

A big fat NO for this. Firewall is the first thing to deal with when you connect something directly to the internet, always, no exceptions. The filth from the net is incredibly fast to squat in. Since the VPN works, you should be able to only allow connection to the router itself via the VPN and via LAN as backup and block everything else except what you know needs to be open.

I am pleased to say that our VPN connection is working again and I have verified that I can access all local devices in Unalakleet remotely.

Great.

Now I just need to get our Sonar Billing instance re-established...
The only settings needed for the MikroTik are: WAN IP, Username, Password and Port Number as shown in attached screenshot. This was all working prior just like the VPN so just need some guidance on what to change on the MikroTik. The difference is this is a connection coming into the MikroTik.

I assume I will need separate routing table and address list so those I have already created. I would like to use the same interfaces 9 and 10 to keep things simple if possible.

The router always answers incoming connections from the same address to which they have arrived, but you need to make sure it will also send them via the correct interface because it does not choose a routing table up to a source address automatically. We need to let it use a routing table that contains a single route through the correct interface or at least has such a "correct" route as the most preferred one. If the WAN addresses were static, you could use routing rules for that, but that's not possible here (unless we would make the lease script update them as well) so we need mangle rules again. The generic way to do this is to use connection marking where we save the information about the in-interface of the initial packet of the incoming connection to the context of that connection maintained by the connection tracking module and use it to assign a routing mark to the response packets. So add a rule to the top of chain prerouting of mangle, and keep the rest of rules in that chain disabled for now so that none of them would overwrite the connection mark assigned by this one.

/ip firewall mangle print chain=prerouting where !dynamic
/ip/firewall/mangle/add chain=prerouting place-before=0 in-interface=sfp28-10-wan10 connection-state=new action=mark-connection new-connection-mark=use-wan10

Then, add a rule to chain output of mangle that will assign an appropriate routing mark based on this connection mark:

/ip/firewall/mangle add chain=output connection-mark=use-wan10 action=mark-routing new-routing-mark=for-l2tp passthrough=no

Now the router should respond via wan10 any connections that arrive to the IP address assigned to wan10.

It is just a quick solution because I am almost asleep; tomorrow I'll give you some additional information regarding possible redundancy scenarios.

jaysen · Thu Nov 16, 2023 2:50 am

Thanks again for the help Today and I do agree with the points you made regarding the firewall. Now that VPN is working again, I will do that. I am going to work on fixing the Sonar connection and then setup firewall like I have with all my Oregon sites. Have a good rest. I'll look for any responses here Tomorrow.

sindy · Thu Nov 16, 2023 7:41 pm

A couple of questions regarding Sonar:

can it use an FQDN of the router it manages rather than an IPv4 number?
does it require a continuous connection or it is not an issue if it loses contact for minutes?

jaysen · Thu Nov 16, 2023 7:57 pm

A couple of questions regarding Sonar:
can it use an FQDN of the router it manages rather than an IPv4 number?

does it require a continuous connection or it is not an issue if it loses contact for minutes?

Hello Sindy,

Sonar is working again as of last night. Unfortunately it does not have the capability to use FQDN and not sure if they will add that in future updates. I would like that option though. I don't think a few minutes would be too big of an issue but it's preferred that it be continuous as it controls the DHCP server leases and plan speeds. When it's offline I have to do everything manually on the MIkroTik when adding/removing customers.

gotsprings · Thu Nov 16, 2023 8:35 pm

I would look at BigLeaf.

Let them Bond and Distribute Across the feeds back to their VPS. Then on to the internet.

jaysen · Thu Nov 16, 2023 9:28 pm

I would look at BigLeaf.

Let them Bond and Distribute Across the feeds back to their VPS. Then on to the internet.

Thank you for the suggestion. I will look into them. In the meantime, I hope to get the rest of these routes working today and the rest of the terminals into my NMS monitor. I do appreciate everyone's patience and assistance with all of these issues but progress is being made and client is happy. I have approx 14+ more villages in Alaska coming up so this is quite a learning experience. Unalakleet is the first 2.5 Ghz broadband installation in the state of Alaska (So I've been told) so they are going to be the example that we showcase to the rest of the villages. This is huge and people's lives are going to be positively impacted due to all this work. This will enable access to distance learning opportunities, Access to TeleMedicine and of course the occasional Call of Duty match.

anav · Thu Nov 16, 2023 9:41 pm

gotsprings looks like your trying to put mikrotik wan solutions under the bus LOL. Here I am trying to figure out optimal failover WAN approaches and it turns out I just need to use BigLeaf.....................
Please send $$$$

sindy · Thu Nov 16, 2023 10:49 pm

Unfortunately it does not have the capability to use FQDN and not sure if they will add that in future updates.

The reason for the question was that the router could update a DNS record in your company DNS if the latter has an API for that, or using the Mikrotik "ip cloud" service, or using some 3rd party dynamic DNS service that has a simple enough API. The Mikrotik service has a limitation of a single IP address per device, a fixed hostname generated from the serial number of the device (so to stay safe when using the FQDN to reach that device from too many other devices or if a VPN connection established towards that FQDN is the only way how those other devices can be reached, you still need another DNS with a CNAME pointing to that fixed FQDN based on device serial number), and I have seen it to be down for days in the past which apparently wasn't Mikrotik's fault but for some it was a really tough time.

The use case here would be that if the chosen terminal would fail, the router would use a backup one to update the dynamic DNS with the public address of the public one, and the Sonar could reconnect.

I don't think a few minutes would be too big of an issue but it's preferred that it be continuous as it controls the DHCP server leases and plan speeds. When it's offline I have to do everything manually on the MIkroTik when adding/removing customers.

Well, my question was whether it was a provisioning tool (which your explanation seems to confirm) or whether it directly controlled the traffic (in terms of e.g. cutting off a client if they run off their quota, I have no clue what your business offer is up there). A provisioning tool only has to work when you actively use it, so you can manually change the address if the currently configured one changes or dies; a traffic policing tool needs a constant connection.

What you can do to make VPN traffic switch to a backup WAN far sooner than the VPN client detects an outage and re-establishes the connection via the backup WAN is to establish two VPN tunnels, one using strictly the preferred WAN and the other one using strictly the backup one, and let one of these VPN tunnels be a backup for the other one. This approach doesn't suffer from the issue of src-nated connection surviving an outage of the uplink until the lease is lost as mentioned earlier, but it requires a compatible setup at the VPN server side.

With such a setup, you can also set up port forwarding for Sonar on one of your machines in a datacenter with a static public address to the CCR in Unalakleet via this pair of VPN tunnels. But it is still only a protection against failure of a terminal, it cannot handle an absence of a functional satellite within reach or a failure of the gateway machine.

Totally unrelated, you have mentioned that people keep getting security warnings. Some web sites like to handle requests within the same application session by different servers at their end, but check whether all the requests come from the same public address at client side and either reject them or at least issue security warnings if not. To avoid this, the hash in the per-connection-classifier matcher must be calculated solely from the src-address. This means that all outgoing connections of a given LAN address will always get mapped to the same WAN address (unless it fails of course), so the traffic will not be distributed as evenly as when you hash both addresses and ports, but it may be bearable (and also controllable to some extent, you can change the addresses of the clients that generate most traffic to evenly distribute them over the WANs manually).

jaysen · Fri Nov 17, 2023 1:12 am

Unfortunately it does not have the capability to use FQDN and not sure if they will add that in future updates.

The reason for the question was that the router could update a DNS record in your company DNS if the latter has an API for that, or using the Mikrotik "ip cloud" service, or using some 3rd party dynamic DNS service that has a simple enough API. The Mikrotik service has a limitation of a single IP address per device, a fixed hostname generated from the serial number of the device (so to stay safe when using the FQDN to reach that device from too many other devices or if a VPN connection established towards that FQDN is the only way how those other devices can be reached, you still need another DNS with a CNAME pointing to that fixed FQDN based on device serial number), and I have seen it to be down for days in the past which apparently wasn't Mikrotik's fault but for some it was a really tough time.

The use case here would be that if the chosen terminal would fail, the router would use a backup one to update the dynamic DNS with the public address of the public one, and the Sonar could reconnect.

Sonar Support has said that everything that I see in Sonar can be controlled via webhooks so I am wondering if there is some way to automate updating the IP address via a webhook should it change. I don't have much experience with that but I will reach out to Sonar to see if that is possible.

I don't think a few minutes would be too big of an issue but it's preferred that it be continuous as it controls the DHCP server leases and plan speeds. When it's offline I have to do everything manually on the MIkroTik when adding/removing customers.

Well, my question was whether it was a provisioning tool (which your explanation seems to confirm) or whether it directly controlled the traffic (in terms of e.g. cutting off a client if they run off their quota, I have no clue what your business offer is up there). A provisioning tool only has to work when you actively use it, so you can manually change the address if the currently configured one changes or dies; a traffic policing tool needs a constant connection.

Correct, Sonar is an all-in-one solution that handles provisioning and it ties in with the Mikrotik and a Preseem appliance to apply shaping to keep customers on their provision package speeds as well as cut off delinquent customers. The only thing Preseem does not do is throttle down a customer when they exceed their data cap. However, I am working with Sonar and have a set of scripts that work with address lists and queues to do the Throttling which I will be implementing at a later time once all the rest of this is setup.

The way I currently handle any IP change is by creating multiple inline device and dhcp server entries in sonar. I would add all the terminal IP addresses in there and leave all disabled except for the current active one. Then all I need to do should a change happen is switch between them. Sonar says this is fine to do but to not have more than one enabled at a time as that could cause problems with having duplicate leases. I've only had to do it once when we still had OneWeb and a backup Starlink set as a failover. The OneWeb terminal failed and the Mikrotik failed over to the Starlink as expected. I then disabled the OneWeb and enabled the Starlink in Sonar and all was good.

What you can do to make VPN traffic switch to a backup WAN far sooner than the VPN client detects an outage and re-establishes the connection via the backup WAN is to establish two VPN tunnels, one using strictly the preferred WAN and the other one using strictly the backup one, and let one of these VPN tunnels be a backup for the other one. This approach doesn't suffer from the issue of src-nated connection surviving an outage of the uplink until the lease is lost as mentioned earlier, but it requires a compatible setup at the VPN server side.

With such a setup, you can also set up port forwarding for Sonar on one of your machines in a datacenter with a static public address to the CCR in Unalakleet via this pair of VPN tunnels. But it is still only a protection against failure of a terminal, it cannot handle an absence of a functional satellite within reach or a failure of the gateway machine.

We have been exploring this option of providing a static via VPN tunnels and do own several blocks of IP addresses that we could use but we are not quite ready to set that up yet. We are in the process of building a new NOC in Anchorage that will house monitoring systems and the ability to provide these static addresses. With so many projects going on in multiple states, that process has been slowed down.

Totally unrelated, you have mentioned that people keep getting security warnings. Some web sites like to handle requests within the same application session by different servers at their end, but check whether all the requests come from the same public address at client side and either reject them or at least issue security warnings if not. To avoid this, the hash in the per-connection-classifier matcher must be calculated solely from the src-address. This means that all outgoing connections of a given LAN address will always get mapped to the same WAN address (unless it fails of course), so the traffic will not be distributed as evenly as when you hash both addresses and ports, but it may be bearable (and also controllable to some extent, you can change the addresses of the clients that generate most traffic to evenly distribute them over the WANs manually).

Could you provide some assistance with getting this going? I have read a bit about PCC and started working on it with the help of anov who was the first to respond to my post. I do have many mangle rules and static routes created but they are currently disabled until I am certain they are configured correctly. I think my most recent config that I provided will show them but if not I can provide it again.

Thank You!

sindy · Fri Nov 17, 2023 2:12 am

Could you provide some assistance with getting this going? I have read a bit about PCC and started working on it with the help of anov who was the first to respond to my post. I do have many mangle rules and static routes created but they are currently disabled until I am certain they are configured correctly. I think my most recent config that I provided will show them but if not I can provide it again.

I can assist, just bear in mind the time shift. But your last config posted should work if you enable the mangle rules and the routes and redo the QoS-related mangle rules in chain forward so that they would not use connection marks, because they overwrite those assigned in prerouting.

To do so, take all the match conditions of the mangle forward rule that assigns the connection mark, add them to the mangle forward rule that currently translates that connection mark to a packet mark, remove the match on connection mark from the latter rule, and remove the former rule completely. Do this 4 times and that's it.

If no route is active in a routing table indicated by the routing mark, the system uses routing table main instead (unless a routing rule explicitly prohibits that). So the existing configuration does contain a backup for the case that a single Starlink terminal stops working.

As I wrote earlier, it is better to replace both-addresses by src-address in the per-connection-classifier.

jaysen · Fri Nov 17, 2023 2:54 am

Could you provide some assistance with getting this going? I have read a bit about PCC and started working on it with the help of anov who was the first to respond to my post. I do have many mangle rules and static routes created but they are currently disabled until I am certain they are configured correctly. I think my most recent config that I provided will show them but if not I can provide it again.
I can assist, just bear in mind the time shift. But your last config posted should work if you enable the mangle rules and the routes and redo the QoS-related mangle rules in chain forward so that they would not use connection marks, because they overwrite those assigned in prerouting.

To do so, take all the match conditions of the mangle forward rule that assigns the connection mark, add them to the mangle forward rule that currently translates that connection mark to a packet mark, remove the match on connection mark from the latter rule, and remove the former rule completely. Do this 4 times and that's it.

If no route is active in a routing table indicated by the routing mark, the system uses routing table main instead (unless a routing rule explicitly prohibits that). So the existing configuration does contain a backup for the case that a single Starlink terminal stops working.

As I wrote earlier, it is better to replace both-addresses by src-address in the per-connection-classifier.

Hi Sindy,

It might be best to start fresh with the mangle rules. The ones I had originally created were based on another posters advice which is different. While I appreciate their help. What you and I have done up to this point, things are beginning to work. Since they are all disabled, It might be easier for me to follow what you are saying if I start fresh. I've been pulled in all different directions the last few days with multiple clients so my train of thought on this is a little off. I am thinking we get rid of all the disabled ones in the screenshot or at least the ones that are not needed. Then make corrections to the ones I do need based on what you are saying. It'll make more sense to me that way. I appreciate your understanding as this is new to me.

Thank You

gotsprings · Fri Nov 17, 2023 3:23 am

gotsprings looks like your trying to put mikrotik wan solutions under the bus LOL. Here I am trying to figure out optimal failover WAN approaches and it turns out I just need to use BigLeaf.....................
Please send $$$$

Speedify has a lab that ran that at one point.

But Big Leaf seams a bit more business oriented.

sindy · Fri Nov 17, 2023 5:58 pm

It might be best to start fresh with the mangle rules.

Let's go that way if you like, but I'd prefer a more interactive communication channel than the forum. This kind of "share the wisdom" sites is great to describe typical setups and principles so that others could follow them, but there are so many topics dealing with load distribution&backup here that I can't see any point in documenting the process here for the 500th time, so only the waste of time remains. So please consider following this post.

Fri Nov 17, 2023 6:12 pm

FWIW the PCC youtube vid made by MT is quite good.
https://www.youtube.com/watch?v=nlb7XAv57tw

Used it again to clean up an AC3 LTE setup for PCC sharing across VDSL and LTE.
Only, THIS time I disabled the subtitles which all of a sudden made me see a couple of important things I missed the previous time I saw that video.
And now it works stable and reliably

Just saying ...

jaysen · Fri Nov 17, 2023 6:22 pm

Hello Sindy,

I did more work on it last night. I updated and activated all the mangle rules plus added a few more. I also added in all the masquerade rules for each interface. So far things appear to be working great. All interfaces are responding to ping and SNMP queries so I am now monitoring them in our NMS. Sonar and VPN connections are still working and no complaints so far from the village that internet isn't working. I'm out of the office most of the morning but plan to work more on it later Today.

I did setup some basic firewall rules using the section on that in the Wiki but it still needs a little more work which I also plan to complete Today. I'll post an updated config later tonight and you can make any suggestions on how to improve things.

A question I have on that lease script. I assume that is meant to be placed inside the script section in each DHCP client? It makes sense to me that it would be appropriate to put it there but figured I would ask.

I do have a remote windows box at the tower as a backup way into the network in case VPN fails for any reason. I am going to play around with some DDNS clients then work with Sonar to see if they have a backend way of making things work with an FQDN.

Thanks again for all the help

jaysen · Fri Nov 17, 2023 6:22 pm

FWIW the PCC youtube vid made by MT is quite good.
https://www.youtube.com/watch?v=nlb7XAv57tw

Used it again to clean up an AC3 LTE setup for PCC sharing across VDSL and LTE.
Only, THIS time I disabled the subtitles which all of a sudden made me see a couple of important things I missed the previous time I saw that video.
And now it works stable and reliably

Just saying ...

Awesome, I'll take a look at it. Thanks for sharing

anav · Fri Nov 17, 2023 7:20 pm

FWIW the PCC youtube vid made by MT is quite good.
https://www.youtube.com/watch?v=nlb7XAv57tw

Used it again to clean up an AC3 LTE setup for PCC sharing across VDSL and LTE.
Only, THIS time I disabled the subtitles which all of a sudden made me see a couple of important things I missed the previous time I saw that video.
And now it works stable and reliably

Just saying ...

Actually, I just viewed this video and I think it has some serious flaws LOL, and no I am not just saying that to contradict holvoe, as much as fun as that is.

.

sindy · Fri Nov 17, 2023 7:20 pm

A question I have on that lease script. I assume that is meant to be placed inside the script section in each DHCP client? It makes sense to me that it would be appropriate to put it there but figured I would ask.

To be precise, it is meant to be placed just once into the /system script section, and its name to be placed to the script item of each DHCP client. You could put the complete script to the script item of each DHCP client (that item is interpreted in a contextual way, i.e. if the contents is a single word, it is interpreted as a script name, otherwise it is interpreted as a script code), but that would be a maintenance nightmare.

Referring to the script this way still allows you to create a modified instace for some special purpose or just for testing new features and let just one of the DHCP clients use it.

Fri Nov 17, 2023 7:37 pm

Ok Anav.
Eagerly awaiting your version then ...
Let's see if it is better.

anav · Fri Nov 17, 2023 8:11 pm

I'm too sexy for youtube video ;-PP { he did a decent job of reading the teleprompter which was slightly to the left of the camera, above is better LOL )

Nothing major but without discussing really what the heck his purpose was for putting distances on the routes......... Assuming some sort of very rudimentary failover??
But why........ it clouds the PCC thrust of the video. Failover in PCC is a much different beast and people will get the wrong idea.

The other bone of contention I have is the fact that he uses check-gateway=ping on the fib routes AND the default main table routes.
My understanding was that there is no need for check-gateway=ping on fib routes if already entered for the corresponding default routes (main table).
Maybe I am wrong??

He used passthrough=yes vice no for expediency I suppose (if there are no other output chain rules, passthrough=no is technically better).

jaysen · Fri Nov 17, 2023 11:40 pm

Hello Sindy,

I am now getting reports from the village that speeds are super slow and web pages are not loading since enabling all the routes and mangle rules. Both myself and my Chief Engineer have been looking at it and can't seem to figure out what might have caused this problem. I can still access everything over the VPN and the connection seems fast for me. I was able to login and browse from remote PC so i know the internet is working. I am including my latest config to see if maybe you see something I am missing.

Thank You!

Jaysen

# 2023-11-17 10:43:01 by RouterOS 7.12
# software id = DVNU-PTAC
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HE808NY9TF8
/interface bridge
add admin-mac=48:A9:8A:85:65:FB auto-mac=no comment="Link to NV1560 Port 1" \
    name=bridge-lan
/interface ethernet
set [ find default-name=ether1 ] disabled=yes
set [ find default-name=qsfp28-1-1 ] disabled=yes
set [ find default-name=qsfp28-1-2 ] disabled=yes
set [ find default-name=qsfp28-1-3 ] disabled=yes
set [ find default-name=qsfp28-1-4 ] disabled=yes
set [ find default-name=qsfp28-2-1 ] disabled=yes
set [ find default-name=qsfp28-2-2 ] disabled=yes
set [ find default-name=qsfp28-2-3 ] disabled=yes
set [ find default-name=qsfp28-2-4 ] disabled=yes
set [ find default-name=sfp28-1 ] comment="Link to Starlink Terminal 1" name=\
    sfp28-1-wan1
set [ find default-name=sfp28-2 ] comment="Link to Starlink Terminal 2" name=\
    sfp28-2-wan2
set [ find default-name=sfp28-3 ] comment="Link to Starlink Terminal 3" name=\
    sfp28-3-wan3
set [ find default-name=sfp28-4 ] comment="Link to Starlink Terminal 4" name=\
    sfp28-4-wan4
set [ find default-name=sfp28-5 ] comment="Link to Starlink Terminal 5" name=\
    sfp28-5-wan5
set [ find default-name=sfp28-6 ] comment="Link to Starlink Terminal 6" name=\
    sfp28-6-wan6
set [ find default-name=sfp28-7 ] comment="Link to Starlink Terminal 7" name=\
    sfp28-7-wan7
set [ find default-name=sfp28-8 ] comment="Link to Starlink Terminal 8" name=\
    sfp28-8-wan8
set [ find default-name=sfp28-9 ] comment="Link to Starlink Terminal 9" name=\
    sfp28-9-wan9
set [ find default-name=sfp28-10 ] comment="Link to Starlink Terminal 10" \
    name=sfp28-10-wan10
set [ find default-name=sfp28-11 ] disabled=yes
set [ find default-name=sfp28-12 ] comment="Link to Adtran Switch Port 1" \
    name=sfp28-12-lan
/interface l2tp-client
add connect-to=vpn.richesineng.com disabled=no name=RE use-ipsec=yes user=\
    unalakleet
/interface vlan
add comment=Customers interface=sfp28-12-lan name=vlan2 vlan-id=2
/interface list
add comment="contains WAN interfaces" name=WAN
add comment="contains LAN interfaces" name=LAN
add comment="Contains VPN interfaces" name=VPN
add name=WAN1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add authoritative=after-2sec-delay interface=bridge-lan lease-time=5m name=\
    vlan2-dhcp
/ip pool
add name=vlan2-pool ranges=10.130.2.10-10.130.3.254
/port
set 0 name=serial0
/queue type
add kind=pcq name=MaxDownloadSpeed pcq-classifier=src-address pcq-rate=10M
add kind=pcq name=MaxUploadSpeed pcq-classifier=dst-address pcq-rate=1M
add kind=pcq name=Standard_Download_Speed pcq-burst-rate=30M pcq-burst-time=\
    15s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=25M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=Standard_Upload_Speed pcq-burst-rate=3600k pcq-burst-time=\
    15s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=3M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Download_Speed pcq-burst-rate=30M \
    pcq-burst-time=15s pcq-classifier=dst-address pcq-dst-address6-mask=64 \
    pcq-rate=25M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Upload_Speed pcq-burst-rate=3600k \
    pcq-burst-time=15s pcq-classifier=src-address pcq-dst-address6-mask=64 \
    pcq-rate=3M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
/queue tree
add max-limit=300M name=Standard_Download_Speed packet-mark=\
    Standard_Traffic_Down parent=bridge-lan priority=7 queue=\
    Standard_Download_Speed
add max-limit=50M name=Standard_Upload_Speed packet-mark=Standard_Traffic_Up \
    parent=bridge-lan priority=7 queue=Standard_Upload_Speed
add max-limit=300M name=High_Capacity_Download_Speed packet-mark=\
    High_Capacity_Traffic_Down parent=bridge-lan priority=7 queue=\
    High_Capacity_Download_Speed
add max-limit=50M name=High_Capacity_Upload_Speed packet-mark=\
    High_Capacity_Traffic_Up parent=bridge-lan priority=7 queue=\
    High_Capacity_Upload_Speed
/routing table
add fib name=to_wan1
add fib name=to_wan2
add fib name=to_wan3
add fib name=to_wan4
add fib name=to_wan5
add fib name=to_wan6
add fib name=to_wan7
add fib name=to_wan8
add fib name=to_wan9
add fib name=to_wan10
add disabled=no fib name=for_l2tp
add disabled=no fib name=for_sonar
/snmp community
set [ find default=yes ] disabled=yes
add addresses=::/0 name=atbunal
/interface bridge port
add bridge=bridge-lan interface=vlan2
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=bridge-lan list=LAN
add interface=sfp28-1-wan1 list=WAN
add interface=sfp28-2-wan2 list=WAN
add comment="RE VPN" interface=RE list=VPN
add interface=sfp28-3-wan3 list=WAN
add interface=sfp28-4-wan4 list=WAN
add interface=sfp28-5-wan5 list=WAN
add interface=sfp28-6-wan6 list=WAN
add interface=sfp28-7-wan7 list=WAN
add interface=sfp28-8-wan8 list=WAN
add interface=sfp28-9-wan9 list=WAN
add interface=sfp28-10-wan10 list=WAN
/ip address
add address=10.130.1.1/24 comment="Servers - vlan1" interface=sfp28-12-lan \
    network=10.130.1.0
add address=10.130.2.1/23 comment="Subscribers - CPE" interface=vlan2 \
    network=10.130.2.0
add address=10.130.4.1/23 comment="Subscribers - SM" interface=vlan2 network=\
    10.130.4.0
/ip dhcp-client
add add-default-route=no interface=sfp28-1-wan1 use-peer-dns=no
add add-default-route=no interface=sfp28-2-wan2 use-peer-dns=no
add add-default-route=no interface=sfp28-3-wan3 use-peer-dns=no
add add-default-route=no interface=sfp28-4-wan4 use-peer-dns=no
add add-default-route=no interface=sfp28-5-wan5 use-peer-dns=no
add add-default-route=no interface=sfp28-6-wan6 use-peer-dns=no
add add-default-route=no interface=sfp28-7-wan7 use-peer-dns=no
add add-default-route=no interface=sfp28-8-wan8 use-peer-dns=no
add add-default-route=no interface=sfp28-9-wan9 use-peer-dns=no
add add-default-route=no interface=sfp28-10-wan10 use-peer-dns=no
/ip dhcp-server alert
add comment="Rogue DHCP Detect" disabled=no interface=bridge-lan on-alert=\
    ":log error message=\"Rogue DHCP Server Discovered\"" valid-server=\
    48:A9:8A:85:65:FB

/ip firewall filter
add action=accept chain=input comment=\
    "Accept established,related connections" connection-state=\
    established,related disabled=yes
add action=accept chain=input comment=\
    "Accept all connections from allowed_to_router address list" disabled=yes \
    src-address-list=allowed_to_router
add action=accept chain=input comment="Accept ICMP connections" disabled=yes \
    protocol=icmp
add action=accept chain=input comment="Accept SNMP connections" disabled=yes \
    dst-port=161 protocol=udp
add action=accept chain=input comment=\
    "Accept L2TP/IPSEC connections (500,1701,4500/udp)" disabled=yes \
    dst-port=1701,500,4500 protocol=udp
add action=accept chain=input comment="Accept WinBox connections (8291/tcp)" \
    disabled=yes dst-port=8291 protocol=tcp
add action=accept chain=input comment=\
    "For L2TP/IPSEC Encryption and Authentication" disabled=yes protocol=\
    ipsec-esp
add action=drop chain=input comment="Drop everything else" disabled=yes
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-1-wan1 new-connection-mark=to_wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-2-wan2 new-connection-mark=to_wan2 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-3-wan3 new-connection-mark=to_wan3 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-4-wan4 new-connection-mark=to_wan4 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-5-wan5 new-connection-mark=to_wan5 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-6-wan6 new-connection-mark=to_wan6 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-7-wan7 new-connection-mark=to_wan7 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-8-wan8 new-connection-mark=to_wan8 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-9-wan9 new-connection-mark=to_wan9
add action=mark-connection chain=prerouting connection-state=new \
    in-interface=sfp28-10-wan10 new-connection-mark=to_wan10 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan1 passthrough=yes per-connection-classifier=src-address:10/0
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan2 passthrough=yes per-connection-classifier=src-address:10/1
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan3 passthrough=yes per-connection-classifier=src-address:10/2
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan4 passthrough=yes per-connection-classifier=src-address:10/3
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan5 passthrough=yes per-connection-classifier=src-address:10/4
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan6 passthrough=yes per-connection-classifier=src-address:10/5
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan7 passthrough=yes per-connection-classifier=src-address:10/6
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan8 passthrough=yes per-connection-classifier=src-address:10/7
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan9 passthrough=yes per-connection-classifier=src-address:10/8
add action=mark-connection chain=prerouting connection-state=new \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=\
    to_wan10 passthrough=yes per-connection-classifier=src-address:10/9
add action=mark-routing chain=prerouting connection-mark=to_wan1 \
    in-interface=bridge-lan new-routing-mark=to_wan1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan2 \
    in-interface=bridge-lan new-routing-mark=to_wan2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan3 \
    in-interface=bridge-lan new-routing-mark=to_wan3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan4 \
    in-interface=bridge-lan new-routing-mark=to_wan4 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan5 \
    in-interface=bridge-lan new-routing-mark=to_wan5 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan6 \
    in-interface=bridge-lan new-routing-mark=to_wan6 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan7 \
    in-interface=bridge-lan new-routing-mark=to_wan7 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan8 \
    in-interface=bridge-lan new-routing-mark=to_wan8 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan9 \
    in-interface=bridge-lan new-routing-mark=to_wan9 passthrough=no
add action=mark-routing chain=prerouting connection-mark=to_wan10 \
    in-interface=bridge-lan new-routing-mark=to_wan10 passthrough=no
add action=mark-connection chain=forward comment="For Sonar" \
    dst-address-list="Standard Internet" new-connection-mark=\
    Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list=\
    "Standard Internet"
add action=mark-connection chain=forward dst-address-list=\
    "High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=High_Capacity_Traffic_Up passthrough=yes \
    src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" connection-mark=\
    Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=\
    yes
add action=mark-packet chain=forward connection-mark=\
    High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up \
    passthrough=yes
add action=mark-routing chain=output connection-mark=to_wan1 \
    new-routing-mark=to_wan1 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan2 \
    new-routing-mark=to_wan2 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan3 \
    new-routing-mark=to_wan3 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan4 \
    new-routing-mark=to_wan4 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan5 \
    new-routing-mark=to_wan5 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan6 \
    new-routing-mark=to_wan6 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan7 \
    new-routing-mark=to_wan7 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan8 \
    new-routing-mark=to_wan8 passthrough=no
add action=mark-routing chain=output dst-address-list=re-vpn \
    new-routing-mark=for_l2tp passthrough=no
add action=mark-routing chain=output connection-mark=to_wan9 \
    new-routing-mark=for_sonar passthrough=no
add action=mark-routing chain=output connection-mark=to_wan10 \
    new-routing-mark=to_wan10 passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp28-1-wan1
add action=masquerade chain=srcnat out-interface=sfp28-2-wan2
add action=masquerade chain=srcnat out-interface=sfp28-3-wan3
add action=masquerade chain=srcnat out-interface=sfp28-4-wan4
add action=masquerade chain=srcnat out-interface=sfp28-5-wan5
add action=masquerade chain=srcnat out-interface=sfp28-6-wan6
add action=masquerade chain=srcnat out-interface=sfp28-7-wan7
add action=masquerade chain=srcnat out-interface=sfp28-8-wan8
add action=masquerade chain=srcnat out-interface=sfp28-9-wan9
add action=masquerade chain=srcnat out-interface=sfp28-10-wan10
add action=dst-nat chain=dstnat comment="Cacti HTTP" dst-port=180 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=80
add action=dst-nat chain=dstnat comment="Cacti HTTPS" dst-port=1443 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.10 to-ports=443
add action=dst-nat chain=dstnat comment="Preseem SSH" dst-port=22222 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=22
add action=dst-nat chain=dstnat comment="Preseem Cockpit" dst-port=9090 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=9090
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
/ip route
add disabled=no dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add disabled=no dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-1-wan1 routing-table=main suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-2-wan2 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-3-wan3 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-4-wan4 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-5-wan5 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-6-wan6 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-7-wan7 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-8-wan8 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-9-wan9 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-10-wan10 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-1-wan1 routing-table=to_wan1 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-2-wan2 routing-table=to_wan2 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-3-wan3 routing-table=to_wan3 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-4-wan4 routing-table=to_wan4 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-5-wan5 routing-table=to_wan5 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp28-6-wan6 routing-table=to_wan6 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-7-wan7 routing-table=to_wan7 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-8-wan8 routing-table=to_wan8 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-9-wan9 routing-table=to_wan9 suppress-hw-offload=no
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-10-wan10 routing-table=to_wan10 suppress-hw-offload=no
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp28-9-wan9 pref-src="" routing-table=for_l2tp scope=30 \
    suppress-hw-offload=no target-scope=10
add distance=1 gateway=98.97.96.1%sfp28-9-wan9 routing-table=for_sonar
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=myCa disabled=no port=8443
set api disabled=yes
set api-ssl certificate=myCa
/ip ssh
set strong-crypto=yes
/snmp
set contact="Richesin Engineering <jaysen@richesinengineering.com>" enabled=\
    yes location="Communications Shelter, Unalakleet, AK [63.89890275724056, -\
    160.7788028412428]" trap-community=atbunal
/system clock
set time-zone-autodetect=no time-zone-name=America/Anchorage
/system identity
set name=UnalakleetRouter
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.pool.ntp.org
add address=1.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/system scheduler
add disabled=yes interval=5m name=QueueTreeUsageCheck on-event=":log info \"Qu\
    eueTreeUsageCheck script started\"\r\
    \nQueueTreeUsageCheck\r\
    \n:log info \"QueueTreeUsageCheck script finished\"" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-03 start-time=11:04:00
add disabled=yes interval=4w2d name=datareset on-event=\
    "/queue tree reset-counters numbers=[find]" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-11-01 start-time=09:33:17
add disabled=yes interval=5m name=runThrottle on-event=\
    "/system script run throttle;" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-23 start-time=00:00:00
/system script
add dont-require-permissions=no name=rogue-dhcp-scan owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    log info \"Found rogue DHCP server at \$[/system clock get date] \$[/syste\
    m clock get time] on interface bridge-trunk]\"\r\
    \n#Email\r\
    \n/tool e-mail send to=\"jaysen@richesinengineering.com\" subject=\"Found \
    rogue DHCP server at \$[/system clock get date] \$[/system clock get time]\
    \_on interface bridge-trunk\""
add dont-require-permissions=no name=throttle owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local thresholdBytes 161061273600; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"Standard_Download_Sp\
    eed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"Standard_Upload_Speed\
    \"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}\r\
    \n\r\
    \n:local thresholdBytes 268435456000; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"High_Capacity_Downlo\
    ad_Speed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"High_Capacity_Upload_S\
    peed\"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}"
add dont-require-permissions=no name=lease-script owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="i\
    f (\$bound=1) do={\
    \n \r\
    \n/ip route {\
    \n \r\
    \n\r\
    \n:if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interfac\
    e\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n \r\
    \nadd gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n \r\
    \n}\
    \n \r\
    \n\r\
    \n:foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"inter\
    face\") !dynamic !blackhole] do={\
    \n \r\
    \nset \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n \r\
    \n}\
    \n \r\
    \n}\r\
    \n\
    \n}\
    \n"
/tool bandwidth-server
set enabled=no
/tool e-mail
set from="Unalakleet Router" port=2525 server=mail.smtp2go.com tls=yes user=\
    alerts@richesinengineering.com
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
/tool traffic-monitor
add interface=sfp28-3-wan3 name=tmon1

anav · Fri Nov 17, 2023 11:59 pm

(1) I personally like to use a different connection mark for the PCC LAN traffic, then what I used for the WANS, just for ease of readings, but thats personal preference.

I personally do not understand what are the ramifications, if any, by
having the first set of mangle rule group for routing marks be in-interface=bridge-lan
and the second set of mangle rule group for routing marks just be solely based on the same connections marks?.

add action=mark-routing chain=prerouting connection-mark=to_wan1 \
in-interface=bridge-lan new-routing-mark=to_wan1 passthrough=no
add action=mark-routing chain=output connection-mark=to_wan1 \
new-routing-mark=to_wan1 passthrough=no

it may very well be you dont need the first set if all covered by the more widely scoped second set???.
In any case shouldnt be causing any issues your users are potentially experiencing, as the setup looks pretty standard.

(2) I believe its probably better to leave passthrough=yes for all the Routing Mark rules for PCC Lan traffic.
For the WAN associated Routing-Mark rules, since you dont seem to have other output rules, passthrough=no seems to be fine.

(3) For the IP routes, you may want to consider for the Main (Table) WAN Routes, to put a different distance between each route so that the router has some direction in terms of what happens when a starlink associated gateway is inactive. Without any distance, I am not sure what the Routter will do?

(4) Lastly, I am not sure of the value in putting check-gateway=ping on all the EXTRA routes associated with the specific tables. My impression is that if the MAIN table route with the same gateway, was inactive based on its check-gateway=ping, all associated routes (aka with same gateway) would also be considered inactive. I could be wrong though.

sindy · Sat Nov 18, 2023 12:20 am

Hello Sindy,

I've sent you an e-mail, check your spam folder if you haven't received it.

jaysen · Sat Nov 18, 2023 12:32 am

Hello Sindy,
I've sent you an e-mail, check your spam folder if you haven't received it.

Hello Sindy,

I got your email and tried disabling the forward mangles as you suggested. Although those are needed for Sonar per their documentation so I'll need to figure that out if that is the source of the issue. I am still waiting to hear back from the village but in any case it seems to have broken our VPN connection as the logs are showing it trying but failing to establish a connection.

sindy · Sat Nov 18, 2023 12:39 am

I ... tried disabling the forward mangles as you suggested. ... in any case it seems to have broken our VPN connection as the logs are showing it trying but failing to establish a connection.

The only way how disabling only the mangle rules in forward chain could affect the L2TP would be that it would stop limiting the LAN->WAN traffic and therefore the L2TP packets would get lost due to bandwidth exhaustion caused by the user traffic now not throttled by the QoS queues, and even that is just a speculation.

Anyway, easy to test - re-enable them and see whether that makes L2TP happy again.

The reason why I've sent na e-mail is that the notifications from the forum are currently coming even 30 minutes after the post, and I'm not stuck to the screen.

sindy · Sat Nov 18, 2023 12:47 pm

... tried disabling the forward mangles as you suggested. Although those are needed for Sonar per their documentation ...

A very important question here is whether Sonar needs just the effect of these rules or these rules must be there literally because Sonar's API scripts check or even manipulate them.

The very purpose of a connection mark is to be stored in the context data of a connection only once, when handling any of its packets, and be given as an additional attribute to match on to the firewall rules handling all subsequent packets of the same connection, regardless their direction. Its uses are basically two:

to speed up the processing of packets in the firewall by only evaluating a complex set of match conditions once, typically when handling the initial packet of each connection, and storing the result in the form of a connection mark. This becomes even more important if connections whose packets match different sets of match conditions need to be treated the same, so instead of having e.g. three mangle rules handling every packets of every connection, you have a single rule for mid-connection packets that matches on the connection mark alone and the three ones matching on complex sets of conditions only handle the initial ones.
to "remember" information that can only be obtained when handling some of the packets of the connection so that you could handle other packets of that connection according to that information.

The connection marks assigned by the rules in prerouting that match on per-connection-classifier represent the first case (although if the per connection classification rules were the only element of the configuration to be used to choose a routing table, the connection marking would not be necesary); the connection marks assigned by the rules in prerouting that match on in-interface represent the second case (but they are actually only required in setups where you use port forwarding of requests coming in from the internet to some servers in the LAN, which doesn't seem to be your case).

The issue with RouterOS is that although the netfilter itself can treat the connection mark attribute as a bitmap, Mikrotik uses it as a scalar value, so you can have many distinct connection marks but only one can be assigned to a given connection at a time; if you assign a new one, you rewrite the previous one.

In your particular setup, the overwriting does not affect the own traffic of the router because it doesn't pass through mangle chain forward.

Now back to the question regarding Sonar, if it is enough that the firewall assigns the proper packet marks and it is not important how exactly it does so, you can simplify the Sonar rules from the existing

add action=mark-connection chain=forward comment="For Sonar" dst-address-list="Standard Internet" new-connection-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list="Standard Internet"
add action=mark-connection chain=forward dst-address-list="High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan new-connection-mark=High_Capacity_Traffic_Up passthrough=yes src-address-list="High Capacity Internet"

add action=mark-packet chain=forward comment="For Sonar" connection-mark=Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up passthrough=yes

to just

add action=mark-packet chain=forward comment="For Sonar" dst-address-list="Standard Internet" new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=yes src-address-list="Standard Internet"
add action=mark-packet chain=forward dst-address-list="High Capacity Internet" new-packet-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up passthrough=yes src-address-list="High Capacity Internet"

The effect will be the same except that they will not destroy the function of the use of connection marks for choice of WAN.

You can also save some extra nanoseconds per packet if you arrange the rules assigning a packet mark in such an order that most packets only got handled by least rules and set passthrough to no; this is possible because there are (currently!) no additional rules in mangle chain forward that would have to be applied once the packet mark has been assigned. We assume there will be more download packets than upload ones, and we want to handle the "High Capacity Internet" ones even a tiny bit faster than the "Standard Internet" ones, so the outcome would look as follows:

add action=mark-packet chain=forward dst-address-list="High Capacity Internet" new-packet-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=no
add action=mark-packet chain=forward in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up passthrough=no src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" dst-address-list="Standard Internet" new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=no
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=no src-address-list="Standard Internet"

have been looking at it and can't seem to figure out what might have caused this problem

What helps a lot are the live counters that show you the traffic volume on the individual interfaces and on the firewall rules, and also the possibility to list the tracked connections up to multiple criteria. So if you can see that the action=mark-routing rules handle about the same amount of packets like the action=per-connection-classifier rules, it clearly indicates a problem as the former ones should handle way more than the latter ones, as the former ones handle each LAN->WAN packet of every connection whereas the latter ones only handle the first packet of each connection.

Sniffing the traffic of a particular public address that is rarely used by any of the customers (like one of a web site about Egyptian pyramids hosted in a small data center in southern America) without limiting the sniff to a particular interface can show you whether all the traffic of a TCP sesssion indeed uses the same WAN interface as required - of course in cooperation with a real or simulated customer who will access that site for you.

gotsprings · Sat Nov 18, 2023 2:57 pm

I have to admit that there are many here who could mop the floor with me when it comes to "Carrier Grade" networks.

But seriously... I really don't get how this would work like at all.

When I work on something like a an office with 300 users... With multiple ISPs... I can't get load balancing to work 100%. With every single page these days pulling in feeds from multiple locations... And sites requiring log ins from a specific IP address... And if the IP changes... You have to log back in. I mean voip and video calling take it the worst, actually wifi calling takes it the worst.

As one feed overloads or drops services fail. Then they need to move to another connection which breaks up or needs to be reestablished.

I mean really help me out... Because I just don't get it.

I really fight to get some sort of dynamic stryping to a VPS provider. That way all connections come from the same public IP. And if a feed fails or doesn't work... The other feeds pick up the slack and the other side still sees the one singular connection. This keeps calls from dropping. Services from seeing an IP change and dropping the connection. Etc.

None of this seems to be addressed in this configuration.

sindy · Sat Nov 18, 2023 3:08 pm

With every single page these days pulling in feeds from multiple locations... And sites requiring log ins from a specific IP address... And if the IP changes... You have to log back in. I mean voip and video calling take it the worst, actually wifi calling takes it the worst.

As for src-nating the same user always to the same public IP address when spreading the load across WANs, per-connection-classifier taking only src-address into account is sufficient. Failures and WAN address changes are another thing of course. In my experience so far, WiFi calling normally recovers quite fast from a public address change, incoming SIP calls to phones with long re-registration intervals may be a different story indeed.

I really fight to get some sort of dynamic stryping to a VPS provider. That way all connections come from the same public IP. And if a feed fails or doesn't work... The other feeds pick up the slack and the other side still sees the one singular connection. This keeps calls from dropping. Services from seeing an IP change and dropping the connection. Etc.

This would work of course, the VPS being the SPOF (but that's not a big deal as decent data centers automatically respawn VMs on another hardware in case of a failure). What might be a problem would be running out of ephemeral TCP ports on a single public IP if too many clients would access the same service.

jaysen · Sat Nov 18, 2023 3:19 pm

... tried disabling the forward mangles as you suggested. Although those are needed for Sonar per their documentation ...
A very important question here is whether Sonar needs just the effect of these rules or these rules must be there literally because Sonar's API scripts check or even manipulate them.

The very purpose of a connection mark is to be stored in the context data of a connection only once, when handling any of its packets, and be given as an additional attribute to match on to the firewall rules handling all subsequent packets of the same connection, regardless their direction. Its uses are basically two:
to speed up the processing of packets in the firewall by only evaluating a complex set of match conditions once, typically when handling the initial packet of each connection, and storing the result in the form of a connection mark. This becomes even more important if connections whose packets match different sets of match conditions need to be treated the same, so instead of having e.g. three mangle rules handling every packets of every connection, you have a single rule for mid-connection packets that matches on the connection mark alone and the three ones matching on complex sets of conditions only handle the initial ones.

to "remember" information that can only be obtained when handling some of the packets of the connection so that you could handle other packets of that connection according to that information.
The connection marks assigned by the rules in prerouting that match on per-connection-classifier represent the first case (although if the per connection classification rules were the only element of the configuration to be used to choose a routing table, the connection marking would not be necesary); the connection marks assigned by the rules in prerouting that match on in-interface represent the second case (but they are actually only required in setups where you use port forwarding of requests coming in from the internet to some servers in the LAN, which doesn't seem to be your case).

The issue with RouterOS is that although the netfilter itself can treat the connection mark attribute as a bitmap, Mikrotik uses it as a scalar value, so you can have many distinct connection marks but only one can be assigned to a given connection at a time; if you assign a new one, you rewrite the previous one.

In your particular setup, the overwriting does not affect the own traffic of the router because it doesn't pass through mangle chain forward.

Now back to the question regarding Sonar, if it is enough that the firewall assigns the proper packet marks and it is not important how exactly it does so, you can simplify the Sonar rules from the existing

add action=mark-connection chain=forward comment="For Sonar" dst-address-list="Standard Internet" new-connection-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list="Standard Internet"
add action=mark-connection chain=forward dst-address-list="High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan new-connection-mark=High_Capacity_Traffic_Up passthrough=yes src-address-list="High Capacity Internet"

add action=mark-packet chain=forward comment="For Sonar" connection-mark=Standard_Traffic_Down new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up passthrough=yes

to just

add action=mark-packet chain=forward comment="For Sonar" dst-address-list="Standard Internet" new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=yes src-address-list="Standard Internet"
add action=mark-packet chain=forward dst-address-list="High Capacity Internet" new-packet-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up passthrough=yes src-address-list="High Capacity Internet"

The effect will be the same except that they will not destroy the function of the use of connection marks for choice of WAN.

You can also save some extra nanoseconds per packet if you arrange the rules assigning a packet mark in such an order that most packets only got handled by least rules and set passthrough to no; this is possible because there are (currently!) no additional rules in mangle chain forward that would have to be applied once the packet mark has been assigned. We assume there will be more download packets than upload ones, and we want to handle the "High Capacity Internet" ones even a tiny bit faster than the "Standard Internet" ones, so the outcome would look as follows:

add action=mark-packet chain=forward dst-address-list="High Capacity Internet" new-packet-mark=High_Capacity_Traffic_Down out-interface=bridge-lan passthrough=no
add action=mark-packet chain=forward in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up passthrough=no src-address-list="High Capacity Internet"
add action=mark-packet chain=forward comment="For Sonar" dst-address-list="Standard Internet" new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan passthrough=no
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=no src-address-list="Standard Internet"

have been looking at it and can't seem to figure out what might have caused this problem
What helps a lot are the live counters that show you the traffic volume on the individual interfaces and on the firewall rules, and also the possibility to list the tracked connections up to multiple criteria. So if you can see that the action=mark-routing rules handle about the same amount of packets like the action=per-connection-classifier rules, it clearly indicates a problem as the former ones should handle way more than the latter ones, as the former ones handle each LAN->WAN packet of every connection whereas the latter ones only handle the first packet of each connection.

Sniffing the traffic of a particular public address that is rarely used by any of the customers (like one of a web site about Egyptian pyramids hosted in a small data center in southern America) without limiting the sniff to a particular interface can show you whether all the traffic of a TCP sesssion indeed uses the same WAN interface as required - of course in cooperation with a real or simulated customer who will access that site for you.

Hello Sindy,

This is the documentation I used when setting up the Sonar connection marks. I am not sure if it answers your question but this is all I really know about it.

https://docs.sonar.expert/networking/mi ... ing-speeds

I ended up rolling back all of my changes to get everything back online but now we have another problem. Apparently all of our 10 Starlink terminals are offline so I have no remote access. I am working with the village to find out more on what happened there. No change that I made should have taken the terminals down and they were all up when I ended my work day. Unfortunately the only tech that I work with left yesterday for a week so I am not sure how soon I will have it resolved but I am going to try to get someone on-site Today when everyone wakes up. Otherwise I might be catching a plane up there to investigate.

sindy · Sat Nov 18, 2023 3:43 pm

This is the documentation ... I am not sure if it answers your question but this is all I really know about it.

It does. The good news is that as they ask you to create the rules manually, they are interested in their ultimate effect, not in the particular rules themselves.

Apparently all of our 10 Starlink terminals are offline so I have no remote access. I am working with the village to find out more on what happened there.

The fact that you can work with the people there suggests that at least a telephone service is available. Is there a chance to use that service to get a good old dial-up channel to be able to access the network if something like this happens, or is it a satellite phone as well so data service is not available, or at least not quickly?

jaysen · Sat Nov 18, 2023 4:28 pm

This is the documentation ... I am not sure if it answers your question but this is all I really know about it.
It does. The good news is that as they ask you to create the rules manually, they are interested in their ultimate effect, not in the particular rules themselves.

Apparently all of our 10 Starlink terminals are offline so I have no remote access. I am working with the village to find out more on what happened there.
The fact that you can work with the people there suggests that at least a telephone service is available. Is there a chance to use that service to get a good old dial-up channel to be able to access the network if something like this happens, or is it a satellite phone as well so data service is not available, or at least not quickly?

Unfortunately not, The phone service there is VOIP and normally runs on our network but had to be switched over to their very slow backup GCI cellular link. The tower site is about 3 miles away and does not have any phone lines into the shelter. I suspect the starlink issue is something physical either with power or cabling and I hope to have the other tech on-site this morning to work with him on bringing those back online. I do have the Remote PC at the site and another single Starlink which can get me access. We have tried to get the village to pay for a GCI connection to use as an OOB circuit but that was not approved. Hopefully after this experience they change their mind. I am sick of not having redundancy in place for these reasons alone.

Once things are back online. I am hoping we can change the mangles for Sonar to work. I'll post another update later today once I get the outage resolved.

Thank You

sirbryan · Sat Nov 18, 2023 9:02 pm

I know I'm coming in on this conversation late, but I'd like to throw in a couple ideas that you can look at once you get Starlink squared away.

1) I would use Wireguard (if possible) for the management VPN. If one end (your office) has a static IP, the remote end(s) (the router(s) behind Starlink) can be configured to always connect to that IP, and it reconnects really fast.

2) With one or more (Wireguard) tunnels from the router over each Starlink terminal back to a well-homed router, you can load balance more "cleanly" across all connections, eliminate PCC complexities, and protect against losing a number of active connections as a result of terminal failure (as suggested in an earlier post). This works behind whatever kind of connection you have, be it Starlink, LTE, old-school satellite, etc.

On my network, I use wireless gear as a primary link and a VPN back to my datacenter-hosted router over residential cable or FTTH as a secondary link to some of my relay (micro pop) sites. With OSPF and BFD, the failover is quick and the additional latency is minimal. The only other thing I have to add is a mangle rule to the VPN tunnel to clamp TCP MSS.

If this were given to me, I'd have one "border" router handling all the aggregation and tunneling back to a central location, and a second "core" handling all the CGNAT, traffic queuing, DHCP reservations, Sonar integration, etc. The VPN could be on either router, but probably the CGNAT router would be best for establishing a tunnel back to the office, giving the office router (and Sonar) direct access to the customer networks.

(I haven't read all the nitty gritty of the rest of the network internals, so I might be missing or overlooking a couple bits.)

Sindy's offered some amazing assistance, so I don't want to step on any toes. But I am in Utah, so timezones are less of an issue should you need more timely assistance.

anav · Sat Nov 18, 2023 9:13 pm

(1) Personally recommend you dont use the same entry names for connection marks and routing marks. It gets terribly hard to read.

For instance.
For the initial set of rules use connection marks incoming-WAN1 ( reflect traffic is originating from outside the router )
For the PCC traffic use connection marks : through-WAN1 (reflect traffic going through each wan to www )

All the Routing Marks are fine ( use is accurate ))

In the end it doesn't matter as all that traffic will be going to the same Route-Marks and thus personal preference.
(It probably is slightly more efficient not to ask the router to keep track of a different set of connection marks?)

(2) Missing one line in the mangle config
add action=mark-routing chain=output connection-mark=to_wan9 \
new-routing-mark=to_wan9 passthrough=no[/b]

sindy · Sat Nov 18, 2023 9:17 pm

1) I would use Wireguard (if possible) for the management VPN. If one end (your office) has a static IP, the remote end(s) (the router(s) behind Starlink) can be configured to always connect to that IP, and it reconnects really fast.

Event better, strictly speaking it even isn't a reconnection - if the peer on the static address receives a packet from the remote peer from a different address than before, it still recognizes it belongs to an already existing session and updates the peer address in its configuration so it sends the subsequent packets to the new address, no renegotiation etc. is required.

Sindy's offered some amazing assistance, so I don't want to step on any toes.

No problem with this on my side

anav · Sat Nov 18, 2023 10:01 pm

@Sir Bryan sounds fascinating!
Any chance we could see how you setup OSPF+BDF in RoS 7 for this to work??
An examples of the type of network arrangement your espousing may be of great potential use by the OP as an alternative approach to engineer in slow time, and of utmost interest to me as well.

[ I have been looking at a more simple Primary/Secondary WAN (non PCC or balancing) scenario with disparate WANs.
- one wireguard to CHR Server, L2TP plain (mrru to handle packet fragmentation)
- OSPF+BDF to monitor uplinks and decide which routes are available/active. ]
{email avail at my username if more comfortable sharing that way.}

jaysen · Sat Nov 18, 2023 11:01 pm

All systems are back online now. Turns out it was power issue. I think with the added terminals it put too much strain on the power system and tripped the breaker. We managed to get it all reset and I had the local tech unplug some servers that I am decommissioning. Also going to have the local utility increase the Amps to the facility so this doesn't happen again. Now I can read through all the previous posts and work on some changes that some of you suggested that I thought were great.

Larsa · Sun Nov 19, 2023 12:15 am

Just a side note before jumping on the SD-WAN train with solutions like BigLeaf, first make sure your objectives are in order before making any decisions.

Fwiw, ZeroTier is already integrated into ROS v.7. As for remote management, I’d choose ZeroTier anytime over WireGuard but the latter might serve well as backup access.

Good luck with the remaining installations in Alaska, it has been very interesting reading.

jaysen · Sun Nov 19, 2023 12:32 am

(1) Personally recommend you dont use the same entry names for connection marks and routing marks. It gets terribly hard to read.

For instance.
For the initial set of rules use connection marks incoming-WAN1 ( reflect traffic is originating from outside the router )
For the PCC traffic use connection marks : through-WAN1 (reflect traffic going through each wan to www )

All the Routing Marks are fine ( use is accurate ))

In the end it doesn't matter as all that traffic will be going to the same Route-Marks and thus personal preference.
(It probably is slightly more efficient not to ask the router to keep track of a different set of connection marks?)

(2) Missing one line in the mangle config
add action=mark-routing chain=output connection-mark=to_wan9 \
new-routing-mark=to_wan9 passthrough=no[/b]

(1) I like that idea and I am implementing that now. Just want to confirm, for all the marks I will create separate route tables (ie: incoming-wanX, through-wanX, outgoing-wanX). I agree this way will be easier to read and understand where all the traffic is going. I am also renaming all my interfaces to shorten them up.

(2) That has been fixed. Not sure how I missed that one. Thank You!

sindy · Sun Nov 19, 2023 12:53 am

(1) I like that idea and I am implementing that now.

Well, I don't like that idea, and we already had a private discussion about it with @anav.

Your configuration is already complicated enough, and creating multiple connection marks and routing marks/tables that get ultimately translated to the same route set is just and additional level of complication to me, which doesn't even improve readability (here we disagree with @anav) and creates additional load to the router.

The namespaces for connection marks and routing marks are isolated so the same string used in both causes no conflict. If the ultimate goal of assigning a connection mark is that packets with that connection mark would be sent via WAN 1, there is little point in having the connection mark indicate not only that WAN 1 should be used but also why it should be used. Nor is there any point in stressing out that it is a connection mark. But OK, that's still an issue of personal preferences, but if you use two different connection marks that get translated to the same routing mark, you need two different rules to perform the translation, as a single rule can only match on a single connection mark.

So in the simplest case, assigning connection mark "wan1" based on in-interface=wan1 for connections to the router itself initiated by clients in the internet that have to be responded via wan1 and assigning the same connection mark "wan1" to connections to servers in the internet initiated by LAN clients based on the per-connection-classifier "dispatching", and later translating connection mark "wan1" to a routing mark "wan1" regardless why the former one has been assigned just makes things more clear to me, rather than having to bear in mind that two connection mark to routing mark translation rules are necessary for each WAN. And forcing every single packet to run through twice more mangle rules would cause a pointless extra CPU load.

Having even multiple routing tables with a single default route via wan1, i.e. only differing by name, is again just adding more space for an error which, in my personal opinion, the "better readability" cannot justify.

jaysen · Sun Nov 19, 2023 1:11 am

(1) I like that idea and I am implementing that now.
Well, I don't like that idea, and we already had a private discussion about it with @anav.

Your configuration is already complicated enough, and creating multiple connection marks and routing marks/tables that get ultimately translated to the same route set is just and additional level of complication to me, which doesn't even improve readability (here we disagree with @anav) and creates additional load to the router.

The namespaces for connection marks and routing marks are isolated so the same string used in both causes no conflict. If the ultimate goal of assigning a connection mark is that packets with that connection mark would be sent via WAN 1, there is little point in having the connection mark indicate not only that WAN 1 should be used but also why it should be used. Nor is there any point in stressing out that it is a connection mark. But OK, that's still an issue of personal preferences, but if you use two different connection marks that get translated to the same routing mark, you need two different rules to perform the translation, as a single rule can only match on a single connection mark.

So in the simplest case, assigning connection mark "wan1" based on in-interface=wan1 for connections to the router itself initiated by clients in the internet that have to be responded via wan1 and assigning the same connection mark "wan1" to connections to servers in the internet initiated by LAN clients based on the per-connection-classifier "dispatching", and later translating connection mark "wan1" to a routing mark "wan1" regardless why the former one has been assigned just makes things more clear to me, rather than having to bear in mind that two connection mark to routing mark translation rules are necessary for each WAN. And forcing every single packet to run through twice more mangle rules would cause a pointless extra CPU load.

Having even multiple routing tables with a single default route via wan1, i.e. only differing by name, is again just adding more space for an error which, in my personal opinion, the "better readability" cannot justify.

I can certainly see both sides of this and while I do like things to be easier to read. I need things to be simple and not increase load on the router so I will have to go with Sindy's recommendation on this issue. Since I only manage this network. I want to set it up for success in the event something happens to me and someone else takes over. I still appreciate all input on this. Thank You! I'll post an updated config once I finish cleaning things up so we can review and make any adjustments before I activate it again. I still need to figure out how to do the mangles for Sonar as they were clearly an issue yesterday and village wasn't happy. Plus the power issue certainly didn't make things any better but at least I got that worked out.

sirbryan · Sun Nov 19, 2023 1:14 am

Fwiw, ZeroTier is already integrated into ROS v.7. As for remote management, I’d choose ZeroTier anytime over WireGuard but the latter might serve well as backup access.

ZeroTier is a great tool/service for automagically creating Wireguard tunnels between peers, especially for management & telemetry, as opposed to doing it manually. But it's overkill if you only need a simple PTP link and have control and stability at least at one end.

Larsa · Sun Nov 19, 2023 2:02 am

Well, I beg to differ. I believe that there is absolutely no exaggeration in striving to achieve simple configuration and administration of VPN links for network management.

On the contrary, SD-WAN like ZerTier is way much easier to manage compared to manually configured static links like WireGuard, especially considering dynamic addresses in CGNAT (double nat) networks for commercial and business-critical environments.

It's the essence of Software-Defined Networks. Once you’ve been there, you’ll never want to switch to anything else.

sirbryan · Sun Nov 19, 2023 2:55 am

ZeroTier is Wireguard, with their additional relay tech. And now MikroTik is doing their own thing with Back to Home. I'm not disputing the benefits, especially for the double NAT scenarios you speak of (both ends), where there is no choice but to relay. ZT and BTH make that easy.

But I also know my network requirements (and those of some of my clients), and if you have control of at least one end, if not both, reliance on a third party for critical or secure infrastructure (not to mention the additional latency) may not be tolerable.

As an ISP, I would not leverage ZeroTier to handle backbone traffic via VPN for my customers like I do with my vanilla Wireguard tunnels. For management of a few off-net customer routers, it's a great fit for me for all the reasons you say. For OP's Sonar case, the pros of simplicity/convenience for them would have to outweigh the cons of dependence upon a third party system. Standard risk assessment and cost/benefit analysis exercise.

anav · Sun Nov 19, 2023 3:16 am

Jaysen, just to be clear, I was ONLY talking about the connection-mark nomenclature for Mangle rules! There is no change to either the mangles routing-mark nomenclature or especially to any naming in the IP Routes. The mangle rules for marking routes should remain as is --> useWANX, as do the IP Routes.

You will note I specifically stated this.
Quote: " All the Routing Marks are fine ( use is accurate )) " unquote.

Apologies for any confusion caused, and yes I still find this small change (only to connection marks) helpful in reading and understanding the mangle rules.

jaysen · Sun Nov 19, 2023 4:38 am

Hello Sindy,

Posting an updated config after a bunch of cleanup Today. I am sure there is some changes I could make to improve things so please do let me know what you find that needs to be changed. I kept the PCC mangles for the bridge-lan disabled for now until those can be fixed but so far I don't see any problems at this time.

Thank You

# 2023-11-18 17:22:41 by RouterOS 7.12
# software id = DVNU-PTAC
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HE808NY9TF8
/interface bridge
add admin-mac=48:A9:8A:85:65:FB auto-mac=no comment="Access Bridge" name=\
    bridge-lan
/interface ethernet
set [ find default-name=ether1 ] comment="Not Used" disabled=yes
set [ find default-name=qsfp28-1-1 ] disabled=yes
set [ find default-name=qsfp28-1-2 ] disabled=yes
set [ find default-name=qsfp28-1-3 ] disabled=yes
set [ find default-name=qsfp28-1-4 ] disabled=yes
set [ find default-name=qsfp28-2-1 ] disabled=yes
set [ find default-name=qsfp28-2-2 ] disabled=yes
set [ find default-name=qsfp28-2-3 ] disabled=yes
set [ find default-name=qsfp28-2-4 ] disabled=yes
set [ find default-name=sfp28-1 ] comment="Link to Starlink Terminal 1" name=\
    sfp1-wan1
set [ find default-name=sfp28-2 ] comment="Link to Starlink Terminal 2" name=\
    sfp2-wan2
set [ find default-name=sfp28-3 ] comment="Link to Starlink Terminal 3" name=\
    sfp3-wan3
set [ find default-name=sfp28-4 ] comment="Link to Starlink Terminal 4" name=\
    sfp4-wan4
set [ find default-name=sfp28-5 ] comment="Link to Starlink Terminal 5" name=\
    sfp5-wan5
set [ find default-name=sfp28-6 ] comment="Link to Starlink Terminal 6" name=\
    sfp6-wan6
set [ find default-name=sfp28-7 ] comment="Link to Starlink Terminal 7" name=\
    sfp7-wan7
set [ find default-name=sfp28-8 ] comment="Link to Starlink Terminal 8" name=\
    sfp8-wan8
set [ find default-name=sfp28-9 ] comment="Link to Starlink Terminal 9" name=\
    sfp9-wan9
set [ find default-name=sfp28-10 ] comment="Link to Starlink Terminal 10" \
    name=sfp10-wan10
set [ find default-name=sfp28-11 ] comment="Not Used" disabled=yes name=sfp11
set [ find default-name=sfp28-12 ] comment=\
    "Attached to bridge-lan.  Link to NV1560 Switch - Port 1" name=sfp12-lan
/interface l2tp-client
add comment="Link to Richesin Engineering VPN" connect-to=vpn.richesineng.com \
    disabled=no name=RE use-ipsec=yes user=unalakleet
/interface vlan
add comment="Customer VLAN" interface=sfp12-lan name=vlan2 vlan-id=2
/interface list
add comment="contains WAN interfaces" name=WAN
add comment="contains LAN interfaces" name=LAN
add comment="Contains VPN interfaces" name=VPN
add name=WAN1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add authoritative=after-2sec-delay comment="Controlled by Sonar" interface=\
    bridge-lan lease-time=5m name=vlan2-dhcp
/port
set 0 name=serial0
/queue type
add kind=pcq name=MaxDownloadSpeed pcq-classifier=src-address pcq-rate=10M
add kind=pcq name=MaxUploadSpeed pcq-classifier=dst-address pcq-rate=1M
add kind=pcq name=Standard_Download_Speed pcq-burst-rate=30M pcq-burst-time=\
    15s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=25M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=Standard_Upload_Speed pcq-burst-rate=3600k pcq-burst-time=\
    15s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=3M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Download_Speed pcq-burst-rate=30M \
    pcq-burst-time=15s pcq-classifier=dst-address pcq-dst-address6-mask=64 \
    pcq-rate=25M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Upload_Speed pcq-burst-rate=3600k \
    pcq-burst-time=15s pcq-classifier=src-address pcq-dst-address6-mask=64 \
    pcq-rate=3M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
/queue tree
add max-limit=300M name=Standard_Download_Speed packet-mark=\
    Standard_Traffic_Down parent=bridge-lan priority=7 queue=\
    Standard_Download_Speed
add max-limit=50M name=Standard_Upload_Speed packet-mark=Standard_Traffic_Up \
    parent=bridge-lan priority=7 queue=Standard_Upload_Speed
add max-limit=300M name=High_Capacity_Download_Speed packet-mark=\
    High_Capacity_Traffic_Down parent=bridge-lan priority=7 queue=\
    High_Capacity_Download_Speed
add max-limit=50M name=High_Capacity_Upload_Speed packet-mark=\
    High_Capacity_Traffic_Up parent=bridge-lan priority=7 queue=\
    High_Capacity_Upload_Speed
/routing table
add disabled=no fib name=wan1
add disabled=no fib name=wan2
add disabled=no fib name=wan3
add disabled=no fib name=wan4
add disabled=no fib name=wan5
add disabled=no fib name=wan6
add disabled=no fib name=wan7
add disabled=no fib name=wan8
add disabled=no fib name=wan9
add disabled=no fib name=wan10
add disabled=no fib name=l2tp-sonar
/snmp community
set [ find default=yes ] disabled=yes
add addresses=::/0 name=atbunal
/interface bridge port
add bridge=bridge-lan interface=vlan2
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=bridge-lan list=LAN
add interface=sfp1-wan1 list=WAN
add interface=sfp2-wan2 list=WAN
add comment="RE VPN" interface=RE list=VPN
add interface=sfp3-wan3 list=WAN
add interface=sfp4-wan4 list=WAN
add interface=sfp5-wan5 list=WAN
add interface=sfp6-wan6 list=WAN
add interface=sfp7-wan7 list=WAN
add interface=sfp8-wan8 list=WAN
add interface=sfp9-wan9 list=WAN
add interface=sfp10-wan10 list=WAN
/ip address
add address=10.130.1.1/24 comment="Servers - vlan1" interface=sfp12-lan \
    network=10.130.1.0
add address=10.130.2.1/23 comment="Subscribers - CPE" interface=vlan2 \
    network=10.130.2.0
add address=10.130.4.1/23 comment="Subscribers - SM" interface=vlan2 network=\
    10.130.4.0
/ip dhcp-client
add add-default-route=no interface=sfp1-wan1 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp2-wan2 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp3-wan3 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp4-wan4 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp5-wan5 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp6-wan6 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp7-wan7 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp8-wan8 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp9-wan9 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp10-wan10 script=lease-script \
    use-peer-dns=no

/ip firewall filter
add action=accept chain=input comment=\
    "Accept established,related connections" connection-state=\
    established,related disabled=yes
add action=accept chain=input comment=\
    "Accept all connections from allowed_to_router address list" disabled=yes \
    src-address-list=allowed_to_router
add action=accept chain=input comment="Accept ICMP connections" disabled=yes \
    protocol=icmp
add action=accept chain=input comment="Accept SNMP connections" disabled=yes \
    dst-port=161 protocol=udp
add action=accept chain=input comment=\
    "Accept L2TP/IPSEC connections (500,1701,4500/udp)" disabled=yes \
    dst-port=1701,500,4500 protocol=udp
add action=accept chain=input comment="Accept WinBox connections (8291/tcp)" \
    disabled=yes dst-port=8291 protocol=tcp
add action=accept chain=input comment=\
    "For L2TP/IPSEC Encryption and Authentication" disabled=yes protocol=\
    ipsec-esp
add action=drop chain=input comment="Drop everything else" disabled=yes
/ip firewall mangle
add action=mark-connection chain=prerouting comment=Incoming in-interface=\
    sfp1-wan1 new-connection-mark=wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp2-wan2 new-connection-mark=wan2 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp3-wan3 new-connection-mark=wan3 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp4-wan4 new-connection-mark=wan4 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp5-wan5 new-connection-mark=wan5 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp6-wan6 new-connection-mark=wan6 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp7-wan7 new-connection-mark=wan7 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp8-wan8 new-connection-mark=wan8 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp9-wan9 new-connection-mark=wan9 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp10-wan10 new-connection-mark=wan10 passthrough=yes
add action=mark-connection chain=prerouting comment=PCC connection-state=new \
    disabled=yes dst-address-type=!local in-interface=bridge-lan \
    new-connection-mark=wan1 passthrough=yes per-connection-classifier=\
    src-address:10/0
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan2 \
    passthrough=yes per-connection-classifier=src-address:10/1
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan3 \
    passthrough=yes per-connection-classifier=src-address:10/2
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan4 \
    passthrough=yes per-connection-classifier=src-address:10/3
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan5 \
    passthrough=yes per-connection-classifier=src-address:10/4
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan6 \
    passthrough=yes per-connection-classifier=src-address:10/5
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan7 \
    passthrough=yes per-connection-classifier=src-address:10/6
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan8 \
    passthrough=yes per-connection-classifier=src-address:10/7
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan9 \
    passthrough=yes per-connection-classifier=src-address:10/8
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan10 \
    passthrough=yes per-connection-classifier=src-address:10/9
add action=mark-routing chain=prerouting connection-mark=wan1 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan2 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan3 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan4 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan5 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan5 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan6 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan6 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan7 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan7 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan8 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan8 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan9 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan9 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan10 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan10 passthrough=yes
add action=mark-connection chain=forward comment="For Sonar" \
    dst-address-list="Standard Internet" new-connection-mark=\
    Standard_Traffic_Down out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=Standard_Traffic_Up passthrough=yes src-address-list=\
    "Standard Internet"
add action=mark-connection chain=forward dst-address-list=\
    "High Capacity Internet" new-connection-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-connection chain=forward in-interface=bridge-lan \
    new-connection-mark=High_Capacity_Traffic_Up passthrough=yes \
    src-address-list="High Capacity Internet"
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Down \
    new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan \
    passthrough=yes
add action=mark-packet chain=forward connection-mark=Standard_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=Standard_Traffic_Up passthrough=\
    yes
add action=mark-packet chain=forward connection-mark=\
    High_Capacity_Traffic_Down new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=yes
add action=mark-packet chain=forward connection-mark=High_Capacity_Traffic_Up \
    in-interface=bridge-lan new-packet-mark=High_Capacity_Traffic_Up \
    passthrough=yes
add action=mark-routing chain=output comment=Outgoing connection-mark=wan1 \
    new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=\
    wan2 passthrough=no
add action=mark-routing chain=output connection-mark=wan3 new-routing-mark=\
    wan3 passthrough=no
add action=mark-routing chain=output connection-mark=wan4 new-routing-mark=\
    wan4 passthrough=no
add action=mark-routing chain=output connection-mark=wan5 new-routing-mark=\
    wan5 passthrough=no
add action=mark-routing chain=output connection-mark=wan6 new-routing-mark=\
    wan6 passthrough=no
add action=mark-routing chain=output connection-mark=wan7 new-routing-mark=\
    wan7 passthrough=no
add action=mark-routing chain=output connection-mark=wan8 new-routing-mark=\
    wan8 passthrough=no
add action=mark-routing chain=output connection-mark=wan9 new-routing-mark=\
    wan9 passthrough=no
add action=mark-routing chain=output connection-mark=wan10 new-routing-mark=\
    wan10 passthrough=no
add action=mark-routing chain=output connection-mark=wan10 dst-address-list=\
    l2tp-sonar new-routing-mark=l2tp-sonar passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp1-wan1
add action=masquerade chain=srcnat out-interface=sfp2-wan2
add action=masquerade chain=srcnat out-interface=sfp3-wan3
add action=masquerade chain=srcnat out-interface=sfp4-wan4
add action=masquerade chain=srcnat out-interface=sfp5-wan5
add action=masquerade chain=srcnat out-interface=sfp6-wan6
add action=masquerade chain=srcnat out-interface=sfp7-wan7
add action=masquerade chain=srcnat out-interface=sfp8-wan8
add action=masquerade chain=srcnat out-interface=sfp9-wan9
add action=masquerade chain=srcnat out-interface=sfp10-wan10
add action=dst-nat chain=dstnat comment="Preseem SSH" dst-port=22222 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=22
add action=dst-nat chain=dstnat comment="Preseem Cockpit" dst-port=9090 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=9090
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
/ip route
add disabled=no distance=1 dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp1-wan1 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp2-wan2 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp3-wan3 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp4-wan4 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp5-wan5 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp6-wan6 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp7-wan7 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp8-wan8 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp9-wan9 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp10-wan10 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp1-wan1 \
    pref-src="" routing-table=wan1 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp2-wan2 \
    pref-src="" routing-table=wan2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp3-wan3 \
    pref-src="" routing-table=wan3 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp4-wan4 \
    pref-src="" routing-table=wan4 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp5-wan5 \
    pref-src="" routing-table=wan5 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp6-wan6 \
    pref-src="" routing-table=wan6 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp7-wan7 \
    pref-src="" routing-table=wan7 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp8-wan8 \
    pref-src="" routing-table=wan8 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp9-wan9 \
    pref-src="" routing-table=wan9 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp10-wan10 pref-src="" routing-table=wan10 scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp5-wan5 \
    pref-src="" routing-table=*40C scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp10-wan10 \
    routing-table=l2tp-sonar suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=myCa disabled=no port=8443
set api disabled=yes
set api-ssl certificate=myCa
/ip ssh
set strong-crypto=yes
/snmp
set contact="Richesin Engineering <jaysen@richesinengineering.com>" enabled=\
    yes location="Communications Shelter, Unalakleet, AK [63.89890275724056, -\
    160.7788028412428]" trap-community=atbunal
/system clock
set time-zone-autodetect=no time-zone-name=America/Anchorage
/system identity
set name=UnalakleetRouter
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.pool.ntp.org
add address=1.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/system scheduler
add disabled=yes interval=5m name=QueueTreeUsageCheck on-event=":log info \"Qu\
    eueTreeUsageCheck script started\"\r\
    \nQueueTreeUsageCheck\r\
    \n:log info \"QueueTreeUsageCheck script finished\"" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-03 start-time=11:04:00
add disabled=yes interval=4w2d name=datareset on-event=\
    "/queue tree reset-counters numbers=[find]" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-11-01 start-time=09:33:17
add disabled=yes interval=5m name=runThrottle on-event=\
    "/system script run throttle;" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-23 start-time=00:00:00
/system script
add dont-require-permissions=no name=rogue-dhcp-scan owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    log info \"Found rogue DHCP server at \$[/system clock get date] \$[/syste\
    m clock get time] on interface bridge-trunk]\"\r\
    \n#Email\r\
    \n/tool e-mail send to=\"jaysen@richesinengineering.com\" subject=\"Found \
    rogue DHCP server at \$[/system clock get date] \$[/system clock get time]\
    \_on interface bridge-trunk\""
add dont-require-permissions=no name=throttle owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local thresholdBytes 161061273600; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"Standard_Download_Sp\
    eed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"Standard_Upload_Speed\
    \"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}\r\
    \n\r\
    \n:local thresholdBytes 268435456000; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"High_Capacity_Downlo\
    ad_Speed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"High_Capacity_Upload_S\
    peed\"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}"
add dont-require-permissions=no name=lease-script owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="i\
    f (\$bound=1) do={\
    \n \r\
    \n/ip route {\
    \n \r\
    \n\r\
    \n:if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interfac\
    e\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n \r\
    \nadd gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n \r\
    \n}\
    \n \r\
    \n\r\
    \n:foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"inter\
    face\") !dynamic !blackhole] do={\
    \n \r\
    \nset \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n \r\
    \n}\
    \n \r\
    \n}\r\
    \n\
    \n}\
    \n"
/tool bandwidth-server
set enabled=no
/tool e-mail
set from="Unalakleet Router" port=2525 server=mail.smtp2go.com tls=yes user=\
    alerts@richesinengineering.com
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
/tool traffic-monitor
add interface=sfp3-wan3 name=tmon1

sindy · Sun Nov 19, 2023 3:28 pm

I am sure there is some changes I could make to improve things so please do let me know what you find that needs to be changed. I kept the PCC mangles for the bridge-lan disabled for now until those can be fixed but so far I don't see any problems at this time.

What I can see is that you have decided to use a common routing table for both the outbound L2TP connection and the inbound Sonar connection. A good idea as such, a bit degraded by the fact that the way how I proposed linking the Sonar connection to a WAN was a bandaid one intended to quickly restore that connection until the final configuration would have been done. A minor collateral damage of this change is that you have removed one of the original routing tables from the list before removing a route that belonged to that table; in such cases, RouterOS doesn't automatically wipe the dependent objects from the configuration, so now you have an orphaned route with routing-table=*40C. No big deal, just remove it manually.

You also seem to have moved to the route backup concept used in the Mikrotik video - if the WAN chosen by mangle rules is unavailable, the backup route becomes whichever one is currently active (based on distance and availability) in table main. This concept is based on the fact that if no route to a destination is available in the requested table, the routing automatically uses table main as a backup for that destination unless you explicitly prohibit that using a routing rule.

For inbound connections to a WAN address of the router coming from the internet (like the Sonar one), it doesn't matter how exactly a backup route is chosen for a given routing mark, because if that WAN goes down or its IP address changes/is lost, no backup route can keep such an inbound connection alive and a new connection has to be established via another WAN (Wireguard is one of few exceptions here, which is extremely useful in some cases and extremely annoying in others). But it may affect the L2TP client - when pure ECMP was used, the client was unable to connect, so explicitly indicating a backup route within table l2tp-sonar (with a higher distance than the one via WAN 10) might be a more reliable approach. It is still not fully clear to me what exactly the problem was in that ECMP-only case, but I'm waiting for the more important issues to get resolved before asking you to investigate this using a test L2TP server.

The purpose of the mangle/prerouting rules that assign connection marks based on in-interface is to let any inbound connection from the internet (like the Sonar one) be responded via the same WAN through which the initial request has arrived (because leaving other aspects aside, the TCP client would be unable to change the address of the responder within an existing session). Once these rules start working, the individual treatment of the Sonar connection based on an address list will not be required any more. If you look at it from another perspective, if you don't expect any other inbound connections from the internet than the Sonar one to ever be used, you don't need to assign connection marks based on in-interface at all, which in turn means you don't need to use the connection marks as an intermediate step between the PCC and the routing marks - always the same per-connection-classified condition matches on all packets of a given direction of any given connection. Which, in summary, would mean that you wouldn't need connection marks for routing at all.

In any case, before starting to use the connection marks for routing by enabling the mark-connection rules that are currently disabled, the mandatory first step is to get rid of the Sonar mark-connection rules the way I've suggested in this post. If you opt for not using connection marks for routing, it is still a good idea to simplify the Sonar rules just to lower the CPU load per packet. Just to be clear - matching a packet against the full list of tracked connections, which is an essential step of connection tracking, is way more CPU intensive than processing the packet by any single rule, but every drop counts in the grand total.

Also, I still deem both the ideas of @sirbryan outlined in this post worth considering. The first one did not come to my mind at all as I am a little biased against Wireguard, but in this particular case replacing L2TP by Wireguard would elegantly solve the issue of the potentially changing IP address of the peer (pity that Mikrotik still has not implemented MOBIKE, which does the same thing for IPsec); the second one did but given your lukewarm reaction to my suggestion to use an external server on a fixed IP to provide a stable IP for the Sonar to connect to, I did not dare to suggest spawning a VPS with double the aggregate bandwidth of the 10 Starlinks just to provide the comfort of client sessions not breaking if one Starlink terminal dies. On the other hand, the fact that you need to run housekeeping scripts to remove NATed UDP connections if their reply-dst-address disappears from the router in order that WiFi calling/VoWiFi (Apple/Android name), SIP phone connections to PBXes in the internet, and other perpetually updated UDP connections could ever recover is annoying. What holds me back is the price of bandwith in the data centers as I don't know what the traffic volumes will be once people start using the full potential of the service. Except that I would use L2TP tunnels rather than Wireguard ones for this purpose, for several reasons I will name if you eventually start thinking in that direction.

jaysen · Sun Nov 19, 2023 6:46 pm

I am sure there is some changes I could make to improve things so please do let me know what you find that needs to be changed. I kept the PCC mangles for the bridge-lan disabled for now until those can be fixed but so far I don't see any problems at this time.

What I can see is that you have decided to use a common routing table for both the outbound L2TP connection and the inbound Sonar connection. A good idea as such, a bit degraded by the fact that the way how I proposed linking the Sonar connection to a WAN was a bandaid one intended to quickly restore that connection until the final configuration would have been done. A minor collateral damage of this change is that you have removed one of the original routing tables from the list before removing a route that belonged to that table; in such cases, RouterOS doesn't automatically wipe the dependent objects from the configuration, so now you have an orphaned route with routing-table=*40C. No big deal, just remove it manually.

The orphaned route and table have been removed. I saw that right after I posted the config and took care of it. To ensure I understand correctly. Once final configuration is done, I shouldn't need to link Sonar and the VPN to a WAN?

Unfortunately there will always be a manual adjustment on the Sonar side due to the IP address being hard coded and currently no support for FQDN and DDNS. I plan to submit feedback to Sonar to hopefully add this feature in future releases. The way I've been advised by Sonar is to create multiple, identical inline device and DHCP server entries for each WAN IP and leave all but the current active one disabled. Should the IP address change then I go into Sonar and toggle them on and off as needed. Personally I hate this approach because it requires me to keep an eye on it. The plan we have been discussing is to use a VPN tunnel back to our office to assign a static IP as we have our own IP allocations from ARIN. Then we point Sonar to that which should solve that issue. I think I also failed to mention earlier in this conversation that our VPN address is in fact on a static IP. It was assumed that it was dynamic hence the reason we used an address list and FQDN and that's my fault for not stating that. It's working the way it is setup now but will change it if necessary.

You also seem to have moved to the route backup concept used in the Mikrotik video - if the WAN chosen by mangle rules is unavailable, the backup route becomes whichever one is currently active (based on distance and availability) in table main. This concept is based on the fact that if no route to a destination is available in the requested table, the routing automatically uses table main as a backup for that destination unless you explicitly prohibit that using a routing rule.

For inbound connections to a WAN address of the router coming from the internet (like the Sonar one), it doesn't matter how exactly a backup route is chosen for a given routing mark, because if that WAN goes down or its IP address changes/is lost, no backup route can keep such an inbound connection alive and a new connection has to be established via another WAN (Wireguard is one of few exceptions here, which is extremely useful in some cases and extremely annoying in others). But it may affect the L2TP client - when pure ECMP was used, the client was unable to connect, so explicitly indicating a backup route within table l2tp-sonar (with a higher distance than the one via WAN 10) might be a more reliable approach. It is still not fully clear to me what exactly the problem was in that ECMP-only case, but I'm waiting for the more important issues to get resolved before asking you to investigate this using a test L2TP server.

The backup route has been put back in place. That was simply an oversight but has been fixed.

The purpose of the mangle/prerouting rules that assign connection marks based on in-interface is to let any inbound connection from the internet (like the Sonar one) be responded via the same WAN through which the initial request has arrived (because leaving other aspects aside, the TCP client would be unable to change the address of the responder within an existing session). Once these rules start working, the individual treatment of the Sonar connection based on an address list will not be required any more. If you look at it from another perspective, if you don't expect any other inbound connections from the internet than the Sonar one to ever be used, you don't need to assign connection marks based on in-interface at all, which in turn means you don't need to use the connection marks as an intermediate step between the PCC and the routing marks - always the same per-connection-classified condition matches on all packets of a given direction of any given connection. Which, in summary, would mean that you wouldn't need connection marks for routing at all.

That's great!!! I don't expect any other inbound connections. We don't plan on hosting any servers so if I am understanding correctly. I can remove those and are we talking about all of the connection marks that I've labeled "incoming" and "PCC"? What about connections to our NMS monitor that use ICMP and SNMP? It's currently on a cloud-hosted VPS but my plan is to move that into our office so it will be on our VPN. That will also allow us to monitor the customer devices which our current NMS setup can't do as those devices are all on private IP ranges. I have made attempts to connect the cloud VPS to our VPN but haven't had much success with that Also, the Preseem engineer who supports the QoE appliance uses an ssh connection and it also has a web GUI called cockpit which I have port forwards for but that can also be reached over VPN. There is a cnMaestro server running on-site that controls all the Cambium equipment which is accessible only by VPN but I am also working to migrate that to the cloud hosted version like our other clients in Oregon.

In any case, before starting to use the connection marks for routing by enabling the mark-connection rules that are currently disabled, the mandatory first step is to get rid of the Sonar mark-connection rules the way I've suggested in this post. If you opt for not using connection marks for routing, it is still a good idea to simplify the Sonar rules just to lower the CPU load per packet. Just to be clear - matching a packet against the full list of tracked connections, which is an essential step of connection tracking, is way more CPU intensive than processing the packet by any single rule, but every drop counts in the grand total.

My plan today is to modify the Sonar rules to your suggestions so I can get those tested and working. Getting ready to do that now. I would prefer to not use connection marks if it simplifies things and makes it easier on the router.

Also, I still deem both the ideas of @sirbryan outlined in this post worth considering. The first one did not come to my mind at all as I am a little biased against Wireguard, but in this particular case replacing L2TP by Wireguard would elegantly solve the issue of the potentially changing IP address of the peer (pity that Mikrotik still has not implemented MOBIKE, which does the same thing for IPsec); the second one did but given your lukewarm reaction to my suggestion to use an external server on a fixed IP to provide a stable IP for the Sonar to connect to, I did not dare to suggest spawning a VPS with double the aggregate bandwidth of the 10 Starlinks just to provide the comfort of client sessions not breaking if one Starlink terminal dies. On the other hand, the fact that you need to run housekeeping scripts to remove NATed UDP connections if their reply-dst-address disappears from the router in order that WiFi calling/VoWiFi (Apple/Android name), SIP phone connections to PBXes in the internet, and other perpetually updated UDP connections could ever recover is annoying. What holds me back is the price of bandwith in the data centers as I don't know what the traffic volumes will be once people start using the full potential of the service. Except that I would use L2TP tunnels rather than Wireguard ones for this purpose, for several reasons I will name if you eventually start thinking in that direction.

My apologies if my reaction was lukewarm. I was most likely distracted by other issues. This week was a crazy one and I was being pulled in multiple directions by just about every client. Phone just kept ringing. I am open to either VPN option whichever will best solve that issue. I have thought about setting up the Wireguard seeing that it's supported. I never noticed it in Winbox until I upgraded to the latest RouterOS. Has that always been supported? We have always used L2TP but certainly open to change. However, the VPS option might not get a great reaction from our Owner/CTO solely based on cost. He's a little conservative when it comes to money and I admit I can be as well. He is family but my influence only goes so far.

On another note, I noticed an issue this morning when our NMS was showing a couple of the starlink terminals as down. Looking through the router logs it appears the terminals lost their lease temporarily and received a CGNAT IP. I am not sure why they obtain a CGNAT IP as we have public IP policy enabled on all terminals. I've seen this happen before when starlink is doing software updates and reboots the terminals. When they came back up. They did keep their same public IP and the lease-script did trigger. However, I noticed when it updated the gateway, it only put in the interface name (ie: sfp1-wan1) and there were several of these in the routing table. It looks like it also did create the correct gateway (ie: x.x.x.x%sfp1-wan1) but the routes with just the interface name were not removed. It was easy to fix by manually correcting the static routes and removing the incorrect ones but I was wondering if there was something in the script that needs adjusting to fix that problem? I've already had to manually fix the routes twice this morning. Log file screenshot is attached for context.

Thank You!

jaysen · Sun Nov 19, 2023 7:22 pm

Hello Sindy,

The connection marks for the Sonar mangles have now been removed. I have also modified and rearranged the packet marks per your suggestions. PCC mangles are still disabled but if all looks good and I can enable those then I am ready to test them out. Config and screenshot attached.

# 2023-11-19 08:10:59 by RouterOS 7.12
# software id = DVNU-PTAC
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HE808NY9TF8
/interface bridge
add admin-mac=48:A9:8A:85:65:FB auto-mac=no comment="Access Bridge" name=\
    bridge-lan
/interface ethernet
set [ find default-name=ether1 ] comment="Not Used" disabled=yes
set [ find default-name=qsfp28-1-1 ] disabled=yes
set [ find default-name=qsfp28-1-2 ] disabled=yes
set [ find default-name=qsfp28-1-3 ] disabled=yes
set [ find default-name=qsfp28-1-4 ] disabled=yes
set [ find default-name=qsfp28-2-1 ] disabled=yes
set [ find default-name=qsfp28-2-2 ] disabled=yes
set [ find default-name=qsfp28-2-3 ] disabled=yes
set [ find default-name=qsfp28-2-4 ] disabled=yes
set [ find default-name=sfp28-1 ] comment="Link to Starlink Terminal 1" name=\
    sfp1-wan1
set [ find default-name=sfp28-2 ] comment="Link to Starlink Terminal 2" name=\
    sfp2-wan2
set [ find default-name=sfp28-3 ] comment="Link to Starlink Terminal 3" name=\
    sfp3-wan3
set [ find default-name=sfp28-4 ] comment="Link to Starlink Terminal 4" name=\
    sfp4-wan4
set [ find default-name=sfp28-5 ] comment="Link to Starlink Terminal 5" name=\
    sfp5-wan5
set [ find default-name=sfp28-6 ] comment="Link to Starlink Terminal 6" name=\
    sfp6-wan6
set [ find default-name=sfp28-7 ] comment="Link to Starlink Terminal 7" name=\
    sfp7-wan7
set [ find default-name=sfp28-8 ] comment="Link to Starlink Terminal 8" name=\
    sfp8-wan8
set [ find default-name=sfp28-9 ] comment="Link to Starlink Terminal 9" name=\
    sfp9-wan9
set [ find default-name=sfp28-10 ] comment="Link to Starlink Terminal 10" \
    name=sfp10-wan10
set [ find default-name=sfp28-11 ] comment="Not Used" disabled=yes name=sfp11
set [ find default-name=sfp28-12 ] comment=\
    "Attached to bridge-lan.  Link to NV1560 Switch - Port 1" name=sfp12-lan
/interface l2tp-client
add comment="Link to Richesin Engineering VPN" connect-to=vpn.richesineng.com \
    disabled=no name=RE use-ipsec=yes user=unalakleet
/interface vlan
add comment="Customer VLAN" interface=sfp12-lan name=vlan2 vlan-id=2
/interface list
add comment="contains WAN interfaces" name=WAN
add comment="contains LAN interfaces" name=LAN
add comment="Contains VPN interfaces" name=VPN
add name=WAN1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add authoritative=after-2sec-delay comment="Controlled by Sonar" interface=\
    bridge-lan lease-time=5m name=vlan2-dhcp
/port
set 0 name=serial0
/queue type
add kind=pcq name=MaxDownloadSpeed pcq-classifier=src-address pcq-rate=10M
add kind=pcq name=MaxUploadSpeed pcq-classifier=dst-address pcq-rate=1M
add kind=pcq name=Standard_Download_Speed pcq-burst-rate=30M pcq-burst-time=\
    15s pcq-classifier=dst-address pcq-dst-address6-mask=64 pcq-rate=25M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=Standard_Upload_Speed pcq-burst-rate=3600k pcq-burst-time=\
    15s pcq-classifier=src-address pcq-dst-address6-mask=64 pcq-rate=3M \
    pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Download_Speed pcq-burst-rate=30M \
    pcq-burst-time=15s pcq-classifier=dst-address pcq-dst-address6-mask=64 \
    pcq-rate=25M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
add kind=pcq name=High_Capacity_Upload_Speed pcq-burst-rate=3600k \
    pcq-burst-time=15s pcq-classifier=src-address pcq-dst-address6-mask=64 \
    pcq-rate=3M pcq-src-address6-mask=64 pcq-total-limit=250000KiB
/queue tree
add max-limit=300M name=Standard_Download_Speed packet-mark=\
    Standard_Traffic_Down parent=bridge-lan priority=7 queue=\
    Standard_Download_Speed
add max-limit=50M name=Standard_Upload_Speed packet-mark=Standard_Traffic_Up \
    parent=bridge-lan priority=7 queue=Standard_Upload_Speed
add max-limit=300M name=High_Capacity_Download_Speed packet-mark=\
    High_Capacity_Traffic_Down parent=bridge-lan priority=7 queue=\
    High_Capacity_Download_Speed
add max-limit=50M name=High_Capacity_Upload_Speed packet-mark=\
    High_Capacity_Traffic_Up parent=bridge-lan priority=7 queue=\
    High_Capacity_Upload_Speed
/routing table
add disabled=no fib name=wan1
add disabled=no fib name=wan2
add disabled=no fib name=wan3
add disabled=no fib name=wan4
add disabled=no fib name=wan5
add disabled=no fib name=wan6
add disabled=no fib name=wan7
add disabled=no fib name=wan8
add disabled=no fib name=wan9
add disabled=no fib name=wan10
add disabled=no fib name=l2tp-sonar
/snmp community
set [ find default=yes ] disabled=yes
add addresses=::/0 name=atbunal
/interface bridge port
add bridge=bridge-lan interface=vlan2
/ip neighbor discovery-settings
set discover-interface-list=none lldp-med-net-policy-vlan=1
/ipv6 settings
set disable-ipv6=yes
/interface list member
add interface=bridge-lan list=LAN
add interface=sfp1-wan1 list=WAN
add interface=sfp2-wan2 list=WAN
add comment="RE VPN" interface=RE list=VPN
add interface=sfp3-wan3 list=WAN
add interface=sfp4-wan4 list=WAN
add interface=sfp5-wan5 list=WAN
add interface=sfp6-wan6 list=WAN
add interface=sfp7-wan7 list=WAN
add interface=sfp8-wan8 list=WAN
add interface=sfp9-wan9 list=WAN
add interface=sfp10-wan10 list=WAN
/ip address
add address=10.130.1.1/24 comment="Servers - vlan1" interface=sfp12-lan \
    network=10.130.1.0
add address=10.130.2.1/23 comment="Subscribers - CPE" interface=vlan2 \
    network=10.130.2.0
add address=10.130.4.1/23 comment="Subscribers - SM" interface=vlan2 network=\
    10.130.4.0
/ip dhcp-client
add add-default-route=no interface=sfp1-wan1 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp2-wan2 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp3-wan3 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp4-wan4 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp5-wan5 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp6-wan6 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp7-wan7 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp8-wan8 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp9-wan9 script=lease-script \
    use-peer-dns=no
add add-default-route=no interface=sfp10-wan10 script=lease-script \
    use-peer-dns=no

/ip firewall filter
add action=accept chain=input comment=\
    "Accept established,related connections" connection-state=\
    established,related disabled=yes
add action=accept chain=input comment=\
    "Accept all connections from allowed_to_router address list" disabled=yes \
    src-address-list=allowed_to_router
add action=accept chain=input comment="Accept ICMP connections" disabled=yes \
    protocol=icmp
add action=accept chain=input comment="Accept SNMP connections" disabled=yes \
    dst-port=161 protocol=udp
add action=accept chain=input comment=\
    "Accept L2TP/IPSEC connections (500,1701,4500/udp)" disabled=yes \
    dst-port=1701,500,4500 protocol=udp
add action=accept chain=input comment="Accept WinBox connections (8291/tcp)" \
    disabled=yes dst-port=8291 protocol=tcp
add action=accept chain=input comment=\
    "For L2TP/IPSEC Encryption and Authentication" disabled=yes protocol=\
    ipsec-esp
add action=drop chain=input comment="Drop everything else" disabled=yes
/ip firewall mangle
add action=mark-connection chain=prerouting comment=Incoming in-interface=\
    sfp1-wan1 new-connection-mark=wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp2-wan2 new-connection-mark=wan2 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp3-wan3 new-connection-mark=wan3 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp4-wan4 new-connection-mark=wan4 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp5-wan5 new-connection-mark=wan5 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp6-wan6 new-connection-mark=wan6 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp7-wan7 new-connection-mark=wan7 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp8-wan8 new-connection-mark=wan8 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp9-wan9 new-connection-mark=wan9 passthrough=yes
add action=mark-connection chain=prerouting connection-state="" in-interface=\
    sfp10-wan10 new-connection-mark=wan10 passthrough=yes
add action=mark-connection chain=prerouting comment=PCC connection-state=new \
    disabled=yes dst-address-type=!local in-interface=bridge-lan \
    new-connection-mark=wan1 passthrough=yes per-connection-classifier=\
    src-address:10/0
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan2 \
    passthrough=yes per-connection-classifier=src-address:10/1
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan3 \
    passthrough=yes per-connection-classifier=src-address:10/2
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan4 \
    passthrough=yes per-connection-classifier=src-address:10/3
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan5 \
    passthrough=yes per-connection-classifier=src-address:10/4
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan6 \
    passthrough=yes per-connection-classifier=src-address:10/5
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan7 \
    passthrough=yes per-connection-classifier=src-address:10/6
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan8 \
    passthrough=yes per-connection-classifier=src-address:10/7
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan9 \
    passthrough=yes per-connection-classifier=src-address:10/8
add action=mark-connection chain=prerouting connection-state=new disabled=yes \
    dst-address-type=!local in-interface=bridge-lan new-connection-mark=wan10 \
    passthrough=yes per-connection-classifier=src-address:10/9
add action=mark-routing chain=prerouting connection-mark=wan1 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan2 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan3 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan4 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan5 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan5 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan6 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan6 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan7 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan7 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan8 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan8 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan9 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan9 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan10 disabled=yes \
    in-interface=bridge-lan new-routing-mark=wan10 passthrough=yes
add action=mark-packet chain=forward comment="For Sonar" dst-address-list=\
    "High Capacity Internet" new-packet-mark=High_Capacity_Traffic_Down \
    out-interface=bridge-lan passthrough=no
add action=mark-packet chain=forward in-interface=bridge-lan new-packet-mark=\
    High_Capacity_Traffic_Up passthrough=no src-address-list=\
    "High Capacity Internet"
add action=mark-packet chain=forward dst-address-list="Standard Internet" \
    new-packet-mark=Standard_Traffic_Down out-interface=bridge-lan \
    passthrough=no
add action=mark-packet chain=forward in-interface=bridge-lan new-packet-mark=\
    Standard_Traffic_Up passthrough=no src-address-list="Standard Internet"
add action=mark-routing chain=output comment=Outgoing connection-mark=wan1 \
    new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=\
    wan2 passthrough=no
add action=mark-routing chain=output connection-mark=wan3 new-routing-mark=\
    wan3 passthrough=no
add action=mark-routing chain=output connection-mark=wan4 new-routing-mark=\
    wan4 passthrough=no
add action=mark-routing chain=output connection-mark=wan5 new-routing-mark=\
    wan5 passthrough=no
add action=mark-routing chain=output connection-mark=wan6 new-routing-mark=\
    wan6 passthrough=no
add action=mark-routing chain=output connection-mark=wan7 new-routing-mark=\
    wan7 passthrough=no
add action=mark-routing chain=output connection-mark=wan8 new-routing-mark=\
    wan8 passthrough=no
add action=mark-routing chain=output connection-mark=wan9 new-routing-mark=\
    wan9 passthrough=no
add action=mark-routing chain=output connection-mark=wan10 new-routing-mark=\
    wan10 passthrough=no
add action=mark-routing chain=output connection-mark=wan10 dst-address-list=\
    l2tp-sonar new-routing-mark=l2tp-sonar passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp1-wan1
add action=masquerade chain=srcnat out-interface=sfp2-wan2
add action=masquerade chain=srcnat out-interface=sfp3-wan3
add action=masquerade chain=srcnat out-interface=sfp4-wan4
add action=masquerade chain=srcnat out-interface=sfp5-wan5
add action=masquerade chain=srcnat out-interface=sfp6-wan6
add action=masquerade chain=srcnat out-interface=sfp7-wan7
add action=masquerade chain=srcnat out-interface=sfp8-wan8
add action=masquerade chain=srcnat out-interface=sfp9-wan9
add action=masquerade chain=srcnat out-interface=sfp10-wan10
add action=dst-nat chain=dstnat comment="Preseem SSH" dst-port=22222 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=22
add action=dst-nat chain=dstnat comment="Preseem Cockpit" dst-port=9090 \
    in-interface-list=WAN protocol=tcp to-addresses=10.130.1.20 to-ports=9090
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
/ip route
add disabled=no distance=1 dst-address=192.168.132.0/24 gateway=192.168.133.1 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=192.168.133.0/24 gateway=192.168.133.1 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp1-wan1 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp2-wan2 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp3-wan3 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp4-wan4 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp5-wan5 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.64.1%sfp6-wan6 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp7-wan7 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp8-wan8 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp9-wan9 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp10-wan10 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp1-wan1 \
    pref-src="" routing-table=wan1 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp2-wan2 \
    pref-src="" routing-table=wan2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp3-wan3 \
    pref-src="" routing-table=wan3 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp4-wan4 \
    pref-src="" routing-table=wan4 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp5-wan5 \
    pref-src="" routing-table=wan5 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.64.1%sfp6-wan6 \
    pref-src="" routing-table=wan6 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp7-wan7 \
    pref-src="" routing-table=wan7 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp8-wan8 \
    pref-src="" routing-table=wan8 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp9-wan9 \
    pref-src="" routing-table=wan9 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    98.97.96.1%sfp10-wan10 pref-src="" routing-table=wan10 scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp10-wan10 \
    routing-table=l2tp-sonar suppress-hw-offload=no
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=98.97.96.1%sfp9-wan9 \
    routing-table=l2tp-sonar suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set www-ssl certificate=myCa disabled=no port=8443
set api disabled=yes
set api-ssl certificate=myCa
/ip ssh
set strong-crypto=yes
/snmp
set contact="Richesin Engineering <jaysen@richesinengineering.com>" enabled=\
    yes location="Communications Shelter, Unalakleet, AK [63.89890275724056, -\
    160.7788028412428]" trap-community=atbunal
/system clock
set time-zone-autodetect=no time-zone-name=America/Anchorage
/system identity
set name=UnalakleetRouter
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.pool.ntp.org
add address=1.pool.ntp.org
/system routerboard settings
set enter-setup-on=delete-key
/system scheduler
add disabled=yes interval=5m name=QueueTreeUsageCheck on-event=":log info \"Qu\
    eueTreeUsageCheck script started\"\r\
    \nQueueTreeUsageCheck\r\
    \n:log info \"QueueTreeUsageCheck script finished\"" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-03 start-time=11:04:00
add disabled=yes interval=4w2d name=datareset on-event=\
    "/queue tree reset-counters numbers=[find]" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-11-01 start-time=09:33:17
add disabled=yes interval=5m name=runThrottle on-event=\
    "/system script run throttle;" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-10-23 start-time=00:00:00
/system script
add dont-require-permissions=no name=rogue-dhcp-scan owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    log info \"Found rogue DHCP server at \$[/system clock get date] \$[/syste\
    m clock get time] on interface bridge-trunk]\"\r\
    \n#Email\r\
    \n/tool e-mail send to=\"jaysen@richesinengineering.com\" subject=\"Found \
    rogue DHCP server at \$[/system clock get date] \$[/system clock get time]\
    \_on interface bridge-trunk\""
add dont-require-permissions=no name=throttle owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local thresholdBytes 161061273600; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"Standard_Download_Sp\
    eed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"Standard_Upload_Speed\
    \"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}\r\
    \n\r\
    \n:local thresholdBytes 268435456000; \r\
    \n\r\
    \n:local downloadQueue [/queue tree find where name=\"High_Capacity_Downlo\
    ad_Speed\"]\r\
    \n:local downloadBytes [/queue tree get \$downloadQueue bytes]\r\
    \n:if (\$downloadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$downloadQueue max-limit=12M\r\
    \n}\r\
    \n\r\
    \n:local uploadQueue [/queue tree find where name=\"High_Capacity_Upload_S\
    peed\"]\r\
    \n:local uploadBytes [/queue tree get \$uploadQueue bytes]\r\
    \n:if (\$uploadBytes > \$thresholdBytes) do={\r\
    \n    /queue tree set \$uploadQueue max-limit=1M\r\
    \n}"
add dont-require-permissions=no name=lease-script owner=sysadm policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="i\
    f (\$bound=1) do={\
    \n \r\
    \n/ip route {\
    \n \r\
    \n\r\
    \n:if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interfac\
    e\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n \r\
    \nadd gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n \r\
    \n}\
    \n \r\
    \n\r\
    \n:foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"inter\
    face\") !dynamic !blackhole] do={\
    \n \r\
    \nset \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n \r\
    \n}\
    \n \r\
    \n}\r\
    \n\
    \n}\
    \n"
/tool bandwidth-server
set enabled=no
/tool e-mail
set from="Unalakleet Router" port=2525 server=mail.smtp2go.com tls=yes user=\
    alerts@richesinengineering.com
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
/tool traffic-monitor
add interface=sfp3-wan3 name=tmon1

Thank You!

sindy · Sun Nov 19, 2023 8:00 pm

To ensure I understand correctly. Once final configuration is done, I shouldn't need to link Sonar and the VPN to a WAN?

Yes for the Sonar if you keep connection marks in use (which is what I was expecting), maybe for the VPN. I cannot say why the L2TP was struggling initially and then later after you enabled some mangle rules until we do some tests with an L2TP server dedicated for the testing. The difference between the two is that Sonar is an "inbound call" (Sonar is the initiator, the 2216 is a responder) whereas the L2TP transport session is an "outbounnd call" (the 2216 is an initiator, your company VPN server is a responder). So I have my theories why the outgoing call was struggling before but I want to see them confirmed or denied before giving out my speculations to the internet to remember them forever.

The plan we have been discussing is to use a VPN tunnel back to our office to assign a static IP as we have our own IP allocations from ARIN. Then we point Sonar to that which should solve that issue.

This is what I was suggesting and saw the "lukewarm reaction"

So apparently it was just a misunderstanding.

I think I also failed to mention earlier in this conversation that our VPN address is in fact on a static IP. ... It's working the way it is setup now but will change it if necessary.

It's not necessary, as changing that to a numeric IP address would not simplify anything due to other factors. Plus this way you can change the VPN server address if needed without losing the L2TP VPN, even if you may not plan on doing in foreseeable future.

I don't expect any other inbound connections. We don't plan on hosting any servers so if I am understanding correctly. I can remove those and are we talking about all of the connection marks that I've labeled "incoming" and "PCC"?

It's again a question how much foreseeable the future is.If you don't mind identifying Sonar connections by the Sonar IP address as you do now, and no requirement for incoming connections landing directly on the WAN IPs will arise, then yes, you can remove connection marks used for routing. In such case yes, we are talking about all the rules that assign connection marks up to in-interface and up to per-connection-classifier. The former can be removed completely, the latter can be changed to assign routing marks directly instead of connection marks. But given what you write just below, let's postpone that change.

What about connections to our NMS monitor that use ICMP and SNMP? It's currently on a cloud-hosted VPS but my plan is to move that into our office so it will be on our VPN. That will also allow us to monitor the customer devices which our current NMS setup can't do as those devices are all on private IP ranges. I have made attempts to connect the cloud VPS to our VPN but haven't had much success with that Also, the Preseem engineer who supports the QoE appliance uses an ssh connection and it also has a web GUI called cockpit which I have port forwards for but that can also be reached over VPN. There is a cnMaestro server running on-site that controls all the Cambium equipment which is accessible only by VPN but I am also working to migrate that to the cloud hosted version like our other clients in Oregon.

Inbound connections via VPN are still inbound connections that need to be responded via the correct interface, but it may be possible to handle this using normal routing so it may not require connection marking; in any case, as soon as private addresses come into play, the issue of overlapping private ranges may pop up and require a solution. No idea what Preseem and QoE is, but SSH from the support guy is an inbound connection, and a cnMaestro in the cloud is another couple of incoming connections. All of them better served if port-forwarded from a fixed IP via the VPN tunnel. SNMP and ICMP may be actually both - traps would likely be sent from 2216 to your NMS server, GETs would be sent by the NMS server to the 2216 so inbound connections to 2216 again. ICMP - I guess this means that the NMS actively pings the 2216 (as I cannot see any pinging of the NMS from the 2216) so yet another inbound connection.

My apologies if my reaction was lukewarm. I was most likely distracted by other issues.

No need to apologize, I know this so well...

I am open to either VPN option whichever will best solve that issue. I have thought about setting up the Wireguard seeing that it's supported. I never noticed it in Winbox until I upgraded to the latest RouterOS. Has that always been supported?

Wireguard has been added early in RouterOS 7, if not from the very beginning. So on a 2216 that cannot run RouterOS 6 I guess it must have been available since unpacking.

We have always used L2TP but certainly open to change. However, the VPS option might not get a great reaction from our Owner/CTO solely based on cost.

Distinguish between the two applications - the VPN for management purposes would benefit from Wireguard's ability to accommodate to the IP address changes on the Unalakleet end, and would not require a dedicated VPS with a high bandwidth. The tunnel for providing a static public IP for everything behind the 2216 (I hesitate to call it a VPN because I think there is no need for privacy as its payload traffic would go directly to internet anyway if not for the tunnel) would require a dedicated VPS or physical server with enough bandwidth and monthly allowance, hence the additional OPEX.

On another note,... I noticed when it updated the gateway, it only put in the interface name (ie: sfp1-wan1) and there were several of these in the routing table.

It is not a CGNAT address (only 100.64.0.0/10 are Carrier Grade NAT addresses, it's a dedicated range), it is a mere private one, and yes, it looks like the case when the connection to the satellite has not been established yet. I assume the DHCP lease simply contains no default gateway item at all, which I did not anticipate when writing the script. So most likely the $"gateway-address" variable is empty and the rest is a consequence.

I'll have to imitate that case and give you an update for the script.

sindy · Sun Nov 19, 2023 8:12 pm

screenshot attached.

Now hold up. The 10.5 GiB of data shown in most mark-routing rules in chain output mean that your device itself is sending tons of its own traffic (not one forwarded from the customers); as it has no reason to do that actively, and as these rules translate connection marks to routing marks so connection marks must have been assigned and they are only assigned up to in-interface, it must be responses to some incoming requests. The concluion is that as you forgot to enable the firewall rules, your 2216 is now probably taking part in DDoS attacks (such as sending long answers to short DNS queries, except that the DNS proxy function is disabled so it must be something else). Fix that ASAP, please, but it may still be too late and those public addresses may have already made it to some public blacklists.

I'll comment on the rest once this is resolved.

sindy · Sun Nov 19, 2023 9:31 pm

I'll have to imitate that case and give you an update for the script.

Remove the existing lease-script and copy-paste the following to the terminal:

/system script add name=lease-script source=":if (\$bound=1) do={\
    \n  /ip route {\
    \n    :if ([:typeof \$\"gateway-address\"]!=\"nothing\") do={\
    \n      :if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n        add gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n      }\
    \n      :foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole] do={\
    \n        set \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n      }\
    \n    }\
    \n  }\
    \n}\
    \n"

jaysen · Sun Nov 19, 2023 9:43 pm

screenshot attached.
Now hold up. The 10.5 GiB of data shown in most mark-routing rules in chain output mean that your device itself is sending tons of its own traffic (not one forwarded from the customers); as it has no reason to do that actively, and as these rules translate connection marks to routing marks so connection marks must have been assigned and they are only assigned up to in-interface, it must be responses to some incoming requests. The concluion is that as you forgot to enable the firewall rules, your 2216 is now probably taking part in DDoS attacks (such as sending long answers to short DNS queries, except that the DNS proxy function is disabled so it must be something else). Fix that ASAP, please, but it may still be too late and those public addresses may have already made it to some public blacklists.

I'll comment on the rest once this is resolved.

Hello Sindy,

I have now went through the "Building your first firewall" section in the Mikrotik Wiki and activated all those rules which seem to be the defaults. I had started on the Firewall on Friday but then the problems occurred with customers complaining that the internet was extremely slow and pages wouldn't load at all. During my troubleshooting I had temporarily turned it off as I wasn't sure exactly what the issues were at the time.

Do you have any recommendations on other firewall rules that may be required that are not on the Wiki or will the defaults be sufficient enough?

jaysen · Sun Nov 19, 2023 9:44 pm

I'll have to imitate that case and give you an update for the script.

Remove the existing lease-script and copy-paste the following to the terminal:

/system script add name=lease-script source=":if (\$bound=1) do={\
    \n  /ip route {\
    \n    :if ([:typeof \$\"gateway-address\"]!=\"nothing\") do={\
    \n      :if ([:len [find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole routing-table=main]]=0) do={\
    \n        add gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n      }\
    \n      :foreach r in=[find where dst-address=0.0.0.0/0 gateway~(\"%\".\$\"interface\") !dynamic !blackhole] do={\
    \n        set \$r gateway=(\$\"gateway-address\".\"%\".\$\"interface\")\
    \n      }\
    \n    }\
    \n  }\
    \n}\
    \n"

This has been done. Thank You

sindy · Sun Nov 19, 2023 10:05 pm

Do you have any recommendations on other firewall rules that may be required that are not on the Wiki or will the defaults be sufficient enough?

If you haven't changed anything in chain input of /ip/firewall/filter since the last export you've posted, please do the following steps until the first unexpected outcome (or till the end which we hope will be the case).

enable "safe mode"
enable all the action=accept rules in chain input of filter (i.e. all of them except the last action=drop chain=input comment="Drop everything else" one)
without closing the existing one or logging off, run another instance of Winbox and connect to the 2216 from there; if successful, close it again
in the first Winbox instance, check that the rule action=accept chain=input comment="Accept WinBox connections (8291/tcp)" dst-port=8291 protocol=tcp or the action=accept chain=input comment="Accept all connections from allowed_to_router address list" src-address-list=allowed_to_router one did count at least one packet
if it did, enable the action=drop chain=input comment="Drop everything else" rule
once again without closing the existing one or logging off, run another instance of Winbox and connect to the 2216 from there; if successful, close it again
disable "safe mode" in the first Winbox instance

Let me know the outcome before proceeding.

jaysen · Sun Nov 19, 2023 10:17 pm

Do you have any recommendations on other firewall rules that may be required that are not on the Wiki or will the defaults be sufficient enough?
If you haven't changed anything in chain input of /ip/firewall/filter since the last export you've posted, please do the following steps until the first unexpected outcome (or till the end which we hope will be the case).
enable "safe mode"

enable all the action=accept rules in chain input of filter (i.e. all of them except the last action=drop chain=input comment="Drop everything else" one)

without closing the existing one or logging off, run another instance of Winbox and connect to the 2216 from there; if successful, close it again

in the first Winbox instance, check that the rule action=accept chain=input comment="Accept WinBox connections (8291/tcp)" dst-port=8291 protocol=tcp or the action=accept chain=input comment="Accept all connections from allowed_to_router address list" src-address-list=allowed_to_router one did count at least one packet

if it did, enable the action=drop chain=input comment="Drop everything else" rule

once again without closing the existing one or logging off, run another instance of Winbox and connect to the 2216 from there; if successful, close it again

disable "safe mode" in the first Winbox instance
Let me know the outcome before proceeding.

That's all done and nothing unexpected happened. Everything appears to be working.

sindy · Sun Nov 19, 2023 11:41 pm

OK. So now, move the action=mark-routing chain=output connection-mark=wan10 dst-address-list=l2tp-sonar new-routing-mark=l2tp-sonar passthrough=no rule to the top of the chain output. Then you can enable all the action=mark-routing rules in prerouting, but keep the action=mark-connection ones matching on per-connection-classifier still disabled. Then enter /ip/firewall/mangle/reset-counters-all on the command line and watch the mangle rule counters grow on the GUI.

Most of the ones in chain output should now grow only slowly, the one now on the top (for l2tp-sonar) should grow faster than the rest.

The action=mark-routing ones in prerouting should stay at 0 packets, 0 bytes at this stage.

The next step will be to enable the action=mark-connection rules matching on per-connection-classifier. I would recommend to start with only one of them (e.g. for WAN 1); its counters should start growing slowly and those of the corresponding mark-routing rule in prerouting should start growing much faster.

/ip/firewall/connection/print interval=1s where connection-mark=wan1 repl-packets>0 should show some live connections, and no complaints should start coming from Unalakleet.

jaysen · Mon Nov 20, 2023 12:21 am

OK. So now, move the action=mark-routing chain=output connection-mark=wan10 dst-address-list=l2tp-sonar new-routing-mark=l2tp-sonar passthrough=no rule to the top of the chain output. Then you can enable all the action=mark-routing rules in prerouting, but keep the action=mark-connection ones matching on per-connection-classifier still disabled. Then enter /ip/firewall/mangle/reset-counters-all on the command line and watch the mangle rule counters grow on the GUI.

Most of the ones in chain output should now grow only slowly, the one now on the top (for l2tp-sonar) should grow faster than the rest.

The action=mark-routing ones in prerouting should stay at 0 packets, 0 bytes at this stage.

The next step will be to enable the action=mark-connection rules matching on per-connection-classifier. I would recommend to start with only one of them (e.g. for WAN 1); its counters should start growing slowly and those of the corresponding mark-routing rule in prerouting should start growing much faster.

/ip/firewall/connection/print interval=1s where connection-mark=wan1 repl-packets>0 should show some live connections, and no complaints should start coming from Unalakleet.

Okay I have done that. The counters in the output chain do appear to grow slower as you stated. The l2tp-sonar rule actually grows slower than the others. Also, the action=mark-routing ones in prerouting do not stay at 0 and I lost my connection. I had safe mode enabled so I am back in. The l2tp rule is still at the top of output but the mark routing rules in prerouting reverted back to disabled.

Another thing I just noticed is most of my customer subscriber modules are going offline in cnMaestro. I am received many alerts about that. I have not attempted to re-enable the routing marks in prerouting. Only the l2tp-sonar outgoing routing mark is still at the top of that chain. I also just got booted out of Winbox again but it did reconnect.

sindy · Mon Nov 20, 2023 1:05 am

the action=mark-routing ones in prerouting do not stay at 0 and I lost my connection. I had safe mode enabled so I am back in.

So much about a low-profile testing with minimum impact on the traffic

I've missed a small difference between the action=mark-connection rules. In the ones that match on in-interface, the connection-state=new condition is missing. However, while this explains why the action=mark-routing rules did count, it does not explain why you have lost your connection, since the connections have been marked properly - as the per-connection-classifier rules and the mark-routing rules were disabled, the LAN->internet initial packet of each connection went out via some interface chosen by ECMP, so it got src-nated (by masquerade) to the IP of that interface, so the response came in via that interface and the connection got marked accordingly. Hence enforcing that interface by means of the corresponding action=mark-routing rule and the default route in the corresponding routing table should have had no impact on the connection.

How are you connecting the Winbox? Via L2TP, via one of the public IP addresses, or via the PC on the LAN?

I'm afraid I'll have to see it live to understand what is going on.

jaysen · Mon Nov 20, 2023 1:27 am

the action=mark-routing ones in prerouting do not stay at 0 and I lost my connection. I had safe mode enabled so I am back in.
So much about a low-profile testing with minimum impact on the traffic

I've missed a small difference between the action=mark-connection rules. In the ones that match on in-interface, the connection-state=new condition is missing. However, while this explains why the action=mark-routing rules did count, it does not explain why you have lost your connection, since the connections have been marked properly - as the per-connection-classifier rules and the mark-routing rules were disabled, the LAN->internet initial packet of each connection went out via some interface chosen by ECMP, so it got src-nated (by masquerade) to the IP of that interface, so the response came in via that interface and the connection got marked accordingly. Hence enforcing that interface by means of the corresponding action=mark-routing rule and the default route in the corresponding routing table should have had no impact on the connection.

How are you connecting the Winbox? Via L2TP, via one of the public IP addresses, or via the PC on the LAN?

I'm afraid I'll have to see it live to understand what is going on.

I will fix the input connection marks and add the "new" connection state. The reason my customer SMs lost connection as did I was due to firewall being too restrictive. I had gone further in the Mikrotik Wiki https://help.mikrotik.com/docs/display/ ... t+Firewall and added in the rules under Protect the LAN devices. Turns out I probably shouldn't have done that yet. Removing those rules restored customer connectivity and I kept the ones under Protect the router enabled. I really need to learn to slow down and not do more than necessary. Lesson learned. I got a bit paranoid when you mentioned that our router might be participating in DDOS so that was me trying to make sure that wasn't happening. I re-enabled all the mark routing rules under prerouting and this time I remain connected.

As to how I connect to the router. I connect to our VPN and then connect to WinBox using it's private IP. I also have a windows jumpbox at the site running remote-pc as a backup

sindy · Mon Nov 20, 2023 1:43 am

The reason my customer SMs lost connection as did I was due to firewall being too restrictive. I had gone further in the Mikrotik Wiki https://help.mikrotik.com/docs/display/ ... t+Firewall and added in the rules under Protect the LAN devices. Turns out I probably shouldn't have done that yet. Removing those rules restored customer connectivity and I kept the ones under Protect the router enabled.

Haven't you by chance saved the configuration while these rules were in place? Referring to a manual only gives a rough idea but there may have been some minor mistake in what you have actually set.

There are ways how to send traffic to devices on private addresses behind a NAT but they require a very cooperative ISP, so normally, the fact that everyting on LAN is on a private address should prevent the LAN devices from getting involved in a DDoS attack as reflectors/amplifiers of traffic getting to them via WAN.

I re-enabled all the mark routing rules under prerouting and this time I remain connected.

As to how I connect to the router. I connect to our VPN and then connect to WinBox using it's private IP. I also have a windows jumpbox at the site running remote-pc as a backup

If you connect Winbox to the private IP assigned to the L2TP client by your VPN server, I can see no way how forward filter rules could affect that path, as both the L2TP payload and the L2TP transport packets are received and sent by the router itself, so they run through input and output chains, respectively. Mystery.

Going offline for next few hours.

jaysen · Mon Nov 20, 2023 2:05 am

The reason my customer SMs lost connection as did I was due to firewall being too restrictive. I had gone further in the Mikrotik Wiki https://help.mikrotik.com/docs/display/ ... t+Firewall and added in the rules under Protect the LAN devices. Turns out I probably shouldn't have done that yet. Removing those rules restored customer connectivity and I kept the ones under Protect the router enabled.
Haven't you by chance saved the configuration while these rules were in place? Referring to a manual only gives a rough idea but there may have been some minor mistake in what you have actually set.

There are ways how to send traffic to devices on private addresses behind a NAT but they require a very cooperative ISP, so normally, the fact that everyting on LAN is on a private address should prevent the LAN devices from getting involved in a DDoS attack as reflectors/amplifiers of traffic getting to them via WAN.

I re-enabled all the mark routing rules under prerouting and this time I remain connected.

As to how I connect to the router. I connect to our VPN and then connect to WinBox using it's private IP. I also have a windows jumpbox at the site running remote-pc as a backup
If you connect Winbox to the private IP assigned to the L2TP client by your VPN server, I can see no way how forward filter rules could affect that path, as both the L2TP payload and the L2TP transport packets are received and sent by the router itself, so they run through input and output chains, respectively. Mystery.

Going offline for next few hours.

I did not save that config since they were just the ones listed in the Wiki with the interfaces and LAN network changed to match our configuration.

I should have mentioned I use it's LAN IP, not the IP assigned by L2TP. Perhaps that clears up the mystery?

I am going to jump off here myself. Perhaps if you are around later we can proceed with testing the PCC connection marks. I'm not certain where you are in the world or the time difference between you and me (PST). Feel free to reply here or shoot me an email when you have time and I'll be available. I'm very flexible so what ever works for you will work for me. I just appreciate the help in getting this mess (mostly created by me) fixed. I've just recently jumped back into the networking world having received my degree about 10 years ago. I used to work in Emergency Medicine although I do have a background in IT and ISP tech support. Thankfully I didn't treat my patients the same way I do networks. That would be bad.

anav · Mon Nov 20, 2023 3:35 am

Just to have a sample practical, minimalist but secure (shortNsweet) firewall available for your perusal. Its based on allowing only authorized traffic and dropping everything else.
The input chain rule allowed admin access is based on a firewall address list one create and which is comprised of local admin IP addresses (set statically in dhcp leases) and any remote VPN type IP addresses. You will note the forward chain fastrack rule is disabled due to using queues and mangles.

/ip firewall filter
{Input Chain}
(default rules to keep)
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
( admin rules )
add action=accept chain=input src-address-list=Admin comment="Config Access"
add action=accept chain=input comment="Allow LAN DNS queries-UDP" \ {and NTP *** services if required etc}
dst-port=53,123  in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN DNS queries - TCP" \
dst-port=53 in-interface-list=LAN protocol=tcp
add action=drop chain=input comment="drop all else"
{forward chain}
(default rules to keep)
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related disabled=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
(user rules)
add action=accept chain=forward comment="allow internet traffic" in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="allow dst-nat from both WAN and LAN (including port forwarding)" connection-nat-state=dstnat
add action=drop chain=forward comment="drop all else"

aoakeley · Mon Nov 20, 2023 3:55 am

I know this is throwing a bunch of your work out the window, but this might make your life a lot simpler....

Have you thought about buying 10 more small mikrotik routers?
- one for each starlink.
Then put the big router you are using for load balancing and traffic management behind them?

That way each WAN interface on your load balancer could have a different gateway IP, you could more easily assign static routes for traffic you only want to go out one interface, etc etc.

Just a thought...

Screenshot 2023-11-20 094932.png

gotsprings · Mon Nov 20, 2023 4:20 pm

I know this is throwing a bunch of your work out the window, but this might make your life a lot simpler....

I tried to get him to just subscribe to a service that can actually strype connections down multiple links and provides him a public IP.

jaysen · Mon Nov 20, 2023 5:26 pm

I know this is throwing a bunch of your work out the window, but this might make your life a lot simpler....

Have you thought about buying 10 more small mikrotik routers?
- one for each starlink.
Then put the big router you are using for load balancing and traffic management behind them?

That way each WAN interface on your load balancer could have a different gateway IP, you could more easily assign static routes for traffic you only want to go out one interface, etc etc.

Just a thought...

Screenshot 2023-11-20 094932.png

I don't know about simpler by adding 10 more routers to the network.

However, that does look like it would be fun to setup in a lab. I might do that sometime just not with so many WANs. Unfortunately changing everything up at this stage isn't an option.

This is for a small tribal village on the Alaskan coastline and the current design and equipment was paid for with federal grant funds to build out broadband in underserved populations. Additionally the tower site doesn't have enough space or outlets to support the extra devices plus every change and purchase must be approved by tribal council which only meets once per month. This project has already taken longer than necessary due to issues with the previous vendors. We were brought in to take over management and fix all the mistakes. I've been working on this for almost a year and customer billing was to start back in May but keeps getting pushed further out but I think we are just about ready to go live with billing hopefully by Dec 1st. I really hope this project survives but residents have already grown impatient that many have already given up an went with their own Starlinks. Every trip to the village I see more and more Starlinks on homes. Only time will tell I suppose.

gotsprings · Tue Nov 21, 2023 4:42 pm

10 Starlink accounts with Public IP $140 each = $1,400
VPS service with one Public IP and 1 Gig of bandwidth. = $900

That gives you a monthly of $2,300.

You typed before that the Monthly used to be $36,000 for 100Mb.

I am still confused why my idea was rejected out of hand. I must be missing something.

Amm0 · Tue Nov 21, 2023 5:06 pm

You typed before that the Monthly used to be $36,000 for 100Mb.

That's was for a geosynchronous (GEO) satellite transponder (~36Mhz), which be in the ballpark. We use GEO for temp internet and satellite time cost $300-500/hour if not a TDMA shared service.

He switch to 10 starlink (LEO). Now business/premium starlink might run you $250/month per starlink — I'm not sure they allow static IP on $125 service, maybe – there always changing stuff

I am still confused why my idea was rejected out of hand. I must be missing something.

To your point, Bigfoot isn't doing anything different here, they just hide the config for you.
Now you do have a point if was $36k/month...that would cover something like a Pepwave at either end, although they are expensive, you at least own them. Although L2TP (or other VPN) with a VPS/datacenter and Mikrotik does same thing, it's obviously just more config to manage yourself on Mikrotik.

gotsprings · Tue Nov 21, 2023 7:16 pm

To your point, BIGLEAF isn't doing anything different here, they just hide the config for you.
Now you do have a point if was $36k/month...that would cover something like a Pepwave at either end, although they are expensive, you at least own them. Although L2TP (or other VPN) with a VPS/datacenter and Mikrotik does same thing, it's obviously just more config to manage yourself on Mikrotik.

Mikrotik does the same thing as Big Leaf?

Amm0 · Tue Nov 21, 2023 7:26 pm

Mikrotik does the same thing as Big Leaf?

In any bonded/striped multi-wan approach...
It's tunnels all the way down .

As on display in this thread... there just isn't a friendly UI to do it on a Mikrotik. But you do have choice of tunnels types on Mikrotik to use for various situation. Never used Big Leaf, but under-the-covers there is still some VPN tunnel (just managed by them, with QoS, etc applied).

While load balancing does not aggregate bandwidth... it still has a benefit that it has lower latency (in most cases) than striping/bonding since there no re-assembly needed.

gotsprings · Wed Nov 22, 2023 1:41 pm

I hadn't really heard any complaints about latency.

I got an email from Big Leaf During the World Series Broadcast. The uplink truck for the crew outside the stadium was connected to our network . Seems Comcast decided to go out right then. The Verizon feed was still running. Comcast came back up.

Fox didn't even notice.

Larsa · Wed Nov 22, 2023 9:27 pm

@Amm0, BigLeaf is just a regular SD-WAN solution with options like public internet access branded “cloud routing”.

anav · Wed Nov 22, 2023 9:38 pm

No word of a lie, but I was out running on friggin mountain in Spain recently when my bowels told me I was in a dire very dire short fused situation.
I went off the beaten path to ensure isolation, just in case, and was just in time. What a relief,,,,,,
However, I could have really used a BIG LEAF, or a few of them at that moment. I had to make do, ( or wipe the doodoo ) with dried pine needles.............

+1 for Big leaf ;-PPPP

Amm0 · Wed Nov 22, 2023 10:18 pm

@Amm0, BigLeaf is just a regular SD-WAN solution with options like public internet access branded “cloud routing”.

Totally. I'm sure it's fine. Only complaint is there website was light on algorithms/protocol they use, performance specs, and pricing... I just find "SD-WAN" is so loosely-defined to be meaningless & BigLeaf lack of technical details beyond that is what worries me. e.g. If Mikrotik marketing was more aggressive, you could call RouterOS tunnels+mangle+scripts as a "software-defined WAN" too

.

But we digress here... Mikrotik certainly be a cheaper solution for the load balancing 10 starlink case ...

Larsa · Wed Nov 22, 2023 11:14 pm

If Mikrotik marketing was more aggressive, you could call RouterOS tunnels+mangle+scripts as a "software-defined WAN" too .

Yup, so it is!

Regarding 'black box' solutions like B.L, Gartner also expressed concern about the lack of technical details.

gotsprings · Mon Dec 11, 2023 3:51 pm

@Amm0, BigLeaf is just a regular SD-WAN solution with options like public internet access branded “cloud routing”.
.... If Mikrotik marketing was more aggressive, you could call RouterOS tunnels+mangle+scripts as a "software-defined WAN" too .

I had been hoping that somehow Zerotier would have give me some sort of load balanced single IP.

This thread has been bothering me for a while.

Has someone managed to set up something like Big Leaf using Mikrotik and some sort of cloud VPS?

I really want to have multiple ISPs running to one PUBLIC IP before presenting to services. Then have the VPS Split up the traffic coming back to me over the multiple feed and reassemble them at my router.

This would provide for ZERO IP CHANGE FAILOVER. Which is the only reason I am looking at Big Leaf.

I built a poor mans version using Speedify and a linux box. But the services kept identifying the VPS Speedify gave me as a VPN. Causing some services to be blocked. So I would have to go back into the Speedify interface and try to write bypass rules.

But... when I was using it for voip and wifi calling and Signal calls... it worked perfectly when I would rip one or the other ISP circuits out of the box.

anav · Mon Dec 11, 2023 4:29 pm

Way ahead of me gotsprings.
I am thinking of using a CHR cloud router and connecting a ground site to it via multiple ISPs
Then using L2TP plain over transparent wireguard to connect the ground site to the CHR ( L2Tp allows mrru adjustment for packet fragmentation).
THEN using OSPF and BFD to monitor link connectivity and then to switch which routes should be used....

Another poor mans attempt at better switching between ISPs.......... ( avoiding the check-gateway=ping, or netwatch script variants )
If you are any good at the above functionalities..............................

Larsa · Mon Dec 11, 2023 10:05 pm

@Gotsprings, I don’t get what you mean by "load-balanced single IP." Zerotier, Tailscale, and most other SD-WAN solutions can utilize multipath and internal load balancing with an "exit node" to a public IP on internet. It’s just a matter of configuration.

gotsprings · Tue Dec 12, 2023 3:44 am

Larsa

I want multiple ISPs to be connected to my site. I want all my traffic to go across multiple ISPs to a single VPS. I want the VPS to be my PUBLIC IP to any IP or service I connect to. I want to do this actually spread my traffic across multiple connections and be fault tolerant to nearly zero... As long as atleast one feed is active.

The paid services allow for redundancy on the VPS and provide info about "what's going on" behind the connections.

Larsa · Tue Dec 12, 2023 9:00 am

As I mentioned earlier, it’s feasible with SD-WAN in general using open-source or paid solutions.

There is no magic with the VPS; it's simply another node employing the SD-WAN protocol that might be used as default gateway to internet for the SD-WAN network.

SD-WAN is by design fault-tolerant and utilizes all available connections by default, although this behavior is configurable.

Traffic monitoring is conducted through the built-in protocol using either the web GUI or a standalone utility. Usually there’s also an API available for configuration, deployment and monitoring.

This is somewhat OT so I suggest we continue this discussion in another thread.

Who is online