Multi Starlink WANs, VOIP and live stream broadcast

daves · May 22, 2024, 5:35am

I have 2 Starlinks and a hAP ac2.
I want to do load balancing with failover while prioritizing video broadcasting and VOIP.

You cannot change a Starlink’s IP address/subnet. So both are 192.168.1.1/24 and hand out addresses with DHCP in that subnet. So I end up with two wan interfaces with addresses in the same subnet (possibly the same IP for both wan interfaces) and the gateway for each is the same. I think I could set static IPs if needed.

Is that a problem for setting up load balancing with failover?
I want to set up 3 vlans. (easy part)
30 Guest 10.30.0.1/16
60 VOIP 192.168.60.1/24
70 Broadcast 192.168.70.1/24
I need to guarantee priority for the broadcast stream first and then VOIP (both with no limit) and the rest to other traffic.
The challenge is I don’t see quite how to fit that traffic shaping with dual wan and load balancing for the remaining traffic.
Is there a way to do QOS for voip and video calls for devices not on the VOIP VLAN. Like Zoom, Slack, MS Teams? Trigger on their tcp/udp ports, tag them, and put them in the VOIP QOS queue?
Starlink download can vary quite a bit so to guarantee downward priority for VOIP would require setting limit to the worst download speed because it has to push back non-priority traffic. Whereas upward priorities it just holds back sending lower-prioritized traffic till higher priority stuff is satisfied. Right?
Could I do VOIP/broadcast QOS on one and other traffic is load balanced on what is left of that Starlink and the full out on the other?

What I’ve gathered from reading other posts is to use the voip and broadcast vlans (IP ranges) to tag connections/packets to prioritize them.
But load balancing seems to also require tagging connections/packets.
And it seems like it would require a separate queue per WAN per direction.

Any guidance would be so appreciated!
Thanks!

Larsa · May 22, 2024, 6:40am

Set up your Starlinks in “Bypass Mode” to obtain distinct WAN addresses, use the Cake queue manager to automatically prioritize traffic like VOIP and use Mikrotik Starlink load balancing.

If you’re thinking about setting up VLANs, a good place to start is by reading the user article “Using RouterOS to VLAN your network” courtesy of @pcunite.

sindy · May 26, 2024, 2:42pm

Not any more with RouterOS 7; with RouterOS 6, you need some unusual tricks if you want to use failover based on recursive next hop search and multiple gateways with the same IP addresses.

Indeed.

The key is to treat not only both uploads (easy) but also both downloads separately. So you’ll have to use two groups of packet marks, one per each WAN, for the download. For best throughput, Mikrotik recommends to use the interface queues as root ones. The queues are part of postrouting, so for upload, the same packet mark may be used because the use of the interface queue as a parent one will take care about the separation:

[me@myTik] > queue tree export
...
/queue tree
add limit-at=100M max-limit=100M name=wan1-root parent=ether1
add limit-at=10 max-limit=10 name=wan1-slow packet-mark=slow parent=wan1-root
add limit-at=100M max-limit=100M name=wan2-root parent=ether2
add limit-at=10 max-limit=10 name=wan2-slow packet-mark=slow parent=wan2-root

[me@myTik] > queue tree print stats
Flags: X - disabled, I - invalid
 0   name="wan1-root" parent=ether1 packet-mark="" rate=0 packet-rate=0 queued-bytes=0 queued-packets=0 bytes=280 packets=4 dropped=0
 1   name="wan1-slow" parent=wan1-root packet-mark=slow rate=0 packet-rate=0 queued-bytes=0 queued-packets=0 bytes=280 packets=4 dropped=0
 2   name="wan2-root" parent=ether2 packet-mark="" rate=416 packet-rate=1 queued-bytes=0 queued-packets=0 bytes=490 packets=7 dropped=0
 3   name="wan2-slow" parent=wsn2-root packet-mark=slow rate=416 packet-rate=1 queued-bytes=0 queued-packets=0 bytes=490 packets=7 dropped=0

Yes, provided that the provider of the respective service gives you information about the addresses and ports the service uses for media delivery. Microsoft does, I’m not sure about the rest.

Upload varies too. And pushing back only works for traffic that takes care, so TCP and those protocols that use UDP as transport but use some kind of feedback on application level (such as QUIC). So the only way how to stay more or less safe is to consider the worst bandwidth per direction.
for upload, it is indeed enough to give priority to the “broadcasting” (in networking, that has a distinct meaning, you probably actually want to stream online to some distribution server) and VoIP/online conferencing, and to limit the total bandwidth to the worst case one
for download, you have to see what margin you’ll have to leave for traffic that doesn’t care about pushback, subtract that from the minimum download bandwidth, and set the result as the max-limit for a common queue for the VoIP and the “throttlable” traffic.

Although marking of connections is only necessary for some scenarios of load distribution, it usually helps to be ready for the worst. So here we go - since each connection can have at most one connection mark, you must be ready to use “combined” connection marks, expressing both the WAN to be used (by means of translating that connection-mark to a routing-mark) and the QoS class (by means of translating that connection-mark to a packet-mark).

The advantage of bypass mode on the Starlink is rather the availability of IPv6 than the ability to obtain a unique address; for me, a more important aspect would be the design quality of the add-on Ethernet adaptor if you happen to have a Gen 2 router. Experience has shown that if long-term reliability is your concern, is is necessary to connect to the Starlink routers wirelessly (using a different frequency band for each terminal, as a single wireless interface cannot be a client of two APs simultaneously) or to remove the router from the setup completely and replace it by a connector adaptor, a PoE injector with a quite unusual pinout, and a DC power adaptor.

Also, don’t be surprised if the hAP ac² turns out to be the bottleneck in the overall setup. With so much mangling, its overall throughput may be significantly limited as compared to the lightweight home use scenario.

Larsa · May 26, 2024, 3:48pm

I think IPv6 is a good way to go, but since there are still plenty of pure IPv4 apps you probably want to run, I’d consider adding dual stack support, i.e. also add ipv4.

Regarding queues, I might have missed something, but I still think CAKE is particularly well-suited for use with Starlink to handle things like fairness, bufferbloat, and dynamically managing queue lengths, etc. It might be a bit more complex to set up, but I think it’s worth trying. Once you’ve got everything up and running, you rarely need to bother with fine-tuning things, as most of it is handled automatically, which is the whole idea behind CAKE. Performance-wise, I’m pretty confident it won’t be any problem with a quad-core hAP ac2, whatever you choose.

But to get started quickly, @Sindy showed some great examples using device queues.

sindy · May 26, 2024, 6:57pm

I tried to understand whether CAKE is the right way when it comes to the specific requirements of prioritizing VoIP and streaming over anything else, and the following contrast made me laugh:
https://www.bufferbloat.net/projects/codel/wiki/CakeTechnical says: Cake’s fundamental design goal is perceived simplicity.
https://help.mikrotik.com/docs/display/ROS/Queue+types says: CAKE (Common Applications Kept Enhanced) … Disadvantages: More complex to configure than simpler queuing disciplines. Not ideal for all use cases, particularly where fine-tuned control is required.

On a serious note, what I’ve understood from the manual is that if using CAKE, you don’t need to (or rather must not) deal with one leaf queue (selected using a corresponding packet mark) for each priority class; instead, you set the DSCP headers to tell the single common CAKE queue which “tin” to put the packet to.

This post seems to have it all, except the way to handle the WAN->LAN traffic independently for each source WAN; there, the packet marking remains necessary in addition to the DSCP marking. What makes me more cautious is the fact that in the OP’s requirements outline multiple LAN interfaces but the WAN download bandwidth is (naturally) common for all of them, which does not play well with the need to use the interface queues as the parent ones. So I wonder whether it would make sense to use a hairpin tunnel to overcome this conflict of requirements.

daves · May 31, 2024, 10:16am

Thanks much for the responses.
FYI: I’m a volunteer network admin for the summer camp I’m doing this for. My primary work is programming and devops.
I’m fairly new to Mikrotik. We used PFSense for many years.

I have been making a proof of concept on a basic test router RB941-2nD. It is on v6.
I have the VLANs and PCC setup. They seem to be working.
I was tripped up for a while on routing not working without specifying the gateway IP and interface. Might be a v6 thing. Tutorials said to just add the appropriate interface but that didn’t work.
gateway=192.168.68.1%ether1_WAN1,192.168.68.1%ether2_WAN2
Upgrade to V7 is recommended to have 64MB RAM. My test device has 32MB.
I have a HEx or HAp ac2 for the production deploy.
We got the 2 starlinks not for redundancy but to dedicate one to the broadcasting, VOIP, and working from “home”. I dedicated one port to the guest VLAN connecting the 2nd starlink to it. Not using mikrotik for DHCP, routing, etc for Guest VLAN.

I was looking into CAKE per your recommendations.
Seems the CPU load may be a problem per discussion on topic:
Which mikrotik for 1Gbps WAN, SOHO, and queue enabled (fqcodel / cake)
If CAKE and/or other QAS doesn’t work I figure I can use the connection mark the VOIP and Broadcast VLANs and specific destinations, like MS Teams.

If CPU is a problem, maybe I can apply CAKE to just one WAN and direct the VOIP, broadcast, work-from-home stuff to that WAN.

I prefer not to use bypass mode in Starlink so I can use the starlink app to monitor and troubleshoot.
I’m pretty new to Mikrotik and have not used IPV6. I don’t understand the benefit to using IPV6 or if there would be problems.

I’m thinking to set it up when I use the Starlink app and connect to the staff vlan it directs to wan1_conn and when I connect to base vlan it uses wan2_conn. I’ve put in the rules but I don’t have the starlinks available for my test setup.
For the tests, I’m just connecting the two WAN ports the the LAN ports of another router. That provide the same gateway but different IPs assigned to the WAN interfaces.

I’m not clear on the routing table failover but seems failover is working as I unplugged the wan ports one at a time while running ping from mikrotik and a computer.

I was thinking about automating adjustment of PCC mangle rules if a connection became unavailable. Not really practical for just two WAN connections. But I love automating things like that. So just including info here for posterity.
How to auto-start a script at interface link up / down ?
Archived sample scripts as a starting point

Here is the config I have so far.

# may/30/2024 23:13:13 by RouterOS 6.49.15
# software id = H4V8-DQ6B
#
# model = RB941-2nD
/interface bridge
add name=BR1 protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] name=ether1_WAN1
set [ find default-name=ether2 ] name=ether2_WAN2
set [ find default-name=ether3 ] name=ether3_Trunk
set [ find default-name=ether4 ] name=ether4_Guest
/interface pwr-line
set [ find default-name=pwr-line1 ] disabled=yes
/interface vlan
add interface=BR1 name=VLAN10_Guest vlan-id=10
add interface=BR1 name=VLAN20_Staff vlan-id=20
add interface=BR1 name=VLAN50_BASE vlan-id=50
add interface=BR1 name=VLAN60_VOIP vlan-id=60
add interface=BR1 name=VLAN70_Broadcast vlan-id=70
/interface list
add name=WAN
add name=VLAN
add name=Manage

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa2-psk mode=dynamic-keys name=Staff_WiFi_Sec supplicant-identity=MikroTik
add authentication-types=wpa2-psk mode=dynamic-keys name=Guest_WiFi_Sec supplicant-identity=MikroTik
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-onlyn channel-width=20/40mhz-XX country="united states" disabled=no frequency=\
    auto mode=ap-bridge name=wlan1_Staff security-profile=Staff_WiFi_Sec ssid=Staff wps-mode=push-button-virtual-only
add disabled=no keepalive-frames=disabled mac-address=0A:55:31:4A:BB:61 master-interface=wlan1_Staff multicast-buffering=\
    disabled name=wlan2_Guest security-profile=Guest_WiFi_Sec ssid=Redwood wds-cost-range=0 wds-default-cost=0 wps-mode=\
    disabled

/ip pool
add name=V10_Guest_Pool ranges=10.10.1.1-10.10.254.255
add name=V20_Staff_Pool ranges=10.20.1.1-10.20.254.255
add name=V50_BASE_Pool ranges=192.168.50.100-192.168.50.199
add name=V60_VOIP_Pool ranges=192.168.60.2-192.168.60.254
add name=V70_Broadcast_Pool ranges=192.168.70.2-192.168.70.254
/ip dhcp-server
add address-pool=V10_Guest_Pool disabled=no interface=VLAN10_Guest name=V10_Guest_DHCP
add address-pool=V20_Staff_Pool disabled=no interface=VLAN20_Staff name=V20_Staff_DHCP
add address-pool=V50_BASE_Pool disabled=no interface=VLAN50_BASE name=V50_BASE_DHCP
add address-pool=V60_VOIP_Pool disabled=no interface=VLAN60_VOIP name=V60_VOIP_DHCP
add address-pool=V70_Broadcast_Pool disabled=no interface=VLAN70_Broadcast name=V70_Broadcast_DHCP

# Log interface changes to periodically run a script that checks and adjusts PCC to handle wan availability changes.
# TODO write script. Starting resources:
#   https://web.archive.org/web/20210416175553/https://wiki.mikrotik.com/wiki/Log_Parser_-_Event_Trigger_Script
#   http://forum.mikrotik.com/t/how-to-auto-start-a-script-at-interface-link-up-down/139637/1
/system logging action
add name=InterfaceUpDown target=memory
/system logging
add topics=interface

/user group
set full policy=\
    local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp

/interface bridge port
add bridge=BR1 frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes interface=ether4_Guest
add bridge=BR1 ingress-filtering=yes interface=ether3_Trunk pvid=50
add bridge=BR1 interface=wlan1_Staff pvid=20
add bridge=BR1 interface=wlan2_Guest pvid=10
/interface bridge vlan
add bridge=BR1 tagged=BR1 untagged=VLAN50_BASE vlan-ids=50
add bridge=BR1 tagged=BR1,ether3_Trunk untagged=wlan1_Staff vlan-ids=20
add bridge=BR1 tagged=BR1,ether3_Trunk untagged=ether4_Guest,wlan2_Guest vlan-ids=10
add bridge=BR1 tagged=BR1,ether3_Trunk vlan-ids=60,70

/interface list member
add interface=ether1_WAN1 list=WAN
add interface=ether2_WAN2 list=WAN
add interface=VLAN50_BASE list=VLAN
add interface=VLAN10_Guest list=VLAN
add interface=VLAN20_Staff list=VLAN
add interface=VLAN50_BASE list=Manage
add interface=VLAN20_Staff list=Manage

/ip dhcp-client
add add-default-route=no disabled=no interface=ether1_WAN1
add add-default-route=no disabled=no interface=ether2_WAN2
/ip address
add address=10.10.0.1/16 interface=VLAN10_Guest network=10.10.0.0
add address=10.20.0.1/16 interface=VLAN20_Staff network=10.20.0.0
add address=192.168.50.1/24 interface=VLAN50_BASE network=192.168.50.0
add address=192.168.60.1/24 interface=VLAN60_VOIP network=192.168.60.0
add address=192.168.70.1/24 interface=VLAN70_Broadcast network=192.168.70.0
/ip dhcp-server network
add address=10.10.0.0/16 dns-server=10.10.0.1 gateway=10.10.0.1
add address=10.20.0.0/16 dns-server=10.20.0.1 gateway=10.20.0.1
add address=192.168.10.0/24 gateway=192.168.10.1
add address=192.168.50.0/24 dns-server=192.168.50.1 gateway=192.168.50.1
add address=192.168.60.0/24 gateway=192.168.60.1
add address=192.168.70.0/24 gateway=192.168.70.1
/ip dns
set allow-remote-requests=yes servers=8.8.4.4,9.9.9.9,8.8.8.8,1.1.1.1

# Define local networks for mangle rules so we only do PCC on internet bound traffic.
/ip firewall address-list
add address=192.168.50.0/24 list=Local_Networks
add address=192.168.60.0/24 list=Local_Networks
add address=192.168.70.0/24 list=Local_Networks
add address=10.10.0.0/16 list=Local_Networks
add address=10.20.0.0/16 list=Local_Networks
add address=192.168.1.1 list=StarlinkDevIPs
add address=192.168.100.1 list=StarlinkDevIPs


/ip firewall filter
add comment="Allow Estab & Related" \
    chain=input action=accept connection-state=established,related
add comment="Allow VLAN" \
    chain=input action=accept in-interface-list=VLAN
add chain=input action=drop comment=Drop
add comment="Allow Estab & Related" \
    chain=forward action=accept connection-state=established,related
add comment="Allow Staff & BASE VLANs Access other VLANs" \
    chain=forward action=accept in-interface-list=Manage
add comment="VLAN Internet Access only" \
    chain=forward action=accept connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=drop comment=Drop

# PCC for dual WAN.
# For efficiency on new connection jump to more complex checks.
/ip firewall mangle

add comment="From External (off Mikrotik, LAN & WAN)" \
    chain=prerouting action=jump connection-state=new jump-target=new_extrnal

add comment="From WANs" \
    chain=new_extrnal action=mark-connection connection-mark=no-mark in-interface=ether1_WAN1 new-connection-mark=WAN1_conn passthrough=yes

add comment="Starlink APP Per VLAN"\
    chain=new_extrnal action=mark-connection connection-mark=no-mark dst-address-list=StarlinkDevIPs in-interface=VLAN20_Staff new-connection-mark=WAN1_conn passthrough=yes
add chain=new_extrnal action=mark-connection connection-mark=no-mark dst-address-list=StarlinkDevIPs in-interface=VLAN50_BASE new-connection-mark=WAN2_conn passthrough=yes

add comment="From LAN" \
    chain=new_extrnal action=jump connection-mark=no-mark jump-target=masq_load_balance

add comment="To/From Local" \
    chain=new_extrnal action=mark-connection connection-mark=no-mark new-connection-mark=Local_conn passthrough=yes

add comment="From Mikrotik" \
    chain=output action=jump connection-state=new jump-target=masq_load_balance

add comment="PCC LoadBalance" \
    chain=masq_load_balance action=mark-connection connection-mark=no-mark dst-address-list=\
    !Local_Networks new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add chain=masq_load_balance action=mark-connection connection-mark=no-mark dst-address-list=!Local_Networks \
    new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1

add comment="Mark Route Table" \
    chain=output action=mark-routing connection-mark=WAN1_conn new-routing-mark=WAN1_route passthrough=yes
add chain=output action=mark-routing connection-mark=WAN2_conn new-routing-mark=WAN2_route passthrough=yes

/ip firewall nat
add comment="Default masquerade" \
    chain=srcnat action=masquerade out-interface-list=WAN

#Gateway requires both gateway IP and interface (at least for firmware v6)
/ip route
add check-gateway=ping distance=1 gateway=192.168.68.1%ether1_WAN1 routing-mark=WAN1_route
add check-gateway=ping distance=1 gateway=192.168.68.1%ether2_WAN2 routing-mark=WAN2_route
add check-gateway=ping distance=2 gateway=192.168.68.1%ether1_WAN1,192.168.68.1%ether2_WAN2
/system clock
set time-zone-name=America/Los_Angeles