Yes, *but* whatever you are using for wireless needs to be able to read that. If your wireless is MikroTik, and you are doing "set priority" on the radio itself, it will work as long as you are using NV2 or WMM. If the wireless radio is not MikroTik, it will need to support QoS. Most wireless radios support QoS using CoS (vlan priority) which "set priority" on the router also sets. So if you are using a non-MikroTik radio your router will need to set priority with a mangle rule and tag the packet with a VLAN tag in order for the radio's QoS to kick in and prioritize the packet.So for all our routers just add a rule at the top of mangle with passthrough ticked
new priority: from dscp
And that's all thats needed? (Assuming DSCP is already set, otherwise add more mangle rules to set DSCP bits)
No queue's added?
Well, when the radio is UBNT (quite common as they operate in the same market segment as MikroTik), the whole QoS thing will work automatically, also without VLAN tagging.If the wireless radio is not MikroTik, it will need to support QoS. Most wireless radios support QoS using CoS (vlan priority) which "set priority" on the router also sets. So if you are using a non-MikroTik radio your router will need to set priority with a mangle rule and tag the packet with a VLAN tag in order for the radio's QoS to kick in and prioritize the packet.
This is not true for all of their radios. We have mostly AirFiber and they do not read DSCP so we use the VLAN priority. Even their radios that do read DSCP (AirMax, AirMax ac, LTU) are unable to read the DSCP if there are extra L3 headers on the packet besides just the IP header (ex. PPPoE frame, MPLS label). Those same radios will however also read VLAN priority so it remains the most reliable way to do QoS that works across the Ubiquiti line. I do not like the Ubiquiti radios that read DSCP without being able to disable that feature (and have no QoS config options) because in a PtMP setup there is the potential for a single end customer to abuse that by tagging all of their packets as priority to get better rates in a congestion situation. With VLAN priority if the radio is adding the VLAN tag, the customer has no control over the priority set in the tag and therefore cannot use that to prop up their service to the detriment of other customers.Well, when the radio is UBNT (quite common as they operate in the same market segment as MikroTik), the whole QoS thing will work automatically, also without VLAN tagging.
It uses the WMM defined queue mapping based on DSCP high 3 bits with 4 queues.
You don't need use-ip-firewall=yes if you set priority using a bridge filter. I prefer working with bridge filters to set priority with MikroTik wireless bridges instead of "use IP firewall" for performance reasons - if I am already using firewall rules for other purposes on the same device, I shouldn't need all bridged wireless traffic to pass through those same rules unnecessarily.- setting wmm-support=enabled on the wireless interface, likely you also want ampdu-priorities=0,1,2
- adding the mangle rule to set priority based on DSCP high 3 bits
- when it is operating in bridge mode: adding the use-ip-firewall=yes setting to the global bridge configuration
We extend the VLAN over the WiFi link - we have hundreds of such setups and haven't had problems.With the UBNT devices, do you use VLAN tagging only on ethernet and then strip it in the radio, or extend VLAN all over the WiFi link?
I have not-so-good experience with the latter when it is not in PtP mode. Sometimes it works fine, sometimes it fails in strange ways.
The problem with using "bridge filter" for setting the priority is there is no "set priority from DSCP high 3 bits" option so it requires custom matching and possibly many rules.
This doesn't help when customers abuse with UDP on upload for traffic that has no kind of congestion control mechanism. We would have to use the traffic shaping settings on the Ubiquiti radio which is awkward to manage. Even if you did that, the problem would still be that latency for other customers is higher because the one abusive customer is marking everything with DSCP EF or whatever instead of just actual VoIP traffic.Problems with customers abusing DSCP (fortunately I don't have that experience... my customers are not primarily wanting performance) could be overcome with a queue tree with some carefully selected limit-at and max-limit values, so it is not possible to monopolize the link from a single DSCP level. That is also the reason I like queue trees more than priority queuing. I have queue trees in the routers and priority queues in the APs.
Ok, my experience with UBNT radios that are not in "WDS" mode has been that tagged VLAN traffic over the link does not always work correctly.We extend the VLAN over the WiFi link - we have hundreds of such setups and haven't had problems.With the UBNT devices, do you use VLAN tagging only on ethernet and then strip it in the radio, or extend VLAN all over the WiFi link?
I have not-so-good experience with the latter when it is not in PtP mode. Sometimes it works fine, sometimes it fails in strange ways.
Ok that is a nice setup, I could try that. With MikroTik AP I normally run the user traffic over a VLAN on ethernet, with UBNT AP usually not (I feel less comfortable with separating the management and data on those, fearing to lock myself out at some time).Have the router set priority from dscp high 3 bits with a mangle rule and apply the VLAN tag. Then on the radio (that is doing the bridging), have the bridge filter rule set priority to ingress priority - only one rule needed. This will result in the radio prioritization being read from the VLAN priority of the incoming frame, and the VLAN priority was in turn set by the preceding router based on DSCP. This is how we do it and it works great. No need for "Use IP firewall" anywhere, and no need for custom matching and many rules.
Fortunately until now it has not happened. In our network, there are not really "customers", we are all part of a co-operating club.This doesn't help when customers abuse with UDP on upload for traffic that has no kind of congestion control mechanism. We would have to use the traffic shaping settings on the Ubiquiti radio which is awkward to manage. Even if you did that, the problem would still be that latency for other customers is higher because the one abusive customer is marking everything with DSCP EF or whatever instead of just actual VoIP traffic.
Ok, my experience with UBNT radios that are not in "WDS" mode has been that tagged VLAN traffic over the link does not always work correctly.
As we have a mix of UBNT/MikroTik in het network (both at the AP and client side) we often cannot run in "WDS" mode.
This should be the same difference as between "bridge" and "pseudobridge" mode in MikroTik but still they do not appear to be compatible.
(I think the operation of the non-WDS mode relies on ARP spoofing which does not work for VLAN tagged traffic across the link)
In our case the UBNT APs have WDS enabled and the MikroTik client connected is in "station WDS" mode and it works. I have not tried the other way.The other way around, UBNT AP no WDS with MikroTik client connected can often pass VLAN tagged traffic but sometimes it stops and needs to be re-associated to continue.
Of course between 2 UBNT devices in WDS mode, and between 2 MikroTik devices in bridge mode there is no issue.
We have a similar setup to you only we don't have Cambium - we only have Ubiquiti and MikroTik. We have a central PPPoE concentrator as well with MPLS.Ok so i'm a bit confused as to which method to use here. So lets step it back and i'll give a couple of different scenario's that may need different methods
We primarily use Cambium radio's but do use some Ubiquiti and a few Mikrotik
I'm going to talk about our backbone infrastructure and ignore customer facing radio's, as all of our customer radio links exceed their provisioned bandwidth so theoretically shouldn't need QoS on those links. All customers connect with PPPoE back to a central PPPoE concentrator. I'm strongly considering changing this design to instead terminate the PPPoE session as close to the customer as possible because it solves some other issues, QoS should be one of them (as I know some radio's won't read QoS tags if there is a PPPoE header)
It is hard to do QoS for PPPoE traffic certainly. We do QoS for PPPoE sometimes over VPLS but it is an all-or-nothing situation - either we prioritize all of a customer's packets or none of them. You would need to "set priority" on the traffic before it has the MPLS label applied, then the priority would be carried across the network via MPLS experimental bits all the way to the destination. Do you really need to run VoIP over PPPoE though? Why not put the VoIP on a different VLAN? Assuming you have central VoIP servers, you only need a few firewall rules to allow the traffic to the VoIP servers and vice versa. The phones would not need complete access to the Internet, so it shouldn't require too many firewall rules on the side of the other routers. Then you can tag just the VoIP packets quite easily - either they could run over a different VPLS tunnel (and then you could prioritize everything in the VPLS tunnel with EXP bits) or they could be locally routed. We are planning to roll out VoIP for our customers in the next year and are not planning to run the VoIP over the PPPoE tunnel.First Scenario:
So our backbone wireless links are all run as a L2 bridge. MikroTik routers on each side do all the routing
At the moment due to the central PPPoE concentrator. Customer connections all come in on a VLAN and are bridged to a VPLS interface that goes back to the concentrator. So I actually don't know if we can QoS their traffic between these 2 end points because it's all encapsulated in VPLS then in PPPoE, so tags assigned by the customer router are most likely going to be ignored? no way to prioritize their VoIP over data through our backbone right? This is another reason i'd like to move PPPoE termination closer to the customer, cause then their traffic can transit our backbone as normal IP traffic, no need for VPLS or PPPoE running through the backbone, we can mark traffic as close to the customer as possible
So the question is, with our current setup (PPPoE over VPLS) does infrastructure QoS sit in the 'too hard/not viable' basket and so we shouldn't bother with it and instead wait until moving away from PPPoE?
Yes - tag all packets from router to router where they go over a radio link. VLAN priority should be your 1 solution fits all QoS solution. For packets that are not MPLS, you can use set priority on the routers when the VLAN tag is applied and all radios can read the VLAN priority. MikroTik will copy MPLS EXP bits to VLAN priority so if you set priority on your MPLS packets before the label is applied, the VLAN priority will be set when that packet has a VLAN tag added. This transfer of VLAN priority is supposed to be fully automatic, but I found that this only works out of the box with CHR (cloud hosted router). With hardware routers I needed extra steps - I had to create extra single-port bridges on any ports that an MPLS packet can arrive or leave on with STP turned off and a bridge filter rule (chain=output action=passthrough ingress-priority=!0) in order for the automatic setting of VLAN priority from MPLS EXP bits to work. I believe it is due to a bug in the MPLS fastpath handler, and there is no config option to turn off MPLS fastpath. Although adding these extra bridges and the bridge filter is annoying, it works around the issue so that QoS works. I think what happens is MPLS fastpath is switched off by adding this config, which allows the automatic copying of MPLS EXP bits to VLAN priority to work as originally intended.Second Scenario
Lets ignore the customer traffic and focus on 'our' traffic (SNMP, ICMP, OSPF/BGP/MPLS, Keepalives etc between routers and for monitoring)
Currently we don't use a separate VLAN for router-router connections so we can't set VLAN priority. But that doesn't mean we can't change this design if its beneficial
As above we have different equipment connecting these routers together, Cambium PMP/ePMP/PTP series, Ubiquiti Airfiber and some MikroTik 60ghz. And we occasionally change out equipment. So it would be nice if we can have a 1 solution fits all QoS implementation so we don't have to make further changes in each router if we are replacing the radio link
You will need to configure your router-router links to use VLANs by placing VLAN interfaces on the ether5 (RouterA) and ether7 (RouterB) ports. The router needs to apply a VLAN tag in order to have somewhere to store the priority.The router-router links don't use VLAN's though
They just speak to each other on the ethernet link i.e. ether5 on RouterA connects to PTP670 link connects to ether7 on RouterB
So using the set priority mangle rule wouldn't do anything? Or would it still tag packets with native VLAN id so that priority is applied?
The router will not do anything with DSCP tagged packets by default.Ok but I have heard its best practice to use QoS tags at Layer3 as opposed to Layer2 so why not use DSCP tags instead of CoS?
And does a MikroTik router actually do anything with DSCP tagged packets by default or does it need to configured with mangle or queue's to apply prioritization to traffic?
/ip firewall mangle add action=jump chain=postrouting jump-target=QoS add action=change-dscp chain=QoS comment="QoS: 56 - VRRP" new-dscp=56 passthrough=yes protocol=vrrp add action=change-dscp chain=QoS comment="QoS: 56 - OSPF" new-dscp=56 passthrough=yes protocol=ospf add action=change-dscp chain=QoS comment="QoS: 56 - BFD" dst-port=3784,3785 new-dscp=56 passthrough=yes protocol=udp add action=change-dscp chain=QoS comment="QoS: 56 - MPLS LDP" dst-port=646 new-dscp=56 passthrough=yes protocol=udp add action=change-dscp chain=QoS comment="QoS: 46 - VoIP" dst-address-list=KnownVoIPServers new-dscp=46 passthrough=yes add action=change-dscp chain=QoS comment="QoS: 43 - BGP" dst-port=179 new-dscp=43 passthrough=yes protocol=tcp add action=change-dscp chain=QoS comment="QoS: 43 - ICMP" new-dscp=43 passthrough=yes protocol=icmp add action=set-priority chain=QoS dscp=!0 new-priority=from-dscp-high-3-bits passthrough=yes add action=return chain=QoS
Answer to First question: No we aren't. One thing you need to realize is that, at least on MikroTik routers, ethernet interfaces do not consider the packet priority at all when deciding what packets to drop - only NV2 and WMM wireless interfaces do this. Changing the hardware queue type does not change this behavior. I generally do not bother doing QoS on two routers cabled together because if it is a cable and it is at 1G and it is maxed out, you can always bring it up to 10gig by moving to an SFP+ port or increase it by combining two links in a LAG pair, and remove the possibility of congestion.First question: Are you changing the hardware queue type on the MikroTik's? What are you using and what settings?
Second question: Are you using a common template for QoS settings and would you care to share it?
This is the primary reason for this post. We want better QoS for backhaul wireless links that we own, and the bandwidth varies it cannot be guaranteed. Real world is not perfect, radio frequencies get crowded, new constructions go up and partially block signal, a bin chicken fly's into the radio and knocks it out of alignment etcWe only do this queue tree setup on links from 3rd party connectivity vendors where they guarantee us a certain bandwidth amount where we are at risk of actually maxing out that amount. It doesn't make sense to set up these queue trees and packet marks if the router is only connected to radio links and perhaps direct cables to other routers, because the radios will automatically do the QoS based on CoS (and the maximum rate can be variable depending on modulation), and the cables to other routers generally won't be maxed out so the queueing becomes an unnecessary complexity with those links.
Are you sure? Because my tests with bfifo show packet loss on the saturated link drops from 30% to 0%. My thoughts are that if it's not dropping low priority packets (since I don't think bfifo/pfifo does any reordering) then packet loss should remain at 30% if it's sustained saturation, since the bucket would remain full. But this doesn't seem to be the caseOne thing you need to realize is that, at least on MikroTik routers, ethernet interfaces do not consider the packet priority at all when deciding what packets to drop - only NV2 and WMM wireless interfaces do this. Changing the hardware queue type does not change this behavior
Yes, that is all we are doing. Ubiquiti uses CoS by default, no config changes needed, dropping lower priority packets if necessary to make room for high priority. MikroTik radios need an extra bridge filter rule, chain=forward ingress-priority=!0 action=set-priority new-priority=from-ingress, and then it drops lower priority packets over the radio link too. Cambium probably uses it by default, but you should check to see if there is a setting that needs to be turned on, just in case.This is the primary reason for this post. We want better QoS for backhaul wireless links that we own, and the bandwidth varies it cannot be guaranteed. Real world is not perfect, radio frequencies get crowded, new constructions go up and partially block signal, a bin chicken fly's into the radio and knocks it out of alignment etc
So for this you are just using mangle rules to set L2 CoS/VLAN priority from L3 DSCP values?
I think the only way that you could handle a cable issue dropping an ethernet port to 100Mbps is to do the aforementioned queue tree setup (8 child queues, 1 parent) and use a script to automate lowering the parent queue limit to 100M in the event of the ethernet link dropping to 100Mbps.Most radio links will be below the throughput of ethernet, but 60ghz and 80ghz can exceed 1gbit. And/or it is possible for a cable issue to drop the connection from 1gbit to 100mbit. Both scenario's result in the ethernet being the bottleneck not the radio. So i'm thinking it may be worthwhile using SFQ on those. Or have you found that it really doesn't matter?
Yes - I think the problem is you are doing your testing with btest UDP. This does not really flood the link. Use a specialized UDP packet flooder, they are used to simulate DDoS situations. We do our QoS testing in DDoS simulation situations. In these situations the wireless radios will not drop higher priority packets (they will drop lower priority as necessary in congestion situations).Are you sure? Because my tests with bfifo show packet loss on the saturated link drops from 30% to 0%. My thoughts are that if it's not dropping low priority packets (since I don't think bfifo/pfifo does any reordering) then packet loss should remain at 30% if it's sustained saturation, since the bucket would remain full. But this doesn't seem to be the case